15:30:03 <DinaBelova> #startmeeting Performance Team
15:30:03 <openstack> Meeting started Tue Mar 14 15:30:03 2017 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:07 <openstack> The meeting name has been set to 'performance_team'
15:30:13 <tovin07_> hi o/
15:30:21 <DinaBelova> hello folks!
15:30:22 <ronana> o/
15:30:23 <DinaBelova> let's wait for a few moments :)
15:30:30 <DinaBelova> ronana tovin07_ o/
15:30:37 <DinaBelova> ronana new faces :)
15:30:45 <rcherrueau> :)
15:30:48 <DinaBelova> rcherrueau heh :)
15:30:49 <DinaBelova> non new :D
15:31:43 <DinaBelova> akrzos o/
15:33:11 <DinaBelova> ok, so let's get started
15:33:17 <DinaBelova> #topic Action Items
15:33:43 <DinaBelova> the only action item we had was on myself regarding asking Vipin about his future participance in OSprofiler
15:33:59 <DinaBelova> sadly I had really hard weeks, so I did not manage to do this
15:34:21 <DinaBelova> #action DinaBelova ping Vipin on his participance in OSprofiler - he seemed not to be really active last few weeks, need to check if he's going to continue work
15:34:21 <DinaBelova> I'll keep this till next week
15:34:29 <DinaBelova> so we can jump to the current progress
15:34:30 <DinaBelova> #topic Current progress on the planned tests
15:34:34 <DinaBelova> rcherrueau ?
15:34:52 <tovin07_> DinaBelova: :D
15:34:53 <rcherrueau> Nothing really new on our side
15:35:17 <DinaBelova> rcherrueau ok :) just usual work and smooth progress I guess
15:35:28 <tovin07_> last week, I had some disscusions with rcherrueau about kolla
15:35:46 <rcherrueau> We started some test in a WAN environment
15:35:57 <akrzos> DinaBelova:  o/
15:36:03 <DinaBelova> tovin07_ indeed. rcherrueau afair tovin07_ has proposed to push BP to kolla regarding osprofiler support
15:36:25 <rcherrueau> there is a WAN connection between keystone and an OpenStack
15:36:44 <DinaBelova> rcherrueau ok, so we should expect some results soon :)
15:37:05 <rcherrueau> yes
15:37:16 <msimonin> all o/
15:37:29 <DinaBelova> rcherrueau ok, thank you. What's about pushing new Bp to kolla about osprofiler support?
15:37:31 <DinaBelova> msimonin o/
15:37:46 <rcherrueau> hey msimonin, can you give some info about the WAN experiment?
15:38:01 <rcherrueau> such as the latency
15:38:02 <msimonin> sure
15:38:31 <msimonin> First of all did you know that the summit talk is accepted ?
15:38:32 <DinaBelova> rcherrueau msimonin btw congrats on your presentation choosing for the summit
15:38:38 <DinaBelova> ahaha
15:38:43 <msimonin> heh :)
15:38:45 <DinaBelova> I'm just in time with my congrats
15:38:48 <rcherrueau> DinaBelova: thx
15:38:53 <DinaBelova> #link https://www.openstack.org/summit/boston-2017/summit-schedule/events/17952/toward-fog-edge-and-nfv-deployments-evaluating-openstack-wanwide?BackURL=https%3A%2F%2Fwww.openstack.org%2Fsummit%2Fboston-2017%2Fsummit-schedule%2Fspeakers%2F11238%23eventId%3D17952
15:39:41 <msimonin> I guess the talk will be on the methodology we use and some results
15:40:32 <tovin07_> congrats rcherrueau
15:40:43 <DinaBelova> msimonin any details on the experiment for now?
15:40:50 <rcherrueau> tovin07_: thx
15:40:59 <DinaBelova> or do you think some more time is needed for the sharing?
15:41:13 <msimonin> about the experiments we are using  some of the rally  scenarios with various latency/bandwidth constraints
15:41:33 <msimonin> and actually
15:42:25 <DinaBelova> msimonin yes sir?
15:42:30 <msimonin> the granularity of the results (I mean the timing of the steps) is to big
15:42:50 <msimonin> so we'll use osprofiler + rally to get more fine grained results
15:43:08 <DinaBelova> msimonin gotcha
15:43:24 <DinaBelova> msimonin ack, so I think we need to wait for some results :) hope to see them soon :)
15:43:32 <msimonin> and extract what operations/functions suffer of high latency
15:43:47 <rcherrueau> DinaBelova: on that point of osprofiler + rally, some students here change the rally client to an `--os-profile YOUR_KEY`
15:44:05 <rcherrueau> DinaBelova: to add*
15:44:26 <rcherrueau> so now we can trace a specific scenario :)
15:44:29 <tovin07_> if you use openstackclient, —os-profile is the option
15:44:46 <tovin07_> `—os-profile`
15:44:53 <msimonin> I think they changed in the rally code directly
15:45:04 <DinaBelova> msimonin I believe you're right
15:45:12 <rcherrueau> tovin07_: this an implementation of the `--os-profile` into the rally client
15:45:17 <tovin07_> oh
15:45:20 <tovin07_> got it
15:45:31 <DinaBelova> rcherrueau I believe not only to the client, but to the rally itself as well :)
15:45:34 <tovin07_> it’s strange with other projects
15:45:54 <DinaBelova> to pass the header further
15:46:21 <DinaBelova> rcherrueau msimonin anyway, it's a good activity
15:46:22 <DinaBelova> thanks for the news
15:46:37 <rcherrueau> DinaBelova: in fact, the change is really small and located into the cli part
15:46:52 <DinaBelova> #info work on adding osprofiler support to rally has been started by students in Inria to trace specific scenario
15:46:59 <msimonin> Could it be interesting to push this change to rally ?
15:47:22 <DinaBelova> msimonin I believe yes, but we need to ping Andrey Kurilin (Rally PTL)
15:47:35 <DinaBelova> msimonin can you please do this?
15:47:37 <rcherrueau> tovin07_: yes you are right, we did it like that because it suited our needs, but I'm not sure this makes sens for a PR for instance
15:47:40 <msimonin> sure
15:47:46 <msimonin> I can take the action
15:47:55 <tovin07_> yup
15:47:55 <DinaBelova> #action msimonin ping Rally PTL regarding adding osprofiler support to rally code base
15:48:01 <DinaBelova> msimonin thank you sir
15:48:09 <msimonin> DinaBelova: ack :)
15:48:21 <DinaBelova> msimonin rcherrueau anything else to share today?
15:48:41 <msimonin> maybe one
15:48:50 <rcherrueau> DinaBelova: just to be clear, the idea is to trace OS based on a scenario executed by, not rally itself
15:49:00 <DinaBelova> rcherrueau sure, I meant this :)
15:49:22 <msimonin> we'll add the support for a new provider to Enos : chameleon cloud
15:49:27 <DinaBelova> rcherrueau "osprofiler support" sounds confusing, I agree
15:49:28 <DinaBelova> :D
15:49:40 <msimonin> https://www.chameleoncloud.org/
15:49:51 <tovin07_> yes, too confusing :v
15:50:33 <DinaBelova> msimonin ack, so your tool is covering more and more :)
15:50:42 <msimonin> :)
15:51:19 <msimonin> I guess that's all
15:51:30 <DinaBelova> ok, akrzos any news?
15:51:48 <DinaBelova> btw several talks by RedHat is accepted as well if i'm not mistaken
15:51:49 <akrzos> So Ceilometer Collector appears to be my biggest bottleneck
15:51:54 <akrzos> Yes
15:51:57 <DinaBelova> #link https://www.openstack.org/summit/boston-2017/summit-schedule/events/17530/openstack-telemetry-and-the-10000-instances
15:52:12 <akrzos> Also one lightening talk for browbeat
15:52:20 <DinaBelova> #link https://www.openstack.org/summit/boston-2017/summit-schedule/events/18245/performance-and-scale-analysis-of-openstack-using-browbeat?BackURL=https%3A%2F%2Fwww.openstack.org%2Fsummit%2Fboston-2017%2Fsummit-schedule%2Fspeakers%2F9457%23eventId%3D18245
15:52:25 <DinaBelova> yeah, posted it as well
15:53:02 <DinaBelova> akrzos I did not go through most recent Ceilo architecture, but that has been an issue since Kilo at least
15:53:03 <akrzos> I added a collectd plugin for rabbitmq monitoring since ceilometer collector appears to be choking on how many messages it is trying to consume and push into Gnocchi backlog
15:53:12 <akrzos> also Gnocchi is not behaving well with Ceph
15:53:21 <akrzos> if it's not configured it will create many small objects
15:53:28 <akrzos> (16 byte objects)
15:53:41 <DinaBelova> akrzos so some recommendations are needed on how to tune Ceph, indeed
15:53:45 <akrzos> and can push other more important data out of caches because it can create so many
15:54:23 <akrzos> So still on the battle to get more instances, but we can pretty much blame the collector whcih should be gone by Pike
15:54:46 <akrzos> Also later this week I intend on pushing my rally code into a review
15:54:54 <akrzos> thats it for me
15:55:36 <DinaBelova> akrzos the issue with collector / rabbit looks like the issues we've observed with classical Ceilometer some time ago... I remember issue like "collector is slowly taking messages from rabbitmq (as the speed depends on writing speed to the backend, and it may be slow), therefore queue size is constantly growing and at some point rabbitmq crashes"
15:56:06 <akrzos> so the collector will grab too much off all at once
15:56:17 <akrzos> if the prefetch is not limited
15:56:21 <DinaBelova> yeah
15:56:25 <akrzos> so if you limit prefetch you can prevent that
15:56:34 <akrzos> but that doesn't stop it from being too slow
15:56:38 <DinaBelova> indeed
15:56:45 <akrzos> there is some settings surrounding the thread count and worker count
15:57:01 <akrzos> which could use a bit more playing with to see if it improves much
15:57:26 <DinaBelova> akrzos thanks for your update
15:58:21 <DinaBelova> from Mirantis side: we're currently stressing k8s API and ETCD with huge number of pods (on 450 nodes). At the same time we're trying to evaluate k8s benefits from LCM point of view - we've started working on upgrades testing of containerized OpenStack environment
15:58:37 <DinaBelova> sadly progress is slow due to several issues with the lab itself from hw perspective and from deployment perspective as well
15:58:56 <DinaBelova> so nothing to share so far
15:59:30 <DinaBelova> it looks like we're done with the progress update
15:59:42 <DinaBelova> so I believe we may jump to the open discussion
15:59:44 <DinaBelova> #topic Open Discussion
15:59:58 <tovin07_> from me
15:59:58 <tovin07_> continue to work on these patches
16:00:05 <tovin07_> #link OSprofiler in Ironic https://review.openstack.org/#/c/358142/
16:00:06 <tovin07_> #link OSprofiler in Murano https://blueprints.launchpad.net/murano/+spec/osprofiler-support-in-murano
16:00:06 <tovin07_> #link review OSprofiler in Zun https://review.openstack.org/#/c/444776/
16:00:21 <DinaBelova> #info also I propose to add tovin07_ to OSprofiler core team
16:00:22 <DinaBelova> tovin07_ ^^
16:00:35 <DinaBelova> I'll discuss this with Vipin, but I think he'll be really glad
16:00:39 <tovin07_> patches in Murano will be push this week
16:00:44 <tovin07_> DinaBelova:
16:00:51 <tovin07_> DinaBelova: thanks :D
16:00:54 <DinaBelova> tovin07_ good to know!
16:01:00 <tovin07_> i think that I’m ready for that
16:01:07 <DinaBelova> :D you should yeah :D
16:01:27 <tovin07_> hope to see it soon
16:01:29 <tovin07_> :D
16:01:36 <DinaBelova> tovin07_ ;)
16:01:48 <tovin07_> besides, just realize that we bump version of OSprofiler recently (current 1.7.0)
16:02:32 <DinaBelova> tovin07_ yeah, so it includes all nice things that were done about new drivers, etc.
16:02:41 <DinaBelova> and some of your improvements as well
16:02:52 <tovin07_> yes
16:03:25 <DinaBelova> tovin07_ anything else to share today?
16:03:36 <tovin07_> oh
16:03:58 <DinaBelova> tovin07_ it might be not, do not worry :)
16:03:59 <tovin07_> Boris said that he came back
16:04:27 <tovin07_> however, I did not see him around
16:04:30 <tovin07_> :D
16:04:59 <rcherrueau> tovin07_:, DinaBelova: I have a question about osprofiler, can I?
16:05:07 <tovin07_> yes sir
16:05:08 <DinaBelova> tovin07_ yeah, but upcoming summit rally update will be processed by him and Andrey together
16:05:10 <DinaBelova> #link https://www.openstack.org/summit/boston-2017/summit-schedule/events/18612/project-update-rally?BackURL=https%3A%2F%2Fwww.openstack.org%2Fsummit%2Fboston-2017%2Fsummit-schedule%2Fglobal-search%3Ft%3Dpavlovic%23eventId%3D18612
16:05:16 <DinaBelova> rcherrueau sure :)
16:05:36 <rcherrueau> tovin07_: In the *-stop part of a trace, is it possible to get the result of the function or sql evaluation?
16:05:38 <tovin07_> DinaBelova: yes, I see :D
16:06:30 <DinaBelova> rcherrueau hm, I believe it's possible, but it will require addtional developer work in osprofiler
16:06:37 <tovin07_> rcherrueau: I think yes
16:06:37 <DinaBelova> rcherrueau right now it's not possible
16:07:04 <DinaBelova> tovin07_ can you please evaluate this work? what needs to be done?
16:07:05 <tovin07_> however, current implementation mainly focus on request parameters and arguments (and kwargs)
16:07:22 <tovin07_> DinaBelova: sure
16:07:39 <DinaBelova> #action tovin07_ evaluate what needs to be done to add result of the function or sql evaluation in the *-stop part of osprofiler trace
16:07:46 <rcherrueau> tovin07_: yes you are right, but during debug it could be helpful to get the result also
16:07:57 <DinaBelova> rcherrueau yeah, it's good use case
16:08:00 <DinaBelova> thanks for sharing
16:08:07 <tovin07_> rcherrueau: yes
16:08:16 <tovin07_> thanks
16:08:21 <DinaBelova> let's try to get some answer next week
16:08:28 <tovin07_> (y)
16:08:32 <rcherrueau> great
16:08:58 <DinaBelova> msimonin akrzos anything else to share?
16:09:17 <akrzos> i'm good
16:09:33 <DinaBelova> ok, so thanks everyone for participating and see you next week!
16:09:37 <DinaBelova> #endmeeting