15:30:00 <DinaBelova> #startmeeting Performance Team
15:30:00 <openstack> Meeting started Tue Apr 11 15:30:00 2017 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:01 <openstackgerrit> Merged openstack/performance-docs master: Ceph RBD test plan  https://review.openstack.org/453600
15:30:03 <openstack> The meeting name has been set to 'performance_team'
15:30:07 <openstackgerrit> Merged openstack/performance-docs master: Kube-proxy performance test plan and report  https://review.openstack.org/452257
15:30:14 <DinaBelova> hello folks!
15:30:29 <DinaBelova> rcherrueau tovin07 akrzos o/
15:30:32 <rcherrueau> o/
15:30:35 <tovin07> o/
15:30:49 <akrzos> DinaBelova: o/
15:31:04 <DinaBelova> #info today I have only 30 minutes - have to run to the next meeting >_<
15:31:14 <DinaBelova> so let's get started
15:31:18 <DinaBelova> #topic Action Items
15:31:30 <DinaBelova> last time we had two action items, both on rcherrueau :)
15:32:00 <DinaBelova> I see you have no objections to tovin07 's commit to osprofiler
15:32:02 <rcherrueau> Yep. my first priority is finalising wan experiments, but I make a WIP test plan for OpenStack under WAN.
15:32:08 <DinaBelova> rcherrueau ack
15:32:13 <openstackgerrit> Merged openstack/performance-docs master: Add Fuel CCP upgrade testing  https://review.openstack.org/451419
15:32:16 <DinaBelova> so lemme keep this action item for now
15:32:19 <openstackgerrit> Merged openstack/performance-docs master: Fuel-CCP OpenStack reliability testing report  https://review.openstack.org/449087
15:32:20 <rcherrueau> #link https://github.com/rcherrueau/performance-docs/tree/openstack-wan
15:32:35 <rcherrueau> I will iterate on that at the same time I iterate on experiments.
15:32:44 <rcherrueau> You know, the protocol become more and more precise as I deep into experiments.
15:32:48 <DinaBelova> #action rcherrueau add OpenStack testing under networking delays (e.g. multisite deployment) methodology to performance docs (openstack under WAN)
15:32:51 <DinaBelova> rcherrueau true
15:33:00 <DinaBelova> #topic Current progress on the planned tests
15:33:01 <rcherrueau> Anyway, at some point, I expect to provide in the document the `tc` program that helps you set up traffic shaping.
15:33:17 <DinaBelova> rcherrueau I guess it makes sense for you to share current progress now :)
15:33:30 <rcherrueau> For the WAN experiments, we have the felling that high latency and traffic shaping will be a problem for REST call and oslo_messaging.
15:33:43 <rcherrueau> To investigate this, we need a good knowledge of API/RPC workflow and OSProfiler is a good friend here :)
15:33:52 <rcherrueau> So we decided to go a bit further and develop a tool that automatically generate a Sequence Diagram based on a Trace.
15:33:55 <DinaBelova> yay :)
15:33:58 <rcherrueau> #link https://github.com/rcherrueau/osp-utils
15:34:07 <rcherrueau> You can find an sequence diagram of `openstack server create ...` in test/rsc/out.svg
15:34:13 <rcherrueau> #link https://github.com/rcherrueau/osp-utils/blob/1af8c4e0a0c3e5f5722fe7cf33109eafb77c2c74/tests/rsc/out.svg
15:34:22 <rcherrueau> From now, it's not really readable. The diagram is really huge because of tons of calls to the DB.
15:34:36 <rcherrueau> But I'm writing a folding function that let you aggregate similar traces based on a predicate.
15:34:41 <rcherrueau> Maybe one last remark, the tool has been written in Haskell.
15:34:54 <rcherrueau> I know this is not OpenStack compliant, but I always say to my students: the right language for the right tool ;)
15:35:03 <DinaBelova> rcherrueau yeah, I noticed this already :)
15:35:05 <rcherrueau> And I was afraid to go with python regarding the huge size of traces
15:35:26 <rcherrueau> In addition, depending how fast I go on WAN experiments, I hope I can add a Query Language to filter traces.
15:35:36 <rcherrueau> For instance, to only keep RPC calls.
15:35:50 <rcherrueau> that's all for this week on my side.
15:35:58 <DinaBelova> rcherrueau ack, thanks
15:35:59 <DinaBelova> btw
15:36:25 <DinaBelova> if I'm not mistaken, tovin07, we were talking one time about adding filtration to the osprofiler report?
15:36:39 <DinaBelova> to filter rpc / db / other types of calls
15:36:52 <tovin07> yes
15:37:04 <tovin07> we were talking about that
15:37:08 <DinaBelova> tovin07 did you have a chance to look at this already?
15:37:11 <tovin07> a very long time ago
15:37:16 <DinaBelova> :D
15:37:29 <tovin07> maybe :D
15:37:37 <tovin07> I will investigate
15:37:45 <DinaBelova> okay, I just think that this thing becomes even more actual
15:37:49 <DinaBelova> thank you
15:37:56 <rcherrueau> tovin07: thanks
15:37:57 <tovin07> btw
15:38:19 <DinaBelova> tovin07 yes?
15:38:27 <tovin07> I think that’s a necessary feature
15:38:42 <DinaBelova> it is :)
15:38:45 <tovin07> sometimes, we’ve got alot of db trace :v
15:39:04 <rcherrueau> tovin07: too much db traces ...
15:39:12 <tovin07> yup
15:39:28 <DinaBelova> true :) OpenStack communicates with DB in an extensive manner :)
15:39:41 <tovin07> okay, we should move on to the next topic :D
15:39:49 <DinaBelova> ack :)
15:39:50 <DinaBelova> akrzos sir, how's your summit preparation is going on?
15:40:22 <akrzos> running telemetry with Ocata now
15:40:32 <akrzos> also removing the collector
15:40:46 <akrzos> i expect some solid results on this by this time next week
15:40:53 <akrzos> also
15:41:00 <akrzos> we identified a big issue with gnocchi and ceph
15:41:12 <akrzos> especially evident when gnocchi is untuned
15:41:17 <DinaBelova> akrzos oh, can you please share it?
15:41:38 <akrzos> a single object in ceph is used to hold a list of keys
15:41:52 <akrzos> each key is a small object in ceph for an unprocessed measure
15:41:59 <akrzos> so two issues
15:42:02 <akrzos> the list of keys
15:42:06 <akrzos> can grow too large
15:42:15 <akrzos> and this can cause ceph to fail
15:42:20 <akrzos> whatever your replica is
15:42:26 <akrzos> those osds will fail
15:42:33 <DinaBelova> oh, unpleasant :D
15:42:34 <akrzos> so if you have replica 3 on the metrics pool
15:42:38 <akrzos> 3 osds will go out
15:42:52 <akrzos> so this occurs when the backlog grows
15:43:10 <DinaBelova> okay, so this definitely looks as issues #1 and issue #2 ?
15:43:12 <akrzos> so if you don't have sufficent capacity to process all your measures
15:43:18 <akrzos> that can occur
15:43:21 <akrzos> issue #2
15:43:31 <akrzos> is ceph as a filestore is bad for small objects
15:43:45 <akrzos> resolution basiclly won't come until ceph bluestore is shipping
15:43:51 <akrzos> and integrated
15:44:10 <akrzos> so pretty much if you scale up and don't tune gnocchi
15:44:15 <akrzos> you hit ceph problems
15:44:21 <akrzos> if you tune gnocchi
15:44:25 <akrzos> and then scale up
15:44:25 <DinaBelova> yeah, I remember concerns about it when gnocchi was starting
15:44:31 <akrzos> you'll hit ceilometer collector issues
15:44:48 <akrzos> so need to test with ceilometer collector removed
15:45:06 <DinaBelova> akrzos ack, thank you sir for the update
15:45:09 <akrzos> basiclly in ocata and > you can change the publisher to gnocchi so agent-notification can publish directly to gnocchi
15:45:20 <akrzos> so thats what i am working on
15:45:21 <akrzos> also
15:45:29 <akrzos> i got WIP gnocchi benchmarks in rally
15:45:34 <akrzos> needs a bit more touch up
15:45:59 <DinaBelova> do you have some links to them?
15:47:28 <DinaBelova> akrzos ^^
15:47:52 <akrzos> yes
15:48:05 <akrzos> #link https://review.openstack.org/#/c/453861/
15:49:04 <DinaBelova> akrzos thanks!
15:49:06 <DinaBelova> from Mirantis side - we're still working on the next cycle of testing scope, as product testing priorities strike back :) so last week was mostly around planning and some more time is required there.... from what I see right now we'll be even more k8s centric this quarter in terms of RnD
15:49:29 <DinaBelova> but it's not defined yet, sadly
15:49:36 <DinaBelova> hope to give more details next time
15:49:52 <DinaBelova> #topic Open Discussion
15:49:59 <DinaBelova> tovin07 I see items from you sir :)
15:50:02 <DinaBelova> please proceed
15:50:10 <tovin07> yup
15:50:21 <tovin07> (tovin07) OSprofiler show sql/function results (continue)
15:50:30 <tovin07> rcherrueau: do you have any comment?
15:50:43 <rcherrueau> yep
15:51:02 <rcherrueau> I wanna try your patch, going with a devstack AIO deployment and `enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler refs/changes/72/450072/2` but devstack always install osprofiler master.
15:51:08 <rcherrueau> Don't know why!
15:51:44 <rcherrueau> By the way, I wrote a review based on the demo you provided in your commit message.
15:52:07 <tovin07> yes, I see
15:52:17 <DinaBelova> rcherrueau shouldn't that be some flag in devstack? to set up specific branch for the library?
15:52:39 <tovin07> thank you rcherrueau, I will re-recheck it
15:53:00 <rcherrueau> DinaBelova: no idea, I rarely use devstack
15:53:09 <rcherrueau> tovin07: cool
15:53:21 <tovin07> okay
15:53:22 <tovin07> next item
15:53:30 <tovin07> (tovin07) About Rally + OSprofiler for OpenStack services (rcherrueau had discussed with Rally PTL about this)
15:53:40 <tovin07> this one is related to rcherrueau (too)
15:54:19 <tovin07> #link Rally + OSprofiler http://eavesdrop.openstack.org/meetings/rally/2017/rally.2017-04-10-14.05.log.html#l-39
15:54:23 <tovin07> as I discussed in Rally IRC meeting yesterday
15:54:39 <tovin07> <andreykurilin> Ok, I'll start a spec about it.
15:55:07 <rcherrueau> tovin07: nice
15:55:22 <tovin07> so, I think that I can help a little bit on that work
15:56:11 <DinaBelova> tovin07 thank you sir :)
15:56:23 <rcherrueau> Speaking about OSProfiler, I finally managed to get it works into kolla-ansible. Briefly, osprofiler dependency was missing in nova/keystone/glance/... containers.
15:56:46 <rcherrueau> So the import of osprofiler always failed and services never called osprofiler to put trace events into oslo_messaging
15:56:50 <tovin07> rcherrueau: cool
15:56:59 <rcherrueau> This error was really silent!
15:57:01 <tovin07> yes
15:57:04 <DinaBelova> rcherrueau ah, so that was about right configuration in the end :)
15:57:09 <DinaBelova> 95% of all issues :D
15:57:12 <tovin07> osprofiler is not a required dependency
15:57:29 <tovin07> it’s optional
15:57:34 <rcherrueau> DinaBelova: yep :D
15:58:13 <DinaBelova> ok, folks, I have to run to other meeting... tovin07 rcherrueau akrzos anything to share at last minutes?
15:58:23 <tovin07> nope
15:58:26 <rcherrueau> no
15:58:35 <DinaBelova> thank you folks! have a nice day / evening!
15:58:37 <DinaBelova> #endmeeting