15:30:02 <DinaBelova> #startmeeting Performance Team
15:30:03 <openstack> Meeting started Tue Sep  6 15:30:02 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:07 <openstack> The meeting name has been set to 'performance_team'
15:30:14 <DinaBelova> hey folks :)
15:30:22 <DinaBelova> let's see who's around :)
15:30:29 <gokrokve> o/
15:30:33 <DinaBelova> rcherrueau, rohanion ? :)
15:30:35 <DinaBelova> gokrokve o/
15:30:38 <rcherrueau> hello
15:30:49 <DinaBelova> rcherrueau o/
15:30:53 <ilyashakhat> hi!
15:30:54 <DinaBelova> msimonin o/
15:30:58 <DinaBelova> ilyashakhat wow
15:31:10 <DinaBelova> ilyashakhat really glad to see you :)
15:31:13 <JorgeCardoso> hi all
15:31:28 <rohanion> yep!
15:31:28 <gokrokve> hi
15:31:29 <msimonin> Hey DinaBelova, I can't attend today :(
15:31:38 <msimonin> But rcherrueau is here :)
15:31:43 <DinaBelova> JorgeCardoso o/ would you like to present yourself yo the audience?
15:31:46 <DinaBelova> msimonin ck
15:32:11 <JorgeCardoso> yes
15:32:14 <DinaBelova> to the audience*** :) sorry
15:32:26 <JorgeCardoso> My name is Jorge Cardoso.
15:32:26 <DinaBelova> please go ahead
15:33:05 <JorgeCardoso> I am Chief Architect at Huawei Research and working on CLoud Reliability and Resiliency
15:33:31 <JorgeCardoso> I am currently workign in a project with Deutsche Telekom on OpenStack Reliability
15:34:05 <gokrokve> Very nice to meet you!
15:34:19 <JorgeCardoso> You can find more about my interests on reliability at: http://jorge-cardoso.github.io/research/
15:34:29 <DinaBelova> JorgeCardoso nice to see you here :) do you have something to share regarding this activity? We're really interested in OpenStack
15:34:35 <JorgeCardoso> Like wise!
15:34:35 <DinaBelova> reliability
15:34:41 <DinaBelova> JorgeCardoso indeed :)
15:35:03 <DinaBelova> #info JorgeCardoso 's research regarding reliability at: http://jorge-cardoso.github.io/research/
15:35:15 <DinaBelova> JorgeCardoso I'll definitely take a look on this
15:35:27 <DinaBelova> thank you so much
15:35:51 <DinaBelova> ok, so it looks like we may jump to the agenda
15:36:05 <JorgeCardoso> We are looking into Chaos Engineering and Google SRE to udnerstadn how a Fault Injection framework would look like for openstack
15:36:16 <DinaBelova> #link https://wiki.openstack.org/wiki/Meetings/Performance#Agenda_for_next_meeting
15:36:29 <DinaBelova> JorgeCardoso I think you may also speak to ilyashakhat
15:36:53 <DinaBelova> as we re working on the same thing right now - and we came up with rally+os-faults library
15:37:25 <DinaBelova> #link https://github.com/openstack/os-faults
15:37:54 <ilyashakhat> right now we mostly interested in investigating how particular operations survive the single fault
15:37:58 <JorgeCardoso> I looked at it, and it is a good starting point IMHO
15:38:59 <DinaBelova> JorgeCardoso ok, thanks for the feedback, lemme take a look on your points described in doc, it looks like we can somehow merge our efforts on this
15:39:55 <DinaBelova> so we can jump to current progress on the planned tests
15:39:59 <DinaBelova> #topic Current progress on the planned tests
15:40:05 <DinaBelova> rcherrueau would you like to start?
15:40:10 <rcherrueau> yes
15:40:21 <rcherrueau> so we continue on the 1000 node experiment
15:40:32 <rcherrueau> There is a pad with our roadmap
15:40:42 <rcherrueau> #link https://pad.inria.fr/p/FK18ZSyjFyLPITfE
15:41:22 <rcherrueau> we curently collect a lot of information
15:41:28 <DinaBelova> rcherrueau it looks like you had several runs
15:41:54 <rcherrueau> result of rally / network traffic / cpu & me usage / memcache usage ...
15:42:08 <DinaBelova> several timeouts were observed?
15:42:21 <rcherrueau> DinaBelova several runs ?
15:42:34 <DinaBelova> rcherrueau I see pass #1, pass #2, etc
15:42:38 <DinaBelova> in the doc :)
15:42:57 <rcherrueau> Oh yes, you should look at ligne 169
15:43:05 <rcherrueau> 163 sorry
15:43:14 <rcherrueau> line*
15:43:23 <DinaBelova> rcherrueau ack, thanks
15:43:37 <DinaBelova> rcherrueau any issues / interesting results observed so far?
15:44:22 <rcherrueau> nothing special, so far we mainly spend time on developing our framework to run test and collect information
15:44:56 <rcherrueau> now we have to interpret results of these experimentations
15:45:15 <DinaBelova> rcherrueau ack
15:45:30 <DinaBelova> rcherrueau btw do you have any monitoring for the containers?
15:45:38 <rcherrueau> yes
15:45:48 <DinaBelova> rcherrueau what are you using for this purpose?
15:45:59 <DinaBelova> what kind of tool?
15:46:06 <rcherrueau> we go with cadvisor
15:46:42 <rcherrueau> we also use some plugins from collectd
15:46:46 <DinaBelova> rcherrueau ack, I'm asking as I was experimenting with various pods / containers monitoring for openstack on top of k8s deployments
15:46:53 <gokrokve> Did you integrate cadvisor with InfluxDB+Graphana?
15:46:59 <rcherrueau> especially to monitor rabbitmq and mariadb
15:47:10 <rcherrueau> gokrokve yes
15:47:11 <gokrokve> That what we did in our 1000 nodes experiments
15:47:15 <rcherrueau> the same for collectd
15:47:31 <DinaBelova> rcherrueau and frankly speaking monitoring of ALL containers for all OpenStack services looks a bit overkill sometimes
15:47:45 <gokrokve> Did you see problems with InfluxDB slowness?
15:48:22 <rcherrueau> no not me, but I will ask to msimonin tomorrow
15:48:24 <gokrokve> When we used grafana sometimes Influx used 100 CPU and was not very responsive.
15:49:02 <DinaBelova> rcherrueau I can confirm
15:49:06 <rcherrueau> I'm sure msimonin tune grafana is such manner that there isn't too much writing into the database
15:49:40 <gokrokve> That is cool.
15:49:44 <DinaBelova> rcherrueau ok, probably it'll be nice to include this information (about what monitoring tools were used and ho tuned) to this documnet
15:49:55 <DinaBelova> probably at the very end of it, just for information
15:49:57 <gokrokve> We did just dirty wiring to have some graphs.
15:50:03 <rcherrueau> DinaBelova yes you'r right
15:50:40 <DinaBelova> rcherrueau I'll add an action item
15:51:10 <DinaBelova> #action rcherrueau msimonin add information about what monitoring tools were used to the https://pad.inria.fr/p/FK18ZSyjFyLPITfE
15:51:24 <DinaBelova> rcherrueau thank you for sharing this
15:51:38 <rcherrueau> you are welcome
15:52:02 <DinaBelova> from our side: we have merged 400 nodes baseline testing results to the performance docs
15:52:06 <DinaBelova> #link http://docs.openstack.org/developer/performance-docs/test_results/control_plane_400_nodes/index.html
15:52:22 <DinaBelova> dataplane and Neutron testing results are under construction
15:52:26 <DinaBelova> to get published
15:52:47 <DinaBelova> probably some first part of them will be shared before next meeting
15:53:42 <DinaBelova> we're having the plans to start fuel-ccp (containerized control plane) + k8s scale testing this week
15:53:48 <DinaBelova> let's see how it'll go
15:54:29 <DinaBelova> jfyi: I'm visiting k8s Scale Special Interest Group meetings
15:55:08 <DinaBelova> to understand how k8s itself as a basement for OpenStack deployments can scale and what kind of performance to support
15:55:39 <DinaBelova> so if anyone is interested in that kind of information, I really advice you to attend these meetings - they are quite interesting
15:56:21 <DinaBelova> also reliability tests with os-faults library are coming soon
15:56:32 <DinaBelova> so I think that's all from our side right now
15:56:58 <DinaBelova> let's proceed to the osprofiler
15:57:04 <DinaBelova> #topic OSProfiler weekly update
15:57:16 <DinaBelova> we have several new interesting changes on review
15:57:39 <DinaBelova> #info new Redis-based driver for OSprofiler among them
15:57:41 <DinaBelova> #link https://review.openstack.org/#/c/364487/
15:58:12 <DinaBelova> frankly speaking did not have chance to try it yet on the test env, I'm having it in my todo list for this wee
15:58:14 <DinaBelova> kweek*
15:58:50 <DinaBelova> #info ElasticSearch driver still under development https://review.openstack.org/#/c/340936/
15:59:53 <DinaBelova> afaik rohanion you did not have a chance to publish your script for enabling profiling on vanilla OpenStack yet?
16:00:39 <DinaBelova> rohanion ? :)
16:00:46 <rohanion> actually, there are 2 scripts
16:00:58 <rohanion> a script that patches conf files
16:01:05 <rohanion> and a script that restarts the services
16:01:19 <rohanion> the latter requires Fuel
16:01:36 <DinaBelova> rohanion well yeah, that is the deployment-specific thing
16:01:39 <rohanion> the first just hacks the config files so I can publish it somewhere
16:01:53 <DinaBelova> rohanion yes please, this will be useful
16:01:54 <rohanion> it's really short btw
16:02:02 <DinaBelova> rohanion I believe it should :)
16:02:18 <rohanion> #action @rohanion publish osprofiler config patcher
16:02:40 <DinaBelova> ok, so it looks like that's all regarding the profiler
16:03:00 <DinaBelova> for today we did not more specific items in agenda
16:03:03 <DinaBelova> #topic Open Discussion
16:03:12 <DinaBelova> anything to share, folks?
16:04:01 <DinaBelova> it looks like nope
16:04:16 <DinaBelova> so thanks everyone for coming, have a nice day/evening
16:04:22 <DinaBelova> #endmeeting