15:30:03 <DinaBelova> #startmeeting Performance Team
15:30:04 <openstack> Meeting started Tue Sep 20 15:30:03 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:08 <openstack> The meeting name has been set to 'performance_team'
15:30:15 <DinaBelova> ok, so hello everyone!
15:30:22 <DinaBelova> let's see who's around :)
15:30:24 <ad_rien_> o/
15:30:36 <DinaBelova> ad_rien_ o/
15:30:41 <bvanhav> hello
15:30:48 <DinaBelova> rohanion, bvanhav o/
15:31:01 <ad_rien_> but I will be the only one from Inria today (rcherrueau is finalizing his Phd thesis and msimonin had to go)
15:31:15 <DinaBelova> ad_rien_ ack
15:31:30 <rohanion> heeeey
15:31:32 <rohanion> o/
15:32:07 <DinaBelova> from my side I need to apologize as well: Mirantis folks are having big all hands meeting right now, so not so much tracktion from our side as well
15:32:27 <DinaBelova> traction *
15:32:37 <DinaBelova> ok, so let's get started
15:32:48 <openstack> DinaBelova: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
15:32:58 <DinaBelova> oh sorry :)
15:32:59 <DinaBelova> #topic Action Items
15:33:03 <DinaBelova> wrong command :)
15:33:18 <DinaBelova> so geragding action items - we had only one last time
15:33:33 <DinaBelova> ad_rien_ it was about adding monitoring info to your etherpad
15:33:55 <DinaBelova> ad_rien_ I see some details were added - cadvisor/influx/grafana
15:34:20 <ad_rien_> yes but this is internal information
15:34:40 <ad_rien_> From our side, we do not have much information to share today
15:34:49 <DinaBelova> ad_rien_ ack
15:34:56 <DinaBelova> #topic Current progress on the planned tests
15:34:58 <ad_rien_> we are working on analyzing the traces we collected for
15:35:11 <ad_rien_> (DinaBelova sorry you were too fast ;))
15:35:25 <ad_rien_> the IDLE deployments
15:35:49 <ad_rien_> and are still fighting issues with the 1000 fake drivers deployment
15:35:53 <DinaBelova> ad_rien_ well, the test plans section is better for what you're discussing, so I switched to it
15:35:56 <DinaBelova> :)
15:35:57 <ad_rien_> running rally benchmark suite
15:36:02 <ad_rien_> :-)
15:36:33 <ad_rien_> We arranged a meeting on thursday with msimonin to prepare content for the presentation (and before our friday meeting)
15:37:04 <DinaBelova> ad_rien_ cool! I think that I'll fill my part as well
15:37:35 <ad_rien_> rcherreau developed a basic script to be able to see the influxdb informations we collected (i.e. this enables us to make postmorten analysis in a friendly way)
15:37:35 <DinaBelova> ad_rien_ as for Alex part, he's helping one of our customers right now, so not sure if he'll be able to finish everything
15:37:51 <ad_rien_> DinaBelova:  ack
15:37:55 <sai> hello, kinda late :)
15:38:01 <DinaBelova> sai o/
15:38:13 <DinaBelova> ok, so what can I share from our side
15:38:34 <DinaBelova> lots of work in progress regarding preparing test plans and test results
15:38:42 <DinaBelova> that were performed during the q3
15:38:55 <DinaBelova> + we're still fighting with OpenStack on top of k8s
15:39:18 <DinaBelova> right now we're having 350 nodes baremetal k8s installation
15:39:22 <DinaBelova> with Calico overlay
15:39:34 <DinaBelova> + OpenStack installed on top of this k8s cluster
15:39:39 <openstackgerrit> Merged openstack/performance-docs: modify the home-page info with the developer documentation  https://review.openstack.org/372923
15:39:59 <DinaBelova> we planned to run baseline tests against this cloud, but constantly observing the issues with k8s / docker
15:40:27 <DinaBelova> docker is not that scalable when we're talking about huge load running at once
15:40:29 <DinaBelova> :)
15:40:38 <ad_rien_> :-)
15:40:55 <DinaBelova> as for 1000 nodes experiment, we've decided to give a first try to non-containerized deployment
15:41:11 <DinaBelova> so we're having 1000 nodes being under installation with fuel + some dirty hacks
15:41:16 <DinaBelova> to make it work smoothly
15:41:17 <ad_rien_> it would be great if you can share the issues you faced
15:41:32 <DinaBelova> ad_rien_ sure, first set of issues was related to installer itself
15:41:55 <DinaBelova> we used Kargo for this purpose (due to the right topology these folks are proposing)
15:42:00 <DinaBelova> lemme share the links
15:42:15 <DinaBelova> #link https://github.com/kubespray/kargo
15:42:21 <DinaBelova> so that's an installer
15:43:05 <DinaBelova> we filed two original issues
15:43:12 <DinaBelova> #link https://github.com/kubespray/kargo/issues/478
15:43:19 <DinaBelova> #link https://github.com/kubespray/kargo/issues/479
15:43:27 <DinaBelova> that later leaded to several more :)
15:43:32 <DinaBelova> https://github.com/kubespray/kargo/issues/487
15:43:32 <DinaBelova> https://github.com/kubespray/kargo/issues/485
15:43:33 <DinaBelova> https://github.com/kubespray/kargo/issues/480
15:43:57 <DinaBelova> so the installation part is now ok, and we have observed several issues with the installed cluster
15:44:07 <DinaBelova> the most annoying is the following:
15:44:20 <DinaBelova> cluster seem to degrade with time
15:44:47 <ad_rien_> what do you mean by cluster ''degrades''  ?
15:45:04 <DinaBelova> OpenStack VMs were stuck in scheduling state - with more probability of this the more time k8s cluster is existing
15:45:45 <DinaBelova> so OpenStack was reinstalled several times with no issues, but for some reasons OpenStack services running as k8s apps began not to reach each other
15:45:51 <ad_rien_> ok …. maybe some issues we observed previously with kolla (docker) but we did succeed to reproduce them on our side :( …. did you find a way to reproduce such issues ?
15:46:09 <ad_rien_> s/we did succeed/did not succeed
15:46:30 <DinaBelova> ad_rien_ only running OpenStack... one of our k8s engineers is suspecting the Docker in this behaviour
15:46:33 <DinaBelova> under research right now
15:46:41 <DinaBelova> but nothing specific today, sorry
15:46:48 <ad_rien_> the same on our side....
15:46:57 <DinaBelova> ad_rien_ ok, so this seems to be common
15:47:14 <DinaBelova> I'll share details once (and I hope I will!) I'll have them
15:47:20 <ad_rien_> thanks
15:47:32 <DinaBelova> so that's all from our side regarding the plans
15:47:52 <DinaBelova> does anyone has anything else to share in this section?
15:48:20 <DinaBelova> it looks like we may jump to the osprofiler
15:48:36 <DinaBelova> #topic OSProfiler weekly update
15:48:41 <DinaBelova> rohanion the floor is your
15:48:44 <DinaBelova> :)
15:48:44 <rohanion> hey guys
15:48:56 <rohanion> so what we have done for the last week
15:49:08 <rohanion> 1. neutron patch is still waiting for CR
15:49:34 <rohanion> 2. cinder patch is documented and tested
15:49:55 <rohanion> 3. glance patch is finished, verified, waiting to merge
15:50:10 <DinaBelova> rohanion do you need any help pinging the core reviewrs?
15:50:13 <rohanion> 4. sahara - still nothing, no updates
15:50:18 <rohanion> YES PLEASE
15:50:21 <DinaBelova> rohanion ack
15:50:37 <rohanion> because it's a challenge finding who is responsible enough to merge the changes
15:50:53 <DinaBelova> rohanion I'll make sure core reviewers are aware about these patches and ask for the help
15:51:04 <DinaBelova> rohanion ack :)
15:51:18 <rohanion> 5. elasticsearch driver - reviewed, ready to merge, harlowja please look at it
15:51:36 <sai> rohanion: mind if i ask u what the elastic search dirver is abt
15:51:41 <sai> or if u can point me to some docs
15:52:03 <rohanion> sure, I'll post a link right here
15:52:14 <rohanion> https://review.openstack.org/#/c/340936/
15:52:30 <rohanion> this driver is basically another way of storing internal osprofiler data
15:52:56 <sai> gotcha
15:52:58 <sai> thanks
15:53:17 <DinaBelova> rohanion anything else?
15:53:19 <rohanion> 6. redis driver - I reviewed it, I didn't like it, the setup will go nuts if the database server goes down
15:53:55 <rohanion> I asked Ben if he could add a subclass that will use sentinel instead of a single redis instance
15:54:07 <rohanion> no response since thursday
15:54:13 <DinaBelova> bvanhav ^^
15:54:33 <bvanhav> I'm working on making the suggested change
15:54:39 <DinaBelova> bvanhav can you please take a look?
15:54:40 <DinaBelova> ack!
15:54:51 <DinaBelova> bvanhav good to know, thanks you :)
15:54:58 <rohanion> thank you Ben!
15:55:18 <rook> rohanion what about nova -- sorry if I missed it
15:55:31 <rohanion> oh you mean Dina's patches?
15:55:47 <rook> osprofiler w/ nova
15:55:57 <DinaBelova> yeah, rohanion was going to continue my patches
15:56:13 <rohanion> I have not started looking at them yet
15:56:29 <rohanion> but thanks for reminding me that, I'll start doing that tomorrow
15:56:36 <DinaBelova> as I am simply drown in the q3 deliverables we (Mirantis Scale RnD team) need to achieve
15:56:45 <DinaBelova> so thanks rohanion for grabbing a flag here
15:56:54 <rohanion> hope to finish work on some of them till the next meeting
15:57:11 <DinaBelova> rohanion ack
15:57:55 <DinaBelova> ok, so it looks like we may proceed
15:58:04 <DinaBelova> #topic Open Discussion
15:58:14 <DinaBelova> so I have some small summit news :)
15:58:32 <DinaBelova> #info I was told we're goingto have separated time slot on summit for performance team needs
15:58:34 <sai> DinaBelova: yeh wanted to ask abt that, are the perf guys meeting at summit?
15:58:47 <DinaBelova> by summit authorities
15:58:51 <DinaBelova> sai yes we are :)
15:59:00 <sai> we are goin to have some presence from redhat perf, would be great to sync
15:59:11 <DinaBelova> and now (after some conversations) we'll have a separated time slot
15:59:14 <DinaBelova> sai sure
15:59:34 <DinaBelova> #info now exact time not available so far, most probably Thursday morning
15:59:46 <DinaBelova> will keep you folks updated
15:59:47 <sai> awesome
16:00:09 <sai> DinaBelova: so wrt DVR work seeing issues when trying to ping VM with floating ip when router is DVR
16:00:10 <DinaBelova> sai ;)
16:00:17 <sai> some cases takes 200 seconds
16:00:26 <DinaBelova> sai ack
16:00:26 <sai> filed https://bugs.launchpad.net/neutron/+bug/1625333
16:00:28 <openstack> Launchpad bug 1625333 in neutron "Booting VM with a Floating IP and pinging it via that takes a long time with errors in L3-Agent logs when using DVR" [Undecided,New]
16:00:30 <DinaBelova> anything to share?
16:00:32 <DinaBelova> ok, I see
16:00:40 <DinaBelova> #link https://bugs.launchpad.net/neutron/+bug/1625333
16:01:08 <DinaBelova> sai lemme share this bug with our neutron team
16:01:10 <sai> rook: and i looked at it, seems to be an issue with floatingip assignment at l3-agent
16:01:17 <sai> sure
16:02:02 <DinaBelova> sai cool, thank you for the update
16:02:24 <sai> also heloing some folks at other places get started with shaker
16:02:29 <sai> helping*
16:02:35 <sai> seems to be good amount of interest
16:03:04 <DinaBelova> sai our neutron team did not jump into the issue you filed during their experiments
16:03:07 <DinaBelova> it's interesting
16:03:19 <DinaBelova> as they had plenty of tests done on various topologies
16:03:25 <DinaBelova> including DVR-based
16:03:25 <sai> DinaBelova: I ahve linked the rally-plugin we used and rally results too
16:03:36 <DinaBelova> sai ack, I'll take a look
16:03:45 <sai> DinaBelova: our neutron team is also looking at it
16:03:56 <DinaBelova> :) good to have neutron teams :)
16:03:58 <DinaBelova> :D
16:04:16 <DinaBelova> ok, anything else to share?
16:04:29 <sai> nothing from my end
16:04:38 <DinaBelova> ad_rien_ rook ?
16:04:42 <ad_rien_> nothing thanks
16:04:47 <rook> nada
16:04:59 <DinaBelova> ok, so thanks everyone for participating, stay tuned!
16:05:02 <DinaBelova> #endmeeting