15:30:02 <DinaBelova> #startmeeting Performance Team
15:30:03 <openstack> Meeting started Tue Aug 30 15:30:02 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:06 <openstack> The meeting name has been set to 'performance_team'
15:30:12 <DinaBelova> hey folks :)
15:30:18 <AugieMena> hey
15:30:21 <ad_rien_> hi
15:30:22 <DinaBelova> hope new time will be more comfortable :)
15:30:37 <DinaBelova> rohanion msimonin o/
15:30:58 <DinaBelova> let's wait for a few moments :)
15:31:09 <msimonin> Hey, hello :)
15:31:46 <DinaBelova> so let's get started with action items
15:31:47 <DinaBelova> #topic Action Items
15:32:01 <DinaBelova> last time we had only one in fact: find out what's slowing down OpenStack REST APIs in the Inria 1000 nodes experiment
15:32:24 <msimonin> yes
15:32:26 <DinaBelova> msimonin rcherrueau afaiu the issue was in DB settings?
15:32:35 <msimonin> yes it was
15:32:52 <msimonin> kolla deployment configure the mariadb with galera replication out of the box
15:33:12 <msimonin> in our primary tests we only have 1 mariadb instance
15:33:30 <rohanion> hiii
15:33:40 <DinaBelova> msimonin yep, to have comparable stuff
15:33:55 <DinaBelova> ok, so you found what was the difference even without any help :)
15:33:57 <DinaBelova> rohanion o/
15:34:10 <DinaBelova> so we can jump to the nice part :)
15:34:13 <DinaBelova> #topic Upcoming summit
15:34:28 <rook> o/
15:34:34 <DinaBelova> so from what I know: 1000 nodes emulation testing sesison was approved
15:34:40 <DinaBelova> lemme find a link
15:35:10 <DinaBelova> #link https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15977/chasing-1000-nodes-scale
15:35:30 <DinaBelova> so this means msimonin and Alex and I are going to have funny time preparing :)
15:35:42 <msimonin> yes :)
15:35:59 <ad_rien_> rcherrueau should contribute too
15:36:06 <DinaBelova> ad_rien_ ack
15:36:06 <ad_rien_> (even if his name does not appear ;) )
15:36:47 <DinaBelova> regarding the separated time slot for performance team discussions: WG sessions were not yet approved, although as technically we're part of LDT we may not capture separated slot as we were lucky to do last summit
15:36:48 <msimonin> Is there any other talk accepted for the performance WG ?
15:37:43 <DinaBelova> not that I know about
15:37:46 <DinaBelova> :)
15:38:19 <DinaBelova> so going back to the WG stuff: as we're part of LDT, we may not have separated slot
15:38:36 <ad_rien_> What do you mean ?
15:38:37 <DinaBelova> so I wonder if LDT will give us some time to present the results and collect feedback
15:38:41 <DinaBelova> klindgren ? ^^
15:38:59 <ad_rien_> ok
15:38:59 <klindgren> so
15:39:06 <DinaBelova> klindgren do you think it's possible?
15:39:12 <klindgren> I think our schedulue is fairly open right now.
15:39:34 <ad_rien_> What is your idea DinaBelova ? you would like to present twice   ?
15:39:52 <DinaBelova> klindgren ok, so I just want to have some backup variant if separated session/time slot won't be available
15:40:02 <ad_rien_> Or you plan to give a more synthetic presentation just for the LDT WG?
15:40:07 <DinaBelova> ad_rien_ the thing is that official talk != design summit
15:40:11 <DinaBelova> and usual discussions
15:40:15 <ad_rien_> yes
15:40:23 <DinaBelova> + 1000 nodes is only one set of tests
15:40:27 <DinaBelova> from what would be done
15:40:29 <ad_rien_> but we can expect that LDT key persons will attend the ''official'' presentation ?
15:40:39 <DinaBelova> so sharing the whole status will be nice I think
15:40:53 <klindgren> I think that should be fine.  I haven't looked at the schedule to see the LDT sloted times.
15:41:00 <DinaBelova> ad_rien_ this depends much on the schedule (that's not defined yet :))
15:41:14 <DinaBelova> klindgren I did not see these slots being added and defined yet
15:41:19 <DinaBelova> klindgren that's the issue :)
15:42:12 <DinaBelova> klindgren so I have 0 idea on if my slot separated request for performance team was accepted or not :)
15:42:19 <DinaBelova> and that's also an issue :D
15:43:19 <klindgren> Ah I - C
15:43:40 <DinaBelova> #info right now schedule is not yet finalized, not clear if performance team will gather the separated time slot for discussions, or we can present our results using small part of LDT time slot: LDT agenda is fairly open right now
15:44:10 <DinaBelova> ok, so that's pretty all I know about upcoming summit in terms of performance team :)
15:44:21 <DinaBelova> does someone know something else?
15:45:03 <DinaBelova> it looks like nope :)
15:45:05 <AugieMena> fyi, someone from my team (Chris Kirkland) had a performance presentation accepted
15:45:06 <DinaBelova> rook AugieMena ?
15:45:20 <AugieMena> #link ttps://www.openstack.org/summit/barcelona-2016/summit-schedule/global-search?t=nova+scheduler
15:45:34 <rook> We have a Performance talk accepted as well /me gets link
15:45:43 <DinaBelova> rook ack
15:45:54 <DinaBelova> #link https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15424/a-nova-scheduler-for-public-cloud-scale
15:46:28 <rook> #link https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/15279
15:46:45 <DinaBelova> rook ack, thanks :)
15:46:57 <DinaBelova> so we have some summit agenda already defined :)
15:47:25 <DinaBelova> btw, folks, who's going to attend the summit?
15:47:33 <ad_rien_> o/
15:47:42 <DinaBelova> msimonin I suppose you'll, and what's about rcherrueau ?
15:47:50 <DinaBelova> ad_rien_ ;)
15:47:50 <ad_rien_> should be there too
15:47:57 <DinaBelova> ok, cool, that's nice
15:48:00 <DinaBelova> AugieMena, rook ?
15:48:02 <rcherrueau> yep
15:48:05 <AugieMena> I won't be there, but Chris will
15:48:12 <rook> xnegative
15:48:22 <DinaBelova> AugieMena ack, thanks for letting know
15:48:27 <rook> I do have a bunch of people from my team attending
15:48:32 <rook> Too close to the birth of my second son :)
15:48:46 <DinaBelova> rook wow, congratulations on upcoming EVENT
15:49:04 <rook> thanks!
15:49:18 <AugieMena> congrats rook... also close to the birth of my first grandson :)
15:49:57 <DinaBelova> ahaha, it's an epidemic :)
15:50:14 <DinaBelova> ok, so it looks like we may proceed
15:50:15 <DinaBelova> #topic Current progress on the planned tests
15:50:27 <DinaBelova> msimonin rcherrueau may you please start?
15:50:42 <msimonin> yes sure
15:51:08 <msimonin> so with rcherrueau we deployed a 1000 compute nodes openstack
15:51:23 <msimonin> to make some premilinary tests and validate our experiment workflow
15:51:59 <msimonin> as we said before with the patch on the mariadb cofiguration, things were better
15:52:07 <DinaBelova> ok, so it looks like you're having nice progress
15:52:12 <DinaBelova> that's really cool
15:52:30 <DinaBelova> msimonin thanks
15:52:38 <msimonin> moreover :)
15:52:54 <msimonin> Next week we will try different topologies
15:53:07 <msimonin> with this 1000 computes (fake driver)
15:53:21 <msimonin> like adding more controllers
15:53:50 <msimonin> maybe we should discuss what could be relevant ?
15:53:58 <DinaBelova> msimonin good idea
15:54:27 <msimonin> because actually I discussed briefly with Steve Dake
15:54:40 <msimonin> from Kolla
15:55:10 <msimonin> and they are making some experimentation as well using a kolla deployed openstack and a set of rally benchmarks
15:55:52 <DinaBelova> msimonin ok, any insights from them?
15:56:16 <msimonin> let me share a link
15:56:26 <msimonin> #link https://etherpad.openstack.org/p/kolla-N-midcycle-osic
15:57:23 * DinaBelova trying to find the deployment topology in the doc
15:57:36 <msimonin> line 35
15:57:41 <msimonin> for the first scenario
15:57:58 <DinaBelova> a-ha, I see
15:58:07 <DinaBelova> and what's meant by controller in this case btw?
15:58:51 <msimonin> in the kolla terminology
15:59:29 <msimonin> control = nova-[api|scheduler], horizon, rabbitmq, mariadb, keystone
15:59:40 <DinaBelova> I'm asking as personally from what we've seen several control plane components need to be removed from controller nodes to separated ones if we're going up with scale (rabbitmq, mysql, conductor, neutron server, keystone)
15:59:56 <DinaBelova> so for their case (130 nodes) just all-in-one controller should be ok
16:00:15 <DinaBelova> but for 1000+ nodes scale I believe something needs to be separated
16:00:29 <msimonin> yes 1 control should be able to handle 100 compute node scale
16:00:49 <msimonin> yes actually that's the purpose of the experiment I'd like to conduct next week
16:01:23 <msimonin> having separated DB, rabbitmq, conductor, …
16:01:44 <DinaBelova> and see how it'll go :)
16:01:52 <DinaBelova> that's nice, thanks from doing this :)
16:02:18 <msimonin> To your knowledge is their some similar evaluation in the Openstack community ,
16:02:18 <msimonin> ?
16:02:50 <DinaBelova> msimonin I know operators were discussing the deployment topologies. LDT in particular
16:03:12 <DinaBelova> and everyone is having it's own opinion on this :)
16:03:30 <msimonin> I can imagine easily :)
16:03:47 <DinaBelova> so we can just measure the numbers and present them and create some recommendations - but HOW to do it it's still up to operator
16:04:10 <DinaBelova> and sometimes there are internal reasons not to follow these recommendations :)
16:04:15 <ad_rien_> Maybe we can add such a point in the agenda of the working session in Barcelona
16:04:25 <DinaBelova> ad_rien_ good point
16:04:31 <ad_rien_> we have different scenarios in mind @Inria
16:04:56 <ad_rien_> the one msimonin explained and also somes that include regions/multi sites deployments
16:05:13 <DinaBelova> #info let's go thorough 1000 nodes experiment result on Barcelona summit and present the deployment recommendations we'll come up with as a result of it
16:05:31 <DinaBelova> ad_rien_ ack
16:06:06 <DinaBelova> ok, so from Mirantis side: we have finished 400 nodes control plane / dataplane test runs, currently collecting the results
16:06:21 <DinaBelova> the thing is that in fact due to several hw issues it were 378 nodes, not 400 :(
16:06:37 <DinaBelova> but we decided not to wait till they will be fixed and run tests now
16:06:50 <DinaBelova> if we'll be able, we'll rerun them on 500 nodes at the end of sept
16:07:40 <DinaBelova> so what's upcoming: k8s + fuel-ccp (containerized control plane) evaluation, 1000 nodes emulation on 250 nodes
16:08:33 <DinaBelova> as for the first item: it's something requested by Mirantis folks, who are working on fuel-ccp and and k8s, so we're interested on how k8s itself and openstack on top of it can scale
16:08:54 <ad_rien_> could you please clarify
16:08:58 <ad_rien_> I'm bit lost:
16:09:08 <ad_rien_> k8s + fuel ccp
16:09:23 <ad_rien_> are you deploying VMs or containers to emulate your 1000 nodes ?
16:09:25 <DinaBelova> ad_rien_ there is much effort here in Mirantis is spent on containeraized OpenStack development
16:09:39 <ad_rien_> ok
16:09:42 <DinaBelova> and we (as scale team inside Mirantis) got a request to try it
16:09:47 <ad_rien_> ok
16:10:02 <ad_rien_> I'm just trying to  understand
16:10:15 <ad_rien_> how are you emulating your 1000 nodes on top of 250 physical servers
16:10:37 <DinaBelova> ad_rien_ so that's something I was going to talk about now
16:10:47 <ad_rien_> cool
16:10:48 <DinaBelova> last time on small lab we used containers for this purpose
16:11:05 <DinaBelova> this time we really wanted to have non-fake driver
16:11:52 <DinaBelova> so we have several options now: use kolla/ccp for this purpose (on top of 1000 vms run against 250 nodes) or just fuel (again, on top of 1000 vms)
16:12:32 <DinaBelova> due to the fact we're pushed to use ccp, we're currently evaluating it on small scale
16:12:57 <DinaBelova> and we'll see what will be chosen in next 2 weeks or so
16:13:53 <DinaBelova> ad_rien_ this is much about politics and what's more useful for Mirantis as a company, but I really hope we'll grad these 1000 nodes cluster :D
16:14:01 <DinaBelova> grab*
16:14:32 <DinaBelova> so the main moment: we think there will be enough HW to run these 1000 nodes with usual libvirt Nova driver, not fake one
16:14:48 <ad_rien_> ok
16:14:56 <ad_rien_> it's clear thanks
16:15:14 <DinaBelova> and see how the picture will change comparing with what we've done previously on small lab with fake river
16:15:19 <DinaBelova> ok, cool
16:15:27 <DinaBelova> so it looks like we can jump to the OSprofiler
16:15:32 <DinaBelova> #topic OSProfiler weekly update
16:15:38 <DinaBelova> rohanion the floor is yours :)
16:15:49 <rohanion> ok cool
16:16:41 <rohanion> I'm still working on a script that changes the config files and restarts the services
16:16:51 <rohanion> will finish it by the end of today
16:17:03 <rohanion> nothing besides that, unfortunately :(
16:17:19 <DinaBelova> rohanion you are about automation of osprofiler usage on Fuel-installed clouds?
16:17:33 <rohanion> yes
16:17:53 <DinaBelova> rohanion ack, thanks
16:18:00 <rohanion> but it will work with vanilla OS too
16:18:18 <DinaBelova> rohanion ok, thanks for mentioning this :)
16:18:20 <rohanion> I decided not to work with hiera and detect the role based on the services
16:19:15 <DinaBelova> rohanion any updates from Alex? (I know he could not attend today meeting)
16:19:34 <rohanion> No, he switched to another project
16:19:38 <rohanion> afair
16:20:03 <DinaBelova> rohanion ack
16:20:10 <DinaBelova> #topic Open Discussion
16:20:21 <DinaBelova> so do we have something else to cover?
16:21:15 <DinaBelova> it looks like nope :)
16:21:22 <DinaBelova> thanks everyone for participating :)
16:21:24 <DinaBelova> bye!
16:21:26 <ad_rien_> bye
16:21:28 <DinaBelova> #endmeeting