15:30:02 <DinaBelova> #startmeeting Performance Team
15:30:03 <openstack> Meeting started Tue Nov 22 15:30:02 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:06 <openstack> The meeting name has been set to 'performance_team'
15:30:17 <DinaBelova> ok, let's see who's around :)
15:30:25 <DinaBelova> msimonin ad_rien_ rcherrueau o/
15:30:28 <msimonin> o/
15:30:51 <DinaBelova> sai are you around?
15:31:17 <rcherrueau> o/
15:31:43 <ad_rien_> o/ (but in no so proactive mode sorry,  few pendings todos on my side)
15:31:52 <DinaBelova> ack, got it
15:32:24 <DinaBelova> so let's get started I guess
15:32:41 <DinaBelova> let's got through action items first
15:32:50 <DinaBelova> and I think people may join in progress
15:32:56 <DinaBelova> #topic Action Items
15:33:22 <DinaBelova> one of the items was on me regarding the nova cells V2 status
15:33:43 <DinaBelova> msimonin rcherrueau ad_rien_ I've posted that info recently, but only msimonin was online :)
15:33:50 <DinaBelova> so lemme repaste it :)
15:34:23 <rook> o/
15:34:36 <DinaBelova> #info Andrew Laski is not working anymore at Mirantis -> most probably he won't take part in OpenStack dev -> Dan Smith and Melanie Witt at Red Hat are picking up the remaining cellsv2 tasks
15:34:41 <DinaBelova> rook o/
15:35:17 <DinaBelova> msimonin ad_rien_ rcherrueau so you guys need to ping those two ^^ about what's going on right now in the Nova Cells V2
15:35:36 <msimonin> thanks DinaBelova
15:35:44 <ad_rien_> thanks
15:35:54 <DinaBelova> okay, so let's proceed with action items
15:36:37 <DinaBelova> one more was on ad_rien_ - regarding adding info about the nfv group info to the summit recaps section of performance docs
15:36:47 <ad_rien_> Sorry I missed it
15:36:54 <ad_rien_> (too many todos)
15:36:58 <DinaBelova> I guess it was a big load on you
15:37:00 <DinaBelova> yeah
15:37:04 <ad_rien_> I will do it by tomorrow evening
15:37:07 <DinaBelova> lemme keep this action item
15:37:10 <ad_rien_> you can count on me
15:37:11 <DinaBelova> thanks
15:37:22 <DinaBelova> #action ad_rien_ add info about the nfv group info to the summit recaps section of performance docs
15:37:26 <DinaBelova> thanks again :)
15:37:36 <DinaBelova> and the last action item was on sai
15:38:03 <DinaBelova> as he did not answer yet and he seems to be in silent mode right now, lemme keep his action item as well
15:38:11 <DinaBelova> sai find out when information about telemetry researched Red Hat has done can be published to performance docs
15:38:14 <DinaBelova> #action sai find out when information about telemetry researched Red Hat has done can be published to performance docs
15:38:26 <DinaBelova> so we are done with action items!
15:38:33 <DinaBelova> and may jump to the next section
15:38:43 <rook> DinaBelova: he is in a way different timezone
15:38:44 <DinaBelova> #topic Current progress on the planned tests
15:38:50 <rook> DinaBelova: akrzos can speak to that
15:38:57 <DinaBelova> rook yay
15:39:05 <DinaBelova> thanks for letting me know
15:39:10 <rook> akrzos re: Telemetry work you have done
15:39:46 <DinaBelova> akrzos the thing is that OpenStack operators were really excited to hear that you folks have done telemetry researches on the summit
15:40:03 <DinaBelova> and LDT kindly asked to publish this info to the performance-docs
15:40:30 <DinaBelova> when I checked that with sai he told he need to check if and when this can be done
15:40:45 <akrzos> DinaBelova: gotcha, yeah I need to revisit it with a ceph backend
15:41:41 <DinaBelova> akrzos ok, so any understanding of when you're going to finish those experiments? and to publish the results of course! folks were really excited - telemetry is on of the pain points for a while :)
15:41:42 <akrzos> i have looked at it with file and swift backends but with not great disks so i'd say the results are less than exciting wrt to Gnocchi vs database
15:42:05 <DinaBelova> akrzos ack, got that
15:42:30 <DinaBelova> #info akrzos looked at Gnocchi with file and swift backends but with not great disks so i'd say the results are less than exciting wrt to Gnocchi vs database, need to revist with Ceph backend
15:42:46 <akrzos> also here is a link to my initial strobe of it https://browbeatproject.org/2016/09/12/performance-testing-openstack-telemetry-ceilometer-and-gnocchi/
15:43:00 <DinaBelova> #link https://browbeatproject.org/2016/09/12/performance-testing-openstack-telemetry-ceilometer-and-gnocchi/
15:43:36 <DinaBelova> ok, so the next steps are: good HW, Ceph backend, retest
15:43:52 <DinaBelova> akrzos so again - any plans to publish that to performance-docs?
15:44:23 <akrzos> DinaBelova: I can start with the test plan
15:44:35 <DinaBelova> akrzos sure
15:44:42 <DinaBelova> this will be a good start
15:44:49 <DinaBelova> ok, so lemme create an action item
15:45:04 <akrzos> and then post what results we currently have and build on it when more hardware better disks are available for me again
15:45:08 <DinaBelova> if you won't be able to finish it till next week, it's ok, let's just have that pinned
15:45:18 <akrzos> so let me take that action on the test plan for now
15:45:48 <DinaBelova> #action akrzos publish OpenStack telemetry teat plan to performance-docs
15:45:53 <DinaBelova> ok, cool
15:45:56 <DinaBelova> thanks akrzos
15:46:15 <DinaBelova> msimonin any news about most recent experiments at Inria?
15:46:54 <DinaBelova> or ad_rien_ or rcherrueau  :)
15:47:05 <msimonin> Actually we are writing a paper about our evaluation framework
15:47:26 <msimonin> #link https://github.com/BeyondTheClouds/enos
15:47:29 <DinaBelova> good thing to be done! and to be shared once finished :)
15:47:31 <DinaBelova> ack
15:47:48 <msimonin> we are also performing energy evaluation of openstack services
15:48:01 <msimonin> (no link to that yet I think)
15:48:19 <DinaBelova> #info energy evaluation of openstack services by Inria in progress
15:48:21 <msimonin> We are adapting the tool to support multi site deployment
15:48:47 <msimonin> DinaBelova: sure we'll share it
15:48:56 <msimonin> deadline is next week for the paper :s
15:48:56 <DinaBelova> msimonin yeah, any plans on what Grid'5000 locations will be first tested in multi-site deployment?
15:49:01 <DinaBelova> just curious
15:49:13 <DinaBelova> msimonin ahaha, good luck with paper work :)
15:49:25 <ad_rien_> What do you mean DinaBelova?
15:49:28 <ad_rien_> Can you elaborate ?
15:49:29 <msimonin> :)
15:49:31 <ad_rien_> Thanls
15:49:50 <ad_rien_> Our idea is to define with the help of the NFV working group
15:49:58 <ad_rien_> a minimalist openstack deployment
15:50:10 <ad_rien_> that can federate resources spread across different locations
15:50:19 <ad_rien_> and then performs the same experiments we made in order to understand
15:50:28 <ad_rien_> major bottlenecks/performance limitations
15:50:50 <DinaBelova> well, Grid'5000 contains lots of servers in multiple cities - so I was just curious - last time for 1000 nodes experiment you used only one location of servers, this time you want to have multi-site deployment - so I was just curious what servers locations are affected
15:51:04 <ad_rien_> G5K is already a distributed infrastructure. So we have two possibilities: either deploy the control plane on one site and the other (i.e. compute nodes) on other sites.
15:51:08 <ad_rien_> yes
15:51:21 <ad_rien_> but since we want to ensure reproducibility
15:51:34 <DinaBelova> anyway, forgot what I was asking about :) I seem to confuse you
15:51:41 <DinaBelova> don't worry :)
15:51:45 <ad_rien_> we usually prefer to have the experiment in one site (and emulate the locations)
15:52:16 <DinaBelova> ok, so you're extending your experience
15:52:23 <ad_rien_> This way enables to control everything and collect relevant results (i.e. that do not have noise coming from another experiment)
15:52:59 <DinaBelova> indeed
15:53:12 <ad_rien_> the issue right now is that none knows what can be the minimalist deployment
15:53:36 <ad_rien_> we are exchanging with the NFV folks and colleagues from Orange Labs, BT,...
15:53:48 <ad_rien_> it is unclear what can be the right deployment scenario.
15:54:10 <ad_rien_> so the idea is probably to identify two or three and make comparisons between them
15:54:19 <ad_rien_> by using performance and functional metrics
15:54:29 <ad_rien_> (I hope I've been clear enough)
15:54:52 <DinaBelova> yeah, thank you!
15:55:11 <DinaBelova> and sorry again for confusion
15:55:15 <ad_rien_> the only thing that looks quite sure is that more and more operators would like to supervise several DCs in an unified manner (i.e. with a single openstack system)
15:55:19 <ad_rien_> no problem
15:55:42 <DinaBelova> ok, so let's proceed
15:56:44 <DinaBelova> on Mirantis side we're almost done with our advanced monitoring :) hope to publish this info to our perf docs soon :) we have crazy combination of Prometheus, Telefraf and CollectD to collect all info about system, k8s and OpenStack on top of it
15:57:12 <DinaBelova> we're going to use that approach during our k8s and OpenStack testing soon
15:57:28 <DinaBelova> we'll start with k8s most probably
15:57:48 <DinaBelova> with k8s density and kubelet capacity testing
15:58:01 <DinaBelova> test plans should appear on review soon
15:58:21 <DinaBelova> and that's all for now from our side
15:58:35 <DinaBelova> anything else to cover in this topic?
15:58:37 <msimonin> one question
15:58:40 <DinaBelova> sure
15:59:05 <msimonin> Your deployment method here : will it be part of fuel ?
15:59:23 <DinaBelova> deployment of OpenStack == fuel-ccp
15:59:37 <DinaBelova> and we have to install k8s separately - right now using Kargo tool
16:00:04 <DinaBelova> all steps are automated in our Scale team Jenkins
16:00:26 <msimonin> ok I see, thanks
16:00:52 <DinaBelova> that manages all the steps - e.g. "provision nodes (usually via cobbler)", "install k8s", "run fuel-ccp", "run tests"
16:01:03 <DinaBelova> we found that approach convenient for us
16:01:16 <DinaBelova> but for OpenStack yes - we're using fuel-ccp
16:01:16 <msimonin> Is the monitoring stack included in the tool (sorry for my ignorance)
16:01:19 <msimonin> ?
16:01:33 <DinaBelova> right now in progress of integration
16:01:45 <DinaBelova> we were playing with different methodologies mostly manually
16:02:04 <DinaBelova> right now - when even all Grafana boards are almost finished - we'll proceed with automation
16:02:14 <DinaBelova> of the monitoring installation and usage
16:02:36 <DinaBelova> I hope that answers your question :)
16:02:54 <msimonin> do you intend to publish the grafana dashboards ?
16:02:59 <msimonin> :)
16:03:22 <DinaBelova> we have that wish, we'll check if Intel folks (who're on other side of our co-dev activity) will agree
16:03:36 <DinaBelova> but they are usually very happy with upstreaming things
16:03:54 <msimonin> cool :)
16:03:56 <DinaBelova> #action DinaBelova find out if and when we're going to publish our Grafana dashboards
16:04:06 <DinaBelova> yeah ;)
16:04:35 <DinaBelova> okay, anything else in "test plans and results" section?
16:05:08 <DinaBelova> it looks like nope :)
16:05:14 <DinaBelova> #topic PTG in Atlanta
16:05:21 <DinaBelova> okay :D summit has just passed
16:05:32 <DinaBelova> and we need to think already about next one :D
16:05:59 <DinaBelova> as all you know starting after Barcelona summit design summit (PGT, project team gathering) ir separated from main summit event
16:05:59 <ad_rien_> DinaBelova:  can you please explain what our WG can achieve in this PTG meeting.
16:06:04 <ad_rien_> Still not clear from my side.
16:06:21 <DinaBelova> ad_rien_ the idea is to discuss if we need that :)
16:06:36 <DinaBelova> Rally team is going to have separated room on the summit
16:06:50 <ad_rien_> ok but I mean. From my understanding the meeting is mainly for core-devs ?
16:07:07 <ad_rien_> But our meeting, what non developers can do ?
16:07:41 <DinaBelova> ad_rien_ the thing is that if something needs to be improved in the projects - e.g. their performance and scalability - it's a good idea to have this planned for the next cycle..
16:07:56 <ad_rien_> We can come but it should be relevant (i.e. not  only for our meeting…. a bit expensive to bring 4/5 persons in one place if it is for one day working session).
16:08:17 <DinaBelova> so right now andreykurilin asked ttx to name the room "performance team and tools"
16:08:28 <ad_rien_> ok
16:08:35 <DinaBelova> so basically what I was thinking about is tools development
16:08:39 <ad_rien_> this has sense
16:08:45 <DinaBelova> we have rally, osprofiler, browbeat
16:08:56 <DinaBelova> and we're in fact developing them ourselves :)
16:09:14 <msimonin> :)
16:09:30 <DinaBelova> so something I was thinking about is to have some discussions about how things might be improved
16:09:49 <DinaBelova> we all are Rally users and there is plenty of things that need to be done there
16:10:05 <DinaBelova> can we help rally team somehow here? can we leave feedback?
16:10:19 <DinaBelova> do we have any wishes on what needs to be done there?
16:10:25 <DinaBelova> and so on
16:10:29 <ad_rien_> ok
16:10:33 <ad_rien_> thanks for the clarification
16:10:40 <msimonin> The same for shaker ? We are actually using it aswell
16:10:47 <DinaBelova> yes and shaker!
16:10:54 <DinaBelova> I forget about all of them :)
16:11:02 <DinaBelova> os-faults for destructive testing as well.
16:11:08 <msimonin> sure they are plenty of them
16:11:10 <DinaBelova> so bunch of tools
16:11:27 <DinaBelova> and it looks like this PGT is a good place to work on their progress
16:11:36 <DinaBelova> or even have kinda workshops
16:11:53 <DinaBelova> with ideas -> development -> reviews being done in plane
16:11:59 <DinaBelova> so what am I suggesting here
16:12:12 <DinaBelova> let's have 1 week for ideas gathering
16:12:16 <DinaBelova> and share them next time
16:12:24 <DinaBelova> on what can be discussed and how
16:12:35 <DinaBelova> akrzos ^^ fiy
16:13:11 <ad_rien_> DinaBelova:  can you please remind the dates ?
16:13:14 <DinaBelova> #info let's think if PGT will be useful for our group and if we can use this time for tools roadmaps discussions
16:13:15 <DinaBelova> sure
16:13:25 <DinaBelova> #info The first Project Teams Gathering (PTG) will be held in Atlanta, Feb 20-24, 2017.
16:13:49 <ad_rien_> thanks
16:14:11 <DinaBelova> so... I'm not insisting that we really need to have that... but let's think about if we will benefit from that
16:14:19 <ad_rien_> ok
16:14:26 <ad_rien_> the idea is also to attend the summit
16:14:45 <ad_rien_> (i.e. one face-to-face meeting in Atlanta and one in Boston)?
16:15:36 <DinaBelova> yeah, main summit is still an EVENT :) my company likes PGT more than summit mostly because PGT locations should be usually cheaper
16:15:47 <DinaBelova> but ideally - yes
16:16:20 <DinaBelova> two face-to-face meetings should be a good idea
16:16:43 <DinaBelova> as usually it's all question of money to be spent
16:17:03 <DinaBelova> so that's floating and vague - and that's ok
16:17:36 <DinaBelova> so again - please think if and how we can benefit from PGT
16:17:57 <ad_rien_> ok
16:18:07 <msimonin> ack DinaBelova
16:18:17 <DinaBelova> that's all from my side for this topic
16:18:35 <DinaBelova> so I think we can jump to open discussions
16:18:40 <DinaBelova> #topic Open Discussion
16:18:47 <DinaBelova> anything else to cover folks?
16:18:59 <DinaBelova> questions / ideas / notes?
16:19:12 <msimonin> not from our side I guess
16:19:24 <DinaBelova> ack, cool
16:19:29 <DinaBelova> it looks like we're done for today
16:19:34 <DinaBelova> thanks everyone!
16:19:38 <msimonin> thanks
16:19:38 <DinaBelova> #endmeeting