15:30:00 <DinaBelova> #startmeeting Performance Team
15:30:01 <openstack> Meeting started Tue Dec  6 15:30:00 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:05 <openstack> The meeting name has been set to 'performance_team'
15:30:15 <DinaBelova> ok, good evening / morning :)
15:30:30 <DinaBelova> let's see who's around :)
15:30:34 <msimonin1> Hi DinaBelova
15:30:44 <DinaBelova> akrzos, msimonin1 o/
15:31:00 <DinaBelova> oh :) msimonin left us :)
15:31:09 <DinaBelova> msimonin has joined :)
15:31:12 <DinaBelova> :D
15:31:14 <msimonin> yep :)
15:31:22 <msimonin> fixed my nick
15:31:35 <DinaBelova> well, you'll be still number one msimonin :)
15:31:48 <msimonin> :D
15:32:02 <akrzos> here!
15:32:05 <DinaBelova> ok, let's wait for a while and see if someone else is gonna join :)
15:32:06 <DinaBelova> yay :)
15:32:10 <DinaBelova> ad_rien_ o/
15:32:13 <ad_rien_> o/
15:32:35 <DinaBelova> ok, it looks like we may smoothly get started - I believe we'll have short meeting today
15:32:41 <DinaBelova> #topic Action Items
15:32:50 <DinaBelova> starting with action items
15:32:58 <DinaBelova> thanks akrzos for the Telemetry test plan!
15:33:02 <DinaBelova> #link https://review.openstack.org/#/c/405542/
15:33:17 <DinaBelova> I liked the way how it was written and described
15:33:24 <DinaBelova> so thanks again :)
15:33:28 <akrzos> thanks
15:33:46 <akrzos> I'm hoping to get some hardware to run that with Newton w/ ceph as  a backend
15:33:52 <DinaBelova> any chance you know when the results can be shared? as said, LDT was really excited about having them in public openstack docs
15:33:59 <DinaBelova> ack, got it
15:34:14 <akrzos> Well I have results posted with file as a backend but alot has already changed
15:34:15 <DinaBelova> akrzos please keep us updated :)
15:34:21 <DinaBelova> akrzos indeed
15:34:23 <akrzos> thats on our blog
15:34:29 <DinaBelova> I agree things need to be rerun
15:34:56 <DinaBelova> and in fact that was only action item we had for today
15:35:11 <DinaBelova> so thanks akrzos again and we may proceed
15:35:22 <DinaBelova> #topic Current progress on the planned tests
15:35:42 <DinaBelova> msimonin ad_rien_ - how are you guys doing? did you finish your whitepaper?
15:36:00 <ad_rien_> yes
15:36:01 <msimonin> yes we finished last thursday
15:36:02 <ad_rien_> sory
15:36:05 <ad_rien_> yes please msimonin
15:36:09 <msimonin> :)
15:36:10 <ad_rien_> just go on
15:36:13 <ad_rien_> :)
15:36:22 <msimonin> so
15:36:37 <msimonin> now we are moving to multisite deployment and experimentation
15:36:44 <DinaBelova> yay! :D
15:37:04 <ad_rien_> we should be able to make a technical report and share it soon
15:37:20 <msimonin> maybe we can propose a test plan as well for the performance doc
15:37:31 <DinaBelova> #info Inria starting multisite deployment and experimentation
15:37:36 <DinaBelova> msimonin that will be really cool
15:37:45 <DinaBelova> again, lots of operators might be interested
15:37:57 <msimonin> yes apparently
15:37:59 <DinaBelova> so it makes sense to ask some folks from LDT  to review that
15:38:04 <msimonin> cool
15:38:33 <DinaBelova> msimonin so let's proceed with the test plan, and I'll ping several folks from LDT to review it
15:38:38 <DinaBelova> I believe they'll be interested
15:38:57 <msimonin> DinaBelova: ack
15:39:18 <DinaBelova> also msimonin dunno if you saw ilyashakhat 's email in openstack-dev
15:39:19 <msimonin> What is challenging is to be able to cover many different deployment topologies
15:39:29 <msimonin> yeah I just saw it
15:39:37 <DinaBelova> oh, ok :)
15:39:44 <msimonin> I'll try to give my feed back as soon as possible
15:39:54 <DinaBelova> so he was not able to attend the meeting - but kindly asks you to review his approach
15:40:00 <DinaBelova> and if that fits your needs
15:40:08 <msimonin> yes I'll f
15:40:09 <DinaBelova> so thanks in advance :)
15:40:14 <msimonin> I'll do it*
15:40:18 <DinaBelova> ack
15:40:34 <DinaBelova> anything else to share?
15:41:02 <DinaBelova> ok, so I may share what's going on with our experiments
15:41:17 <DinaBelova> the whole lat week we were tuning our monitoring solution
15:41:18 <msimonin> nothing more from me
15:41:23 <DinaBelova> ack
15:41:52 <DinaBelova> so it turned out that our Prometheus is not surviving through the 900 nodes k8s+openstack deployment :)
15:42:01 <DinaBelova> and not actually collecting / storing all measurements
15:42:23 <DinaBelova> we experienced error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped." chunksToPersist=1525260 maxChunksToPersist=1524288 maxToleratedMemChunks=55553014 memoryChunks=50539186 source="storage.go:908"
15:42:32 <DinaBelova> Prometheus now eat 206G RAM, Local storage memory series is 13 079 271 :D
15:42:42 <msimonin> :D
15:42:53 <DinaBelova> so we're going to scale it to two prometheuses :D
15:43:10 <DinaBelova> to separate system measutements and k8s measurements
15:43:18 <DinaBelova> let's see if that's gonna help
15:43:29 <DinaBelova> right now we have a good feeling regarding this approach
15:43:36 <DinaBelova> so I'll keep you folks updated
15:43:42 <akrzos> Do you have a document that can be shared descibing the issues you ran into?
15:43:51 <DinaBelova> akrzos we're composing it right now
15:44:03 <akrzos> *nod*
15:44:13 <DinaBelova> as obviously we had issues not only with monitoring during last 2 months :D
15:44:24 <DinaBelova> and we decided to have one doc for all of them :D
15:44:36 <DinaBelova> I think we'll more or less fill it during this week
15:44:49 <DinaBelova> and I'll share it
15:45:02 <DinaBelova> I think most probably that will be google spreadsheet
15:45:12 <DinaBelova> at the first approximation
15:45:29 <DinaBelova> if info will be interesting, we can share it in performance docs as well
15:45:56 <DinaBelova> so that's all from my side obviously :)
15:46:07 <DinaBelova> akrzos anything you think will be interesting to share?
15:46:34 <akrzos> Not at this time
15:46:38 <DinaBelova> ack :)
15:46:48 <DinaBelova> so it looks like we may jump to Open Discussion
15:46:58 <DinaBelova> #topic Open Discussion
15:47:09 <DinaBelova> I have only one thing to mention in this section
15:47:36 <DinaBelova> #info there will be a room prepared for Rally / Performance team on PTG
15:47:42 <DinaBelova> #link https://www.openstack.org/ptg/
15:47:57 <DinaBelova> it'll be Monday - Tuesday (Feb 20-Feb 21)
15:48:45 <DinaBelova> so I really think it'll be beneficial for us to gather together and discuss tools roadmap and testing plans, as well as the presentations that can be shared on the main summit
15:48:47 <ad_rien_> we marked that on our agenda @inria.
15:48:53 <DinaBelova> ad_rien_ ack
15:49:09 <ad_rien_> we need to validate that it makes sense 'economically/financially' speaking
15:49:21 <DinaBelova> indeed
15:49:26 <DinaBelova> totally understand
15:49:30 <ad_rien_> PTG + Summit less than two months after then we have our internal meeting ...
15:49:36 <ad_rien_> at the end it costs a lot.
15:49:54 <ad_rien_> In any case, we will attend the meeting at least by remote webconf system if this can be envisionned
15:49:55 <ad_rien_> ?
15:50:03 <DinaBelova> I agree - esp keeping in mind that both will be happening in US this time
15:50:06 <DinaBelova> totally agree
15:50:08 <ad_rien_> yes
15:50:11 <ad_rien_> you get the point
15:50:18 <DinaBelova> from Mirantis side I'm going to attend as well as andreykurilin
15:50:25 <DinaBelova> who's rally PTL :)
15:50:33 <ad_rien_> I think we need to discuss with the Openstack foundation folks
15:51:07 <ad_rien_> because this PTG is one more event that should occur twice a year if I'm correct … which means 4 events in a year where we should go
15:51:18 <ad_rien_> at least...
15:51:48 <DinaBelova> yeah, that can be an issue, esp if locations will be chosen orienting on US-based contributors
15:51:58 <DinaBelova> at least for PTG
15:52:24 <DinaBelova> anyway, please keep that in mind and let's see if you'll be able to attend in person
15:52:33 <akrzos> Well will PTG be one in US and one outside US (same as summit)?
15:53:39 <DinaBelova> akrzos don't know exactly - as the only motivation for PGT locations is to be cheap enough and locating in transport cities with international airports preferrably, etc.
15:53:58 <DinaBelova> to make teams gathering easier
15:54:16 <DinaBelova> but as said - most of the contributors are us-based, that means this might influence
15:54:29 <akrzos> yea
15:54:55 <DinaBelova> afair there was no assumption that there will be "US - non US" approach for PGT initially - although it might appear as a needed thing
15:55:15 <DinaBelova> and in fact  that's all I wanted to mention
15:55:26 <DinaBelova> does anyone has anything else to discuss?
15:56:03 <msimonin> not me ;)
15:56:06 <DinaBelova> it looks like we were super quick this time :) thanks akrzos msimonin ad_rien_
15:56:13 <DinaBelova> see you guys
15:56:17 <DinaBelova> #endmeeting