15:30:01 <DinaBelova> #startmeeting Performance Team
15:30:01 <openstack> Meeting started Tue Jan  3 15:30:01 2017 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:30:05 <openstack> The meeting name has been set to 'performance_team'
15:30:13 <DinaBelova> hello folks!
15:30:27 <DinaBelova> let's see who has survived after holidays :D
15:30:32 <tovin07> hello :D
15:30:39 <DinaBelova> akrzos tovin07 msimonin o/
15:31:13 <akrzos> DinaBelova: hello
15:31:20 <msimonin> Hello
15:31:22 <spn> o/
15:31:38 <DinaBelova> wow, we have quorum :D
15:31:38 <spn> Happy new year to all :)
15:32:22 <DinaBelova> yeah, first of all Happy New Year to all of you folks :) I hope holidays were as merry as they should :)
15:32:43 <DinaBelova> and it looks like we an start our first 2017 brand new meeting :)
15:32:55 <DinaBelova> #topic Action Items
15:33:06 <DinaBelova> last time we only had one action item on myself
15:33:33 <DinaBelova> regarding creating an etherpad to start collecting PGT agenda
15:33:35 <DinaBelova> here it is
15:33:38 <DinaBelova> #link https://etherpad.openstack.org/p/ptg-performance-team
15:33:45 <DinaBelova> @akrzos @msimonin ^^
15:34:00 <DinaBelova> let's have two weeks or so to collect ideas
15:34:27 <DinaBelova> #info let's use https://etherpad.openstack.org/p/ptg-performance-team to collect ideas to discuss on PTG
15:34:41 <spn> yup
15:35:02 <DinaBelova> #info let's have two weeks timeframe to prepare it
15:35:22 <tovin07> (y)
15:35:33 <akrzos> *nod*
15:35:50 <DinaBelova> @msimonin do you have any news about Inria participance? are you guys going to attend in person?
15:36:32 <msimonin> Hum I don't think we'll go in person
15:36:53 <DinaBelova> msimonin ok, so we need to plan somehow your remote participance
15:37:02 <msimonin> yes if it's possible
15:37:13 <msimonin> technically speaking this would be great
15:37:19 <spn> is there a list to tell who all are attending in person?
15:37:35 <akrzos> can we use that etherpad to track
15:37:45 <DinaBelova> spn I and andreykurilin (Rally PTL)
15:37:47 <DinaBelova> akrzos  sure
15:37:53 <msimonin> +1 to track on the etherpad
15:37:57 <spn> I am attending in person too
15:38:37 <spn> +1 to etherpad. I will update it
15:39:19 <DinaBelova> ok, cool
15:39:38 <DinaBelova> I've added who's going to attend from mirantis side
15:40:10 <akrzos> We have budget for it. just not sure if who is going just yet
15:40:20 <DinaBelova> akrzos cool
15:40:34 <DinaBelova> please update the etherpad once you'll know for sure
15:40:36 <DinaBelova> :)
15:40:38 <akrzos> I'll try and confirm before End of today
15:40:42 <DinaBelova> akrzos ack
15:41:08 <DinaBelova> so it looks like we may proceed to researches and updates
15:41:14 <DinaBelova> #topic Current progress on the planned tests
15:42:04 <DinaBelova> msimonin any updates from you sir? (I do not think we all hav something sagnificant to talk about as it ere holidays, but still)
15:42:14 <msimonin> Yes :)
15:42:31 <msimonin> Holliday was productive for other things than current actions :)
15:42:44 <DinaBelova> ** significant, sorry for the typo
15:42:45 <DinaBelova> :D
15:42:56 <DinaBelova> ok, I believe akrzos you have pretty the same situation :D
15:43:29 <DinaBelova> from our side I have some items to share
15:43:35 <akrzos> pretty much just trying to get gnocchi on swift on the same hardware as ceph i did about two weeks ago
15:43:41 <akrzos> though already changes in gnocchi
15:43:44 <akrzos> 's defaults
15:43:46 <akrzos> (again)
15:43:54 <DinaBelova> oh, I got it
15:44:41 <rook> DinaBelova et all any more feedback on #link https://review.openstack.org/#/c/412554/
15:44:43 <DinaBelova> so waiting for the some updates next week I guess :)
15:45:09 <DinaBelova> rook I see, not more reviews there
15:45:14 <DinaBelova> lemme just merge it
15:45:15 <rook> yup
15:45:20 <rook> i want to post the results
15:45:24 <DinaBelova> as I feel pretty confident about it
15:45:36 <DinaBelova> cool, lemme click merge button
15:45:37 <rook> thx
15:45:49 <DinaBelova> rook it should land in few moments
15:46:12 <DinaBelova> from our side: several documents have been published on review: https://review.openstack.org/#/q/project:openstack/performance-docs+status:open
15:46:44 <DinaBelova> I guess one of the most interesting parts for you folks will be monitoring methodology
15:46:45 <DinaBelova> akrzos msimonin https://review.openstack.org/#/c/415563/
15:47:21 <rook> oh fun, prometheus
15:47:30 <DinaBelova> please review, we're going to continue this effort this year to improve it, so some feedback it welcome :D
15:47:34 <DinaBelova> rook yeah :D
15:47:38 <msimonin> DinaBelova: ack
15:48:09 <akrzos> interesting
15:48:31 <msimonin> I see Heka for log collection
15:48:44 <DinaBelova> msimonin true
15:48:57 <msimonin> Kolla is dropping Heka probably in favour of fluentd
15:49:03 <tovin07> Heka is now deprecated in some other projects as I see
15:49:08 <msimonin> yes
15:49:08 <tovin07> Yup, Kolla
15:49:09 <akrzos> DinaBelova: I'll review and provide feedback though i don't have experience with promethous, definitely interested in it vs graphite or gnocchi or any other monitoring solution :)
15:49:22 <DinaBelova> akrzos ack, thank you sir :)
15:49:27 <tovin07> Kolla plans to move to Fluentd
15:49:44 <DinaBelova> msimonin tovin07 well, that's all matter of just what o choose :)
15:49:59 <DinaBelova> lemme ask team go through this option as well :)
15:50:03 <DinaBelova> thanks for letting know
15:50:22 <msimonin> yes it was just  to give the information
15:50:44 <DinaBelova> #action DinaBelova discuss Fluentd option for logs collecting in our monitoring solution
15:50:48 <DinaBelova> thank you folks for the info :)
15:51:09 <openstackgerrit> Merged openstack/performance-docs: Performance of OpenStack API Workers test plan  https://review.openstack.org/412554
15:51:28 <rook> interesting...
15:51:41 <rook> DinaBelova: do you guys have scale/perf data on these solutions?
15:52:05 <DinaBelova> rook we have run this type of monitoring for 900 nodes environment
15:52:17 <rook> hm ok cool, what are you collecting?
15:52:25 <rook> all metrics? or a subset
15:52:44 <DinaBelova> all of them :)
15:52:45 <DinaBelova> in that caae it was 13,5 million time series were collected
15:52:54 <rook> per second?
15:53:04 <DinaBelova> and we needed to have two prometheus instances to survive this load
15:53:10 <rook> hm ok
15:53:14 <rook> cool!!
15:53:41 <DinaBelova> rook not per second, but at all in Prometheus "list" of what needs to be collected
15:53:58 <DinaBelova> although prometheus is first of all monitoring and alerting tool
15:54:17 <DinaBelova> and that means that there is high data retention being used by us
15:54:26 <akrzos> so promethues is not just time-series data, but also collecting logs?
15:54:39 <akrzos> or receiving logs from fluentd*
15:54:46 <DinaBelova> as if data was stored there for a long time, it became less responsive
15:55:25 <DinaBelova> akrzos moniotirng only, we used elastic search for the logs in most common way. weka was used to parse some of the data from it and push as measurements to prometheus
15:55:35 <DinaBelova> like Kubernetes API response time
15:55:47 <DinaBelova> the only way you can gather this metric - calculate it from logs
15:56:26 <akrzos> oh so some of the performance metrics from services is emitted only into the logs thus using the log collection component to pull the performance data and graph in time-series, thats very cool
15:56:43 <DinaBelova> akrzos indeed
15:56:44 <akrzos> thought I'm an advocate for the performance data to be exposed outside logs (to begin with)
15:56:48 <akrzos> though*
15:57:19 <DinaBelova> well, I'll agree with you personally, but we were really interested in k8s API response time :D
15:57:30 <DinaBelova> so we had to find a way to collect it
15:57:39 <akrzos> gotta do what you gotta do
15:58:44 <DinaBelova> we plan to continue this effort in new year - at very first we need to configure some external time series DB to collect all this data for longer time for post-experiments researches and calculations
15:58:47 <DinaBelova> like Cassandra or OpenTSDB
15:59:05 <DinaBelova> as right now we use pure promethues, and it's not really good in storing data for a ong time
15:59:10 <DinaBelova> long*
15:59:16 <akrzos> same problem with graphite
15:59:22 <DinaBelova> akrzos yeah
15:59:53 <DinaBelova> we had issues like "oh, let's take a look on a data we had a month ago, oh sh*t no this data anymore"
16:00:19 <DinaBelova> so I really would like not having this issue in 2017 :D
16:00:29 <akrzos> yeah or downsampling or what was done to "aggregation" of samples
16:00:37 <akrzos> samples/measurements
16:01:02 <spn> what is the backend for promethues?
16:01:23 <akrzos> grafana can even "change" what the measurements look likes based on the number of pixels you have available to display :O
16:01:51 * akrzos assumes your using grafana for the visualization portion
16:02:17 <DinaBelova> spn prometheus has its own prometheus db (not sure what technology are they using under the hood), but if you want to have this data persistent - use some persistent DB prometheus can work with - it might be cassandra, opentsdb, influx
16:02:23 <DinaBelova> akrzos true
16:02:47 <spn> DinaBelova: ok cool
16:02:49 <DinaBelova> so grafana is sometimes a challenge as well, that's true :D
16:03:19 <DinaBelova> anyway, so folks I'm kindly asking to review it
16:03:45 <spn> akrzos: can we export the visualization portion of grafana so that it can be imported on a newer grafana elsewhere?
16:04:15 <akrzos> spn: should be able to export the json and then push it into another grafana via the api
16:04:41 <akrzos> i haven't tried the latest grafana to be honest
16:04:44 <DinaBelova> my team is going to be back around Jan 9th (Russian winder holidays are after the New year for about a week) so I really hope they'll have the feedback from you :)
16:05:54 <DinaBelova> ok, so it looks like we may jump to open discussions
16:06:01 <DinaBelova> #topic Open Discussion
16:06:12 <DinaBelova> anything else to share folks?
16:06:22 <spn> can we discuss about spec in our face to face?
16:06:37 <spn> I am using spec @ Dell for openstack
16:06:58 <DinaBelova> I believe we can
16:07:18 <DinaBelova> spn please propose this item to the list of what to discuss
16:07:29 <spn> ok sure
16:08:05 <DinaBelova> ok, anything else folks?
16:08:40 <msimonin> Yep
16:08:45 <msimonin> From Inria side
16:09:09 <DinaBelova> sure, please go ahead :)
16:09:11 <msimonin> Among other things we'll use osprofiler
16:09:28 <msimonin> more precisely some interns will use it
16:09:32 <msimonin> ;)
16:09:46 <DinaBelova> yay :)
16:09:51 <DinaBelova> I'm glad to read this
16:10:06 <tovin07> That’s a good new
16:10:07 <msimonin> I may be wrong but I thing there is still a review for its intergration in nova
16:10:24 <DinaBelova> msimonin that's right
16:10:45 <msimonin> DinaBelova: ack
16:10:49 <DinaBelova> although right now it's tracked by our (Mirnatis) Nova team - they are really interesting in landing it
16:10:54 <msimonin> ok :)
16:11:00 <DinaBelova> this change is up to date in fact
16:11:02 <tovin07> Not just Mirantis :))
16:11:08 <DinaBelova> and can be used
16:11:15 <DinaBelova> tovin07 yes, you're right ;)
16:11:36 <msimonin> ok good to know
16:12:08 <DinaBelova> msimonin please keep me updated if any issues will occur, let's fix them :)
16:12:18 <msimonin> Thanks !
16:12:51 <DinaBelova> msimonin anything else to share?
16:13:25 <msimonin> I think that's all for today
16:13:34 <DinaBelova> yeah, thank you folks!
16:13:41 <DinaBelova> have a nice 2017 :)
16:13:44 <DinaBelova> #endmeeting