16:00:01 <DinaBelova> #startmeeting Performance Team
16:00:01 <openstack> Meeting started Tue Apr  5 16:00:01 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:04 <openstack> The meeting name has been set to 'performance_team'
16:00:14 <DinaBelova> o/
16:00:17 <DinaBelova> hey folks!
16:00:19 <rohanion> о/
16:00:22 <rohanion> hi
16:00:27 <lezbar__> hi
16:00:29 <rook> o/
16:00:30 <rook> howdy
16:01:08 <DinaBelova> ok, let's wait for a few moments
16:01:14 <DinaBelova> and let others join as well
16:01:22 <bsilverman> Hi Dina
16:01:27 <DinaBelova> bsilverman o/
16:01:33 <ilyashakhat> o/
16:01:39 <bsilverman> :)
16:01:52 <DinaBelova> we have standard agenda for today
16:01:57 <DinaBelova> https://wiki.openstack.org/wiki/Meetings/Performance#Agenda_for_next_meeting
16:02:33 <DinaBelova> due to the fact I was sick last time and klindgren was replacing me, there are no much to discuss in the action items, but still
16:02:34 <DinaBelova> :)
16:02:50 <DinaBelova> #topic Action Items
16:03:07 <klindgren> DinaBelova, you going to be in Austin?
16:03:16 <DinaBelova> klindgren yes sir :)
16:03:50 <DinaBelova> hope to see you there as well :)
16:04:08 <klindgren> I will be there
16:04:14 <bsilverman> DinaBelova: Let me know if you want to talk about new items after the action items.
16:04:15 <rook> will there be a performance meetup @ Austin?
16:04:20 <DinaBelova> btw, about action items: the only one left was on myself regarding keystone profiling results sharing
16:04:56 <DinaBelova> rook we'll have OpenStack summit @Austin, and some our session will be there
16:05:21 <DinaBelova> so for Keystone: here are my changes
16:05:25 <DinaBelova> #link https://review.openstack.org/#/q/status:merged+project:openstack/performance-docs+branch:master+topic:keystone-res
16:05:54 <DinaBelova> some of them were merged to give comfortable possibility to go through the results, one change still on review
16:06:21 <DinaBelova> with raw results - mostly because it'll ruin all openstack contribution stats
16:06:24 <DinaBelova> :D
16:06:30 <DinaBelova> we're trying to understand what to do with them
16:06:37 <DinaBelova> dims ^^
16:07:14 <DinaBelova> so in short on upstream OpenStack with Mitaka keystone on my env I've observed very strange caching behaviour
16:07:15 <rook> DinaBelova: are you tuning the wsgi workers?
16:07:30 <rook> DinaBelova: Processes/Threads wrt to your keystone work?
16:07:50 <DinaBelova> rook - my idea was to profile DB/memcached usage for caching of keystone and keystone auth middleware
16:07:56 <DinaBelova> and compare liberty vs mitaka
16:08:12 <DinaBelova> as Keystone was observing huge changes during Mitaka timeframe
16:08:19 <dims> DinaBelova : liberty env was eventlet or Apache?
16:08:29 <DinaBelova> dims both apache
16:08:39 <dims> both with same apache configuration?
16:09:06 <DinaBelova> dims when I was looking on it - yes, but I'll recheck
16:09:10 <DinaBelova> good idea sir
16:09:35 <DinaBelova> #action DinaBelova ensure Apache server had the same config for both Liberty and Mitaka envs keystone profiling was done on
16:09:44 <dims> Dina, which stats to compare? "SELECT Keystone DB queries count" in "User list request stats"?
16:10:21 <DinaBelova> dims - for human readability please take a look on this change
16:10:23 <DinaBelova> #link https://review.openstack.org/#/c/299514/
16:10:37 <DinaBelova> you can see OSprofiler html reports there
16:10:46 <DinaBelova> for all 8 topologies I was looking on
16:11:28 <dims> DinaBelova : there's a lot in there, which one should we look at first as an example to see difference in behavior between liberty and mitaka
16:11:50 <DinaBelova> Liberty+Fernet+CacheOff/user_list.html and Mitaka+Fernet+CacheOff/user_list.html
16:11:52 <DinaBelova> for instance
16:12:42 <DinaBelova> behaviour will be common if you'll take a look on any pure keystone operations with different Liberty/Mitaka prefix
16:13:14 <DinaBelova> so you'll observe the following: on Liberty htmls all cache points are leaved in the requests tree
16:13:28 <DinaBelova> as I'm doing two identical API calls to keystone one after another
16:13:37 <DinaBelova> and the second one I'm profiling
16:14:05 <DinaBelova> and on Mitaka cache point as rarely leaves - in most of the cases taking from cach was not successful
16:14:08 <dims> DinaBelova : k let's take that. "User list request stats" shows "SELECT Keystone DB queries count" as 56 vs 95 in liberty vs Mitaka. right?
16:14:10 <DinaBelova> and we're going to the DB
16:14:50 <DinaBelova> oh, dims I'm sorry
16:15:01 <DinaBelova> please use CacheOn htmls
16:15:08 <rook> Was this only with Fernet?
16:15:22 <DinaBelova> rook - nope, UUID as well
16:15:39 <DinaBelova> Liberty+Fernet+CacheOn and Mitaka+Fernet+CacheOn for instance << dims
16:15:44 <dims> ok "SELECT Keystone DB queries count" 26 vs 80
16:16:14 <dims> so 56 went down to 26 and 95 went down only to 80
16:16:33 <dims> am i reading this right Dina?
16:16:52 <DinaBelova> yes sir
16:17:01 <DinaBelova> I was rechecking the numbers
16:17:02 <DinaBelova> yes
16:17:20 <DinaBelova> and the htmls in the https://review.openstack.org/#/c/299514/1 will show where exactly
16:17:41 <DinaBelova> I'll add links to them once we'll merge it without making stackalytics crazy
16:18:10 <dims> DinaBelova : where is the osprofiler output? for say  Mitaka+Fernet+CacheOn/user_list.html
16:18:24 <DinaBelova> dims right now here https://review.openstack.org/#/c/299514/1
16:18:36 <DinaBelova> you'll need to cherry-pick it locally
16:18:41 <DinaBelova> and take a look for now
16:18:50 <DinaBelova> as said, too crazy change to merge it now
16:19:12 <DinaBelova> dims sorry for inconvinience
16:19:35 <dims> DinaBelova : no worries, just trying to understand
16:19:42 <DinaBelova> dims yeah, sure
16:20:02 <DinaBelova> I may upload them now quickly to the github.io
16:20:03 <dims> so i can pull down user_list.html to see the calls
16:20:07 <DinaBelova> yes, sure
16:20:35 <dims> it's ok no worries will do it in a little bit
16:20:38 <dims> thanks for the details
16:20:40 <DinaBelova> with info like what function was chached with what args for instance
16:20:50 <DinaBelova> dims thank you for the effort anyway
16:21:08 <rook> based on the output they are both in wsgi (Apache) DinaBelova dims ( my guess they share the same config as well )
16:21:11 <dims> DinaBelova : only other thing to ask for would be the logs for each run
16:21:29 <DinaBelova> my later plan will be to take closer look on this and on the keystone authtoken middleware chaching as well
16:21:35 <DinaBelova> rook yes, they are under apache
16:21:37 <DinaBelova> both
16:21:39 <DinaBelova> dims ack
16:21:53 <DinaBelova> when I'll be rerunning it, I'll collect the logs
16:21:59 <dims> perfect
16:22:03 <dims> thanks Dina
16:22:06 <DinaBelova> np
16:22:20 <DinaBelova> ok, so it looks like we're done with action items :D
16:22:30 <DinaBelova> #action DinaBelova continue Keystone caching research
16:22:46 <DinaBelova> going forward if no objections
16:23:06 <DinaBelova> ok
16:23:07 <DinaBelova> #topic Test plans statuses
16:23:26 <DinaBelova> in fact my Keystone stuff was not the only one landed in performance-docs repo this week
16:23:38 <DinaBelova> ilyashakhat, listomin - please share your progress as well
16:23:57 <listomin> Hello!
16:24:23 <listomin> Here you can find results of Cobler:
16:24:35 <listomin> https://review.openstack.org/#/c/299516/ https://review.openstack.org/#/c/299510/
16:24:51 <DinaBelova> #info https://review.openstack.org/#/c/299516/ https://review.openstack.org/#/c/299510/
16:25:21 <listomin> human readable link http://docs-draft.openstack.org/16/299516/3/check/gate-performance-docs-docs/b3b7d2a//doc/build/html/test_results/provisioning/cobbler/index.html
16:25:21 <DinaBelova> ok, so that's the results of the provisioning performance - in case if we're using Cobbler
16:25:28 <DinaBelova> listomin - is that correct?
16:25:37 <listomin> Right
16:25:47 <DinaBelova> listomin ack
16:25:52 <rook> might i suggest adding that detail to the commit?
16:26:01 <DinaBelova> rook - sure, please review
16:26:13 <DinaBelova> we have Gerrit exactly for this purpose :)
16:26:34 <rook> roger, done.
16:27:10 <DinaBelova> rook ack - listomin please modify a bit commit message, I believe rook 's comments are valid
16:27:20 <DinaBelova> ok, cool
16:27:30 <DinaBelova> listomin I'll review you commits, thank you sir
16:27:37 <DinaBelova> ilyashakhat - are you around?
16:29:08 <DinaBelova> hm :)
16:29:09 <ilyashakhat> DinaBelova, what's up?
16:29:25 <DinaBelova> ilyashakhat any news about test plans results merged after last IRC meeting?
16:29:58 <ilyashakhat> hm, all RabbitMQ results are merged
16:30:24 <DinaBelova> ok, cool
16:30:38 <DinaBelova> am I wrong something was done for the DB as well?
16:30:42 <ilyashakhat> ZeroMQ are pending, need to re-run them with reduced message generation rate to avoid losses on server
16:30:56 <DinaBelova> or probably I just mess things up :)
16:31:02 <ilyashakhat> yes, there are also sysbench results for MySQL + Galera
16:31:08 <bsilverman> DinaBelova: We've done some scale testing, was wondering if we should bring the results to this group or if it would be better somewhere else?
16:31:25 <DinaBelova> bsilverman I believe this  one :)
16:32:00 <DinaBelova> please share details about what testing you were doing, what env against and what were the results
16:32:29 <DinaBelova> this can be done as a first estimation as you'd like, but ideally I think we can use the same pattern performance WG is using
16:32:32 <DinaBelova> that will be
16:32:40 <DinaBelova> - publish test plan
16:32:45 <DinaBelova> - publish env details
16:32:51 <DinaBelova> - publish the results
16:33:04 <DinaBelova> all to the https://github.com/openstack/performance-docs
16:33:36 <DinaBelova> bsilverman - if you feel uncomfortable with this right now, let's create an etherpad as a first approximation
16:33:51 <bsilverman> DinaBelova: Great, I will get with you offline with any questions. Is there any specific test plan template we have to follow?
16:34:23 <DinaBelova> bsilverman ack
16:34:27 <DinaBelova> yes, it is
16:34:30 <DinaBelova> one moment sir
16:34:37 <bsilverman> DinaBelova: Sure, the goal of our project is to see how far we can push a single control plane RA.
16:34:44 <DinaBelova> bsilverman : https://github.com/openstack/performance-docs/blob/master/doc/source/test_plans/template.rst
16:34:54 <bsilverman> DinaBelova: Thanks.
16:35:11 <DinaBelova> bsilverman - perfect, thank you sir
16:35:17 <lezbar__> DinaBelova: hi, I'm also doing some scale tests with MidoNet, Neutron and OpenStack, and we'll also use the Intel lab for scalability Mirantis is using
16:35:41 <bsilverman> We'd like to use the scale lab too :)
16:35:50 <DinaBelova> ahaha :D
16:35:58 <DinaBelova> all needs to be discussed with gosha :)
16:35:59 <lezbar__> we've had a couple of chats with Gosha already, he suggested me to assist to this meeting :-)
16:36:06 <DinaBelova> lezbar__ sure
16:36:18 <DinaBelova> in short lezbar__ I'll propose to have the same flow for you sir
16:36:35 <DinaBelova> publish test plan - publish env details -  publish the results
16:36:37 <lezbar__> sounds good to me
16:36:43 <bsilverman> We are working with Gosha, it's early but Jim is on-board.
16:36:48 <DinaBelova> let's share our experienca and collect feedback
16:37:01 <DinaBelova> bsilverman ack
16:37:27 <bsilverman> However, I want to include the results in this group and get feedback/share results so we don't re-invent the wheel.
16:37:56 <DinaBelova> bsilverman, lezbar__ - thank you guys for pushing your tests against this community effort
16:37:58 <DinaBelova> bsilverman +
16:38:10 <DinaBelova> ok, cool
16:38:13 <lezbar__> I have the same question as rook had, is there going to be any performance team meetup at the summit?
16:38:40 <DinaBelova> lezbar__, rook in short we're going to have working group session
16:38:56 <DinaBelova> as well as probably some time in oslo space - harlowja ?
16:39:26 <lezbar__> sorry DinaBelova, when is the working group session scheduled?
16:39:27 <rook> DinaBelova where will this working group session be held?
16:39:28 <DinaBelova> lezbar__ rook - and, sure, we can conquer some table and have free form discussion ther anyway :)
16:39:40 <rook> ok
16:39:42 <DinaBelova> moment, folks
16:40:07 <bsilverman> Make sure it doesn't conflict with any of my talks :)
16:40:08 <harlowja> DinaBelova i'm working through the schedules but possibly :)
16:40:15 <DinaBelova> harlowja ack
16:40:27 <harlowja> if not a direct oslo space/time, we can all get dinner together or something :-P
16:40:33 <DinaBelova> rook lezbar__ https://www.openstack.org/summit/austin-2016/summit-schedule/events/7399?goback=1
16:40:42 <DinaBelova> #info summit WG session https://www.openstack.org/summit/austin-2016/summit-schedule/events/7399?goback=1
16:40:52 <rook> DinaBelova++
16:40:54 <rook> thanks
16:40:59 <DinaBelova> harlowja - that's always a solution :)
16:41:04 <DinaBelova> rook you're welcome
16:41:04 <lezbar__> DinaBelova: thanks!
16:41:10 <bsilverman> DinaBelova: Nice!
16:41:16 <rook> harlowja yup, dinner and drinks also sounds good
16:41:23 <harlowja> ya, meetings after the meetings :-P
16:41:26 <DinaBelova> :D
16:41:49 <harlowja> summit-dinner-and-drinks-meeting(s) (not on the schedules), lol
16:41:55 <DinaBelova> in case of summits in general and Austin summit in particular that is going to be part after party after party :)
16:42:08 <harlowja> after after party?
16:42:09 <harlowja> woah
16:42:09 <harlowja> lol
16:42:17 <DinaBelova> indeed sir :)
16:42:26 * rook prepares liver now.
16:42:29 <harlowja> lol
16:42:34 <DinaBelova> anyway, it looks like we may jump into the next section
16:42:34 <rohanion> ehehe
16:42:36 <DinaBelova> a-haha
16:42:43 <DinaBelova> #topic OSProfiler weekly update
16:42:47 <DinaBelova> rohanion a-ha!
16:42:50 <DinaBelova> I see you sir :)
16:42:51 <rohanion> oops
16:42:53 <DinaBelova> :D
16:42:55 <bsilverman> Ain't no party like a performance party cause a performance party don't stop!
16:42:57 <rohanion> caught in the middle
16:43:04 <DinaBelova> rohanion - updates about drivers?
16:43:07 <rohanion> sure
16:43:44 <rohanion> I've deprecated the Notifier class from osprofiler
16:44:06 <DinaBelova> #link https://review.openstack.org/#/c/247005/
16:44:09 <rohanion> osprofiler notifiers are switched to our new architecture
16:44:20 <rohanion> they use Driver class instead
16:44:22 <DinaBelova> any chance you checked them already?
16:44:39 <rohanion> yes, I've run cinder --profile with my changes
16:44:44 <rohanion> it's working
16:44:47 <DinaBelova> ack, cool
16:45:09 <DinaBelova> #action DinaBelova harlowja boris-42 review latest rohanion's changes to https://review.openstack.org/#/c/247005/
16:45:16 <DinaBelova> rohanion thank you sir
16:45:30 <DinaBelova> also i have small update from the OPW
16:45:31 <rohanion> thank you
16:45:43 <DinaBelova> mentors began to vote for the candidates
16:46:04 <DinaBelova> one of them - buttercup - is wishing to work on OSprofiler next round
16:46:22 <rohanion> I've talked to her - she's cool
16:46:23 <DinaBelova> let's see if she'll gain the slot, but I'm pushing her :)
16:46:25 <DinaBelova> rohanion indeed
16:46:35 <DinaBelova> very nice possible contributor for sure :)
16:46:57 <DinaBelova> so let's hope she'll be able to join us
16:47:15 <rohanion> she'll probably work with elastic and json files if she succeeds
16:47:22 <DinaBelova> rohanion ack
16:47:36 <DinaBelova> it looks like that's all updates about profiler for now
16:47:41 <DinaBelova> #topic Open Discussion
16:47:48 <harlowja> woot, put on my todo lists
16:47:49 <rohanion> I've seen her patches in Ironic and she's familiar with json
16:47:53 <harlowja> i got to many todo(s)
16:48:05 <DinaBelova> harlowja you're not the only one :D
16:48:09 <harlowja> :)
16:48:31 <DinaBelova> I wanted to discuss briefly summit stuff, but we did it during the meeting
16:48:34 <DinaBelova> so let's sum up
16:48:34 <rohanion> therefore she doesn't need to know how actually does elasticsearch work, she'll have to dump all our output to json
16:48:44 <rohanion> a perfect job for an intern afaik
16:48:55 <rook> I did not add my item, however, my team is working on a project called Browbeat to combine best of breed performance and scale tools.
16:49:20 <DinaBelova> #info [Summit][Austin] we're going to have WG session https://www.openstack.org/summit/austin-2016/summit-schedule/events/7399?goback=1 + possible some oslo space if harlowja will be kind :)
16:49:31 <DinaBelova> rook - it's very interesting, btw
16:49:38 <rook> rohanion what I have found is we have to manipulate the data a bit to have elasticsearch work -- well.... if you want to use Kibana to viz it.
16:49:41 <DinaBelova> I saw you had a brief conversation with boris-42
16:49:56 <DinaBelova> rook - that's true
16:49:58 <rook> yeah, i can't seem to get on the same time as him... he is still in Ru time?
16:50:00 <rohanion> what about plain graphite?
16:50:15 <DinaBelova> rook he's in PST, but lots of stuff to do this week
16:50:16 <rook> Elastic -> Graphite?
16:50:23 <rohanion> Kibana -> Graphite
16:50:24 <DinaBelova> rook - possibly that's the reason
16:50:30 <rook> DinaBelova: ah, ok did he move to SJ?
16:50:39 <rohanion> Kibana cannot use logarithmic scales
16:50:47 <DinaBelova> rook afair yes
16:51:15 <rook> rohanion I havent tried to viz elastic data w/ Graphite... Just timeseries.
16:51:30 <rook> and for the most part we found that Grafana works better with timeseries viz.
16:51:34 <rohanion> yes
16:51:42 <rohanion> indeed it does
16:51:53 * bsilverman needs to run to another meeting, DinaBelova , I will ping you a little later to work on getting setup with the test plans and sharing.
16:52:09 <DinaBelova> rook rohanion - that's all discussible, although I have a probably crazy dream to have all needed stuff encapsulated in one tool - for now the best candidate for this is rally and what I see from their roadmap they are moving the the right direction
16:52:12 <DinaBelova> bsilverman ack
16:52:17 <rook> anyhoo... for people interested in browbeat : https://github.com/jtaleric/browbeat
16:52:25 <DinaBelova> #link https://github.com/jtaleric/browbeat
16:52:37 <rook> it stands up the infra to capture data -- it also runs Rally / Shaker / PerfKit benchmarks
16:52:38 <rohanion> yes, that's probably a topic for next discussions
16:53:00 <rook> DinaBelova yeah agreed one tool to rule them all :)
16:53:03 <DinaBelova> rook - thanks, I'll try to have a deep dive asap
16:53:23 <DinaBelova> rook - because in case if we want to reach this one-tool-perferct-future we need to start thinking now :)
16:53:35 <DinaBelova> anyway, that's all from my side
16:53:40 <DinaBelova> anything specific, folks?
16:53:46 <rook> Agreed... That is what we are trying to build with Browbeat :)
16:53:46 <rohanion> one tool to parse them all and in Kibana bind them
16:53:56 <rook> yup push all the metrics!
16:54:02 <DinaBelova> :D
16:54:22 <DinaBelova> ok, if nothing specific -thank you folks for taking part in meeting :)
16:54:30 <DinaBelova> special thanks to first-time attendees :)
16:54:31 <harlowja> browbeat, intersting
16:54:35 <harlowja> beat your brow?
16:54:38 <harlowja> eyebrow?
16:54:48 <rohanion> loooool
16:54:58 <DinaBelova> harlowja - you always make nice jokes close to the meeting end :)
16:54:59 <rohanion> nice band name btw
16:54:59 <harlowja> (do not try at home)
16:55:09 <harlowja> ;)
16:55:19 <lezbar__> bye o/
16:55:26 <DinaBelova> bye \o
16:55:28 <DinaBelova> #endmeeting