16:01:00 <mlavalle> #startmeeting neutron_performance
16:01:01 <openstack> Meeting started Mon Dec  3 16:01:00 2018 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:04 <openstack> The meeting name has been set to 'neutron_performance'
16:01:18 <slaweq> hi
16:01:20 <rubasov> o/
16:01:40 <mlavalle> how are you guys?
16:02:05 <rubasov> just started my deputy week, went fine so far
16:02:11 <mlavalle> lol
16:02:19 <rubasov> and you?
16:02:31 <slaweq> fine
16:03:09 <mlavalle> catching up with a lot of things. The combination of the trip to Berlin plus the Thanksgiving Holidays (I took one additional day off) really slowed me down
16:03:41 <mlavalle> #topic osprofiler and Rally
16:04:20 <mlavalle> Let's start talking a bit on where we are with adding osprofiler to the Rally job...
16:04:40 <rubasov> finally I made some progress with the rally job extension: https://review.openstack.org/620952
16:04:51 <rubasov> adding some floating IP operations
16:05:20 <mlavalle> Super nice. Thanks!
16:05:31 <rubasov> I still should upload another patch covering the floating IP association api ops
16:05:42 <rubasov> hoping to do that this week
16:05:48 <njohnston> o/
16:06:09 <haleyb> hi
16:06:36 <mlavalle> rubasov: so wait for the next revision before reviewing?
16:06:54 <rubasov> mlavalle: that will be a 2nd change
16:07:02 <rubasov> feel free to go ahead with this one
16:07:08 <mlavalle> ah ok, then I'll review this one today
16:07:30 <rubasov> also please leave response if you think other API ops are still missing
16:07:45 <mlavalle> yes of course
16:08:03 <mlavalle> slaweq: what's the situation with https://review.openstack.org/#/c/615350/?
16:08:15 <slaweq> so I didn't work on it too much
16:08:29 <slaweq> basically this would work but there is still performance problem
16:08:38 <slaweq> and osprofiler reports are generated too slow
16:08:47 <slaweq> I talked with andreykurilin about it today
16:09:10 <slaweq> and he told me that changing osprofiler backend to something else then redis may help with it
16:09:34 <slaweq> so I will need to add support for other backends in osprofiler devstack plugin first and test it
16:10:00 <mlavalle> do you have bandwidth?
16:10:15 <slaweq> I will try to do it this week
16:10:25 <slaweq> but if You want to take it, feel free :)
16:12:00 <mlavalle> if you think you can work on it this week, then why don't I ping you on Friday and see if you made any progress? I you don't I can look at it
16:12:04 <mlavalle> does that worlk?
16:12:21 <slaweq> yes, sounds good for me
16:12:24 <slaweq> thx
16:12:35 <mlavalle> on the contrary, thanks for working on this
16:12:55 <mlavalle> I think that getting osprofiler enabled in Rally is a very important step
16:12:55 <slaweq> yw :)
16:13:14 <slaweq> yes, and it may help a lot with many things in future
16:13:23 <mlavalle> that will start providing us with baseline data on where the bottlenecks are
16:14:03 <mlavalle> and from there we can start drawing some conclusions and even enable us to enlist other people who might help us.....
16:14:14 <mlavalle> which leads me to the next topic
16:15:16 <mlavalle> #topic Neutron performance conversations during Berlin Summit
16:16:40 <openstack> qwebirc62878: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
16:17:27 <mlavalle> Over the past few months I had been watching some videos of presentation on message brokers performance given by Ken Giusti (kgiusti I believe is his irc nickname)
16:18:02 <njohnston> qwebirc62878: cloudkitty meetings occur in #cloudkitty: http://eavesdrop.openstack.org/#CloudKitty_Team_Meeting
16:18:05 <mlavalle> qwebirc62878: we are running the Neutron performance meeting, which is scheduled at this time http://eavesdrop.openstack.org/#Neutron_Performance_sub-team_Meeting
16:18:49 <mlavalle> anyway, kgiusti works for Red Hat out of the same office where haleyb is located
16:19:14 <mlavalle> he is also a core team meber of oslo.messaging
16:19:45 <mlavalle> I was able to track him down during the Summit and had lunch with him
16:20:55 <mlavalle> he is willing to help us with performance issues in Neutron if they happen to involve the RPC channel
16:21:12 <mlavalle> I copied all of you in an email that I sent him last night
16:21:30 <njohnston> yes, thanks!
16:21:57 <mlavalle> I spent some time yesterday revieiwing the places where we have RPC channel related calls to osprofiler
16:22:15 <mlavalle> it seems to me that we are pretty much covering all the important components
16:22:43 <mlavalle> but as usual, I might be wrong, so please feel free to respond and correct me
16:23:15 <mlavalle> this is where I say that it is important to get osprofiler enabled in the Rally job
16:23:58 <mlavalle> is can see / show in the data of Rally that the RPC channl is indeed a problem. kgiusti is willing to help to dig further
16:24:41 <mlavalle> haleyb: he is a short, slim guy with a pony tail, in his mid 50s, in case you see him around
16:25:08 <haleyb> mlavalle: ack, i don't think i've met him, maybe someone else in his group though
16:25:44 <haleyb> so is there a specific bug, or just room for improvement here?
16:25:51 <rubasov> so you suspect raw messaging performance to be a bottleneck, right? but we need data of course to confirm or deny
16:26:23 <rubasov> would that lead to rabbitmq tuning, or ditching rabbit for something else?
16:26:41 <mlavalle> haleyb: at this point no bug report yet. as rubasov says, I would like to get some data first from Rally
16:27:21 * slaweq needs to go afk for a while, will read log from meeting later, sorry
16:27:57 <mlavalle> rubasov: yes, that may lead to a range of possibilities: code optimization on our part, rabbitmq tunning, changing the messaging broker
16:28:18 <rubasov> mlavalle: got it, thanks
16:28:53 <haleyb> i know there are people looking at different message brokers as well
16:29:32 <mlavalle> yes, kgiusti has been cooperating with a group in a French reasearch institute, Inria
16:30:03 <mlavalle> the presentations he has ben giving on the subject are in cooperation with them
16:30:23 <mlavalle> so that leads me to the other part of my update...
16:31:04 <mlavalle> during the lunch I had with kgiusti, he introduced me to members of the Inria team
16:31:27 <mlavalle> and it turns out that they have about 10 people full time working on OpenStack performance
16:32:05 <mlavalle> They have this EnOS framework (https://enos.readthedocs.io/en/stable/)
16:32:36 <njohnston> that looks interesting
16:32:37 <mlavalle> Repo here: https://github.com/BeyondTheClouds/enos
16:32:50 <rubasov> wow, that sounds like serious work
16:33:32 <mlavalle> which is part of a larger initiative, which Red Hat is sponsoring: https://beyondtheclouds.github.io/\
16:34:59 <mlavalle> This reasearch institute, Inria, seems to be closely associated with https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home, which is a testbed for cloud infrastructure
16:35:43 <mlavalle> earlier today I copied all of you in an email I sent to Javier Balderrama, one of the Inria researchers
16:36:21 <mlavalle> Based on all this, I have an initial vision of our performance work as follows:
16:36:49 <mlavalle> 1) we use rally and osprofiler to get an initial characterization of Neutron performance
16:37:11 <mlavalle> 2) Based on ^^^^, we ask Ken to help us improve the rpc channel behavior
16:38:09 <mlavalle> 3) with enough data and a well defined test, we might conduct a large scale experiment with the inria guys. One of them told over lunch that they regularly conduct tests with a 1000 conducts on Grid'5000
16:38:26 <mlavalle> 1000 computes^^^^
16:38:56 <njohnston> wow
16:39:30 <rubasov> congrats for organizing this
16:40:03 <njohnston> yes indeed, this is great collaboration
16:40:32 <mlavalle> Their interest is purely research. So if we can organize something where thay and we can learn, it might be a great partnership
16:41:38 <mlavalle> This is my update for today
16:42:20 <mlavalle> so for the next meeting:
16:42:32 <mlavalle> 1) Let's make progress with osprofiler and Rally
16:42:45 <mlavalle> 2) Finish adding resources to the Rally test
16:42:55 <rubasov> ack
16:43:00 <mlavalle> 3) See the feedback we get from Ken and the Inria team
16:43:17 <mlavalle> does this sound like a plan?
16:43:23 <rubasov> it does
16:44:07 <mlavalle> ok....
16:44:16 <mlavalle> #topic On demand agenda
16:44:36 <mlavalle> Anything else we should discuss today?
16:44:58 <rubasov> I don't have anything else for today
16:45:37 <njohnston> me neither
16:45:52 <mlavalle> ok, thanks for attending
16:45:58 <mlavalle> Have a great week
16:46:03 <mlavalle> #endmeeting