16:00:10 <DinaBelova> #startmeeting Performance Team
16:00:11 <openstack> Meeting started Tue May 17 16:00:10 2016 UTC and is due to finish in 60 minutes.  The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:14 <openstack> The meeting name has been set to 'performance_team'
16:00:20 <DinaBelova> hey folks o/
16:00:32 <ad_rien_> Hi
16:00:32 <rohanion> Hi Dina
16:00:38 <rook> hey
16:00:48 <DinaBelova> wow, I'm glad people did not forget about the meeting :)
16:00:52 <rook> lol
16:00:54 <DinaBelova> it's a after-summit recovery :)
16:01:10 <DinaBelova> let's wait for a few more moments
16:01:39 <gokrokve> 0/
16:01:42 <DinaBelova> it's sad I do not see AugieMena yet
16:01:58 <rook> hm
16:02:01 <DinaBelova> let's hope he'll join
16:02:08 <rook> I can see if I can find someone who might know where he is
16:02:17 <DinaBelova> rook ack
16:03:20 <DinaBelova> anyway, let's get started
16:03:30 <DinaBelova> with action items first
16:03:31 <DinaBelova> #topic Action Items
16:04:19 <DinaBelova> in fact we had two of them
16:04:31 <DinaBelova> one on myself gerarding Keystone performance multi node testing
16:04:45 <luzC> o/
16:04:53 <DinaBelova> sadly i do not have something specific to share regarding summit and my vacation >_<
16:05:02 <DinaBelova> I'll leave this on myself for the next time
16:05:19 <rook> o/
16:05:25 <DinaBelova> #action DinaBelova share status of Keystone multinode testing
16:05:26 <rook> no word from Augie or Marcio
16:05:27 <DinaBelova> luzC o/
16:05:31 <DinaBelova> rook ack
16:05:41 <DinaBelova> so I'll leavie his action itme with no changes
16:05:48 <DinaBelova> #action AugieMena share details about some v3 limitations in Rally
16:06:15 <DinaBelova> he planned to share his results about some v3 limitations and related difficulties in rally
16:06:24 <DinaBelova> so let's keep this till next time
16:06:29 <rook> I have heard of other users having v3 issues with Rally
16:06:34 <rook> mostly around keystone.
16:06:43 <DinaBelova> rook - any specific issues?
16:06:43 <rook> I haven't personally hit it.
16:06:49 <rook> negative.
16:06:57 * rook wonders if there are launchpads for the issues?
16:07:09 <DinaBelova> rook can you please point the people boris-42 and I should talk about these issues with?
16:07:54 <DinaBelova> rook it is - both for keystone and rally, but if there are some specific ones...
16:08:25 <DinaBelova> ok, so anyway i guess it'll be better to find AugieMena :)
16:08:42 <DinaBelova> so I'll probably jump to the next topic
16:08:48 <DinaBelova> #topic Summit follow-up
16:09:17 <DinaBelova> in fact I loved how this summit went from the performance topic perspective
16:09:38 <DinaBelova> I want to quickly go through the interesting sessions
16:09:54 <DinaBelova> for folks who had no chance to attend - like rohanion and gokrokve
16:10:23 <DinaBelova> first of all the working group session we had on Wednesday
16:10:25 <DinaBelova> #link https://etherpad.openstack.org/p/newton-performance-team
16:11:10 <DinaBelova> I loved the way how it went - people seemed to be enthusiastic about the progress made by team and suggested several things to be done during Newton
16:11:58 <DinaBelova> I need to go through the suggestions and notes about some other performance benchmarks people are running
16:12:18 <DinaBelova> for instance there was a suggestion to take a look on the SPECCloud - a benchmark from the SPEC organization
16:12:44 <DinaBelova> sadly I did not have a chance to take a look on it yet, but that's in my todo list for sure
16:13:44 <rook> DinaBelova I helped build that workload so feel free to ask questions :)
16:13:54 <gokrokve> There are a lot of ideas for Newton. Do we have priorities for them?
16:14:08 <DinaBelova> rook cool
16:14:11 <DinaBelova> gokrokve - there were several suggestions about what can be tested in the networking test as well - like we planned to focus on Neutron, Contrail, Midokura, Calico - but folks asked to take a look on the OpenDaylight and DragonFlow as well if possible
16:14:32 <rook> OVN too :D
16:14:38 <DinaBelova> rook indeed
16:15:08 <DinaBelova> gokrokve - I'll check this with ilyashakhat - as we surely have limited working cycles on this task
16:15:22 <DinaBelova> I'm sure we won't cover everything, so let's decide what can be covered
16:15:53 <DinaBelova> #action DinaBelova ilyashakhat prioritize Networking testing ideas from https://etherpad.openstack.org/p/newton-performance-team
16:15:59 <gokrokve> Sure. We can actually vote for each item in this etherpad.
16:16:21 <DinaBelova> gokrokve - good idea, let's do it
16:16:29 <gokrokve> Just put +1 for an item which looks interesting and important for you.
16:16:47 <DinaBelova> gokrokve ++
16:17:08 <DinaBelova> #info let's vote for most interesting Networking testing ideas from https://etherpad.openstack.org/p/newton-performance-team with simple +1
16:17:40 <DinaBelova> in fact all other ideas were very welcomed
16:17:44 <DinaBelova> as well :)
16:18:03 <DinaBelova> I should mention several other sessions as well
16:18:32 <DinaBelova> two of them were held as a pat of Large Deployment Team effort during the summit
16:18:37 <DinaBelova> #link https://etherpad.openstack.org/p/AUS-ops-Large-Deployments-Team
16:19:43 <DinaBelova> there were mostly deployment and scale issues and interesting moments discussed, several performance related questions as well
16:20:15 <DinaBelova> I just recommend to go through this etherpad and take a look on the issues real operators and active LDT members are focused on
16:20:18 <DinaBelova> to be on track
16:20:40 <DinaBelova> there was interesting test idea for our team proposed as well
16:21:01 <DinaBelova> folks were concerned about number of messages Neutron can generate
16:21:57 <DinaBelova> Things like security group changes can generate LOTS of messages - we can try to create some synthetic test via rally and measure number of messages that are trying to use messaging bus
16:22:08 <DinaBelova> we have synthetic numbers already
16:22:32 <rook> I have a test to look at ovsct (new firewall driver) vs the iptables work.
16:22:41 <DinaBelova> rook good idea
16:22:42 <rook> We have throughput numbers, but now we are looking at the control-plane as well.
16:23:03 <rook> Using Rally-plugins i created a scenario to create N networks and attachN networks to the guest.
16:23:17 <rook> which will also bloat the firewall rules since they are mostly mac address driven.
16:23:59 <DinaBelova> #info let's play around the Networking/Messaging test ideas proposed in https://etherpad.openstack.org/p/AUS-ops-Large-Deployments-Team
16:24:04 <DinaBelova> rook ack, thank you sir
16:25:02 <DinaBelova> there is one more session that looks really interesting for myself
16:25:15 <DinaBelova> not the session in fact, but operators meetup
16:25:18 <DinaBelova> #link https://etherpad.openstack.org/p/AUS-ops-informal-meetup
16:25:33 <DinaBelova> it has super long etherpad as a result :D
16:26:26 <DinaBelova> in fact I'll just recommend to go through the etherpad at least and see what topics are hot for now from the operators perspective
16:27:06 <DinaBelova> lots of issues are related to upgradability and deployment, but at the same time there is a bunch of scalability / performance issues mentioned and discussed
16:27:48 <DinaBelova> right now I'm not ready to point on the specific ones, as lots of them (as it turned out) are either fixed in the latest OpenStack or have nice workarounds
16:28:02 <DinaBelova> so I need additional research on this etherpad now
16:28:12 <DinaBelova> hope to do it soon
16:28:55 <DinaBelova> I think that's all I wanted to share after a summit - at least for now
16:29:01 <DinaBelova> any comments?
16:29:15 <ad_rien_> Hi DinaBelova
16:29:21 <DinaBelova> ad_rien_ hey :)
16:29:36 <ad_rien_> Maybe we can discuss a bit about scalability evaluations we would like to perform at Inria
16:29:52 <DinaBelova> ad_rien_ - good idea, let's move it to the end of meeting
16:29:56 <ad_rien_> on top of Grid'5000 (see link available in the pad on just google for grid'5000)
16:29:58 <DinaBelova> to the Open Discussions part
16:30:02 <ad_rien_> thanks
16:30:08 <DinaBelova> ack
16:30:21 <DinaBelova> ok, so let's jump to the OSprofiler
16:30:26 <DinaBelova> #topic OSProfiler weekly update
16:30:30 <DinaBelova> rohanion hey o/
16:30:34 <rohanion> Hiiii
16:30:38 <rohanion> so let's begin
16:30:39 <DinaBelova> looks like you had 3 weeks :D
16:30:42 <DinaBelova> not just one :)
16:30:50 <DinaBelova> sure, the floor is yours
16:31:28 <rohanion> Ok, first of all - I've completed writing the new architecture
16:31:39 <rohanion> now it's time for code review
16:31:50 <DinaBelova> new drivers architecture?
16:31:52 <rohanion> https://review.openstack.org/#/c/247005/ here's the base commit
16:31:55 <rohanion> yes
16:32:02 <DinaBelova> #link https://review.openstack.org/#/c/247005/
16:32:22 <rohanion> also I've split the ceilometer driver from the base commit
16:32:31 <rohanion> #link https://review.openstack.org/#/c/251343/
16:33:03 <rohanion> MongoDB driver is working and only small fixes have to be written
16:33:07 <rohanion> #link https://review.openstack.org/#/c/310530/
16:33:48 <DinaBelova> rohanion any numbers about MongoDB vs Ceilometer ?
16:33:57 <rohanion> also, I've commited changes to cinder, glance and horizon for them to support the new osprofiler api
16:34:02 <rohanion> #link https://review.openstack.org/#/c/315676/
16:34:12 <rohanion> #link https://review.openstack.org/#/c/316799/
16:34:25 <rohanion> #link https://review.openstack.org/#/c/273085/
16:34:26 <DinaBelova> boris-42 harlowja ^^ we need to go through this bunch of commits :S
16:35:00 <DinaBelova> rohanion thank you sir :D
16:35:01 <rohanion> DinaBelova: yes, building a full report with mongodb driver takes 0.7 seconds
16:35:14 <DinaBelova> it looks like I have looooooots of stuff to review after the vacation :D
16:35:17 <rohanion> compare that with 5 seconds for ceilometer
16:35:43 <DinaBelova> rohanion I guess horizon folks are much happier with these numbers :)
16:36:20 <DinaBelova> rohanion ok, I'll start reviewing on Thursday
16:36:43 <DinaBelova> rohanion please ping other core reviewers as well - boris-42 and harlowja
16:37:06 <rohanion> Yes, Timur is very happy but there's an issue with messaging in horizon, we have to create the RPC transport ourselves and it's tricky as for me
16:37:22 <rohanion> Will do, thank you!
16:37:22 <DinaBelova> rohanion that's true
16:37:26 <DinaBelova> ack
16:37:35 <DinaBelova> ok, it looks like that's all for OSprofiler part
16:37:47 <DinaBelova> let's jump to the Open Discussions
16:37:52 <DinaBelova> #topic Open Discussion
16:38:05 <DinaBelova> I have no specific items, but we have ad_rien_ from Inria today :))
16:38:11 <ad_rien_> ;)
16:38:12 <DinaBelova> #link https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home
16:38:12 <ad_rien_> Thanks
16:38:50 <ad_rien_> To make a long story short, we are working on massively distributed clouds at Inria (from the research viewpoint) and we are interested to understand the current limit of the OpenStack core components
16:39:11 <ad_rien_> We briefly exchanged with some folks during the summit and it looks that evaluating the zeroMQ solution
16:39:21 <ad_rien_> make sense as a first TODO.
16:39:45 <DinaBelova> ad_rien_ - in fact rohanion did several 0MQ researches already
16:39:55 * DinaBelova looking if that was published to the performance-docs
16:40:05 <rohanion> it was, as I remember
16:40:19 <rohanion> I pushed the changes a long time ago
16:40:31 <DinaBelova> rohanion ok, but I do not see them on the http://docs.openstack.org/developer/performance-docs/#
16:40:43 <DinaBelova> probably because ilyashakhat was finishing your work
16:40:50 <ad_rien_> It would be great if we can have a link to such results
16:41:23 <ad_rien_> So we have few questions regarding that point: What can be a representative enough experimental protocol ?
16:41:23 <DinaBelova> rohanion - can you please investigate if the 0MQ results will be publiched soon with Ilya?
16:42:00 <rohanion> ah, I remember. there's a bug that blocks HA for 0mq
16:42:08 <DinaBelova> ad_rien_ we have composed Messaging Bus testing plan some time ago - http://docs.openstack.org/developer/performance-docs/test_plans/mq/plan.html
16:42:25 <rohanion> sure I will
16:42:55 <ad_rien_> Ok so what is the best way to move forward on that point ?
16:43:00 <DinaBelova> with several test cases - RPC Call Throughput Test, RPC Cast Throughput Test, Notification Throughput Test
16:43:11 <ad_rien_> 1./ wait for the rohanion's answer regarding the previous experiments
16:43:38 <rohanion> I'll look for my notes, they have to be published somewhere
16:43:43 <DinaBelova> 2./ ad_rien_ - please go through the test plan http://docs.openstack.org/developer/performance-docs/test_plans/mq/plan.html and see if it looks ok for you
16:43:45 <ad_rien_> 2./ perform preliminary experiments according to the link you pasted DinaBelova
16:43:54 <rohanion> I suggest doing both
16:43:55 <DinaBelova> yes, indeed
16:43:58 <rohanion> ad_rien_
16:44:10 <DinaBelova> probably you'll have some ideas to be added to this test plan
16:44:10 <ad_rien_> ok if we have questions, is there a particular person to contact ?
16:44:29 <DinaBelova> myself, ilyashakhat and rohanion can answer them :)
16:44:32 <DinaBelova> ad_rien_ ^^
16:44:34 <ad_rien_> cool thanks
16:44:49 <rook> ad_rien_: might i ask why the ZeroMQ direction?
16:45:07 <rook> ad_rien_ is it due to the distributed environment you have?
16:45:10 <rohanion> the best openstack doctors recommend giving up on zmq until the sockets issue is fixed
16:45:19 <rook> rohanion: exactly.
16:45:27 <DinaBelova> ad_rien_ - please notice that this specific test plan is playing around oslo.messaging + SOME_DRIVER testing via oslo simulator tool
16:45:44 <ad_rien_> We are working on a particular use-case related to Fog/Edge cloud computing solutions (i.e. a cloud that is spread over a significant number of small data centers)
16:46:01 <ad_rien_> DinaBelova:  ok thanks
16:46:13 <DinaBelova> probably you'll find out this is not 100% suitable for you - we can think either on the tool improvement or test plan extension, etc.
16:46:33 <ad_rien_> We will give a look to the different information you gave us and come back to you soon
16:46:54 <ad_rien_> The second experiment we would like to perform is a a scalability test of OpenStack
16:47:01 <ad_rien_> with Rally.
16:47:21 <ad_rien_> We are working on that right now
16:47:41 <DinaBelova> ad_rien_ - currently 0MQ driver has one socket issue rohanion and rook mentioned ^^ - with node number growing up, 0MQ eats more sockets on the cloud controller nodes... too many
16:47:42 <ad_rien_> we have several ways to achieve it on top of Grid'5000
16:47:50 <DinaBelova> rohanion - can you find a link to the bug?
16:47:55 <DinaBelova> ad_rien_ ack
16:48:17 <msimonin> the socket issue is it a bug from zeroMQ ?
16:48:26 <ad_rien_> or a way OpenStack is using it ?
16:48:32 <ad_rien_> (thanks msimonin)
16:49:21 <DinaBelova> msimonin ad_rien_ - it's a logical continuation of 0MQ usage
16:49:23 <DinaBelova> in openstack
16:49:36 <rohanion> https://bugs.launchpad.net/oslo.messaging/+bug/1555007
16:49:37 <openstack> Launchpad bug 1555007 in oslo.messaging "[zmq] ZMQ driver eats too many TCP sockets" [Undecided,Fix released] - Assigned to Oleksii Zamiatin (ozamiatin)
16:49:39 <rohanion> here we go
16:49:52 <ad_rien_> Ok thanks
16:49:58 <DinaBelova> too many peer-to-peer connectivity without any proxies from the 0mq driver that's written right now
16:50:11 <ad_rien_> we are going to give a careful look
16:50:13 <rohanion> proxies don't help as much as we want them
16:50:22 <DinaBelova> rohanion that's true
16:50:30 <DinaBelova> I just used this as an example
16:51:07 <rohanion> we have spotted a slight decrease in the number of sockets, but still it eats tons of them
16:51:23 <rohanion> but ten thousand less than without using proxies
16:51:24 <rook> I saw some blueprints for reusing sockets
16:51:39 <rook> ah ha.
16:51:49 * rook sees rohanion posted it.
16:51:56 <DinaBelova> and that's normal for the 0MQ protocol, but not appropriate yet for the real usage
16:52:01 <rohanion> was it me? O_o
16:52:02 <DinaBelova> rohanion, rook indeed
16:52:03 <DinaBelova> :D
16:52:14 <ad_rien_> Ok is there any other information we should know about the couple zeroMQ/openStack  before starting on that action :-P ?
16:52:39 <DinaBelova> ad_rien_ nothing specific yet
16:52:48 <rohanion> for now bunny is much more reliable.
16:52:53 <msimonin> :)
16:52:57 <DinaBelova> we'll check if something else can be found to help you
16:53:20 <ad_rien_> ok just for your information we plan to work on that aspect during this summer (we are currently hiring an engineer that will work on scalability evaluations)
16:53:52 <DinaBelova> ack, cool
16:53:56 <DinaBelova> anything else?
16:53:57 <ad_rien_> so regarding the second experiments about the scalability evaluation, there are few items on the pad
16:54:17 <ad_rien_> (see New tests planned section)
16:54:27 <ad_rien_> #link https://etherpad.openstack.org/p/newton-performance-team
16:54:50 <msimonin> We had some discussions about qpid dispatch router at the summit
16:54:57 <DinaBelova> ad_rien_ some of them we'll cover on Mirantis-Intel lab, but I think two labs will be even better :D
16:55:09 <ad_rien_> yes that's exactly my poin
16:55:11 <ad_rien_> t
16:55:16 <ad_rien_> how can we merge the efforts.
16:55:25 <msimonin> any thought on qpid dispatch router as brokerless communication solutioj
16:55:28 <msimonin> solution*
16:55:30 <harlowja> DinaBelova u alive???!!??
16:55:41 <DinaBelova> harlowja hahahha
16:55:46 <rohanion> hehey joshua
16:55:51 <rohanion> how have u been?
16:56:05 <DinaBelova> msimonin - I dunno myself
16:56:06 <ad_rien_> As it is mentioned on the pad, there are different ways to perform such evaluations (using docker, VM, tools like infrastructure emulators)
16:56:19 <DinaBelova> I guess we need to ping some oslo.messaging guys for this
16:56:22 <DinaBelova> dims_ ?
16:57:16 <DinaBelova> msimonin - I'll check it with our oslo.messaging folks
16:57:17 <dims_> hey DinaBelova
16:57:22 <DinaBelova> hey :)
16:57:28 <ad_rien_> it would be nice if we can use the same deployment scripts
16:57:30 <DinaBelova> dims_ - it was a question from msimonin
16:57:30 <dims_> what's up?
16:58:28 <dims_> msimonin : DinaBelova : qpid dispatch router? we'll need kgiusti
16:58:33 <DinaBelova> dims_ about 0MQ current driver limitations and other drivers and solutions like qpid dispatch router as brokerless communication and it's comparison
16:58:36 <DinaBelova> dims_ ack
16:58:48 <msimonin> dims_:  ok thanks
16:58:59 <DinaBelova> ad_rien_ - sure
16:59:09 <ad_rien_> did you start to work on that on your side ?
16:59:10 <DinaBelova> once we'll finalize test plans
16:59:20 <dims_> DinaBelova : 0MQ will support direct connections for smaller number of nodes and broker-ed for larger number of nodes
16:59:34 <DinaBelova> ad_rien_ - well, I did not find this out yet :D my vacation is till thursday
16:59:39 <DinaBelova> I'm a bit out of context
16:59:49 <DinaBelova> ad_rien_ - I'll update you once I'll have any info
16:59:55 <ad_rien_> ok so the first action is to work on the experimental protocol ?
17:00:15 <dims_> DinaBelova : we should call kgiusti and ozamiatin for next meeting :)
17:00:45 <DinaBelova> dims_ ack
17:00:59 <DinaBelova> ad_rien_ i believe yes
17:01:05 <DinaBelova> together with rohanion from our side
17:01:13 <DinaBelova> ok, so we're out of time in fact
17:01:25 <DinaBelova> so let's end the discussion for now
17:01:27 <ad_rien_> yes I understand, do not worry
17:01:28 <ad_rien_> ok
17:01:39 <DinaBelova> and collect all info about the items discussed today :)
17:01:42 <DinaBelova> thanks!
17:01:45 <DinaBelova> #endmeeting