| *** hockeynut has quit IRC | 01:36 | |
| *** markvoelker has joined #openstack-performance | 01:44 | |
| *** markvoelker_ has joined #openstack-performance | 01:46 | |
| *** markvoelker has quit IRC | 01:50 | |
| *** rfolco has quit IRC | 02:18 | |
| *** markvoelker has joined #openstack-performance | 04:21 | |
| *** ilyashakhat has joined #openstack-performance | 04:22 | |
| *** markvoelker_ has quit IRC | 04:22 | |
| *** markvoelker has quit IRC | 04:28 | |
| *** markvoelker has joined #openstack-performance | 05:29 | |
| *** ilyashakhat has quit IRC | 05:31 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** ilyashakhat has joined #openstack-performance | 05:40 | |
| *** ilyashakhat has quit IRC | 05:50 | |
| *** msimonin has joined #openstack-performance | 05:54 | |
| *** msimonin has quit IRC | 05:57 | |
| *** msimonin has joined #openstack-performance | 05:57 | |
| *** msimonin has quit IRC | 05:59 | |
| *** msimonin has joined #openstack-performance | 05:59 | |
| *** msimonin has quit IRC | 06:00 | |
| *** pcaruana has joined #openstack-performance | 06:14 | |
| *** markvoelker has joined #openstack-performance | 06:36 | |
| *** markvoelker has quit IRC | 06:42 | |
| *** f13o has quit IRC | 06:53 | |
| *** msimonin has joined #openstack-performance | 07:20 | |
| *** markvoelker has joined #openstack-performance | 07:38 | |
| *** markvoelker has quit IRC | 07:43 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-performance | 08:04 | |
| *** f13o has joined #openstack-performance | 08:07 | |
| *** dewanee has joined #openstack-performance | 08:25 | |
| *** f13o has quit IRC | 08:31 | |
| *** markvoelker has joined #openstack-performance | 08:39 | |
| *** markvoelker has quit IRC | 08:44 | |
| *** markvoelker has joined #openstack-performance | 09:40 | |
| *** markvoelker has quit IRC | 09:44 | |
| *** markvoelker has joined #openstack-performance | 11:09 | |
| *** bvanhav has joined #openstack-performance | 11:19 | |
| *** markvoelker has quit IRC | 11:20 | |
| *** rfolco has joined #openstack-performance | 11:47 | |
| *** eandersson_ has quit IRC | 13:47 | |
| *** eandersson has joined #openstack-performance | 13:56 | |
| *** pcaruana has quit IRC | 14:02 | |
| *** hieulq_ has joined #openstack-performance | 14:08 | |
| *** zz_dimtruck is now known as dimtruck | 14:17 | |
| *** pcaruana has joined #openstack-performance | 14:17 | |
| *** tinbar has joined #openstack-performance | 14:25 | |
| *** tinbar has quit IRC | 14:25 | |
| *** hieulq__ has joined #openstack-performance | 14:31 | |
| *** hieulq_ has quit IRC | 14:33 | |
| *** bwallis has quit IRC | 14:55 | |
| openstackgerrit | Ilya Shakhat proposed openstack/performance-docs: Add Neutron ML2 OVS plugin performance test plan https://review.openstack.org/359255 | 14:56 |
|---|---|---|
| *** hockeynut has joined #openstack-performance | 14:58 | |
| *** hieulq__ has quit IRC | 15:01 | |
| *** hieulq_ has joined #openstack-performance | 15:01 | |
| *** klindgren has quit IRC | 15:17 | |
| *** bwallis has joined #openstack-performance | 15:21 | |
| *** klindgren has joined #openstack-performance | 15:30 | |
| *** bwallis has quit IRC | 15:37 | |
| *** rohanion has joined #openstack-performance | 15:50 | |
| *** aignatev has joined #openstack-performance | 15:51 | |
| DinaBelova | rohanion, aignatev o/ | 15:56 |
| DinaBelova | klindgren wow, you joined the channel right before the meeting :) does it mean you're going to attend? | 15:56 |
| DinaBelova | :) | 15:56 |
| *** pcaruana has quit IRC | 15:56 | |
| klindgren | um | 15:57 |
| klindgren | more like I had to fix some stuff before I have to attend 2 meetings | 15:57 |
| klindgren | at once | 15:57 |
| *** ad_rien_ has joined #openstack-performance | 15:57 | |
| klindgren | (this would be #3) | 15:58 |
| DinaBelova | :D heh, I know you're having the daily meetings at this time slot | 15:58 |
| DinaBelova | I hoped to discuss new time slots this meeting | 15:58 |
| ad_rien_ | 15:30 sounds good on our side ;) | 15:58 |
| ad_rien_ | (Hi all BTW) | 15:58 |
| DinaBelova | klindgren and for you? is 15:30 UTC (8:30 PST better? as we usually have meeting for 30-45 mins) | 15:59 |
| DinaBelova | ad_rien_ o/ | 15:59 |
| ad_rien_ | o/ | 15:59 |
| DinaBelova | #startmeeting Performance Team | 16:00 |
| openstack | Meeting started Tue Aug 23 16:00:07 2016 UTC and is due to finish in 60 minutes. The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| openstack | The meeting name has been set to 'performance_team' | 16:00 |
| DinaBelova | I think we may start with organizational questions | 16:00 |
| DinaBelova | as the new time for the meeting :) | 16:00 |
| DinaBelova | #topic New time for the meeting | 16:00 |
| DinaBelova | so we have several options | 16:01 |
| DinaBelova | 16:00 UTC (current time, really tough) vs 15:30 UTC vs 17:30 UTC vs 18:00 UTC | 16:01 |
| *** AugieMena has joined #openstack-performance | 16:01 | |
| rook | 1200 EST isn't great.... :) | 16:02 |
| rook | o/ | 16:02 |
| DinaBelova | rook indeed | 16:02 |
| DinaBelova | from what I heard from people 30 minutes earlier (15:30 UTC) seems to be the best option for now | 16:02 |
| rohanion | hey everyone :) | 16:02 |
| DinaBelova | rohanion o/ | 16:03 |
| *** rcherrueau has joined #openstack-performance | 16:03 | |
| DinaBelova | 15:30 UTC (11:30 EST, 8:30 PST) is something I've heard from AugieMena and ad_rien_ and other Inria folks at least | 16:03 |
| ad_rien_ | for French Inria Guys 15:30 UTC is a good choice - (17:30 can be a second one) | 16:03 |
| rook | +1 to 1530 | 16:04 |
| DinaBelova | rook ack | 16:04 |
| DinaBelova | I want to grad confirmation from klindgren - if at least he'll be able to attend from GoDaddy side on more or less weekly basis it'll be cool | 16:04 |
| DinaBelova | I know that for harlowja it'll be too early :D | 16:05 |
| DinaBelova | glad -> grab ** | 16:05 |
| DinaBelova | although these two are having right now a daily meeting ^^, so let's wait for their response later | 16:05 |
| DinaBelova | #info preliminary agreed on moving the meeting 30 minutes earlier on Tuesdays, 15:30 UTC (11:30 EST, 8:30 PST) | 16:06 |
| DinaBelova | rohanion am I right that for Moscow it'll be convenient as well? | 16:06 |
| DinaBelova | rohanion it'll be 18:30 | 16:07 |
| rohanion | yeah, no problem | 16:07 |
| rohanion | 18:30 RTZ-2 | 16:07 |
| DinaBelova | ok, cool, I suspected this, but wanted to collect the information | 16:07 |
| *** bwallis has joined #openstack-performance | 16:07 | |
| DinaBelova | ok, so I'll send a final update via email thread | 16:07 |
| DinaBelova | so let's jump to the current progress on test plans | 16:08 |
| DinaBelova | #topic Current progress on the planned tests | 16:08 |
| DinaBelova | from Mirantis side we (finally) were able to move forward with labs configuration | 16:08 |
| DinaBelova | our 200 nodes lab is under intensive Neutron testing for now (dataplane tests with huge enough number of Neutron objects and VMs) | 16:09 |
| *** pcaruana has joined #openstack-performance | 16:09 | |
| DinaBelova | and it looks like we finally were able to filter all non-working nodes from 500 nodes lab, so we'll have 400+ nodes Openstack cluster (installed by MOS) | 16:10 |
| DinaBelova | so we'll run http://docs.openstack.org/developer/performance-docs/test_plans/control_plane/plan.html against it | 16:10 |
| DinaBelova | hopefully this will start tomorrow | 16:10 |
| rook | MOS ? | 16:10 |
| DinaBelova | Mirantis OpenStack (way of deploying) | 16:10 |
| rook | Ah MOS != Fuel ? | 16:11 |
| DinaBelova | rook yep :D | 16:11 |
| DinaBelova | MOS includes OpenStack packages with more fixes and backports | 16:11 |
| msimonin | From inria side we made some progress on the 1000 nodes test plan | 16:11 |
| DinaBelova | than usual stable/* usually have | 16:11 |
| DinaBelova | but deployment tool itself is Fuel, yes | 16:11 |
| DinaBelova | msimonin cool! | 16:12 |
| DinaBelova | please share some details | 16:12 |
| msimonin | but … | 16:12 |
| msimonin | :) | 16:12 |
| DinaBelova | :D | 16:12 |
| msimonin | we still have some performance issue when scaling with 500+ computes | 16:12 |
| DinaBelova | msimonin you use fake computes? | 16:12 |
| msimonin | for the record we are using a kolla based deployment | 16:12 |
| luzC | o/ | 16:12 |
| DinaBelova | luzC o/ | 16:12 |
| msimonin | DinaBelova: yes fake drivers | 16:12 |
| rook | msimonin: what sort of scaling issues? Control plane? | 16:13 |
| DinaBelova | msimonin ok, so kolla + fake driver | 16:13 |
| DinaBelova | what kind of issues? | 16:13 |
| luzC | hello Dina | 16:13 |
| msimonin | actually, let me share a document that we are maintaining | 16:13 |
| DinaBelova | msimonin it'll be cool | 16:13 |
| msimonin | #link https://pad.inria.fr/p/FK18ZSyjFyLPITfE | 16:13 |
| DinaBelova | msimonin ok, I see " very slow apis when using 500 computes" | 16:14 |
| msimonin | apis + mariadb are behind the haproxy | 16:14 |
| msimonin | yep | 16:14 |
| msimonin | that's the issue | 16:14 |
| rook | the issue is haproxy? | 16:14 |
| DinaBelova | does it relate to all OpenStack REST APIs? | 16:14 |
| msimonin | rook: maybe haproxy, but we need more insight | 16:15 |
| DinaBelova | msimonin I suspect some parameters of HAproxy may influence here indeed on the performance, I'll agree with rook here | 16:16 |
| msimonin | yes it's my feeling too | 16:16 |
| msimonin | do you have some experience with haproxy ? | 16:16 |
| DinaBelova | msimonin not too big sadly | 16:16 |
| rook | msimonin do you know at what # of computes you start seeing the problem? | 16:16 |
| msimonin | 100 computes is ok, 500 isn't | 16:17 |
| rook | well you go 5x :) | 16:17 |
| msimonin | sure :) | 16:17 |
| DinaBelova | msimonin I can suggest you to ping alwex (Alex Shaposhnikov) and try to discuss this with him, he may have some ideas | 16:17 |
| msimonin | cool, I'll do that | 16:17 |
| rook | msimonin we have a monitoring solution to help track down what might be slowing down | 16:17 |
| rook | using collectd/graphite/grafana | 16:18 |
| rook | might provide insight into the slowness or the possible culprit. | 16:18 |
| msimonin | actually we are using cadvisor/influx/grafana | 16:18 |
| msimonin | but it just keep track of basics metrics | 16:18 |
| DinaBelova | #info Mirantis: Neutron testing on 200 nodes in progress, 400 nodes cluster almost installed for baseline performance testing | 16:18 |
| DinaBelova | #info Inria facing issues with 1000 nodes emulation experiment: https://pad.inria.fr/p/FK18ZSyjFyLPITfE | 16:19 |
| rook | msimonin: sure, we break out per-process | 16:19 |
| DinaBelova | msimonin I think you may try the same method rook is talking about + let's ask alwex to help | 16:20 |
| DinaBelova | #action DinaBelova msimonin alwex find out what's slowing down OpenStack REST APIs in the Inria 1000 nodes experiment | 16:20 |
| msimonin | rook: do you have what metrics can help tracking the issue ? | 16:21 |
| msimonin | do you know* | 16:21 |
| ad_rien_ | Since we built our deployment framework on top of Kolla, we get the ha-proxy by default…. DinaBelova do you think it can be possible to see how you are deploying the different containers on your side ? | 16:21 |
| ad_rien_ | For us it is not so complex/time consuming to redeploy a set of containers on top of G5K | 16:22 |
| DinaBelova | ad_rien_ I think that's possible, let's anyway communicate this with Alex as he was originating the experiment | 16:22 |
| rook | msimonin: since it seems to be based on the # of computes... we could look at the msging services (rabbit?), haproxy, and possibly the individual nova services. since each of the compute nodes will be checking in on a interval. | 16:22 |
| DinaBelova | ad_rien_ fyi now we're playing with k8s and fuel-ccp (containeraized control plane) to see if we can use it for the experiments and testing | 16:22 |
| msimonin | rook: ack, rabbit may be the issue as well indeed | 16:23 |
| rook | msimonin: honestly, it wouldn't shock me. | 16:23 |
| rook | msimonin: what errors are you seeing? | 16:23 |
| DinaBelova | ad_rien_ although I believe HAproxy is either not an issue at all or this can be fixed even without redeployment | 16:23 |
| msimonin | rook: WARNING nova.openstack.common.loopingcall [-] task <bound method DbDriver._report_state of <nova.servicegroup.drivers.db.DbDriver object at 0x7f454c8db710>> run outlasted interval | 16:24 |
| msimonin | in the nova-compute.lgo | 16:24 |
| rook | yeah - could be msging for sure. | 16:24 |
| msimonin | yes | 16:24 |
| DinaBelova | msimonin so let's check this option as well | 16:25 |
| * rook isn't familiar with your control-plane setup | 16:25 | |
| ad_rien_ | DinaBelova: actually to be fair with your experiments, we should remove it…. ok let's see whether we can fix that point by discussing with Alex. | 16:25 |
| DinaBelova | ad_rien_ ack | 16:25 |
| ad_rien_ | (to be faire, i.e. to provide fair comparisons) | 16:25 |
| DinaBelova | ad_rien_ msimonin I'll start separated email thread with Alex | 16:26 |
| msimonin | DinaBelova: ack | 16:26 |
| ad_rien_ | DinaBelova: ack thanks | 16:26 |
| DinaBelova | it looks like we may proceed for now | 16:26 |
| rcherrueau | DinaBelova: could you also put me in CC | 16:26 |
| DinaBelova | rcherrueau sure | 16:26 |
| rcherrueau | thanks | 16:26 |
| DinaBelova | ok, so let's jump to the OSprofiler | 16:26 |
| DinaBelova | #topic OSProfiler weekly update | 16:27 |
| DinaBelova | rohanion aignatev - the floor is yours | 16:27 |
| aignatev | need review for https://review.openstack.org/#/c/340936/ | 16:27 |
| aignatev | other than that I'm not currently working with osprofiler, so no updates | 16:27 |
| rohanion | ok, will look at it today | 16:27 |
| aignatev | thanks! | 16:28 |
| *** msimonin has quit IRC | 16:28 | |
| DinaBelova | aignatev I'll take a look as well, thanks | 16:28 |
| aignatev | thank you | 16:28 |
| DinaBelova | rohanion any updates from your side? | 16:28 |
| rohanion | yep, writing :) | 16:28 |
| rohanion | I'm working on enabling osprofiler in Fuel. basically I have to code a PoC that updates configuration and reloads the services. | 16:28 |
| rohanion | this PoC will be turned into a Fuel plugin | 16:29 |
| rohanion | Also I'm thinking about a set of profiling scenarios with single entry-point and results in table format. | 16:29 |
| DinaBelova | rohanion I believe you found out that's a bit overkill solution? | 16:29 |
| DinaBelova | I mean fule plugin? | 16:29 |
| DinaBelova | fuel* | 16:29 |
| rohanion | yes, a little. but still it will be a good option to set all the needed variables in a nice UI | 16:30 |
| DinaBelova | or you decided it's not too difficult to develop this profiling plugin? | 16:30 |
| DinaBelova | rohanion ack | 16:31 |
| rohanion | it will be tough but I think I'll handle that. | 16:31 |
| rook | bummer, will it be written to only work with Fuel? | 16:31 |
| rook | not OOO? | 16:31 |
| rohanion | no updates regarding osprofiler itself | 16:31 |
| rohanion | rook, no | 16:31 |
| rook | no to only Fuel ? | 16:32 |
| rohanion | the script will work with vanilla openstack | 16:32 |
| rook | ah ha cool | 16:32 |
| DinaBelova | rook it's just way for us to deploy it automatically using fuel, but it workd with any OpenStack and is automated to be used with devtack | 16:32 |
| rohanion | the plugin will just launch this script, sending all necessary parameters | 16:32 |
| rook | cool rohanion - have a link to this work | 16:32 |
| *** AugieMena has quit IRC | 16:33 | |
| DinaBelova | ok, rohanion is this all for now? | 16:33 |
| rohanion | I'll publish the changes as soon as I write the code :) | 16:33 |
| rohanion | yes, that's everything | 16:33 |
| DinaBelova | ok, cool | 16:33 |
| DinaBelova | so we may jump to open discussion :) | 16:34 |
| DinaBelova | #topic Open Discussion | 16:34 |
| DinaBelova | anything else to share? | 16:34 |
| rcherrueau | We really want to contribute to the performance documentation | 16:35 |
| DinaBelova | rcherrueau you're really welcome :) | 16:35 |
| rcherrueau | do you think it's possible | 16:35 |
| DinaBelova | rcherrueau of course :) | 16:35 |
| DinaBelova | that's usual enough contribution process | 16:35 |
| DinaBelova | to the repository | 16:35 |
| rcherrueau | for instance, do you think it's possible to add a specific subsection in the labs section with information about our grid? | 16:35 |
| DinaBelova | #link https://github.com/openstack/performance-docs | 16:35 |
| DinaBelova | rcherrueau for sure | 16:35 |
| DinaBelova | to have all info shared | 16:36 |
| rcherrueau | cool | 16:36 |
| DinaBelova | and you'll mention this lab in the test plans/test results | 16:36 |
| rcherrueau | pleaze what is the process to do so | 16:36 |
| DinaBelova | rcherrueau you need to upload several commits | 16:36 |
| DinaBelova | commit #1 - regarding the lab description | 16:36 |
| DinaBelova | lemme find an example | 16:37 |
| rcherrueau | yes thanks | 16:37 |
| DinaBelova | rcherrueau like this https://review.openstack.org/#/c/292835/ | 16:37 |
| DinaBelova | but regarding your lab | 16:37 |
| ad_rien_ | ok DinaBelova I think we got the information we were looking for | 16:37 |
| rcherrueau | ok thank you DinaBelova | 16:38 |
| DinaBelova | and second one with your results, as you're going to use the same test plan as we did | 16:38 |
| DinaBelova | rcherrueau https://review.openstack.org/#/c/324806/ - test results part | 16:38 |
| ad_rien_ | yes modulo the way we perform(ed) the experiments | 16:38 |
| DinaBelova | ad_rien_ indeed | 16:38 |
| DinaBelova | the overall documentation is free form, the only rule is to set test plans regarding the template https://github.com/openstack/performance-docs/blob/master/doc/source/test_plans/template.rst | 16:39 |
| DinaBelova | but you're not going to propose your own test plan :) | 16:39 |
| DinaBelova | so feel free to skip my last message :) | 16:39 |
| ad_rien_ | ok we gonna give a look | 16:39 |
| rcherrueau | OK great, that's all for us. | 16:40 |
| DinaBelova | ad_rien_ sure, will take a look with great pleasure :) | 16:40 |
| DinaBelova | anything else here? from my side I have nothing more to share | 16:40 |
| ad_rien_ | not sure whether it will be done for the next week as right now we are focusing on kolla related issues but once we get relevant results we will commit them | 16:41 |
| DinaBelova | ad_rien_ ack, thanks for the update | 16:41 |
| DinaBelova | ok, it looks like we're done for now | 16:41 |
| DinaBelova | please wait the meeting time update email | 16:41 |
| DinaBelova | I think we'll stop with 15:30 UTC | 16:42 |
| DinaBelova | thanks folks for coming and participation | 16:42 |
| DinaBelova | see you | 16:42 |
| DinaBelova | #endmeeting | 16:42 |
| openstack | Meeting ended Tue Aug 23 16:42:25 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:42 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-08-23-16.00.html | 16:42 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-08-23-16.00.txt | 16:42 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/performance_team/2016/performance_team.2016-08-23-16.00.log.html | 16:42 |
| *** rohanion has quit IRC | 16:47 | |
| *** aignatev has quit IRC | 16:52 | |
| *** Guest25180 is now known as med_ | 17:06 | |
| *** med_ has joined #openstack-performance | 17:06 | |
| *** med_ is now known as medberry | 17:06 | |
| *** medberry is now known as med_ | 17:06 | |
| *** ilyashakhat has joined #openstack-performance | 17:14 | |
| *** rcherrueau has quit IRC | 17:16 | |
| *** hieulq_ has quit IRC | 17:20 | |
| *** ilyashakhat has quit IRC | 17:21 | |
| *** ad_rien_ has quit IRC | 17:25 | |
| *** hockeynut has quit IRC | 17:33 | |
| *** bwallis has quit IRC | 17:36 | |
| *** ilyashakhat has joined #openstack-performance | 17:39 | |
| *** ilyashakhat has quit IRC | 17:40 | |
| *** ilyashakhat has joined #openstack-performance | 17:40 | |
| *** dimtruck is now known as zz_dimtruck | 17:59 | |
| *** zz_dimtruck is now known as dimtruck | 18:01 | |
| *** ilyashakhat has quit IRC | 18:07 | |
| *** hockeynut has joined #openstack-performance | 18:13 | |
| *** ilyashakhat has joined #openstack-performance | 18:36 | |
| *** ilyashakhat has quit IRC | 18:57 | |
| *** markvoelker has joined #openstack-performance | 18:58 | |
| *** markvoelker has quit IRC | 19:01 | |
| *** pcaruana has quit IRC | 19:20 | |
| *** hockeynut has quit IRC | 19:28 | |
| *** bvanhav has quit IRC | 19:31 | |
| *** ad_rien_ has joined #openstack-performance | 19:46 | |
| *** msimonin has joined #openstack-performance | 19:55 | |
| *** markvoelker has joined #openstack-performance | 19:57 | |
| *** ad_rien_ has quit IRC | 20:06 | |
| *** ilyashakhat has joined #openstack-performance | 20:07 | |
| *** hockeynut has joined #openstack-performance | 20:27 | |
| *** ad_rien_ has joined #openstack-performance | 20:38 | |
| *** markvoelker has quit IRC | 20:52 | |
| *** ilyashakhat has quit IRC | 20:53 | |
| *** rfolco has quit IRC | 21:01 | |
| *** dimtruck is now known as zz_dimtruck | 21:21 | |
| *** zz_dimtruck is now known as dimtruck | 21:23 | |
| *** hockeynut has quit IRC | 21:31 | |
| *** ad_rien_ has quit IRC | 21:49 | |
| *** AnarchyAo has joined #openstack-performance | 22:22 | |
| *** dimtruck is now known as zz_dimtruck | 22:30 | |
| *** msimonin has quit IRC | 22:53 | |
| *** eandersson_ has joined #openstack-performance | 23:40 | |
| *** eandersson has quit IRC | 23:43 | |
| *** AnarchyAo has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!