21:02:57 <markmcclain> #startmeeting Networking 21:02:58 <openstack> Meeting started Mon Dec 16 21:02:57 2013 UTC and is due to finish in 60 minutes. The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:02:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:03:02 <openstack> The meeting name has been set to 'networking' 21:03:10 <marun> hi 21:03:12 <markmcclain> #link https://wiki.openstack.org/wiki/Network/Meetings 21:03:20 <markmcclain> #topic Announcements 21:03:42 <markmcclain> Last week was an interesting week for the OpenStack gate 21:04:06 <marun> 'may you live in interesting times' 21:04:15 <Makdaam> hello 21:04:19 <markmcclain> We merged a small change that had exacerbated an existing race condition 21:04:47 <enikanorov> you mean moving rpc to the end of __init__? 21:04:58 <markmcclain> yes.. that revert 21:05:38 <markmcclain> that patch will be a good one to dive into because it could better reveal the root cause of the race we've been battlting 21:05:38 <enikanorov> interesting 21:05:54 <markmcclain> the other takeaway from the experience 21:06:19 <markmcclain> Is that we need to trust the gate when it fails 21:06:40 <markmcclain> and investigate the logs and cause before issuing rechecks or reverificiations 21:07:32 <markmcclain> if you have a patch that fails don't be surprised if a member of the core team asks for a reason why the proposed patch was unrelated to any failures that required a recheck or reverify 21:08:38 <markmcclain> also if your patch requires multiple rechecks just to get a +1 from testing infrastructure 21:08:51 <markmcclain> expect to get questions during the review 21:09:58 <markmcclain> #link https://launchpad.net/neutron/+milestone/icehouse-2 21:10:07 <markmcclain> Icehouse-2 is rapidly approaching 21:10:22 <markmcclain> along with that 3rd party testing 21:10:34 <markmcclain> mestery led a meeting last week to share info 21:11:02 <mestery> Yes: The hope is to get past likely hurdles together rather than individually. 21:11:02 <markmcclain> #link http://eavesdrop.openstack.org/meetings/networking_third_party_testing/2013/networking_third_party_testing.2013-12-12-17.00.log.html 21:11:05 <mestery> And share information on that. 21:11:10 <mestery> We have another one this week. 21:11:24 <markmcclain> Dec 19th at 1700UTC? 21:11:32 <mestery> 2200 UTC Thursday on #openstack-meeting-alt 21:12:02 <markmcclain> #info 3rd Party Testing Dec 19th at 2200 UTC in -alt 21:12:36 <markmcclain> Lastly the end of the year holidays are approaching, so many take time out off 21:13:01 <markmcclain> I'm thinking we'll meet December 23rd and then skip December 30th 21:13:28 <marun> +1 21:13:33 <Sukhdev> +1 21:14:06 <markmcclain> Any other objections? 21:14:13 <markmcclain> s/other// 21:14:22 <mestery> +1 21:14:34 <mlavalle> +1 21:15:04 <markmcclain> #info No Neutron team meeting December 30th 21:15:15 <markmcclain> Ok let's move onto bugs 21:15:18 <markmcclain> #topic Bugs 21:15:31 <markmcclain> #link http://not.mn/gate_status.html 21:16:39 * notmyname can answer question about that graph 21:16:49 <anteaya> notmyname: awesome graph 21:17:15 <markmcclain> notmyname: definitely helps to visualize what is happening 21:17:48 <salv-orlando> it reminds me a keith haring's painting 21:18:17 <markmcclain> haha 21:18:31 <markmcclain> anteaya: want to highlight any of the bugs you have listed? 21:18:36 <anteaya> sure 21:19:00 <anteaya> marun has a patch in review: https://review.openstack.org/#/c/61168/ 21:19:09 <notmyname> this bold red line is the chance that a new patch, 100% correct itself, has of passing the 6 gate jobs tracked. and for a gate queue that is 30 deep, that means that it is that percentage (eg 62% right now) is raised to the depth: .62**38 == 1.290875032e-06% chance of all passes clearing the gate right now, even if they are all perfect 21:19:11 <anteaya> for bug https://bugs.launchpad.net/neutron/+bug/1192381 21:19:12 <salv-orlando> notmyname: how's the patch pass chance calculated? [can answer in open discussion time to avoid collisions] 21:19:13 <uvirtbot> Launchpad bug 1192381 in neutron "dhcp dnsmasq lost port in host config file" [Critical,In progress] 21:19:26 <salv-orlando> notmyname: thanks for your answer 21:19:37 <anteaya> would be great if we could cross that off the list today, marun can you address comments and submit a new patchset? 21:19:53 <marun> anteaya: can do 21:20:02 <anteaya> nati_ueno: what is the obstacle on https://bugs.launchpad.net/neutron/+bug/1112912, this has been in existance for 1 year? 21:20:04 <uvirtbot> Launchpad bug 1112912 in neutron "get_firewall_required should use VIF parameter from neutron" [Critical,In progress] 21:20:10 <anteaya> marun, thanks 21:20:10 <notmyname> salv-orlando: patch pass chance is the multiplication of the pass chance for each of the 6 jobs (they are independent variables for that calc) 21:20:19 <markmcclain> notmyname: ouch .62**38 hurts 21:20:25 <nati_ueno> anteaya: That's one is still active 21:20:40 <anteaya> nati_ueno: yes, what more to do you need to bring it to closed? 21:20:48 <notmyname> markmcclain: ya. exponents mean that even a 5% drop _dramatically_ reduces the chance that patches actually land 21:21:09 <nati_ueno> anteaya: get reviewed https://review.openstack.org/#/c/21946/ 21:21:35 <notmyname> markmcclain: eg a 95% chance means that a 10-deep queue has less than 60% chance to clear 21:22:04 <anteaya> nati_ueno: okay, let's work on getting Jenkins happy with the patch and then getting reviews on ti 21:22:14 <nati_ueno> anteaya: sure 21:22:29 <anteaya> markmcclain: this is the ssh bug: https://bugs.launchpad.net/neutron/+bug/1253896 21:22:31 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] 21:22:46 <anteaya> you are the champion on it right now, more to say on it? 21:23:06 <markmcclain> not too much other than it went a bad to really bad and now back to bad again 21:23:18 <anteaya> nati_ueno: also can you update the report for the bug with that patch url? I couldn't see it when I read the report, I may have missed it 21:23:23 <markmcclain> there are others who have been digging on it too and most things are wired properly 21:24:14 <anteaya> markmcclain: are you willing to drive some discussions about it this week to see what can be done to make it even less bad that it is now? 21:24:17 <nati_ueno> anteaya: Thanks. I'll add the url 21:24:22 <anteaya> nati_ueno: thank you 21:24:32 <amotoki> hi, sorry for late 21:24:39 <markmcclain> anteaya: yes I won't be on any airplanes this week :) 21:24:50 <anteaya> markmcclain: great, thanks 21:24:57 <anteaya> want to give yourself an action item? 21:25:19 <markmcclain> #action markmcclain to drive 1253896 work 21:25:24 <anteaya> thanks 21:25:29 <anteaya> next confirmed bug 21:25:37 <anteaya> this one needs a champion: https://bugs.launchpad.net/neutron/+bug/1210483 21:25:38 <uvirtbot> Launchpad bug 1210483 in neutron "ServerAddressesTestXML.test_list_server_addresses FAIL" [Critical,Confirmed] 21:25:47 <salv-orlando> markmcclain: I will hand over to you all my knowledge on bug 1253896 so far then 21:25:49 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] https://launchpad.net/bugs/1253896 21:25:57 <anteaya> please volunteer before I track someone down 21:26:22 <markmcclain> salv-orlando: sounds good I figure many of us can collaborate on it 21:26:25 <anteaya> markmcclain: you again with https://bugs.launchpad.net/neutron/+bug/1230407 21:26:26 <uvirtbot> Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed] 21:26:35 <anteaya> any thoughts on it currently? 21:27:14 <anteaya> armax: is this bug still open? https://bugs.launchpad.net/neutron/+bug/1243726 21:27:17 <uvirtbot> Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] 21:27:17 <markmcclain> It's been somewhat infrequent 21:27:17 <markmcclain> http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkFzc2VydGlvbkVycm9yOiBTdGF0ZSBjaGFuZ2UgdGltZW91dCBleGNlZWRlZCFcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDE1NzA5OTYwMiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ== 21:27:34 <anteaya> you have two patches merged against, is it still an issue? 21:27:39 <salv-orlando> There have been only 9 occurrence of 1230407 in a lot of time. And all while the gate was brokeb 21:27:46 <armax> it does not look like a burning issue 21:27:56 <salv-orlando> broken. Note that 153896 can manifest also as 1230407 21:28:04 <salv-orlando> sorry I mean 1253896 21:28:18 <markmcclain> since it still occurs that's why I have not closed it 21:28:24 <anteaya> well sdague has asked that once critical bugs remain in that status unless they are completely gone 21:28:33 <salv-orlando> My understanding is that the bug has been isolated and markmcclain has the definitive for it, which is splitting API/RPC servers 21:28:39 <armax> however there are related bugs open against this one 21:28:56 <anteaya> salv-orlando: that was the last update I posted to that bug yes 21:29:04 <mlavalle> anteaya: I can take a stab at 1210483. If I need help, I'll yell 21:29:12 <anteaya> armax: are you able to update the bug report to reflect that? 21:29:21 <anteaya> mlavalle: awesome thank you 21:29:23 <marun> salv-orlando: doesn't having wsgi out of process get us halfway there? 21:29:30 <armax> of course I am, you're asking me if I will? 21:29:30 <armax> :) 21:29:42 <anteaya> armax: okay, will you? 21:29:44 <markmcclain> marun: yes 21:29:45 * salv-orlando chops armax's finger 21:29:45 <armax> :) 21:29:48 <armax> yup 21:29:52 <anteaya> thank you 21:30:05 <anteaya> this is mine https://bugs.launchpad.net/neutron/+bug/1250168 21:30:08 <uvirtbot> Launchpad bug 1250168 in neutron "gate-tempest-devstack-vm-neutron-large-ops is failing" [Critical,Confirmed] 21:30:19 <anteaya> but I need someone to take over, I simply offered a revert for it 21:30:36 <anteaya> can someone else take this and make it disappear? 21:30:40 <salv-orlando> arosen? 21:30:44 <anteaya> as in no longer occuring 21:31:03 <arosen> sure i can take a look at it. 21:31:07 <salv-orlando> arosen is our nova/neutron guy. This bug pertains to nova/neutron interface 21:31:08 <markmcclain> arosen: thanks 21:31:11 <anteaya> arosen: thanks 21:31:13 <anteaya> great 21:31:17 <anteaya> sorry for taking so long 21:31:25 <anteaya> https://bugs.launchpad.net/neutron/+bug/1251784 needs a volunteer 21:31:29 <uvirtbot> Launchpad bug 1251784 in tripleo "nova+neutron scheduling error: Connection to neutron failed: Maximum attempts reached" [Critical,Fix released] 21:31:34 <anteaya> and to be triaged 21:32:21 <markmcclain> this is in the neutron/nova interface 21:32:48 <anteaya> anyone in addition to arosen available to help? 21:32:58 <markmcclain> not showing any occurrences of this in the last 7 days 21:33:19 <anteaya> it is still critical is it not? 21:33:28 <arosen> markmcclain: it looks like that one is a timeout in neutron 21:33:35 <markmcclain> the notes in the bug say no hits since Nov 28th 21:33:48 <markmcclain> I think it might be safe to close this one 21:33:53 <anteaya> okay 21:34:07 <anteaya> last two that need volunteers: https://bugs.launchpad.net/nova/+bug/1210483 21:34:08 <uvirtbot> Launchpad bug 1210483 in neutron "ServerAddressesTestXML.test_list_server_addresses FAIL" [Critical,Confirmed] 21:34:16 <anteaya> and https://bugs.launchpad.net/neutron/+bug/1254890 21:34:17 <uvirtbot> Launchpad bug 1254890 in tempest ""Timed out waiting for thing" causes tempest-dsvm-neutron-* failures" [Low,In progress] 21:34:29 <enikanorov> i've taken 1210483 21:34:38 <anteaya> enikanorov: thank you 21:34:46 <anteaya> let me know how your progress goes 21:34:51 <markmcclain> ok I also thought mlavalle said he was working on it too 21:35:18 <anteaya> markmcclain: you are correct, sorry 21:35:24 <mlavalle> Yes 21:35:57 <anteaya> enikanorov: how do you feel about looking at 1254890? 21:36:28 <enikanorov> ok, I'll take a look 21:36:32 <anteaya> thank you 21:36:33 <anteaya> done 21:36:45 <markmcclain> anteaya: thanks for reviewing the bugs 21:37:02 <markmcclain> All any other bugs the team should be tracking? 21:37:15 <salv-orlando> for 1254890 the log stash query refers to large_ops only. I've seen the same error in other places too. 21:37:30 <markmcclain> salv-orlando: good to know 21:37:48 <salv-orlando> If you remove indeed large_ops from the query you'll find more occurences. 21:38:25 <markmcclain> #topic Nova Parity 21:38:28 <markmcclain> beagles: hi 21:38:33 <beagles> hi 21:39:21 <beagles> so, we didn't have much movement on parity specific stuff last week unfortunately 21:39:38 <markmcclain> ok.. what resources do you need to help move things along? 21:40:45 <beagles> I think it would be useful to align the efforts for starters.. obviously there are lots of people doing stuff that is related 21:40:57 <beagles> the gate... the additional tests, etc 21:41:03 <markmcclain> ok 21:41:08 <beagles> if we can carve out some time this week to sync that would be good 21:41:20 <anteaya> beagles: who do you need to sync with? 21:41:41 <markmcclain> I'm guessing at least me + mlavalle? 21:41:42 <salv-orlando> I can help from the testing side 21:41:49 <anteaya> actually I see mlavalle's great etherpad and wiki pages but haven't seen many tests yet 21:41:49 <salv-orlando> but mlavalle holds everything 21:41:55 <markmcclain> oops and salv-orlando 21:41:58 <beagles> mlavalle, rossella_s, markmcclain, if arosen could join that would be great 21:42:11 <markmcclain> #action markmcclain to sechedule a time for syncing this week 21:42:13 <salv-orlando> I have just bits of knowledge about status of testing and a few parity features I've been involved with 21:42:18 <mlavalle> beagles: just propose a time and I'll make myself available 21:42:44 <arosen> ditto 21:43:03 <beagles> that'd be great.. can we do it tomorrow a.m. EST? 21:43:07 <markmcclain> we'll chat in the -neutron channel so it's logged and everyone can participate since it will be an on going discussion once we kick it off 21:43:24 <beagles> something like 10:00 am EST.. (what is that UTC?) 21:43:43 <mestery> 1500 UTC 21:43:45 <mlavalle> mlavalle: yes, that is 17:00 UTC 21:43:52 <mlavalle> sorry, 15:00 UTC 21:44:00 <salv-orlando> folks UTC = EST +5 21:44:07 <arosen> beagles: that's a little early for me. I'm on the west coast. Could we push for 11:00 EST instead? 21:44:26 <beagles> arosen: sure 21:44:34 <mlavalle> so 16:00 UTC 21:44:36 <arosen> thanks :) 21:44:50 <rossella_s> ok for me 21:44:56 <markmcclain> #info parity+testing 1600UTC in openstack-neutron 21:45:08 <mlavalle> fine with me 21:45:16 <markmcclain> #topic Tempest 21:45:25 <markmcclain> mlavalle: anyything add that's not on the agenda? 21:45:36 <mlavalle> yeah 21:46:06 <mlavalle> first of all, continued gap analysis for API tests. Added List available extensions, provider extended attributes for networks, binding extended attributes for ports, external network extension, configurable external gateway modes extension, quotas, security groups and rules, agent management extension, extraroute extension to the etherpad 21:46:36 <mlavalle> several people have already assigned themselves work from there 21:46:39 <salv-orlando> in % how many of these tasks have assignees? 21:46:41 <enikanorov> mlavalle: whos doing the work on tempest side? 21:46:43 <markmcclain> cool… thanks for adding them 21:47:06 <mlavalle> and enikanorov has volunteered to refactor the rest client for neutron 21:47:26 <enikanorov> yeah, asking because i want to make it less of copy-paste work 21:47:36 <mlavalle> i'll finish the gap analysis this week, covering the whole api and all the extensions 21:47:54 <mlavalle> for the sake of time i'll stop here 21:48:15 <mlavalle> enikanorov: i'm also the tempest side 21:48:22 <salv-orlando> tempest parallel tests: just a quick update that I'm starting to push all the relevant patches upstream 21:48:23 <anteaya> enikanorov: I've seen your patch, nice work so far 21:48:30 <enikanorov> thx 21:48:34 <salv-orlando> in my internal server I have a 80% success rate on parallel job 21:48:59 <salv-orlando> which is not worse than other gate jobs apparently 21:49:03 <anteaya> salv-orlando: great progress on your parallel blueprint 21:49:09 <markmcclain> salv-orlando: cool 21:49:31 <markmcclain> hopefully we'll be able to close that last 20% 21:49:34 <salv-orlando> only issue I am tracking and can't explain so far is that at some point some VM do not sent DHCPDISCOVER even if they're perfectly wired 21:49:55 <salv-orlando> this happens also on the upstream gate (just happened on one of my patches), and therefore I will file a bug soon 21:50:04 <salv-orlando> Or perhaps is a bug that haunts just me? 21:50:07 <markmcclain> interesting 21:50:49 <markmcclain> ok we're running short on time again 21:51:21 <mlavalle> salv-orlando: when you file the bug, please let me know and I will try to reproduce in my dev system 21:51:31 <salv-orlando> malavalle: sure 21:51:42 <mlavalle> so we can compare notes 21:51:46 <markmcclain> #topic IPv6 21:52:03 <markmcclain> There's a mailing list thread on hairpinning per vif 21:52:23 <markmcclain> please chime in on that thread if you have thoughts on it 21:52:26 <markmcclain> #topic ML2 21:52:46 <rkukura> nothing critical to discuss today 21:53:15 <mestery> The only item I have is that we will be canceling the meetings on 12-25 and 1-1. 21:53:19 <markmcclain> Seems that there is discussion still to be had on providernet vs multi-provider? 21:53:24 <mestery> Will send email with a note to openstack-dev. 21:53:50 <markmcclain> mestery: thanks for the heads u 21:53:52 <markmcclain> up 21:53:58 <markmcclain> #topic Open Discussion 21:54:08 <geekinutah> salv-orlando, enikanorov http://lists.openstack.org/pipermail/openstack-dev/2013-December/021984.html https://bugs.launchpad.net/neutron/+bug/1214115 21:54:10 <uvirtbot> Launchpad bug 1214115 in neutron "ipavailabilityranges race condition when allocating from same range on multiple neutron-servers" [High,In progress] 21:54:24 <geekinutah> what are next steps on this bug? I'm happy to do leg work 21:55:07 <geekinutah> not sure if the original patch submitter is still active, but also happy to pick that up and massage it if needed 21:55:09 <markmcclain> geekinutah: contact the current bug assignee 21:55:12 <enikanorov> it's interesting if carl baldwin's patch could address this issue 21:55:14 <salv-orlando> geekinutah: I thought I have already removed my -2 there. 21:55:32 <geekinutah> hmmm, still shows -2 21:55:35 <enikanorov> geekinutah: https://review.openstack.org/#/c/58017/ 21:55:56 <geekinutah> ahh, okay thx 21:55:56 <markmcclain> the patchset is abandoned 21:56:08 <markmcclain> so the state cannot change until the review is active again 21:56:09 <salv-orlando> I am happy to reconsider. You might understand that if we suspect a risk of introducing an issue worse that the bugs being fixed we act a big conservative in the 3rd milestone 21:56:11 <carl_baldwin> enikanorov: My impression was that my patch probably would not address that bug. 21:56:16 <salv-orlando> markmcclain: correct 21:57:08 <markmcclain> geekinutah: reach out and see if the original person is still interested in the work if not offer to take it on 21:57:24 <geekinutah> I will do that, also I'll look at carl_baldwin's patch 21:57:35 <markmcclain> if you have any questions feel free to ask in the IRC channel or mailing list 21:57:44 <markmcclain> Any other open discussion items? 21:57:46 <anteaya> Please consider me offline from Jan 1. until code sprint in Montreal 21:57:50 <enikanorov> yeah, it seems that retries are needed 21:58:07 <enikanorov> I also remember we did something similar for generating tunnel ids 21:58:13 <anteaya> any detail questions about the code sprint, ensure you have the answers you need prior to Holiday Break 21:58:15 <markmcclain> anteaya: thanks for the heads up 21:59:49 <dkehn> anybody experience issue with devstack on a local system, beyond migration issues , like the sudo /usr/local/bin/neutron-rootwrap 21:59:52 <dkehn> 12-16 14:55[ dkehn]: /etc/neutron/rootwrap.conf ip netns exec qprobe-83b5c2ea-ca44-4b1\ 21:59:55 <dkehn> 12-16 14:55[ dkehn]: 12-16 14:55[ dkehn]: c-a7d6-46235386b287 ping -w 1 -c 1 10.1.0.4; failures 21:59:58 <dkehn> pinging namespaces 22:00:39 <sc68cal> dkehn: +1 22:00:49 <markmcclain> I have not 22:01:03 <marun> dkehn: is that the debug agent? 22:01:04 <markmcclain> We're at time for this week 22:01:07 <markmcclain> Ok remember tomorrow at 1600UTC in #openstack-neutron we'll kick off the discussion on Nova Parity and Testing overlapping 22:01:12 <markmcclain> #endmeeting