21:02:57 <markmcclain> #startmeeting Networking
21:02:58 <openstack> Meeting started Mon Dec 16 21:02:57 2013 UTC and is due to finish in 60 minutes.  The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:03:02 <openstack> The meeting name has been set to 'networking'
21:03:10 <marun> hi
21:03:12 <markmcclain> #link https://wiki.openstack.org/wiki/Network/Meetings
21:03:20 <markmcclain> #topic Announcements
21:03:42 <markmcclain> Last week was an interesting week for the OpenStack gate
21:04:06 <marun> 'may you live in interesting times'
21:04:15 <Makdaam> hello
21:04:19 <markmcclain> We merged a small change that had exacerbated an existing race condition
21:04:47 <enikanorov> you mean moving rpc to the end of __init__?
21:04:58 <markmcclain> yes.. that revert
21:05:38 <markmcclain> that patch will be a good one to dive into because it could better reveal the root cause of the race we've been battlting
21:05:38 <enikanorov> interesting
21:05:54 <markmcclain> the other takeaway from the experience
21:06:19 <markmcclain> Is that we need to trust the gate when it fails
21:06:40 <markmcclain> and investigate the logs and cause before issuing rechecks or reverificiations
21:07:32 <markmcclain> if you have a patch that fails don't be surprised if a member of the core team asks for a reason why the proposed patch was unrelated to any failures that required a recheck or reverify
21:08:38 <markmcclain> also if your patch requires multiple rechecks just to get a +1 from testing infrastructure
21:08:51 <markmcclain> expect to get questions during the review
21:09:58 <markmcclain> #link https://launchpad.net/neutron/+milestone/icehouse-2
21:10:07 <markmcclain> Icehouse-2 is rapidly approaching
21:10:22 <markmcclain> along with that 3rd party testing
21:10:34 <markmcclain> mestery led a meeting last week to share info
21:11:02 <mestery> Yes: The hope is to get past likely hurdles together rather than individually.
21:11:02 <markmcclain> #link http://eavesdrop.openstack.org/meetings/networking_third_party_testing/2013/networking_third_party_testing.2013-12-12-17.00.log.html
21:11:05 <mestery> And share information on that.
21:11:10 <mestery> We have another one this week.
21:11:24 <markmcclain> Dec 19th at 1700UTC?
21:11:32 <mestery> 2200 UTC Thursday on #openstack-meeting-alt
21:12:02 <markmcclain> #info 3rd Party Testing Dec 19th at 2200 UTC in -alt
21:12:36 <markmcclain> Lastly the end of the year holidays are approaching, so many take time out off
21:13:01 <markmcclain> I'm thinking we'll meet December 23rd and then skip December 30th
21:13:28 <marun> +1
21:13:33 <Sukhdev> +1
21:14:06 <markmcclain> Any other objections?
21:14:13 <markmcclain> s/other//
21:14:22 <mestery> +1
21:14:34 <mlavalle> +1
21:15:04 <markmcclain> #info No Neutron team meeting December 30th
21:15:15 <markmcclain> Ok let's move onto bugs
21:15:18 <markmcclain> #topic Bugs
21:15:31 <markmcclain> #link http://not.mn/gate_status.html
21:16:39 * notmyname can answer question about that graph
21:16:49 <anteaya> notmyname: awesome graph
21:17:15 <markmcclain> notmyname: definitely helps to visualize what is happening
21:17:48 <salv-orlando> it reminds me a keith haring's painting
21:18:17 <markmcclain> haha
21:18:31 <markmcclain> anteaya: want to highlight any of the bugs you have listed?
21:18:36 <anteaya> sure
21:19:00 <anteaya> marun has a patch in review: https://review.openstack.org/#/c/61168/
21:19:09 <notmyname> this bold red line is the chance that a new patch, 100% correct itself, has of passing the 6 gate jobs tracked. and for a gate queue that is 30 deep, that means that it is that percentage (eg 62% right now) is raised to the depth: .62**38 == 1.290875032e-06% chance of all passes clearing the gate right now, even if they are all perfect
21:19:11 <anteaya> for bug https://bugs.launchpad.net/neutron/+bug/1192381
21:19:12 <salv-orlando> notmyname: how's the patch pass chance calculated? [can answer in open discussion time to avoid collisions]
21:19:13 <uvirtbot> Launchpad bug 1192381 in neutron "dhcp dnsmasq lost port in host config file" [Critical,In progress]
21:19:26 <salv-orlando> notmyname: thanks for your answer
21:19:37 <anteaya> would be great if we could cross that off the list today, marun can you address comments and submit a new patchset?
21:19:53 <marun> anteaya: can do
21:20:02 <anteaya> nati_ueno: what is the obstacle on https://bugs.launchpad.net/neutron/+bug/1112912, this has been in existance for 1 year?
21:20:04 <uvirtbot> Launchpad bug 1112912 in neutron "get_firewall_required should use VIF parameter from neutron" [Critical,In progress]
21:20:10 <anteaya> marun, thanks
21:20:10 <notmyname> salv-orlando: patch pass chance is the multiplication of the pass chance for each of the 6 jobs (they are independent variables for that calc)
21:20:19 <markmcclain> notmyname: ouch .62**38 hurts
21:20:25 <nati_ueno> anteaya: That's one is still active
21:20:40 <anteaya> nati_ueno: yes, what more to do you need to bring it to closed?
21:20:48 <notmyname> markmcclain: ya. exponents mean that even a 5% drop _dramatically_ reduces the chance that patches actually land
21:21:09 <nati_ueno> anteaya: get reviewed https://review.openstack.org/#/c/21946/
21:21:35 <notmyname> markmcclain: eg a 95% chance means that a 10-deep queue has less than 60% chance to clear
21:22:04 <anteaya> nati_ueno: okay, let's work on getting Jenkins happy with the patch and then getting reviews on ti
21:22:14 <nati_ueno> anteaya: sure
21:22:29 <anteaya> markmcclain: this is the ssh bug: https://bugs.launchpad.net/neutron/+bug/1253896
21:22:31 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress]
21:22:46 <anteaya> you are the champion on it right now, more to say on it?
21:23:06 <markmcclain> not too much other than it went a bad to really bad and now back to bad again
21:23:18 <anteaya> nati_ueno: also can you update the report for the bug with that patch url? I couldn't see it when I read the report, I may have missed it
21:23:23 <markmcclain> there are others who have been digging on it too and most things are wired properly
21:24:14 <anteaya> markmcclain: are you willing to drive some discussions about it this week to see what can be done to make it even less bad that it is now?
21:24:17 <nati_ueno> anteaya: Thanks. I'll add the url
21:24:22 <anteaya> nati_ueno: thank you
21:24:32 <amotoki> hi, sorry for late
21:24:39 <markmcclain> anteaya: yes I won't be on any airplanes this week :)
21:24:50 <anteaya> markmcclain: great, thanks
21:24:57 <anteaya> want to give yourself an action item?
21:25:19 <markmcclain> #action markmcclain to drive 1253896 work
21:25:24 <anteaya> thanks
21:25:29 <anteaya> next confirmed bug
21:25:37 <anteaya> this one needs a champion: https://bugs.launchpad.net/neutron/+bug/1210483
21:25:38 <uvirtbot> Launchpad bug 1210483 in neutron "ServerAddressesTestXML.test_list_server_addresses FAIL" [Critical,Confirmed]
21:25:47 <salv-orlando> markmcclain: I will hand over to you all my knowledge on bug 1253896 so far then
21:25:49 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress] https://launchpad.net/bugs/1253896
21:25:57 <anteaya> please volunteer before I track someone down
21:26:22 <markmcclain> salv-orlando: sounds good I figure many of us can collaborate on it
21:26:25 <anteaya> markmcclain: you again with https://bugs.launchpad.net/neutron/+bug/1230407
21:26:26 <uvirtbot> Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed]
21:26:35 <anteaya> any thoughts on it currently?
21:27:14 <anteaya> armax: is this bug still open? https://bugs.launchpad.net/neutron/+bug/1243726
21:27:17 <uvirtbot> Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed]
21:27:17 <markmcclain> It's been somewhat infrequent
21:27:17 <markmcclain> http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkFzc2VydGlvbkVycm9yOiBTdGF0ZSBjaGFuZ2UgdGltZW91dCBleGNlZWRlZCFcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MDE1NzA5OTYwMiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==
21:27:34 <anteaya> you have two patches merged against, is it still an issue?
21:27:39 <salv-orlando> There have been only 9 occurrence of 1230407 in a lot of time. And all while the gate was brokeb
21:27:46 <armax> it does not look like a burning issue
21:27:56 <salv-orlando> broken. Note that 153896 can manifest also as 1230407
21:28:04 <salv-orlando> sorry I mean 1253896
21:28:18 <markmcclain> since it still occurs that's why I have not closed it
21:28:24 <anteaya> well sdague has asked that once critical bugs remain in that status unless they are completely gone
21:28:33 <salv-orlando> My understanding is that the bug has been isolated and markmcclain has the definitive for it, which is splitting API/RPC servers
21:28:39 <armax> however there are related bugs open against this one
21:28:56 <anteaya> salv-orlando: that was the last update I posted to that bug yes
21:29:04 <mlavalle> anteaya: I can take a stab at 1210483. If I need help, I'll yell
21:29:12 <anteaya> armax: are you able to update the bug report to reflect that?
21:29:21 <anteaya> mlavalle: awesome thank you
21:29:23 <marun> salv-orlando: doesn't having wsgi out of process get us halfway there?
21:29:30 <armax> of course I am, you're asking me if I will?
21:29:30 <armax> :)
21:29:42 <anteaya> armax: okay, will you?
21:29:44 <markmcclain> marun: yes
21:29:45 * salv-orlando chops armax's finger
21:29:45 <armax> :)
21:29:48 <armax> yup
21:29:52 <anteaya> thank you
21:30:05 <anteaya> this is mine https://bugs.launchpad.net/neutron/+bug/1250168
21:30:08 <uvirtbot> Launchpad bug 1250168 in neutron "gate-tempest-devstack-vm-neutron-large-ops is failing" [Critical,Confirmed]
21:30:19 <anteaya> but I need someone to take over, I simply offered a revert for it
21:30:36 <anteaya> can someone else take this and make it disappear?
21:30:40 <salv-orlando> arosen?
21:30:44 <anteaya> as in no longer occuring
21:31:03 <arosen> sure i can take a look at it.
21:31:07 <salv-orlando> arosen is our nova/neutron guy. This bug pertains to nova/neutron interface
21:31:08 <markmcclain> arosen: thanks
21:31:11 <anteaya> arosen: thanks
21:31:13 <anteaya> great
21:31:17 <anteaya> sorry for taking so long
21:31:25 <anteaya> https://bugs.launchpad.net/neutron/+bug/1251784 needs a volunteer
21:31:29 <uvirtbot> Launchpad bug 1251784 in tripleo "nova+neutron scheduling error: Connection to neutron failed: Maximum attempts reached" [Critical,Fix released]
21:31:34 <anteaya> and to be triaged
21:32:21 <markmcclain> this is in the neutron/nova interface
21:32:48 <anteaya> anyone in addition to arosen available to help?
21:32:58 <markmcclain> not showing any occurrences of this in the last 7 days
21:33:19 <anteaya> it is still critical is it not?
21:33:28 <arosen> markmcclain:  it looks like that one is a timeout in neutron
21:33:35 <markmcclain> the notes in the bug say no hits since Nov 28th
21:33:48 <markmcclain> I think it might be safe to close this one
21:33:53 <anteaya> okay
21:34:07 <anteaya> last two that need volunteers: https://bugs.launchpad.net/nova/+bug/1210483
21:34:08 <uvirtbot> Launchpad bug 1210483 in neutron "ServerAddressesTestXML.test_list_server_addresses FAIL" [Critical,Confirmed]
21:34:16 <anteaya> and https://bugs.launchpad.net/neutron/+bug/1254890
21:34:17 <uvirtbot> Launchpad bug 1254890 in tempest ""Timed out waiting for thing" causes tempest-dsvm-neutron-* failures" [Low,In progress]
21:34:29 <enikanorov> i've taken 1210483
21:34:38 <anteaya> enikanorov: thank you
21:34:46 <anteaya> let me know how your progress goes
21:34:51 <markmcclain> ok I also thought mlavalle said he was working on it too
21:35:18 <anteaya> markmcclain: you are correct, sorry
21:35:24 <mlavalle> Yes
21:35:57 <anteaya> enikanorov: how do you feel about looking at 1254890?
21:36:28 <enikanorov> ok, I'll take a look
21:36:32 <anteaya> thank you
21:36:33 <anteaya> done
21:36:45 <markmcclain> anteaya: thanks for reviewing the bugs
21:37:02 <markmcclain> All any other bugs the team should be tracking?
21:37:15 <salv-orlando> for 1254890  the log stash query refers to large_ops only. I've seen the same error in other places too.
21:37:30 <markmcclain> salv-orlando: good to know
21:37:48 <salv-orlando> If you remove indeed large_ops from the query you'll find more occurences.
21:38:25 <markmcclain> #topic Nova Parity
21:38:28 <markmcclain> beagles: hi
21:38:33 <beagles> hi
21:39:21 <beagles> so, we didn't have much movement on parity specific stuff last week unfortunately
21:39:38 <markmcclain> ok.. what resources do you need to help move things along?
21:40:45 <beagles> I think it would be useful to align the efforts for starters.. obviously there are lots of people doing stuff that is related
21:40:57 <beagles> the gate... the additional tests, etc
21:41:03 <markmcclain> ok
21:41:08 <beagles> if we can carve out some time this week to sync that would be good
21:41:20 <anteaya> beagles: who do you need to sync with?
21:41:41 <markmcclain> I'm guessing at least me + mlavalle?
21:41:42 <salv-orlando> I can help from the testing side
21:41:49 <anteaya> actually I see mlavalle's great etherpad and wiki pages but haven't seen many tests yet
21:41:49 <salv-orlando> but mlavalle holds everything
21:41:55 <markmcclain> oops and salv-orlando
21:41:58 <beagles> mlavalle, rossella_s, markmcclain, if arosen could join that would be great
21:42:11 <markmcclain> #action markmcclain to sechedule a time for syncing this week
21:42:13 <salv-orlando> I have just bits of knowledge about status of testing and a few parity features I've been involved with
21:42:18 <mlavalle> beagles: just propose a time and I'll make myself available
21:42:44 <arosen> ditto
21:43:03 <beagles> that'd be great.. can we do it tomorrow a.m. EST?
21:43:07 <markmcclain> we'll chat in the -neutron channel so it's logged and everyone can participate since it will be an on going discussion once we kick it off
21:43:24 <beagles> something like 10:00 am EST.. (what is that UTC?)
21:43:43 <mestery> 1500 UTC
21:43:45 <mlavalle> mlavalle: yes, that is 17:00 UTC
21:43:52 <mlavalle> sorry, 15:00 UTC
21:44:00 <salv-orlando> folks UTC = EST +5
21:44:07 <arosen> beagles:  that's a little early for me. I'm on the west coast. Could we push for 11:00 EST instead?
21:44:26 <beagles> arosen: sure
21:44:34 <mlavalle> so 16:00 UTC
21:44:36 <arosen> thanks :)
21:44:50 <rossella_s> ok for me
21:44:56 <markmcclain> #info parity+testing 1600UTC in openstack-neutron
21:45:08 <mlavalle> fine with me
21:45:16 <markmcclain> #topic Tempest
21:45:25 <markmcclain> mlavalle: anyything add that's not on the agenda?
21:45:36 <mlavalle> yeah
21:46:06 <mlavalle> first of all, continued gap analysis for API tests. Added List available extensions, provider extended attributes for networks, binding extended attributes for ports, external network extension, configurable external gateway modes extension, quotas, security groups and rules, agent management extension, extraroute extension to the etherpad
21:46:36 <mlavalle> several people have already assigned themselves work from there
21:46:39 <salv-orlando> in % how many of these tasks have assignees?
21:46:41 <enikanorov> mlavalle: whos doing the work on tempest side?
21:46:43 <markmcclain> cool… thanks for adding them
21:47:06 <mlavalle> and enikanorov has volunteered to refactor the rest client for neutron
21:47:26 <enikanorov> yeah, asking because i want to make it less of copy-paste work
21:47:36 <mlavalle> i'll finish the gap analysis this week, covering the whole api and all the extensions
21:47:54 <mlavalle> for the sake of time i'll stop here
21:48:15 <mlavalle> enikanorov: i'm also the tempest side
21:48:22 <salv-orlando> tempest parallel tests: just a quick update that I'm starting to push all the relevant patches upstream
21:48:23 <anteaya> enikanorov: I've seen your patch, nice work so far
21:48:30 <enikanorov> thx
21:48:34 <salv-orlando> in my internal server I have a 80% success rate on parallel job
21:48:59 <salv-orlando> which is not worse than other gate jobs apparently
21:49:03 <anteaya> salv-orlando: great progress on your parallel blueprint
21:49:09 <markmcclain> salv-orlando: cool
21:49:31 <markmcclain> hopefully we'll be able to close that last 20%
21:49:34 <salv-orlando> only issue I am tracking and can't explain so far is that at some point some VM do not sent DHCPDISCOVER even if they're perfectly wired
21:49:55 <salv-orlando> this happens also on the upstream gate (just happened on one of my patches), and therefore I will file a bug soon
21:50:04 <salv-orlando> Or perhaps is a bug that haunts just me?
21:50:07 <markmcclain> interesting
21:50:49 <markmcclain> ok we're running short on time again
21:51:21 <mlavalle> salv-orlando: when you file the bug, please let me know and I will try to reproduce in my dev system
21:51:31 <salv-orlando> malavalle: sure
21:51:42 <mlavalle> so we can compare notes
21:51:46 <markmcclain> #topic IPv6
21:52:03 <markmcclain> There's a mailing list thread on hairpinning per vif
21:52:23 <markmcclain> please chime in on that thread if you have thoughts on it
21:52:26 <markmcclain> #topic ML2
21:52:46 <rkukura> nothing critical to discuss today
21:53:15 <mestery> The only item I have is that we will be canceling the meetings on 12-25 and 1-1.
21:53:19 <markmcclain> Seems that there is discussion still to be had on providernet vs multi-provider?
21:53:24 <mestery> Will send email with a note to openstack-dev.
21:53:50 <markmcclain> mestery: thanks for the heads u
21:53:52 <markmcclain> up
21:53:58 <markmcclain> #topic Open Discussion
21:54:08 <geekinutah> salv-orlando, enikanorov http://lists.openstack.org/pipermail/openstack-dev/2013-December/021984.html https://bugs.launchpad.net/neutron/+bug/1214115
21:54:10 <uvirtbot> Launchpad bug 1214115 in neutron "ipavailabilityranges race condition when allocating from same range on multiple neutron-servers" [High,In progress]
21:54:24 <geekinutah> what are next steps on this bug? I'm happy to do leg work
21:55:07 <geekinutah> not sure if the original patch submitter is still active, but also happy to pick that up and massage it if needed
21:55:09 <markmcclain> geekinutah: contact the current bug assignee
21:55:12 <enikanorov> it's interesting if carl baldwin's patch could address this issue
21:55:14 <salv-orlando> geekinutah: I thought I have already removed my -2 there.
21:55:32 <geekinutah> hmmm, still shows -2
21:55:35 <enikanorov> geekinutah: https://review.openstack.org/#/c/58017/
21:55:56 <geekinutah> ahh, okay thx
21:55:56 <markmcclain> the patchset is abandoned
21:56:08 <markmcclain> so the state cannot change until the review is active again
21:56:09 <salv-orlando> I am happy to reconsider. You might understand that if we suspect a risk of introducing an issue worse that the bugs being fixed we act a big conservative in the 3rd milestone
21:56:11 <carl_baldwin> enikanorov: My impression was that my patch probably would not address that bug.
21:56:16 <salv-orlando> markmcclain: correct
21:57:08 <markmcclain> geekinutah: reach out and see if the original person is still interested in the work if not offer to take it on
21:57:24 <geekinutah> I will do that, also I'll look at carl_baldwin's patch
21:57:35 <markmcclain> if you have any questions feel free to ask in the IRC channel or mailing list
21:57:44 <markmcclain> Any other open discussion items?
21:57:46 <anteaya> Please consider me offline from Jan 1. until code sprint in Montreal
21:57:50 <enikanorov> yeah, it seems that retries are needed
21:58:07 <enikanorov> I also remember we did something similar for generating tunnel ids
21:58:13 <anteaya> any detail questions about the code sprint, ensure you have the answers you need prior to Holiday Break
21:58:15 <markmcclain> anteaya: thanks for the heads up
21:59:49 <dkehn> anybody experience issue with devstack on a local system, beyond migration issues , like the sudo /usr/local/bin/neutron-rootwrap
21:59:52 <dkehn> 12-16 14:55[ dkehn]:           /etc/neutron/rootwrap.conf ip netns exec qprobe-83b5c2ea-ca44-4b1\
21:59:55 <dkehn> 12-16 14:55[ dkehn]: 12-16 14:55[ dkehn]: c-a7d6-46235386b287 ping -w 1 -c 1 10.1.0.4; failures
21:59:58 <dkehn> pinging namespaces
22:00:39 <sc68cal> dkehn: +1
22:00:49 <markmcclain> I have not
22:01:03 <marun> dkehn: is that the debug agent?
22:01:04 <markmcclain> We're at time for this week
22:01:07 <markmcclain> Ok remember tomorrow at 1600UTC in #openstack-neutron we'll kick off the discussion on Nova Parity and Testing overlapping
22:01:12 <markmcclain> #endmeeting