21:04:05 <markmcclain> #startmeeting Networking
21:04:05 <armax> here
21:04:05 <openstack> Meeting started Mon Nov 18 21:04:05 2013 UTC and is due to finish in 60 minutes.  The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:04:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:04:08 <openstack> The meeting name has been set to 'networking'
21:04:13 <markmcclain> #link https://wiki.openstack.org/wiki/Network/Meetings
21:04:22 <markmcclain> #topic Announcements
21:05:11 <markmcclain> Special agenda this week to go over priorities for Icehouse
21:05:22 <markmcclain> #info Icehouse release schedule is available
21:05:27 <markmcclain> #link https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
21:05:58 <markmcclain> Icehouse-1 is rapidly approaching: December 5th
21:06:45 <arosen> whoops sorry i'm late.
21:06:57 <markmcclain> Also one modification to the agenda: going to insert Plugin/Driver testing after Bug topic
21:07:05 <mestery> I'm just happy you made it back to the US in one piece arosen. :)
21:07:05 <markmcclain> arosen: no worries
21:07:14 <arosen> mestery: haha no kidding ;)
21:07:27 <markmcclain> sounds like there is a good story somewhere
21:07:32 <markmcclain> #topic Bugs
21:07:40 <markmcclain> #link https://bugs.launchpad.net/neutron/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Confirmed&field.status=Triaged&field.status=In+Progress
21:08:10 <markmcclain> we're still carrying 3 critical bugs
21:08:38 <markmcclain> I'm currently working on: https://bugs.launchpad.net/neutron/+bug/1230407
21:08:41 <uvirtbot> Launchpad bug 1230407 in neutron "VMs can't progress through state changes because Neutron is deadlocking on it's database queries, and thus leaving networks in inconsistent states" [Critical,Confirmed]
21:09:23 <markmcclain> I'm inclined to close: https://bugs.launchpad.net/neutron/+bug/1236439
21:09:26 <uvirtbot> Launchpad bug 1236439 in ubuntu-release-notes "switch to use hostnames like nova breaks upgrades of l3-agent" [Undecided,Fix released]
21:09:35 <markmcclain> I know mestery looked into this one
21:10:00 <mestery> markmcclain: Yes, was reading that one, looks like it was release noted in Ubuntu already.
21:10:09 <markmcclain> ok
21:10:23 <markmcclain> last one: https://bugs.launchpad.net/neutron/+bug/1243726
21:10:26 <uvirtbot> Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed]
21:10:31 <mestery> markmcclain: Is that related to that other one you assigned me to look at before the Summit?
21:10:36 <markmcclain> yes
21:10:38 * mestery can't recall the number, searching now.
21:10:39 <mestery> OK
21:10:54 <markmcclain> armax: you have 1243726 and then released it
21:11:09 <armax> bug 1243726?
21:11:10 <uvirtbot> Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] https://launchpad.net/bugs/1243726
21:11:18 <markmcclain> was that intentional since the code merged?
21:11:59 <armax> I am still looking into the root cause
21:12:20 <markmcclain> ok.. should we put you as the assignee on the bug?
21:12:25 <armax> the partial fix to nova was to address an infinite loop
21:12:42 <markmcclain> ok
21:13:13 <armax> ah..sorry I confused bug 1243726 for bug 1235435
21:13:15 <uvirtbot> Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] https://launchpad.net/bugs/1243726
21:13:16 <uvirtbot> Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,Fix committed] https://launchpad.net/bugs/1235435
21:14:15 <armax> I will take bug 1243726
21:14:16 <uvirtbot> Launchpad bug 1243726 in neutron "tempest failure: No more IP addresses available on network" [Critical,Confirmed] https://launchpad.net/bugs/1243726
21:14:23 <markmcclain> armax: thanks
21:14:55 <markmcclain> Anyone have any other critical/high bugs that are rated too low?
21:15:06 <salv-orlando> jog0 has shared a list of gate offenders
21:15:17 <salv-orlando> two at least are neutron
21:15:23 <anteaya> that is on the agenda under tempest
21:15:30 <anteaya> we can talk about them now if you want
21:15:32 <salv-orlando> ok, forget about it
21:15:36 <salv-orlando> let's talk at the right time
21:15:59 <salv-orlando> markmcclain: let's move to the next topic then
21:16:30 <markmcclain> yeah.. we'll cover it then
21:16:58 <markmcclain> #topic Plugin/Driver Testing
21:17:09 <markmcclain> #link http://lists.openstack.org/pipermail/openstack-dev/2013-November/019219.html
21:17:40 <markmcclain> #info 3rd Party Plugin/Driver Requirements are now in effect
21:18:51 <markmcclain> Any questions about it?
21:18:59 <marun> what about lb?
21:19:16 <marun> technically not 3rd party, but we don't have integration testing in the gate at present...
21:19:27 <marun> (or rather, ml2+lb)
21:19:39 <mestery> markmcclain: This applies for all plugins, not just core plugins right?
21:19:55 <enikanorov> mestery: for services as well
21:20:03 <markmcclain> marun: yes we will need to add load balancing testing
21:20:07 <markmcclain> mestery: yes
21:20:10 <salv-orlando> technically we'd first have some tests for services, and then we'll enforce them for drivers
21:20:22 <enikanorov> markmcclain: i think marun is talking abut linux bridge
21:20:32 <marun> enikanorov: I am
21:20:47 <marun> I'm suggesting we should target full integration testing of all plugins, 3rd party and oss.
21:20:47 <markmcclain> too many LBs :)
21:21:00 <markmcclain> marun: agreed
21:21:00 <marun> sorry, linuxbridge then
21:21:05 <enikanorov> but sure we'll add tests from loadbalancing (and actually started working on them already)
21:21:20 <enikanorov> *for
21:21:20 <markmcclain> we should be able to add a linuxbridge variant to the gate list
21:21:42 <markmcclain> we're currently working on getting stability for I1
21:21:45 <mestery> Why add LB tests if it's going to be deprecated soon?
21:21:53 <markmcclain> and then in I2 we can add another gate check
21:21:55 <mestery> I think adding more tests which exercise the LB agent when used with ML2 would be good.
21:22:03 <marun> mestery: I meant ml2+lb, sorry
21:22:09 <rkukura> seems ml2+linuxbridge could run internally as part of the gate
21:22:10 <mestery> marun: OK, cool, no worries.
21:22:11 <marun> ml2+linuxbridge driver, sorry
21:22:25 <marun> rkukura: we need separate jobs, is all.  that's my point.
21:22:33 <markmcclain> rkukura: right it can be run as one of the gate checks
21:22:36 <marun> anyway, moving on
21:22:42 <marun> didn't mean to derail
21:22:52 <markmcclain> marun: no worries
21:22:53 <pcm_> Is the assumption that new core features will be having tests as well (e.g. VPNaaS)?
21:22:54 <mestery> marun: Not derail, that's a good topic IMHO.
21:23:11 <markmcclain> pcm_: yes
21:23:13 <pcm_> for icehouse-2?
21:23:33 <salv-orlando> pcm_: the scenarios should probably land by icehouse-1, if possible
21:23:40 <salv-orlando> but I agree it seems not that easy
21:23:59 <salv-orlando> and possibly not super-high priority
21:24:20 <marun> One more thing...
21:24:37 <markmcclain> I'd say the screnarios for VPNaaS should follow us getting the core tests stable
21:24:56 <markmcclain> s/VPNaaS/L3 Services/
21:25:04 <pcm_> I'll want to do vendor service, so wondering if there will be core there as reference.
21:25:33 <marun> pcm_: no mention in the email, but vendors should be participating in creation of representative integration tests.
21:25:44 <markmcclain> ++
21:25:45 <marun> pcm_: It shouldn't all be on the opensource side of things.
21:25:52 <enikanorov> pcm_: i guess the simple way is to start with API tests for the service, if there's none
21:25:55 <markmcclain> the tests are community work item
21:26:44 <markmcclain> that we'll all need to contribute
21:26:54 <markmcclain> marun: what was the other item you had?
21:27:08 <marun> markmcclain: sorry, the bit about vendor participation in test creation was it
21:27:18 <pcm_> markmcclain: gotcha.
21:27:35 <markmcclain> marun: thanks for making sure that was raised
21:27:40 <marun> :)
21:27:57 <salv-orlando> I heard no comment/concern on the integration testing policy draft. Can we consider it a done deal?
21:28:08 * marun keeps beating the dead horse
21:28:15 <markmcclain> yes
21:28:17 * mestery assumed it was a done deal. :)
21:28:46 <salv-orlando> ok so it's up to the PTL to cast it in the meeting minutes.
21:29:14 <markmcclain> #info 3rd Party Plugin/Driver Test Requirements in effect
21:29:47 * mestery notes the date and time. :)
21:30:05 <gongysh> what is 3rd party plugin/Driver? ml2+linux bridge is third party plugin/driver?
21:30:40 <markmcclain> gongysh: any plugin or driver that is not the reference implementation for Neutron
21:30:52 <markmcclain> includes vendor plugins/drivers
21:31:19 <markmcclain> ML2+Linuxbridge is something we maintain that we'll need to add a gate job for
21:31:48 <markmcclain> #topic Icehouse Project Plan
21:32:46 <markmcclain> I'm sure the summit was a whirlwind for everyone, so I wanted to map out the high level plan for this cycle
21:33:26 <markmcclain> Several sessions and conference talks really upon a few critical areas we need to address this cycle:
21:33:36 <markmcclain> Stability, Testing, and Parity
21:34:12 <markmcclain> These are important within our community and even more so outside of our community
21:34:15 <jog0> markmcclain: nova is re-evaulating nova-network at icehouse-2 or Jan 23rd
21:34:32 <markmcclain> jog0: good to know
21:35:18 <jog0> so ideally these would be goals for milestone 2  as well
21:35:54 <markmcclain> yeah we certainly have a lot of work to get done
21:36:17 <markmcclain> I've mapped out 10 or so high level area for us to work on this cycle:
21:36:17 <markmcclain> https://etherpad.openstack.org/p/NeutronIcehouseProjectPlan
21:37:05 <markmcclain> The first 5 all work address stability, testing, and parity
21:38:33 <mestery> markmcclain: This is a very good high level plan, nice work.
21:39:07 <markmcclain> mestery: thanks
21:39:47 <markmcclain> I'll be working with of the subteam leads to fill in the details for there subteam to make sure we're working towards the main objectives
21:39:54 <geoffarnold> Clarifying: (7) and (8) delivering in Icehouse, (9) and (10) post-Icehouse?
21:40:10 <geoffarnold> Otherwise this is ambiguous as an "Icehouse plan"
21:40:35 <markmcclain> geoffarnold: so on 7 I expect there will be Icehouse deliverables
21:41:09 <markmcclain> they team needs to distill out into clearer blueprints so that we can better track progress
21:41:25 <anteaya> 19 minutes of meeting time remaining
21:41:41 <markmcclain> 8) I expect the teams will be able to improve testing in their area and deliver the new features listed
21:41:45 <iwamoto> no service-chain-insertion in Icehouse?
21:42:14 <geoffarnold> I'd prefer it if the plan explicitly called out those areas where there are Icehouse deliverables (by I-2, right?) and those that are explicitly out of scope for Icehouse integration
21:42:16 <markmcclain> 9) are features that may or may not make the cut because they involve a bit of speculative work or require legal clearances to release code
21:42:26 <marun> fyi, i've posted a link to that etherpad that outlines a preliminary plan for the parity effort from beagles (I don't see him on here)
21:43:07 <salv-orlando> thanks marun. I think we can shift the discussion of the plan to etherpad/mailing list given that we all agree at least on priorities for items 1-7?
21:43:08 <geoffarnold> I'm unhappy about a "may or may not make the cut" in a release cycle focussed on stability
21:43:15 <markmcclain> the 10 are subteams that need to keep working with an eye towards J because there's consensus on a vague direction but not on how to do implement
21:43:56 <markmcclain> geoffarnold: all of the items in that bucket are new features
21:44:28 <marun> salv-orlando: +1
21:44:45 <markmcclain> stability needs to focus on what we have today as the new features are not drop in replacements
21:44:58 <markmcclain> I'll be sending a message to mailing list and we can discuss further
21:45:25 <jog0> markmcclain: this sounds like a really ambitious list, I am concerned the last 5 will reduce the amount of work being done on the first 5
21:45:26 <mestery> Only other concern I have is new subteam work being higher priority than existing subteam work, but the order may not matter there.
21:46:05 <geoffarnold> Can we also call out all API changes as early as possible?
21:46:09 <markmcclain> jog0: that's why I listed the others first as dev and review resources should be focused on 1-4
21:46:11 <markmcclain> -5
21:46:37 <salv-orlando> Can I use the 'shiny' adjective?
21:46:41 <jog0> markmcclain: what about just dropping some of the bottom ones until after I-2?
21:47:22 <salv-orlando> I would say 1-6 are must have; 7 and 8 are expected new features due to community request; 9 and 10 are the shiny new?
21:47:49 <mestery> salv-orlando: +1 to your organization thought.
21:47:58 <jog0> salv-orlando: makes sense to me, although 8 sounds big
21:48:10 <jog0> and could distact from 106
21:48:13 <jog0> 1-6*
21:48:29 <marun> how about saying the subteams should focus on bugfix/parity stuff only until 1-5 are deemed ready?
21:48:36 <markmcclain> jog0: most the blueprints for 8+ will be targets for I3 or J depending the speed of the everything elese
21:48:41 <salv-orlando> jog0: each sub-item of 8 has a sub team of at least 3 devs independently from the devs busy on core stuff (and btw ml2 is core stuff)
21:48:53 <jog0> markmcclain: cool, thanks for the clarification
21:48:58 <jog0> marun: ++
21:49:02 <marun> 990495
21:49:25 <markmcclain> ok.. we've got 10mins and anteaya has a lots of stuff to cover
21:49:28 <amotoki> I am not sure what 4) "nova-neutron integration" covers. is it related to nova-network deprecation?
21:49:48 <salv-orlando> amotoki: nova/network/neutronv2/api.py
21:49:49 <markmcclain> amotoki: nova-parity will cover the items to deprecate nova-net
21:50:04 <salv-orlando> or the module from hell
21:50:09 <markmcclain> nova-neutron integration is the pain and performance points in the integration between the systems
21:50:09 <amotoki> got it.
21:50:34 <markmcclain> I'll send out the email and we can resume on ML
21:50:40 <markmcclain> #topic Neutron Tempest
21:50:53 <anteaya> okay
21:51:10 <anteaya> starting with the bugs that are causing rechecks
21:51:26 <anteaya> #link https://bugs.launchpad.net/swift/+bug/1224001
21:51:31 <uvirtbot> Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [Critical,Fix released]
21:51:44 <anteaya> salv-orlando: how is this one going?
21:51:48 <salv-orlando> well, let's start by reopening it.
21:51:58 <salv-orlando> it should have never been marked as fix released.
21:52:05 <anteaya> okay, let's do that
21:52:31 <salv-orlando> The manifestation is different from the one that originally caused the high number of failures. From the logs I clearly see the floating IP being configured.
21:52:33 <markmcclain> I've reopened
21:52:38 <anteaya> thank you
21:52:52 <anteaya> anything more on this one now?
21:52:57 <anteaya> or shall i move on?
21:53:00 <salv-orlando> So I guess the failure in connectivity is due to the vm not booting, but I did not find a trace in the logs for that
21:53:16 <anteaya> salv-orlando: what more do you need to address this bug?
21:53:24 <salv-orlando> anteaya: time?
21:53:27 <anteaya> okay
21:53:34 <anteaya> let's talk next week, next
21:53:36 <anteaya> https://bugs.launchpad.net/neutron/+bug/1251448
21:53:37 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New]
21:53:51 <anteaya> again salv-orlando
21:54:09 <anteaya> thoughts on this one?
21:54:17 <salv-orlando> investigating this currently. The problem is that some tempest tests fail to delete a network under some circumstances, leading with multiple networks with the same name.
21:54:33 <anteaya> anything you need to help you with this one?
21:54:37 <salv-orlando> I can fix tempest for this, but the real issue is understanding why in some cases the network does not get deleted.
21:54:43 <marun> hmmm, that seems like a dumb bug.  should be easy to always use ids in tempest
21:54:44 <anteaya> also can you delegate this one to anyone?
21:55:01 <salv-orlando> marun: correct. This would be a tempest fix - an easy one.
21:55:08 <anteaya> marun: are you able to help salv-orlando with this one?
21:55:16 <anteaya> since he has the other one as well?
21:55:22 <mestery> salv-orlando: When I run tempest tests locally I consistently have leftover networks, ports, subnets, and routers. Not sure if it's related or not.
21:55:29 <rkukura> not sure masking the underlying issue is the best way forward
21:55:33 <marun> anteaya: I can take it
21:55:34 <salv-orlando> But there's an underlying neutron issue - the networks from time to time do not get deleted (remember we still do not have parallel testing so if there are two nets with the same name it's weird)
21:55:40 <anteaya> marun: great thanks
21:55:43 <salv-orlando> marun - all yours then
21:56:02 <marun> salv-orlando: unless you want to keep it?
21:56:10 <anteaya> 5 minutes left, can I cover the other bugs?
21:56:22 <enikanorov> salv-orlando: i wonder if we temporarily add db uniq constraint for the network name
21:56:27 <enikanorov> just for catching purposes
21:56:45 <enikanorov> that might help
21:56:48 <markmcclain> enikanorov: names are not unique
21:56:51 <salv-orlando> enikanorov: idk frankly
21:56:51 <enikanorov> right
21:56:52 <markmcclain> can change
21:57:04 <anteaya> next bug?
21:57:05 <markmcclain> *can't change that without pain for people
21:57:13 <markmcclain> i say next bug
21:57:16 <anteaya> https://bugs.launchpad.net/neutron/+bug/1235435
21:57:17 <marun> enikanorov: -1, i'd say we fix the symptom and worry about the root cause separately
21:57:19 <uvirtbot> Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,Fix committed]
21:57:44 <anteaya> incomplete and unassigned for neutron
21:57:49 <enikanorov> marun: that was suggestion how to catch it, not how to fix it
21:57:55 <armax> is this one still showing up?
21:57:57 <anteaya> tests are failing
21:57:58 <marun> enikanorov: fair enough
21:57:59 <anteaya> yes
21:58:12 <armax> cool, can you point me to them, plz?
21:58:22 <anteaya> see the log stash link at teh bottom of the bug report
21:58:27 <armax> k
21:58:47 <anteaya> last 2
21:58:49 <anteaya> https://bugs.launchpad.net/nova/+bug/1249065
21:58:50 <uvirtbot> Launchpad bug 1249065 in nova "Tempest failure: tempest/scenario/test_snapshot_pattern.py" [Undecided,Confirmed]
21:58:59 <anteaya> https://bugs.launchpad.net/neutron/+bug/1251784
21:59:02 <uvirtbot> Launchpad bug 1251784 in nova "nova+neutron scheduling error: Connection to neutron failed: Maximum attempts reached" [Undecided,New]
21:59:21 <markmcclain> looks like salv-orlando is triaging this one
21:59:29 <anteaya> both unassigned
21:59:35 <anteaya> I'm out of time, thanks
21:59:39 <anteaya> more info on the agenda
21:59:49 <salv-orlando> I am triaging 1249065 but no progress so far
22:00:07 <markmcclain> Sorry we ran short on time
22:00:11 <anteaya> salv-orlando: can you delegate?
22:00:21 <markmcclain> anteaya posted lots of good info so make sure to read
22:00:27 <markmcclain> I want to highlight a few things:
22:00:40 <salv-orlando> anteaya: I will reassign in the next few hours.
22:00:47 <anteaya> salv-orlando: thanks
22:00:49 <markmcclain> Code sprint in Montreal in January.. please let her know if you're interested
22:00:53 <marun> isn't 1251784 a duplicate?
22:01:35 <salv-orlando> marun: you're referring to bug 1211915?
22:01:37 <uvirtbot> Launchpad bug 1211915 in neutron/havana "Connection to neutron failed: Maximum attempts reached" [High,Fix committed] https://launchpad.net/bugs/1211915
22:01:42 <marun> salv-orlando: ye
22:01:43 <marun> s
22:01:54 <salv-orlando> jog0 said it's a different manifestation; perhaps different root cause.
22:01:59 <emagana> sorry, I missed the meeting, will catch up with the minutes
22:02:17 <marun> salv-orlando: ah, ok
22:03:37 <markmcclain> Ok talk to everyone on IRC and ML and have a great week
22:03:40 <markmcclain> #endmeeting