21:01:38 <markmcclain> #startmeeting Neutron
21:01:38 <salv-orlando> aloha
21:01:39 <openstack> Meeting started Mon Nov 25 21:01:38 2013 UTC and is due to finish in 60 minutes.  The chair is markmcclain. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:43 <openstack> The meeting name has been set to 'neutron'
21:02:29 <markmcclain> #link https://wiki.openstack.org/wiki/Network/Meetings
21:02:38 <markmcclain> #topic Announcements
21:02:49 <markmcclain> #info Icehouse-1 is December 5th
21:03:12 <emagana> hi all!
21:03:19 <markmcclain> It's next week which means code needs to merged by Tuesday Dec 3rd
21:03:26 <Swami> hi
21:03:37 <markmcclain> #link https://launchpad.net/neutron/+milestone/icehouse-1
21:03:45 <safchain> hi
21:03:49 <colinmcnamara> hi
21:03:59 * salv-orlando regrets the week off taken during I-1
21:04:20 <markmcclain> the first milestone always seems to feel rushed
21:04:35 * mestery nods in agreement.
21:04:43 <markmcclain> and even more so since many will be taking a few days off at the end of this week
21:05:24 <markmcclain> #topic Bugs
21:05:27 <markmcclain> anteaya: around?
21:05:53 <markmcclain> #link https://bugs.launchpad.net/neutron/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Confirmed&field.status=Triaged&field.status=In+Progress
21:06:31 <armax> I'd add bug #1254236 to the list
21:06:33 <uvirtbot> Launchpad bug 1254236 in neutron "TestNetworkQuotaBasic tearDownClass  fails while deleting port" [Undecided,New] https://launchpad.net/bugs/1254236
21:06:37 <markmcclain> not sure if anteaya is active at the moment.. she has gone through and listed the gate blocking bugs in the agenda
21:06:44 <clarkb> I am going to mention really quickly that the way jenkins processes test logs after tests run has been updated to speed up the neutron unittest jobs. py26 is ~40 minutes now and py27 is ~27 minutes
21:06:49 <armax> I found out about it last week
21:06:50 <armax> it's getting hot pretty fast
21:06:55 <markmcclain> clarkb: awesome
21:06:57 <clarkb> shouldn't see anymore job timeouts on those
21:07:02 <sc68cal> woooo!
21:07:13 <markmcclain> armax: thanks
21:07:15 <nati_ueno> cool!
21:07:17 <armax> clarkb: thanks
21:07:29 <armax> clarkb: so do you think that review https://review.openstack.org/#/c/58022/ is no longer worth it?
21:08:30 <clarkb> armax: correct, we should keep an eye on it but I think we are well below variability in test times
21:08:45 <clarkb> *well below the timeouts considering variability in test times
21:09:06 <armax> clarkb: ok sounds good
21:09:15 <markmcclain> salv-orlando: is working on https://bugs.launchpad.net/swift/+bug/1224001
21:09:17 <uvirtbot> Launchpad bug 1224001 in neutron "test_network_basic_ops fails waiting for network to become available" [High,In progress]
21:09:33 <salv-orlando> I think I and otherwise guy merged the patches that should fix it
21:09:41 <markmcclain> cool
21:09:43 <salv-orlando> we hope it won't show up again, but I know it will
21:09:52 <salv-orlando> (more later when we discuss parallel testing)
21:10:12 <markmcclain> k
21:10:35 <markmcclain> https://bugs.launchpad.net/neutron/+bug/1251448
21:10:37 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New]
21:10:43 <markmcclain> marun: any progress?
21:11:00 <marun> mlavalle seems to have narrowed the issue to not having tenant isolation turned on
21:11:19 <marun> is there a reason why any of the gates turn it off?
21:11:41 <markmcclain> when it was activate earlier it actually caused more gate failures
21:11:52 <marun> i thought that was parallelization
21:11:55 <marun> :\
21:12:10 <marun> damned if we do, damned it we don't
21:12:43 <markmcclain> yeah well as we work on improving the test situation we will uncover a few issue along the way
21:13:12 <beagles> what are we considering the fundamental error there?
21:13:15 <salv-orlando> marun, markmcclain, on this note of tenant isolation I might be dumb, but the 'isolated' job we run
21:13:20 <beagles> that the tearDown fails?
21:13:33 <marun> it is a cleanup issue, yeah
21:13:34 <salv-orlando> which set allow_tenant_isolation = True - aren't they using isolation?
21:13:37 <markmcclain> beagles: yes
21:14:06 <salv-orlando> or does bug 1251448 occur only on non-isolated jobs?
21:14:08 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New] https://launchpad.net/bugs/1251448
21:14:14 <markmcclain> sdague or jog0: ^
21:15:03 <marun> salv-orlando: but if any job doesn't use isolation, then we have potential races in cleanup
21:15:13 <armax> it looks like it occurs on non-isolated jobs
21:15:40 <salv-orlando> armax: that was what I was looking for. If we can confirm that it does not happen with isolation, then the issue is nailed
21:15:44 <marun> so the question is, are the non-isolated jobs going to continue?
21:15:55 <sdague> salv-orlando: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiIHBvc3NpYmxlIG5ldHdvcmtzIGZvdW5kLCB1c2UgYSBOZXR3b3JrIElEIHRvIGJlIG1vcmUgc3BlY2lmaWMuIChIVFRQIDQwMClcIiAgICAgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODU0MTQxMjYzNjcsIm1vZGUiOiJ0ZXJtcyIsImFuYWx5emVfZmllbGQiOiJidWlsZF9uYW1lIn0=
21:15:57 <salv-orlando> marun: should be discontinued, this is my understanding
21:16:08 <sdague> yes, only "non issolated" jobs
21:16:19 <marun> so yeah, not sure we want to bother fixing.  isolation is the ideal fix
21:16:34 <salv-orlando> sdague: thanks for showing me how to the query - I was hopelessly trying to do that by myself
21:16:34 <sdague> marun: so we're not fully isolated on the isolated jobs right now
21:16:56 <marun> sdague: ?
21:17:20 <sdague> last I knew the API tests aren't actually properly issolated, because the load pushed neutron over the edge
21:17:31 <sdague> mtreinish has the details
21:17:40 <marun> sdague: ah, ok
21:17:55 <marun> so it will be necessary to ensure name uniqueness
21:18:03 <markmcclain> no that we've activated minimize polling we might was to look and see if the load is still too high
21:18:11 <markmcclain> s/was/want/
21:18:15 <mtreinish> sdague: yeah the neutron api tests don't have isolation turned on
21:18:36 <mtreinish> last time I did that the gate for neutron failed basically 100% of the time
21:18:56 <mtreinish> it needs a rebase but: https://review.openstack.org/#/c/53459/
21:19:43 <salv-orlando> mtreinish, sdague, marun, markmclain and whoever else: I'm working on that too as a part of the parallel testing effort
21:19:52 <markmcclain> cool
21:20:00 <marun> the polling switch has made a difference in runtimes at least
21:20:12 <markmcclain> Any other critical bugs we need to discuss?
21:20:13 <marun> i'm seeing 20-25% less execution time
21:20:22 <jog0> so neutron gate is failing at 33% right now
21:20:32 <jog0> .gate-tempest-devstack-vm-neutron
21:21:42 <markmcclain> jog0: good to know.. was hoping for a lower number with all of the recent changes
21:22:17 <jog0> there were some outstanding patches in the nova side that should help
21:22:21 <jog0> let me hunt those down
21:22:41 <markmcclain> me and arosen pinged the nova room earlier with those
21:22:42 <arosen> https://review.openstack.org/#/c/57711/ , https://review.openstack.org/#/c/57069/6
21:23:00 <arosen> jog0: ^
21:23:03 <jog0> arosen: thanks thats it
21:23:12 <jog0> arosen: I will  review those
21:23:18 <markmcclain> jog0: thanks!
21:23:22 <arosen> jog0:  awesome thanks
21:23:38 <markmcclain> Any other critical bugs the team needs to track?
21:23:53 <markmcclain> #topic Docs
21:24:01 <jog0> https://bugs.launchpad.net/neutron/+bug/1250168  https://bugs.launchpad.net/neutron/+bug/1251448  https://bugs.launchpad.net/neutron/+bug/1235435
21:24:04 <uvirtbot> Launchpad bug 1250168 in neutron "gate-tempest-devstack-vm-neutron-large-ops is failing" [Critical,Confirmed]
21:24:07 <markmcclain> #undo
21:24:07 <openstack> Removing item from minutes: <ircmeeting.items.Link object at 0x359b710>
21:24:14 <jog0> are all gate bugs, none are listed as critical ATM
21:24:28 <jog0> not sure where the root causes of those bugs are though
21:25:06 <armax> Partial fix for bug 1235435 is review 57069
21:25:07 <uvirtbot> Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,In progress] https://launchpad.net/bugs/1235435
21:25:20 <armax> one of the two that aaron pointed you at
21:25:32 <armax> s/aaron/arosen
21:26:06 <markmcclain> bug 1235435 is because isolations isn't turned on
21:26:08 <uvirtbot> Launchpad bug 1235435 in nova "'SubnetInUse: Unable to complete operation on subnet UUID. One or more ports have an IP allocation from this subnet.'" [Medium,In progress] https://launchpad.net/bugs/1235435
21:26:15 <markmcclain> salv-orlando: wanted to talk about parallel testing
21:26:17 <jog0> armax: cool thanks
21:26:30 <markmcclain> work may help solve that issue
21:26:33 <salv-orlando> markmcclain: I'm fine either way if you want to move it oto open discussion
21:26:43 <salv-orlando> and continue with the normal agenda
21:26:49 <jog0> markmcclain: we just saw bug 435 or the query we have is bad https://review.openstack.org/#/c/58352/
21:26:50 <uvirtbot> Launchpad bug 435 in ethereal "ethereal uses gksu instead of gksudo for root menuitem" [Low,Invalid] https://launchpad.net/bugs/435
21:26:57 <jog0> anyway thanks
21:26:57 <armax> markmclain: not in all cases
21:27:11 <armax> but we can talk about it offline if you want to learn more
21:27:31 <markmcclain> ok.. we can discuss the actually bug in teh room after the meeting, lets keep moving so that we don't run out of time
21:27:48 <markmcclain> #topic Docs
21:27:55 <emagana> hi
21:28:11 <markmcclain> Metering docs have merged.. thanks to sfchain
21:28:22 <markmcclain> emagana: anything else to add that's not on the agenda?
21:28:28 <emagana> yes!
21:28:56 <emagana> just a head ups about a Doc Bug Day
21:29:19 <emagana> #info https://wiki.openstack.org/wiki/Documentation/BugDay Dec 20th
21:29:43 <emagana> It could be great to have some neutrons participating
21:29:53 <safchain> I also added a section about the ml2 scenarios, with to start 2 scenarios with the l2 population mechanism driver
21:30:16 <salv-orlando> emagana: fine but I really don't feel comfortable around electrons
21:30:39 <markmcclain> emagana: thanks for the reminder about the doc bug day
21:30:46 <mestery> salv-orlando: heh
21:30:47 <emagana> salv-orlando: keep neutral.. I mean neutron
21:31:02 <emagana> nothing else markmcclain
21:31:07 <markmcclain> thanks for the update
21:31:10 <salv-orlando> emagana: so we don't have network bugs open against either the admin-guide or netconn-api?
21:31:11 <markmcclain> #topic Nova Parity
21:31:16 <markmcclain> #undo
21:31:17 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x35d8110>
21:31:33 <emagana> Yes, we do!
21:32:01 <emagana> I did not believe we want to discuss all of them, at least we have a high priority ones
21:32:14 <emagana> as we had with metring part
21:32:24 <emagana> s/metring/metering
21:32:46 <emagana> just wanted to keep it short, but we can go over more details if needed, advise!
21:33:10 <markmcclain> we still have a lot of material to cover, so let's keep moving for now
21:33:15 <markmcclain> #topic Nova Parity
21:33:51 <beagles> somebody is playing my song
21:34:00 <markmcclain> yep
21:34:36 <beagles> I was on PTO last week so I apologize for my silence... however before I left someone proposed creating a draft list of milestones related to parity
21:34:54 <markmcclain> yes will be helpful to track progress
21:35:00 <beagles> I've posted a link to the ethepad in the agenda... it was a complete off the cuff affair,but it fits the schedule (I think)
21:35:03 <dkehn> agreed
21:35:04 <markmcclain> and we can fill it in with action items
21:35:16 <markmcclain> #link https://etherpad.openstack.org/p/icehouse-neutron-nova-parity-prelim-outline
21:35:17 <beagles> I'd appreciate suggestions and refinements
21:35:37 <salv-orlando> thanks a lot beagles
21:36:01 <beagles> my pleasure: I've mentioned some things in there that warrant conversations offline
21:36:07 <salv-orlando> do you reckong we are on schedule with your plane?
21:36:10 <beagles> or out-of-the-meeting
21:36:11 <salv-orlando> plane/plan
21:36:22 <beagles> we are a little behind but not dire I think
21:36:48 <markmcclain> ok.. we can discuss offline farm out the work if necessary to get us back on track
21:37:26 <markmcclain> Anything else for this week?
21:37:46 <sc68cal> ipv6?
21:38:23 <markmcclain> sc68cal: yeah we'll have to look at gaps there too
21:38:29 <markmcclain> #topic Tempest
21:39:02 <markmcclain> Lots of updates to the etherpad
21:39:04 <markmcclain> #link https://etherpad.openstack.org/p/icehouse-summit-qa-neutron
21:39:22 <markmcclain> mlavalle or salv-orlando who wants to update first?
21:39:43 <salv-orlando> I'd let mlavalle go first - he's the topic lead, I think
21:39:59 * salv-orlando at least that's what the wiki page says :)
21:40:02 <mlavalle> markmcclain: as indicated in the ether pad, I have taken responsibility  for the api testing
21:40:19 <markmcclain> ok
21:40:40 <mlavalle> my plan is to have a plan by this week, so we can let other contrabute to this effort
21:41:13 <markmcclain> great… I'll follow up with you later in the week
21:41:18 <mlavalle> I also worked over the weekend on https://bugs.launchpad.net/neutron/+bug/1251448
21:41:20 <uvirtbot> Launchpad bug 1251448 in neutron "BadRequest: Multiple possible networks found, use a Network ID to be more specific. " [High,New]
21:41:42 <mlavalle> I've updated the but in launchpad with my analysis
21:41:48 <markmcclain> great thanks
21:42:01 <mlavalle> it seem to me it's a race condition
21:42:18 <mlavalle> when deleting ports
21:42:36 <markmcclain> yeah the test changes have definitely revealed several races
21:42:53 <mlavalle> so by the time the tea tries to delete a subnet, it finds ports still active
21:43:11 <mlavalle> so a net and a subnet are left alive in the demo user
21:43:21 <mlavalle> that cascades to the other tests after that
21:43:42 <markmcclain> yeah.. that will do it
21:44:00 <mlavalle> tonight I will attempt to create a fix….. i might ask a lot of questions tomorrow
21:44:07 <markmcclain> mlavalle: sounds great
21:44:12 <salv-orlando> mlavalle: I'll be around
21:44:16 <mlavalle> that's all I have today
21:44:22 <markmcclain> mlavalle: thanks
21:44:23 <salv-orlando> in terms of "items" pertaining to testing
21:44:44 <salv-orlando> on the etherpad we have at least three we need to discuss here: parallel, full, and grenade testing
21:45:08 <salv-orlando> as I do things without any logic, let's start from the bottom
21:45:13 <markmcclain> ok
21:45:28 <salv-orlando> grenade - the issue is simple, there's no grenade testing, and we're looking for an owner
21:45:41 <salv-orlando> last week I said "we have no owner" and somebody said "hold on"
21:45:46 <salv-orlando> does this mean we have an owner now?
21:46:23 <markmcclain> I don't think we've had anyone step up
21:46:30 <salv-orlando> silence == no owner, I guess. markmcclain, can we prioritise finding a volunteer?
21:46:38 <beagles> nobody wants to dive on the grenade?
21:46:44 <dkehn> cue crickets
21:46:50 <markmcclain> #action markmcclain to find grenade lead
21:47:03 <markmcclain> I work on that this week
21:47:04 <salv-orlando> full neutron devstack test
21:47:17 * beagles nods
21:47:20 <sc68cal> Does grenade piece also include the OVS -> ML2 migrations?
21:47:21 <salv-orlando> This work item is a mix of issues with neutron and features missing from neutron.
21:47:22 <marun> markmcclain: please coordinate with jlibosva
21:47:32 <markmcclain> marun: will do
21:47:34 <sc68cal> not sure if that's out of scope for grenade
21:47:38 <salv-orlando> At the moment we have rossella_s performing an assessment
21:48:00 <markmcclain> sc68cal: the ml2 team will need to add migration scripts
21:48:02 <salv-orlando> she's not online, but as she's in my timezone I will ping her tomorrow
21:48:04 <beagles> salv-orlando, there is an overlap with the parity related activities outlined for this week
21:48:34 <markmcclain> beagles can you coordinate with rossella_s?
21:48:37 <mestery> sc68cal: Yes, we're aware of the migration requirements from OVS to ML2, but have no specific plans at the moment.
21:48:38 <salv-orlando> beagles: I am aware of that; I thought I was good to get someone started anyway on the assessment of what needs to be done to get that test passing
21:48:42 <beagles> absolutely :)
21:48:45 <mlavalle> salv-orlando: a couple of months ago I put together (with the help of others) this https://etherpad.openstack.org/p/gate-tempest-devstack-vm-quantum-full
21:48:47 <markmcclain> thanks
21:48:53 <beagles> salv-orlando, no argument there :)
21:49:06 <salv-orlando> mlavalle: thanks for that I will add it to other etherpad
21:49:13 <salv-orlando> and finally… parallel testing
21:49:38 <mlavalle> salv-oralando: rosella_s can ping me if clarification is needed
21:49:49 <markmcclain> salv-orlando: you've been digging into this one right?
21:49:51 <salv-orlando> mlavalle: I will ell there
21:50:08 <salv-orlando> Basically there are 3 issues, 1 in devstack/tempest, and 2 in neutron
21:51:05 <salv-orlando> devstack/tempest give neutron only 16 public IPs. this is fine for nova-network, but neutron uses a public IP for each tenant router. If you add the fact that some tests do not cleanup routers, you can easily run out of IPs
21:51:06 <EmilienM> salv-orlando: i could help with grenade, even if i  have never contributed yet
21:51:24 <salv-orlando> armax has a patch for 2 test cases leaving a router behind
21:51:36 <salv-orlando> I have a patch proposing to start defaulting the public network to /27
21:51:53 <salv-orlando> any questions before we move to the neutron side?
21:52:10 <markmcclain> we have 8 mins, so I think we have to move on
21:52:15 <mlavalle> salv-orlando: I think the number of ip's is configurable in the tempest conf file
21:52:41 <armax> salv-orlando: the patches you're talking about are: https://review.openstack.org/#/c/57805/ and https://review.openstack.org/#/c/58054/
21:52:43 <salv-orlando> mlavalle: this is about the public network created by devstack I think, but we can take this detail offline I think
21:52:52 <mlavalle> cool
21:53:06 <lifeless> SpamapS: hey - whats the bug number about that race with new networks?
21:53:12 * salv-orlando is funny to say offline while you'll still be online talking about this
21:53:17 <lifeless> SpamapS: thinking it would be topical to raise it here as something tempest should be testing
21:53:55 <salv-orlando> I think we're out of time for the tempest discussion - I will update on neutron issues separately, unless you feel it's important
21:54:26 <markmcclain> yeah.. I think we discuss on this on the mailing list
21:54:42 * markmcclain fingers won't work today
21:55:08 <markmcclain> #topic Subteams
21:55:31 <markmcclain> We've got several new subteams in addition to our existing ones.
21:55:43 <markmcclain> Please check the agenda for updates
21:56:05 <markmcclain> #topic Open Discussion
21:56:28 <markmcclain> Any items need to cover in 4 minutes or less?
21:57:48 <nati_ueno> Is lbbas api still experimental?
21:58:09 <markmcclain> no with Havana that api will need to follow the deprecation cycle
21:58:31 <nati_ueno> markmcclain: Thanks
21:58:45 <nati_ueno> it looks there is a patch which change beheivior
21:58:49 <nati_ueno> of LBaaS API
21:59:01 <markmcclain> link?
21:59:13 <nati_ueno> https://review.openstack.org/#/c/53821/
21:59:47 <nati_ueno> health monitor was cascate deleted before
21:59:56 <nati_ueno> This patch changes that beheivior
22:00:06 <markmcclain> thanks for the link.. I'll follow up with teh sub-team
22:00:12 <nati_ueno> markmcclain: Thanks
22:00:19 <markmcclain> we're out of time for this week
22:00:24 <markmcclain> I want to thank salv-orlando, armax, arosen, jog0, marun, anteaya for digging into the gate problems
22:00:40 <jog0> markmcclain: thank you
22:00:50 <jog0> to everyone fixing them
22:00:58 <armax> I hope I just don't add more ;)
22:01:05 <arosen> ditto :)
22:01:10 <markmcclain> Have a great week and for everyone traveling for the holidays later this week.. safe travels
22:01:12 <markmcclain> #endmeeting