14:00:14 <armax> #startmeeting networking 14:00:15 <openstack> Meeting started Tue Nov 17 14:00:14 2015 UTC and is due to finish in 60 minutes. The chair is armax. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:18 <openstack> The meeting name has been set to 'networking' 14:00:21 <carl_baldwin> Hi 14:00:26 <hoangcx> Hi 14:00:31 <pc_m> hi 14:00:32 <kevinbenton> Hi 14:00:35 <obondarev_> o/ 14:00:41 <SridharG> hi 14:00:47 <wwriverrat> hi 14:00:56 <jschwarz> \o/ 14:01:11 <rossella_s> hi 14:01:18 <armax> hi folks 14:01:36 <mestery> Good morning armax! 14:01:53 <ajo> o/ :) 14:01:55 <garyk1> mestery: is that a wake up call? 14:02:01 <armax> #topic Announcements 14:02:02 <regXboi> WAKE UP 14:02:05 <mestery> garyk1: I think armax needs one. 14:02:08 <regXboi> ^^^^^^^ that is a wake up call 14:02:09 <armax> I do 14:02:13 * mestery hands armax espresso 14:02:15 <ajo> good morning :-) 14:02:29 <garyk1> he should be triaging bugs and not sleeping 14:02:32 <salv-orlando> aloha 14:02:45 * xgerman wonders who cam up with this 6 am time slot 14:02:51 <armax> #link https://wiki.openstack.org/wiki/Network/Meetings 14:03:07 <regXboi> xgerman: blame daylight savings time 14:03:09 <armax> #link https://wiki.openstack.org/wiki/Network/Meetings#Announcements_.2F_Reminders 14:03:19 <mhickey> Hey 14:03:23 <armax> a couple of things worth sharing 14:03:25 <jschwarz> xgerman, it's 16:00 here in Europe.. this meeting is for us as well as you American guys ;-) 14:03:27 <salv-orlando> gcc 14:03:29 <garyk1> xgerman: you can always read the logs and send a mail to the list if there are issues that trouble you 14:03:32 <salv-orlando> meh 14:03:41 <regXboi> salv-orlando: gcc: not found 14:04:13 <ajo> lol regXboi salv-orlando 14:04:15 * armax gives people time to go through the reminders while he snoozes a bit 14:04:37 <akamyshnikova> hi! sorry being late 14:04:47 <hichihara> hi 14:04:51 <amotoki> just announcement of sharing neutron meetup photos in Tokyo https://goo.gl/photos/HHTwgCSXoPuG5iqt5 14:04:59 <ajo> thanks for the reminders armax 14:05:21 <hichihara> amotoki: Nice! :) 14:05:21 <armax> it’s worth noting the creation of the assert tags 14:05:31 <armax> mestery: can tell us more, since he’s the TC dude 14:05:46 <armax> I can’t see markmclain 14:05:51 <armax> markmcclain: ping? 14:05:53 <mestery> armax: Which tags did you have qusetions about? 14:06:09 <mestery> armax: For the upgrade tags, we don't qualify for rolling-upgrade yet 14:06:13 <mestery> But supports upgrade we can define 14:06:16 <armax> mestery: right 14:06:19 <mestery> BEcause we clearly support upgrades 14:06:19 <ajo> amotoki : oh guys you had fun :') 14:06:25 <mestery> I'll propose a patch for that today 14:06:28 <armax> mestery: and I did: https://review.openstack.org/#/c/246242/ 14:06:34 <mestery> armax: You beat me to it 14:06:38 <armax> mestery: of course 14:06:40 <armax> :) 14:06:46 <mestery> armax: I hadn't reviewed governance patches yet today 14:06:49 <armax> that said, the rolling upgrade is technically not htat far off 14:07:01 <mestery> right 14:07:16 <armax> my wishful thinking is that garyk and sc68cal_ are gonna help with the partial grenade upgrade 14:07:42 * regXboi wonders if it's time to rename grenade 14:07:57 <garyk1> armax: the coming week i am going to be on the bugs 14:07:58 <armax> there has been a thread going on lately 14:08:00 <armax> #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/079344.html 14:08:24 <armax> garyk1: multitasking? :) 14:08:47 <garyk1> :) 14:09:10 <ajo> garyk1: I can keep helping a bit in bg :), we need a deputy volunteer for the week after 14:09:20 <korzen> the patch introducing the multinode grenade for neutron: https://review.openstack.org/#/c/245862 14:09:22 <armax> anyhow, it sounds like we could easily grasp the rolling-upgrade tag if we pull ourselves together and manage to have CI for the grenade partial job 14:09:36 <garyk1> i will be in touch with sc68cal_ and go from there 14:09:58 <armax> korzen: thanks for the link! 14:10:24 <regXboi> unfortunately, multinode jobs still look ill 14:10:51 <armax> regXboi: true, but that shouldn’t stop us from making progress 14:10:53 <kevinbenton> regXboi: is 'ill' a hip word for 'cool'? 14:11:21 <kevinbenton> As in, 'bro, check out this ill job' 14:11:23 <regXboi> no, in this case "ill" == "still appear to be at 100% error rate" 14:11:26 <armax> as for other reminders, there have been a couple of suggested improvements to the way we handle blueprints 14:11:30 <armax> and rfe's 14:11:44 <armax> full process detailed on: 14:11:45 <armax> #link http://docs.openstack.org/developer/neutron/policies/blueprints.html 14:12:27 <armax> regXboi: do we have a culprit? 14:12:42 <regXboi> armax: there was yesterday - I'm looking to see if it still exists 14:12:49 <hichihara> Who tries to fix multinode job issue? 14:13:43 <regXboi> ok, I *think* that the code to fix merged in the last hour - I'll keep an eye on it 14:14:01 <armax> regXboi: link? 14:14:16 <armax> regXboi: it looks like both multi-jobs broke 14:14:33 <regXboi> armax: all the multinode jobs broke - getting the links from eavesdrop now 14:14:44 <armax> regXboi: ok thanks 14:15:06 <obondarev> armax: yeah, both for the same reason: live migration test 14:15:07 <garyk1> regXboi: http://logs.openstack.org/18/245518/5/check/gate-tempest-dsvm-neutron-multinode-full/5a3c901/logs/testr_results.html.gz (live migration test) 14:15:50 <armax> obondarev: looks like that’s a tempest issue 14:15:53 <garyk1> why do we need livemigration for neutron jobs? 14:16:15 <obondarev> garyk1: it's a part of full tempest suite I guess 14:16:17 <armax> garyk1: why do we need to test neutron at all? 14:16:19 <garyk1> it is something that is usually broken at the best of times in nova 14:16:47 <armax> garyk1: even in the nova net case you mean? 14:16:53 <garyk1> armax: why the cynicism? 14:17:10 <ajo> recently I found a live migration issue in nova 14:17:12 <armax> garyk1: I was making a joke 14:17:13 <garyk1> armax: yes, there is even a working group setup to address live migrations 14:17:21 <regXboi> ok patch that broke multinode: https://review.openstack.org/245996 14:17:25 <ajo> they don't wait on port notifications from neutron, before bringing up the VM at destination 14:17:30 <ajo> that only shows up in mutlinode 14:17:32 <regXboi> patch that fixed multinode: https://review.openstack.org/#/c/233711/ 14:17:54 <ajo> VM is resumed before network is wired, RARP messages get blackholed, so the nodes don't know where the MAC address belongs to 14:17:57 <armax> regXboi: how can 245996 break multi node? 14:18:06 <armax> regXboi: you sure it’s the right link 14:18:15 <regXboi> ugh 14:18:17 <regXboi> no it isn't 14:18:23 <kevinbenton> ajo: any reason why it hadn't been fixed? 14:18:24 <regXboi> 233711 broke multinode 14:18:34 <obondarev> ajo: related https://bugs.launchpad.net/neutron/+bug/1414559 ? 14:18:34 <openstack> Launchpad bug 1414559 in neutron "OVS drops RARP packets by QEMU upon live-migration - VM temporarily disconnected" [Undecided,Invalid] - Assigned to sean mooney (sean-k-mooney) 14:18:37 <regXboi> https://review.openstack.org/#/c/245697/ supposedly fixed it 14:18:41 <ajo> yeah 14:18:43 <ajo> obondarev : exactly 14:18:45 <obondarev> ajo: I'm currently working on https://bugs.launchpad.net/neutron/+bug/1414559 14:18:46 <ajo> thanks, it's that one 14:19:04 <obondarev> ajo: have patches for nova and neutron, testing 14:19:05 <ajo> obondarev , but as I saw, it was more a nova thing, because they start the VM before network is wired 14:19:09 <regXboi> amuller and I discussed this yesterday in channel: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-11-16.log.html#t2015-11-16T21:42:10 14:19:09 <ajo> they should use notification 14:19:09 <ajo> ack 14:19:18 <ajo> that's likely to be the culprit of live migration failures 14:19:24 <armax> ok, it looks like we have a handle on this then 14:19:26 <ajo> I investigated that for 1 week.. 14:19:31 <armax> but the failure I observed 14:19:34 <armax> that garyk1 shared 14:19:44 <armax> was related to a setup issue 14:19:49 <armax> nothing to do with the dataplane 14:19:50 <obondarev> ajo: neutron should send notification in the first, we can take it offline to not wast meeting time 14:20:00 <ajo> obondarev: ack 14:20:28 <armax> ajo, obondarev: even though the change you’re referring to may be addressing some other instability 14:20:43 <salv-orlando> ajo: I seem to recall that upon certain events nova does not wait for a notification from neutron before restaring the VM 14:20:57 <armax> salv-orlando: how rude 14:20:59 <ajo> armax : ack 14:21:03 <salv-orlando> the logic is buried deep in the compute manager so it's not easy to grasp 14:21:17 <salv-orlando> but this was over a year ago, maybe in the meanwhile things have changed 14:21:19 <ajo> yes, this was a race condition by nova not honoring the notification, does not happen 100% of the time. 14:21:49 <salv-orlando> ajo: which might be consistent with nova ignoring the notification and therefore things being racey 14:21:57 <armax> ok, so someone maybe be interested in going to the nova mid-cycle meetup and yell at the nova people? the event is scheduled for Jan 26 2016 14:22:00 <armax> details here: 14:22:02 <armax> #link https://www.eventbrite.com.au/e/openstack-mitaka-nova-mid-cycle-meetup-tickets-19326224257 14:22:14 <kevinbenton> I can't get :'( 14:22:19 <kevinbenton> Go* 14:22:22 <mestery> It's in London, right armax? 14:22:27 <armax> mestery: bristol 14:22:31 <armax> that was the last of the reminders 14:22:32 <ajo> I think rdopiera from redhat was looking at it, 14:22:38 <armax> let’s move on to the next section 14:22:38 <salv-orlando> armax: I think some discussion in IRC and a bug report will lead to a patch and a fix 14:22:42 <ajo> but I could eventually try to speedup it, or help 14:22:46 <salv-orlando> no need to put yourself on a plane ;) 14:23:03 <ajo> specially if it's hurting our jobs, but it seems obondarev had already a good look on the issue 14:23:09 <armax> salv-orlando: I know, I found that lack of sleeps makes me particular prone to making jokes 14:23:26 <armax> #topic Blueprints 14:23:43 <armax> #link https://launchpad.net/neutron/+milestone/mitaka-1 14:23:46 <salv-orlando> armax: this is because I have poured scotch in your nespresso capsules 14:24:07 <garyk1> sorry to barge in but this may be what happens with the live migrate - https://github.com/openstack/nova/commit/150406679c14778f0479bc6d4e77df5119c0bdae 14:24:08 <armax> salv-orlando: all makes sense now! 14:24:44 <armax> obondarev: just filed https://blueprints.launchpad.net/neutron/+spec/improve-dvr-l3-agent-binding 14:25:08 <obondarev> armax: right, it's for already approved spec 14:25:26 <obondarev> armax: bp was missing I guess 14:25:27 <armax> obondarev: technically something was something dangling from past days 14:25:35 <armax> obondarev: we need an approver 14:26:04 <obondarev> armax: I think carl_baldwin is the right person 14:26:05 <carl_baldwin> o/ 14:26:08 <armax> I can’t be one, I am already approver on a bunch of other bp’s and I won’t be able to help you with reviews and stuf f as much 14:26:23 <obondarev> armax: k, changing to carl_baldwin 14:26:47 <armax> carl_baldwin: do you have fire power? 14:27:01 <carl_baldwin> Yes, I can take this 14:27:07 <obondarev> carl_baldwin: thanks 14:27:29 <pc_m> armax: The multipel subnets to connect to vpn commits are all in (though it says needs code-review) 14:27:31 <shihanzhang> this one https://review.openstack.org/#/c/196959/ was missed 14:28:55 <amotoki> pc_m: I should update the status of yours. 14:29:04 <garyk1> shihanzhang: any reason why the norifictaion do not suffice? 14:29:28 <armax> anyone wants to raise flags over the progress of any of the blueprints targeted for M1? 14:30:00 <pc_m> armax: I'll help part-time on splitting neutron into base library 14:30:18 <pc_m> (already talking to dougwig 14:30:20 <armax> pc_m: thanks 14:30:28 <garyk1> armax: i would like to know from other what they think of the timestamp bp? 14:30:30 <mestery> pc_m: Any progress there? 14:30:40 <mestery> Because that one doesn't look like it's making a ton of progress according to LP at least 14:30:49 <mestery> pc_m: Regarding https://blueprints.launchpad.net/neutron/+spec/neutron-lib 14:30:50 <pc_m> Doug and I have some protoypes, but need to get test working. 14:30:54 <mestery> ACk 14:31:20 <armax> garyk1: faik, that must be resubmitted for mitaka, but that aside, there’s a change from kevinbenton that will help add this type of info to neutron resources 14:31:32 <armax> and it’s a requirement to timestamps, tags and the likes 14:31:34 <garyk1> armax: ok, thanks 14:31:52 <garyk1> that makes more sense that it be something generic 14:31:55 <garyk1> fyo - https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp 14:32:02 <armax> that said, we currently have 21 blueprints targeting ‘neutron’ for M1 14:32:04 <garyk1> kevinbenton: can you please send a link to the work if possible 14:32:57 <wwriverrat> armax: requesting for direction from this group: Was looking for your guidance on how to proceed with "Add API extension for reporting IP address usage statistics": https://review.openstack.org/#/c/212955/ 14:32:58 <armax> garyk1: https://review.openstack.org/#/c/222079/ 14:33:02 <xgerman> armax is flavor among them? 14:33:13 <armax> xgerman: yes 14:33:14 <shihanzhang> garyk1, this is the link https://review.openstack.org/222079 14:33:39 <armax> we had 28 blueprints targeted for the whole of Liberty 14:33:57 <armax> right now I am targeting everything for the first release 14:34:20 <armax> and will let stuff spill over as events unfold 14:34:48 <armax> bottom line: I am thinking that once we reached the 30-ish mark. We’re most likely shut for business for the Mitaka timeframe 14:35:02 <armax> I can’t think how we can take much more work than that 14:35:23 <xgerman> red bull? 14:35:35 <anteaya> have you seen what it does to your heart? 14:35:54 <salv-orlando> 30-ish makes sense 14:36:26 <armax> the drivers team will still continue to review regularly 14:36:46 <armax> rfe’s and specs 14:37:43 <armax> but we need to have a reasonable expectation of how much we can chew in a single release 14:38:12 <armax> anyhow, I think I am probably stating the obvious 14:38:14 <armax> next up 14:38:18 <armax> #topic Bugs 14:38:21 <armax> ajo: you up 14:38:28 <armax> bugs keep piling up 14:38:36 <armax> :) 14:38:42 <ajo> hi, sorry, I ad an interruption 14:38:47 <ajo> I think garyk1 and I have a list 14:38:51 <garyk1> a lot of the bugs opened recently are lbaas specific 14:38:54 <armax> it looks like we had ~70 reports only during last week 14:39:18 <ajo> yeah, I found this one concerning: https://bugs.launchpad.net/neutron/+bug/1513765 shihanzhang is working on it and needs some review 14:39:18 <openstack> Launchpad bug 1513765 in neutron "bulk delete of ports cost iptables-firewall too much time" [High,In progress] - Assigned to shihanzhang (shihanzhang) 14:39:27 <garyk1> should those be in the neutron bugs or should we create a lbaas launchpad? just an idea? it will help focus more on the core bugs 14:39:39 <ajo> it's due conntrack manipulation. 14:39:55 <ajo> I'm concerned the fix is not backportable, and I'd love if we could have amotoki's eyes on the patch 14:39:56 <armax> garyk1: advs services and neutron are tracked by the same lp 14:40:05 <armax> garyk1: we use tags to differentiate 14:40:23 <garyk1> armax: i understand the tags, but i think that maybe we should consider splitting them out. 14:40:27 <xgerman> yep, and octavia has it’s own tag 14:40:30 <garyk1> that would give a truer picture 14:40:47 <garyk1> but sorry, back to the bugs 14:40:53 <wwriverrat> Regarding GoDaddy/Rax api extension for "IP usage" we plan to re-introduce into m-release: How would you like us to proceed? RFE->code? RFE->Spec->code? Re: https://review.openstack.org/#/c/180803/ 14:41:31 <wwriverrat> Patch exists here: https://review.openstack.org/#/c/212955/ 14:42:00 <ajo> this one needs lbaas eyes on it: https://bugs.launchpad.net/neutron/+bug/1163569 it's been hanging there since 2013, not urgent btw, just needs some re-triage. 14:42:01 <openstack> Launchpad bug 1163569 in neutron "security groups don't work with LBaaS vip and ovs plugin" [High,New] - Assigned to Doug Wiegley (dougwig) 14:42:31 <armax> wwriverrat: I think we have an rfe for that, possibly raised by someone else 14:43:01 <wwriverrat> https://bugs.launchpad.net/neutron/+bug/1457986 14:43:01 <openstack> Launchpad bug 1457986 in neutron "RFE: Need API to provide network IP allocation counts" [Wishlist,In progress] - Assigned to David Bingham (wwriverrat) 14:43:33 <armax> wwriverrat: let me look into that 14:43:41 <ajo> enikanorov : could you update about: https://bugs.launchpad.net/neutron/+bug/1453008 ? 14:43:41 <openstack> Launchpad bug 1453008 in neutron "Deadlock on update_port_status" [High,In progress] - Assigned to Eugene Nikanorov (enikanorov) 14:44:20 <wwriverrat> armax: My question is around how to proceed. Will do as little or as much as you see fit. 14:44:55 <armax> wwriverrat: at one point I saw to competing proposals 14:45:13 <armax> wwriverrat: I don’t have a clear picture right now…I need to go back and look before I can advise you further 14:45:33 <armax> going back to bugs... 14:45:36 <wwriverrat> thanks. I’ll try to pull up that other proposal 14:45:47 <armax> ajo: anything else worth raising? 14:45:56 <armax> any gate failure worth noting? 14:46:01 <armax> we have a massive stroke last week 14:46:08 <ajo> https://bugs.launchpad.net/neutron/+bug/1513782 14:46:08 <openstack> Launchpad bug 1513782 in neutron "API response time degradation" [High,New] 14:46:10 <armax> as someone might recall 14:46:12 <ajo> this one is important I think 14:46:40 <ajo> we had some important performance degradation 14:47:07 <armax> kevinbenton seems to have volunteered on 1513782 14:47:20 <armax> I wonder if that falls on the larger performance effort that regXboi is leading? 14:47:29 <regXboi> armax: that is on the list *eventually* 14:47:32 <salv-orl_> it must be because he caused it with some clumsy db op he added ;) 14:47:37 <ajo> armax , I think you guys handled gate failures faster than my wake up time, everytime I was up things were handled 14:47:53 * regXboi just tagged 1513782 as loadimpact 14:48:48 <armax> I already did ;) 14:48:54 <armax> ajo: anything else? 14:49:02 <armax> the torch is with garyk1 this week 14:49:07 <armax> any volunteer for next week? 14:49:23 <rossella_s> armax, me! 14:49:23 <ajo> I think garyk1 spotted a couple of bugs, garyk1 , did I name all? 14:49:31 <armax> rossella_s: super 14:49:34 <wwriverrat> armax: Ref “competing proposal”: https://review.openstack.org/#/c/234541/ targeting our RFE: https://bugs.launchpad.net/neutron/+bug/1457986 14:49:34 <openstack> Launchpad bug 1457986 in neutron "RFE: Need API to provide network IP allocation counts" [Wishlist,In progress] - Assigned to David Bingham (wwriverrat) 14:49:50 <armax> wwriverrat: can we take this offline, please? 14:49:51 <garyk1> armax: ack. 14:50:08 <armax> wwriverrat: it feels like you’re simply hijacking the meeting 14:50:38 <ajo> armax : I think we're done with the important ones I'm aware of 14:50:45 <armax> ajo: ok, cool 14:50:53 <regXboi> wow - where does the time go 14:51:14 <armax> regXboi: indeed 14:51:15 <ajo> sorry: https://bugs.launchpad.net/neutron/+bug/1514810 (noted by garyk1 ) 14:51:15 <openstack> Launchpad bug 1514810 in neutron "Turning on 'enable_dhcp' on subnet update cause request failure for pluggable IPAM" [Medium,New] - Assigned to Pavel Bondar (pasha117) 14:51:20 <ajo> but I think carl_baldwin is already on top of it 14:51:45 <armax> ajo: right, the assigned ones are not the ones that worry me 14:51:55 <armax> :) 14:51:58 <armax> anyhow 14:52:01 <ajo> correct 14:52:24 <armax> emagana is not here so le’ts skip the docs section and go straight to the open discussion one 14:52:40 <armax> #topic Open Discussion 14:52:52 <armax> there are two topics of which one seems already handled 14:53:18 <armax> https://bugs.launchpad.net/neutron/+bug/1507776 14:53:18 <openstack> Launchpad bug 1507776 in neutron "Wrong OVS flows created for new networks" [High,Fix committed] - Assigned to YAMAMOTO Takashi (yamamoto) 14:53:26 <armax> claudiub: ping? 14:53:33 <claudiub> pong 14:53:45 <armax> you raised this for the neutron meeting? 14:53:53 <armax> what’s there to be discussed? 14:53:55 <claudiub> that was weeks ago. 14:54:03 <claudiub> before the summit, i think. 14:54:08 <armax> oh, ack 14:54:12 <armax> it’s left over then? 14:54:31 <claudiub> sorry, I did not get that 14:54:36 <armax> claudiub: never mind 14:54:41 <armax> claudiub: thanks 14:54:47 <claudiub> np. :) 14:54:55 <amotoki> it seems we need to change the title to "On Demand Agenda for <date>". 14:54:59 <ajo> yamamoto++ claudiub++ 14:55:08 <xgerman> amotoki +1 14:55:12 <armax> amotoki: I’ll fix that 14:55:50 <armax> btw the other, I am sure more current one, was the request to consider the flavors service plugin be enabled by default 14:56:03 <armax> so that we can test it in the gate on a continuous basis 14:56:23 <ajo> armax , I had a talk with the nova PTL about port flavors, he asked me to sync with you on this. 14:56:49 <ajo> he found it interesting in regards making nova-scheduler aware of neutron stuff 14:57:17 <armax> ajo: I talked to him about last week 14:57:25 <ajo> its a long TL;DR to topic, so I'll probably translate it into an RFE, but he asked me to sync with you 14:57:45 <ajo> " long TL;DR to topic" .... wow ':D 14:58:16 <ajo> I guess we can sync offline 14:58:23 <xgerman> ok, so can we reach a consensus if flavors should be enabled by default or not? 14:58:34 <armax> ajo: let’s take this offline, but the idea is worth exploring especially if that helps addressing a large set of use cases that involve unstructured port bindings 14:58:34 <ajo> (sorry for deviating the topic) 14:58:38 <armax> xgerman: yes 14:59:13 <armax> anyone with an opinion? 14:59:16 <armax> we have 1 min left? 14:59:30 <xgerman> #startvote? 14:59:31 <openstack> Only the meeting chair may start a vote. 14:59:54 <armax> we should take that to the ML, I guess 15:00:04 <armax> I’ll end the meeting now and give you a few secs back 15:00:06 <armax> #endmeeting