14:00:14 #startmeeting networking 14:00:15 Meeting started Tue Nov 17 14:00:14 2015 UTC and is due to finish in 60 minutes. The chair is armax. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:18 The meeting name has been set to 'networking' 14:00:21 Hi 14:00:26 Hi 14:00:31 hi 14:00:32 Hi 14:00:35 o/ 14:00:41 hi 14:00:47 hi 14:00:56 \o/ 14:01:11 hi 14:01:18 hi folks 14:01:36 Good morning armax! 14:01:53 o/ :) 14:01:55 mestery: is that a wake up call? 14:02:01 #topic Announcements 14:02:02 WAKE UP 14:02:05 garyk1: I think armax needs one. 14:02:08 ^^^^^^^ that is a wake up call 14:02:09 I do 14:02:13 * mestery hands armax espresso 14:02:15 good morning :-) 14:02:29 he should be triaging bugs and not sleeping 14:02:32 aloha 14:02:45 * xgerman wonders who cam up with this 6 am time slot 14:02:51 #link https://wiki.openstack.org/wiki/Network/Meetings 14:03:07 xgerman: blame daylight savings time 14:03:09 #link https://wiki.openstack.org/wiki/Network/Meetings#Announcements_.2F_Reminders 14:03:19 Hey 14:03:23 a couple of things worth sharing 14:03:25 xgerman, it's 16:00 here in Europe.. this meeting is for us as well as you American guys ;-) 14:03:27 gcc 14:03:29 xgerman: you can always read the logs and send a mail to the list if there are issues that trouble you 14:03:32 meh 14:03:41 salv-orlando: gcc: not found 14:04:13 lol regXboi salv-orlando 14:04:15 * armax gives people time to go through the reminders while he snoozes a bit 14:04:37 hi! sorry being late 14:04:47 hi 14:04:51 just announcement of sharing neutron meetup photos in Tokyo https://goo.gl/photos/HHTwgCSXoPuG5iqt5 14:04:59 thanks for the reminders armax 14:05:21 amotoki: Nice! :) 14:05:21 it’s worth noting the creation of the assert tags 14:05:31 mestery: can tell us more, since he’s the TC dude 14:05:46 I can’t see markmclain 14:05:51 markmcclain: ping? 14:05:53 armax: Which tags did you have qusetions about? 14:06:09 armax: For the upgrade tags, we don't qualify for rolling-upgrade yet 14:06:13 But supports upgrade we can define 14:06:16 mestery: right 14:06:19 BEcause we clearly support upgrades 14:06:19 amotoki : oh guys you had fun :') 14:06:25 I'll propose a patch for that today 14:06:28 mestery: and I did: https://review.openstack.org/#/c/246242/ 14:06:34 armax: You beat me to it 14:06:38 mestery: of course 14:06:40 :) 14:06:46 armax: I hadn't reviewed governance patches yet today 14:06:49 that said, the rolling upgrade is technically not htat far off 14:07:01 right 14:07:16 my wishful thinking is that garyk and sc68cal_ are gonna help with the partial grenade upgrade 14:07:42 * regXboi wonders if it's time to rename grenade 14:07:57 armax: the coming week i am going to be on the bugs 14:07:58 there has been a thread going on lately 14:08:00 #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/079344.html 14:08:24 garyk1: multitasking? :) 14:08:47 :) 14:09:10 garyk1: I can keep helping a bit in bg :), we need a deputy volunteer for the week after 14:09:20 the patch introducing the multinode grenade for neutron: https://review.openstack.org/#/c/245862 14:09:22 anyhow, it sounds like we could easily grasp the rolling-upgrade tag if we pull ourselves together and manage to have CI for the grenade partial job 14:09:36 i will be in touch with sc68cal_ and go from there 14:09:58 korzen: thanks for the link! 14:10:24 unfortunately, multinode jobs still look ill 14:10:51 regXboi: true, but that shouldn’t stop us from making progress 14:10:53 regXboi: is 'ill' a hip word for 'cool'? 14:11:21 As in, 'bro, check out this ill job' 14:11:23 no, in this case "ill" == "still appear to be at 100% error rate" 14:11:26 as for other reminders, there have been a couple of suggested improvements to the way we handle blueprints 14:11:30 and rfe's 14:11:44 full process detailed on: 14:11:45 #link http://docs.openstack.org/developer/neutron/policies/blueprints.html 14:12:27 regXboi: do we have a culprit? 14:12:42 armax: there was yesterday - I'm looking to see if it still exists 14:12:49 Who tries to fix multinode job issue? 14:13:43 ok, I *think* that the code to fix merged in the last hour - I'll keep an eye on it 14:14:01 regXboi: link? 14:14:16 regXboi: it looks like both multi-jobs broke 14:14:33 armax: all the multinode jobs broke - getting the links from eavesdrop now 14:14:44 regXboi: ok thanks 14:15:06 armax: yeah, both for the same reason: live migration test 14:15:07 regXboi: http://logs.openstack.org/18/245518/5/check/gate-tempest-dsvm-neutron-multinode-full/5a3c901/logs/testr_results.html.gz (live migration test) 14:15:50 obondarev: looks like that’s a tempest issue 14:15:53 why do we need livemigration for neutron jobs? 14:16:15 garyk1: it's a part of full tempest suite I guess 14:16:17 garyk1: why do we need to test neutron at all? 14:16:19 it is something that is usually broken at the best of times in nova 14:16:47 garyk1: even in the nova net case you mean? 14:16:53 armax: why the cynicism? 14:17:10 recently I found a live migration issue in nova 14:17:12 garyk1: I was making a joke 14:17:13 armax: yes, there is even a working group setup to address live migrations 14:17:21 ok patch that broke multinode: https://review.openstack.org/245996 14:17:25 they don't wait on port notifications from neutron, before bringing up the VM at destination 14:17:30 that only shows up in mutlinode 14:17:32 patch that fixed multinode: https://review.openstack.org/#/c/233711/ 14:17:54 VM is resumed before network is wired, RARP messages get blackholed, so the nodes don't know where the MAC address belongs to 14:17:57 regXboi: how can 245996 break multi node? 14:18:06 regXboi: you sure it’s the right link 14:18:15 ugh 14:18:17 no it isn't 14:18:23 ajo: any reason why it hadn't been fixed? 14:18:24 233711 broke multinode 14:18:34 ajo: related https://bugs.launchpad.net/neutron/+bug/1414559 ? 14:18:34 Launchpad bug 1414559 in neutron "OVS drops RARP packets by QEMU upon live-migration - VM temporarily disconnected" [Undecided,Invalid] - Assigned to sean mooney (sean-k-mooney) 14:18:37 https://review.openstack.org/#/c/245697/ supposedly fixed it 14:18:41 yeah 14:18:43 obondarev : exactly 14:18:45 ajo: I'm currently working on https://bugs.launchpad.net/neutron/+bug/1414559 14:18:46 thanks, it's that one 14:19:04 ajo: have patches for nova and neutron, testing 14:19:05 obondarev , but as I saw, it was more a nova thing, because they start the VM before network is wired 14:19:09 amuller and I discussed this yesterday in channel: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-11-16.log.html#t2015-11-16T21:42:10 14:19:09 they should use notification 14:19:09 ack 14:19:18 that's likely to be the culprit of live migration failures 14:19:24 ok, it looks like we have a handle on this then 14:19:26 I investigated that for 1 week.. 14:19:31 but the failure I observed 14:19:34 that garyk1 shared 14:19:44 was related to a setup issue 14:19:49 nothing to do with the dataplane 14:19:50 ajo: neutron should send notification in the first, we can take it offline to not wast meeting time 14:20:00 obondarev: ack 14:20:28 ajo, obondarev: even though the change you’re referring to may be addressing some other instability 14:20:43 ajo: I seem to recall that upon certain events nova does not wait for a notification from neutron before restaring the VM 14:20:57 salv-orlando: how rude 14:20:59 armax : ack 14:21:03 the logic is buried deep in the compute manager so it's not easy to grasp 14:21:17 but this was over a year ago, maybe in the meanwhile things have changed 14:21:19 yes, this was a race condition by nova not honoring the notification, does not happen 100% of the time. 14:21:49 ajo: which might be consistent with nova ignoring the notification and therefore things being racey 14:21:57 ok, so someone maybe be interested in going to the nova mid-cycle meetup and yell at the nova people? the event is scheduled for Jan 26 2016 14:22:00 details here: 14:22:02 #link https://www.eventbrite.com.au/e/openstack-mitaka-nova-mid-cycle-meetup-tickets-19326224257 14:22:14 I can't get :'( 14:22:19 Go* 14:22:22 It's in London, right armax? 14:22:27 mestery: bristol 14:22:31 that was the last of the reminders 14:22:32 I think rdopiera from redhat was looking at it, 14:22:38 let’s move on to the next section 14:22:38 armax: I think some discussion in IRC and a bug report will lead to a patch and a fix 14:22:42 but I could eventually try to speedup it, or help 14:22:46 no need to put yourself on a plane ;) 14:23:03 specially if it's hurting our jobs, but it seems obondarev had already a good look on the issue 14:23:09 salv-orlando: I know, I found that lack of sleeps makes me particular prone to making jokes 14:23:26 #topic Blueprints 14:23:43 #link https://launchpad.net/neutron/+milestone/mitaka-1 14:23:46 armax: this is because I have poured scotch in your nespresso capsules 14:24:07 sorry to barge in but this may be what happens with the live migrate - https://github.com/openstack/nova/commit/150406679c14778f0479bc6d4e77df5119c0bdae 14:24:08 salv-orlando: all makes sense now! 14:24:44 obondarev: just filed https://blueprints.launchpad.net/neutron/+spec/improve-dvr-l3-agent-binding 14:25:08 armax: right, it's for already approved spec 14:25:26 armax: bp was missing I guess 14:25:27 obondarev: technically something was something dangling from past days 14:25:35 obondarev: we need an approver 14:26:04 armax: I think carl_baldwin is the right person 14:26:05 o/ 14:26:08 I can’t be one, I am already approver on a bunch of other bp’s and I won’t be able to help you with reviews and stuf f as much 14:26:23 armax: k, changing to carl_baldwin 14:26:47 carl_baldwin: do you have fire power? 14:27:01 Yes, I can take this 14:27:07 carl_baldwin: thanks 14:27:29 armax: The multipel subnets to connect to vpn commits are all in (though it says needs code-review) 14:27:31 this one https://review.openstack.org/#/c/196959/ was missed 14:28:55 pc_m: I should update the status of yours. 14:29:04 shihanzhang: any reason why the norifictaion do not suffice? 14:29:28 anyone wants to raise flags over the progress of any of the blueprints targeted for M1? 14:30:00 armax: I'll help part-time on splitting neutron into base library 14:30:18 (already talking to dougwig 14:30:20 pc_m: thanks 14:30:28 armax: i would like to know from other what they think of the timestamp bp? 14:30:30 pc_m: Any progress there? 14:30:40 Because that one doesn't look like it's making a ton of progress according to LP at least 14:30:49 pc_m: Regarding https://blueprints.launchpad.net/neutron/+spec/neutron-lib 14:30:50 Doug and I have some protoypes, but need to get test working. 14:30:54 ACk 14:31:20 garyk1: faik, that must be resubmitted for mitaka, but that aside, there’s a change from kevinbenton that will help add this type of info to neutron resources 14:31:32 and it’s a requirement to timestamps, tags and the likes 14:31:34 armax: ok, thanks 14:31:52 that makes more sense that it be something generic 14:31:55 fyo - https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp 14:32:02 that said, we currently have 21 blueprints targeting ‘neutron’ for M1 14:32:04 kevinbenton: can you please send a link to the work if possible 14:32:57 armax: requesting for direction from this group: Was looking for your guidance on how to proceed with "Add API extension for reporting IP address usage statistics": https://review.openstack.org/#/c/212955/ 14:32:58 garyk1: https://review.openstack.org/#/c/222079/ 14:33:02 armax is flavor among them? 14:33:13 xgerman: yes 14:33:14 garyk1, this is the link https://review.openstack.org/222079 14:33:39 we had 28 blueprints targeted for the whole of Liberty 14:33:57 right now I am targeting everything for the first release 14:34:20 and will let stuff spill over as events unfold 14:34:48 bottom line: I am thinking that once we reached the 30-ish mark. We’re most likely shut for business for the Mitaka timeframe 14:35:02 I can’t think how we can take much more work than that 14:35:23 red bull? 14:35:35 have you seen what it does to your heart? 14:35:54 30-ish makes sense 14:36:26 the drivers team will still continue to review regularly 14:36:46 rfe’s and specs 14:37:43 but we need to have a reasonable expectation of how much we can chew in a single release 14:38:12 anyhow, I think I am probably stating the obvious 14:38:14 next up 14:38:18 #topic Bugs 14:38:21 ajo: you up 14:38:28 bugs keep piling up 14:38:36 :) 14:38:42 hi, sorry, I ad an interruption 14:38:47 I think garyk1 and I have a list 14:38:51 a lot of the bugs opened recently are lbaas specific 14:38:54 it looks like we had ~70 reports only during last week 14:39:18 yeah, I found this one concerning: https://bugs.launchpad.net/neutron/+bug/1513765 shihanzhang is working on it and needs some review 14:39:18 Launchpad bug 1513765 in neutron "bulk delete of ports cost iptables-firewall too much time" [High,In progress] - Assigned to shihanzhang (shihanzhang) 14:39:27 should those be in the neutron bugs or should we create a lbaas launchpad? just an idea? it will help focus more on the core bugs 14:39:39 it's due conntrack manipulation. 14:39:55 I'm concerned the fix is not backportable, and I'd love if we could have amotoki's eyes on the patch 14:39:56 garyk1: advs services and neutron are tracked by the same lp 14:40:05 garyk1: we use tags to differentiate 14:40:23 armax: i understand the tags, but i think that maybe we should consider splitting them out. 14:40:27 yep, and octavia has it’s own tag 14:40:30 that would give a truer picture 14:40:47 but sorry, back to the bugs 14:40:53 Regarding GoDaddy/Rax api extension for "IP usage" we plan to re-introduce into m-release: How would you like us to proceed? RFE->code? RFE->Spec->code? Re: https://review.openstack.org/#/c/180803/ 14:41:31 Patch exists here: https://review.openstack.org/#/c/212955/ 14:42:00 this one needs lbaas eyes on it: https://bugs.launchpad.net/neutron/+bug/1163569 it's been hanging there since 2013, not urgent btw, just needs some re-triage. 14:42:01 Launchpad bug 1163569 in neutron "security groups don't work with LBaaS vip and ovs plugin" [High,New] - Assigned to Doug Wiegley (dougwig) 14:42:31 wwriverrat: I think we have an rfe for that, possibly raised by someone else 14:43:01 https://bugs.launchpad.net/neutron/+bug/1457986 14:43:01 Launchpad bug 1457986 in neutron "RFE: Need API to provide network IP allocation counts" [Wishlist,In progress] - Assigned to David Bingham (wwriverrat) 14:43:33 wwriverrat: let me look into that 14:43:41 enikanorov : could you update about: https://bugs.launchpad.net/neutron/+bug/1453008 ? 14:43:41 Launchpad bug 1453008 in neutron "Deadlock on update_port_status" [High,In progress] - Assigned to Eugene Nikanorov (enikanorov) 14:44:20 armax: My question is around how to proceed. Will do as little or as much as you see fit. 14:44:55 wwriverrat: at one point I saw to competing proposals 14:45:13 wwriverrat: I don’t have a clear picture right now…I need to go back and look before I can advise you further 14:45:33 going back to bugs... 14:45:36 thanks. I’ll try to pull up that other proposal 14:45:47 ajo: anything else worth raising? 14:45:56 any gate failure worth noting? 14:46:01 we have a massive stroke last week 14:46:08 https://bugs.launchpad.net/neutron/+bug/1513782 14:46:08 Launchpad bug 1513782 in neutron "API response time degradation" [High,New] 14:46:10 as someone might recall 14:46:12 this one is important I think 14:46:40 we had some important performance degradation 14:47:07 kevinbenton seems to have volunteered on 1513782 14:47:20 I wonder if that falls on the larger performance effort that regXboi is leading? 14:47:29 armax: that is on the list *eventually* 14:47:32 it must be because he caused it with some clumsy db op he added ;) 14:47:37 armax , I think you guys handled gate failures faster than my wake up time, everytime I was up things were handled 14:47:53 * regXboi just tagged 1513782 as loadimpact 14:48:48 I already did ;) 14:48:54 ajo: anything else? 14:49:02 the torch is with garyk1 this week 14:49:07 any volunteer for next week? 14:49:23 armax, me! 14:49:23 I think garyk1 spotted a couple of bugs, garyk1 , did I name all? 14:49:31 rossella_s: super 14:49:34 armax: Ref “competing proposal”: https://review.openstack.org/#/c/234541/ targeting our RFE: https://bugs.launchpad.net/neutron/+bug/1457986 14:49:34 Launchpad bug 1457986 in neutron "RFE: Need API to provide network IP allocation counts" [Wishlist,In progress] - Assigned to David Bingham (wwriverrat) 14:49:50 wwriverrat: can we take this offline, please? 14:49:51 armax: ack. 14:50:08 wwriverrat: it feels like you’re simply hijacking the meeting 14:50:38 armax : I think we're done with the important ones I'm aware of 14:50:45 ajo: ok, cool 14:50:53 wow - where does the time go 14:51:14 regXboi: indeed 14:51:15 sorry: https://bugs.launchpad.net/neutron/+bug/1514810 (noted by garyk1 ) 14:51:15 Launchpad bug 1514810 in neutron "Turning on 'enable_dhcp' on subnet update cause request failure for pluggable IPAM" [Medium,New] - Assigned to Pavel Bondar (pasha117) 14:51:20 but I think carl_baldwin is already on top of it 14:51:45 ajo: right, the assigned ones are not the ones that worry me 14:51:55 :) 14:51:58 anyhow 14:52:01 correct 14:52:24 emagana is not here so le’ts skip the docs section and go straight to the open discussion one 14:52:40 #topic Open Discussion 14:52:52 there are two topics of which one seems already handled 14:53:18 https://bugs.launchpad.net/neutron/+bug/1507776 14:53:18 Launchpad bug 1507776 in neutron "Wrong OVS flows created for new networks" [High,Fix committed] - Assigned to YAMAMOTO Takashi (yamamoto) 14:53:26 claudiub: ping? 14:53:33 pong 14:53:45 you raised this for the neutron meeting? 14:53:53 what’s there to be discussed? 14:53:55 that was weeks ago. 14:54:03 before the summit, i think. 14:54:08 oh, ack 14:54:12 it’s left over then? 14:54:31 sorry, I did not get that 14:54:36 claudiub: never mind 14:54:41 claudiub: thanks 14:54:47 np. :) 14:54:55 it seems we need to change the title to "On Demand Agenda for ". 14:54:59 yamamoto++ claudiub++ 14:55:08 amotoki +1 14:55:12 amotoki: I’ll fix that 14:55:50 btw the other, I am sure more current one, was the request to consider the flavors service plugin be enabled by default 14:56:03 so that we can test it in the gate on a continuous basis 14:56:23 armax , I had a talk with the nova PTL about port flavors, he asked me to sync with you on this. 14:56:49 he found it interesting in regards making nova-scheduler aware of neutron stuff 14:57:17 ajo: I talked to him about last week 14:57:25 its a long TL;DR to topic, so I'll probably translate it into an RFE, but he asked me to sync with you 14:57:45 " long TL;DR to topic" .... wow ':D 14:58:16 I guess we can sync offline 14:58:23 ok, so can we reach a consensus if flavors should be enabled by default or not? 14:58:34 ajo: let’s take this offline, but the idea is worth exploring especially if that helps addressing a large set of use cases that involve unstructured port bindings 14:58:34 (sorry for deviating the topic) 14:58:38 xgerman: yes 14:59:13 anyone with an opinion? 14:59:16 we have 1 min left? 14:59:30 #startvote? 14:59:31 Only the meeting chair may start a vote. 14:59:54 we should take that to the ML, I guess 15:00:04 I’ll end the meeting now and give you a few secs back 15:00:06 #endmeeting