14:00:14 <armax> #startmeeting networking
14:00:15 <openstack> Meeting started Tue Nov 17 14:00:14 2015 UTC and is due to finish in 60 minutes.  The chair is armax. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:18 <openstack> The meeting name has been set to 'networking'
14:00:21 <carl_baldwin> Hi
14:00:26 <hoangcx> Hi
14:00:31 <pc_m> hi
14:00:32 <kevinbenton> Hi
14:00:35 <obondarev_> o/
14:00:41 <SridharG> hi
14:00:47 <wwriverrat> hi
14:00:56 <jschwarz> \o/
14:01:11 <rossella_s> hi
14:01:18 <armax> hi folks
14:01:36 <mestery> Good morning armax!
14:01:53 <ajo> o/ :)
14:01:55 <garyk1> mestery: is that a wake up call?
14:02:01 <armax> #topic Announcements
14:02:02 <regXboi> WAKE UP
14:02:05 <mestery> garyk1: I think armax needs one.
14:02:08 <regXboi> ^^^^^^^ that is a wake up call
14:02:09 <armax> I do
14:02:13 * mestery hands armax espresso
14:02:15 <ajo> good morning :-)
14:02:29 <garyk1> he should be triaging bugs and not sleeping
14:02:32 <salv-orlando> aloha
14:02:45 * xgerman wonders who cam up with this 6 am time slot
14:02:51 <armax> #link https://wiki.openstack.org/wiki/Network/Meetings
14:03:07 <regXboi> xgerman: blame daylight savings time
14:03:09 <armax> #link https://wiki.openstack.org/wiki/Network/Meetings#Announcements_.2F_Reminders
14:03:19 <mhickey> Hey
14:03:23 <armax> a couple of things worth sharing
14:03:25 <jschwarz> xgerman, it's 16:00 here in Europe.. this meeting is for us as well as you American guys ;-)
14:03:27 <salv-orlando> gcc
14:03:29 <garyk1> xgerman: you can always read the logs and send a mail to the list if there are issues that trouble you
14:03:32 <salv-orlando> meh
14:03:41 <regXboi> salv-orlando: gcc: not found
14:04:13 <ajo> lol regXboi salv-orlando
14:04:15 * armax gives people time to go through the reminders while he snoozes a bit
14:04:37 <akamyshnikova> hi! sorry being late
14:04:47 <hichihara> hi
14:04:51 <amotoki> just announcement of sharing neutron meetup photos in Tokyo https://goo.gl/photos/HHTwgCSXoPuG5iqt5
14:04:59 <ajo> thanks for the reminders armax
14:05:21 <hichihara> amotoki: Nice! :)
14:05:21 <armax> it’s worth noting the creation of the assert tags
14:05:31 <armax> mestery: can tell us more, since he’s the TC dude
14:05:46 <armax> I can’t see markmclain
14:05:51 <armax> markmcclain: ping?
14:05:53 <mestery> armax: Which tags did you have qusetions about?
14:06:09 <mestery> armax: For the upgrade tags, we don't qualify for rolling-upgrade yet
14:06:13 <mestery> But supports upgrade we can define
14:06:16 <armax> mestery: right
14:06:19 <mestery> BEcause we clearly support upgrades
14:06:19 <ajo> amotoki : oh guys you had fun :')
14:06:25 <mestery> I'll propose a patch for that today
14:06:28 <armax> mestery: and I did: https://review.openstack.org/#/c/246242/
14:06:34 <mestery> armax: You beat me to it
14:06:38 <armax> mestery: of course
14:06:40 <armax> :)
14:06:46 <mestery> armax: I hadn't reviewed governance patches yet today
14:06:49 <armax> that said, the rolling upgrade is technically not htat far off
14:07:01 <mestery> right
14:07:16 <armax> my wishful thinking is that garyk and sc68cal_ are gonna help with the partial grenade upgrade
14:07:42 * regXboi wonders if it's time to rename grenade
14:07:57 <garyk1> armax: the coming week i am going to be on the bugs
14:07:58 <armax> there has been a thread going on lately
14:08:00 <armax> #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/079344.html
14:08:24 <armax> garyk1: multitasking? :)
14:08:47 <garyk1> :)
14:09:10 <ajo> garyk1: I can keep helping a bit in bg :), we need a deputy volunteer for the week after
14:09:20 <korzen> the patch introducing the multinode grenade for neutron: https://review.openstack.org/#/c/245862
14:09:22 <armax> anyhow, it sounds like we could easily grasp the rolling-upgrade tag if we pull ourselves together and manage to have CI for the grenade partial job
14:09:36 <garyk1> i will be in touch with sc68cal_ and go from there
14:09:58 <armax> korzen: thanks for the link!
14:10:24 <regXboi> unfortunately, multinode jobs still look ill
14:10:51 <armax> regXboi: true, but that shouldn’t stop us from making progress
14:10:53 <kevinbenton> regXboi: is 'ill' a hip word for 'cool'?
14:11:21 <kevinbenton> As in, 'bro, check out this ill job'
14:11:23 <regXboi> no, in this case "ill" == "still appear to be at 100% error rate"
14:11:26 <armax> as for other reminders, there have been a couple of suggested improvements to the way we handle blueprints
14:11:30 <armax> and rfe's
14:11:44 <armax> full process detailed on:
14:11:45 <armax> #link http://docs.openstack.org/developer/neutron/policies/blueprints.html
14:12:27 <armax> regXboi: do we have a culprit?
14:12:42 <regXboi> armax: there was yesterday - I'm looking to see if it still exists
14:12:49 <hichihara> Who tries to fix multinode job issue?
14:13:43 <regXboi> ok, I *think* that the code to fix merged in the last hour - I'll keep an eye on it
14:14:01 <armax> regXboi: link?
14:14:16 <armax> regXboi: it looks like both multi-jobs broke
14:14:33 <regXboi> armax: all the multinode jobs broke - getting the links from eavesdrop now
14:14:44 <armax> regXboi: ok thanks
14:15:06 <obondarev> armax: yeah, both for the same reason: live migration test
14:15:07 <garyk1> regXboi: http://logs.openstack.org/18/245518/5/check/gate-tempest-dsvm-neutron-multinode-full/5a3c901/logs/testr_results.html.gz (live migration test)
14:15:50 <armax> obondarev: looks like that’s a tempest issue
14:15:53 <garyk1> why do we need livemigration for neutron jobs?
14:16:15 <obondarev> garyk1: it's a part of full tempest suite I guess
14:16:17 <armax> garyk1: why do we need to test neutron at all?
14:16:19 <garyk1> it is something that is usually broken at the best of times in nova
14:16:47 <armax> garyk1: even in the nova net case you mean?
14:16:53 <garyk1> armax: why the cynicism?
14:17:10 <ajo> recently I found a live migration issue in nova
14:17:12 <armax> garyk1: I was making a joke
14:17:13 <garyk1> armax: yes, there is even a working group setup to address live migrations
14:17:21 <regXboi> ok patch that broke multinode: https://review.openstack.org/245996
14:17:25 <ajo> they don't wait on port notifications from neutron, before bringing up the VM at destination
14:17:30 <ajo> that only shows up in mutlinode
14:17:32 <regXboi> patch that fixed multinode: https://review.openstack.org/#/c/233711/
14:17:54 <ajo> VM is resumed before network is wired, RARP messages get blackholed, so the nodes don't know where the MAC address belongs to
14:17:57 <armax> regXboi: how can 245996 break multi node?
14:18:06 <armax> regXboi: you sure it’s the right link
14:18:15 <regXboi> ugh
14:18:17 <regXboi> no it isn't
14:18:23 <kevinbenton> ajo: any reason why it hadn't been fixed?
14:18:24 <regXboi> 233711 broke multinode
14:18:34 <obondarev> ajo: related https://bugs.launchpad.net/neutron/+bug/1414559 ?
14:18:34 <openstack> Launchpad bug 1414559 in neutron "OVS drops RARP packets by QEMU upon live-migration - VM temporarily disconnected" [Undecided,Invalid] - Assigned to sean mooney (sean-k-mooney)
14:18:37 <regXboi> https://review.openstack.org/#/c/245697/ supposedly fixed it
14:18:41 <ajo> yeah
14:18:43 <ajo> obondarev : exactly
14:18:45 <obondarev> ajo: I'm currently working on https://bugs.launchpad.net/neutron/+bug/1414559
14:18:46 <ajo> thanks, it's that one
14:19:04 <obondarev> ajo: have patches for nova and neutron, testing
14:19:05 <ajo> obondarev , but as I saw, it was more a nova thing, because they start the VM before network is wired
14:19:09 <regXboi> amuller and I discussed this yesterday in channel: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-11-16.log.html#t2015-11-16T21:42:10
14:19:09 <ajo> they should use notification
14:19:09 <ajo> ack
14:19:18 <ajo> that's likely to be the culprit of live migration failures
14:19:24 <armax> ok, it looks like we have a handle on this then
14:19:26 <ajo> I investigated that for 1 week..
14:19:31 <armax> but the failure I observed
14:19:34 <armax> that garyk1 shared
14:19:44 <armax> was related to a setup issue
14:19:49 <armax> nothing to do with the dataplane
14:19:50 <obondarev> ajo: neutron should send notification in the first, we can take it offline to not wast meeting time
14:20:00 <ajo> obondarev: ack
14:20:28 <armax> ajo, obondarev: even though the change you’re referring to may be addressing some other instability
14:20:43 <salv-orlando> ajo: I seem to recall that upon certain events nova does not wait for a notification from neutron before restaring the VM
14:20:57 <armax> salv-orlando: how rude
14:20:59 <ajo> armax : ack
14:21:03 <salv-orlando> the logic is buried deep in the compute manager so it's not easy to grasp
14:21:17 <salv-orlando> but this was over a year ago, maybe in the meanwhile things have changed
14:21:19 <ajo> yes, this was a race condition by nova not honoring the notification, does not happen 100% of the time.
14:21:49 <salv-orlando> ajo: which might be consistent with nova ignoring the notification and therefore things being racey
14:21:57 <armax> ok, so someone maybe be interested in going to the nova mid-cycle meetup and yell at the nova people? the event is scheduled for  Jan 26 2016
14:22:00 <armax> details here:
14:22:02 <armax> #link https://www.eventbrite.com.au/e/openstack-mitaka-nova-mid-cycle-meetup-tickets-19326224257
14:22:14 <kevinbenton> I can't get  :'(
14:22:19 <kevinbenton> Go*
14:22:22 <mestery> It's in London, right armax?
14:22:27 <armax> mestery: bristol
14:22:31 <armax> that was the last of the reminders
14:22:32 <ajo> I think rdopiera from redhat was looking at it,
14:22:38 <armax> let’s move on to the next section
14:22:38 <salv-orlando> armax: I think some discussion in IRC and a bug report will lead to a patch and a fix
14:22:42 <ajo> but I could eventually try to speedup it, or help
14:22:46 <salv-orlando> no need to put yourself on a plane ;)
14:23:03 <ajo> specially if it's hurting our jobs, but it seems obondarev had already a good look on the issue
14:23:09 <armax> salv-orlando: I know, I found that lack of sleeps makes me particular prone to making jokes
14:23:26 <armax> #topic Blueprints
14:23:43 <armax> #link https://launchpad.net/neutron/+milestone/mitaka-1
14:23:46 <salv-orlando> armax: this is because I have poured scotch in your nespresso capsules
14:24:07 <garyk1> sorry to barge in but this may be what happens with the live migrate - https://github.com/openstack/nova/commit/150406679c14778f0479bc6d4e77df5119c0bdae
14:24:08 <armax> salv-orlando: all makes sense now!
14:24:44 <armax> obondarev: just filed https://blueprints.launchpad.net/neutron/+spec/improve-dvr-l3-agent-binding
14:25:08 <obondarev> armax: right, it's for already approved spec
14:25:26 <obondarev> armax: bp was missing I guess
14:25:27 <armax> obondarev: technically something was something dangling from past days
14:25:35 <armax> obondarev: we need an approver
14:26:04 <obondarev> armax: I think carl_baldwin is the right person
14:26:05 <carl_baldwin> o/
14:26:08 <armax> I can’t be one, I am already approver on a bunch of other bp’s and I won’t be able to help you with reviews and stuf f as much
14:26:23 <obondarev> armax: k, changing to carl_baldwin
14:26:47 <armax> carl_baldwin: do you have fire power?
14:27:01 <carl_baldwin> Yes, I can take this
14:27:07 <obondarev> carl_baldwin: thanks
14:27:29 <pc_m> armax: The multipel subnets to connect to vpn commits are all in (though it says needs code-review)
14:27:31 <shihanzhang> this one https://review.openstack.org/#/c/196959/ was missed
14:28:55 <amotoki> pc_m: I should update the status of yours.
14:29:04 <garyk1> shihanzhang: any reason why the norifictaion do not suffice?
14:29:28 <armax> anyone wants to raise flags over the progress of any of the blueprints targeted for M1?
14:30:00 <pc_m> armax: I'll help part-time on splitting neutron into base library
14:30:18 <pc_m> (already talking to dougwig
14:30:20 <armax> pc_m: thanks
14:30:28 <garyk1> armax: i would like to know from other what they think of the timestamp bp?
14:30:30 <mestery> pc_m: Any progress there?
14:30:40 <mestery> Because that one doesn't look like it's making a ton of progress according to LP at least
14:30:49 <mestery> pc_m: Regarding https://blueprints.launchpad.net/neutron/+spec/neutron-lib
14:30:50 <pc_m> Doug and I have some protoypes, but need to get test working.
14:30:54 <mestery> ACk
14:31:20 <armax> garyk1: faik, that must be resubmitted for mitaka, but that aside, there’s a change from kevinbenton that will help add this type of info to neutron resources
14:31:32 <armax> and it’s a requirement to timestamps, tags and the likes
14:31:34 <garyk1> armax: ok, thanks
14:31:52 <garyk1> that makes more sense that it be something generic
14:31:55 <garyk1> fyo - https://blueprints.launchpad.net/neutron/+spec/add-port-timestamp
14:32:02 <armax> that said, we currently have 21 blueprints targeting ‘neutron’ for M1
14:32:04 <garyk1> kevinbenton: can you please send a link to the work if possible
14:32:57 <wwriverrat> armax: requesting for direction from this group: Was looking for your guidance on how to proceed with "Add API extension for reporting IP address usage statistics": https://review.openstack.org/#/c/212955/
14:32:58 <armax> garyk1: https://review.openstack.org/#/c/222079/
14:33:02 <xgerman> armax is flavor among them?
14:33:13 <armax> xgerman: yes
14:33:14 <shihanzhang> garyk1, this is the link https://review.openstack.org/222079
14:33:39 <armax> we had 28 blueprints targeted for the whole of Liberty
14:33:57 <armax> right now I am targeting everything for the first release
14:34:20 <armax> and will let stuff spill over as events unfold
14:34:48 <armax> bottom line: I am thinking that once we reached the 30-ish mark. We’re most likely shut for business for the Mitaka timeframe
14:35:02 <armax> I can’t think how we can take much more work than that
14:35:23 <xgerman> red bull?
14:35:35 <anteaya> have you seen what it does to your heart?
14:35:54 <salv-orlando> 30-ish makes sense
14:36:26 <armax> the drivers team will still continue to review regularly
14:36:46 <armax> rfe’s and specs
14:37:43 <armax> but we need to have a reasonable expectation of how much we can chew in a single release
14:38:12 <armax> anyhow, I think I am probably stating the obvious
14:38:14 <armax> next up
14:38:18 <armax> #topic Bugs
14:38:21 <armax> ajo: you up
14:38:28 <armax> bugs keep piling up
14:38:36 <armax> :)
14:38:42 <ajo> hi, sorry, I ad an interruption
14:38:47 <ajo> I think garyk1 and I have a list
14:38:51 <garyk1> a lot of the bugs opened recently are lbaas specific
14:38:54 <armax> it looks like we had ~70 reports only during last week
14:39:18 <ajo> yeah, I found this one concerning: https://bugs.launchpad.net/neutron/+bug/1513765 shihanzhang is working on it and needs some review
14:39:18 <openstack> Launchpad bug 1513765 in neutron "bulk delete of ports cost iptables-firewall too much time" [High,In progress] - Assigned to shihanzhang (shihanzhang)
14:39:27 <garyk1> should those be in the neutron bugs or should we create a lbaas launchpad? just an idea? it will help focus more on the core bugs
14:39:39 <ajo> it's due conntrack manipulation.
14:39:55 <ajo> I'm concerned the fix is not backportable, and I'd love if we could have amotoki's eyes on the patch
14:39:56 <armax> garyk1: advs services and neutron are tracked by the same lp
14:40:05 <armax> garyk1: we use tags to differentiate
14:40:23 <garyk1> armax: i understand the tags, but i think that maybe we should consider splitting them out.
14:40:27 <xgerman> yep, and octavia has it’s own tag
14:40:30 <garyk1> that would give a truer picture
14:40:47 <garyk1> but sorry, back to the bugs
14:40:53 <wwriverrat> Regarding GoDaddy/Rax api extension for "IP usage" we plan to re-introduce into m-release: How would you like us to proceed?  RFE->code? RFE->Spec->code? Re: https://review.openstack.org/#/c/180803/
14:41:31 <wwriverrat> Patch exists here: https://review.openstack.org/#/c/212955/
14:42:00 <ajo> this one needs lbaas eyes on it: https://bugs.launchpad.net/neutron/+bug/1163569  it's been hanging there since 2013, not urgent btw, just needs some re-triage.
14:42:01 <openstack> Launchpad bug 1163569 in neutron "security groups don't work with LBaaS vip and ovs plugin" [High,New] - Assigned to Doug Wiegley (dougwig)
14:42:31 <armax> wwriverrat: I think we have an rfe for that, possibly raised by someone else
14:43:01 <wwriverrat> https://bugs.launchpad.net/neutron/+bug/1457986
14:43:01 <openstack> Launchpad bug 1457986 in neutron "RFE: Need API to provide network IP allocation counts" [Wishlist,In progress] - Assigned to David Bingham (wwriverrat)
14:43:33 <armax> wwriverrat: let me look into that
14:43:41 <ajo> enikanorov : could you update about: https://bugs.launchpad.net/neutron/+bug/1453008  ?
14:43:41 <openstack> Launchpad bug 1453008 in neutron "Deadlock on update_port_status" [High,In progress] - Assigned to Eugene Nikanorov (enikanorov)
14:44:20 <wwriverrat> armax: My question is around how to proceed. Will do as little or as much as you see fit.
14:44:55 <armax> wwriverrat: at one point I saw to competing proposals
14:45:13 <armax> wwriverrat: I don’t have a clear picture right now…I need to go back and look before I can advise you further
14:45:33 <armax> going back to bugs...
14:45:36 <wwriverrat> thanks. I’ll try to pull up that other proposal
14:45:47 <armax> ajo: anything else worth raising?
14:45:56 <armax> any gate failure worth noting?
14:46:01 <armax> we have a massive stroke last week
14:46:08 <ajo> https://bugs.launchpad.net/neutron/+bug/1513782
14:46:08 <openstack> Launchpad bug 1513782 in neutron "API response time degradation" [High,New]
14:46:10 <armax> as someone might recall
14:46:12 <ajo> this one is important I think
14:46:40 <ajo> we had some important performance degradation
14:47:07 <armax> kevinbenton seems to have volunteered on 1513782
14:47:20 <armax> I wonder if that falls on the larger performance effort that regXboi is leading?
14:47:29 <regXboi> armax: that is on the list *eventually*
14:47:32 <salv-orl_> it must be because he caused it with some clumsy db op he added ;)
14:47:37 <ajo> armax , I think you guys handled gate failures faster than my wake up time, everytime I was up things were handled
14:47:53 * regXboi just tagged 1513782 as loadimpact
14:48:48 <armax> I already did ;)
14:48:54 <armax> ajo: anything else?
14:49:02 <armax> the torch is with garyk1 this week
14:49:07 <armax> any volunteer for next week?
14:49:23 <rossella_s> armax, me!
14:49:23 <ajo> I think garyk1 spotted a couple of bugs, garyk1 , did I name all?
14:49:31 <armax> rossella_s: super
14:49:34 <wwriverrat> armax: Ref “competing proposal”: https://review.openstack.org/#/c/234541/ targeting our RFE: https://bugs.launchpad.net/neutron/+bug/1457986
14:49:34 <openstack> Launchpad bug 1457986 in neutron "RFE: Need API to provide network IP allocation counts" [Wishlist,In progress] - Assigned to David Bingham (wwriverrat)
14:49:50 <armax> wwriverrat: can we take this offline, please?
14:49:51 <garyk1> armax: ack.
14:50:08 <armax> wwriverrat: it feels like you’re simply hijacking the meeting
14:50:38 <ajo> armax : I think we're done with the important ones I'm aware of
14:50:45 <armax> ajo: ok, cool
14:50:53 <regXboi> wow - where does the time go
14:51:14 <armax> regXboi: indeed
14:51:15 <ajo> sorry: https://bugs.launchpad.net/neutron/+bug/1514810  (noted by garyk1 )
14:51:15 <openstack> Launchpad bug 1514810 in neutron "Turning on 'enable_dhcp' on subnet update cause request failure for pluggable IPAM" [Medium,New] - Assigned to Pavel Bondar (pasha117)
14:51:20 <ajo> but I think carl_baldwin  is already on top of it
14:51:45 <armax> ajo: right, the assigned ones are not the ones that worry me
14:51:55 <armax> :)
14:51:58 <armax> anyhow
14:52:01 <ajo> correct
14:52:24 <armax> emagana is not here so le’ts skip the docs section and go straight to the open discussion one
14:52:40 <armax> #topic Open Discussion
14:52:52 <armax> there are two topics of which one seems already handled
14:53:18 <armax> https://bugs.launchpad.net/neutron/+bug/1507776
14:53:18 <openstack> Launchpad bug 1507776 in neutron "Wrong OVS flows created for new networks" [High,Fix committed] - Assigned to YAMAMOTO Takashi (yamamoto)
14:53:26 <armax> claudiub: ping?
14:53:33 <claudiub> pong
14:53:45 <armax> you raised this for the neutron meeting?
14:53:53 <armax> what’s there to be discussed?
14:53:55 <claudiub> that was weeks ago.
14:54:03 <claudiub> before the summit, i think.
14:54:08 <armax> oh, ack
14:54:12 <armax> it’s left over then?
14:54:31 <claudiub> sorry, I did not get that
14:54:36 <armax> claudiub: never mind
14:54:41 <armax> claudiub: thanks
14:54:47 <claudiub> np. :)
14:54:55 <amotoki> it seems we need to change the title to "On Demand Agenda for <date>".
14:54:59 <ajo> yamamoto++ claudiub++
14:55:08 <xgerman> amotoki +1
14:55:12 <armax> amotoki: I’ll fix that
14:55:50 <armax> btw the other, I am sure more current one, was the request to consider the flavors service plugin be enabled by default
14:56:03 <armax> so that we can test it in the gate on a continuous basis
14:56:23 <ajo> armax , I had a talk with the nova PTL about port flavors, he asked me to sync with you on this.
14:56:49 <ajo> he found it interesting in regards making nova-scheduler aware of neutron stuff
14:57:17 <armax> ajo: I talked to him about last week
14:57:25 <ajo> its a long TL;DR to topic, so I'll probably translate it into an RFE, but he asked me to sync with you
14:57:45 <ajo> " long TL;DR to topic" .... wow ':D
14:58:16 <ajo> I guess we can sync offline
14:58:23 <xgerman> ok, so can we reach a consensus if flavors should be enabled by default or not?
14:58:34 <armax> ajo: let’s take this offline, but the idea is worth exploring especially if that helps addressing a large set of use cases that involve unstructured port bindings
14:58:34 <ajo> (sorry for deviating the topic)
14:58:38 <armax> xgerman: yes
14:59:13 <armax> anyone with an opinion?
14:59:16 <armax> we have 1 min left?
14:59:30 <xgerman> #startvote?
14:59:31 <openstack> Only the meeting chair may start a vote.
14:59:54 <armax> we should take that to the ML, I guess
15:00:04 <armax> I’ll end the meeting now and give you a few secs back
15:00:06 <armax> #endmeeting