14:00:07 <armax> #startmeeting networking
14:00:08 <openstack> Meeting started Tue Jun 16 14:00:07 2015 UTC and is due to finish in 60 minutes.  The chair is armax. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:10 <janonymous_> hi
14:00:11 <openstack> The meeting name has been set to 'networking'
14:00:14 <miyagishi_t> hi
14:00:32 <armax> #link https://wiki.openstack.org/wiki/Network/Meetings
14:00:33 <haleyb> hi
14:00:37 <john-davidge> hi
14:00:37 <mlavalle> hi
14:00:44 <amotoki> hi
14:00:51 <rkukura> hi
14:00:57 <HenryG> o/
14:01:07 <armax> hi everyone!
14:01:40 <armax> let’s dive into the meeting! we have a packed agenda
14:01:42 <ihrachyshka> o/
14:01:52 <armax> #topic Announcements
14:02:05 <sc68cal> o/
14:02:45 <armax> there was a meeting on VLAN aware VMs yesterday
14:03:12 <armax> does anyone who attended care to brief the ones who didn’t?
14:03:22 <Sukhdev> armax: etherpad says it is today
14:03:40 <amotoki> 6/16 1700UTC is the time.
14:03:48 <armax> Sukhdev: thanks, it looks like the agenda is wrong
14:03:54 <pc_m> amotoki: Which channel?
14:04:03 <armax> pc_m: In #openstack-meeting-4 at 1700UTC
14:04:07 <Sukhdev> pc_m: meeting-4
14:04:11 <pc_m> armax: thanks
14:04:34 <xek> o/
14:05:02 <armax> ok, next….there’s still a number of spects and RFE’s in flight
14:05:33 <armax> #link https://review.openstack.org/#/q/status:open+project:openstack/neutron-specs,n,z
14:05:46 <armax> the ones that do not make the cutover date of Liberty 1
14:05:50 <markmcclain> o/
14:06:01 <armax> will be abandoned and need to be resubmitted as RFE bugs
14:06:05 <armax> as outlined here:
14:06:14 <Sukhdev> does anyone know when is Liberty-1?
14:06:21 <armax> #link https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#cutover-to-rfes-from-pure-specs
14:06:35 <armax> Sukhdev: liberty-1 (Jun 23-25)
14:06:45 <armax> Sukhdev: https://wiki.openstack.org/wiki/Liberty_Release_Schedule
14:06:48 <Sukhdev> armax: thanks
14:06:54 <ajo> makes sense,
14:07:19 <armax> next on the announcmement list is Feature branch setup for QoS
14:07:30 <armax> ajo: care to spend a few words?
14:07:38 <ajo> armax: ok :)
14:08:21 <ajo> We have started to work on the feature branch to develop the QoS API extension, agent functionality and anything related to that work.
14:08:30 <armax> it looks like mestery also added a job for the pecan branch
14:08:32 <armax> #link https://review.openstack.org/#/c/190756/
14:08:49 <armax> ajo: is there any intention of doing something along the same lines?
14:08:54 <ajo> we have chosen also the neutron-qos topic to put together anything related to neutron-qos (like neutronclient work) so it's easy to see everything together
14:08:57 <ajo> armax, let me check
14:09:25 <ajo> armax, yes, sc68cal is going to prepare an experimental work for the branch,
14:09:42 <armax> ajo: yes, experimental is best IMO
14:09:58 <ajo> ok, will do that way.
14:10:42 <ajo> related to the QoS branch, I've put a little devref that may deserve some review time: https://review.openstack.org/190635 (about a generic RPC callback)
14:10:45 <armax> I have made a comment on the pecan job, I think check is a bit an overkill…perhaps kevinbenton knows more about this job configuration?
14:10:50 <ajo> which could be equally reused for security groups..
14:10:53 <armax> ajo: yes, I am on it
14:11:05 <armax> ajo: I was distracted by the pymysql switch
14:11:07 <ajo> armax, thanks
14:11:17 <ajo> I'm going to put a new revision on 30 minutes
14:11:24 <ajo> end_of_meeting+30m
14:11:32 <armax> ajo: ok
14:11:57 <armax> ok, on to the next announcement
14:12:04 <armax> next week there’s the Neutron Liberty mid-cycle
14:12:17 <armax> #link https://etherpad.openstack.org/p/neutron-liberty-mid-cycle
14:12:46 <salv-orlando> armax: my opinion on the pecan work is that it either works or not. cahnces of races are limited, so perhaps it can stay in the experimental queue
14:12:54 <armax> there’s also one in Europe not long after that
14:12:55 <armax> June 30 - July 2 in Raanana, Israel (Red Hat offices)
14:13:03 <armax> https://etherpad.openstack.org/p/neutron-liberty-qos-code-sprint
14:13:07 <armax> #link https://etherpad.openstack.org/p/neutron-liberty-qos-code-sprint
14:13:10 <armax> mainly focused on qos
14:13:18 <armax> salv-orlando: agreed
14:13:24 <armax> salv-orlando: I made that comment on mestery’s patch
14:13:35 <ajo> anybody wanting to late-join the IL one, ping me anytime
14:14:24 <armax> ok, moving to the next topic
14:14:34 <armax> unless there’s anyone who wants to add anything?
14:14:38 <Sukhdev> armax: I have an announcement
14:14:42 <armax> Sukhdev: sure
14:15:02 <Sukhdev> I would like to direct team's attention to L2 GW work
14:15:17 <Sukhdev> The wiki is here - https://wiki.openstack.org/wiki/Neutron/L2-GW
14:15:39 <Sukhdev> Please come and join us and provide us with additional use cases that can be implemented
14:16:00 <armax> thanks Sukhdev
14:16:02 <Sukhdev> Next meeting is on 6/22 at 10am PT
14:16:20 <armax> anyone else?
14:16:34 <armax> ...
14:16:38 <armax> #topic Bugs
14:16:57 <armax> I recall that enikanorov is no longer on the frontline for this
14:17:14 <armax> there are a couple of bugs that bit us last week
14:17:35 <armax> I got here https://bugs.launchpad.net/neutron/+bug/1359523 and https://bugs.launchpad.net/neutron/+bug/1335375
14:17:37 <openstack> Launchpad bug 1359523 in neutron "Security group rules are erroneously applied to all ports having same ip addresses in different networks" [High,In progress] - Assigned to shihanzhang (shihanzhang)
14:17:38 <openstack> Launchpad bug 1335375 in neutron "ping still working after security group rule is created, updated, or deleted" [High,In progress] - Assigned to shihanzhang (shihanzhang)
14:17:46 <armax> does anyone have an update?
14:18:25 * armax waits a tad longer
14:18:39 <salv-orlando> I think our rtef ctl pln ltn should
14:18:39 <armax> ok
14:18:40 <ajo> armax, looking, I was tracking that work, but I think there's no update
14:18:55 <armax> salv-orlando: care to explain the acronyms?
14:19:04 <armax> I only got LTN
14:19:11 <armax> :)
14:19:13 <salv-orlando> armax: kevinbenton
14:19:26 <HenryG> I suspect he is asleep
14:19:27 <armax> salv-orlando: oh
14:19:39 <armax> reference control plane Lieutanant
14:19:42 <armax> salv-orlando: gee
14:19:54 <armax> salv-orlando: it’s still 7am here, you know? ;)
14:20:04 <armax> ok
14:20:11 <armax> I’ll bug him when he wakes up
14:20:23 <armax> last week there was a change that startled us a bit
14:20:24 <markmcclain> I thought salv was having a little kid help at the keyboard
14:20:27 <ajo> #action ajo contact shihanzhang about the SG bugs #1335375 and #1359523
14:20:29 <openstack> bug 1359523 in neutron "Security group rules are erroneously applied to all ports having same ip addresses in different networks" [High,In progress] https://launchpad.net/bugs/1359523 - Assigned to shihanzhang (shihanzhang)
14:20:30 <openstack> bug 1335375 in neutron "ping still working after security group rule is created, updated, or deleted" [High,In progress] https://launchpad.net/bugs/1335375 - Assigned to shihanzhang (shihanzhang)
14:20:37 <armax> markmcclain: lol
14:20:41 <ajo> lol :)
14:20:46 <armax> thanks ajo!
14:20:52 <ajo> I will talk to him, adding a reminder here not to forget..
14:21:12 <armax> this change https://review.openstack.org/#/c/184383/
14:21:20 <armax> caused a bit of instability in the gate
14:21:39 <armax> and caused DB functional tests to stop working
14:22:10 <armax> what you need to be aware of, is that the situation was quickly reverted by disabling the driver gloabally
14:22:24 <armax> with https://review.openstack.org/#/c/191010/
14:22:39 <armax> but that still kept the functional job broken
14:22:45 <armax> amuller: keep me honest here
14:22:52 <amuller> you're honest
14:22:54 <amuller> carry on
14:22:58 <ZZelle-backup> :)
14:23:07 <armax> ZZelle-backup has a fix for it at https://review.openstack.org/#/c/190342/
14:23:17 <armax> as for the mysql driver
14:23:37 <armax> we reverted the global switch here:
14:23:46 <armax> https://review.openstack.org/#/c/191113/
14:24:05 <armax> but we selectively bring sanity back to the Tempest Neutron jobs by overriding the mysql driver
14:24:26 <armax> I filed a bunch of patches that introduce an unstable neutron job
14:24:27 <armax> https://review.openstack.org/#/q/topic:neutron-unstable,n,z
14:24:45 <armax> where we can test Neutron with the newer driver, without too much of an havoc
14:24:55 <armax> the failure rate trend is available here: http://goo.gl/YM7gUC
14:24:57 <armax> #link http://goo.gl/YM7gUC
14:25:32 <armax> as usual kevinbenton beated us by filing https://review.openstack.org/#/c/191540/
14:25:46 <armax> but I suspect that needs to bake a bit
14:26:01 <ajo> nice :)
14:26:31 <armax> anyhow, this ramble of mine is to inform you that we’re keeping an unstable job to triage some of the stuff that threw us off
14:26:51 <armax> to test the new driver as well as the # of API workers that was reverted late in the Kilo cycle
14:27:05 <ajo> It's a good approach, so we can sort out those instabilities without affecting everyone else
14:27:40 <armax> once we’re happy with the failure rate, we’ll go back to using pymysql driver (the one that allows us to get to Py3) and we’ll finally have multiple API workers in the gate
14:27:49 <armax> for now, we’re a bit cautious
14:27:57 <armax> anything else
14:27:57 <armax> ?
14:28:26 <HenryG> Thanks armax for the update and for keeping on top of all this
14:29:04 <armax> HenryG: sure thing
14:29:10 <armax> on if nothing else bugs us
14:29:13 <salv-orlando> What's the opinion of the DB lieutinant on the matter? henryg?
14:29:24 <armax> oh, salv-orlando bugs us
14:29:29 * armax waits
14:29:55 <salv-orlando> I think the approach adopted in patch #191540 pretty much uses napalm all over the problem
14:29:56 <HenryG> salv-orlando: We see if kevinbenton's fix works for pymysql
14:30:12 <armax> salv-orlando: I agree
14:30:18 <salv-orlando> HenryG: napalm fixes everything ;)
14:30:39 <salv-orlando> HenryG: I think we'd need to seek help from zzzeeek
14:30:47 <armax> salv-orlando: I think that kevinbenton’s rationale is that until we get rid of the lockmode for update we’re bound to these types of races
14:30:51 <salv-orlando> (after all he wrote he was willing to assist)
14:31:01 <HenryG> salv-orlando: I have engaged him
14:31:07 <salv-orlando> HenryG: awesome
14:31:10 <armax> salv-orlando: and he made an interesting comparison with the nova code base where they use the retry decorator extensively
14:31:40 <armax> salv-orlando: that doesn’t mean that the retry mechanism is a good thing, btw
14:32:00 <HenryG> He was surprised that we are getting deadlocks on insert
14:32:01 <salv-orlando> armax: I agree at 50%. There are also cases where LOCK FOR UPDATE works fine but eventlet screws up. For those cases using the retry decorator is still far from optimal as the timeout of 50secs would still expire
14:32:21 <ajo> ouch
14:32:28 <salv-orlando> armax: cool, I see you guys are up to date on all pro and cons of the decorator
14:32:32 <armax> salv-orlando: agreed…that’s why I mentioned that review 191540 needs to bake a little more
14:32:34 <salv-orlando> I can go back in my graveyard
14:32:45 <armax> salv-orlando: your expertise on the matter would be invaluable
14:32:45 <ihrachyshka> salv-orlando, we should not get those timeouts with the new driver
14:32:55 <armax> salv-orlando: so please consider reviewing the patch if you haven’t already
14:33:04 <armax> salv-orlando: RIP
14:33:14 <armax> #topic Docs
14:33:15 <ajo> ihrachyshka, salv-orlando : definitely 50 sec timeouts on ops.. don't seem good
14:33:15 <salv-orlando> ihrachyshka: indeed, that's what I was told too, but then looking at the problem with sdague and zzzeeek it seems they were there
14:33:29 <ihrachyshka> meh, ok
14:33:33 <armax> it looks like emagana is not around
14:33:43 <salv-orlando> ajo: that's the default mysql lock wait timeout, but it gets triggered by eventlet not being smart enough in switching threads
14:34:28 <salv-orlando> ihrachyshka, armax, ajo, HenryG: we might need to stop wasting time about this issue now... the agenda is packed. We just need to keep eyes on that lock wait timeout are not present even if the job succeeds
14:34:40 <armax> salv-orlando: agreed
14:34:49 <ajo> salv-orlando: correct
14:34:57 <armax> anyhow, emagana is not here so we are left with
14:35:02 <armax> #topic On Demand Agenda
14:35:15 <armax> got a few items here
14:35:38 <armax> nova-network compatibility tasks
14:35:48 <armax> #link v
14:35:51 <armax> #link https://etherpad.openstack.org/p/YVR-nova-network
14:35:54 <neiljerram> I also have a thing, after yours
14:36:22 <armax> anyone care to update on tasks being tracked here?
14:36:48 <armax> I was looking at russellb’s devstack patches
14:36:52 <armax> I need to go back to those
14:37:15 <sc68cal> I'm working on the LB job, making some progress
14:37:27 <mlavalle> armax: I am working on priority number 4, dns
14:37:37 <armax> mlavalle: thanks
14:37:51 <armax> mlavalle: how is it going?
14:38:08 <mlavalle> the rfe is approved on the neutron side
14:38:21 <mlavalle> I am also cleaning up a spec
14:38:40 <mlavalle> there is also a spec on the nova side that I am working with johnthetubaguy
14:38:47 <armax> mlavalle: great
14:38:55 <neiljerram> mlavalle: Worth adding links to that etherpad?
14:38:55 <Sukhdev> mlavalle: link?
14:38:59 <mlavalle> and I am already coding the neutron side.
14:39:23 <mlavalle> nova: https://review.openstack.org/#/c/90150/
14:39:40 <mlavalle> neutron: https://review.openstack.org/#/c/88623/
14:40:17 <mlavalle> should push WIP code tomorrow
14:41:14 <armax> perhaps russellb is not around
14:41:55 <armax> there are a bunch of patches in flight to ensure we test rolling upgrades continously
14:42:24 <armax> they are captured on the etherpad…if there are eyes who are interested in looking at those, that’d be great
14:42:49 <armax> Distributed SNAT with DVR is still unclaimed, is it not?
14:43:09 <amuller> correct
14:43:27 <miyagishi_t> I'd like to implement Distributed SNAT in Liberty.
14:43:59 <armax> miyagishi_t: please put your name next to the item on the etherpad
14:44:01 <armax> like 92
14:44:16 <miyagishi_t> armax: okay
14:44:45 <armax> miyagishi_t: have you started to look into this already?
14:45:01 <miyagishi_t> currently under discussion in our team.
14:45:16 <miyagishi_t> armax: I'd like to submit blueprint as soon as possible.
14:45:21 <armax> miyagishi_t: please sync up with kevinbenton and myself
14:45:44 <armax> miyagishi_t: and we may be able to help you stay on the right track
14:46:42 <armax> miyagishi_t: get yourself familiar with the new submission process for Liberty
14:46:44 <armax> #link https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst
14:46:55 <armax> miyagishi_t: if you aren’t already
14:47:23 <armax> last item I have on the nova-net laundry list is the ‘get me a network’ spec
14:47:58 <armax> this is still unclaimed, even though I was going to look into it as soon as I had the chance :)
14:48:19 <armax> sc68cal: anything on your front?
14:48:49 <sc68cal> armax: I think last week there were volunteers to take over the spec to fill out more of the detail?
14:49:07 <armax> sc68cal: ok, so you’d like someone to take over the spec as well?
14:49:23 <salv-orlando> sc68cal: volunteers are the most volatile substance ever seen
14:49:33 <sc68cal> armax: I'll have to pull up the logs to remember the context - but there were volunteers
14:49:36 <haleyb> sc68cal: do you need me to add my comments?  I'm hoping I remembered things correctly
14:49:39 <salv-orlando> armax: I can offer some part-time help
14:50:14 <armax> first step would be to get the spec agreed on and merged
14:50:26 <sc68cal> haleyb: I think your comments were correct, I think if you want to add them to the spec that'd be good
14:50:26 <amotoki> haleyb: feel free to add your comments.
14:50:35 <armax> for that we’d need a submitter that refresh the spec everytime a reviewer makes one, so for that we’d need reviewers too
14:51:12 <haleyb> armax: i will update the spec and try to address all the comments
14:51:17 <ZZelle-backup> i can
14:51:18 <armax> sc68cal: are you going to stop updating the spec yourself?
14:51:23 <ZZelle-backup> oups too late
14:51:37 <haleyb> darn, i knew i should have waited :)
14:51:45 <armax> ok, I gather that haleyb is going to take over the submission side of things?
14:51:50 <sc68cal> armax: I think others can update the spec based on what they remembered
14:52:09 <armax> sc68cal: true, but if we don’t coordinate it becomes a bit messy
14:52:20 <ZZelle-backup> haleyb: mouarf
14:52:31 <armax> we might as well have people fill in the specs with comments and a single person respin the patch
14:52:40 <armax> but either way
14:52:48 <haleyb> i had the largest comment so i will do the next update and incorporate other comments
14:52:49 <sc68cal> armax: ok - I'll coordinate with haleyb
14:53:01 <armax> sc68cal: thanks
14:53:07 <armax> haleyb: thanks
14:53:29 <armax> we’ll leave the last item on the nova-net agenda for next week
14:53:42 <armax> #action armax reminds mestery to read the meeting log
14:54:04 <armax> we got a few minutes to spare, is there anything else someone would like to bring up to the team’s attention?
14:54:15 <neiljerram> Another Nova/Neutron thing, although not nova-net....
14:54:27 <neiljerram> https://review.openstack.org/#/c/162468/
14:54:38 <pc_m> Can folks kindly look at https://review.openstack.org/191944? Would like to get community feedback on direction here.
14:54:42 <neiljerram> This is about the future of VIF plugging...
14:54:51 <hichihara> Can I ask core team about metaplugin deprecation?
14:55:52 <john-davidge> Would appreciate some more attention on the IPv6 Prefix Delegation work as well https://review.openstack.org/#/c/158697
14:55:54 <john-davidge> thanks
14:56:00 <neiljerram> Nova cores are finding it hard to reach consensus on this, and I wonder if some Neutron eyes might help.
14:56:15 <armax> neiljerram: noted
14:56:28 <HenryG> Plugin/driver decomposition phase 2: https://review.openstack.org/187267
14:56:34 <armax> hichihara: iirc the metaplugin is going to be remoted soon
14:56:39 <armax> *removed
14:56:51 <salv-orlando> neiljerram:  I can trime and chime in with my opinions, but there's a chance I'll end up generating even more confusion
14:56:55 <armax> hichihara: reach out to mestery, I am sure he knows the latest
14:57:15 <hichihara> armax: really? I want to remove it in Liberty not M
14:57:18 <neiljerram> salv-orlando: I'm sure your contribution would be net positive!
14:57:26 <armax> hichihara: ok, so be it
14:57:29 <salv-orlando> neiljerram: the point is that nova made a decision some time ago to make vif plugging completely generic and exclusively driver by parameters dictated from port bindings
14:57:38 <hichihara> armax: Thanks!
14:57:48 <salv-orlando> probably introducing a "script" kind of reverts that decision
14:57:52 <numan> https://review.openstack.org/#/c/189741 request to please review
14:58:28 <neiljerram> salv-orlando: It sounds like you know more of the history here than me...
14:59:22 <neiljerram> salv-orlando: Would you like to discuss this a little further in openstack-neutron after the meeting?
14:59:24 <armax> neiljerram, salv-orlando: it looks like this is the type of conversation that should be tracked on the review page, perhaps?
14:59:29 <armax> or in-channel
14:59:47 <armax> okay, guys, we’re a minute away from the end of the meeting
14:59:53 <armax> I’ll give you a minute back
14:59:58 <salv-orlando> armax: you're right. Anyway I have another meeting now, I will be back available in openstack-neutron in 40 mins
15:00:00 <armax> thanks for joining!
15:00:03 <armax> #endmeeting