21:00:52 <armax> #startmeeting networking
21:00:53 <openstack> Meeting started Mon Jan 23 21:00:52 2017 UTC and is due to finish in 60 minutes.  The chair is armax. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:56 <openstack> The meeting name has been set to 'networking'
21:01:02 <njohnston> o/
21:01:04 <annp> hi
21:01:13 <asingh_> hello
21:01:19 <ihrachys> o/
21:01:45 <dasanind> o/
21:01:54 <armax> #topic announcements and reminders
21:02:09 <armax> This week is a crucial week
21:02:24 <mlavalle> indeed
21:02:45 <armax> #link https://releases.openstack.org/ocata/schedule.html
21:02:50 <electrocucaracha> o/
21:03:05 <armax> it’s Ocata-3 week
21:03:22 <armax> this means we’re gonna go in feature freeze
21:03:37 <armax> we’ll have to wrap up whatever we have outstanding
21:03:57 <armax> and make sure we give absolute priority to things are close to crossing the finishing line
21:04:22 <armax> it’s also Pike PTL self-nomination week
21:04:35 <Sukhdev> Hi
21:05:15 <armax> which means we will have a new PTL starting the week of Feb 06
21:05:30 <armax> I’ll work with him/her to ensure a smooth transition
21:05:41 <manjeets> o/
21:05:43 <armax> in relation to the release and PTG organization
21:07:34 <armax> it’s also the week where we need to nail down our client release
21:08:22 <armax> hopefully we’ll be able to cut an RC1 as early as next week so that we create a stable branch and open master for Pike immediately after that
21:08:58 <armax> anyone else has anything to add?
21:09:05 <ihrachys> armax: question on the last one
21:09:11 <armax> ihrachys: yes, sir
21:09:15 <sindhu> o/
21:09:25 <ihrachys> armax: once stable branch is cut off, are we free to land Pike stuff, or you want some cool off period still?
21:10:20 <armax> ihrachys: so long as what we land in pike is not gonna screw up our ability to backport cleanly
21:10:34 <johndperkins> +
21:10:43 <ihrachys> that's hard to assess sometimes, but gotcha
21:10:46 <armax> ihrachys: you raise a good point, we just need to be careful to what we merge in pike
21:11:05 <armax> ihrachys: agreed
21:11:24 <armax> we have the option of taking the hard line and block everything until we’re absolutely sure we have a good RC
21:11:35 <armax> but that seems to defeat the point of cutting a stable branch
21:11:57 <armax> or merge everything as it was business as usual, but that’s risky as well
21:12:01 <ihrachys> yea, I wouldn't like that. two branches are intended to be able to land in both. :)
21:12:18 <armax> I think targeted changes should be fine
21:12:30 <armax> things like API and DB changes are always risky to land asap
21:12:32 <armax> for instance
21:12:51 <ihrachys> ok, the usual 'be cautious' approach, sounds good to me
21:12:51 <armax> I mean API or DB related changes
21:13:50 <armax> yeah, and we’ve done this for a while now, I think we have a pretty good handle on it
21:14:37 <armax> dasm and I will work on cutting O-3 and the client release by Thurday this week
21:14:54 <dasm> aye aye, sir
21:15:15 <armax> let’s make sure we line all our ducks in the right row :)
21:15:45 <armax> any other announcement/reminder?
21:16:29 <armax> looks like we’re good
21:16:34 <armax> moving on
21:16:42 <armax> #topic blueprints
21:17:05 <armax> Any patch worth raising awareness for?
21:17:31 <mlavalle> I need a final nudge for this one:
21:17:37 <mlavalle> #link https://review.openstack.org/#/c/358658/
21:17:45 <armax> hang on
21:17:51 <armax> I feel we have a gate failure
21:17:53 <armax> going on
21:18:00 <armax> was looking at zuul
21:18:05 <armax> to see how light the load was
21:18:36 <armax> http://logs.openstack.org/78/423578/2/check/gate-grenade-dsvm-neutron-ubuntu-xenial/2a25430/logs/grenade.sh.txt.gz#_2017-01-23_20_35_26_914
21:18:41 <dasm> jenkins seems to beis crazy lately. random -1/-2 without any reason
21:18:43 <armax> not sure if it’s me feaking out
21:18:50 <armax> or there’s something going on
21:19:04 <dasm> armax: this seems to be related to osc, if i recall correctly
21:19:12 <dasm> osc is almost entirely red right now
21:19:30 <dasm> and python-neutronclient had osc-check-plugins gate failing
21:19:32 <armax> I see this
21:19:34 <armax> http://logs.openstack.org/93/404293/41/check/gate-grenade-dsvm-neutron-ubuntu-xenial/0b6f384/logs/grenade.sh.txt.gz#_2017-01-23_19_02_20_389
21:19:35 <armax> at least twice
21:19:39 <armax> so STOP the merging!
21:20:05 <dasm> abandon the ship!
21:20:22 <kevinbenton> okay :(
21:20:26 * armax runs to get a life vest
21:20:32 <kevinbenton> or maybe merging more code will make it go faster?
21:20:33 <kevinbenton> :)
21:21:04 <armax> I wonder if a G-R requirement bump went undetected
21:21:15 <ihrachys> I have some OVO patches worth a look even if not merging right away, not sure if I get a separate slot for that though
21:22:09 * armax looking
21:22:25 * manjeets just reported a bug for grenade failure over gate
21:22:51 <armax> https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L257
21:23:24 <njohnston> I hope to have this patch passing all tests tomorrow - it's the last piece that fwaas v2 depends on outside the neutron-fwaas repository: https://review.openstack.org/#/c/348177/
21:23:33 <armax> the packaging requirment issue seems a red-herring
21:24:13 <armax> manjeets: bug no?
21:24:54 <manjeets> https://bugs.launchpad.net/neutron/+bug/1658806
21:24:54 <openstack> Launchpad bug 1658806 in neutron "Broken grenade jobs" [Undecided,New]
21:25:06 <manjeets> i see same failure on 3 patches
21:25:15 <manjeets> which were in merge queue
21:25:30 <armax> manjeets: neutron changes?
21:25:55 <manjeets> just openned a ticket for investigation
21:26:12 <manjeets> its failing at deployment phase
21:26:40 <armax> ok, I guess this fire has just brought us to the next section
21:26:45 <armax> #topic Bugs and Gate issues
21:27:02 <armax> is there anyone interested in looking into bug 1658806
21:27:02 <openstack> bug 1658806 in neutron "Broken grenade jobs" [Critical,Confirmed] https://launchpad.net/bugs/1658806
21:27:02 <armax> ?
21:28:15 <armax> as far as I can tell this happens for multinode too
21:28:21 <armax> and in the gate
21:28:26 <armax> http://logs.openstack.org/01/301801/2/gate/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/793c1bd/logs/grenade.sh.txt.gz
21:28:41 <manjeets> i think its version mismatch issue packaging 16.7 on machine if it will be 16.8
21:28:58 <manjeets> i hope issue will be resolved on all
21:30:08 <dasm> armax: i'm looking into that. trying to reach the source (requirements team) to double check this. i believe it's far beyond just neutron
21:31:08 <armax> talked to infra
21:31:14 <armax> it looks like we have a way out
21:31:15 <armax> https://review.openstack.org/#/c/424323/
21:31:32 <manjeets> that'll fix all
21:31:50 <kevinbenton> so approve all the things now?
21:31:59 <dasm> armax: ack
21:32:00 <armax> it’s a requirement change
21:32:25 <armax> let’s wait for things to go calm again before opening the flood gates
21:33:03 <armax> the top of the queue
21:33:04 <armax> https://review.openstack.org/#/q/project:openstack/requirements+status:open
21:33:04 <kevinbenton> upper constraints we don't have have to wait for requirements bump though from the bot, right?
21:33:10 <armax> was also pretty revealing
21:33:30 <armax> no
21:33:34 <ihrachys> kevinbenton: no but the patch is not merged
21:33:41 <kevinbenton> right
21:33:43 <ihrachys> kevinbenton: so let's say it's bumped off gate due to some failure
21:33:56 <ihrachys> then all we put into gate starts respinning without it, and failing
21:34:02 <kevinbenton> no, i just mean we don't have to wait to approve one on neutron as well
21:34:21 <ihrachys> yeah, we should only wait until this lands before putting more work in gate.
21:34:22 <armax> right, but we better wait for the upper constraint to land
21:34:34 <armax> until we can run on the right constraint
21:34:45 <armax> the problem is that the issue happens on the stable branch setup
21:36:27 <armax> unfortunately the patch is not good in the gate
21:36:30 <armax> 424323
21:36:35 <armax> http://logs.openstack.org/23/424323/1/check/gate-requirements-integration-dsvm-ubuntu-xenial/ca816a0/console.html
21:36:54 <armax> it’ll probably be a while until we put stuff in the gate
21:37:34 <ihrachys> I guess we can track that in async mode
21:37:43 <kevinbenton> shall we just call what we have now the ocata release then :)
21:38:21 <armax> ok, let’s move on
21:38:26 <armax> I’ll send an email to the ML
21:38:33 <armax> any other gate failure we want to track here?
21:39:43 <armax> no?
21:40:04 <janzian> There is #link https://bugs.launchpad.net/neutron/+bug/1656386
21:40:04 <openstack> Launchpad bug 1656386 in neutron "Memory leaks on Neutron jobs" [Critical,Confirmed]
21:40:06 <armax> this link begs to differ
21:40:12 <armax> #link https://bugs.launchpad.net/neutron/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.tag=gate-failure
21:41:00 <armax> janzian: no assignee on that one
21:41:38 <janzian> Looks like root cause has proven elusive so far
21:41:57 <ihrachys> the size of the list of gate failures corresponds to sad state of our gate
21:42:07 <armax> this was flagged as a linuxbridge failure
21:42:41 <kevinbenton> it hits the linuxbridge job the most, but it's not clear neutron is at fault
21:42:45 <armax> but it looks like the jobs involved go beyond linuxbridge alone
21:42:54 <armax> kevinbenton: it does not look like
21:43:21 <armax> I see 12 occurences against 2 between full and linuxbridge in the latest logstash run I am looking at
21:44:17 <armax> what’s interesting is that it’s reported 41 times on change https://review.openstack.org/#/c/421593/
21:45:06 <armax> but it might be a red-herring
21:45:22 <kevinbenton> what would an abandoned change have to do with it?
21:45:32 <armax> kevinbenton: that’s what I am saying
21:46:02 <janzian> There are a couple other bugs reporting random failures across jobs, they might be stemming from the same problem
21:46:10 <janzian> Namely #link https://bugs.launchpad.net/neutron/+bug/1657761
21:46:10 <openstack> Launchpad bug 1657761 in neutron "ovsdb native interface timeouts sometimes causing random functional failures" [High,New]
21:46:17 <janzian> And #link https://bugs.launchpad.net/neutron/+bug/1657935
21:46:17 <openstack> Launchpad bug 1657935 in neutron "intermittent failures in setup_physical_bridges" [High,Confirmed]
21:46:27 <armax> we really need to get to the bottom of the memory leak bug
21:46:42 <armax> janzian: bug 1657935 might be solved already
21:46:58 <armax> I have seen no occurences of it over the past few days, I provided my findings on the bug report
21:47:10 <janzian> Ah yes, I missed that update
21:47:22 <armax> anyhow we should probably move along, we took most of the meeting time talking about gate woes
21:48:16 <armax> #topic Docs
21:48:58 <armax> With feature freeze and the release approaching, let me kindly remind the team that we should take some time document what we did over the past few months
21:49:17 <armax> so please take the time to review  https://bugs.launchpad.net/neutron/+bugs?field.tag=doc to see if there’s any doc bug that can be squashed
21:49:31 <mlavalle> armax: I just mrged the update for the networking guide on the work I've been doing for routed networks:
21:49:39 <mlavalle> #link https://review.openstack.org/#/c/423723/
21:49:40 <armax> mlavalle: sweet
21:49:49 <ankur-gupta-f1> armax: would Feature Classification fall under the feature freeze?
21:49:56 <mlavalle> it got merged in a couple of days
21:50:04 <armax> ankur-gupta-f1: what do you mean?
21:50:27 <ankur-gupta-f1> its nearing completion could it be merged this week potentially, or will i have to wait for stable to be cut
21:50:58 <armax> ankur-gupta-f1: I am okay merging when it’s ready irrespective of the feature freeze
21:51:03 <ankur-gupta-f1> okay thanks
21:51:41 <armax> ankur-gupta-f1: thanks for sticking to it
21:52:12 <ankur-gupta-f1> np. it give me lots of joy
21:52:12 <armax> #topic OpenStack Client
21:52:20 <armax> anything worth sharing here?
21:52:44 <ankur-gupta-f1> OSC held up on reviews still
21:53:13 <ankur-gupta-f1> https://etherpad.openstack.org/p/osc-transition-priorities
21:54:24 <armax> ankur-gupta-f1: ack
21:56:20 <armax> ankur-gupta-f1: I’ll go over the list of patches to see if there’s anything I missed
21:56:33 <armax> #topiuc Neutron-lib and planned refactoring
21:56:53 <mlavalle> armax: topic^^^
21:57:26 <armax> #topic Neutron-lib and planned refactoring
21:58:11 <armax> at this point I’d defer most of LibImpact changes to Pike
21:58:24 <armax> as we’re approaching feature freeze
21:59:23 <armax> ok guys
21:59:29 <armax> and ladies
21:59:35 <armax> it looks like we ran out of time
21:59:48 <njohnston> \o
21:59:54 <armax> thanks for joining
21:59:58 <armax> #endmeeting