21:00:52 <armax> #startmeeting networking 21:00:53 <openstack> Meeting started Mon Jan 23 21:00:52 2017 UTC and is due to finish in 60 minutes. The chair is armax. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:54 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:56 <openstack> The meeting name has been set to 'networking' 21:01:02 <njohnston> o/ 21:01:04 <annp> hi 21:01:13 <asingh_> hello 21:01:19 <ihrachys> o/ 21:01:45 <dasanind> o/ 21:01:54 <armax> #topic announcements and reminders 21:02:09 <armax> This week is a crucial week 21:02:24 <mlavalle> indeed 21:02:45 <armax> #link https://releases.openstack.org/ocata/schedule.html 21:02:50 <electrocucaracha> o/ 21:03:05 <armax> it’s Ocata-3 week 21:03:22 <armax> this means we’re gonna go in feature freeze 21:03:37 <armax> we’ll have to wrap up whatever we have outstanding 21:03:57 <armax> and make sure we give absolute priority to things are close to crossing the finishing line 21:04:22 <armax> it’s also Pike PTL self-nomination week 21:04:35 <Sukhdev> Hi 21:05:15 <armax> which means we will have a new PTL starting the week of Feb 06 21:05:30 <armax> I’ll work with him/her to ensure a smooth transition 21:05:41 <manjeets> o/ 21:05:43 <armax> in relation to the release and PTG organization 21:07:34 <armax> it’s also the week where we need to nail down our client release 21:08:22 <armax> hopefully we’ll be able to cut an RC1 as early as next week so that we create a stable branch and open master for Pike immediately after that 21:08:58 <armax> anyone else has anything to add? 21:09:05 <ihrachys> armax: question on the last one 21:09:11 <armax> ihrachys: yes, sir 21:09:15 <sindhu> o/ 21:09:25 <ihrachys> armax: once stable branch is cut off, are we free to land Pike stuff, or you want some cool off period still? 21:10:20 <armax> ihrachys: so long as what we land in pike is not gonna screw up our ability to backport cleanly 21:10:34 <johndperkins> + 21:10:43 <ihrachys> that's hard to assess sometimes, but gotcha 21:10:46 <armax> ihrachys: you raise a good point, we just need to be careful to what we merge in pike 21:11:05 <armax> ihrachys: agreed 21:11:24 <armax> we have the option of taking the hard line and block everything until we’re absolutely sure we have a good RC 21:11:35 <armax> but that seems to defeat the point of cutting a stable branch 21:11:57 <armax> or merge everything as it was business as usual, but that’s risky as well 21:12:01 <ihrachys> yea, I wouldn't like that. two branches are intended to be able to land in both. :) 21:12:18 <armax> I think targeted changes should be fine 21:12:30 <armax> things like API and DB changes are always risky to land asap 21:12:32 <armax> for instance 21:12:51 <ihrachys> ok, the usual 'be cautious' approach, sounds good to me 21:12:51 <armax> I mean API or DB related changes 21:13:50 <armax> yeah, and we’ve done this for a while now, I think we have a pretty good handle on it 21:14:37 <armax> dasm and I will work on cutting O-3 and the client release by Thurday this week 21:14:54 <dasm> aye aye, sir 21:15:15 <armax> let’s make sure we line all our ducks in the right row :) 21:15:45 <armax> any other announcement/reminder? 21:16:29 <armax> looks like we’re good 21:16:34 <armax> moving on 21:16:42 <armax> #topic blueprints 21:17:05 <armax> Any patch worth raising awareness for? 21:17:31 <mlavalle> I need a final nudge for this one: 21:17:37 <mlavalle> #link https://review.openstack.org/#/c/358658/ 21:17:45 <armax> hang on 21:17:51 <armax> I feel we have a gate failure 21:17:53 <armax> going on 21:18:00 <armax> was looking at zuul 21:18:05 <armax> to see how light the load was 21:18:36 <armax> http://logs.openstack.org/78/423578/2/check/gate-grenade-dsvm-neutron-ubuntu-xenial/2a25430/logs/grenade.sh.txt.gz#_2017-01-23_20_35_26_914 21:18:41 <dasm> jenkins seems to beis crazy lately. random -1/-2 without any reason 21:18:43 <armax> not sure if it’s me feaking out 21:18:50 <armax> or there’s something going on 21:19:04 <dasm> armax: this seems to be related to osc, if i recall correctly 21:19:12 <dasm> osc is almost entirely red right now 21:19:30 <dasm> and python-neutronclient had osc-check-plugins gate failing 21:19:32 <armax> I see this 21:19:34 <armax> http://logs.openstack.org/93/404293/41/check/gate-grenade-dsvm-neutron-ubuntu-xenial/0b6f384/logs/grenade.sh.txt.gz#_2017-01-23_19_02_20_389 21:19:35 <armax> at least twice 21:19:39 <armax> so STOP the merging! 21:20:05 <dasm> abandon the ship! 21:20:22 <kevinbenton> okay :( 21:20:26 * armax runs to get a life vest 21:20:32 <kevinbenton> or maybe merging more code will make it go faster? 21:20:33 <kevinbenton> :) 21:21:04 <armax> I wonder if a G-R requirement bump went undetected 21:21:15 <ihrachys> I have some OVO patches worth a look even if not merging right away, not sure if I get a separate slot for that though 21:22:09 * armax looking 21:22:25 * manjeets just reported a bug for grenade failure over gate 21:22:51 <armax> https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L257 21:23:24 <njohnston> I hope to have this patch passing all tests tomorrow - it's the last piece that fwaas v2 depends on outside the neutron-fwaas repository: https://review.openstack.org/#/c/348177/ 21:23:33 <armax> the packaging requirment issue seems a red-herring 21:24:13 <armax> manjeets: bug no? 21:24:54 <manjeets> https://bugs.launchpad.net/neutron/+bug/1658806 21:24:54 <openstack> Launchpad bug 1658806 in neutron "Broken grenade jobs" [Undecided,New] 21:25:06 <manjeets> i see same failure on 3 patches 21:25:15 <manjeets> which were in merge queue 21:25:30 <armax> manjeets: neutron changes? 21:25:55 <manjeets> just openned a ticket for investigation 21:26:12 <manjeets> its failing at deployment phase 21:26:40 <armax> ok, I guess this fire has just brought us to the next section 21:26:45 <armax> #topic Bugs and Gate issues 21:27:02 <armax> is there anyone interested in looking into bug 1658806 21:27:02 <openstack> bug 1658806 in neutron "Broken grenade jobs" [Critical,Confirmed] https://launchpad.net/bugs/1658806 21:27:02 <armax> ? 21:28:15 <armax> as far as I can tell this happens for multinode too 21:28:21 <armax> and in the gate 21:28:26 <armax> http://logs.openstack.org/01/301801/2/gate/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/793c1bd/logs/grenade.sh.txt.gz 21:28:41 <manjeets> i think its version mismatch issue packaging 16.7 on machine if it will be 16.8 21:28:58 <manjeets> i hope issue will be resolved on all 21:30:08 <dasm> armax: i'm looking into that. trying to reach the source (requirements team) to double check this. i believe it's far beyond just neutron 21:31:08 <armax> talked to infra 21:31:14 <armax> it looks like we have a way out 21:31:15 <armax> https://review.openstack.org/#/c/424323/ 21:31:32 <manjeets> that'll fix all 21:31:50 <kevinbenton> so approve all the things now? 21:31:59 <dasm> armax: ack 21:32:00 <armax> it’s a requirement change 21:32:25 <armax> let’s wait for things to go calm again before opening the flood gates 21:33:03 <armax> the top of the queue 21:33:04 <armax> https://review.openstack.org/#/q/project:openstack/requirements+status:open 21:33:04 <kevinbenton> upper constraints we don't have have to wait for requirements bump though from the bot, right? 21:33:10 <armax> was also pretty revealing 21:33:30 <armax> no 21:33:34 <ihrachys> kevinbenton: no but the patch is not merged 21:33:41 <kevinbenton> right 21:33:43 <ihrachys> kevinbenton: so let's say it's bumped off gate due to some failure 21:33:56 <ihrachys> then all we put into gate starts respinning without it, and failing 21:34:02 <kevinbenton> no, i just mean we don't have to wait to approve one on neutron as well 21:34:21 <ihrachys> yeah, we should only wait until this lands before putting more work in gate. 21:34:22 <armax> right, but we better wait for the upper constraint to land 21:34:34 <armax> until we can run on the right constraint 21:34:45 <armax> the problem is that the issue happens on the stable branch setup 21:36:27 <armax> unfortunately the patch is not good in the gate 21:36:30 <armax> 424323 21:36:35 <armax> http://logs.openstack.org/23/424323/1/check/gate-requirements-integration-dsvm-ubuntu-xenial/ca816a0/console.html 21:36:54 <armax> it’ll probably be a while until we put stuff in the gate 21:37:34 <ihrachys> I guess we can track that in async mode 21:37:43 <kevinbenton> shall we just call what we have now the ocata release then :) 21:38:21 <armax> ok, let’s move on 21:38:26 <armax> I’ll send an email to the ML 21:38:33 <armax> any other gate failure we want to track here? 21:39:43 <armax> no? 21:40:04 <janzian> There is #link https://bugs.launchpad.net/neutron/+bug/1656386 21:40:04 <openstack> Launchpad bug 1656386 in neutron "Memory leaks on Neutron jobs" [Critical,Confirmed] 21:40:06 <armax> this link begs to differ 21:40:12 <armax> #link https://bugs.launchpad.net/neutron/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.tag=gate-failure 21:41:00 <armax> janzian: no assignee on that one 21:41:38 <janzian> Looks like root cause has proven elusive so far 21:41:57 <ihrachys> the size of the list of gate failures corresponds to sad state of our gate 21:42:07 <armax> this was flagged as a linuxbridge failure 21:42:41 <kevinbenton> it hits the linuxbridge job the most, but it's not clear neutron is at fault 21:42:45 <armax> but it looks like the jobs involved go beyond linuxbridge alone 21:42:54 <armax> kevinbenton: it does not look like 21:43:21 <armax> I see 12 occurences against 2 between full and linuxbridge in the latest logstash run I am looking at 21:44:17 <armax> what’s interesting is that it’s reported 41 times on change https://review.openstack.org/#/c/421593/ 21:45:06 <armax> but it might be a red-herring 21:45:22 <kevinbenton> what would an abandoned change have to do with it? 21:45:32 <armax> kevinbenton: that’s what I am saying 21:46:02 <janzian> There are a couple other bugs reporting random failures across jobs, they might be stemming from the same problem 21:46:10 <janzian> Namely #link https://bugs.launchpad.net/neutron/+bug/1657761 21:46:10 <openstack> Launchpad bug 1657761 in neutron "ovsdb native interface timeouts sometimes causing random functional failures" [High,New] 21:46:17 <janzian> And #link https://bugs.launchpad.net/neutron/+bug/1657935 21:46:17 <openstack> Launchpad bug 1657935 in neutron "intermittent failures in setup_physical_bridges" [High,Confirmed] 21:46:27 <armax> we really need to get to the bottom of the memory leak bug 21:46:42 <armax> janzian: bug 1657935 might be solved already 21:46:58 <armax> I have seen no occurences of it over the past few days, I provided my findings on the bug report 21:47:10 <janzian> Ah yes, I missed that update 21:47:22 <armax> anyhow we should probably move along, we took most of the meeting time talking about gate woes 21:48:16 <armax> #topic Docs 21:48:58 <armax> With feature freeze and the release approaching, let me kindly remind the team that we should take some time document what we did over the past few months 21:49:17 <armax> so please take the time to review https://bugs.launchpad.net/neutron/+bugs?field.tag=doc to see if there’s any doc bug that can be squashed 21:49:31 <mlavalle> armax: I just mrged the update for the networking guide on the work I've been doing for routed networks: 21:49:39 <mlavalle> #link https://review.openstack.org/#/c/423723/ 21:49:40 <armax> mlavalle: sweet 21:49:49 <ankur-gupta-f1> armax: would Feature Classification fall under the feature freeze? 21:49:56 <mlavalle> it got merged in a couple of days 21:50:04 <armax> ankur-gupta-f1: what do you mean? 21:50:27 <ankur-gupta-f1> its nearing completion could it be merged this week potentially, or will i have to wait for stable to be cut 21:50:58 <armax> ankur-gupta-f1: I am okay merging when it’s ready irrespective of the feature freeze 21:51:03 <ankur-gupta-f1> okay thanks 21:51:41 <armax> ankur-gupta-f1: thanks for sticking to it 21:52:12 <ankur-gupta-f1> np. it give me lots of joy 21:52:12 <armax> #topic OpenStack Client 21:52:20 <armax> anything worth sharing here? 21:52:44 <ankur-gupta-f1> OSC held up on reviews still 21:53:13 <ankur-gupta-f1> https://etherpad.openstack.org/p/osc-transition-priorities 21:54:24 <armax> ankur-gupta-f1: ack 21:56:20 <armax> ankur-gupta-f1: I’ll go over the list of patches to see if there’s anything I missed 21:56:33 <armax> #topiuc Neutron-lib and planned refactoring 21:56:53 <mlavalle> armax: topic^^^ 21:57:26 <armax> #topic Neutron-lib and planned refactoring 21:58:11 <armax> at this point I’d defer most of LibImpact changes to Pike 21:58:24 <armax> as we’re approaching feature freeze 21:59:23 <armax> ok guys 21:59:29 <armax> and ladies 21:59:35 <armax> it looks like we ran out of time 21:59:48 <njohnston> \o 21:59:54 <armax> thanks for joining 21:59:58 <armax> #endmeeting