16:00:33 <ihrachys> #startmeeting neutron_ci
16:00:34 <openstack> Meeting started Tue Oct 31 16:00:33 2017 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:37 <openstack> The meeting name has been set to 'neutron_ci'
16:00:40 <mlavalle> o/
16:00:50 <ihrachys> wassuuuup
16:01:05 <haleyb> hi
16:01:09 <davidsha> Hi
16:01:24 <ihrachys> #topic Actions from prev meeting
16:01:27 <jlibosva> o/
16:01:50 <ihrachys> "haleyb to follow up with infra on missing grafite data for zuulv3 jobs"
16:02:03 <ihrachys> grafana is still empty
16:02:14 <haleyb> done - board is happy and neutron-lib merged as well
16:02:25 <ihrachys> huh?
16:02:25 <mlavalle> \o/
16:02:34 <haleyb> ihrachys: there are some that have no data, was told due to no failures (yet)
16:02:40 <ihrachys> oh ok
16:02:50 <haleyb> http://grafana.openstack.org/dashboard/db/neutron-failure-rate ?
16:03:07 <ihrachys> yeah, that one. I was misled by empty boards
16:03:30 <ihrachys> thanks for getting it done!
16:03:32 <haleyb> i put a note at the top since i was confused as well, but graphite data seems to match
16:03:49 <haleyb> if we see a failure but don't see it in the boards then we have a problem
16:04:39 <ihrachys> next AI was "ihrachys to follow up with Chandan on progress to split off tempest plugin"
16:04:40 <jlibosva> does it mean it can't show 0 with zuulv3?
16:05:22 <jlibosva> weird that periodic functional works even without failure
16:06:17 <haleyb> jlibosva: i think it can, just that it needs a FAILURE stat to do the math, and it's not present
16:06:30 <jlibosva> ok, I trust you :)
16:07:02 <ihrachys> ok. back to tempest plugin
16:07:03 <haleyb> i was told if i found a discrepancy to ping infra, that's all i nkow
16:07:04 <jlibosva> sorry for noise, go on
16:07:34 <ihrachys> I reached out to Chandar, and he told me he is going to work on it asap. that being said, I haven't seen any updates in gerrit since then
16:08:20 <ihrachys> tempest plugin: https://review.openstack.org/#/c/506553/ and project-config: https://review.openstack.org/#/c/507038/
16:08:31 <ihrachys> the latter was abandoned because the job will now live in another repo
16:09:06 <ihrachys> I guess I will need to check with him again
16:09:18 <ihrachys> at least we could speed up the tempest repo change
16:09:21 <mlavalle> I haven't seen updates in gerrit either
16:09:30 <ihrachys> #action ihrachys to follow up with Chandan about tempest split again
16:09:46 <ihrachys> those are all AIs we had
16:09:49 <ihrachys> #topic Grafana
16:09:52 <ihrachys> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:10:01 <ihrachys> now that we have the board, let's have a look
16:10:09 <mlavalle> are we going to talk about job migration later?
16:10:27 <ihrachys> mlavalle, yeah I suppose
16:10:43 <mlavalle> I can cover it during open topics if you want
16:10:51 <mlavalle> carry on
16:11:18 <ihrachys> fullstack and scenarios seem to be the old close-to-100%-failure fellas we all love
16:11:45 <ihrachys> there is also high failure rate on dvr-ha and ovsfw jobs but I guess we will let them slide for now
16:12:09 <ihrachys> #topic Fullstack
16:12:48 <ihrachys> I believe we still have two major issues there: trunk cleanup and rootwrap returning wrong data from other commands
16:13:07 <ihrachys> for the former, slaweq has this: https://review.openstack.org/#/c/514586/
16:13:25 <jlibosva> slaweq couldn't attend this meeting so I'm forwarding his message he's going to work on that one asap :)
16:13:55 <ihrachys> ok. do we have final decision of the path forward?
16:14:06 <ihrachys> I remember there was disagreement the last time
16:14:23 <ihrachys> armax pushing for fullstack-patched executables and slaweq for config option
16:14:46 <jlibosva> but slaweq has a point that we'd need to run all patched services
16:15:05 <jlibosva> so I assume we'll go with the config option
16:15:10 <ihrachys> ok that's good.
16:15:17 <jlibosva> I'm also working slowly on the namespace isolation in parallel so then we can remove the config option
16:15:54 <ihrachys> for the other issue - rootwrap daemon mixing outputs of commands - we have this attempt from toshii: https://review.openstack.org/#/c/514547/
16:16:44 <ihrachys> I think the direction is correct. there were some smaller questions about whether we can avoid eventlet dependency somehow
16:18:33 <ihrachys> if someone can have another look, that would be great. also pulling someone from oslo at this point would probably help. I recollect they wanted to see a test case reproducing the issue to move forward.
16:18:38 <ihrachys> which we have now
16:18:48 <jlibosva> I'll have a look
16:19:25 <ihrachys> I guess we can focus on those issues for fullstack and revisit any other failures next time if we make progress
16:19:30 <ihrachys> #topic Scenarios
16:20:06 <ihrachys> a latest dvr flavor result: http://logs.openstack.org/31/513831/1/check/legacy-tempest-dsvm-neutron-dvr-multinode-scenario/24754c9/logs/testr_results.html.gz
16:20:32 <ihrachys> that's covered with https://bugs.launchpad.net/neutron/+bug/1717302
16:20:33 <openstack> Launchpad bug 1717302 in neutron "Tempest floatingip scenario tests failing on DVR Multinode setup with HA" [High,Confirmed]
16:20:49 <ihrachys> haleyb, I recollect l3 subteam was going to discuss that
16:21:13 <haleyb> ihrachys: we have, it's still on swami's plate
16:21:40 <haleyb> "we have our best people working on it" :)
16:22:03 * ihrachys feels relieved now
16:22:08 <mlavalle> swami will be in Sydney next week
16:22:25 * ihrachys stops feeling relieved
16:23:14 <ihrachys> as for linuxbridge job, here is another result: http://logs.openstack.org/31/513831/1/check/legacy-tempest-dsvm-neutron-scenario-linuxbridge/057e83a/logs/testr_results.html.gz
16:23:42 <ihrachys> "Cannot 'detach_interface' instance 7722f257-3946-4a05-baee-4b69739d6547 while it is in vm_state building" in a bunch of failures
16:24:43 <ihrachys> nova compute log is quite red: http://logs.openstack.org/31/513831/1/check/legacy-tempest-dsvm-neutron-scenario-linuxbridge/057e83a/logs/screen-n-cpu.txt.gz?level=TRACE
16:24:51 <ihrachys> but all seems related to VIF provisioning
16:26:30 <ihrachys> oh in q-agt, I see this: "IpTablesApplyException: IPTables Rules did not converge"
16:26:40 <ihrachys> I believe it's https://bugs.launchpad.net/neutron/+bug/1719711
16:26:41 <openstack> Launchpad bug 1719711 in neutron "iptables failed to apply when binding a port with AGENT.debug_iptables_rules enabled" [High,Confirmed] - Assigned to Brian Haley (brian-haley)
16:26:54 <ihrachys> haleyb, any updates on it?
16:27:23 <haleyb> ihrachys: i am still looking at that one, had it reproducing then lost power and now it doesn't happen locally
16:28:11 <ihrachys> have you tried turning it off and on again? (c)
16:28:48 <haleyb> :)
16:29:05 <ihrachys> ok, seems like both scenario issues have best people working on them
16:29:06 <mlavalle> yes, that usually helps
16:29:32 <ihrachys> #topic zuulv3 job migration
16:29:36 <ihrachys> mlavalle, your floor
16:30:20 <mlavalle> submiited neutron-dsvm-api move to the Neutron tree:
16:30:36 <mlavalle> #link https://review.openstack.org/#/c/516715
16:30:57 <mlavalle> submitted removal of job definition to project-config:
16:31:24 <mlavalle> #link https://review.openstack.org/#/c/516724/
16:32:04 <mlavalle> and over the next few minutes I will submit a patch to openstac-zuul-jobs removing the old job from our project pipeline
16:32:21 <mlavalle> These are the steps prescribed by the zuul V3 migration guide
16:32:42 <mlavalle> so let's see if it works
16:32:44 <ihrachys> mlavalle, you may want to link patches with needed-by/depends-on for context
16:32:54 <ihrachys> I see andreas is puzzled
16:32:58 <mlavalle> ihrachys: yes, I will do that soon
16:33:32 <mlavalle> he was faster than me
16:33:45 <ihrachys> why do we need two infra patches to replace a job?
16:33:54 <ihrachys> isn't pipeline definition in a single place?
16:34:03 <mlavalle> apparently not
16:34:11 <mlavalle> I'm following the guide
16:34:21 <ihrachys> hm ok, weird :)
16:34:43 <ihrachys> so once we have api job settled, what's next?
16:34:44 <mlavalle> I guess the first patch removes the defintion when they migrated the job
16:35:25 <mlavalle> and then we take step 2
16:35:41 <mlavalle> which is tweking the moved job to take advantage of zuul v3
16:35:48 <mlavalle> hasn't gotten there yet
16:36:24 <ihrachys> advantage? what do you mean? reworking the script triggered in some way?
16:36:44 <mlavalle> well zuul v3 is supposed to have new features
16:36:55 <mlavalle> I don't know if we can exploiut any of that
16:36:59 <mlavalle> but we have to explore
16:37:31 <mlavalle> "Rework the jobs to be native v3 jobs"
16:37:40 <ihrachys> it may take some time
16:37:56 <ihrachys> if I were you, I would first get all jobs in tree then rework them
16:38:06 <mlavalle> https://docs.openstack.org/infra/manual/zuulv3.html#reworking-legacy-jobs-to-be-v3-native
16:38:20 <mlavalle> good advice
16:38:38 <ihrachys> on similar note, the new tempest plugin repo, it will share jobs with neutron
16:38:48 <ihrachys> because they will now need to cross-gate
16:39:05 <ihrachys> I believe those jobs will live in neutron tree?
16:39:08 <ihrachys> or tempest plugin?
16:39:25 <mlavalle> I am moving jobs to the Neutron tree
16:40:07 <ihrachys> but those are gating neutron only, no?
16:40:17 <mlavalle> yes
16:40:20 <ihrachys> the case with tempest plugin jobs is different in that they belong to neutron and the new repo
16:40:51 <mlavalle> I will have to take a look at that
16:43:23 <ihrachys> ok
16:43:29 <ihrachys> #topic Open discussion
16:43:34 <ihrachys> anything else to discuss?
16:44:20 <ihrachys> I guess no!
16:44:29 <ihrachys> thanks everyone fellas
16:44:32 <ihrachys> #endmeeting