16:01:08 <ihrachys> #startmeeting neutron_ci
16:01:11 <openstack> Meeting started Tue Apr 18 16:01:08 2017 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:11 <jlibosva> o/
16:01:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:14 <openstack> The meeting name has been set to 'neutron_ci'
16:01:19 <haleyb> hi
16:01:52 <ihrachys> hi everyone
16:02:03 <ihrachys> #topic Action items from prev meeting
16:02:15 <ihrachys> "ihrachys to report bugs for fullstack race in ovs agent when calling to enable_connection_uri"
16:02:20 <ihrachys> I did, just need to find a link
16:03:26 <ihrachys> https://bugs.launchpad.net/neutron/+bug/1682202
16:03:27 <openstack> Launchpad bug 1681945 in neutron "duplicate for #1682202 Neutron Agent error "constraint violation"" [Undecided,Confirmed]
16:03:43 <ihrachys> oh and I see it's a dup :)
16:03:54 <ihrachys> the right one is then https://bugs.launchpad.net/neutron/+bug/1681945
16:03:56 <openstack> Launchpad bug 1681945 in neutron "Neutron Agent error "constraint violation"" [Undecided,Confirmed]
16:04:31 <ihrachys> otherwiseguy, were there any plans/patches to make ovs agent gracefully handle existing manager?
16:05:51 <ihrachys> ok, I guess he is not avail
16:06:06 <ihrachys> so folks, if someone wants to pick it up, you're welcome
16:06:09 <otherwiseguy> ihrachys, what do you mean?
16:06:17 <ihrachys> otherwiseguy, hi! https://bugs.launchpad.net/neutron/+bug/1681945
16:06:18 <openstack> Launchpad bug 1681945 in neutron "Neutron Agent error "constraint violation"" [Undecided,Confirmed]
16:06:33 <otherwiseguy> That *is* graceful.
16:06:34 <ihrachys> multiple agents trying to add same manager, clashing on constraint
16:06:40 <otherwiseguy> It is supposed to fail.
16:06:43 <ihrachys> it says ERROR
16:06:47 <ihrachys> it shouldn't
16:07:22 <ihrachys> otherwiseguy, does it bubble up anywhere, or it's contained?
16:07:57 <otherwiseguy> Probably possible to adust the log message by doing log_errors=False, but check_error=False so no exception raised.
16:08:18 <ihrachys> ok makes sense
16:08:30 <ihrachys> so it should probably be Low and not gate-failure tagged
16:08:39 <otherwiseguy> In truth, I'd rather we didn't even attempt it. But, that's just my preference in general--let installers do the right thing. :p
16:09:24 <otherwiseguy> I can also see the value in less ideological approaches. :)
16:09:40 <otherwiseguy> But yes, not a gate failure. Just a logging issue.
16:09:55 <ihrachys> otherwiseguy, thanks for clarification, I updated the bug so that it's not tracked as CI issue
16:10:22 <ihrachys> next item was "jlibosva to follow up on py3 plan for pike"
16:10:29 <jlibosva> argh
16:10:34 <ihrachys> we have this ether: https://etherpad.openstack.org/p/py3-neutron-pike
16:10:40 <jlibosva> I didn't get any time though I saw kevinbenton reporting a tracker bug
16:10:57 <ihrachys> and I think kevinbenton acknowledged the goal at https://review.openstack.org/#/c/457200/
16:11:44 <ihrachys> jlibosva, do you think you will have time for that next weeks, or we should find someone to take over?
16:12:23 <jlibosva> ihrachys: I will sync with kevinbenton, I hope I'll have time. I forgot that I was working only 3 out of 5 working days since last mtg
16:12:48 <ihrachys> ok makes sense. keep us in the loop.
16:13:08 <ihrachys> we will revisit the goal later in the run but for the time being it seems like we should not raise it every mtg :)
16:13:18 <ihrachys> next was "ihrachys to update wiki with the link to gerrit CI dashboard"
16:13:20 <ihrachys> I did
16:13:22 <ihrachys> moving on :)
16:13:28 <ihrachys> next was "jlibosva document current openvswitch requirements for fullstack/functional in TESTING.rst"
16:13:33 <ihrachys> jlibosva, I think there was a patch?
16:13:51 <jlibosva> yes, I sent something
16:13:54 <jlibosva> let me find link
16:13:54 <ihrachys> https://review.openstack.org/#/c/456219/
16:14:01 <ihrachys> merged, good
16:14:16 <jlibosva> https://review.openstack.org/#/c/456219/
16:14:18 <jlibosva> yep
16:14:38 <ihrachys> jlibosva, btw we also switched to ubuntu cloud archive in master
16:14:44 <ihrachys> jlibosva, does it change anything for us?
16:14:56 <ihrachys> I think we can e.g. enable the func tests?
16:14:58 <jlibosva> ihrachys: afaik it doesn't have the newer kernel, right?
16:15:32 <ihrachys> wasn't that an issue in userspace?
16:15:35 <ihrachys> appctl
16:16:12 <ihrachys> I talk about test_install_flood_to_tun
16:16:31 <jlibosva> ihrachys: it should be triggered automatically if the version meets the requirement
16:16:42 <ihrachys> oh right
16:16:43 <ihrachys> let's check
16:16:50 <jlibosva> ihrachys: yep, it passes
16:17:01 <jlibosva> e.g. here: http://logs.openstack.org/22/457422/1/check/gate-neutron-dsvm-functional-ubuntu-xenial/41dd5db/testr_results.html.gz
16:17:06 <jlibosva> OVSFlowTestCase
16:17:45 <ihrachys> yeah, executed
16:17:46 <ihrachys> nice
16:17:54 <ihrachys> then we should probably revise the doc again?
16:18:05 <jlibosva> yep :)
16:18:21 <ihrachys> should we leave the skip check, or we can also kill it now?
16:18:27 <jlibosva> I would leave it
16:18:39 <ihrachys> ok ok, maybe someone sticks to stock ubuntu
16:18:40 <jlibosva> it can produce false negative results to those who run with 2.5.0
16:18:45 <ihrachys> ok
16:18:48 <jlibosva> and the error is not in produciton, it's just the test
16:19:08 <ihrachys> #action jlibosva to revise TESTING doc to update about test_install_flood_to_tun gate status with UCA used for Pike
16:19:31 <ihrachys> next was "ihrachys to figure out why gerrit dashboard seems to not show some gate-failure fixes"
16:19:34 <ihrachys> I did not :-x
16:19:54 <ihrachys> lemme follow up after the mtg
16:19:57 <ihrachys> #action ihrachys to figure out why gerrit dashboard seems to not show some gate-failure fixes
16:20:19 <ihrachys> last item was "ihrachys to review fullstack and scenario health before next meeting"
16:20:51 <ihrachys> I checked fullstack, it's 100% failure mode, example http://logs.openstack.org/65/404265/15/check/gate-neutron-dsvm-fullstack-ubuntu-xenial/a48bd9f/testr_results.html.gz
16:21:18 <ihrachys> it seems like the same issue for all failures
16:21:23 <ihrachys> some connectivity breakage
16:21:43 <ihrachys> we need a volunteer to report a bug for that and ideally dig it to a fix
16:21:48 <ihrachys> but at least a bug report
16:23:00 <ihrachys> I see lots of volunteers, but I will probably take it myself!
16:23:11 <ihrachys> #action ihrachys to report a bug for fullstack connectivity failures
16:23:27 <ihrachys> as for scenarios, they are not 100% but close.
16:23:36 <ihrachys> both flavours
16:24:16 <ihrachys> an example of the failure
16:24:18 <ihrachys> http://logs.openstack.org/65/404265/15/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/8f457d5/logs/testr_results.html.gz
16:24:32 <ihrachys> it seems like it's always the same test (test_subport_connectivity for trunks)
16:24:51 <ihrachys> jlibosva, I think you were looking at that one in the past. any lead?
16:24:52 <jlibosva> it seems that vlan interface doesn't get any address
16:25:45 <ihrachys> or could it be that the timed out command breaks connectivity?
16:25:46 <jlibosva> it could be caused by https://bugs.launchpad.net/neutron/+bug/1626010
16:25:47 <openstack> Launchpad bug 1626010 in neutron "OVS Firewall cannot handle non unique MACs" [High,In progress] - Assigned to Thomas Morin (tmmorin-orange)
16:26:07 <jlibosva> I have a fix here: https://review.openstack.org/#/c/385085/
16:26:42 <jlibosva> there is still one scenario that ovs fw won't work with vlan and local network types
16:26:43 <ihrachys> jlibosva, I see the test failing with the patch
16:27:52 <jlibosva> ehm
16:28:03 * jlibosva scratches his head
16:28:09 <jlibosva> it shouldn't be :)
16:28:20 <ihrachys> bad, bad test!
16:28:39 <ihrachys> we probably should start at least with reporting a bug for the failure
16:28:47 <jlibosva> I'll have a closer look
16:28:53 <ihrachys> jlibosva, can you take that?
16:28:59 <jlibosva> I will
16:29:14 <ihrachys> #action jlibosva to report a bug for scenario failures
16:29:15 <ihrachys> thanks
16:29:42 <ihrachys> ok we can now take a look at patches in review
16:29:44 <ihrachys> #topic #topic Patches in review
16:29:46 <ihrachys> #topic Patches in review
16:30:41 <ihrachys> the review dashboard has an issue that I need to look at (discussed before), so I am gonna look through my queue that hopefully captures some fixes
16:31:24 <ihrachys> there were quite some gate bugs in last days, but most are squashed now
16:31:30 <ihrachys> most of the most pressing I mean
16:31:35 <ihrachys> one standing right now is https://bugs.launchpad.net/neutron/+bug/1679775
16:31:37 <openstack> Launchpad bug 1679775 in neutron "test_network_list_queries_constant fails with testtools.matchers._impl.MismatchError: 28 != 13" [Critical,In progress] - Assigned to Kevin Benton (kevinbenton)
16:31:54 <ihrachys> this should be solved by https://review.openstack.org/#/c/457515/
16:32:03 <ihrachys> it's a test only fix
16:32:10 <ihrachys> jlibosva, please have a look
16:32:55 <jlibosva> will do
16:33:47 <ihrachys> I am also cleaning up subunit forks that allowed to slip some test coverage degradation in branches: https://review.openstack.org/#/q/status:open++topic:remove-subunit-trace-fork
16:33:59 <ihrachys> some are stadium projects there so have a look
16:34:40 <ihrachys> we will also need to land https://review.openstack.org/#/q/Icef59a03184473476e0657334bcc30dc7bf4f9ff,n,z to fix periodic neutron-lib jobs for stadium projects
16:36:10 <ihrachys> there is https://review.openstack.org/447781 but I need to have a look why it still fails.
16:37:10 <ihrachys> I know that jlibosva was working on fixing dhcp agent deployment for fullstack at https://review.openstack.org/#/c/455790/
16:37:39 <jlibosva> this one doesn't cause much failures
16:37:57 <ihrachys> yeah, but we need that to unblock work on provisioning blocks
16:38:11 <jlibosva> yep
16:39:29 <ihrachys> I am not aware of other pressing fixes
16:39:49 <ihrachys> I gotta do another cleanup round of gate-failure tagged bugs
16:39:59 <ihrachys> #action ihrachys to clean up gate-failure tagged bugs list
16:40:08 <ihrachys> seems like some should now go since we landed fixes
16:40:31 <ihrachys> also seems like kevinbenton has a lead on linuxbridge multinode instability: https://bugs.launchpad.net/neutron/+bug/1683256
16:40:33 <openstack> Launchpad bug 1683256 in neutron "linuxbridge multinode depending on multicast support of provider" [Critical,New]
16:40:37 <ihrachys> (no patch so far)
16:41:26 <ihrachys> any other patches you are aware that may need attention?
16:41:47 <jlibosva> not from me
16:41:50 <ihrachys> ok
16:41:56 <ihrachys> #topic Grafana
16:42:08 <ihrachys> nothing major there that we haven't discussed
16:42:13 <ihrachys> except one thing that I wanted to run
16:42:27 <ihrachys> I see that functional job is now on par with e.g. -api job in terms of stability
16:42:29 <ihrachys> http://grafana.openstack.org/dashboard/db/neutron-failure-rate?panelId=7&fullscreen
16:42:34 <ihrachys> it leads it pretty close
16:42:48 <ihrachys> I wonder if we are close to the time where we could pull the trigger
16:43:14 <ihrachys> I see that at least 2 weeks in the past
16:43:59 <ihrachys> I am keen to just post a patch against project-config and spur discussion :)
16:44:14 <ihrachys> thoughts?
16:44:16 <jlibosva> there is small a functional peak yesterday
16:45:00 <ihrachys> yeah, I don't recollect what was that
16:45:08 <ihrachys> was busy with other gate breakages yesterday :)
16:45:13 <ihrachys> maybe I should look first
16:45:27 <ihrachys> #action ihrachys to look at what the functional spike yesterday was
16:45:28 <jlibosva> I agree with the patch tho :) we'll see what others say as I think this meeting attendance doesn't belong to the highest
16:45:50 <ihrachys> jlibosva, agreed. (it doesn't, but it helps to have it right?)
16:46:03 <ihrachys> #action ihrachys to propose voting for functional job
16:46:03 <jlibosva> sure, I'm just saying that we will get more voices there
16:46:25 <ihrachys> I will ofc WIP it and ask for comments during a team meeting and ML
16:46:54 <ihrachys> ok I have nothing else
16:46:58 <ihrachys> jlibosva, do you?
16:47:05 <jlibosva> nopes
16:47:10 <ihrachys> great, then ciao! :)
16:47:11 <ihrachys> #endmeeting