16:01:21 <ihrachys> #startmeeting neutron_ci
16:01:22 <openstack> Meeting started Tue Dec 19 16:01:21 2017 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:26 <openstack> The meeting name has been set to 'neutron_ci'
16:01:29 <mlavalle> o/
16:01:39 <ihrachys> hi
16:01:42 <ihrachys> #topic Actions from prev meeting
16:01:44 <haleyb> hi
16:01:50 <ihrachys> "jlibosva to post wishlist bugs for fullstack improvements (reuse db dump; reuse env per class)"
16:01:55 <jlibosva> hi
16:01:58 <jlibosva> I did a few minutes ago
16:02:14 <ihrachys> haha
16:02:16 <ihrachys> links?
16:02:29 <jlibosva> https://bugs.launchpad.net/neutron/+bug/1739075
16:02:29 <openstack> Launchpad bug 1739075 in neutron "fullstack: Improve test suite by creating environment per test class" [Wishlist,New]
16:02:34 <jlibosva> https://bugs.launchpad.net/neutron/+bug/1739078
16:02:34 <openstack> Launchpad bug 1739078 in neutron "fullstack: Use a pre-built database schema" [Wishlist,New]
16:02:35 <slaweq> hi
16:03:16 <ihrachys> jlibosva, is 'rfe' tagneeded?
16:03:33 <jlibosva> I did tag it as rfe since it's an enhancement and not a real bug
16:03:42 <ihrachys> we have Wishlist for that alredy
16:03:45 <jlibosva> but dunno :)
16:03:50 <ihrachys> 'rfe' would trigger drivers discussion
16:03:51 <jlibosva> ok, I'll remove the rfe tags
16:03:55 <ihrachys> +
16:04:04 <mlavalle> and we want to avoid those guys
16:04:15 <ihrachys> yeah they are the worst
16:04:19 <mlavalle> LOL
16:04:25 <jlibosva> removed :)
16:04:37 <ihrachys> ok thanks for the reports! moving on
16:04:48 <ihrachys> "haleyb to post patch disabling east-west tests"
16:05:23 <haleyb> https://review.openstack.org/#/c/527992/ merged
16:05:28 <ihrachys> great
16:05:36 <haleyb> there is another change that seems related
16:05:50 <haleyb> https://review.openstack.org/#/c/528795/
16:06:17 <ihrachys> is it the reason why dvr flavor failed on setup?
16:06:47 <haleyb> yes it could be, i just noticed this morning so haven't looked closely
16:07:22 <ihrachys> I think it is. I saw dvr job failing after ~20mins of setup without triggering tempest
16:07:35 <ihrachys> since the patch passes dvr job it should be ok
16:07:41 <ihrachys> I +2d, thanks for the link
16:07:42 <haleyb> frickler: ^^ ?
16:08:23 <frickler> yes, that was the latest reason I saw for dvr failure
16:08:35 <ihrachys> thanks frickler !
16:08:49 <ihrachys> #topic Tempest plugin
16:08:59 <ihrachys> https://etherpad.openstack.org/p/neutron-tempest-plugin-job-move
16:09:25 <ihrachys> there is still some cleanup work to do
16:09:41 <ihrachys> though I saw Chandan posted a governance patch marking it as completed here: https://review.openstack.org/#/c/524605/2
16:09:51 <ihrachys> I think we should first clean up than claim victory
16:10:12 <frickler> I have a question there related to stable branches
16:10:29 <ihrachys> shoot
16:10:42 <frickler> the jobs using tempest plugin are failing on stable branches for neutron-dynamic-routing
16:11:08 <frickler> is the intention to use tests from neutron repo instead for that instead?
16:11:09 <ihrachys> I don't think they are supposed to be used for any stable branches
16:11:18 <ihrachys> yes, new repo = queens+
16:11:22 <frickler> e.g. https://review.openstack.org/528990
16:11:57 <frickler> so I'd cancel these cherry-picks and do unmodified backports of the "original" legacy jobs?
16:12:18 <ihrachys> yeah. for stable you just move legacy jobs as is, using in-tree tests
16:13:24 <frickler> k
16:14:50 <ihrachys> I remember Chandan said he will update subproject patches consuming new repo but I don't see updates except for midonet that is all red anyway: https://review.openstack.org/#/c/521342/
16:16:37 <ihrachys> what we need to do with all subprojects is to switch to new paths and install plugin using devstack plugin that was merged in https://review.openstack.org/#/c/526044/
16:16:57 <ihrachys> should we ask project owners to take over those patches?
16:17:12 <mlavalle> that's a good idea
16:17:15 <ihrachys> we have midonet, vpnaas and dynamic-routing that are stadium projects
16:17:34 <ihrachys> it looks like frickler drives d-r
16:17:46 <mlavalle> with midonet, yamamoto can help
16:18:00 <mlavalle> with dynamic-routing garyk can help
16:18:03 <ihrachys> who's vpnaas point of contact
16:18:18 <mlavalle> hoangcx
16:18:54 <mlavalle> hoangcx_
16:19:07 <ihrachys> mlavalle, would you mind if I ask you to reach out to yamamoto and hoangcx about those two patches? https://review.openstack.org/#/c/521341/ and https://review.openstack.org/#/c/521342/
16:19:17 <frickler> mlavalle: do you need help with https://review.openstack.org/522931? master is working, so I think having this new patch for stable/pike seems the right approach
16:19:17 <mlavalle> not a problem
16:19:59 <ihrachys> frickler, I believe that's another topic we should discuss now. mlavalle had master patch doing the same that we may want to move forward first.
16:20:14 <ihrachys> this: https://review.openstack.org/#/c/525345/
16:20:26 <mlavalle> yes
16:20:37 <mlavalle> once that merges, I'll take care of the cherrypicks
16:21:04 <mlavalle> https://review.openstack.org/522931 should be abandoned
16:21:20 <frickler> ah, I hadn't seen that one, o.k.
16:21:42 <ihrachys> mlavalle, do you have a patch ready that rips of duplicate jobs from gate?
16:22:00 <mlavalle> ihrachys: no, I will do that next
16:22:06 <ihrachys> I think we should prioritize it right now because x2 number of jobs => unstable gate
16:22:14 <mlavalle> yeah
16:22:15 <ihrachys> ok cool
16:22:29 <ihrachys> #action mlavalle to send patch(es) removing duplicate jobs from neutron gate
16:22:52 <frickler> if you ping me once they are ready, I can push them on the infra side
16:22:56 <ihrachys> you already have https://review.openstack.org/#/c/516724/ for -api so maybe makes sense to reuse it
16:23:03 <mlavalle> frickler: thanks. will do
16:23:17 <mlavalle> ihrachys: yes, I will expand that
16:23:45 <ihrachys> ok cool
16:24:22 <ihrachys> and also once we migrate subprojects to new imports we should be able to kill tempest remainings in neutron tree (the base class etc.)
16:24:44 <ihrachys> seems like that's all there is for migration
16:24:45 <frickler> a bit related: the neutron-api test on devstack-gate currently is doing nothing because the tests it tries to run are removed from neutron repo
16:24:55 <frickler> this is to use the plugin job instead: https://review.openstack.org/529000
16:25:15 <ihrachys> frickler, yeah good catch
16:26:31 <ihrachys> frickler, gerrit suggests it needs a respin
16:26:36 <ihrachys> not sure if that's correct though
16:27:23 <ihrachys> ok now let's move from migration issues to looking at new patches for the plugin
16:27:24 <ihrachys> https://review.openstack.org/#/q/project:openstack/neutron-tempest-plugin+status:open
16:27:34 <chandankumar> ihrachys: hello
16:27:53 <frickler> ihrachys: yes, I'll rebase
16:28:25 <ihrachys> chandankumar, heya, we were discussing remaining migration bits for new tempest plugin for neutron. we decided that subproject owners will take over patches you have for vpnaas, midonet, and dynamic-routing
16:28:39 <chandankumar> ihrachys: sure
16:28:50 <chandankumar> ihrachys: but i have one questions
16:29:18 <ihrachys> shoot
16:29:29 <chandankumar> other neutron dependent projects have also their intree tempest plugins are they also moving to new repo or merging in the same neutron tempest plugin?
16:30:07 <ihrachys> each repo may have its own plugin, it's ok
16:30:30 <ihrachys> but for stadium projects, we may consider adopting their tests too
16:30:34 <ihrachys> mlavalle, what do you think?
16:30:39 <chandankumar> then it will be so many plugins repo again
16:30:52 <chandankumar> ihrachys: i was thinking to merge all in the neutron tempest plugin
16:31:03 <chandankumar> one tempest plugin to rule them all
16:31:09 <mlavalle> yeah, I also thin we can adopt their tests
16:31:22 <mlavalle> bcafarel has been asking about that
16:31:29 <chandankumar> ihrachys: for telemetry projects i have merged aodh, ceilometer, panko in one project telemetry-tempest-plugin
16:32:46 <ihrachys> I think it's a matter of will and effort. from my perspective it makes sense that stadium extensions are part of neutron api and hence covered by unified plugin.
16:32:56 <ihrachys> but it's probably a question for drivers
16:33:14 <ihrachys> because it involves definition of neutron api and stadium structure etc.
16:33:45 <mlavalle> we can discuss in the next drivers meeting
16:33:48 <ihrachys> +
16:33:59 <chandankumar> ihrachys: l2gw tests are moved to a new repo tempest plugin as it does not comes under neutron deliverables
16:34:10 <ihrachys> #action mlavalle to report back about result of drivers discussion on unified tempest plugin for all stadium projects
16:34:33 <ihrachys> chandankumar, yes, anything not in stadium is out of discussion :)
16:34:49 <mlavalle> l2gw won't re-join the stadium
16:35:00 <mlavalle> I had a conversation about it in Sydney
16:35:32 <ihrachys> ok back to plugin patches...
16:35:49 <ihrachys> let's look at the patch adding dns test: https://review.openstack.org/#/c/520233/
16:36:09 <chandankumar> https://review.openstack.org/#/c/510525/ was a patch to move l2gw under stadium but it got abandoned
16:36:11 <ihrachys> I had a concern there about us importing code from designate plugin here: https://review.openstack.org/#/c/520233/11/neutron_tempest_plugin/scenario/test_dns_integration.py@36
16:36:30 <ihrachys> this is needed to get access to all the helpers that inspect dns zones
16:37:04 <ihrachys> I suggested to maybe keep the test inside designate plugin but frickler replied that the functionality under test is in neutron repo
16:37:34 <ihrachys> so I guess we are good to go with it? of course assuming that designate tempest plugin changes will be gated against the job right?
16:37:45 <mlavalle> I am good with it
16:37:57 <frickler> there's a patch up for that already: https://review.openstack.org/522950
16:38:36 <ihrachys> frickler, it's not voting. is it going to change? or you think it's not a risk?
16:38:59 <frickler> ihrachys: well, it is not voting yet on the neutron side either
16:39:03 <ihrachys> ah right
16:39:22 <ihrachys> so it will inherit once we flip in neutron?
16:39:33 <frickler> but I'd envision changing both, once it proves stable, yes
16:40:00 <mlavalle> we talked about that a few weeks ago, didn't we?
16:40:29 <ihrachys> ok. one risk is that currently designate job is not in neutron gate at all. so we may break the dns driver and in result block your gate and plugin gate
16:40:44 <ihrachys> mlavalle, we haven't discussed the tempest cross-import details
16:41:02 <mlavalle> ok
16:41:12 <haleyb> we will also need a dashboard update right?
16:41:28 <ihrachys> haleyb, yeah true
16:41:53 <ihrachys> #action frickler to post patch updating neutron grafana board to include designate scenario job
16:42:01 <frickler> yes, I can do that
16:42:13 <ihrachys> anyway, as for the test itself, I think it's ok to land. we can revisit voting later.
16:43:41 <ihrachys> ok other patches in the queue just need reviewer love but seem innocent to skip them
16:43:47 <ihrachys> #topic Grafana
16:43:52 <ihrachys> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:44:15 <ihrachys> we have dvr-scenario at 100% but it should resolve with brian's and frickler's patches
16:44:30 <ihrachys> I am eager to see result after both land
16:45:04 <ihrachys> besides that, there is linuxbridge scenario and fullstack
16:45:09 <jlibosva> I saw another failure in dvr-scenario related to trunk
16:45:25 <jlibosva> I think it might be a race between ovs firewall and trunk handler. I need to investigate further
16:45:37 <ihrachys> jlibosva, how did you see it if it's supposedly broken? do you have a link to a bug?
16:45:47 * mlavalle stepping away from keyboard for a few minutes
16:46:22 <ihrachys> I guess it's not 100% failure because of devstack-gate issue
16:46:29 <jlibosva> I saw it here http://logs.openstack.org/92/527992/2/check/neutron-tempest-plugin-dvr-multinode-scenario/166eee3/logs/testr_results.html.gz
16:47:15 <ihrachys> same test fails in slightly different but similar way for linuxbridge too: http://logs.openstack.org/88/526488/5/check/neutron-tempest-plugin-scenario-linuxbridge/66f8835/logs/testr_results.html.gz
16:47:27 <frickler> yes, I saw that a couple of times, too. maybe also mark that one as unstable?
16:47:51 <jlibosva> the dvr one fails with SSH failure probably because of an issue in trunk plugging
16:47:53 <ihrachys> we first need to make sure there is a bug report for the failure
16:48:12 <ihrachys> there is https://bugs.launchpad.net/neutron/+bug/1722644 for dvr job
16:48:12 <openstack> Launchpad bug 1722644 in neutron "TrunkTest fails for OVS/DVR scenario job" [High,Confirmed]
16:48:27 <ihrachys> though the error message is different there
16:48:33 <ihrachys> I haven't seen that one for a while
16:48:50 <ihrachys> so we probably should close it and create a new one
16:48:59 <jlibosva> this is my suspected culprit http://logs.openstack.org/92/527992/2/check/neutron-tempest-plugin-dvr-multinode-scenario/166eee3/logs/subnode-2/screen-q-agt.txt.gz#_Dec_14_18_31_25_801432
16:49:09 <jlibosva> I'll do that
16:49:48 <ihrachys> #action jlibosva to close bug 1722644 and open a new one for trunk connectivity failures in dvr and linuxbridge scenario jobs
16:49:50 <openstack> bug 1722644 in neutron "TrunkTest fails for OVS/DVR scenario job" [High,Confirmed] https://launchpad.net/bugs/1722644
16:49:50 <ihrachys> jlibosva, thanks!
16:50:20 <ihrachys> jlibosva, this seems ovs specific, so maybe two bugs for dvr and linuxbridge respectively
16:50:41 <jlibosva> I see
16:51:09 <ihrachys> jlibosva, you will post a disabling patch right?
16:51:30 <jlibosva> ok
16:52:03 <ihrachys> #action jlibosva to disable trunk scenario connectivity tests
16:53:02 * mlavalle back
16:53:18 <ihrachys> another failure in linuxbridge is test_ip_prefix_negative here: http://logs.openstack.org/88/526488/5/check/neutron-tempest-plugin-scenario-linuxbridge/66f8835/logs/testr_results.html.gz
16:53:24 <ihrachys> I don't think we have a bug for that reported either
16:53:54 <slaweq> I think that my patch: https://review.openstack.org/#/c/527965/ fixes this one
16:54:05 <frickler> https://bugs.launchpad.net/neutron/+bug/1736674
16:54:05 <openstack> Launchpad bug 1736674 in neutron "sg rules are sometimes not applied" [High,In progress] - Assigned to Slawek Kaplonski (slaweq)
16:54:17 <slaweq> frickler: thx, I just wanted to post link to it :)
16:54:33 <ihrachys> riiight
16:55:10 <ihrachys> ok let's review it then :)
16:55:15 <slaweq> thx
16:55:18 <ihrachys> I see linuxbridge is green for this patch, good ;)
16:55:25 <ihrachys> #topic Fullstack
16:55:36 <ihrachys> we have little time so let's review it quick
16:55:45 <ihrachys> fullstack job is same 80% as it was for a while
16:55:59 <ihrachys> example: http://logs.openstack.org/65/527965/4/check/legacy-neutron-dsvm-fullstack/12b03b9/logs/testr_results.html.gz
16:56:07 <slaweq> recently I found new issue on fullstack: https://bugs.launchpad.net/neutron/+bug/1737892
16:56:07 <openstack> Launchpad bug 1737892 in neutron "Fullstack test test_qos.TestBwLimitQoSOvs.test_bw_limit_qos_port_removed failing many times" [High,Confirmed]
16:56:16 <slaweq> but I haven't got time to check it yet
16:57:06 <ihrachys> ack
16:57:22 <ihrachys> not enough details to suggest anything there yet. I assume agent log is clean?
16:57:38 <slaweq> to be honest I didn't check it yet
16:57:52 <slaweq> I just reported issue on launchpad
16:58:12 <slaweq> but I will try to debug it ASAP if anyone else will not take it :)
16:58:48 <ihrachys> I bet no one will
16:58:50 <ihrachys> :)
16:59:06 <slaweq> yes, so I will try to debug it :)
16:59:14 <ihrachys> ok there is little time, but I will take it on myself to report the security group failure
16:59:18 <slaweq> it's qos and fullstack so it's what I like most :D
16:59:26 <ihrachys> #action ihrachys to report sec group fullstack failure
16:59:40 <mlavalle> no meeting next week?
16:59:57 <slaweq> ihrachys: You can add action for me that I will check this issue with qos and fullstack
16:59:57 <ihrachys> slaweq, your desires are unconventional ;)
17:00:09 <ihrachys> mlavalle, yeah absolutely
17:00:19 <slaweq> :)
17:00:26 <mlavalle> meaning no meeting?
17:00:36 <ihrachys> #action slaweq to debug qos fullstack failure https://bugs.launchpad.net/neutron/+bug/1737892
17:00:36 <openstack> Launchpad bug 1737892 in neutron "Fullstack test test_qos.TestBwLimitQoSOvs.test_bw_limit_qos_port_removed failing many times" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)
17:00:52 <ihrachys> mlavalle, sure. we will have shutdown as expected
17:00:58 <mlavalle> cool
17:01:05 <ihrachys> next time Jan 2 right?\
17:01:25 <ihrachys> I will send an email, I hope 1/2 is fine with most
17:01:40 <ihrachys> we are out of time, thanks everyone for all the hard work!
17:01:41 <ihrachys> #endmeeting