15:01:37 <slaweq> #startmeeting neutron_ci
15:01:37 <openstack> Meeting started Tue Apr 13 15:01:37 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:40 <openstack> The meeting name has been set to 'neutron_ci'
15:01:40 <slaweq> hi
15:01:41 <ralonsoh> hi
15:02:18 <bcafarel> o/
15:02:59 <slaweq> ping lajoskatona
15:03:48 <slaweq> ok, lets start
15:03:55 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:04:06 <slaweq> #topic Actions from previous meetings
15:04:13 <slaweq> ralonsoh to check failed qos scenario test
15:04:18 <lajoskatona> Hi, I am here
15:04:30 <ralonsoh> yes, I didn't replicate the error
15:04:36 <ralonsoh> and I didn't see it again
15:04:54 <ralonsoh> and reviewing the logs, I didn't see why the QoS was not applied
15:05:05 <ralonsoh> sorry, no solution for this one
15:05:07 <slaweq> ralonsoh: ok, so let's not bother with this for now
15:05:27 <slaweq> we will get back to it when it will be happening more often
15:05:32 <ralonsoh> ok
15:05:37 <slaweq> ralonsoh: thx for checking it
15:05:39 <slaweq> next one
15:05:41 <slaweq> bcafarel to update grafana dashboards with stable/wallaby
15:05:58 <bcafarel> sent and merged (I think), looking for the link
15:06:42 <bcafarel> yep https://review.opendev.org/c/openstack/project-config/+/786008 merged, also good time to ping other for release-related changes
15:07:06 <bcafarel> like https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/785092 (switch to -wallaby variant jobs)
15:07:08 <slaweq> thx bcafarel
15:08:02 <slaweq> ok, next one
15:08:04 <slaweq> ralonsoh to check failed test_get_egress_min_bw_for_port functional test
15:08:18 <ralonsoh> yes, one sec
15:08:33 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron/+/785158
15:08:47 <ralonsoh> this actually a pre-check that must be always true
15:09:00 <ralonsoh> this is not solving the error
15:10:03 <lajoskatona> If I understand well we will have clean feedback if there are policies on port
15:10:31 <ralonsoh> sorry, someone else pinged me
15:10:52 <ralonsoh> this patch will provide info if any other qos is set before the test
15:11:38 <slaweq> so this may help to understand what the issue is there
15:11:42 <ralonsoh> sure
15:11:49 <slaweq> which is good step forward IMO :)
15:11:51 <slaweq> thx ralonsoh
15:12:07 <slaweq> ok, and the last one
15:12:08 <slaweq> ralonsoh to update LB installation guide with info about legacy ebtables
15:12:12 <ralonsoh> done
15:12:15 <ralonsoh> one sec
15:12:49 <ralonsoh> I don't find it
15:13:02 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/785137
15:13:17 <slaweq> thx
15:13:21 <slaweq> it's merged already
15:13:51 <slaweq> and that were all action items from last week
15:13:57 <slaweq> let's move on
15:13:59 <slaweq> #topic Stadium projects
15:14:18 <slaweq> lajoskatona: any updates?
15:14:23 <slaweq> or issues
15:14:24 <lajoskatona> Nothing what I have seen, so seems to be quiet everywhere
15:14:38 <slaweq> ++
15:14:40 <slaweq> thx
15:15:58 <slaweq> #topic Stable branches
15:16:17 <slaweq> as bcafarel  already mentioned there are some patches https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/785092 and https://review.opendev.org/c/openstack/neutron/+/785093
15:16:23 <slaweq> which are for wallaby's ci
15:16:34 <slaweq> and I will need to do similar patches for stadium projects
15:16:45 <slaweq> I will do them this week
15:17:02 <slaweq> #action slaweq to update wallaby's scenario jobs in neutron-tempest-plugin
15:17:21 <slaweq> bcafarel: anything else regarding stable branches ci?
15:18:03 <bcafarel> recent branches look OK overall
15:18:14 <bcafarel> stein has https://bugs.launchpad.net/neutron/+bug/1923412 small fix needed
15:18:15 <openstack> Launchpad bug 1923412 in neutron "[stable/stein] Tempest fails with unrecognized arguments: --exclude-regex" [Critical,Triaged]
15:18:40 <bcafarel> rocky I wonder, it seems iptables_hybrid job is consistently failing (like on https://review.opendev.org/c/openstack/neutron/+/770794 )
15:18:54 <bcafarel> but I did not have time to check this week sorry :(
15:20:00 <ralonsoh> I'll try to check it
15:20:13 <slaweq> there are some failing tests there
15:20:24 <slaweq> but also it fails on collecting logs after job's failure
15:21:18 <slaweq> there is some error in subunit2html.py
15:21:37 <slaweq> I wonder when we will be able to EOL those branches which runs on python 2.7 still
15:21:50 <slaweq> and on Ubuntu Xenial
15:22:34 <slaweq> hmm
15:22:37 <slaweq> interesting
15:22:51 <slaweq> Ubuntu Xenial support ends in April 2021 according to https://lists.ubuntu.com/archives/ubuntu-announce/2016-April/000207.html
15:23:05 <slaweq> and we are still using it for testing Queens and Rocky
15:23:19 <ralonsoh> I think we can try to bump this library to 1.1.0
15:23:31 <ralonsoh> we are still using 1.0.0
15:23:37 <slaweq> sorry, wrong link, good one is https://wiki.ubuntu.com/Releases
15:24:04 <slaweq> ok, bcafarel can You report an issue regarding that rocky jobs?
15:24:28 <bcafarel> slaweq: sure will do later tonight (I will check on other backports too to confirm)
15:24:30 <slaweq> and I will send an email to the community to ask what we should do with that
15:24:37 <slaweq> thx bcafarel
15:24:50 <slaweq> #action bcafarel to report stable/rocky ci failures on LP
15:25:05 <slaweq> #action slaweq to send email about testing older branches on Ubuntu Xenial
15:25:23 <slaweq> ahh, wait
15:25:37 <slaweq> Xenial's standard support ends this month
15:25:42 <slaweq> but EOL will be in 2024
15:25:46 <slaweq> so we still can use it :)
15:25:52 <slaweq> I will not send any email about it
15:25:55 <slaweq> sorry for the noise
15:26:29 <slaweq> ok, I think we can move on
15:26:32 <slaweq> #topic Grafana
15:26:36 <slaweq> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:27:12 <slaweq> it seems we are better with functional job finally
15:27:23 <slaweq> but we have some issue with neutron-tempest-plugin-scenario jobs
15:27:43 <slaweq> as most of them failing at around 20-30 % of times
15:29:12 <slaweq> except that I think that things are pretty ok
15:30:09 <slaweq> let's talk about some specific issues
15:30:11 <slaweq> #topic fullstack/functional
15:30:24 <slaweq> I found (again) this week failure of     test_keepalived_spawns_conflicting_pid_vrrp_subprocess
15:30:28 <slaweq> https://7c4af2962e649c78a0b8-8dbbf393fa4fd2548b6eb8c45bf2a7e6.ssl.cf2.rackcdn.com/785011/1/check/neutron-functional-with-uwsgi/dc727d9/testr_results.html
15:30:38 <slaweq> but I think we marked those tests as unstable, didn't we?
15:30:45 <ralonsoh> yes, same problem as before
15:30:59 <ralonsoh> the sigterm signal do not reply
15:31:08 <ralonsoh> and the thread never ends
15:31:27 <slaweq> yes, but is the patch which markes that as unstable merged already?
15:31:32 <slaweq> do You remember that patch?
15:31:39 <ralonsoh> nope, I think we removed it
15:31:44 <ralonsoh> we'll need to propose it again
15:31:51 <slaweq> ahh, ok
15:32:08 <slaweq> maybe we should then mark them as unstable until we will fix that issue completly
15:32:31 <slaweq> ralonsoh: will You propose it?
15:32:32 <ralonsoh> perfect
15:32:34 <ralonsoh> yes
15:32:37 <slaweq> thx
15:32:54 <slaweq> #action ralonsoh to mark test_keepalived_spawns_conflicting_pid_vrrp_subprocess functional test as unstable
15:33:07 <slaweq> ok, regarding fullstack
15:33:19 <slaweq> I just want to ask You about review of the patch  https://review.opendev.org/c/openstack/neutron/+/783748
15:33:24 <slaweq> it's new fullstack test to add
15:33:29 <slaweq> for segmentation_id updates
15:33:41 <slaweq> no rush but if You will have few minutes, please take a look
15:33:53 <slaweq> next topic
15:33:55 <slaweq> #topic Tempest/Scenario
15:34:07 <slaweq> here there is most "hot" issue for this week
15:34:34 <slaweq> I noticed that in all our neutron-tempest-plugin scenario jobs random tests are failing due to ssh failure
15:34:48 <slaweq> every time there is problem with accessing metadata service
15:35:00 <slaweq> https://f237087178b608be7cbc-a3abe88e6ab2911093ccea19e77537d0.ssl.cf5.rackcdn.com/785749/1/check/neutron-tempest-plugin-scenario-linuxbridge/f746672/testr_results.html
15:35:02 <slaweq> https://5e09181bcc1a50499619-17764b56a5c622705c872e3c7dca2597.ssl.cf2.rackcdn.com/769788/6/check/neutron-tempest-plugin-scenario-openvswitch/32b4182/testr_results.html
15:35:04 <slaweq> https://f3719601e38156e4ec26-626e176c8fffa963d6c301ab5a223935.ssl.cf1.rackcdn.com/785633/2/check/neutron-tempest-plugin-scenario-linuxbridge/4fcd101/testr_results.html
15:35:06 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da7/785895/1/check/neutron-tempest-plugin-scenario-linuxbridge/da7ea5e/testr_results.html
15:35:09 <slaweq> https://5b963027deeaf11430aa-de753611fe18d17817cd3cc57c427456.ssl.cf1.rackcdn.com/785895/1/check/neutron-tempest-plugin-scenario-openvswitch/ce74a5b/testr_results.html
15:35:10 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_785/785889/3/check/neutron-tempest-plugin-scenario-openvswitch/785f8bb/testr_results.html
15:35:12 <slaweq> https://3acd25be4b1698636631-3e445c1a5b8a8c84f71b5a61cff5a09b.ssl.cf5.rackcdn.com/785950/1/check/neutron-tempest-plugin-scenario-linuxbridge/5c9ea3c/testr_results.html
15:35:14 <slaweq> those are examples of the failures
15:35:27 <slaweq> all those jobs runs with "ha=True" so keepalived is involved in the router's namespace
15:35:41 <slaweq> any volunteer to investigate it?
15:35:51 <ralonsoh> I could do it
15:36:24 <slaweq> I think that maybe https://review.opendev.org/c/openstack/neutron/+/782679 may be useful for debugging that issue
15:36:44 <slaweq> ahh, no
15:36:45 <slaweq> sorry
15:36:49 <slaweq> it's only for grenade jobs
15:37:01 <slaweq> we should do similar change for neutron-tempest-plugin jobs also
15:37:06 <ralonsoh> ok
15:37:15 <slaweq> ralonsoh: I will open LP for that issue
15:37:22 <ralonsoh> so you think this is a problem in the iptables rule?
15:37:23 <slaweq> I forgot to do it before the meeting
15:37:32 <slaweq> idk but maybe can be
15:37:39 <ralonsoh> ping me with the LP link
15:37:59 <slaweq> sure, I will
15:39:02 <slaweq> #action slaweq to report LP with metadata issue in scenario jobs
15:39:17 <slaweq> #ralonsoh to check metadata issue in scenario jobs
15:41:29 <slaweq> ok, that's all regarding scenario jobs for today
15:41:35 <slaweq> #topic Periodic
15:41:46 <slaweq> periodic jobs looks pretty ok recently
15:42:01 <slaweq> one thing I want to ask You is to review  https://review.opendev.org/c/openstack/neutron/+/785660
15:42:07 <ralonsoh> sure
15:42:21 <slaweq> as we recently discovered that periodic jobs based on tripleo are testing neutron Victoria every day
15:42:28 <slaweq> which basically doesn't makes any sense :)
15:42:30 <ralonsoh> ups...
15:42:47 <slaweq> so this patch ensures that neutron is from master branch in that jobs always
15:43:14 <slaweq> ok, that's all from my side
15:43:15 <bcafarel> he, nice catch
15:43:28 <slaweq> do You have anything else to discuss today?
15:43:49 <ralonsoh> no, thanks
15:44:18 <slaweq> ahh, one last thing - next week's meeting will be cancelled as we have ptg in the same time
15:44:38 <slaweq> ok, so if there are no other topics, we can finish earlier today
15:44:43 <slaweq> thx for attending the meeting
15:44:49 <slaweq> #endmeeting