15:00:48 <slaweq> #startmeeting neutron_ci
15:00:49 <openstack> Meeting started Tue Mar 30 15:00:48 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:52 <openstack> The meeting name has been set to 'neutron_ci'
15:01:28 <slaweq> hi
15:02:40 <slaweq> ralonsoh: lajoskatona bcafarel: ping
15:02:48 <ralonsoh> hi
15:02:51 <bcafarel> o/
15:03:01 <bcafarel> (sorry got lost in an ansible rabbit hole)
15:03:01 <ralonsoh> (I was in the wrong channel)
15:03:06 <lajoskatona> Hi
15:03:39 <slaweq> :)
15:03:43 <slaweq> ok, lets start
15:03:46 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:03:52 <slaweq> #topic Actions from previous meetings
15:03:59 <slaweq> slaweq to try to backport neutron-tempest-slow-py3 to stable/stein
15:04:03 <slaweq> Train: https://review.opendev.org/c/openstack/neutron/+/783022
15:04:06 <slaweq> Stein: https://review.opendev.org/c/openstack/neutron/+/783024
15:04:28 <slaweq> train already merged
15:04:36 <slaweq> and stein approved but got -2
15:04:41 <slaweq> so it's rechecked
15:04:47 <bcafarel> next recheck it will pass :)
15:04:52 <slaweq> I hope so :)
15:04:54 <slaweq> next one
15:04:56 <slaweq> ralonsoh to check failed qos scenario test
15:05:11 <ralonsoh> sorry, I didn't start
15:05:26 <slaweq> ok, I will move it for next week for You
15:05:28 <slaweq> fine?
15:05:29 <ralonsoh> sure
15:05:32 <slaweq> #action ralonsoh to check failed qos scenario test
15:05:35 <slaweq> thx
15:05:37 <slaweq> next topic
15:05:41 <slaweq> #topic Stadium projects
15:05:45 <slaweq> anything new?
15:07:19 <bcafarel> I think midonet was the only "hot topic"
15:07:29 <bcafarel> (from previous meeting)
15:07:29 <slaweq> but that one is already discussed
15:07:36 <slaweq> it's gate is totally broken now
15:07:54 <slaweq> and I don't think we really have resources here to fix it
15:08:34 <bcafarel> at least the rest of the stadium looks OK CI-wise
15:08:40 <slaweq> ++
15:08:43 <slaweq> thx
15:08:48 <slaweq> #topic Stable branches
15:08:59 <slaweq> we will need to update our stable branches dashboards
15:09:25 <slaweq> but I think we should wait a bit more, until wallaby will be fully done everywhere
15:09:35 <bcafarel> yes, and also it is better to wait for the jobs updates first
15:09:51 <bcafarel> neutron-tempest-plugin-wallaby, remove master jobs etc (where is that doc I updated on that topic?)
15:11:44 <bcafarel> https://docs.openstack.org/neutron/latest/contributor/policies/release-checklist.html
15:11:59 <slaweq> bcafarel: I just wanted to paste it
15:12:03 <slaweq> thx
15:12:48 <slaweq> regarding our currently working stable branches
15:12:56 <slaweq> is there anything to talk about today?
15:13:02 <slaweq> I think all is fine
15:13:10 <slaweq> but maybe I'm missing something
15:13:40 <bcafarel> overall good, I saw a few failures here and there but nothing bad
15:14:08 <slaweq> thx bcafarel
15:14:09 <bcafarel> and neutron-tempest-slow-py3 should help with a few storage test failures
15:14:11 <bcafarel> :)
15:14:18 <slaweq> hopefully
15:14:23 <slaweq> ok, next topic then
15:14:25 <slaweq> #topic Grafana
15:14:31 <slaweq> https://grafana.opendev.org/d/BmiopeEMz/neutron-failure-rate
15:14:52 <slaweq> personally I see only 2 important issues there:
15:15:05 <slaweq> 1. openstack-tox-py38 job
15:15:13 <slaweq> 2. neutron-functional-with-uwsgi job
15:15:32 <ralonsoh> tox
15:15:35 <ralonsoh> sorry
15:15:44 <ralonsoh> tox-py38 was related to os.kill
15:15:49 <ralonsoh> same as FT
15:16:17 <slaweq> but py38 should be good after that revert which You proposed, right?
15:16:22 <ralonsoh> yes
15:16:28 <slaweq> and is it merged already?
15:16:31 <slaweq> I don't remember
15:16:35 <ralonsoh> one sec
15:16:58 <ralonsoh> yes
15:17:02 <slaweq> ++
15:17:33 <slaweq> and for functional job most critical issue is https://launchpad.net/bugs/1921154
15:17:35 <openstack> Launchpad bug 1921154 in neutron "os.kill(SIGTERM) does not finish and timeouts" [Critical,Confirmed] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)
15:17:45 <slaweq> and patch https://review.opendev.org/c/openstack/neutron/+/782986 should at least mittigate it for now
15:17:57 <slaweq> I updated it today to mark both those tests as unstable
15:18:06 <slaweq> as both are failing in the same way
15:18:21 <ralonsoh> even when using shell command kill
15:18:26 <slaweq> when I checked today failed functional jobs from last week it was about 70-80% of all failures
15:18:48 <slaweq> ralonsoh: yes, I think that one was failing even before Your revert
15:19:05 <slaweq> I saw such failures like yesterday or something like that
15:19:23 <ralonsoh> I don't know how to solve this
15:19:32 <ralonsoh> I have a proposal for oslo.privsep
15:19:42 <ralonsoh> to implement a wrapper with timeout
15:20:00 <ralonsoh> of course, that has a problem: the task in the daemon is not stopped
15:20:29 <ralonsoh> there is an alternative, but this could be discussed in the review patch
15:21:36 <slaweq> ok, thx for taking care of it
15:21:47 <slaweq> can You link to that privsep patch in the LP?
15:21:55 <ralonsoh> sure
15:23:25 <slaweq> thx
15:23:45 <slaweq> there are also some other, smaller issues
15:23:51 <slaweq> https://bugs.launchpad.net/neutron/+bug/1921866
15:23:52 <openstack> Launchpad bug 1921866 in neutron "Functional test test_ipv6_router_advts_and_fwd_after_router_state_change_backup failed due to timeout" [High,Confirmed]
15:23:55 <slaweq> I reported that one today
15:24:03 <slaweq> I saw it at least twice last week
15:24:08 <slaweq> anyone wants to check that?
15:24:45 <ralonsoh> i can take a look
15:25:16 <slaweq> thx ralonsoh
15:25:26 <slaweq> #action ralonsoh to check https://bugs.launchpad.net/neutron/+bug/1921866
15:25:27 <openstack> Launchpad bug 1921866 in neutron "Functional test test_ipv6_router_advts_and_fwd_after_router_state_change_backup failed due to timeout" [High,Confirmed]
15:25:41 <slaweq> and the last one which I wanted to mention
15:25:43 <slaweq> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c99/782690/3/check/neutron-functional-with-uwsgi/c99677d/testr_results.html
15:25:48 <slaweq> fail to start metadata agent
15:25:55 <slaweq> did You saw something like that already?
15:26:01 <slaweq> sorry, not agent
15:26:04 <slaweq> metadata proxy
15:26:07 <slaweq> :)
15:26:18 <ralonsoh> no
15:26:44 <slaweq> ok, I will check that
15:26:56 <slaweq> #action slaweq to check failed start metadata proxy issue
15:27:24 <slaweq> that is basically all what I had for today
15:28:03 <slaweq> good news is that our periodic jobs are running fine recently :)
15:28:14 <slaweq> do You have any other topics to discuss today?
15:28:23 <slaweq> or if not, I will give You back about 30 minutes
15:28:38 <ralonsoh> nothing from me
15:28:51 <bcafarel> nothing from me either
15:29:31 <slaweq> ok, so I think we can finish earlier :)
15:29:35 <slaweq> thx for attending the meeting
15:29:38 <slaweq> o/
15:29:41 <lajoskatona> bye
15:29:41 <ralonsoh> bye
15:29:42 <slaweq> #endmeeting