15:00:33 <ralonsoh> #startmeeting neutron_ci
15:00:33 <opendevmeet> Meeting started Tue May 30 15:00:33 2023 UTC and is due to finish in 60 minutes.  The chair is ralonsoh. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:33 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:33 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:37 <ralonsoh> hello all
15:01:12 <mlavalle> o/
15:01:20 <mtomaska> o/ . Looks like this will be a video meeting?
15:01:20 <ralonsoh> lajos and slaweq are not going to attend today
15:01:31 <mlavalle> is it only irc?
15:02:00 <ralonsoh> I don't know what is the schedule
15:02:15 <ralonsoh> this is why I said that this meeting will be in IRC
15:02:30 <ralonsoh> so let's start
15:02:31 <ralonsoh> #topic Actions from previous meetings
15:02:41 <ralonsoh> lajoskatona to check with dnm patch stadium projects with py39
15:03:05 <ralonsoh> there are some issues related mostly with the new RBACs patches in Neutron
15:03:13 <ralonsoh> I'll ping him tomorrow
15:03:18 <ralonsoh> next one
15:03:19 <ralonsoh> ralonsoh to check port creation timeout in functional tests
15:03:25 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron/+/884201
15:03:37 <ralonsoh> the problem, so far, according to the logs
15:03:43 <ralonsoh> was the wait time
15:04:04 <ralonsoh> I also added the vent update just to be sure we receive this event
15:04:15 <ralonsoh> next one
15:04:18 <ralonsoh> ralonsoh  to check dvr_lifecycle timeouts in functional job
15:04:29 <ralonsoh> I didn't have time for this one, I'll try this week
15:04:37 <ralonsoh> next one
15:04:38 <ralonsoh> mlavalle to check ovsdb command timeout in functional job
15:04:41 <ralonsoh> any update?
15:04:44 <mlavalle> Yes
15:05:00 <mlavalle> I followed the job all week long up until today
15:05:12 <mlavalle> didn't find any more occurrences of the issue
15:05:21 <mlavalle> so I think it was a one of
15:05:42 <ralonsoh> ok, there was an open LP?
15:05:46 <mlavalle> it was an osvdb timeout
15:05:53 <mlavalle> not that I know of
15:06:05 <mlavalle> ovsdb^^^
15:06:23 <ralonsoh> ok, so no action is needed then, right?
15:06:28 <mlavalle> correct
15:06:35 <ralonsoh> ok thanks
15:06:42 <ralonsoh> next one is
15:06:44 <ralonsoh> ralonsoh to check failing fips periodic jobs
15:07:02 <ralonsoh> I worked last week with Alfredo Moralejo
15:07:09 <ralonsoh> and he pushed https://review.opendev.org/c/openstack/devstack/+/884277
15:07:25 <ralonsoh> that is solving the issue with the repository certificates
15:07:32 <ralonsoh> the problem is gone now
15:07:39 <ralonsoh> next one
15:07:40 <ralonsoh> ykarel to report grenade issue for nova-api stop failing
15:07:46 <ralonsoh> #link https://bugs.launchpad.net/grenade/+bug/2020643
15:08:00 <ralonsoh> we still don't have any update from Nova folks
15:08:33 <ralonsoh> ykarel, any update on this one?
15:09:04 <ykarel> ralonsoh, i checked this and this is happening due to other issue
15:09:16 <ykarel> https://bugs.launchpad.net/neutron/+bug/2015065
15:10:07 <ykarel> i will push a patch in devstack to handle this failure(nova-api not killed), but fix for requests stuck ^ is still not known
15:10:45 <ralonsoh> but the problem is when Nova sends the request for Neutron, right?
15:10:55 <ralonsoh> at least what is reported in https://bugs.launchpad.net/neutron/+bug/2015065
15:10:56 <ykarel> yes right
15:11:31 <ralonsoh> I though Nova was using wsgi and not importing eventlet anymore
15:11:36 <ralonsoh> but I'm wrong
15:12:19 <ralonsoh> ok, please let's keep an eye on this one because is affecting the CI stability
15:12:30 <ralonsoh> ykarel, thanks for taking care of this complex issue
15:12:55 <ralonsoh> ok, any other pending action?
15:13:19 <ralonsoh> #topic Stable branches
15:13:40 <ralonsoh> bcafarel, told me before this meeting that the stable branches are working fine
15:13:52 <ralonsoh> and there are no critical issues related
15:14:00 <ralonsoh> so I'll move to the next topic then
15:14:07 <ralonsoh> lajos is not here
15:14:26 <ralonsoh> as commented in the agenda, most of the prjects are in red
15:14:33 <ralonsoh> mostly due to the sRBAC changes
15:14:38 <ralonsoh> only 3 are passing
15:14:45 <ralonsoh> openstack/neutron-dynamic-routing
15:14:49 <ralonsoh> openstack/neutron-fwaas
15:14:53 <ralonsoh> openstack/neutron-vpnaas
15:15:12 <ralonsoh> neutron-tempest-plugin-bgpvpn-bagpipe is now failing in n-t-p, but lajos sent a patch
15:15:19 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883838
15:15:55 <ralonsoh> it is not ready but every PS is improving the CI job
15:16:45 <ralonsoh> and I'm blocked with the odl CI
15:16:45 <ralonsoh> https://review.opendev.org/q/topic:tox4+owner:ralonsoh%2540redhat.com+status:open
15:17:04 <ralonsoh> I don't know what is happening in Yoga and older branches
15:17:10 <ralonsoh> #link https://review.opendev.org/c/openstack/networking-odl/+/884359
15:17:26 <ralonsoh> py39 is broken, but not py36
15:17:36 <ralonsoh> and the functional jobs
15:18:23 <ralonsoh> anyway, if I can't solve the CI issues, I'll start deleting the broken jobs
15:18:32 <ralonsoh> we are going to deprecate n-odl project
15:18:38 <mlavalle> yeap
15:18:54 <ralonsoh> ok, next topic
15:18:59 <ralonsoh> #topic Grafana
15:19:06 <ralonsoh> #link https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:19:17 <ralonsoh> I'll need help with these charts ^^
15:19:47 <ralonsoh> I see some spikes but mostly due to known errors and failed patches
15:19:58 <ralonsoh> but nothing permanently on 100% error
15:20:27 <mlavalle> concur. not much going on
15:20:33 <mtomaska> and some show downward plot :) which is good
15:20:42 <ralonsoh> well, neutron-ovn-tempest-ipv6-only-ovs-release was failing too much
15:20:50 <ralonsoh> but seems much better right now
15:21:27 <ralonsoh> and so far I didn't see any critical CI problem so we can jump to the next topic, I think
15:21:37 <ralonsoh> #topic Rechecks
15:21:51 <ralonsoh> last week the recheck rate was 0.8
15:22:06 <ralonsoh> compared to 1.73 from the previous week, this is an improvement
15:22:24 <ralonsoh> so cool, the CI is working a bit better now
15:22:44 <ralonsoh> in the agenda (https://etherpad.opendev.org/p/neutron-ci-meetings) you'll see the most affected patches
15:22:49 <ralonsoh> (most of them, of course, mine)
15:23:06 <ralonsoh> and the bare rechecks % is 6.25
15:23:34 <ralonsoh> please, if you find one, comment on the review asking to check first the CI before recheking the patch
15:23:54 <mlavalle> will do
15:23:55 <ralonsoh> ok, let's go for the next topic
15:23:59 <ralonsoh> #topic fullstack/functional
15:24:04 <ralonsoh> there is one pending issue
15:24:10 <ralonsoh> #link https://dc40d5b24872f3e99b56-a6bd106a0a856fe942ecaed6f7524924.ssl.cf2.rackcdn.com/883421/4/check/neutron-functional-with-uwsgi/6be0e15/testr_results.html
15:24:20 <ralonsoh> that is on my plate, but I didn't have time, as commented
15:24:24 <ralonsoh> I'll check it this week
15:24:38 <ralonsoh> there is nothing else in the agenda
15:24:43 <ralonsoh> do you have something else?
15:24:52 <mlavalle> not me
15:25:09 <ralonsoh> ok, next topic
15:25:09 <ralonsoh> #topic Tempest/Scenario
15:25:14 <ralonsoh> revert cold migration failed (nova issue)
15:25:20 <ralonsoh> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2f1/881535/6/check/neutron-ovs-tempest-dvr-ha-multinode-full/2f17c4b/testr_results.html
15:25:58 <ralonsoh> do you have some time to review this issue
15:25:59 <ralonsoh> ?
15:26:45 <ralonsoh> ok, I'll try to check it this week too
15:26:52 <ralonsoh> next topic
15:26:53 <ralonsoh> #topic grenade
15:26:57 <ralonsoh> nova-api not stopped (again)
15:27:16 <ralonsoh> for this one ykarel opened a LP bug in Nova
15:27:31 <ralonsoh> so it is addressed
15:28:04 <ykarel> actually opened against grenade project
15:28:09 <mlavalle> ralonsoh: I can help with the tempest issue
15:28:15 <ralonsoh> ah sorry, for grenade
15:28:21 <ralonsoh> mlavalle, thanks
15:28:28 <ralonsoh> so you have the link?
15:28:34 <ralonsoh> just to add it to the agenda
15:29:51 <ralonsoh> ok, I'll ping you later
15:30:02 <ralonsoh> the next topic is
15:30:02 <ralonsoh> #topic Periodic
15:30:11 <ralonsoh> #link https://zuul.openstack.org/buildsets?project=openstack%2Fneutron&pipeline=periodic&branch=master
15:30:16 <ralonsoh> we have 2 main issues
15:30:32 <ralonsoh> Tobiko: all jobs are failing and I don't see what the error is
15:30:50 <ralonsoh> actually the job fails because can't start the neutron API but there is no error in the neutron server
15:31:03 <ralonsoh> I'll ping slawek as mantainer of tobiko
15:31:18 <ralonsoh> the second issue is sqlalchemy master
15:31:28 <ralonsoh> the issue with this job is addressed in
15:31:38 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron/+/883421
15:31:48 <ralonsoh> #link https://review.opendev.org/c/openstack/neutron/+/883521
15:31:52 <ralonsoh> ^^ please review the last one
15:32:00 <mlavalle> will do
15:32:08 <ralonsoh> this is the first time this job is passing 100%
15:32:31 <ralonsoh> and that's all!
15:32:46 <ralonsoh> sorry if this meeting was not as good as with slaweq
15:32:53 <ralonsoh> that was my second time chairing it
15:32:56 <mlavalle> it was just fine
15:33:02 <mtomaska> it is great!
15:33:07 <ralonsoh> there are two topic in the on demand agenda
15:33:07 <ralonsoh> #topic On Demand
15:33:12 <ralonsoh> (slaweq) Next week's meeting (6.06.2023) - I will be on PTO so maybe we should cancel ci meeting?
15:33:16 <ralonsoh> (slaweq) Meeting in 2 weeks from now (13.06.2023) - there is Vancover summit - should we cancel ci meeting?
15:33:31 <ralonsoh> so we will cancel this meeting during the next 2 weeks
15:33:38 <mlavalle> ack
15:33:38 <ralonsoh> unless there is any critical issue
15:33:55 <ralonsoh> and of course, any critical CI issue will be addressed on demand
15:34:05 <ralonsoh> so anything else to be added?
15:34:11 <mlavalle> not from me
15:34:37 <ralonsoh> thank you all
15:34:40 <ralonsoh> #endmeeting