15:00:51 <ykarel> #startmeeting neutron_ci
15:00:51 <opendevmeet> Meeting started Tue Jan 30 15:00:51 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:51 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:51 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:01:03 <ralonsoh> hi
15:01:04 <mlavalle> \o
15:01:05 <ykarel> ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:01:10 <haleyb> o/
15:01:12 <mlavalle> is it an irc meeting today?
15:01:25 <slaweq> o/
15:01:35 <ykarel> yes
15:01:42 <mlavalle> ack
15:02:41 <bcafarel> o/
15:03:05 <ykarel> let's start with topics
15:03:07 <ykarel> #topic Actions from previous meetings
15:03:14 <ykarel> ralonsoh to check FOREIGN KEY constraint failed issue in functional test
15:03:44 <ykarel> Logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2e7/903977/7/check/neutron-functional-with-uwsgi/2e76304/testr_results.html
15:03:44 <ykarel> This is a sqlite error: the port binding register is created in the same trasaction as the port; the port ID must be on the transaction commited actions.
15:03:44 <ykarel> This is not an issue related to the patch #903977 nor OVN.
15:04:07 <ralonsoh> ^^ right, that was an issue with sqlite only
15:04:10 <ralonsoh> not the Neutron code
15:04:41 <ykarel> k thx for checking, let's watch if it reproduces again
15:04:52 <ykarel> next one
15:04:52 <ykarel> ralonsoh to check https://bugs.launchpad.net/neutron/+bug/1955008
15:05:07 <ralonsoh> --> https://review.opendev.org/c/openstack/neutron/+/906474
15:05:36 <ykarel> thx
15:05:53 <slaweq> ralonsoh I just approved it
15:05:57 <ralonsoh> thanks
15:06:16 <ykarel> alonsoh to validate fullstack jobs with revert of  https://review.opendev.org/q/I442587e7ef55917f4ea873e190bf8afbc0e911e1
15:06:23 <ralonsoh> --> https://review.opendev.org/q/topic:%22bug/1961740%22: reverts proposed and merged
15:06:23 <ykarel> ralonsoh*
15:06:48 <ykarel> thx, btw i saw similar failure in stable/2023.1 too
15:06:57 <ralonsoh> really? pffff
15:07:02 <ykarel> can look later in failures topic
15:07:11 <ralonsoh> so I don't think this patch is good enough
15:07:59 <ralonsoh> (well, at least for fullstack tests...)
15:08:09 <ykarel> yes not seeing it in master/stable2023.2
15:08:15 <opendevreview> Lajos Katona proposed openstack/tap-as-a-service master: WIP: Documentation for tap-mirrors  https://review.opendev.org/c/openstack/tap-as-a-service/+/907220
15:08:30 <ykarel> may be due to those tests are trimmed in master
15:08:45 <ralonsoh> could be, right
15:08:46 <ykarel> not sure if those patches from slaweq also in stable/2023.2
15:09:04 <ralonsoh> we can backport the fullstack optimizations if needed
15:09:21 <ralonsoh> I think that could benefit stable branches CI too
15:09:27 <bcafarel> +1
15:09:38 <ralonsoh> (I'll add this to my todo list)
15:09:45 <ykarel> ok thx
15:09:58 <ykarel> next one on me
15:09:59 <ykarel> ykarel to check enabling debug logs for ovs for log size
15:10:13 <ykarel> so i checked it with dnm patch https://review.opendev.org/c/openstack/neutron/+/907037
15:10:23 <ykarel> functional job -> 2MB --> 130 MB
15:10:29 <ykarel> fullstack job -> 3MB --> 70 MB
15:10:36 <ralonsoh> (zuul folks are going to kill us...)
15:11:03 <ykarel> increase is significant so doesn't seem good to enable it for all such runs
15:12:00 <slaweq> yes, it's better to not do that
15:12:37 <ykarel> ok
15:12:44 <ralonsoh> but we can document it for debugging purposes
15:12:47 <ykarel> ykarel to check if internal ports removal/addition have impact on bridge add/delete
15:13:40 <ykarel> ralonsoh, okk will check and report doc bug for it
15:13:45 <ralonsoh> cool
15:13:57 <ykarel> #action ykarel to report doc bug for ovs enable debug
15:14:06 <slaweq> what if we would tar.gz those logs?
15:14:14 <slaweq> it should be much smaller, no?
15:14:35 <ralonsoh> 10-20 times smaller
15:14:49 <ykarel> yes should be smaller, need to check how much exactly
15:15:07 <slaweq> maybe worth to do one more experiment then?
15:15:39 <ykarel> ok will do, but iirc if we tar we need to download those to check content or those still browsable?
15:16:02 <slaweq> yes, we will need to download to check content
15:16:37 <ykarel> ohkk
15:17:01 <bcafarel> hmm if it is per single file gz, it should be browsable (may need some tweaks on server side for content-type)
15:17:17 <bcafarel> tarball of all files indeed it would need download
15:17:29 <ykarel> will update that patch to convert those to tar archive and confirm browsable aspects too
15:17:41 <ykarel> bcafarel, thx will check
15:17:53 <ykarel> ykarel to check if internal ports removal/addition have impact on bridge add/delete
15:18:10 <ykarel> i was not able to reproduce that behavior with vsctl commands
15:18:22 <ykarel> ovs command don't allow delete local internal port
15:18:27 <ykarel> ovs-vsctl del-port test-br test-br
15:18:33 <ykarel> fails as ovs-vsctl: cannot delete port test-br because it is the local port for bridge test-br (deleting this port requires deleting the entire bridge)
15:19:19 <ykarel> so that add/delete/add of that internal port seems to be happing internally in ovs and it matches the shy ports issue we seen in past
15:20:24 <ykarel> any idea to simulate this behavior?
15:20:42 <slaweq> nope
15:20:47 <ralonsoh> no sorry
15:21:01 <ykarel> k thx let's move on
15:21:09 <ykarel> #topic Stable branches
15:21:13 <ykarel> bcafarel, any update
15:21:38 <bcafarel> we already mentioned the revert for fullstack sleep, rest looks good
15:21:50 <ykarel> thx
15:21:56 <bcafarel> and with the unmaintained branches coming up we will have a shorter section soon :)
15:22:09 <ykarel> cool :)
15:22:17 <ykarel> #topic Stadium projects
15:22:38 <ykarel> there was one failure in periodic
15:22:46 <ykarel> for bgpvpn https://zuul.openstack.org/buildset/4bc10cd65b28461da28a76fa10553f07
15:23:00 <ykarel> Looks related to django update https://review.opendev.org/c/openstack/requirements/+/904202
15:23:32 <ykarel> lajos is out, so will check if there is already some fixes for it and create a lp
15:23:51 <ykarel> #action ykarel to check and report lp bug for bgpvpn failures
15:24:06 <opendevreview> Merged openstack/neutron stable/2023.2: Set trunk parent port as access port in ovs to avoid loop  https://review.opendev.org/c/openstack/neutron/+/907142
15:24:07 <ykarel> #topic Rechecks
15:24:28 <ykarel> overall good here, there was 0 bare rechecks out of 18 rechecks
15:24:54 <ykarel> there were some known issues which could have resulted into higher rechecks
15:25:12 <ykarel> #topic Unit tests
15:25:25 <ykarel> https://launchpad.net/bugs/2051501
15:25:36 <ykarel> already Fixed with https://review.opendev.org/c/openstack/neutron/+/907040
15:25:39 <ykarel> thx ralonsoh
15:25:44 <ralonsoh> yw
15:25:52 <ykarel> #topic fullstack/functional
15:25:59 <ykarel> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1bf/892599/4/gate/neutron-fullstack-with-uwsgi/1bf74e0/testr_results.html
15:26:10 <ykarel> ^ is the failure i mentioned before in 2023.1
15:26:18 <ykarel> Looks same as for which zed/yoga revert https://review.opendev.org/q/topic:%22bug/1961740%22
15:26:32 <ralonsoh> no no, the error was different
15:26:42 <ralonsoh> this is not the test case that was failing because of this patch
15:27:13 <ralonsoh> (sorry, yes...)
15:27:14 <ykarel> mmm but commit message says it
15:27:23 <ralonsoh> my bad, I had other bug in mind
15:27:24 <ykarel> ok :)
15:28:14 <ykarel> so for fullstack we go with backport of optimization patches or do reverts
15:28:40 <ralonsoh> let me push the backports tomorrow
15:28:47 <ralonsoh> up to zed
15:29:07 <ykarel> because i noticed higher fullstack failures in stable since the revert, so likely the failures for which those sleep patches pushed reappear?
15:29:36 <ykarel> ok trimming those sounds better, thx
15:29:48 <ykarel> #topic Periodic
15:30:15 <ykarel> https://launchpad.net/bugs/2051501
15:30:29 <ykarel> this already Fixed with https://review.opendev.org/c/openstack/neutron/+/907040
15:30:37 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-slow&project=openstack%2Fneutron&branch=master&skip=0
15:30:46 <ykarel> Looks related to recent changes https://review.opendev.org/q/topic:%222023-aa-l3-gw-multihoming%22+status:merged+project:openstack/neutron
15:31:14 <ykarel> any volunteer to check/confirm these failures?
15:31:23 <ralonsoh> I can check it
15:31:30 <ykarel> thx
15:31:45 <ykarel> #action ralonsoh to check failures with neutron-ovn-tempest-slow job in master
15:31:53 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-fullstack-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&skip=0
15:32:05 <ykarel> Fails at dnf install -y kernel-modules-extra-5.14.0-410.el9.x86_64
15:32:23 <ykarel> it's happening due to Outdated infra mirrors http://mirror.iad.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/
15:32:50 <ykarel> rpm exists at centos mirrors https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/
15:33:08 <ralonsoh> then should we switch to this mirror?
15:33:32 <ykarel> i think need to fix infra mirrors
15:33:45 <ralonsoh> yeah, right, much better
15:33:58 <ykarel> will check tomorrow with infra on this, likely the mirror which they follow is outdated
15:35:08 <opendevreview> Merged openstack/neutron stable/2023.1: Set trunk parent port as access port in ovs to avoid loop  https://review.opendev.org/c/openstack/neutron/+/907143
15:35:20 <ykarel> #action ykarel to reach out to infra for outdated mirror if it's not resolved in next periodic run
15:35:34 <ykarel> #topic Grafana
15:35:41 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:37:49 <ykarel> all good in gate queue, some spikes in check around weekend which we normally see due to less runs
15:38:28 <ykarel> a spike is for sqlalchemy master job which already got fixed
15:39:38 <ykarel> there are some recent spikes for tox-pep8
15:39:54 <ykarel> i haven't noticed any consistent error though, so may be patch specific
15:40:08 <slaweq> but it's 1 failure out of 4 if I'm not mistaken for the pep8
15:40:14 <slaweq> so nothing really serious IMO
15:41:08 <ykarel> yes right, then we are good
15:41:14 <ykarel> #topic On Demand
15:41:22 <ykarel> anything else you would like to raise here?
15:41:26 <ralonsoh> nothing from me
15:41:36 <bcafarel> all good here too
15:41:57 <slaweq> nothing here
15:42:03 <mlavalle> nothing from me
15:42:08 <mlavalle> either
15:42:18 <ykarel> k thanks everyone for joining
15:42:29 <ykarel> let's close and have everyone 18 minutes back
15:42:33 <ykarel> #endmeeting