15:00:04 <slaweq> #startmeeting neutron_ci
15:00:04 <opendevmeet> Meeting started Tue Jun 27 15:00:04 2023 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:04 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:07 <mlavalle> o/
15:00:09 <slaweq> ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:00:12 <ralonsoh> hello
15:00:17 <ykarel> o/
15:00:19 <bcafarel> o/
15:00:51 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:00:51 <slaweq> Please open now :)
15:01:02 <slaweq> I think we can start
15:01:04 <slaweq> #topic Actions from previous meetings
15:01:12 <slaweq> ykarel to check test failures  in neutron-ovn-tempest-ipv6-only-ovs-release
15:01:29 <slaweq> Proposed https://review.opendev.org/c/openstack/neutron/+/885074
15:01:38 <ykarel> That's i think is from couple of weeks before
15:01:44 <slaweq> yeah
15:01:49 <opendevreview> Amit Uniyal proposed openstack/os-vif stable/yoga: set default qos policy  https://review.opendev.org/c/openstack/os-vif/+/886710
15:01:55 <slaweq> we haven't have ci meeting in June yet :)
15:02:02 <slaweq> it's our first this month
15:02:09 <ykarel> ack :)
15:02:12 <bcafarel> just in time, it's not July yet
15:02:13 <ykarel> recently i saw more failures, will push a follow up patch for that
15:02:27 <slaweq> ok
15:02:29 <slaweq> thx
15:02:32 <ykarel> https://bugs.launchpad.net/neutron/+bug/2007166/comments/9
15:03:36 <slaweq> thx ykarel
15:03:57 <slaweq> I think we can move on to the next topic then
15:04:10 <slaweq> #topic Stable branches
15:04:25 <bcafarel> a quiet month overall :)
15:05:01 <bcafarel> I have to check the latest backports (one antelope got TIMED_OUT on rally twice) but no open issues that I know of at the moment
15:05:41 <slaweq> those time outs are pretty common issue currently
15:05:53 <slaweq> and don't seems to be related to any branch/change/anything else
15:06:36 <bcafarel> yep that's what I am thinking too
15:06:43 <bcafarel> so tl;dr all good
15:06:50 <ykarel> iirc we bumped timeout in master recently for those job
15:06:58 <ykarel> not sure if backported to antelope
15:07:11 <slaweq> great, thx bcafarel
15:08:06 <ykarel> https://review.opendev.org/c/openstack/neutron/+/885045
15:08:40 <slaweq> nope, it's not backported
15:08:44 <ralonsoh> ok, time to do it
15:08:58 <ykarel> +1
15:09:00 <opendevreview> Rodolfo Alonso proposed openstack/neutron stable/2023.1: Raise the timeout of "neutron-ovn-rally-task" to 9000  https://review.opendev.org/c/openstack/neutron/+/887056
15:09:06 <bcafarel> :)
15:09:33 <slaweq> +2
15:09:48 <slaweq> ok, I think we can move on to the next topic then
15:09:51 <slaweq> #topic Stadium projects
15:09:58 <slaweq> lajoskatona any updates?
15:10:09 <lajoskatona> things seems to be quiet
15:10:27 <lajoskatona> for bagpipe I pushed one patch to fix issues with sqlalchemy 2:
15:10:32 <opendevreview> Amit Uniyal proposed openstack/os-vif stable/xena: set default qos policy  https://review.opendev.org/c/openstack/os-vif/+/886716
15:10:37 <lajoskatona> https://review.opendev.org/c/openstack/networking-bagpipe/+/887024
15:11:01 <slaweq> I also saw that networking-odl periodic jobs are failing and wanted to ask if we still need to run those jobs as project is deprecated
15:11:36 <ralonsoh> no, I'll delete them
15:11:38 <lajoskatona> I have to go back to it as it looked easy just rmeove subtransactions=True, but something is missing and weird things happen sometimes without that keyword
15:11:56 <lajoskatona> yes we can delete those now
15:12:58 <lajoskatona> just one more thing for stadiums: please check the open patches for them, I try to keep an eye on them also :-)
15:12:59 <slaweq> #action ralonsoh to remove networking-odl periodic jobs
15:13:03 <slaweq> thx
15:14:37 <ralonsoh> qq, what jobs? the master branch has been deleted
15:14:43 <ralonsoh> do you have the names?
15:15:24 <slaweq> https://zuul.openstack.org/buildset/bdb18cb84e3e411daaf23f7cb86ad1c5
15:15:43 <slaweq> those are periodic jobs run for networking-odl
15:15:51 <lajoskatona> hmmm, rodolfo is right thoses should be deleted now
15:15:53 <slaweq> at least those jobs were run this saturday
15:17:08 <lajoskatona> the deleting patch was merged yesterday so it should be ok, and last time we saw those jobs :-)
15:17:16 <slaweq> ahh, ok then
15:17:40 <lajoskatona> the ap can be changed and assigned to me to check if we have some common reporisoties with jobs for ODL
15:18:36 <slaweq> #action lajoskatona to check if we have some common reporisoties with jobs for ODL
15:18:50 <slaweq> I will forget about AI on ralonsoh :)
15:18:50 <ralonsoh> ok ok, these are the sable branches
15:19:10 <ralonsoh> I'll remove the periodic executions from the stable branches too
15:21:00 <slaweq> ++
15:21:04 <slaweq> ok, I think we can move on
15:21:08 <slaweq> next topic
15:21:14 <slaweq> #topic Grafana
15:21:19 <slaweq> ok
15:21:39 <slaweq> generally it looks fine
15:22:08 <slaweq> I see that jobs are on pretty high failure rate but in most cases those are timeouts which we see everywhere
15:22:54 <slaweq> anything else regarding grafana anyone wants to add?
15:23:03 <lajoskatona> do yo know perhaps if there is common background for th timeouts?
15:23:36 <lajoskatona> I mean issue in infrastructure or similar
15:23:41 <slaweq> no, I don't know
15:23:49 <slaweq> but I will ask later today on the tc meeting about it
15:24:06 <lajoskatona> ok, thanks
15:24:53 <slaweq> ok, next topic
15:24:54 <slaweq> #topic Rechecks
15:25:05 <slaweq> generally we are doing a lot of rechecks again
15:25:17 <slaweq> https://etherpad.opendev.org/p/neutron-ci-meetings#L41
15:25:30 <ralonsoh> there was an issue in FTs last week
15:25:38 <slaweq> in last few weeks it was more than 2 in average to get anything merged
15:26:43 <slaweq> that's all from me about rechecks
15:27:06 <slaweq> if there are no other questions/comments I think we can move on
15:27:12 <ralonsoh> good for me
15:27:20 <slaweq> #topic fullstack/functional
15:27:31 <slaweq> here I found just one potentially interesting issue in last few days
15:27:36 <slaweq> test_update_minimum_bandwidth_queue
15:27:40 <slaweq> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_709/886992/1/check/neutron-functional-with-uwsgi/70996ce/testr_results.html
15:28:18 <slaweq> I don't think it was related to the patch on which it was run
15:28:22 <slaweq> https://review.opendev.org/c/openstack/neutron/+/886992
15:28:50 <slaweq> but the issue seems like something what should be reproducible (assertion error)
15:29:10 <slaweq> it failed due to different "queue_num" value
15:29:14 <ralonsoh> right
15:29:28 <slaweq> did You saw something like that already?
15:30:03 <ralonsoh> no unless that any other test is interfering, but each OVS instance is independent for each test
15:30:06 <ralonsoh> if I'm not wrong
15:30:20 <slaweq> ovs is the same for all tests
15:30:27 <ralonsoh> even the DB?
15:30:29 <slaweq> it creates separate bridges for tests
15:30:46 <slaweq> yes, I think that ovs db is shared across all tests
15:30:53 <slaweq> which just run it once and then uses it
15:31:09 <ralonsoh> so this could be a problem, let me check if I can find what other test is affecting it
15:31:11 <slaweq> and we are using devstack code to run ovs
15:31:22 <slaweq> thx ralonsoh
15:31:34 <slaweq> You can open LP for that if You think it's valid issue
15:31:45 <ralonsoh> I can do it, thanks!
15:32:03 <slaweq> thank You
15:32:26 <slaweq> #action ralonsoh to check failed test_update_minimum_bandwidth_queue functional test
15:32:32 <slaweq> #topic Tempest/Scenario
15:32:49 <slaweq> first on the list is test_hotplug_nic added by ykarel
15:32:58 <slaweq> but he already mentioned that earlier today
15:33:03 <slaweq> https://bugs.launchpad.net/neutron/+bug/2007166
15:33:36 <slaweq> other than that I saw one issue with connectivity in linuxbridge job:
15:33:38 <slaweq> https://1e3bcd2e249bfc8aee18-7f77ff85d71aba60d48e1d6b772dec0b.ssl.cf2.rackcdn.com/885999/9/check/neutron-tempest-plugin-linuxbridge/93f0305/testr_results.html
15:34:40 <slaweq> now I have a question about this LB job
15:35:30 <slaweq> looking at https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1&viewPanel=16&from=now-30d&to=now it seems that this jobs is a bit less stable than other scenario jobs, especially in last weeks
15:35:46 <slaweq> do You think it's already time to maybe move this job from check to e.g. periodic queue?
15:36:23 <ralonsoh> I'm ok with this, we agreed on that and have make this public during the last cycles
15:36:38 <ralonsoh> that we are no longer actively supporting LB nor fixing CI errors
15:36:43 <lajoskatona> +1
15:36:55 <slaweq> ok, I will propose patch for that
15:36:58 <ralonsoh> perfect
15:37:11 <slaweq> #action slaweq to move LB scenario job to periodic queue
15:37:32 <slaweq> I also noticed some connectivity issues in the grenade job https://zuul.opendev.org/t/openstack/build/5cdd01584e844e839b28fcfc273537ae
15:37:57 <slaweq> it could be (again) slow nodes and nothing else
15:38:05 <ralonsoh> no no
15:38:11 <ralonsoh> these are from a specific patch
15:38:26 <ralonsoh> these test (I don't know why) are always failing
15:38:32 <slaweq> ahh, ok
15:38:36 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/883907
15:38:38 <slaweq> so false alarm then :)
15:38:47 <ralonsoh> I don't know what is wrong in this patch...
15:38:51 <slaweq> I didn't though that this patch can be related
15:38:53 <ralonsoh> (help is welcome)
15:39:22 <slaweq> please ping me tomorrow, I can try to take a look at this patch
15:39:26 <ralonsoh> thanks!
15:39:39 <slaweq> ok, that's all on this topic from me
15:39:44 <slaweq> any other questions/comments?
15:39:51 <slaweq> issues to discuss maybe
15:39:55 <ralonsoh> nope
15:40:18 <slaweq> so lets move on to the last topic
15:40:19 <slaweq> #topic Periodic
15:40:37 <slaweq> here I found out that neutron-functional-with-oslo-master and neutron-functional-with-sqlalchemy-master are failing since few days at least
15:40:42 <slaweq> Bug reported https://bugs.launchpad.net/neutron/+bug/2025126
15:40:48 <slaweq> failure examples:
15:40:49 <ralonsoh> patch proposed
15:40:53 <slaweq> https://zuul.openstack.org/build/55a065238b784ac28e91469d2acce3da
15:40:53 <slaweq> https://zuul.openstack.org/build/2d8d000b62a1448d984eab7059d677a7
15:40:57 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/886961
15:41:01 <slaweq> ralonsoh that's fast :)
15:41:18 <ralonsoh> ^^ this is the problem of testing with master branches
15:41:25 <ralonsoh> that we need to revert previous fixes
15:41:53 <slaweq> at least we are catching such issues fairly quickly without breaking our gate :)
15:42:01 <ralonsoh> exactly
15:42:23 <slaweq> and that's all what I had for today
15:42:37 <slaweq> anything else related to CI You want to discuss maybe?
15:42:49 <ralonsoh> all good
15:42:51 <slaweq> or if not, I will give You back few minutes
15:43:05 <mlavalle> nothing from me either
15:43:42 <slaweq> thx for attending the meeting and see You online
15:43:42 <slaweq> #endmeeting