15:00:31 <ykarel> #startmeeting neutron_ci
15:00:31 <opendevmeet> Meeting started Tue Dec 12 15:00:31 2023 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:31 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:31 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:35 <mlavalle> \o
15:00:41 <ykarel> ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:00:46 <ralonsoh> hi
15:00:47 <ykarel> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:00:47 <ykarel> Please open now :)
15:01:41 <slaweq> hi
15:02:44 <bcafarel> o/
15:02:56 <haleyb> o/
15:03:03 * bcafarel in another meeting, watching from time to time
15:03:10 <ykarel> Let's start
15:03:15 <ykarel> #topic Actions from previous meetings
15:03:17 <mtomaska> o/
15:03:25 <ykarel> ykarel to check and reopen bug for shy ports in functional tests
15:03:38 <ykarel> reopened https://bugs.launchpad.net/neutron/+bug/1961740
15:03:54 <opendevreview> Rodolfo Alonso proposed openstack/ovn-bgp-agent master: [UT] Fix the ``TestLinuxNet`` wrong assert calls  https://review.opendev.org/c/openstack/ovn-bgp-agent/+/903511
15:04:02 <ykarel> ykarel to check/report issue with test_unshelve_to_specific_host
15:04:14 <ykarel> reported https://bugs.launchpad.net/nova/+bug/2045785
15:04:26 <ykarel> mlavalle to check failure with test_port_binding_chassis_create_event
15:04:43 <mlavalle> I did chck it. Using https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi&project=openstack%2Fneutron&branch=master&result=FAILURE&skip=0
15:05:02 <mlavalle> I didn't find another ocuurence since the begining of the month
15:05:10 <mlavalle> so it was probably a one of
15:05:25 <ykarel> Thanks for checking mlavalle
15:05:35 <ykarel> slawek to report bug for fullstack failure
15:05:45 <ykarel> #link https://bugs.launchpad.net/neutron/+bug/2045757
15:06:00 <slaweq> it should be also fixed now
15:06:02 <ykarel> #link https://review.opendev.org/c/openstack/neutron/+/902762
15:06:18 <ykarel> slaweq, so it could be even backported ?
15:06:34 <slaweq> it is up to Zed
15:06:43 <slaweq> all backports merged already
15:06:47 <slaweq> https://review.opendev.org/q/I1c20c90b8abd760f3a53b24024f19ef2bd189b5a
15:06:48 <ykarel> okk good, i missed that
15:06:54 <ykarel> thx for taking care of it
15:07:01 <slaweq> yw
15:07:05 <ykarel> ralonsoh to send fix for ft test test_port_forwarding
15:07:26 <ralonsoh> ups, I think I have the patch but never sent
15:07:30 <ralonsoh> I'll do it today
15:07:35 <ralonsoh> sorry
15:07:43 <ykarel> np, thx for looking into it
15:07:55 <ykarel> #topic Stable branches
15:08:09 <ykarel> bcafarel, all yours ^
15:09:11 <ykarel> i noticed couple of patches merged this week in stable, and periodic lines were good too.
15:09:25 <ykarel> so should be all good in stable
15:09:27 <bcafarel> yes, I have a few reviews not catched on
15:09:33 <bcafarel> but from what I saw all good!
15:09:42 <ykarel> thx bcafarel
15:09:52 <bcafarel> and with ussuri going EOL, one less branch to worry about soon :)
15:09:52 <ykarel> #topic Stadium projects
15:09:59 <ykarel> +1
15:10:19 <ykarel> wrt Stadium weekly periodics were green
15:10:25 <ykarel> so all good here too
15:10:34 <ykarel> #topic Rechecks
15:10:52 <ykarel> quite a good number of patches merged this week across branches
15:11:09 <ykarel> and no bare rechecks out of 36 rechecks in total
15:11:16 <ykarel> let's keep doing it
15:11:33 <ykarel> there were some infra issues too that lead to such rechecks
15:12:18 <ykarel> #topic Unit tests
15:12:28 <ykarel> Just an info
15:12:34 <ykarel> Random POST_FAILURES against jobs, resolved with https://review.opendev.org/c/opendev/base-jobs/+/903351
15:13:11 <ykarel> #topic fullstack/functional
15:13:34 <ykarel> i have not noticed anything new in fullstack/functional so not reported here
15:14:01 <ykarel> it was general fullstack timeout and known functional failures, the one i reopened as linked above
15:14:05 <slaweq> that's good news :)
15:14:26 <ralonsoh> slaweq++ nice patches for fullstack
15:14:43 <ykarel> ++
15:15:05 <ykarel> #topic Tempest/Scenario
15:15:25 <ykarel> Ubuntu image download random issue should be resolved with https://review.opendev.org/q/I7163aea4d121cb27620e4f2a083a543abfc286bf
15:15:42 <slaweq> are You sure?
15:15:57 <ykarel> that was merged last night in master
15:16:08 <ykarel> and since then i have not seen the failure, you seen some?
15:16:10 <slaweq> I think I commented in some patch like that already
15:16:24 <slaweq> timeouts seems to be in the neutron-tempest-plugin's devstack plugin
15:16:53 <slaweq> https://github.com/openstack/neutron-tempest-plugin/blob/master/devstack/customize_image.sh
15:17:36 <ykarel> yes the function here is wrapper to the devstack one, and as fix already in devstack so we should be good now
15:17:36 <slaweq> ahh, ok
15:17:43 <slaweq> now I see
15:17:48 <slaweq> sorry for the noise then :)
15:17:53 <slaweq> ++
15:18:26 <ykarel> There are some other random failures in tempest jobs for which bugs already reported.
15:18:49 <ykarel> So if we start seeing more such failures in those tempest jobs we can consider skipping impacted tests
15:20:05 <ykarel> i mean https://bugs.launchpad.net/neutron/+bug/2039940 and https://bugs.launchpad.net/nova/+bug/2045785
15:20:28 <ykarel> #topic Periodic
15:21:00 <ykarel> https://zuul.openstack.org/builds?job_name=devstack-tobiko-neutron&project=openstack%2Fneutron&branch=master&skip=0
15:21:12 <ykarel> Last run failed with virt-customize errors, need to check if it's one time failure or happens again
15:21:38 <ykarel> i will check next periodic runs to see if it's some consistent new issue or one time thing
15:22:12 <ykarel> #action ykarel to look for next periodic runs of devstack-tobiko-neutron and file bug if consistent issue
15:22:19 <ykarel> Stable periodics
15:22:27 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=stable%2Fyoga&skip=0
15:22:36 <ykarel> Already assigned bug to ralonsoh https://bugs.launchpad.net/neutron/+bug/2039650
15:23:03 <ralonsoh> ok, I have this before my parental leave
15:23:06 <ralonsoh> I'll retake it again
15:23:42 <ykarel> thx ralonsoh
15:24:09 <ykarel> i just noticed as only yoga had consistent failures in periodic
15:24:21 <ykarel> #topic Grafana
15:24:27 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:24:42 <ykarel> let's have a quick look here too
15:25:24 <ralonsoh> some spikes during the weekend
15:26:17 <slaweq> I wouldn't bother with weekends too much, it usually has some spikes as there is less patches tested during that time
15:26:30 <ralonsoh> right
15:26:38 <ykarel> +1
15:26:44 <ykarel> rest all looks good
15:27:00 <ykarel> #topic On Demand
15:27:52 <ralonsoh> I'm good
15:28:02 <slaweq> me too
15:28:58 <mlavalle> nothing from me
15:29:01 <mlavalle> either
15:29:27 <ykarel> thx everyone then, let's close and have everyone 31 minutes back :)
15:29:31 <ykarel> #endmeeting