15:00:11 <slaweq> #startmeeting neutron_ci
15:00:11 <opendevmeet> Meeting started Tue May 31 15:00:11 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:11 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:11 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:58 <bcafarel> o/ again
15:01:12 <slaweq> ralonsoh lajoskatona bcafarel ykarel obondarev mlavalle ping, CI meeting is starting
15:01:17 <ralonsoh> hi sorry
15:01:20 <obondarev> hi
15:01:20 <ykarel> o/
15:01:35 <mlavalle> o/
15:02:16 <lajoskatona> Hi
15:02:27 <slaweq> ok, lets start as we have quorum already
15:02:43 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:03:04 <slaweq> #topic Actions from previous meetings
15:03:09 <slaweq> mlavalle  to propose making things not supported framework in Neutron
15:03:30 <mlavalle> didn't make progress on that last week. I'll work on it this week
15:03:54 <slaweq> ok
15:04:04 <slaweq> #action mlavalle  to propose making things not supported framework in Neutron
15:04:12 <slaweq> lets not forget about it :)
15:04:30 <slaweq> next one
15:04:32 <slaweq> ykarel will propose to use other cirros image to avoid kernel panics
15:04:52 <ykarel> Done https://review.opendev.org/q/I34f3bbf7d30ddce23758ffb513abff0500b1fad6
15:05:02 <ykarel> merged till wallaby
15:05:08 <slaweq> thx a lot
15:05:18 <ykarel> do we need to backport to older branches too
15:05:37 <opendevreview> Rodolfo Alonso proposed openstack/neutron master: [WIP] Skip "PortBindingChassisEvent" if revision number changes  https://review.opendev.org/c/openstack/neutron/+/842147
15:05:53 <ykarel> for singlenodes jobs also it was backported till wallaby, so i did same to multinodes
15:06:05 <slaweq> I think it's enough
15:06:07 <ykarel> k
15:06:39 <slaweq> next one
15:06:43 <slaweq> ykarel to check fips issues in periodic jobs https://zuul.openstack.org/build/8a3af48f193141feb6e5dd496e66e58a
15:07:04 <ykarel> i checked that and pushed https://review.opendev.org/c/openstack/neutron/+/843245 to switch those jobs to c9-stream
15:07:16 <ykarel> but faced couple of issues and reported https://bugs.launchpad.net/neutron/+bug/1976323
15:07:34 <ykarel> and to unblock this job until all issues in c9-stream are fixed pushed https://review.opendev.org/c/openstack/neutron/+/843252
15:07:42 <ykarel> to use python3.8 in c8-stream itself
15:07:55 <ykarel> this is working fine
15:08:14 <lajoskatona> ykarel: nice one
15:08:53 <ykarel> also noticed scenario fips job in yoga broken due to py3.6+ tempest master
15:08:57 <slaweq> ykarel thx, I can assign that bug to myself and work on this centos9 stream issues
15:08:58 <ykarel> pushed https://review.opendev.org/c/openstack/neutron/+/843824
15:09:42 <ykarel> slaweq, okk, i didn't digged much, just checked functional issues are reproducable locally
15:10:08 <slaweq> k, I will deploy box with c9s and will work on it
15:10:21 <slaweq> for now using py38 and c8s should be enough IMO
15:10:29 <ykarel> ack
15:10:56 <slaweq> #action slaweq to fix functiona/fullstack failures on centos 9 stream: https://bugs.launchpad.net/neutron/+bug/1976323
15:11:10 <slaweq> so last AI from last week:
15:11:12 <slaweq> lajoskatona to update UT periodic jobs to use at least py38
15:11:45 <lajoskatona> I checked, but was slow and gmann fixed it before me:
15:11:57 <lajoskatona> https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842646
15:12:11 <slaweq> thx gmann and lajoskatona :)
15:12:25 <lajoskatona> I just run after the events :-)
15:12:55 <slaweq> ok, so that were all AI from last week
15:13:01 <slaweq> I think we can move on
15:13:03 <slaweq> #topic Stable branches
15:13:09 <slaweq> any news here bcafarel
15:13:32 <bcafarel> let's see, wallaby and newer are good
15:13:56 <bcafarel> for older branches you probably saw the gate issue mail (dropped py36 in tempest etc) - thanks gmann for working on these fixes
15:14:20 <gmann> devstack patches merged now
15:14:23 <bcafarel> https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/838053 is close to merge for ussuri/victoria then we will need https://review.opendev.org/c/openstack/neutron/+/843781 for ussuri
15:14:30 <gmann> yeah
15:14:33 <ykarel> train should also be good one of my patch merged today
15:14:44 <gmann> yeah train is already pinned before
15:14:51 <lajoskatona> gmann: thanks
15:15:27 <slaweq> thx for working on this
15:16:27 <slaweq> anything else regarding stable branches or can we move on?
15:16:52 <bcafarel> rest is good :) we can move on
15:16:57 <slaweq> thx
15:17:00 <slaweq> #topic Stadium projects
15:17:12 <slaweq> lajoskatona anything new here?
15:17:13 <lajoskatona> nothing extra from me
15:17:17 <slaweq> ++
15:17:24 <slaweq> no news is good news generally
15:17:28 <slaweq> at least in that meeting :)
15:17:29 <lajoskatona> as mentioned on previous meeting I will send out to EOl old branches
15:17:37 <lajoskatona> slaweq: agree :-)
15:17:44 <ykarel> is issue with networking-sfc known?
15:17:58 <ykarel> i saw it too affected by sqlalchemy 2 changes
15:18:01 <slaweq> ykarel what issue exactly?
15:18:03 <ralonsoh> yes
15:18:05 <ralonsoh> it is reported
15:18:10 <lajoskatona> ykarel: I saw tempest failure, is that you mean?
15:18:22 <ralonsoh> https://bugs.launchpad.net/neutron/+bug/1976360
15:18:38 <ykarel> yes i meant ^ one
15:18:46 <ykarel> seen in neutron-tempest-plugin patch
15:19:14 <ralonsoh> I'll push a patch tomorrow (but you can take it, if you want)
15:19:17 <lajoskatona> thanks, I can check this one
15:19:21 <ralonsoh> cool
15:19:23 <slaweq> ++
15:19:30 <ykarel> i can check but tomorrow only
15:20:09 <lajoskatona> this morning I hoped it is temporary glitch, and havent cheked the logs in detail
15:20:23 <ralonsoh> nope, it is legit
15:20:40 <ykarel> ralonsoh, ok i will assign myself and push a patch in my morning
15:20:49 <ralonsoh> ykarel++
15:21:00 <slaweq> thx ykarel and ralonsoh
15:21:14 <slaweq> I think we can move on to the next topic
15:21:15 <slaweq> #topic Grafana
15:21:44 <slaweq> first of all we need to remove emove lower-constraints and py36 jobs from the graphs
15:21:49 <slaweq> anyone wants to do this?
15:22:00 <ralonsoh> I can
15:22:12 <slaweq> thx ralonsoh
15:22:36 <slaweq> #action ralonsoh to remove lower-constraints and py36 jobs from the graphs in grafana
15:22:54 <slaweq> other than that I still see that functional tests job is our biggest pain
15:23:26 <slaweq> good thing is that our scenario jobs are really good recently
15:24:04 <slaweq> for functional tests still our biggest issue is the one with timeouts during ha router's state transition
15:24:27 <slaweq> so I was thinking today that maybe we can temporary mark all tests which are doing such transition as unstable
15:24:32 <slaweq> wdyt?
15:24:41 <ralonsoh> fine for me
15:24:51 <obondarev> +1
15:24:54 <bcafarel> +1 that's our biggest offender currently
15:24:58 <mlavalle> +1
15:24:59 <ykarel> +1
15:25:09 <lajoskatona> +1
15:25:12 <slaweq> so I will propose patch tomorrow
15:25:31 <slaweq> #action slaweq to mark functional l3 ha tests as unstable temporary
15:25:44 <slaweq> anything else regarding grafana?
15:25:47 <slaweq> or can we move on?
15:26:00 <mlavalle> let's move on
15:26:05 <lajoskatona> +1
15:26:46 <slaweq> #topic Rechecks
15:27:03 <slaweq> rechecks in last weeks were mostly below 1 in average:
15:27:10 <slaweq> | 2022-17 |   0.77   |... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/DzfbWLzrhYzumKNEejtutJsy)
15:27:21 <slaweq> so it's still pretty good IMO
15:27:44 <slaweq> Gate failure bugs: https://tinyurl.com/2p9x6yr2 - we have 27 bugs opened, -1 from last week
15:27:56 <slaweq> thx for working on some of them
15:28:15 <slaweq> last topic for today
15:28:19 <slaweq> #topic Periodic
15:28:29 <slaweq> we have couple of jobs broken there
15:28:36 <slaweq> first one:
15:28:37 <slaweq> propose-translation-update is broken
15:28:50 <slaweq> https://zuul.openstack.org/build/5504f6afb3ce46fdad29afd9e0b59e19
15:29:41 <slaweq> it seems that it's broken since 29.05
15:29:45 <slaweq> https://zuul.openstack.org/buildset/779c07f580cb42fabeba7ad5e6640a1c
15:29:52 <ralonsoh> I'll check installing it manually. Maybe this is just a problem with some pypi versions
15:29:56 <ykarel> i heard it's related to py3.6
15:30:08 <ralonsoh> can we move it to 38?
15:30:17 <ykarel> there are some constraints that i hear
15:30:22 <slaweq> ralonsoh probably we should :)
15:30:28 <ykarel> frickler likely have more context on ^
15:30:47 <ykarel> but it's a known issue, not just for neutron
15:31:13 <slaweq> ykarel ok, thx. So lets monitor it for now :)
15:31:18 <lajoskatona> ykarel: so perhaps common soultion will be for it
15:31:27 <ykarel> yes
15:32:04 <slaweq> next one is about openstack-tox-py39-with-oslo-master which is also broken
15:32:45 <ralonsoh> do you have a link?
15:32:49 <slaweq> https://zuul.openstack.org/build/7aef47045c9649a58d28d87e86839f1e
15:33:29 <ykarel> ERROR: InterpreterNotFound: python3.9
15:33:41 <ykarel> as it's running on ubuntu-focal
15:33:42 <slaweq> yeah
15:33:45 <slaweq> but it's strange for me
15:33:58 <slaweq> similar job with neutron-lib master is fine https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_687/periodic/opendev.org/openstack/neutron/master/openstack-tox-py39-with-neutron-lib-master/6875d5b/job-output.txt
15:34:03 <slaweq> also on focal
15:35:29 <ralonsoh> https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/842646
15:35:34 <ralonsoh> that was added 9 days ago
15:35:43 <ralonsoh> when moving from 36 to 39
15:35:54 <slaweq> yeah together with other jobs
15:36:17 <slaweq> like the one with neutron-lib-master
15:36:23 <ralonsoh> actually there is a comment from frickler saying "This doesn't work, bionic lacks py39."
15:36:34 <lajoskatona> yes as both is on focal it is strange
15:37:16 <ralonsoh> but in focal, the default interpreter is 38, right?
15:37:23 <ralonsoh> you need to manually install 39
15:37:38 <lajoskatona> you are right,
15:37:43 <slaweq> ok, I will compare those jobs and check why it's working differently
15:37:50 <lajoskatona> so in one case we got an image with preinstalled py39?
15:37:50 <slaweq> and I will report bug for it
15:37:55 <ralonsoh> thanks
15:38:10 <slaweq> #action slaweq to report bug and check failing openstack-tox-py39-with-oslo-master periodic job
15:38:16 <slaweq> last one:
15:38:22 <slaweq> neutron-ovn-tempest-ovs-master-fedora is broken, but it seems that it's not related to Neutron
15:38:34 <slaweq> there is one test failing there since at least few days
15:38:40 <slaweq> but it's not really neutron related test
15:38:51 <slaweq> so maybe we should simply blacklist that test?
15:39:06 <slaweq> https://zuul.openstack.org/build/f265a35aa43649d1baa291d9e711b359
15:39:37 <ralonsoh> what's the issue? a missing image?
15:39:46 <slaweq> maybe we can ask glance/nova teams if they know that issue already
15:40:21 <slaweq> ralonsoh I didn't check it but my guess is that this test is creating image from running vm and can't find it later
15:40:51 <ralonsoh> that's curious: this image disappears in a gap of 2 seconds
15:40:52 <ralonsoh> but yes
15:40:59 <ralonsoh> Nova folks could provide more info
15:41:14 <mlavalle> +1
15:41:16 <slaweq> I will talk with them tomorrow and I will report bug if necessary
15:41:20 <lajoskatona> +1
15:41:22 <ykarel> +1
15:41:29 <ralonsoh> because this is not breaking the CI, we can leave it
15:41:32 <slaweq> #action slaweq to talk with nova team about issue in
15:41:33 <opendevreview> Merged openstack/os-vif master: Delete trunk bridges to avoid race with Neutron  https://review.opendev.org/c/openstack/os-vif/+/841499
15:41:57 <slaweq> * issue in neutron-ovn-tempest-ovs-master-fedora
15:42:37 <slaweq> ok, that's all what I had for today
15:42:41 <slaweq> anything else You want to discuss?
15:42:45 <slaweq> ahh, sorry
15:42:45 <ralonsoh> two topics
15:42:46 <ralonsoh> quick
15:42:48 <slaweq> one last thing
15:42:55 <slaweq> but ralonsoh go on with Yours
15:42:58 <ralonsoh> please, go on
15:43:02 <slaweq> #topic On Demand agenda
15:43:08 <slaweq> thx
15:43:12 <slaweq> I will be fast
15:43:18 <slaweq> next week I will be in Berlin
15:43:27 <slaweq> should I cancel ci meeting?
15:43:33 <slaweq> or do You want to do it?
15:43:39 <ralonsoh> ok to cancel it
15:43:42 <slaweq> I will not have time to attend probably
15:43:46 <mlavalle> +1
15:43:49 <ykarel> +1
15:44:00 <lajoskatona> good that you mention it, I will be there too, so I will cancel the team meeting
15:44:09 <slaweq> ++
15:44:11 <slaweq> thx
15:44:19 <mlavalle> ok
15:44:26 <mlavalle> no meetings next Tuesday
15:44:28 <slaweq> ok, ralonsoh: Your turn 😃
15:44:34 <ralonsoh> thanks
15:44:35 <ralonsoh> about the sqlalchemy issues, please add the [sqlachemy-20] prefix in the description
15:44:42 <ralonsoh> and sorry for so many of them
15:44:55 <ralonsoh> but they are difficult to spot
15:45:01 <ralonsoh> and second
15:45:11 <ralonsoh> os-ken UT is based on "nose" library
15:45:20 <ralonsoh> totally unsupported in py310
15:45:28 <ralonsoh> and "nose2" is not a replacement
15:45:39 <ralonsoh> so the idea is to move UT to unittest
15:45:49 <ralonsoh> of course, as usual, this is not trivial nor easy
15:45:55 <ralonsoh> but we have time for this
15:45:58 <ralonsoh> that's all
15:45:59 <lajoskatona> ralonsoh: good catch
15:46:09 <ralonsoh> btw, I'll open a bug for this
15:46:14 <slaweq> ralonsoh++
15:46:24 <slaweq> please remember that we use storyboard for os-ken bugs
15:46:28 <ralonsoh> sure
15:46:49 <slaweq> about prefix [sqlachemy-20]
15:46:57 <slaweq> do You want it to be added to bug titles in LP?
15:47:00 <slaweq> or to the patches?
15:47:00 <ralonsoh> yes
15:47:08 <ralonsoh> just to group the "shame"...
15:47:10 <slaweq> or both?
15:47:21 <ralonsoh> LP is fine, I think
15:47:28 <mlavalle> ok
15:47:29 <slaweq> maybe better would be to add such tag to the bugs in LP?
15:47:37 <slaweq> instead of title's prefix
15:47:37 <ralonsoh> agree with this too
15:47:43 <ralonsoh> perfect then
15:48:08 <lajoskatona> +1 for tag
15:48:20 <slaweq> ok, so lets add tag sqlalchemy-20 for such bugs
15:49:18 <slaweq> if there is nothing else for today, I think we can finish this meeting
15:49:26 <ykarel> just one thing
15:49:28 <ykarel> i have pushed some patches to devstack and grenade to make ovn-grenade job green, testing in https://review.opendev.org/c/openstack/neutron/+/836141
15:49:35 <slaweq> sure ykarel
15:49:46 <ykarel> please check these, this will also be needed for tick-tick upgrades
15:49:54 <slaweq> ++ thx a lot
15:49:59 <slaweq> I will check them tomorrow
15:50:09 <ykarel> and also one backport from lajoskatona https://review.opendev.org/c/openstack/neutron/+/837681
15:50:22 <ykarel> ^ needed for stable/wallaby too as pyroute2 updated there
15:50:56 <mlavalle> I'll check the backport
15:51:23 <slaweq> me too
15:52:41 <slaweq> ok, now I think we can finish the meeting :)
15:52:46 <slaweq> thx for attending
15:52:53 <slaweq> have a great week and see You all online
15:52:56 <bcafarel> o/
15:52:58 <ralonsoh> bye!
15:53:02 <slaweq> and lajoskatona see You in Berlin in person finally :)
15:53:04 <lajoskatona> o/
15:53:08 <ralonsoh> hehehe
15:53:12 <mlavalle> o/
15:53:12 <slaweq> #endmeeting