16:00:25 <bauzas> #startmeeting nova
16:00:25 <opendevmeet> Meeting started Tue Jul 15 16:00:25 2025 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:25 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:25 <opendevmeet> The meeting name has been set to 'nova'
16:00:46 <bauzas> #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
16:01:05 <bauzas> howdy folks, I'll be your humble chair for today's meeting
16:01:08 <bauzas> who's around ?
16:01:18 <elodilles> o/
16:01:31 <fwiesel> o/
16:01:36 <ratailor_> o/
16:01:56 <gmaan> o/
16:02:30 <gibi> o/
16:03:00 <bauzas> okay, let's start
16:03:08 <bauzas> #topic Bugs (stuck/critical)
16:03:10 <bauzas> doh
16:03:15 <bauzas> #topic Bugs (stuck/critical)
16:03:40 <bauzas> #info One Critical bug
16:03:42 <gibi> https://bugs.launchpad.net/nova/+bug/2116852
16:03:46 <gibi> nova-ceph-multistore job fails constantly with memory allocation error Edit
16:03:47 <bauzas> #link https://bugs.launchpad.net/nova/+bug/2116852
16:03:54 <bauzas> yeah
16:03:58 <bauzas> just saw that
16:04:06 <bauzas> any proposed patch ?
16:04:09 <gibi> patch to sidable the single tempest test to unblock the gate: https://review.opendev.org/c/openstack/tempest/+/954949
16:04:21 <bauzas> ack
16:04:29 <gibi> patch that proves that it is caused by ceph package update https://review.opendev.org/c/openstack/nova/+/954956
16:04:40 <gibi> with downgraded packaged the test is green
16:04:40 <gmaan> did recheck again this morning. I think gerrit downtime did not work last night
16:04:49 <bauzas> thanks gmaan
16:05:16 <gibi> I'm trying to report a bug to ceph upstream but I does not seem to be able to register in their tracker successfully
16:05:58 <gmaan> gibi: nice, you want to go with your nova change with downgraded package or tempest change to skip test?
16:06:15 <gibi> I suggest to skip the test until we get some feedback from the ceph folks
16:06:24 <gibi> I don't like forcing packages manually
16:06:28 <gmaan> ok, make sense
16:06:31 <bauzas> cool
16:06:42 <gibi> does anybody have active acc to the ceph bug tracker?
16:06:48 <gibi> https://tracker.ceph.com/login
16:07:00 <gibi> mine is blocked on "Your account was created and is now pending administrator approval.
16:07:20 <gibi> (and yes they disabled the github issues in the repo)
16:07:47 <gmaan> not me, I have not created yet
16:08:16 <gibi> anyhow lets see what happens. If I cannot report a bug upstream I will use my RH powers downstream
16:08:27 <gibi> we can move on :)
16:08:33 <bauzas> ack
16:09:00 <bauzas> #topic Gate status
16:09:07 <bauzas> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:09:11 <gibi> we rae blocked by the above bug
16:09:17 <gibi> as nova-ceph-multistore is red
16:09:25 <bauzas> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status
16:09:31 <bauzas> yeah
16:09:38 <bauzas> and the periodics are in a bad shape too
16:10:26 <gibi> centos jobs failing with
16:10:27 <gibi> ./stack.sh: line 504: /opt/stack/devstack/tools/outfilter.py: Permission denied
16:10:51 <gmaan> latest run seems passing
16:10:51 <gibi> but I did not dig deeper
16:10:53 <gmaan> #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-centos9-stream-fips&skip=0
16:11:27 <gmaan> tempest run  them periodic nightly and at least last two run pass
16:11:44 <gibi> the fip one seems to be different
16:11:47 <gmaan> though did not dig why outfilter.py: Permission denied error
16:12:03 <gibi> https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-9-stream&skip=0
16:12:09 <gibi> this fails with the outfilter
16:12:40 <gmaan> ah another c9s job
16:12:50 <bauzas> all c9s jobs seem impacted
16:13:13 <gmaan> its same error in fips  job also
16:13:15 <gmaan> #link https://zuul.opendev.org/t/openstack/build/e8f309cd649b4f0ca38ab7777e99de9f
16:13:23 <gmaan> which passed in recent run
16:14:27 <elodilles> it seems to be passing on master but not on stable branches
16:15:35 <gmaan> last run on stable was 2025-07-12 and we are seeing passing on master after that
16:15:59 <gmaan> maybe we can run it manually if its there in experimental pipeline or wait for next run
16:17:26 <gmaan> it seems they are not in experimental queue https://github.com/openstack/nova/blob/stable/2025.1/.zuul.yaml#L874
16:17:57 <gmaan> sorry I found it
16:17:59 <gmaan> #link https://github.com/openstack/nova/blob/stable/2025.1/.zuul.yaml#L899
16:18:38 <gmaan> checked experimental in this #link https://review.opendev.org/c/openstack/nova/+/952960
16:19:20 <bauzas> cool
16:19:22 <bauzas> moving on ?
16:19:25 <gibi> one more
16:19:28 <bauzas> k
16:19:59 <gibi> https://bugs.launchpad.net/grenade/+bug/2116336 this happens semi frequently but it is annoying as it makes the zuul feedback very slow
16:20:12 <gibi> grenade times out on dumping DBs
16:20:24 <gibi> so I proposed https://review.opendev.org/c/openstack/grenade/+/954629 to prevent looong timeout
16:20:35 <gibi> but I don't have any idea why it hangs
16:20:41 <gibi> I found nothing in the logs
16:21:33 <gibi> if somebody has ideas then please take a look at the bug
16:21:40 <bauzas> what a lovely week
16:22:23 <gibi> (I have no more)
16:22:45 <bauzas> this is fine (c) dog
16:22:55 <bauzas> okay, moving on
16:23:59 <bauzas> #topic tempest-with-latest-microversion job status
16:24:06 <bauzas> #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-with-latest-microversion&skip=0
16:24:18 <bauzas> red red red red
16:24:22 <gmaan> no updates on this. did not get chance to work last week
16:24:30 <bauzas> no worries
16:24:33 <bauzas> moving on
16:24:45 <bauzas> #topic Release Planning
16:24:51 <bauzas> #link https://releases.openstack.org/flamingo/schedule.html
16:24:53 <gmaan> still 21 more tetts to fix
16:24:59 <bauzas> #info Nova deadlines are set in the above schedule
16:25:29 <bauzas> we have around 7 weeks before ff, if I recount correctly
16:26:09 <bauzas> #topic Review priorities
16:26:29 <bauzas> #link https://etherpad.opendev.org/p/nova-2025.2-status
16:26:36 <bauzas> #topic OpenAPI
16:27:33 <gmaan> last I checked yesterday, still 18 changes yet to review.
16:27:33 <bauzas> #link: https://review.opendev.org/q/topic:%22openapi%22+(project:openstack/nova+OR+project:openstack/placement)+-status:merged+-status:abandoned
16:27:54 <bauzas> #info 18 remaining.
16:27:54 <bauzas> doh, jinxed
16:28:39 <bauzas> #topic Stable Branches
16:28:52 <bauzas> elodilles: happy to speak up ?
16:28:55 <elodilles> yepp
16:28:59 <elodilles> #info stable branches (stable/2025.1 and stable/2024.*) seem to be in OK state
16:29:09 <elodilles> well, i mean i haven't seen any issue
16:29:24 <elodilles> release nova stable versions:
16:29:31 <elodilles> 31.0.1 (2025.1 Epoxy) https://review.opendev.org/c/openstack/releases/+/955058
16:29:37 <elodilles> 30.0.2 (2024.2 Dalmatian) https://review.opendev.org/c/openstack/releases/+/947847
16:29:41 <elodilles> 29.2.2 (2024.1 Caracal) https://review.opendev.org/c/openstack/releases/+/954474
16:29:46 <elodilles> (the latter is preparation for 2024.1 Caracal to move to Unmainained in October)
16:30:08 <elodilles> they need a release liaison / PTL Code-review+1
16:30:43 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:31:00 <elodilles> that's it from me about stable branches
16:31:06 <elodilles> bauzas: back to you
16:31:32 <bauzas> cool
16:32:00 <bauzas> #topic vmwareapi 3rd-party CI efforts Highlights
16:32:05 <bauzas> fwiesel: there ?
16:32:10 <fwiesel> No updates from my side.
16:32:48 <fwiesel> Back to you, bauzas
16:32:54 <bauzas> ack
16:33:03 <bauzas> #topic Gibi's news about eventlet removal.
16:33:12 <gibi> hey that is me
16:33:21 <bauzas> gibi: anything to say except that you're awaiting eagerly for my reviews ?
16:33:30 <gibi> I was busy trying to unblock to land the last approved patch
16:33:50 <gibi> I'm waiting review starting at https://review.opendev.org/c/openstack/nova/+/954990
16:34:23 <gibi> sorry wrong link
16:34:27 <gibi> https://review.opendev.org/c/openstack/nova/+/948187/24
16:34:31 <gibi> this is the good one
16:34:54 <gibi> https://review.opendev.org/c/openstack/nova/+/954990 is valid but it has a rabbit hole we need to go into but maybe not right now.
16:35:13 <dansmith> gibi: I think we should go ahead with removing the eventlet dependency with that patch, no worse than today
16:35:15 <gibi> I have to go back and read dansmith's answer but I think we will procide with it
16:35:24 <gibi> yepp
16:35:38 <gibi> thanks dansmith for helping out
16:35:46 <dansmith> gibi: I'm POCing a proper fix to scatter-gather/multi-list that will make it do the thing we were thinking it does
16:35:57 <gibi> cool
16:36:04 <gibi> that will be an interesting patch :)
16:36:10 <dansmith> I'll try to get that working and proposed and then will try to circle back on reviews
16:36:16 <gibi> awesome
16:36:17 <dansmith> I also have outstanding glance pings
16:36:31 <dansmith> so, I don't want to over-promise :)
16:36:37 <gibi> no worries :)
16:36:46 <bauzas> excellent thanks
16:36:50 <gibi> I have no other news
16:37:21 <bauzas> #topic Open discussion
16:37:25 <bauzas> (crickets)
16:37:30 <bauzas> anything anyone ?
16:38:06 <gibi> -
16:38:09 <bauzas> cool
16:38:31 <bauzas> I'll be honest, I don't have the same energy than Uggla about skimming the long list of untriaged bugs
16:38:40 <bauzas> but I can give the baton to any volunteer
16:38:53 * gibi looks at a distant corner
16:39:13 <bauzas> thanks all then :)
16:39:20 <bauzas> see you next week
16:39:21 <gibi> thanks folks
16:39:22 <bauzas> #endmeeting