16:02:07 <Uggla> #startmeeting nova
16:02:07 <opendevmeet> Meeting started Tue May  6 16:02:07 2025 UTC and is due to finish in 60 minutes.  The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:07 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:02:07 <opendevmeet> The meeting name has been set to 'nova'
16:02:14 <bauzas> \o
16:02:17 <Uggla> Hello everyone
16:02:27 <Uggla> awaiting a moment for people to join.
16:02:50 <elodilles> o/
16:02:54 <fwiesel> o/
16:03:17 <gmaan> o/
16:04:03 <Uggla> thanks bauzas for last week meeting.
16:04:11 <bauzas> np
16:04:11 <gibi> o/
16:04:47 <Uggla> #topic Bugs (stuck/critical)
16:04:53 <Uggla> #info No Critical bug
16:05:01 <Uggla> #topic Gate status
16:05:11 <Uggla> #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs
16:05:19 <Uggla> #link https://etherpad.opendev.org/p/nova-ci-failures-minimal
16:05:33 <Uggla> #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status
16:05:39 <Uggla> #info Please look at the gate failures and file a bug report with the gate-failure tag.
16:05:44 <Uggla> #info Please try to provide meaningful comment when you recheck
16:06:13 <Uggla> If I understood well, the gate was blocked by this https://review.opendev.org/c/openstack/nova/+/948392 last week.
16:06:27 <gibi> jepp
16:06:30 <gibi> fix is landed
16:06:31 <Uggla> It is landed now.
16:06:48 <Uggla> It the gate looks good. Please tell me if I'm wrong.
16:06:54 <sean-k-mooney> the cyborg and barbican jobs are still broken i think
16:06:54 <gibi> I have hit by https://bugs.launchpad.net/glance/+bug/2109428 multiple times recently. Not a blocker but definitly a source of rechecks
16:07:05 <gibi> sean-k-mooney: yepp those non-votings are broken
16:07:08 <gibi> bugs are filed
16:07:25 <sean-k-mooney> i may try and find time to go fix them
16:07:35 <sean-k-mooney> they are both broken on the lack of a pyproject.toml
16:07:40 <gibi> yepp
16:07:47 <Uggla> good to know thx gibi
16:07:51 <gibi> https://bugs.launchpad.net/barbican/+bug/2109584
16:07:54 <gibi> https://bugs.launchpad.net/openstack-cyborg/+bug/2109583
16:07:56 <sean-k-mooney> if i can fix it without having to install the service locally i might give it a try
16:08:01 <sean-k-mooney> i can refence thos bugs
16:08:07 <gibi> thanks
16:08:18 <gmaan> I think we need to make pyproject.toml for all projects in openstack otherwise they will break slowly at some point
16:08:30 <gibi> gmaan: I agree
16:09:21 <Uggla> should we track this ?
16:09:41 <sean-k-mooney> gmaan: yes we will
16:09:56 <sean-k-mooney> Uggla: nova is mostly doen stephen and i tried to do this 2 years ago
16:10:01 <sean-k-mooney> to get ahead of things breaking
16:10:12 <sean-k-mooney> so nova and placment are done
16:10:17 <sean-k-mooney> i need to check os-*
16:10:31 <sean-k-mooney> but for the libs we are responible for it trivial
16:11:03 <sean-k-mooney> ok os-vif is still pending
16:11:09 <Uggla> any bug / blueprint to refer to this work ?
16:11:15 <sean-k-mooney> ill start working on them and ping folks to review
16:11:26 <gmaan> ++
16:11:51 <Uggla> sean-k-mooney++
16:12:05 <Uggla> anything else ?
16:12:23 <Uggla> moving on to next item.
16:12:31 <Uggla> #topic tempest-with-latest-microversion job status
16:12:39 <Uggla> #link https://zuul.opendev.org/t/openstack/builds?job_name=tempest-with-latest-microversion&skip=0
16:12:56 <Uggla> I have just discussed with gmann about it.
16:13:30 <Uggla> gmann is progressing on this periodic job.
16:13:59 <Uggla> gmaan, I let you give a quick status if you wish.
16:14:05 <gmaan> sure
16:14:21 <gmaan> it is not worst than what i suspected, only 23 tests failing
16:14:30 <gmaan> I started fixing those one by one
16:14:32 <gmaan> #link https://review.opendev.org/q/topic:%22latest-microversion-testing%22
16:14:46 <gmaan> hypervisor are fixed, I will fix a few more today and this week
16:15:01 <sean-k-mooney> do you know if some of them are invalid failure
16:15:07 <gmaan> that is ^^ topic I am adding all changes to, feel free to review/comment
16:15:13 <sean-k-mooney> i.e. the test is depending on an older microverion behavior
16:15:25 <sean-k-mooney> and should be skipped when using latest
16:15:30 <gmaan> sean-k-mooney: not invalid, either we need to fix schema or cap the test with min/max microversions
16:15:46 <sean-k-mooney> ok ya that actully what i was wondering
16:15:49 <gmaan> sean-k-mooney: yeah, for example hypervisor uptime test should run till 2.87
16:15:51 <sean-k-mooney> can we express in tempst
16:16:03 <sean-k-mooney> that this test has a max rather then just min verion requirement
16:16:05 <gmaan> yes with 'max_microversion'
16:16:09 <sean-k-mooney> ack
16:16:26 <sean-k-mooney> ah i see that how your fixing it in https://review.opendev.org/c/openstack/tempest/+/948490
16:16:28 <sean-k-mooney> cool
16:16:38 <gmaan> yeah ^^
16:17:12 <gmaan> sometime we need to refactor test but not bug deal.
16:17:46 <gmaan> that is all on these, fell free to review, as I am only active core in tempest I will merge them but keep it open for sometime if anyone would like to review
16:18:14 <gmaan> most probably, i will keep all fixs open till job is green
16:18:14 <Uggla> thx gmaan
16:19:12 <Uggla> #topic Release Planning
16:19:19 <Uggla> #link https://releases.openstack.org/flamingo/schedule.html
16:19:24 <Uggla> #link https://releases.openstack.org/flamingo/schedule.html
16:19:56 <Uggla> The patch about nova deadlines has been merged so I think it is ok.
16:20:41 <Uggla> Please let me know if it is not the case or if something is wrong.
16:20:59 <Uggla> #topic Review priorities
16:21:06 <Uggla> #link https://etherpad.opendev.org/p/nova-2025.2-status
16:22:00 <Uggla> I'd like to progress on openapi and I'll try to check with stephenfin about it.
16:22:52 <Uggla> #topic Stable Branches
16:23:09 <Uggla> elodilles, the mic is yours
16:23:19 <elodilles> thanks Uggla , so
16:23:22 <elodilles> #info stable/2023.2 (bobcat) is End of Life, branch is deleted (tag: bobcat-eol)
16:23:31 <elodilles> #info maintained stable branches: stable/2025.1, stable/2024.2, stable/2024.1
16:23:49 <elodilles> down to 3 maintained branches again ;)
16:24:02 <elodilles> #info nova stable release from stable/2024.1 is out (29.2.1)
16:24:15 <elodilles> #info not aware of any stable gate failure
16:24:30 <elodilles> #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci
16:24:54 <elodilles> we had a broken gate on stable/2025.1 osc-placement,
16:25:07 <elodilles> but it is now fixed
16:25:15 <elodilles> thanks all for the help :)
16:25:30 <elodilles> and i think that's all from me
16:25:36 <elodilles> Uggla: back to you
16:25:39 <Uggla> elodilles, fyi 947847: nova: Release 2024.2 Dalmatian 30.0.1 | https://review.opendev.org/c/openstack/releases/+/947847 should be ok now.
16:25:52 <Uggla> As I have done 948811: Add uc check alternative method | https://review.opendev.org/c/openstack/requirements/+/948811
16:26:04 <elodilles> Uggla: thanks for working on that!
16:26:29 <Uggla> the path is on master with a backport to the stable branch.
16:26:30 <elodilles> i've just added a comment on your patch o:)
16:26:53 <Uggla> so it needs reviews.
16:27:22 <Uggla> but at least the verify is +1
16:27:51 <opendevreview> sean mooney proposed openstack/os-vif master: add pyproject.toml to support pip 23.1  https://review.opendev.org/c/openstack/os-vif/+/899946
16:28:19 <Uggla> #topic vmwareapi 3rd-party CI efforts Highlights
16:28:35 <fwiesel> #info No updates
16:28:46 <fwiesel> Uggla: Back to you
16:28:52 <Uggla> fwiesel, btw welcome back, I hope you are good.
16:29:20 <Uggla> #topic Gibi's news about eventlet removal.
16:29:21 <fwiesel> Thanks! Just still catching up with everything.
16:29:24 <gibi> o/
16:29:31 <Uggla> #link Series: https://gibizer.github.io/categories/eventlet/
16:29:48 <gibi> so last week we saw nova-scheduler running with native threading. There is a blogpost about it
16:29:58 <bauzas> had no time to review those patches yet :(
16:29:59 <gibi> this week I started testing the non happy path of the scatter-gather
16:30:05 * bauzas hardly crying
16:30:25 <gibi> good news both mysql server and pymysql client can be configured with timeouts to avoid handing gather threads
16:30:47 <gibi> I will add a new doc about it in tree
16:31:02 <gibi> bad news, the oslo.service threading backend uses forks
16:31:23 <gibi> which does not play nice with out global threadpools initialized before the service workers are forked off
16:31:32 <gibi> we have workarounds
16:31:41 <gibi> details are in https://review.opendev.org/c/openstack/oslo.service/+/945720/comments/fb5d6632_f0eaf102
16:31:51 <gibi> there are two patches to look at
16:32:09 <gibi> https://review.opendev.org/c/openstack/nova/+/948064?usp=search now with test coverage
16:32:15 <gibi> and
16:32:16 <gibi> https://review.opendev.org/c/openstack/nova/+/948437?usp=search
16:32:40 <gibi> this week I will work on cleaning up the long service to make more patches ready to review
16:32:43 <gibi> that is all
16:32:46 <gibi> Uggla: back to you
16:33:16 <sean-k-mooney> gmaan: Uggla: just a quick update on the pyproject.toml change if i may
16:33:28 <sean-k-mooney> https://etherpad.opendev.org/p/pep-517-and-pip-23 is my old ether pad to track that work
16:33:33 <Uggla> thx gibi
16:33:37 <Uggla> sean-k-mooney, sure go ahead
16:33:40 <sean-k-mooney> and only the os-vif change above is not mergd for nova
16:33:47 <sean-k-mooney> so i rebased and appvoed that
16:33:55 <gmaan> just saw, ++
16:33:57 <sean-k-mooney> once that is landed we shoudl be good
16:34:07 <Uggla> \o/
16:34:43 <sean-k-mooney> anyway that was all on that topic
16:35:09 <Uggla> gibi, just a question you have said : the oslo.service threading backend uses forks  it means real processes and not threads ?
16:35:24 <gibi> Uggla: no we want worker processes with thread pool
16:35:26 <gibi> pools
16:35:44 <gibi> but it uses os.fork to create the worker from the main process
16:35:44 <sean-k-mooney> context is oslo service
16:35:55 <sean-k-mooney> allows you to have multiple workers
16:36:00 <gibi> fork copies the state of the process
16:36:01 <sean-k-mooney> we use that in the schduler and conductor
16:36:12 <sean-k-mooney> but not nava-comptue or the api
16:36:17 <gibi> we would need os.spawn to get a totally new worker process
16:36:22 <gibi> without inheriting state
16:37:04 <Uggla> ok I think I have understood the pb.
16:37:04 <gibi> the wa is to reset the problematic state
16:37:13 <gibi> after the fork
16:37:25 <gibi> but I think the real solution would be os.spawn
16:37:48 <gibi> but changing to that is not super simple as oslo.config has non pickleable lambdas :/
16:37:56 <gibi> details are in the linked gerrit comment
16:38:07 <gibi> and are up in the today IRC log
16:38:16 <dansmith> we can reset after the fork (as you're doing) or reset before the fork, as I suspect will also work and be less janky
16:38:23 <gibi> yepp
16:38:30 <gibi> I will try dansmith's suggestion next
16:38:40 <gibi> while we wait for the oslo folks to respond
16:38:47 <sean-k-mooney> https://github.com/openstack/oslo.config/blob/d6e5c96d6dbeec0db974dfb8afc8e508b74861e5/oslo_config/cfg.py#L1387
16:38:52 <dansmith> I agree the real solution is spawn, but we probably don't want to wait
16:38:55 <sean-k-mooney> that is there only use of lambda i think by the way
16:39:23 <gibi> dansmith: yepp we won't wait, we will go with the WA, and adapt when oslo makes the move
16:39:52 <dansmith> +1
16:40:39 <Uggla> moving on
16:40:49 <Uggla> #topic Open discussion
16:41:30 <Uggla> fwiesel, wants to propose something about Cross hypervisor resize.
16:41:45 <gibi> I guess that was last week's topic
16:42:07 <Uggla> yep but I understood you wanted to discuss it more.
16:42:09 <gibi> or maybe a continuation
16:42:21 <fwiesel> Yes, but there was limited feedback. I mean, if no one feels strongly about it, then I can go forward with a blueprint.
16:43:39 <fwiesel> We would like to allow mobility between two hypervisors, and I was thinking the cross-cell migration might already cover it to a large degree.
16:44:05 <fwiesel> The question is though, if that is a use-case you would feel worthwhile supporting or rather not.
16:44:50 <dansmith> I think it _has_ to be more like cross-cell migration than regular
16:45:10 <dansmith> I'm a bit mixed on whether or not I think this is worth it, because I suspect there will be a lot of gotchas in the image properties
16:45:24 <dansmith> and because you can pretty much do this with snapshot yourself now
16:45:37 <dansmith> and because we won't really be able to test it regularl I don't think
16:46:27 <gibi> yeah testing this will be painful
16:46:32 <fwiesel> Okay, got it. That's fine.
16:46:55 <sean-k-mooney> dansmith: specicly like updating the hw_vif_model for say the vmware one to virtio if they dont happen to have common values already
16:46:55 <dansmith> yeah, all that kind of stuff
16:47:21 <fwiesel> Well, we would set those in the flavours and that would override that.
16:47:39 <sean-k-mooney> so general question. is there anyting we know off that would prevent you form shelving, modifying them in glance and unshleving?
16:47:54 <fwiesel> No, it is more a usability issue.
16:48:02 <fwiesel> For our users.
16:48:08 <dansmith> sean-k-mooney: if the flavor you booted from was vmware-specific you can't unshelve to a libvirt-y one right?
16:48:18 <sean-k-mooney> fwiesel: so historically flavors are for amounts and images propereis are for changing how the devices are presented
16:48:31 <fwiesel> And we were thinking of using shared NFS shares to avoid going through image upload, etc...
16:48:36 <sean-k-mooney> fwiesel: so in the past the precendet was dont put device models in flavors
16:48:49 <sean-k-mooney> dansmith: correct
16:49:01 <sean-k-mooney> dansmith: what im thinkign is we dicussed allowing resize in the sheleved state
16:49:08 <dansmith> to me, snapshot, tweak, boot fresh is the best pattern here
16:49:18 <sean-k-mooney> so the workflow would be shelve. resize , update image properties and then unshleve
16:49:22 <dansmith> sean-k-mooney: that's a whole other conversation
16:49:48 <fwiesel> But I got it... hard to test so, no takers... Perfectly understandable.
16:49:49 <sean-k-mooney> ya im just confirming if there is a way to do that today with the api we have and i think the gaps are resize while shleved
16:50:23 <opendevreview> Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode  https://review.opendev.org/c/openstack/nova/+/948450
16:50:36 <fwiesel> Thanks for the feedback.
16:50:36 <sean-k-mooney> but ya snapshot. detach volumes/ports and boot new vms with thos is a valid workflow too
16:52:49 <Uggla> Shall I say it is ok for fwiesel to propose something that goes in that direction ?
16:53:44 <fwiesel> My understanding is, I won't propose things as it is doable with the existing API and implementing it within Nova is hard to test and maintain.
16:53:48 <dansmith> I think anyone can propose anything they want :)
16:54:02 <dansmith> fwiesel: ++
16:54:32 <Uggla> oh ok good.
16:56:11 <Uggla> We are almost at the top of the hour. Are you ok triaging a couple of bugs or would you prefer to do it next week ?
16:58:00 <Uggla> sean-k-mooney, dansmith ^^
16:59:48 <Uggla> ok no answer so you might be busy. So closing the meeting.
16:59:59 <Uggla> thanks all
17:00:02 <gibi> Uggla: thanks
17:00:06 <fwiesel> thanks everyone
17:00:10 <Uggla> #endmeeting