| *** ykarel__ is now known as ykarel | 04:36 | |
| *** zseguin_ is now known as zseguin | 06:13 | |
| *** gibi_ is now known as gibi | 08:12 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Move py312-threading to py313 https://review.opendev.org/c/openstack/nova/+/980458 | 12:51 |
|---|---|---|
| nicolairuckel | Do I understand it correctly, that this is done? https://blueprints.launchpad.net/nova/+spec/vtpm-live-migration | 14:22 |
| nicolairuckel | And all patches from here that are merged should belong to the implementation? https://review.opendev.org/q/topic:%22bp/vtpm-live-migration%22 | 14:22 |
| dansmith | yes, and "should" | 14:26 |
| nicolairuckel | thanks :) | 14:36 |
| Uggla | melwitt, gmaan, please can you review again https://review.opendev.org/c/openstack/nova/+/981085 it should not take long. | 14:55 |
| Uggla | Upstream meeting in ~1h. | 15:06 |
| opendevreview | Merged openstack/nova stable/2026.1: Fix Flamingo version in qemu matrix. https://review.opendev.org/c/openstack/nova/+/981085 | 15:21 |
| Uggla | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Mon Mar 23 16:00:31 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| gibi | o/ | 16:00 |
| Uggla | Hello everyone | 16:00 |
| nicolairuckel | o/ | 16:01 |
| fwiesel | o/ | 16:01 |
| lajoskatona | o/ | 16:02 |
| Uggla | Let's start | 16:03 |
| Uggla | #topic Bugs (stuck/critical) | 16:03 |
| Uggla | #info No Critical bug | 16:03 |
| bauzas | o/ | 16:04 |
| Uggla | #topic Gate status | 16:04 |
| elodilles | o/ | 16:04 |
| Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:04 |
| Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | 16:04 |
| Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | 16:05 |
| Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:05 |
| Uggla | #info Please try to provide a meaningful comment when you recheck | 16:05 |
| Uggla | TBH I have not looked at the status pages. Anything wrong with the gate ? | 16:05 |
| sean-k-mooney | o/ | 16:06 |
| Uggla | seems not so moving on | 16:07 |
| Uggla | #topic Release Planning | 16:07 |
| Uggla | #link https://releases.openstack.org/gazpacho/schedule.html | 16:07 |
| Uggla | #info Nova deadlines are set in the above schedule | 16:08 |
| Uggla | #info PTG etherpad for 2026.2 is available: https://etherpad.opendev.org/p/nova-2026.2-ptg | 16:08 |
| Uggla | #info This is a "work in progress document", but you can enter you topics at the bottom of the document | 16:08 |
| Uggla | I have seen gibi already entered something about eventlet, so thanks | 16:08 |
| Uggla | #info This week is final RCs, release should happen next week. | 16:09 |
| Uggla | #topic Review priorities | 16:10 |
| Uggla | #link https://etherpad.opendev.org/p/nova-2026.1-status | 16:10 |
| Uggla | I will open the new doc for 2026.2 this week. | 16:10 |
| Uggla | There is 979301: Update min support for Hibiscus | https://review.opendev.org/c/openstack/nova/+/979301 pending waiting for grenade update. | 16:12 |
| Uggla | Then we should be completely ok for 2026.1 | 16:12 |
| Uggla | #topic Stable Branches | 16:13 |
| * Uggla giving the mic to elodilles | 16:13 | |
| elodilles | thanks | 16:13 |
| elodilles | #info nova stable gates should be OK | 16:13 |
| elodilles | #info placement and osc-placement have blocked stable gates, fixes need stable cores review (see: https://review.opendev.org/q/topic:pin-setuptools-81-docs and https://review.opendev.org/q/topic:pin-setuptools-81-via-virtualenv-pkg-resources) | 16:13 |
| elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | 16:14 |
| elodilles | and that's all from me about stable | 16:14 |
| * elodilles is passing back the mic | 16:14 | |
| Uggla | thanks elodilles, that looks in a good shape. | 16:14 |
| Uggla | #topic vmwareapi 3rd-party CI efforts Highlights | 16:14 |
| Uggla | fwiesel anything for us ? | 16:15 |
| fwiesel | no updates from my side | 16:15 |
| Uggla | ok thx | 16:15 |
| Uggla | #topic Gibi's news about eventlet removal | 16:15 |
| gibi | o/ | 16:15 |
| gibi | two things | 16:15 |
| * Uggla I know about a blogpost, but giving the mic to gibi | 16:15 | |
| gibi | fwiesel: could you please take a look at the comment https://review.opendev.org/c/openstack/nova/+/973468/22#message-a3e61c625edd2a7a737962c5ea9913caa335d5fc it is about troubleshooting vmware CI with native threading | 16:16 |
| fwiesel | gibi: Sure, I'll check. | 16:16 |
| gibi | thanks | 16:16 |
| gibi | and the other is what Uggla hinted at. There is finally a new blogpost about Eventlet | 16:16 |
| gibi | https://gibizer.github.io/posts/Eventlet-Removal-Gazpacho/ | 16:16 |
| Uggla | 👍 | 16:17 |
| gibi | and that is it from me | 16:17 |
| Uggla | thanks gibi. | 16:17 |
| Uggla | #topic Nova using openstack sdk for neutron | 16:18 |
| * Uggla giving the mic to lajoskatona | 16:18 | |
| lajoskatona | o/ thanks | 16:18 |
| lajoskatona | I pushed a new ps for security-groups: https://review.opendev.org/c/openstack/nova/+/981141 | 16:18 |
| lajoskatona | and I polished the older patches also to make them ready for review for early H | 16:19 |
| lajoskatona | The security-group one is wip as it seems I still break something :-) | 16:19 |
| lajoskatona | After that I hope I can check what remains, that is for sure the port-binding API and some other small things, so that will come soon I hope | 16:20 |
| lajoskatona | that's it for the SDK work | 16:20 |
| sean-k-mooney | ack. so its breaking in tempest but not in the unit/fucntional tests | 16:21 |
| lajoskatona | exactly, so I have to read more logs | 16:21 |
| Uggla | oh not cool. | 16:22 |
| lajoskatona | life, if you touch something you break something :-( | 16:23 |
| Uggla | sometimes. | 16:24 |
| Uggla | #topic Open discussion | 16:24 |
| Uggla | I forget to say something in review priorities. | 16:25 |
| Uggla | https://review.opendev.org/q/topic:%22bp/generalize-sev-code%22 looks good to me. Except latest patch we agreed to remove from this serie. | 16:25 |
| Uggla | gibi from you point of view anything still not ok with the serie ? | 16:26 |
| gibi | I have to go back to that series | 16:26 |
| gibi | I lost context over last week | 16:26 |
| Uggla | thanks gibi, it will be great to merge that before starting new CC stuffs. | 16:27 |
| gibi | I had some negative feedback before but I guess that was addressed in the meantime | 16:27 |
| gibi | Uggla: sure, but that also needs two cores ;) | 16:28 |
| Uggla | gibi, yes I have not noticed remaining open things. | 16:28 |
| Uggla | I know, bauzas maybe ? | 16:28 |
| * bauzas is in a convo somewhere else | 16:29 | |
| bauzas | OK, drop me as a reviewer | 16:29 |
| Uggla | OK done for this one. | 16:30 |
| Uggla | Second, it seems I will be PTL again for H cycle. | 16:30 |
| gibi | Uggla: thanks for your continued service | 16:31 |
| elodilles | yepp, thanks Uggla \o/ | 16:31 |
| Uggla | thank you. | 16:31 |
| Uggla | Third point, we may have 2 x potential new slots for upstream bug triage. | 16:32 |
| Uggla | It is not settled at the moment. I need to think about how we will organize that. | 16:33 |
| Uggla | I will keep you posted via the ML | 16:34 |
| Uggla | that's all for me. Does anyone else want to discuss something ? | 16:35 |
| sean-k-mooney | am more a quick fyi | 16:36 |
| Uggla | sure | 16:36 |
| sean-k-mooney | so JayF raised a long stanidng issue with the ironic driver (startup time) | 16:36 |
| sean-k-mooney | i pent a few hourse playing aroudn with https://review.opendev.org/q/topic:%22startup-time%22 | 16:37 |
| sean-k-mooney | baisly i create a poc of a simulator to messuer the perfoamcne of it as we scaled out ironic nodes | 16:37 |
| sean-k-mooney | this uses our functional tests infrastucer and then i ran some profileing of the starut and had ai analise it | 16:38 |
| sean-k-mooney | so i just wanted to share that as it found we are don an quadtaric deep copy of the resouce providers | 16:38 |
| * JayF owes you a look on that chain but hasn't gotten there yet, apologies (not that my reviews are likely to be useful on the deep-in-nova stuff) | 16:38 | |
| sean-k-mooney | i dont know if this tool could be useful for other things liek similarting large env for schduler fileter or eventlet debuging | 16:39 |
| sean-k-mooney | btu i just wanted to share that it exists as a review incase anyone found it interesting | 16:39 |
| sean-k-mooney | the code quality/style is not what i woudl normally do | 16:39 |
| sean-k-mooney | since this was just a quick poc | 16:39 |
| sean-k-mooney | but it did produce some interesting results so ya that was all i had | 16:40 |
| gibi | sean-k-mooney: does the deep-copy expensive due to multiple compute nodes per compute host? | 16:40 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/980679 and https://review.opendev.org/c/openstack/nova/+/980676/1 | 16:40 |
| sean-k-mooney | gibi: yep | 16:40 |
| sean-k-mooney | so for 1000 noes we copery the resouce provider for the first one 999 times i blieve | 16:40 |
| gibi | ohh I see why that is bad :) | 16:41 |
| dansmith | this totally doesn't surprise me :) | 16:41 |
| Uggla | (OO° | 16:41 |
| Uggla | (OO) | 16:41 |
| sean-k-mooney | anyway im not sayign the poc patches are correct | 16:41 |
| dansmith | like I bet we probably have some things like that in runtime code too | 16:41 |
| sean-k-mooney | but locally they do sped thing up alot | 16:41 |
| sean-k-mooney | ya so this deepcopy is in get_provider_tree_and_ensure_root | 16:42 |
| gibi | sean-k-mooney: my only question is if this is the real bottleneck, ie the code, or is there any nova-ironic interaction at startup that is inherently slow due to network? | 16:42 |
| gibi | I imagine 1000 tree memcopy is still faster than 1000 network call to ironic | 16:43 |
| sean-k-mooney | on my laptop the delta between the first patch and last goes form 2 and ahalf minute sdown to 40 ish seconds | 16:43 |
| sean-k-mooney | i didnt try tweakign the latency ineject for the api calls | 16:43 |
| sean-k-mooney | but the similater in thory can add in some it defautl to 0 for the get node call at the momemnt | 16:43 |
| gibi | I though you have it in funct test, I'm sure that mock ironic API calls | 16:44 |
| dansmith | sean-k-mooney: and lots of cpu usage I guess? | 16:44 |
| sean-k-mooney | yep as i said i only spent a few hour on this at the weekend but i wanted to see if we coudl guest some emperical data | 16:44 |
| gibi | anyhow. Improvement is good. Question remains if we improved the critical path or not | 16:44 |
| sean-k-mooney | before the ironic cross project session | 16:44 |
| sean-k-mooney | the other atpch that helps is https://review.opendev.org/c/openstack/nova/+/980679/1 | 16:45 |
| sean-k-mooney | that kicks out the per node operation into a thread pool | 16:45 |
| dansmith | well, and if we improved it without introducing other issues (like the cache patch seems the easiest, but as <human> points out it may be problematic) | 16:45 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/980680/1 | 16:45 |
| sean-k-mooney | is a basi update to the reprot before/after the sersise but as i said i have not fully validate this as correct | 16:46 |
| sean-k-mooney | its easy to make thing fast if correctness is not a requirement | 16:46 |
| dansmith | and honestly two minutes to start up with lots of nodes may not be unreasonable, but certainly sucks if we can avoid it (especially if it's just burning cpu time copying stuff) | 16:46 |
| sean-k-mooney | dansmith: so on real ironci system its 15+ minutes | 16:46 |
| dansmith | yeah, uncool :) | 16:46 |
| sean-k-mooney | based on the commit messge i lied N=1000: 42.1 s (42.1 ms/node) — was 4 min 4 s (244 ms/node) | 16:47 |
| sean-k-mooney | so it was 4 minutes even without any network overhead | 16:47 |
| sean-k-mooney | i will also note the bigest boteel neck on the 1000 node case was in placmenet | 16:47 |
| sean-k-mooney | in the jsonschema validation | 16:47 |
| sean-k-mooney | but that partly an artifact fo me using the palcement fixture | 16:47 |
| dansmith | I think we can also put #squarepegroundhole on some of this | 16:48 |
| sean-k-mooney | anyway i just wanted to share one of the experiemts i did. im not sure if we coudl eveolve this simulator into a gerneicly useful perf tool | 16:49 |
| sean-k-mooney | dansmith: you had asked about a schduer simultor in the past so this might be a foundation to build that if we had time | 16:49 |
| Uggla | dansmith #squarepegroundhole what's that ? | 16:49 |
| dansmith | I guess maybe I'd like to have JayF (a mere human) look over this for ironic gotchas before we look too deep at the other things | 16:49 |
| dansmith | sean-k-mooney: yep, for sure | 16:49 |
| JayF | the biggest issue is that it's dangerous to heavily lean on a cache | 16:50 |
| dansmith | Uggla: ironic as a driver in nova is a square peg in a round hole (doesn't fit very well and has lots of ironic-specific issues like this) | 16:50 |
| JayF | because it completely breaks use cases where "node completes cleaning -> immediate re-deploy" which while it's not a very openstack-y use case, is very common | 16:50 |
| sean-k-mooney | Uggla: the reall issues is the compute agent was desgisn to have 1 pre compute node. the ironic driver has 1 agent for 1000s | 16:50 |
| dansmith | this ^ | 16:50 |
| JayF | we'd need ironic to close that loop on more of an event-driven basis vs a poll to fix that concern before relying more heabily on cache | 16:50 |
| JayF | admittedly my initial email was written assuming "ironic updates more stuff about ironic nodes" may be the path, I didn't expect sean-k-mooney to find low-hanging-fruit like he has | 16:51 |
| sean-k-mooney | JayF: ya i saw your coment on the cach. i think we shoudl actully start lookign at riping out the hasring stuff | 16:51 |
| sean-k-mooney | but also have a wider converation on this in general | 16:51 |
| JayF | even just if we put in knobs to let operators make the tradeoff, it might help, idk | 16:52 |
| JayF | I'll ensure I review those before the cross-project session | 16:52 |
| JayF | if a sync between just us two to refine before PTG is valuable, we can do that too | 16:52 |
| sean-k-mooney | ya perhaps. the one caveat ill say is im not activly working on this | 16:52 |
| sean-k-mooney | but you peaked my interest enouch to see if there was anythign obviously wrong | 16:53 |
| sean-k-mooney | or simple improvemtn we coudl make | 16:53 |
| JayF | It's a major issue for Ironic/Nova users; which are 25% of nova users per the survey :) hopefully we can catch the eyes of someone who has expertise, even just to lay the path out for me and/or my team (cid or clif) to implement | 16:54 |
| sean-k-mooney | JayF: i can chat to you after the meetign more if you like but it might be interestign to see if we could perhaps replciate this in devstack with the ironic fake driver | 16:54 |
| JayF | yeah, that sounds like a good path | 16:55 |
| JayF | and I've wanted to make ironic easier to hook up in devstack with fake driver | 16:55 |
| sean-k-mooney | i.e. to see if the imovpments in this emulated env traslate to a real env | 16:55 |
| JayF | maybe I'll spike on that between now:ptg | 16:55 |
| sean-k-mooney | Uggla: sorry that took more time then i was planning but i think this has been productive | 16:56 |
| JayF | we should take Uggla to "why Ironic driver is awesome and terrible 101" some day, I think :P | 16:56 |
| Uggla | JayF sure | 16:56 |
| dansmith | let him enjoy life, gawd | 16:56 |
| dansmith | that's like explaining santa | 16:57 |
| Uggla | Are we good ? | 16:57 |
| JayF | "And here's _tooth_fairy()" | 16:57 |
| JayF | yes, ty Uggla :) | 16:57 |
| Uggla | Anything else ? | 16:57 |
| Uggla | seems not. | 16:58 |
| dansmith | JayF: wait, what's your point about the tooth fairy? If you have a contact, I'm still waiting for $10 | 16:58 |
| Uggla | Quick update on bug scrubbing because we are at he top of the hour. | 16:58 |
| Uggla | #topic Bug scrubbing | 16:58 |
| Uggla | #info up to 199 (-7) | 16:58 |
| JayF | dansmith: I saw her walking out of "Budget Dentures". Scandalous! ;) | 16:59 |
| dansmith | lol | 16:59 |
| Uggla | I manage to go down 200. | 16:59 |
| Uggla | #link: https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:59 |
| Uggla | #link: https://truc.uggla.fr/ if you want to see the trend. | 16:59 |
| Uggla | JayF, dansmith, can tooth fairy do something for bugs ? | 17:00 |
| JayF | If she is a metaphor for the Ironic driver, she can create them ;) | 17:01 |
| Uggla | :) | 17:01 |
| Uggla | Time to close. | 17:01 |
| Uggla | thanks for joining this meeting. Have a nice day/evening and see you next week. | 17:02 |
| gibi | o/ | 17:02 |
| Uggla | #endmeeting | 17:02 |
| opendevmeet | Meeting ended Mon Mar 23 17:02:17 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.html | 17:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.txt | 17:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-03-23-16.00.log.html | 17:02 |
| elodilles | same to you, thanks o/ | 17:02 |
| lajoskatona | o/ | 17:02 |
| sean-k-mooney | melwitt: thanks for taking a look at https://review.opendev.org/c/openstack/nova/+/975872/5 | 18:27 |
| sean-k-mooney | melwitt: gibi do you thinks we could try and get the repoducer patch merged in the next few days https://review.opendev.org/c/openstack/nova/+/975859/4 while we dicuss the fix in https://review.opendev.org/c/openstack/nova/+/975872 | 18:28 |
| sean-k-mooney | it woudl be nice to make some progress on this again. im hoping to complete that bug work in the next week or two | 18:29 |
| melwitt | sean-k-mooney: ok yeah let me focus on that one then, I had started with the fix (obviously) | 18:29 |
| sean-k-mooney | mainly to avoid having to load context on this over and over again | 18:29 |
| sean-k-mooney | melwitt: ill definally look into your feedback on teh fix too | 18:29 |
| sean-k-mooney | just want to see if we can make incremental progress | 18:30 |
| gibi | sean-k-mooney: I don't immediately have capacity for that patch. Please ping me in couple of days | 18:31 |
| melwitt | sean-k-mooney: sure. thanks for letting me know | 18:31 |
| sean-k-mooney | melwitt: since i started wroking on this i have had time to look at the cybrog quota code and foudn its entrily non fucntional https://bugs.launchpad.net/openstack-cyborg/+bug/2143943 | 18:31 |
| sean-k-mooney | gibi: ack will do | 18:31 |
| melwitt | sean-k-mooney: wow that's wild | 18:32 |
| sean-k-mooney | i have a writeups attached https://launchpadlibrarian.net/851194002/quota-system.md but tldr is the safolding for qutas was built in the v1 api but never ported to v2 | 18:32 |
| sean-k-mooney | at this point i think im just oging to rip it out since it does not work and if we need to have native supprot in cybrog indepent of th enova fix go staight to unified limits | 18:33 |
| melwitt | yeah I think that's probably the best move to just go straight to unified limits | 18:35 |
| melwitt | if fixing the current thing is not trivial | 18:35 |
| sean-k-mooney | well its worked for 10 months and has been broken since 2019 | 18:35 |
| sean-k-mooney | so its not so much how hard is fixing it its is it worth addign a seperate quota api to cyborg at all at this point if we can just use unified limits | 18:36 |
| sean-k-mooney | the entiretly of the v1 api was deleted years ago and it never existed in v2 | 18:36 |
| dansmith | unified limits is easy that way.. saves a lot of api effort | 18:36 |
| melwitt | sean-k-mooney: adding a separate API = not trivial :) so yeah | 18:37 |
| sean-k-mooney | in cybrog qutoa was also only configurable via config before when it did work so there is also no prior art to revive | 18:37 |
| dansmith | that's like glance | 18:38 |
| dansmith | like glance _was_ before unified limits, I should say | 18:38 |
| sean-k-mooney | ack | 18:38 |
| dansmith | even if you had prior art, unified limits instantly makes you more modern and compatible with the way the other projects are going, IMHO, so that's a reason right there | 18:39 |
| sean-k-mooney | right now im not fucosing on cybrog as a standalone service "without nova" so if we fix the enfocement on the nova side then that closes the gap although at some point i shoudl add quta checks indepenet of nova spawn/resize code before we add attach/detach supprot | 18:40 |
| sean-k-mooney | anyway out of scope of nova but this will eventually get doen after the other higher priorty items | 18:41 |
| sean-k-mooney | ARQ binding is when a resouce is actully assigne dto an instnace so that is where we should be checking to prevent over quota usage longterm. | 18:42 |
| sean-k-mooney | i.e form parallel nova instance cretes | 18:43 |
| opendevreview | Merged openstack/nova master: Move py312-threading to py313 https://review.opendev.org/c/openstack/nova/+/980458 | 20:06 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!