| opendevreview | Artem Vasilyev proposed openstack/nova master: Add reproducer for bug #2147776 https://review.opendev.org/c/openstack/nova/+/984007 | 08:27 |
|---|---|---|
| opendevreview | Artem Vasilyev proposed openstack/nova master: Fix race between resize and update_available_resource for dedicated CPUs https://review.opendev.org/c/openstack/nova/+/984048 | 08:40 |
| opendevreview | Michel Nederlof proposed openstack/nova master: Add RBD XML update functionality for migration process https://review.opendev.org/c/openstack/nova/+/974032 | 08:53 |
| opendevreview | Michel Nederlof proposed openstack/nova master: Add RBD XML update functionality for migration process https://review.opendev.org/c/openstack/nova/+/974032 | 09:04 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [py313-threading]Reenable last scatter-gather unit test https://review.opendev.org/c/openstack/nova/+/984349 | 09:31 |
| gibi | fyi from #openstack-eventlet-removal | 09:45 |
| gibi | 09:30 < amorin> hey team, I am looking to start an openstack deployment without eventlet, | 09:45 |
| amorin | yup :) for now it's mostly dev purposes | 09:52 |
| sean-k-mooney | amorin: you can mostly do that with devstack today but i dont knwo fi any fo the other installer have an easy button to do it | 10:08 |
| amorin | I have my own way to install anyway :) for nova it's pretty strait forward with this env var, that's perfect | 10:09 |
| amorin | I think I already know the answer, but placement is also eventlet free, right? | 10:12 |
| DominikDanelski[m] | sean-k-mooney: Hello, did you have time recently to look at proposed nova-spec (https://review.opendev.org/c/openstack/nova-specs/+/978570?usp=search), the related code changes, or the smaller bugfix related to allocations (https://review.opendev.org/c/openstack/nova/+/968446?usp=search)? | 10:16 |
| sean-k-mooney | amorin: placement never used eventlet | 10:17 |
| sean-k-mooney | amorin: so yes | 10:17 |
| amorin | make sense | 10:18 |
| sean-k-mooney | watcher uses the same env var or a very similar one and cybrog is eventlet free now too just an fyi | 10:18 |
| amorin | perfect, thanks | 10:18 |
| lajoskatona | Uggla: Hi, welcome back! I have a patch in Neutron for sending notification (/os-server-external-events: https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events ) in case a new subnet is created on a network on which there are already VMs sitting. | 11:07 |
| lajoskatona | Uggla: from user perspective this will mean (most visible at least) that after creating the subnet the server list output will add the extra IP for the given VM. | 11:08 |
| lajoskatona | Uggla: This is a bulk POST from Neutron side, but as I am not familiar with the depths of Nova would be good to check this from a Nova perspective also before I overload some network cache mechanism :-) | 11:09 |
| sean-k-mooney | lajoskatona: i didnt think prot recived ip on subnets automaticlly | 11:20 |
| sean-k-mooney | lajoskatona:my understandignis if you add a subnet to a network that has existing prot if you want to an an ip formthat subnet to the port you have to do it manualy | 11:20 |
| sean-k-mooney | so the server output shoudl not change without addign an addtional fixed ip | 11:20 |
| lajoskatona | sean-k-mooney: good point, I have to double check that | 11:21 |
| sean-k-mooney | so the act of adding a subnet to a netowkr shoudl not require a network-vif-changed event | 11:21 |
| sean-k-mooney | lajoskatona: if you auto added ip to prot by addign a subnet that owuld be a breaking api chagne from my perspective as hostiroclly you did that when you exaused ipis and eneded more | 11:22 |
| sean-k-mooney | so you would not want to auto acllcate ips form the new subnet to exisitng ports | 11:22 |
| lajoskatona | sean-k-mooney: I played only with sending out the network-changed event | 11:24 |
| sean-k-mooney | sorry network change is what i ment | 11:24 |
| sean-k-mooney | the only plasce wehre we cache/use the subnet info is for the metadat api | 11:25 |
| sean-k-mooney | but even so adding a subnet to a network | 11:25 |
| sean-k-mooney | shoudl not chagne the subnets assocatred with a port | 11:25 |
| sean-k-mooney | that is determien based on its fixed ips | 11:25 |
| sean-k-mooney | so i think its only correct to send a network-chagned event ot nova if you added a fixed ip form the new subnet to a port but not for the creation of the subnet itself | 11:27 |
| lajoskatona | sean-k-mooney: ack, I check it, if that can be catched and managed from Neutron side and if that is something to work or just document it somewhere. | 11:30 |
| lajoskatona | sean-k-mooney: did it work ever? Perhaps when nova-network was the networking? | 11:30 |
| opendevreview | Koya Watanabe proposed openstack/nova-specs master: Repropose newly instance metadata/tag protection feature https://review.opendev.org/c/openstack/nova-specs/+/977339 | 11:33 |
| sean-k-mooney | lajoskatona: did what ever work? | 11:41 |
| sean-k-mooney | adding a subnet to a network is not ment to add it to exisitng ports | 11:42 |
| sean-k-mooney | so i dont think that ever "worked" if that is what you mean | 11:42 |
| sean-k-mooney | btu that would be a breakign api change to add now | 11:42 |
| sean-k-mooney | nova-networks only ever supproted 1 subnet per tenant and it was not really exposed as a first class thing | 11:43 |
| sean-k-mooney | at least not in the way that neturon does | 11:43 |
| sean-k-mooney | auto addign subnet ip to prost woudl also likely break l3 routed netowrk for what its worth | 11:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [py313-threading]Reenable the last vmware unit test https://review.opendev.org/c/openstack/nova/+/984369 | 11:44 |
| sean-k-mooney | given you cant assume a subnet is routable to all host in that case | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [py313-threading]Reenable the last vmware unit test https://review.opendev.org/c/openstack/nova/+/984369 | 11:45 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix oslo_svc_fixture.SleepFixture usage https://review.opendev.org/c/openstack/nova/+/984373 | 11:56 |
| Uggla | sean-k-mooney, we have this bug (more a RFE). https://bugs.launchpad.net/nova/+bug/2060916 do you know the status, Do we we have a spec less blue print for it ? | 12:46 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 13:01 |
| opendevreview | Philipp Dreesens proposed openstack/nova-specs master: Add spec for bidirectional RPC liveness handshake https://review.opendev.org/c/openstack/nova-specs/+/984384 | 13:02 |
| sean-k-mooney | Uggla: so the nova part of this | 13:16 |
| sean-k-mooney | Uggla: is that nova should adopt the use fo the new more secure way fo enablet tursted_vf | 13:17 |
| Uggla | sean-k-mooney something to discuss at the PTF ? | 13:18 |
| sean-k-mooney | having an admin modify binding porfile was a security risk espically fi you used custom policy to allow endusers | 13:18 |
| Uggla | *PTG | 13:18 |
| sean-k-mooney | Uggla: this came out of a prior ptg dicussion forfm a year or two ago | 13:18 |
| sean-k-mooney | i asked neutron to develop this | 13:18 |
| sean-k-mooney | so i woudl prefer to handle this as a wishlist bug or at most a specless blueprint ideally | 13:19 |
| sean-k-mooney | Uggla: basicly this just requires use to check 1 addtional localtion with a fallback tothe old way | 13:19 |
| Uggla | yes I'd like a SLBP too, and closing this bug. | 13:19 |
| sean-k-mooney | ack then lets create the blueprint firest then mark ti closed after with a link to the blueprint | 13:20 |
| Uggla | ok do you want me to open the BP ? | 13:21 |
| sean-k-mooney | Uggla: orginally this was a bug prly becuase we might want to backport it to the rlease where neturon added the feature | 13:21 |
| Uggla | oh I guess we can note that in the BP | 13:21 |
| sean-k-mooney | if you have time that woudl be great if not i cna trhy and loop back to it | 13:21 |
| sean-k-mooney | Uggla: the backporting is really jsut a nice to have | 13:22 |
| Uggla | I'm gonna open it, I will ping you for a "review". | 13:22 |
| sean-k-mooney | if we fix it on master i would be happy with that | 13:22 |
| Uggla | ack | 13:22 |
| opendevreview | Koya Watanabe proposed openstack/nova-specs master: Repropose newly instance metadata/tag protection feature https://review.opendev.org/c/openstack/nova-specs/+/977339 | 13:22 |
| sean-k-mooney | Uggla: im not sure fi you saw my message form last week about the cybrog cross project session | 13:23 |
| sean-k-mooney | would 13:00 utc tuesday work for you in the cybrog room | 13:23 |
| Uggla | sean-k-mooney, yep I saw it this morning. I propose to discuss during the upstream meeting. | 13:23 |
| sean-k-mooney | ack that works too | 13:24 |
| Uggla | is 1h ok for you ? | 13:24 |
| sean-k-mooney | yes | 13:24 |
| sean-k-mooney | we might be able to do less time but it shoudl not take more | 13:24 |
| sean-k-mooney | my plan was to then use the next hour in the cyborg room to wrap up the cyborg ptg | 13:24 |
| sean-k-mooney | so we can take any feedback back and resolve it | 13:25 |
| sean-k-mooney | ill be able to join the nova seession then after that wraps up at least on tuesday | 13:25 |
| phildree[m] | Hi Nova team! I’ve proposed a new spec for improving compute node healthchecks: https://review.opendev.org/c/openstack/nova-specs/+/984384 | 13:26 |
| phildree[m] | This is my first time working on OpenStack/Nova, so I’d love any feedback or corrections on the approach. Thanks for the help in advance! | 13:26 |
| Uggla | I have just seen the TC meetings shrink to 16h and 17h. | 13:27 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 13:29 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 13:32 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 14:24 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 14:29 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 14:34 |
| opendevreview | John Garbutt proposed openstack/nova master: scheduler: support shared tenant placement aggregates https://review.opendev.org/c/openstack/nova/+/984383 | 14:37 |
| bauzas | Uggla: I just noticed that the nova sessions are 1 hour later than usual PTGs (2pm-6pm UTC). Was it intutional ? | 15:02 |
| Uggla | bauzas, I have just booked the room before my PTO. So the slots are not well defined. Please do not take them into account yet. | 15:16 |
| bauzas | Uggla: ack, can you then check with sean-k-mooney about the exact timings because he mentioned they scheduled their Watcher sessions based on our own timings | 15:17 |
| Uggla | bauzas, yes we will discuss that at the upstream meeting. Then I will refine the agenda/tools | 15:17 |
| bauzas | cool | 15:17 |
| sean-k-mooney | i have tried to agragne the cybrog and watcher sessions to minimise overlap with nova and the tc sessions | 15:18 |
| sean-k-mooney | basiclly im trying to maxiumis the tiemi can attend the nova session based on how they were orginaly planned | 15:18 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Replace eventlet with threading in novncproxy https://review.opendev.org/c/openstack/nova/+/976089 | 15:19 |
| sean-k-mooney | my intent is if there is not a tc cyborg or watcher seession at the tiem of a nova session ill try and be in the nova sessions | 15:19 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Enable native threading mode for console proxy services https://review.opendev.org/c/openstack/nova/+/976089 | 15:20 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Enable threading mode for proxy services https://review.opendev.org/c/openstack/nova/+/976089 | 15:21 |
| Uggla | Nova upstream meeting in ~20mn | 15:40 |
| opendevreview | Michel Nederlof proposed openstack/nova master: Add RBD XML update functionality for migration process https://review.opendev.org/c/openstack/nova/+/974032 | 15:42 |
| Uggla | #startmeeting nova | 16:02 |
| opendevmeet | Meeting started Mon Apr 13 16:02:24 2026 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:02 |
| opendevmeet | The meeting name has been set to 'nova' | 16:02 |
| Uggla | Hello everyone | 16:02 |
| sean-k-mooney | o/ | 16:02 |
| fwiesel | o/ | 16:02 |
| bauzas | o/ but tired | 16:02 |
| phildree[m] | o/ | 16:03 |
| gibi | o/ | 16:03 |
| tkajinam | o/ | 16:03 |
| elodilles | o/ | 16:03 |
| Uggla | Let's start | 16:04 |
| Uggla | #topic Bugs (stuck/critical) | 16:04 |
| Uggla | #info No Critical bug | 16:04 |
| Uggla | #topic Gate status | 16:04 |
| Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:04 |
| opendevreview | sean mooney proposed openstack/nova master: enable tap creation in nova-live-migration https://review.opendev.org/c/openstack/nova/+/975500 | 16:04 |
| Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | 16:05 |
| Uggla | link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | 16:05 |
| Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:05 |
| Uggla | #info Please try to provide a meaningful comment when you recheck | 16:05 |
| Uggla | TBH, I have not checked the status. Anything to report about the gate ? | 16:05 |
| gibi | Uggla: I saw some tox-cover job timeouts recently but I haven't looked deeper | 16:06 |
| Uggla | gibi ok | 16:07 |
| gibi | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-cover&project=openstack/nova it is not super frequent but it exists | 16:07 |
| gibi | I vaguely remember we changed something there not to long ago... | 16:07 |
| gibi | ahh we added the func test | 16:08 |
| gibi | https://review.opendev.org/c/openstack/nova/+/978652 | 16:09 |
| gibi | we already bumped the timeout once | 16:09 |
| sean-k-mooney | yep which slows things down | 16:09 |
| sean-k-mooney | i think stephen had an idea for makign it faster but it woudl have lost some coverage initally | 16:09 |
| sean-k-mooney | bumping it slightly more might be ok as i think it now only fails on slow rax hosts | 16:09 |
| gibi | yeah that is easy to do | 16:10 |
| gibi | I will put up a patch... | 16:10 |
| Uggla | cool | 16:11 |
| sean-k-mooney | i will say there is like a 30min spread betwen the faster and slowere hosts so it quite senitive to the provider | 16:11 |
| Uggla | something else to add ? | 16:12 |
| sean-k-mooney | not really | 16:12 |
| Uggla | ok so moving on | 16:12 |
| Uggla | #topic Release Planning | 16:12 |
| Uggla | #link https://releases.openstack.org/hibiscus/schedule.html | 16:12 |
| Uggla | #info we will discuss Hibiscus release planning at the PTG. | 16:12 |
| Uggla | #info Nova deadlines are set in the above schedule | 16:12 |
| Uggla | #info PTG etherpad for 2026.2 is available: https://etherpad.opendev.org/p/nova-2026.2-ptg | 16:13 |
| Uggla | #info This is a "work in progress document", but you can enter your topics at the bottom of the document | 16:13 |
| * Uggla currently working to build the agenda. I'll ping when it will be ready. | 16:13 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [openstack-tox-cover]Increase timeout further https://review.opendev.org/c/openstack/nova/+/984417 | 16:14 |
| Uggla | I'll try to pack the X session on Mon / Tue. | 16:14 |
| Uggla | Mon will be shorter to let people go to the tc sessions. | 16:15 |
| Uggla | sean-k-mooney suggested Cross session would tuesday 13:00 utc work for the nova/cyborg cross project time? | 16:16 |
| Uggla | does someone have an objection to that ^ ? | 16:16 |
| sean-k-mooney | the intent was to have it before the nova sessiosn strated since when i propved that nova didnt have monday seession | 16:17 |
| sean-k-mooney | that leave 1 hour left in the cybrog session to reflect on any feedback | 16:17 |
| sean-k-mooney | i have 2 cyborg slots booked on monday which we could also use as an alterniive | 16:18 |
| sean-k-mooney | so im pretty flexible on what works for the wider group | 16:18 |
| gibi | my only concern is overlapping with the tc meetings | 16:19 |
| gibi | (but I don't know the exact timings) | 16:19 |
| sean-k-mooney | yep i have made sure there is none betwen cybrog and tc | 16:19 |
| sean-k-mooney | https://ptg.opendev.org/ptg.html | 16:19 |
| sean-k-mooney | we coudl put up a poll or i can just sync with rene adn we can create an etherpad | 16:21 |
| bauzas | yeah I will need to attend the TC sessions | 16:21 |
| bauzas | usually we also had sessions one hour earlier, that's probably why now we see more conflicts | 16:21 |
| sean-k-mooney | we will also need to squeeze in the other nova cross project session ideally in the first 2 days | 16:22 |
| Uggla | let's start like this, we will adjust during the week if needed. Because atm not all cross session are defined yet. | 16:22 |
| bauzas | (we were usually meeting in between 1-5pm UTC) | 16:22 |
| sean-k-mooney | bauzas: the time change does not impact the tc session | 16:22 |
| sean-k-mooney | orgianlly nova was not having session at all on monday and it finsied beofre them on monday | 16:22 |
| sean-k-mooney | in the current ptg adgendar there is also no conflict | 16:23 |
| Uggla | bauzas again, do not take into account what was defined earlier. | 16:23 |
| bauzas | there is a conflict on FRiday | 16:23 |
| sean-k-mooney | actuly no there is now on firday | 16:23 |
| sean-k-mooney | right there wasnt before the extra session were added the week before last | 16:23 |
| bauzas | and yeah, originally, TC sessions were on Monday with no Nova sessions, and on Friday it was overlapping for one hour | 16:24 |
| Uggla | bauzas, I will try to avoid overlapping for sure. | 16:24 |
| sean-k-mooney | i think one of the tc session might have mvoed form monday to firday | 16:25 |
| sean-k-mooney | as there is now only 2 hours on monday | 16:25 |
| Uggla | ok I propose to move on to next topic, I will ping when the things will be more accurate/settled. | 16:26 |
| bauzas | thanks | 16:26 |
| bauzas | (on Wed, I will have to skip two hours due to personal concerns but I'll speak out once timings settled) | 16:27 |
| tkajinam | (my only hope is that cc-related topics are scheduled in earlier slots | 16:27 |
| tkajinam | (but do not intend to be demanding :-P | 16:27 |
| Uggla | but basically my goal is to pack X session on Mon / Tue avoid overlapping and try to spread the topic in the most convenient hours. | 16:28 |
| Uggla | tkajinam, sure I'll keep that in mind. | 16:28 |
| tkajinam | Uggla, makes sense | 16:28 |
| tkajinam | Uggla, ;-) | 16:28 |
| Uggla | ok next topic | 16:29 |
| Uggla | #topic Review priorities | 16:29 |
| Uggla | #link New file for Hibiscus https://etherpad.opendev.org/p/nova-2026.2-status | 16:29 |
| Uggla | #info I have updated Launchpad and the above doc. Please ping me if you spot something missing. | 16:29 |
| Uggla | #info Starting: https://etherpad.opendev.org/p/nova-2026.2-status#L16 interesting bugs to review. | 16:29 |
| Uggla | #info Noticed https://bugs.launchpad.net/nova/+bug/2147554 bug entered by Gibi : graceful shutdown fails if requested too early in nova-compute startup sequence. Might need follow up by gmaan. | 16:30 |
| Uggla | gibi, do you want to say something about ^ | 16:30 |
| gibi | not much. I noticed some race condition related to graceful shutdown and notified gmaan | 16:30 |
| gibi | nothing serious the shutdown still happens just a bit less gracefully :) | 16:31 |
| Uggla | :) | 16:31 |
| Uggla | Also gouthamr created nice specs for storage. https://review.opendev.org/c/openstack/nova-specs/+/983801 | 16:32 |
| Uggla | and https://review.opendev.org/c/openstack/nova-specs/+/983816 | 16:33 |
| Uggla | We will have a X session with manilla to discuss them. So cool if you can look at them upfront. | 16:33 |
| Uggla | #topic Stable Branches | 16:34 |
| * Uggla giving the mic to elodilles | 16:34 | |
| sean-k-mooney | gibi: why you say failes those it exit early | 16:34 |
| sean-k-mooney | i.e. not wait | 16:34 |
| sean-k-mooney | elodilles: Uggla woudl we want to implemetn the memfd part as well for virtio-fs quality of life | 16:35 |
| gibi | sean-k-mooney: fails to be graceful I guess :) | 16:36 |
| sean-k-mooney | Uggla: that was for you i guess to inlcude that topic in the mainilar sesssion | 16:36 |
| Uggla | yes we discuss to have a topic about it. | 16:36 |
| opendevreview | Merged openstack/nova stable/2026.1: Follow ironic job rename https://review.opendev.org/c/openstack/nova/+/983166 | 16:36 |
| Uggla | however, from internal perspective, all of these are consider low priorities. | 16:36 |
| Uggla | :( | 16:37 |
| sean-k-mooney | well if someone does the work we can try and review but ya... | 16:37 |
| sean-k-mooney | im not sure if elodilles is around at teh moment or stable branches are all good ? | 16:37 |
| elodilles | yepp | 16:37 |
| elodilles | i'm around | 16:37 |
| Uggla | however I said it does not hurt to discuss at the PTG and progress at least on the spec | 16:37 |
| elodilles | just wanted to wait for the prev topic to end o:) | 16:38 |
| sean-k-mooney | :) | 16:38 |
| Uggla | yep sorry elodilles please go ahead | 16:38 |
| elodilles | thx :) | 16:38 |
| elodilles | #info nova stable gates should be OK | 16:38 |
| elodilles | #info thanks for the reviews of placement and osc-placement gate fixes, all but one merged: placement's 2025.1 branch needs a fix for grenade-skip-level job | 16:38 |
| elodilles | i'll try to figure out the issue there, but if that doesn't succeed soon, then we can set it to non-voting (as it is already the case on nova's stable/2025.1) | 16:38 |
| elodilles | (unmaintained/2024.1 -> stable/2025.1) | 16:39 |
| elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | 16:39 |
| elodilles | basically that's it from me | 16:39 |
| * elodilles passes the mic back | 16:40 | |
| sean-k-mooney | well we are not stricly ment to do upgrade testign from unmaintined | 16:40 |
| sean-k-mooney | but if you do get it workign then cool | 16:40 |
| elodilles | sean-k-mooney: yes, that's why i mentioned | 16:40 |
| sean-k-mooney | is this pkg_resouce related out of interst | 16:40 |
| elodilles | sean-k-mooney: though if i can get it working... o:) | 16:40 |
| sean-k-mooney | i knwo we haveing fixed those issues on 2024.2 in the requireemnt repo | 16:40 |
| sean-k-mooney | so i expect 2024.1 is broken too | 16:40 |
| elodilles | sean-k-mooney: yes, it is somewhat related, but afaiu it shouldn't be :/ so still checking why we end up with an error like that :/ | 16:41 |
| sean-k-mooney | ack | 16:41 |
| elodilles | sean-k-mooney: no, that works afaik | 16:41 |
| elodilles | sean-k-mooney: the error is when nova is upgraded from 2024.1 to 2025.1, but i still don't have the full root cause | 16:42 |
| sean-k-mooney | the requiremnts check job failes because the requirement repo depends on pkg_resouces adn we have not pineed setup tools on 2024.2 or bumpt pbr but we can chat about that later if you like | 16:42 |
| elodilles | sean-k-mooney: sure | 16:43 |
| elodilles | thanks in advance o/ | 16:43 |
| * Uggla just noticed you passed the mic back | 16:44 | |
| Uggla | thanks elodilles | 16:44 |
| elodilles | thanks too o:) | 16:44 |
| Uggla | #topic vmwareapi 3rd-party CI efforts Highlights | 16:44 |
| Uggla | fwiesel anything for us ? | 16:44 |
| fwiesel | Hi, so I can confirm that the bugfix in oslo.vmware would fix the image upload timeout observed by gibi | 16:45 |
| fwiesel | #link https://review.opendev.org/c/openstack/oslo.vmware/+/982614 | 16:45 |
| Uggla | \o/ thanks | 16:45 |
| fwiesel | That's from my side for now... I spend the calmer days looking into some random errors, but nothing to report now. | 16:45 |
| gibi | yeah I'm +1 on the fix | 16:46 |
| Uggla | fwiesel thank you. | 16:46 |
| Uggla | #topic Gibi's news about eventlet removal | 16:46 |
| * Uggla giving the mic to gibi | 16:46 | |
| gibi | o/ | 16:46 |
| gibi | no much to report | 16:48 |
| gibi | work ongoing on the console proxies and testing | 16:48 |
| gibi | the PTG etherpad is up to date | 16:48 |
| gibi | we will discuss status and plans there | 16:49 |
| gibi | EOM | 16:49 |
| sean-k-mooney | https://review.opendev.org/c/openstack/oslo.vmware/+/982614 looks indepent of the executor issue we saw in the vmware job so this is just a general urllib3 issue not an eventet oen right | 16:49 |
| sean-k-mooney | im asking because it does not have a bug filed for it and im wodnering if eventlet was masking it | 16:50 |
| gibi | the executor issue was the deadlock on sending dependent tasks to the parent's executor we fixed that in general | 16:51 |
| gibi | this bug apperad after | 16:51 |
| gibi | and was visible as hanging in image tasks in the vmware driver | 16:51 |
| sean-k-mooney | ack so it may have been there but not visable before | 16:52 |
| gibi | yeah | 16:52 |
| sean-k-mooney | i know evently also monkey patches urllib3 | 16:52 |
| sean-k-mooney | so that also complciates things | 16:52 |
| sean-k-mooney | ok we can likely move on | 16:52 |
| Uggla | #topic Bug scrubbing | 16:52 |
| Uggla | #info up to 197 (-2) | 16:52 |
| Uggla | #link: https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:53 |
| Uggla | #link: https://truc.uggla.fr/ to follow the trend. | 16:53 |
| Uggla | Uggla: This week [public] Upstream bug triage. Wednesday, April 15 · 15:30 – 16:00 UTC. Video call link: https://meet.google.com/kmq-eedg-auj | 16:53 |
| Uggla | oops forget to say : Skipping Layos topic as he looks not around and we are a bit late on schedule | 16:53 |
| Uggla | #topic Open discussion | 16:54 |
| Uggla | Uggla: Suggest no meeting next week. 20th April vPTG week. | 16:54 |
| gibi | +1 | 16:54 |
| Uggla | I will notify on the ML too | 16:54 |
| elodilles | +1 | 16:54 |
| Uggla | any last 5 minutes topic to discuss ? | 16:55 |
| phildree[m] | Hello guys, I am not sure if this is the right place and time to bring this up because i am new to OpenStack, but I have created a spec and am looking for some feedback: https://review.opendev.org/c/openstack/nova-specs/+/984384 | 16:57 |
| Uggla | phildree[m], can you add a topic in the ptg etherpad ? then I'll define a slot for you next week. | 16:58 |
| phildree[m] | Ok, thank you :) | 16:58 |
| Uggla | any "prefered" timing ? | 16:58 |
| phildree[m] | Nope | 16:59 |
| Uggla | ok | 17:00 |
| Uggla | so I think we are good. | 17:00 |
| Uggla | time to close, thanks for joining this meeting. Have a nice day/evening and see you next week for the vPTG. | 17:01 |
| Uggla | #endmeeting | 17:01 |
| opendevmeet | Meeting ended Mon Apr 13 17:01:07 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.html | 17:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.txt | 17:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2026/nova.2026-04-13-16.02.log.html | 17:01 |
| fwiesel | o/ thanks everyone | 17:01 |
| elodilles | see you next week o/ | 17:01 |
| sean-k-mooney | phildree[m]: have you talked to oslo about his | 17:01 |
| phildree[m] | No, this IRC is my first point of contact for now | 17:01 |
| sean-k-mooney | phildree[m]: in generally iw oudl prefer if we removed the if we didnt have to have any aplciation level handchakes | 17:01 |
| gibi | thanks | 17:02 |
| sean-k-mooney | we have some existign rpc pings but anything that makes the conductor comptue messageing nosyier by addign background messages direct hurts scalableity fo nova | 17:03 |
| phildree[m] | sean-k-mooney: Ok, I understand. Maybe handshake is a bit much for what i have proposed, its more like an echo | 17:03 |
| sean-k-mooney | right im saying even adding a echo/ping is protically a scaling issues | 17:04 |
| phildree[m] | sean-k-mooney: I have seen the existing ping but it is not applicable to our usecase (I explained why in the spec) | 17:04 |
| phildree[m] | sean-k-mooney: yes you are totally right, I get that. Any other ideas on how to solve the problem we have been facing? | 17:05 |
| opendevreview | Merged openstack/nova stable/2025.1: Fix fill_metadata usage for the ImagePropertiesWeigher https://review.opendev.org/c/openstack/nova/+/964291 | 17:05 |
| sean-k-mooney | i have not read your spec yet but we gerneally perfer to start with the usecase/probelm | 17:05 |
| sean-k-mooney | before going to a feature/design chagne | 17:05 |
| sean-k-mooney | phildree[m]: the prviosu solution which we never implmetned was per binary healthcheck endponts on each nova service | 17:07 |
| sean-k-mooney | phildree[m]: i think teh issue your having is something the outbount rabbit conenction is fucntional | 17:08 |
| sean-k-mooney | but the inbound one is not correct | 17:08 |
| sean-k-mooney | i.e. the compute can call the conductor but the conductor cannot sucessfully call the computes | 17:08 |
| phildree[m] | Yes, we faced this exact issue. | 17:09 |
| sean-k-mooney | do you have the guareented message delivery flag enabled in oslo | 17:09 |
| phildree[m] | I will think about the other approach with individual per binary healthcheck endpoints | 17:09 |
| sean-k-mooney | well we didnt impment that or at least we didnt merge it | 17:10 |
| phildree[m] | That is a good question, which I am not able to answer right now | 17:10 |
| melwitt | bauzas: hi, there are two vtpm related patches that are simple and need a second review if you could get a chance plz https://review.opendev.org/c/openstack/nova/+/978836 and https://review.opendev.org/c/openstack/nova/+/983504 | 17:11 |
| sean-k-mooney | phildree[m]: https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag | 17:11 |
| sean-k-mooney | i woudl check you have that set to true | 17:11 |
| phildree[m] | My issues with the per binary healthcheck is that I still need a way to ensure that full communication from conductor to compute (and back) is up and running, which i cannot check without any actual communication, I think | 17:11 |
| sean-k-mooney | phildree[m]: what that will detect is fi the per compute queue is present | 17:11 |
| phildree[m] | Ok very nice, I will look into it, thank you for the infos :) | 17:12 |
| sean-k-mooney | phildree[m]: well the whole point of the per binary healthcheck si to not use rabbit or the conductor | 17:12 |
| sean-k-mooney | it to move the health moditoring out fo that code path to an external http endpoint | 17:12 |
| sean-k-mooney | that you can have prometheous or whatever scrape instead | 17:12 |
| sean-k-mooney | phildree[m]: the direct_mandatory_flag tuns a slient failrue (the messag eis encued to a new queue that nothign is listenting too) into a driect rpc call failure and failure in teh api | 17:14 |
| phildree[m] | I get it, and it makes sense, but then there is no way to check the health of the communication (and the underlying infrastructure) between conductor and compute (only the individual binaries) | 17:14 |
| sean-k-mooney | right today we do that via the resouce update whcih is a push model | 17:15 |
| sean-k-mooney | its worth a dicussion i guess but my main concern is we dont really run perodic form the conductor to monitor the satues of other agents | 17:16 |
| sean-k-mooney | and we try not to do monitoring in general in nova. so this woudl likely need to be confiugrable and off by defualt | 17:16 |
| sean-k-mooney | at least until we know the overhad of it | 17:16 |
| phildree[m] | I have touched many points of what you have brought up, but as I said, I am not yet very familiar with OpenStack so maybe there are some issues I did not anticipate. | 17:17 |
| phildree[m] | I think I have to read up on what you have sent first before I may rethink the idea. Thanks a lot for the infos so far, that was very helpfull and a lot less complicated then i thought :) | 17:17 |
| sean-k-mooney | phildree[m]: one thing we are currently changing | 17:19 |
| sean-k-mooney | is we will nolonger have a slngle rpc listter per compute agent | 17:19 |
| sean-k-mooney | we will now have two becase fo the graceful shutdown work | 17:19 |
| sean-k-mooney | so gmaan woudl likely need to review your prosal in the context fo that work | 17:20 |
| phildree[m] | Ok, that is something very important as this may have implications on my idea | 17:20 |
| sean-k-mooney | with graceful shutdown we will be shutting down 1 of the 2 lstenter until the inprogress operation complete | 17:20 |
| sean-k-mooney | so any kind fo monitoring would likely have to be on the one we dont shutdown | 17:21 |
| gmaan | sean-k-mooney: reading chat | 17:22 |
| sean-k-mooney | gmaan: context is https://review.opendev.org/c/openstack/nova-specs/+/984384/1/specs/2026.2/approved/bidirectional-rpc-liveness-handshake-for-compute-nodes.rst | 17:22 |
| phildree[m] | At least for the shutdown period, yes | 17:22 |
| sean-k-mooney | phildree[m]: are you goign to present this at the ptg for dicussion? | 17:22 |
| phildree[m] | I have to be honest, I first have to read up on what the PTG exactly is and how it works | 17:23 |
| sean-k-mooney | ack its basically a workign session wehre user/operators/devleoper talk about our plans for the next release and pain points | 17:24 |
| sean-k-mooney | it operats now as a virual event with an eterphad for notes and adgena and jitsi meet as the video confernce tool | 17:24 |
| sean-k-mooney | its free to attend. they used ot be inperson events | 17:25 |
| phildree[m] | Then this would probably make sense from my point of view (even if its just to gain a better understanding of the whole project) | 17:25 |
| sean-k-mooney | if you cant attend thats fine too. | 17:25 |
| gmaan | phildree[m]: I agree with sean-k-mooney that periodic handshake over RPC is expensive especially in scale env | 17:25 |
| sean-k-mooney | gmaan: im kind of wondering if we can maybke marke the node as down sooner by monitoring failed/timedout rpc calls of soemthing liek that | 17:26 |
| gmaan | I also have not read the spec but does compute service up check not enough? that is kind of periodic check if service is down or not | 17:26 |
| sean-k-mooney | i.e. only do a condotor to compute ping in that case to confirm the state | 17:26 |
| sean-k-mooney | gmaan: there was at least one case in the psat where the comptue service is up but the rabbit queue was deleted | 17:27 |
| sean-k-mooney | so it coudl send rpc calles like the status update but not recive them | 17:27 |
| gmaan | " failed/timedout rpc calls" might not give actual result and can end up in false result. | 17:27 |
| sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.direct_mandatory_flag was partly added to prevent that | 17:27 |
| phildree[m] | gmaan: I think if you do the service up check it relies on the DB entries based on heartbeats which is exactly what our issues was because the heartbeat may keep working while the requests from conductor to compute failed | 17:28 |
| gmaan | yeah not RPC | 17:28 |
| phildree[m] | Just like sean-k-mooney just described | 17:28 |
| gmaan | yeah, that is true and existing challenge | 17:29 |
| sean-k-mooney | phildree[m]: the reall issue is take an enve with 500 nodes and 3 conductors. we technically ahve a leader election fetrue in the conductor but if we didnt use that ever 300 second (our default perodic interval) we would send 1500 messages 3 form each conduxtor x 500 compute nodes | 17:30 |
| gmaan | so what is actual issue? conductor to compute RPC call fail the it give status to check if compute service is up or not if yes then RPC is down right? | 17:30 |
| sean-k-mooney | in rare cases the compute can be up enough for the status to be up but a call to the compute form the api will fail to boot a vm | 17:31 |
| phildree[m] | In my proposal the handshake is compute initiated in order to avoid leader stuff and states in the conductor | 17:31 |
| sean-k-mooney | phildree[m]: well we typiclly have 3-5 conductors per cell and can have 300-1500 compute per cell in large clouds | 17:32 |
| gmaan | I am not saying 'service up' is reliable source but my question is do we want to automate it in conductor to check compute before call otherwise asl sch to select other dest? or just for clear troubleshooting and error message that what exactly is down? | 17:33 |
| gmaan | s/asl/ask | 17:33 |
| sean-k-mooney | gmaan: ya im not sayign we shoudl do this im just playing devils advocate | 17:34 |
| gmaan | sean-k-mooney: or the healthcheck proposal is good fit here but is the latest proposal check RPC health also? | 17:35 |
| phildree[m] | it like a tcp Syn (compute send a request to any conductor), Syn-Ack (conductor replies to that tiny request), Ack (compute initiates the regular heartbeat as is) | 17:35 |
| phildree[m] | This way when any part of the communction is broken the node does not appear as up | 17:35 |
| phildree[m] | But in order to reduce RPCs i think also just a periodic ping without compute base initiation might be feasible but will have the issue of coordination etc.. | 17:35 |
| sean-k-mooney | gmaan: it was goign to catch connection closed events and similar on the comptue and report if the rpc bus was working | 17:36 |
| sean-k-mooney | but not by active probing | 17:36 |
| sean-k-mooney | only by recoreding if we got an excption | 17:36 |
| gmaan | k | 17:36 |
| sean-k-mooney | so it woudl not really help in this case | 17:36 |
| gmaan | ++ | 17:36 |
| gmaan | yeah | 17:36 |
| gmaan | do not have any perfect solution as of now but let's discuss it in PTG | 17:38 |
| phildree[m] | Ok, thanks anyway for the engagement, I really appreciate you guys taking your time :) | 17:39 |
| *** dwi_ is now known as dwi | 18:48 | |
| opendevreview | Merged openstack/nova stable/2026.1: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/983989 | 19:20 |
| opendevreview | John Garbutt proposed openstack/nova master: Make PCPUs not land on VCPUs by default https://review.opendev.org/c/openstack/nova/+/975779 | 19:58 |
| opendevreview | John Garbutt proposed openstack/nova master: Fix ironic instance rebuild with API >=2.93 https://review.opendev.org/c/openstack/nova/+/963298 | 19:59 |
| melwitt | dansmith: there is one more small patch that was split out from the 'user' live migration patch that is ready for review at your convenience https://review.opendev.org/c/openstack/nova/+/983505 | 20:24 |
| dansmith | ah yep, the other one, sorry I guess I missed that when hitting the first one | 21:00 |
| dansmith | done now | 21:00 |
| melwitt | dansmith: thanks!! | 21:16 |
| *** bauzas1 is now known as bauzas | 22:40 | |
| opendevreview | melanie witt proposed openstack/nova master: [WIP] how borken is ipv6 only ceph. https://review.opendev.org/c/openstack/nova/+/982302 | 22:46 |
| opendevreview | melanie witt proposed openstack/nova master: [WIP] how borken is ipv6 only ceph. https://review.opendev.org/c/openstack/nova/+/982302 | 22:51 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!