| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 06:31 |
|---|---|---|
| opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | 09:04 |
| * gibi just read back on the metadata server allowing POSTing password and grossed out | 09:50 | |
| opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron Ports https://review.opendev.org/c/openstack/nova/+/969298 | 09:56 |
| opendevreview | OpenStack Release Bot proposed openstack/nova stable/2026.1: Update .gitreview for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980404 | 10:14 |
| opendevreview | OpenStack Release Bot proposed openstack/nova stable/2026.1: Update TOX_CONSTRAINTS_FILE for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980405 | 10:14 |
| opendevreview | OpenStack Release Bot proposed openstack/nova master: Update master for stable/2026.1 https://review.opendev.org/c/openstack/nova/+/980406 | 10:14 |
| opendevreview | ribaudr proposed openstack/nova-specs master: Move Gazpacho implemented specs https://review.opendev.org/c/openstack/nova-specs/+/979283 | 10:51 |
| zigo | Hi there! | 10:58 |
| zigo | From Gazpacho Nova's release notes, I can read: | 10:58 |
| zigo | The default value of [DEFAULT]/graceful_shutdown_timeout has been changed from 60 to 180 seconds for all Nova services. This means that when a Nova service receives SIGTERM, it will now wait up to 180 seconds for a graceful shutdown before being forcefully terminated. | 10:58 |
| zigo | What service is impacted for this? I | 10:58 |
| zigo | I'm guessing: nova-compute and nova-conductor, right? | 10:58 |
| zigo | But scheduler as well? | 10:58 |
| zigo | To me, it looks like, yes (reading the code). | 10:59 |
| zigo | I think it'd be nicer for Nova to read what's in systemd, using the d-bus interface. Would this kind of patch be accepted (if made optional, since not everyone uses systemd) ? | 11:02 |
| gibi | gmaan: ^^ | 11:16 |
| zigo | Also, when attempting to run Nova unit tests, I get failure during test discovery: | 11:16 |
| zigo | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' | 11:16 |
| zigo | Am I missing something? | 11:16 |
| gibi | zigo: I think we agree that synchorizing the timeouts between systemd / k8s liveness probe / etc with the nova's shutdown timout is needed. But it is hard to automate it as it depends on how nova is deployed. It is not always systemd | 11:17 |
| gibi | zigo: which python version are you using for the unit test? | 11:17 |
| gibi | zigo: I think the graceful shutdown timeout change effects all nova sevices except the APIs. But gmaan can confirm | 11:18 |
| gibi | zigo: we see similar unit test issue in py3.14 https://zuul.opendev.org/t/openstack/build/156fe0f67bff4e4da43a31bd0f02e667 | 11:20 |
| gibi | I had not time to dig. But py3.13 is pretty stable | 11:20 |
| gibi | (3.14 is still not voting for us so it is easy to ignore'l | 11:21 |
| gibi | ) | 11:21 |
| zigo | gibi: 3.13 and 3.14. | 11:26 |
| zigo | Flamingo on 3.14 was no issue. | 11:27 |
| zigo | (sorry, in a meeting, wont take too long) | 11:27 |
| gibi | ack. If you see the issue in 3.13 that is strange. But anyho please file a bug I will try to get to it | 11:28 |
| zigo | What I'm seeing: https://paste.opendev.org/show/bB5CJ9SHLPXOjKGyu0N8/ | 11:36 |
| zigo | I'm seeing the issue in 3.14 only, because it's not running 3.13 tests if 3.14 fails. | 11:36 |
| zigo | gibi: Well, I can dig it out, just, do I have a workaround? Maybe export a variable so it uses only threading? | 11:38 |
| zigo | Found it: OS_NOVA_DISABLE_EVENTLET_PATCHING | 11:40 |
| zigo | Ok, tests are running now... Hopefully, no breakage because of this! :) | 11:42 |
| zigo | gibi: How safe is it to use OS_NOVA_DISABLE_EVENTLET_PATCHING in all services? | 11:49 |
| zigo | Looks like one unit test is stuck... :( | 11:49 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | 12:04 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 12:04 |
| nicolairuckel | Could anyone take a look at this? https://zuul.opendev.org/t/openstack/build/59d2bc927a5d4e40936da1ccc05722e6 The job run successfully before with exactly the same code. | 12:08 |
| zigo | gibi: Unit tests are passing, if I set OS_NOVA_DISABLE_EVENTLET_PATCHING=false in d/rules. | 12:16 |
| gibi | zig | 12:49 |
| gibi | zigo: so using native threading via OS_NOVA_DISABLE_EVENTLET_PATCHING=true is experimental. If you can test it the we are eager to hear feedback. All services except the console proxies can be run with native threading | 12:50 |
| zigo | Well, for tests to run, I had to set it to false ! :/ | 12:50 |
| gibi | https://docs.openstack.org/nova/latest/admin/concurrency.html | 12:50 |
| gibi | this is the main doc | 12:50 |
| zigo | I'll try whenever I'm at the tempest phase. | 12:50 |
| gibi | for unit testing we have an exclude list https://github.com/openstack/nova/blob/d5c64e7aba09052601b59bf1297187d85da6b90b/tox.ini#L83 | 12:51 |
| gibi | functional tests does not yet work with native threading | 12:51 |
| gibi | tempest passes in nova-next job where we configure each service with native threading | 12:52 |
| zigo | Oh... Thanks, that's really helpful. | 12:52 |
| sean-k-mooney | zigo: we have optional systemd/debus integration | 14:29 |
| sean-k-mooney | for sending ready | 14:29 |
| sean-k-mooney | vai the notifcaiotn socket | 14:29 |
| sean-k-mooney | but im not sure we shoudl add any mroe integration for the graceful shutdown timeout | 14:30 |
| sean-k-mooney | at least not nativly in nova | 14:30 |
| sean-k-mooney | puting it in oslo and having nova use it might be more reasonable | 14:30 |
| sean-k-mooney | zigo: by the way we see the same issue in the 3.14 tox job upstream | 14:31 |
| sean-k-mooney | we just have nto had time to fix it | 14:31 |
| sean-k-mooney | oslo_service.backend.exceptions.BackendAlreadySelected: Backend already set to 'eventlet', cannot reinitialize with 'threading' is why that job is currently failing | 14:32 |
| sean-k-mooney | zigo: one issue is we have not added py314 here yet https://github.com/openstack/nova/blob/master/tox.ini#L59 | 14:33 |
| zigo | sean-k-mooney: Exactly my thought, which is why I just wrote: https://review.opendev.org/c/openstack/oslo.service/+/980440 | 14:34 |
| sean-k-mooney | so its going to fallback to https://github.com/openstack/nova/blob/master/tox.ini#L55 | 14:34 |
| sean-k-mooney | which does not include the exclude list | 14:34 |
| zigo | I haven't used dbus-python, that'd be an external dependency, so I had to use subprocess to get the TimeoutStopUSec. | 14:34 |
| gmaan | zigo: RE on graceful shutdown: yes that timeout is in effect for the services using oslo.services which are 1. compute 2. sch 3. conductor. The other remaining services api, metadata api, console proxy shutdown path is different and it depends/handled by their underlying wsgi server/websokify lib | 15:10 |
| gmaan | considering the different external system to deploy the services, its not good idea to automate the timeout as per the single deployment model like systemd. We recommend to override this timeout in conf as per the external system timeout | 15:12 |
| sean-k-mooney | we woudl need ot have the config take precedence anyway | 15:15 |
| sean-k-mooney | we coudl perhaps take a defualt form systemd if not set in the config | 15:15 |
| sean-k-mooney | but we need the config when nova is deployed in a continer or on a system without systemd | 15:15 |
| gmaan | yeah, I am not sure how much value it will add to just automate for systemd instead operator checking and setting it in config if current default is not good is better and explicit way | 15:17 |
| gmaan | i think default in systemd is 60 sec which is anyways not enough for nova as we have manager wait time default to 160 | 15:17 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix running test with py314 https://review.opendev.org/c/openstack/nova/+/980457 | 15:20 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Move py312-threading to py313 https://review.opendev.org/c/openstack/nova/+/980458 | 15:20 |
| gibi | sean-k-mooney: zigo: ^^ fix for the py314 unit test job | 15:20 |
| sean-k-mooney | gibi: cool i just didnt have a time to look into that myself but ill review thoese before i leave today | 15:21 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | 15:36 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 15:36 |
| dansmith | nicolairuckel: around by chance? | 15:53 |
| nicolairuckel | dansmith: I just sat down on my computer | 15:55 |
| nicolairuckel | good timing | 15:55 |
| nicolairuckel | at my computer | 15:56 |
| nicolairuckel | on would be weird I guess | 15:56 |
| opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | 15:56 |
| dansmith | nicolairuckel: hey, so I was trying to verify your nvram fix (which maybe I should have done before) but.. I think I must be missing something... | 15:56 |
| dansmith | nicolairuckel: to make sure it works, I was going to just tweak the efibootmgr config a bit and then power cycle the instance to make sure it kept the tweak | 15:57 |
| nicolairuckel | the backport for 2025.2? | 15:57 |
| dansmith | nicolairuckel: actually just the thing we already merged | 15:57 |
| dansmith | so if I create a new fake boot entry and then reboot, it's gone when it comes back, but I see my label string in the VARS.fd for the instance | 15:57 |
| dansmith | what I really wanted to do was just use efivar to poke something in there to look for, but was never able to get that undocumented monstrosity to do anything :P | 15:58 |
| dansmith | anyway, am I missing something about the "template" behavior overwriting the boot config as expected behavior or something? | 15:58 |
| nicolairuckel | that sounds odd to me. In my testing the variable and content were still there. | 15:59 |
| dansmith | all I did was set the firmware type to uefi on the image.. does the platform need to be q35 or something maybe? I wasn't expecting that to matter but... | 15:59 |
| nicolairuckel | did you use efivar to set the variable? | 16:00 |
| dansmith | no, I used efibootmgr | 16:00 |
| dansmith | if you can tell me how to get efivar to work I'd love to hear it :) | 16:00 |
| nicolairuckel | one second | 16:00 |
| nicolairuckel | what I did was the following | 16:01 |
| nicolairuckel | sudo apt update && sudo apt install efivar | 16:01 |
| nicolairuckel | $ echo "Nova UEFI" > foo | 16:01 |
| nicolairuckel | $ sudo efivar -n 56da3878-c379-4f5b-be22-72f6f82b6a52-Nova -w -f ./foo | 16:01 |
| nicolairuckel | $ efivar -n 56da3878-c379-4f5b-be22-72f6f82b6a52-Nova | 16:01 |
| opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | 16:01 |
| nicolairuckel | and then openstack server reboot/start/stop/migrate | 16:02 |
| dansmith | yeah | 16:02 |
| nicolairuckel | and then efivar -l and efivar -n again | 16:02 |
| dansmith | lemme try efivar again.. | 16:02 |
| dansmith | that uuid is just one you generated, right not anything magical? | 16:02 |
| nicolairuckel | ah, right | 16:02 |
| nicolairuckel | that was something I got from OpenStack | 16:03 |
| nicolairuckel | the server ID | 16:03 |
| dansmith | but does it matter? | 16:03 |
| dansmith | I was just using uuidgen to snag one and it never seemed to want to let me do it | 16:03 |
| nicolairuckel | That's a good question. I just wrote in my notes that it should be the server ID | 16:03 |
| nicolairuckel | yeah, I remember something like that | 16:03 |
| dansmith | okay lemme try both | 16:04 |
| dansmith | does not have to be server id | 16:04 |
| dansmith | I dunno why but that just worked.. I spent a stupid amount of time trying to get that to work (granted on a libvirt non-nova VM, maybe there was some config difference) | 16:05 |
| dansmith | so let me see if that sticks after the reboot | 16:05 |
| dansmith | sho'nuf.. works as expected | 16:07 |
| dansmith | I dunno, I got myself very turned around with efivar before and thought I'd just manipulate the boot config since that was less obscure anyway | 16:07 |
| dansmith | there must be something going on that overwrites that boot config on startup or something, I dunno why | 16:07 |
| dansmith | but the efivar method to verify is what I originally wanted, so thanks for that :) | 16:08 |
| nicolairuckel | You're welcome | 16:08 |
| nicolairuckel | I just used efivar because my colleague showed me the commands I needed. | 16:08 |
| nicolairuckel | That's why I never tried efibootmgr. | 16:08 |
| dansmith | yeah, well, efivar is very undocumented and obscure so I had trouble getting ti to behave *shrug* | 16:10 |
| nicolairuckel | I think that's why I never bothered to figure out why at some point the UUID had to be the server ID. | 16:11 |
| nicolairuckel | But I added that to my notes that this isn't actually the case. | 16:11 |
| dansmith | but to be clear, it doesn't.. I just used uuigen to get one | 16:11 |
| dansmith | yeah, cool | 16:11 |
| nicolairuckel | I wonder if that was a problem in my first implementation where I copied the variables manually instead of using libvirt. | 16:12 |
| nicolairuckel | And since using the server ID still works, I never realized that it's not necessary anymore. | 16:12 |
| nicolairuckel | thanks for testing it again | 16:15 |
| dansmith | yup, I was doing something similar for vTPM to prove it works | 16:16 |
| dansmith | again requiring obscure and temperamental tooling | 16:16 |
| dansmith | s/works/persists | 16:16 |
| nicolairuckel | Even though you scared me a bit. For a few minutes I thought I'd have to go back to work on that after telling everyone it was done and merged. :D | 16:16 |
| dansmith | heh | 16:16 |
| gmaan | does anyone knows why we are keeping nova/keymgr/conf_key_mgr.py ? https://github.com/openstack/nova/blob/master/nova/keymgr/conf_key_mgr.py | 18:21 |
| *** erlon2 is now known as erlon | 18:22 | |
| gmaan | I think it is only for testing and because barbican ins not installed by default in devstack, devstack also use this as key_manager backendhttps://github.com/openstack/devstack/blob/master/lib/nova#L460C23-L460C34 | 18:22 |
| gmaan | melwitt: ^^ maybe you know, as enabling the barbican in vTPM jobs modify the key_manager backend to barbican which actually test the secrets store/get etc https://github.com/openstack/barbican/blob/master/devstack/lib/barbican#L94 | 18:23 |
| gmaan | any other place we use this fixed_key backend (nova/keymgr/conf_key_mgr.py) then testing in unit/functional or devstack testing? | 18:24 |
| gmaan | or stephenfin ^^ as you modified it last (though 3 years ago :)) | 18:25 |
| stephenfin | gmaan: yeah, it's purely for testing. I think I had plans to move it to castellan (e.g. castellan.test) but there were a few references in other projects, as you've seen | 18:42 |
| gmaan | stephenfin: ack, thanks. that is enough information for now. but ++ on moving | 18:44 |
| melwitt | Uggla, gmaan: I noticed a pending spec amendment patch and thought it should merge first before the "spec move" patch. it will cause a merge conflict in the "spec move" patch, sorry about that https://review.opendev.org/c/openstack/nova-specs/+/978144 | 20:00 |
| gmaan | melwitt: oh, there is another one. I thought vtpm amendment was the only one left | 20:01 |
| melwitt | yeah, I noticed it :/ | 20:02 |
| melwitt | alternatively we could move the specs and then make the amendment have to update | 20:02 |
| gmaan | as amenedment one is in gate, maybe let's merge that and Uggla can rebase the spec move one | 20:03 |
| melwitt | I thought it should go first and then second guessed it. | 20:05 |
| gmaan | yeah | 20:06 |
| melwitt | I did not expect it to merge conflict bc the spec move is just renaming files. but I guess the file content at the time matters too | 20:07 |
| gmaan | ++ | 20:11 |
| opendevreview | Merged openstack/nova-specs master: Update remove-os-volumes_boot-api to reflect changes https://review.opendev.org/c/openstack/nova-specs/+/978144 | 20:13 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 22:05 |
| opendevreview | Stephen Finucane proposed openstack/nova stable/2025.2: api: Handle empty imageRef alongside null for local BDM check https://review.opendev.org/c/openstack/nova/+/980540 | 22:32 |
| opendevreview | Stephen Finucane proposed openstack/nova stable/2025.1: api: Handle empty imageRef alongside null for local BDM check https://review.opendev.org/c/openstack/nova/+/980541 | 22:32 |
| opendevreview | Stephen Finucane proposed openstack/nova stable/2025.1: api: Handle empty imageRef alongside null for local BDM check https://review.opendev.org/c/openstack/nova/+/980541 | 22:32 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!