| opendevreview | Merged openstack/placement master: Skip functional tests for git config update https://review.opendev.org/c/openstack/placement/+/976907 | 00:00 |
|---|---|---|
| mw_ | Hi folks, could someone please take a look at this Nova change when you have a chance? https://review.opendev.org/c/openstack/nova/+/999841 | 02:01 |
| mw_ | It adds bandwidth limiting for attached volume retypes and migrations to make their network impact more predictable. | 02:01 |
| opendevreview | Merged openstack/placement master: Commonize code for updating and creating allocations https://review.opendev.org/c/openstack/placement/+/808535 | 04:17 |
| *** bauzas4 is now known as bauzas | 05:26 | |
| zigo | Hi team! | 06:25 |
| zigo | With Hibiscus, it looks like the systemd notify doesn't work. My systemd nova-compute.service stays in "Activating" mode until systemd believes it's crashed, and shut it down. | 06:25 |
| zigo | Is it possibly broken ?!? | 06:26 |
| zigo | How can I check? | 06:26 |
| *** bauzas4 is now known as bauzas | 08:03 | |
| gibi | zigo: I'm spinning up a devstack from master to see if I can replicate that | 08:38 |
| gibi | can I see your systemd service file easily somewhere? | 08:43 |
| gibi | zigo: also a quick check, have you tried switching the nova-compute back to eventlet mode (if you can) to see if that resolves the systemd problem? | 08:46 |
| gibi | zigo: I see this https://github.com/openstack/oslo.service/blob/364aafec725c9b98d23426450bc247d838cba611/oslo_service/backend/_eventlet/service.py#L297 but so far I haven't seen the _threading equivalent of it | 08:50 |
| gibi | https://github.com/search?q=repo%3Aopenstack%2Foslo.service+systemd.notify&type=code yeah it feels like that systemd support is missing from oslo.service threading backend | 08:51 |
| gibi | once I confirm it locally in devstack I will file a bug for oslo.service | 08:51 |
| gibi | stephenfin: tkajinam: ^^ | 08:52 |
| gibi | adding Type=notify to the devstack's service file reproduces the problem and switching nova-compute to eventlet mode fixes it | 09:10 |
| gibi | so the reason this is seen now is because we switched the default to threading mode for nova-compute in Hibiscus | 09:11 |
| gibi | but I guess this oslo.service bug affects other services running in threading mode as well | 09:12 |
| zigo | gibi: Thanks for checking for this, I'll switch nova-compute to Type=simple at least temporarily. | 09:24 |
| zigo | If you need to check for my packaging, next time, simply do: | 09:25 |
| zigo | apt-get install extrepo | 09:25 |
| zigo | extrepo openstack_hibiscus | 09:25 |
| zigo | apt-get update | 09:25 |
| zigo | that setup everything. Otherwise it's in Debian Experimental. | 09:25 |
| gibi | zigo: thanks I found the service file | 09:26 |
| gibi | filed https://bugs.launchpad.net/nova/+bug/2167448 | 09:43 |
| opendevreview | Merged openstack/os-vif stable/2026.1: ovs: disable IPv6 on taps created by create_tap before bringing them up https://review.opendev.org/c/openstack/os-vif/+/1005663 | 10:01 |
| gibi | zigo: am I sensing right that you are OK to fix this later, after Hibiscus GA? | 11:00 |
| zigo | gibi: If only Nova is affected, that's ok, yes. | 11:09 |
| gibi | zigo: thanks. I guess you will see in you testing if other services are affected or not | 11:09 |
| zigo | I beI'm currently checking. | 11:10 |
| zigo | gibi: nova-conductor seems ok, in my CI. | 11:11 |
| zigo | It does have Type=notify and system see it as up. | 11:12 |
| gibi | thanks that is a good data point | 11:12 |
| zigo | I'm asking GLM-5 if he knows why... :P | 11:13 |
| gibi | yepp I see that locally too (super conductor is not affected even though running in threading mode) | 11:14 |
| gibi | zigo: it is due to cotyledon + nova-compute using no-fork=True | 11:26 |
| zigo | Ok, the difference, according to my AI, is because nova-conductor calls server(server, workers=N) with default to no_fork=False, but nova-compute calls it with the added no_fork=True, added in https://review.opendev.org/c/openstack/nova/+/965466 (Do not fork compute workers in native threading mode) that actually, you're the patch owner. | 11:26 |
| zigo | Yeah... :P | 11:26 |
| gibi | our LLMs are agreeing :) | 11:26 |
| gibi | I will draft a fix | 11:26 |
| zigo | gibi: Do you think we should fix this is olso.service rather than in Nova ? | 11:27 |
| zigo | (I do...) | 11:27 |
| gibi | I do too | 11:29 |
| gibi | as olso.service give the support for no-fork | 11:29 |
| gibi | but this probably means no fix in Hibiscus GA | 11:29 |
| gibi | or at least it become more complicated | 11:29 |
| gibi | due to req freeze | 11:29 |
| gibi | but it also means this bug only affects services that are not forking | 11:30 |
| gibi | (this is not first bug due to nova-compute being special, so we should also think about working there a single worker and basically load the nova-compute service there) | 11:30 |
| gibi | s/working/forking/ | 11:31 |
| zigo | Well, I can carry the patch in my Debian package. It's really only 2 lines, as much as I can see. | 11:36 |
| zigo | https://review.opendev.org/c/openstack/oslo.service/+/1005864 | 11:39 |
| zigo | :) | 11:39 |
| gibi | you type faster than me | 11:39 |
| gibi | :) | 11:39 |
| gibi | but I tested exactly the same thing now locally | 11:40 |
| gibi | and it resolve the prolbme | 11:40 |
| zigo | I'll try to patch my oslo.service Debian package and see if nova-compute is happy. :P | 11:41 |
| gibi | it is happy for me :) | 11:41 |
| gibi | please add a Closes-Bug: #2167448 line to the commit message otherwise it looks good to me | 11:42 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369 | 11:44 |
| zigo | Right. | 11:45 |
| zigo | Yes, works for me as well ! :) | 11:51 |
| zigo | I can see "Creating RPC server for service: nova-compute on topic: compute start" in my logs, what's that for?!? | 11:51 |
| zigo | Also, I'm seeing this in my log: | 11:53 |
| zigo | https://paste.opendev.org/show/b6Z3DQTexWsKxupCuLit/ | 11:53 |
| zigo | Should I worry? | 11:53 |
| zigo | Missing sentence in my above paste: A dynamic interval looping call should supply either an interval or periodic_interval_max | 11:54 |
| gibi | do you see that during shutdown? | 11:54 |
| gibi | i.e. seeing Skipping periodic tasks during graceful shutdown. | 11:55 |
| zigo | Ah, yes. | 11:55 |
| zigo | Can't see anything now, because I'm restarting my CI from scratch, but yes, it was during shutdown. | 11:55 |
| gibi | then I would not worry much especially if nova-compute exited | 11:56 |
| gibi | which I assume did to be able to start up again | 11:56 |
| gibi | I will look into it | 11:56 |
| gibi | btw it seems only nova-compute and ironic vncproxy service are using no_fork=True https://codesearch.opendev.org/?q=no_fork%3DTrue&i=nope&literal=nope&files=&excludeFiles=&repos= | 11:57 |
| gibi | yepp this is happening to me too during shutdown https://paste.openstack.org/show/bG4BK2BV3lcOqBKtC5ry/ | 11:58 |
| gibi | https://paste.openstack.org/show/bEAmK5c6VltFQ1OmI9JD/ more complete paste | 11:58 |
| gibi | so it is cosmetic, the shutdown happens | 11:59 |
| opendevreview | Merged openstack/placement stable/2025.1: Reproduce bug 2160721 https://review.opendev.org/c/openstack/placement/+/998877 | 12:05 |
| gibi | zigo: we should return a number here https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/manager.py#L131 to fulfill the expected proptocol of oslo and to avoid the stack trace, purely cosmetic | 12:32 |
| *** bauzas5 is now known as bauzas | 12:32 | |
| zigo | Oh ok. | 12:32 |
| zigo | Would be nice though ... :P | 12:32 |
| gibi | yeah I will make a patch | 12:32 |
| zigo | Any stack trace in our production triggers some alerts. :P | 12:33 |
| gibi | understandable | 12:33 |
| opendevreview | Merged openstack/placement stable/2025.1: Check overall capacity before generating candidates https://review.opendev.org/c/openstack/placement/+/998878 | 13:16 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2167481 https://review.opendev.org/c/openstack/nova/+/1005879 | 13:36 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Stop periodic tasks during shutdown https://review.opendev.org/c/openstack/nova/+/1005880 | 13:36 |
| gibi | zigo: ^^ | 13:36 |
| gibi | this removes the stack trace | 13:36 |
| gibi | cc gmaan as this is a shutdown improvement | 13:37 |
| opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | 13:39 |
| eduardomorais[m] | I'm a new contributor and I'm looking for simpler tasks to get familiar with the codebase. I found a very simple TODO and implemented what it asked for. I'd really appreciate it if you could review it! https://review.opendev.org/c/openstack/nova/+/1005881 | 13:46 |
| opendevreview | Stephen Finucane proposed openstack/placement master: Fix tests on Python 3.14 https://review.opendev.org/c/openstack/placement/+/1005883 | 13:46 |
| zigo | Thanks, added to my Debian package. | 13:52 |
| gibi | zigo: thanks for reporting the issues | 13:52 |
| gibi | Uggla: I could not join today's upstream bug triage. Just to summarize the above. We have a regression in systemd service status reporting in nova-compute but as this is limited to that service (and maybe ironic vncproxy) zigo is OK to carry a patch in debian until we can land the fix and backport to H. So this is not an RC2 trigger in my eyes (also it is an oslo.service fix) | 13:54 |
| gibi | Uggla: the secondary bug of having an extra stack trace at shutdown is just cosmetic, can wait | 13:55 |
| gibi | https://bugs.launchpad.net/nova/+bug/2167481 | 13:55 |
| gibi | https://bugs.launchpad.net/oslo.service/+bug/2167448 | 13:55 |
| gibi | this is the two bugs | 13:55 |
| opendevreview | Stephen Finucane proposed openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | 13:55 |
| gibi | both have fix proposed | 13:55 |
| opendevreview | Stephen Finucane proposed openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | 13:56 |
| Uggla | gibi, super cool thanks. | 14:10 |
| dansmith | gibi: gmaan remind me, are we aware of hanging functional tests right now? in my local series (which has some potentially impactful functional tests) I'm getting a hang | 14:12 |
| gibi | dansmith: the tox cover job showed functional hangs and I have suspicion that it might reproducable outside of cover | 14:13 |
| dansmith | oh right right that's what I was thinking of, and as you pointed out, the functional part is all its own command | 14:14 |
| gibi | https://bugs.launchpad.net/nova/+bug/2167088 | 14:14 |
| gibi | yepp | 14:14 |
| gibi | this is the bug | 14:14 |
| gibi | it has some hints where the hang is | 14:14 |
| dansmith | I'm walking up my series a bit to see if it still happens there | 14:14 |
| gibi | or at least one hang I can sometimes reproduce locally | 14:14 |
| dansmith | ah, good to know.. do you have to run cover or just functional alone? | 14:14 |
| gibi | so far I needed to run cover | 14:14 |
| gibi | and severy limit my cpu | 14:15 |
| dansmith | isn't that really bizarre? | 14:15 |
| dansmith | unless it's just the regex filter that is different there or somethin | 14:15 |
| gibi | systemd-run --user --scope -p CPUQuota=10% tox -e cover -- --concurrency 1 nova.tests.functional.libvirt.test_numa_live_migration.NUMALiveMigrationRollbackTests --until-failure | 14:15 |
| gibi | this reproduces to me semi frequently | 14:15 |
| gibi | I haven't run much of this without cover | 14:15 |
| gibi | it is on my todo list | 14:15 |
| dansmith | I'm seeing it on arm64 with 10 asymmetric cores, fwiw | 14:16 |
| dansmith | I'll try master if it repros earlier in my set | 14:16 |
| opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | 14:16 |
| gibi | btw you are touching live migration and my repro points to a live migration test class it cannot be a coincidence | 14:18 |
| dansmith | perhaps | 14:19 |
| dansmith | how did you determine it was that test class? | 14:19 |
| gibi | I looked at multiple failing runs to see what test cases did not run at all and which was the last test case executed successfully by the hanging worker | 14:24 |
| gibi | s/failing/hanging/ | 14:24 |
| gibi | then made a guess | 14:24 |
| gibi | if you reproduce locally you can see the hanging worker's assigned tests in a temp file | 14:25 |
| gibi | if you look at the command line of the hanging stestr process | 14:25 |
| gibi | in the CI we don't have that data saved | 14:25 |
| gibi | so there I can only looked at the subunit partial result to see what was finished and the rest was assumed not started or hanging | 14:26 |
| dansmith | ack | 14:27 |
| dansmith | it didn't hang earlier in the series, so I suspect it's something I've done to the live migration path | 14:28 |
| dansmith | either a total coincidence or maybe exacerbating whatever you're seeing | 14:28 |
| Uggla | Reminder upstream triage in ~30mn. (https://meet.google.com/zjr-rxus-hzj) | 14:59 |
| gibi | Uggla: see my note above for the triage | 15:01 |
| gibi | ahh you sas | 15:01 |
| gibi | saw | 15:01 |
| gibi | I missed the ack | 15:01 |
| Uggla | yep, nw | 15:01 |
| elodilles | Uggla: i'll skip today's meeting as I'm still lagging behind since coming back from summit :/ | 15:14 |
| Uggla | elodilles nw, I do not expect lots of progress due to the release. Anyway thx for letting me know. | 15:16 |
| elodilles | +1 | 15:16 |
| opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977 | 15:48 |
| opendevreview | ribaudr proposed openstack/nova master: Fix evacuate race conditions with periodic task and task_state https://review.opendev.org/c/openstack/nova/+/999978 | 15:48 |
| opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | 17:19 |
| gmaan | gibi_: thanks for periodic_task fix for shutdown case, I commented in gerrit https://review.opendev.org/c/openstack/nova/+/1005880/comment/2facdbfa_bb30dd75/ | 17:33 |
| gmaan | basically instead of raising LoopingCallDone(), I think we should return value which makes oslo.service to gracefully shutdown peropdic task thread | 17:34 |
| gmaan | gibi_: another things, we should backport this fix to stable/2026.2 right? | 17:35 |
| gmaan | btw, other fix for systemd notify in olso.service lgtm. | 17:36 |
| opendevreview | Merged openstack/placement master: CADF audit support for Placement API https://review.opendev.org/c/openstack/placement/+/933116 | 17:40 |
| opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | 21:06 |
| opendevreview | Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881 | 21:13 |
| ashigupt | gmaan: gibi_ Does https://review.opendev.org/c/openstack/nova/+/1001369 looks good to you now ? | 21:46 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!