| opendevreview | Merged openstack/nova master: unified limits: discover service ID and region ID https://review.opendev.org/c/openstack/nova/+/943507 | 01:33 |
|---|---|---|
| opendevreview | Merged openstack/nova stable/2024.1: Reproducer for bug 2098892 https://review.opendev.org/c/openstack/nova/+/942653 | 01:33 |
| opendevreview | Merged openstack/nova stable/2024.1: libvirt: Fix regression of listDevices() return type https://review.opendev.org/c/openstack/nova/+/942654 | 01:33 |
| opendevreview | Merged openstack/nova master: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945028 | 04:35 |
| mikal | sean-k-mooney: I added a section to the PTG etherpad for the SPICE VDI next steps. I am not entirely sure it needs a big discussion, but your thoughts would be welcome. | 08:38 |
| opendevreview | sean mooney proposed openstack/nova stable/2025.1: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945439 | 10:02 |
| opendevreview | sean mooney proposed openstack/nova stable/2024.2: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945440 | 10:04 |
| opendevreview | sean mooney proposed openstack/nova stable/2024.1: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945441 | 10:05 |
| opendevreview | sean mooney proposed openstack/nova stable/2023.2: wrap wsgi_app.init_application with latch_error_on_raise https://review.opendev.org/c/openstack/nova/+/945442 | 10:06 |
| sean-k-mooney | mikal:ack, i think the vdi enhancments are reasonable, you could files a specless blueprint for it now and added it to todays irc meeting we can see if we can get it approved so you can pick up where you left off | 10:26 |
| *** ykarel_ is now known as ykarel | 11:12 | |
| elodilles | sean-k-mooney: regarding yesterday's ping about backportability of the 3 patches till caracal ( https://review.opendev.org/c/openstack/nova/+/918089/3 ): to me it looked OK to backport, but reading dansmith's opinion - he has more insight in this, so - it's completely on nova team decision whether an exception to backport this should be granted or not o:) (based on the discussion, backporting to | 13:04 |
| elodilles | prior version than Caracal is not possible in my opinion). | 13:04 |
| ratailor | elodilles, sean-k-mooney, gibi could you please review these, one more +2 is required https://review.opendev.org/c/openstack/nova/+/933107 https://review.opendev.org/c/openstack/nova/+/873901 https://review.opendev.org/c/openstack/nova/+/938054 ? | 13:09 |
| elodilles | ratailor: i don't have +2 right on master branch o:) | 13:14 |
| ratailor | elodilles, ack. no problem. you can provide your suggestion, if you want to. Thanks! :) | 13:15 |
| *** haleyb|out is now known as haleyb | 13:15 | |
| elodilles | :) | 13:15 |
| sean-k-mooney | ratailor: sure, i need to context switch to somethign else shortly but +2w on https://review.opendev.org/c/openstack/nova/+/933107, im already +2 on https://review.opendev.org/c/openstack/nova/+/938054 so that needs someone else. ill quickly review https://review.opendev.org/c/openstack/nova/+/873901 | 13:25 |
| ratailor | sean-k-mooney, sure. Thanks! | 13:26 |
| sean-k-mooney | my commnets on test coverage are still open on https://review.opendev.org/c/openstack/nova/+/873901 but no one has rasied a concern in 3 months so im sort of inlcined to say we can add more later if needed | 13:27 |
| sean-k-mooney | ratailor: what ill do is the following. ill elevate form +1 to +2w but it would be nice if you could submit a followup after this merges for the happy path test i called out and other nits. | 13:29 |
| ratailor | sean-k-mooney, sure. I will do. | 13:30 |
| opendevreview | ribaudr proposed openstack/nova master: FUP Remove unnecessary PCI check https://review.opendev.org/c/openstack/nova/+/944105 | 13:34 |
| opendevreview | ribaudr proposed openstack/nova master: FUP improve comment accuracy and variable naming for tag removal https://review.opendev.org/c/openstack/nova/+/943124 | 13:34 |
| opendevreview | ribaudr proposed openstack/nova master: FUP Add a warning to make non-explicit live migration request debugging easier https://review.opendev.org/c/openstack/nova/+/944133 | 13:34 |
| opendevreview | ribaudr proposed openstack/nova master: FUP Update pci-passthrough and virtual-gpu documentation https://review.opendev.org/c/openstack/nova/+/944153 | 13:34 |
| opendevreview | ribaudr proposed openstack/nova master: FUP: Improve libvirt fixture for hostdevs https://review.opendev.org/c/openstack/nova/+/945232 | 13:34 |
| opendevreview | ribaudr proposed openstack/nova master: FUP improve and add integration tests for PCI SR-IOV servers https://review.opendev.org/c/openstack/nova/+/944106 | 13:34 |
| elodilles | btw, if any nova core is ready for a 10 sec review (of a bot generated patch): https://review.opendev.org/c/openstack/nova/+/944890 | 13:38 |
| bauzas | elodilles: done | 13:39 |
| elodilles | thanks \o/ | 13:48 |
| dansmith | ...there must be some sort of magic to get failing gabbit tests to actually show me logs... | 14:04 |
| gibi | dansmith: does OS_DEBUG=1 helps? | 14:07 |
| dansmith | it does not | 14:07 |
| gibi | :/ | 14:07 |
| gibi | it was too long ago when last time I debugged gabbit to remember | 14:07 |
| dansmith | yeah, I've been tearing through infrastructure trying to hack it to give me logs or see why it's swallowing it, | 14:08 |
| dansmith | but it's just very focused on the gabbit result | 14:08 |
| dansmith | glance functional tests are sort of similar.. really really hard to get the backend logs when there's a failure | 14:08 |
| dansmith | super annoying | 14:08 |
| gibi | I agree | 14:14 |
| sean-k-mooney | i was going to suggest the same OS_DEBUG=1 i confirmed it is allowed in the tox config so its not that either | 14:16 |
| sean-k-mooney | its usign the normal log fixtures but it also has 2 addtional ones for error and warning | 14:18 |
| sean-k-mooney | https://github.com/openstack/placement/blob/master/placement/tests/functional/base.py#L54-L58 | 14:18 |
| sean-k-mooney | so i think its those last two that are the problem | 14:18 |
| dansmith | yeah, I'm about to just repro the gabbit that is failing in python so I can effing see the problem | 14:18 |
| sean-k-mooney | i woudl ty comemntign out lines 57 and 58 | 14:18 |
| sean-k-mooney | and see fi it woks with os_debug then | 14:18 |
| dansmith | already tried :) | 14:19 |
| sean-k-mooney | well you can do the tried and true Raise RuntimeError instead of logging | 14:20 |
| dansmith | I have no idea where it's failing | 14:22 |
| sean-k-mooney | https://github.com/openstack/placement/blob/master/placement/tests/functional/fixtures/capture.py#L40-L62 | 14:22 |
| dansmith | it's a 500, which should be logging lots | 14:22 |
| sean-k-mooney | oh gabbit also has its own fixtures with its own logging config. it does include that however | 14:24 |
| sean-k-mooney | https://github.com/openstack/placement/blob/master/placement/tests/functional/fixtures/gabbits.py#L62 | 14:24 |
| dansmith | there are multiple layers of log interception in placement, and also in gabbi yeah | 14:24 |
| dansmith | I'm wondering if gabbi has changed in some way and left placement stuck | 14:25 |
| dansmith | to be clear, I've tried to mangle that capture module and I'm not sure it's ever even getting hit | 14:25 |
| dansmith | I can define error, warning, etc on there and raise and nothing changes | 14:25 |
| dansmith | not sure if that's because it ends up with another nameless 500 or what | 14:26 |
| sean-k-mooney | well if you are looging and its intercepted it should show up in one fo those fixtures | 14:27 |
| dansmith | to be clear, I'm not logging | 14:27 |
| dansmith | there's a gabbit test that now fails with 500 after a change I made | 14:28 |
| dansmith | and I'm trying to figure out what the 500 is | 14:28 |
| sean-k-mooney | oh you want ot see that ok | 14:28 |
| dansmith | but I agree, it should be going through that fixture | 14:28 |
| sean-k-mooney | so it has to be realted to this block https://github.com/openstack/placement/blob/master/placement/tests/functional/fixtures/gabbits.py#L62-L71 | 14:28 |
| dansmith | I mean, you'd think, but there are similar fixtures also in functional/base | 14:30 |
| dansmith | and in gabbi itself | 14:30 |
| sean-k-mooney | oh you think that the fact we are inheriting form the gabbit testcase might be altering the behavior | 14:31 |
| sean-k-mooney | looking at git blame https://github.com/openstack/placement/commit/fcd1e538d9b7d315cb3f74c83f1a9e4ddcab180e | 14:31 |
| sean-k-mooney | it same like there was a lot of log output in the past and they added thsi handelign ot silance some of that | 14:31 |
| dansmith | I removed all those fixture setups in gabbits.py and no change | 14:31 |
| dansmith | right, I'm sure it's the attempt to make it pretty, but it has made it completely un-debuggable, AFAICT | 14:32 |
| sean-k-mooney | i proably shoudl be doing somethign else bug you peaked my interest | 14:32 |
| sean-k-mooney | do you have a patch i can pully or aplly and see if i can replicate it | 14:32 |
| dansmith | I can push it up | 14:33 |
| opendevreview | Dan Smith proposed openstack/placement master: Reproduce bug 2104040: allocate in over-capacity https://review.opendev.org/c/openstack/placement/+/945464 | 14:34 |
| opendevreview | Dan Smith proposed openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | 14:34 |
| dansmith | sean-k-mooney: ^ | 14:34 |
| sean-k-mooney | the secodn one is causign the 500 ya | 14:35 |
| dansmith | maybe the actual thing failing will get hit by a tempest test and I can see what the effing problem is :) | 14:35 |
| dansmith | yeah | 14:35 |
| sean-k-mooney | well placmeent does not use eventlet so in theory i can also run this under a debugger in vscoded and set a breakpoint | 14:36 |
| sean-k-mooney | i have never tried that with placment but i have got that to work wiht some of watchers unit tests | 14:37 |
| dansmith | if I knew where to set a breakpoint I could do the same :) | 14:37 |
| sean-k-mooney | you can make the debugger tirgger on uncaught excetpions | 14:37 |
| sean-k-mooney | im going ot make sure i can repoduce the fialure first and then ill try something more fancy | 14:38 |
| dansmith | thanks, maybe I've got tunnel vision on something | 14:39 |
| sean-k-mooney | ... we still have not merget the patch to move form psycopg2 to psycopg2-binary... | 14:44 |
| sean-k-mooney | oh we didnt propose that ot placment | 14:44 |
| sean-k-mooney | ill fix that later | 14:44 |
| sean-k-mooney | if we use psycopg2-binary instead we dont need the postgress development headers to be able to compile it | 14:45 |
| sean-k-mooney | dansmith: so i ran the functional tests and everything passed | 14:47 |
| sean-k-mooney | oh nevermind | 14:47 |
| sean-k-mooney | that with master i have not applied you change yet | 14:47 |
| sean-k-mooney | i did see "TransactionFactory already started, not reconfiguring." pirnted once or twice which is a bit suss but it does nto seam to be breaking anything | 14:48 |
| dansmith | master passes for me too :) | 14:49 |
| dansmith | I think I've just guessed the thing it's failing on by trial and error, which is good, but it's like debugging in the dark | 14:49 |
| sean-k-mooney | so i have output | 14:50 |
| sean-k-mooney | and two failed testcases | 14:50 |
| sean-k-mooney | https://paste.opendev.org/show/baZsKMeiO9mOxyGz2OlY/ | 14:51 |
| dansmith | ...right | 14:51 |
| dansmith | that's just gabbit output | 14:51 |
| dansmith | not the logging from the wsgi app to show you where the 500 was | 14:51 |
| sean-k-mooney | ah you want the api logs form the service right | 14:51 |
| dansmith | yeah, so I can see, you know, the *actual* problem | 14:51 |
| sean-k-mooney | right well now that i know which tests are failing i can see if i can get more output sicn ei an just target those specific tests | 14:53 |
| Uggla | sean-k-mooney, do you think we should have a cross team nova-watcher session at the PTG ? | 15:05 |
| sean-k-mooney | sigh... the way we dynmically load teh gabit tst means that they are not discoverd wehn we use pytest and the way we alias the global fixtures module with our own breaks unittests discoveray | 15:05 |
| sean-k-mooney | Uggla: am we could the topics i wanted to raise are not only relvent to watcher but it does not need to be super formal | 15:06 |
| dansmith | sean-k-mooney: yep, already tried that, but you can use testtools directly, just not for a specific test. you have to run all the gabbit tests | 15:07 |
| sean-k-mooney | did you try the stestr pdb supprot? | 15:07 |
| sean-k-mooney | ... AttributeError: module 'placement.tests.functional.test_api' has no attribute 'allocations-bug-1778743_post_multiple_allocations' | 15:10 |
| sean-k-mooney | right | 15:10 |
| Uggla | sean-k-mooney, ok so I will not create a specific time slot only for watcher and will "distribute" the topic in the agenda where it fits best. Is that ok ? | 15:11 |
| sean-k-mooney | sounds good | 15:11 |
| sean-k-mooney | dansmith: so passing --pdb to stestr didnt help either how fun | 15:12 |
| dansmith | lovely | 15:13 |
| opendevreview | Dan Smith proposed openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | 15:16 |
| sean-k-mooney | dansmith: https://github.com/cdent/gabbi/commit/2469b1617fde2ee62f42ef48fc38a7d075c1dd62#diff-af026a8f8ce5b1d6bf2c2b9a33c71ba0566feb828ca0fbc8ec4f3970c2835c75 | 15:16 |
| sean-k-mooney | appreently you can mark a test as verbose | 15:16 |
| dansmith | and that lets logs go? | 15:17 |
| sean-k-mooney | not exactly https://paste.opendev.org/show/bI2SFFiDmfx51VXu7CrD/ | 15:19 |
| sean-k-mooney | you do get more output | 15:19 |
| dansmith | unhelpful output | 15:19 |
| sean-k-mooney | its jsut increasing the verbosity of the client | 15:20 |
| dansmith | yeah | 15:26 |
| sean-k-mooney | ok i have logs but i need to change code | 15:33 |
| sean-k-mooney | ill make this configuable in a env var | 15:34 |
| sean-k-mooney | tldr set this to false https://github.com/openstack/placement/blob/master/placement/tests/functional/fixtures/capture.py#L55 | 15:34 |
| dansmith | WAT | 15:35 |
| dansmith | set to false? | 15:35 |
| sean-k-mooney | yep to turn off the log capture and allow it to print to stderr | 15:35 |
| dansmith | I have commented out the usage of that thing entirely and I still don't get logs | 15:35 |
| dansmith | do you also have to disable the stdio capture? | 15:36 |
| sean-k-mooney | i also invoked it liek this wiht a few other changes | 15:36 |
| sean-k-mooney | OS_DEBUG=1 OS_LOG_CAPTURE=1 tox -e functional -- --failing | 15:36 |
| dansmith | okay are you going to push up a change so I can see what all? | 15:36 |
| sean-k-mooney | yep | 15:36 |
| dansmith | aight | 15:37 |
| sean-k-mooney | i mean its just this https://termbin.com/gj5q | 15:37 |
| sean-k-mooney | but not all of that is relevent | 15:37 |
| sean-k-mooney | ill past you the output so you can review | 15:38 |
| sean-k-mooney | then ill figure out what bits actully matter | 15:38 |
| sean-k-mooney | https://termbin.com/zacx | 15:39 |
| sean-k-mooney | File "/home/smooney/repos/placement/placement/objects/allocation.py", line 262, in _check_capacity_exceeded | 15:40 |
| sean-k-mooney | prev_used = prev_usage[alloc.resource_provider.id][rc_id] | 15:40 |
| opendevreview | Dan Smith proposed openstack/placement master: Reproduce bug 2104040: allocate in over-capacity https://review.opendev.org/c/openstack/placement/+/945464 | 15:40 |
| opendevreview | Dan Smith proposed openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | 15:40 |
| sean-k-mooney | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ | 15:40 |
| sean-k-mooney | KeyError: 0 | 15:40 |
| dansmith | OS_LOG_CAPTURE=1 sets capture_logs=True on oslo right? | 15:41 |
| sean-k-mooney | i need to see if thats actully needed but proably | 15:41 |
| dansmith | very confusing if we need to set that on the parent *and* =False on the child.. I suspect one of those and not the other is to blame | 15:42 |
| sean-k-mooney | ok so that is not relevnet | 15:42 |
| sean-k-mooney | i can set it to 0 and it still work | 15:42 |
| sean-k-mooney | i was gong to use that to contole it however | 15:42 |
| dansmith | okay, isn't that capture_logs flag telling oslo to capture the logs in memory so they can be reported with the test failure? | 15:43 |
| sean-k-mooney | so instead of hardcoding this to true defualt that to 0 or 1 i dont know what makes the most sense and allow you to override it | 15:43 |
| dansmith | I'm not sure why we'd ever want that to be false unless the reporting is wrong | 15:43 |
| sean-k-mooney | i need to read the related code to answer properly but this is the relevnet code https://github.com/openstack/oslotest/blob/1fe445863e124f8a10c1496fb8244c745e546b21/oslotest/log.py#L35-L91 | 15:45 |
| sean-k-mooney | this is why i have not pushed a patch yet im trying to load context and then figurout what the correct behavior should be before commiting and pushing | 15:46 |
| Uggla | nova meeting in aound 15mn. | 15:47 |
| sean-k-mooney | dansmith: by hardcodign it to ture we were alsways addign the fake loonger | 15:48 |
| sean-k-mooney | which make shte logs aviable in the tests | 15:48 |
| dansmith | right, but that's the intent so it gets reported to the test runner so the runner can report the test fail with logs right? | 15:48 |
| sean-k-mooney | btu im not sure if that will attach them to the tests to print on output | 15:48 |
| dansmith | yeah, idk | 15:48 |
| dansmith | so many layers and years of crust, I'm not even sure how it's supposed to work | 15:49 |
| sean-k-mooney | since we know what to flip we can easly expose that bool as an env var in any case | 15:49 |
| sean-k-mooney | dansmith: so its wrapping a logging fixture form fixtures and nuke_handeler is set to true | 15:51 |
| sean-k-mooney | https://github.com/testing-cabal/fixtures/blob/master/fixtures/_fixtures/logger.py#L39-L40 | 15:51 |
| sean-k-mooney | which prevent it going to stdout | 15:52 |
| sean-k-mooney | dansmith: i have not trast all the code it possibel we ar attachign the logs as an addtional attachment to the test | 15:53 |
| dansmith | right, but I thought things like testr were doing their own capturing of the logs, as long as they don't get swallowed so they can be reported per-test (in the "logging" section) | 15:53 |
| sean-k-mooney | there is an stester flag to print addtional attachments | 15:53 |
| dansmith | right, unittest does that with addSection() or whatever | 15:53 |
| sean-k-mooney | ya let me see if passing that flag helps | 15:53 |
| sean-k-mooney | oh that just for succesful test "--all-attachments" | 15:54 |
| dansmith | yeah | 15:55 |
| opendevreview | Merged openstack/nova master: Fix case sensitive comparison https://review.opendev.org/c/openstack/nova/+/933107 | 15:56 |
| opendevreview | Merged openstack/nova master: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/873901 | 15:57 |
| opendevreview | Merged openstack/nova master: Update master for stable/2025.1 https://review.opendev.org/c/openstack/nova/+/944890 | 15:57 |
| Uggla | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Tue Mar 25 16:00:20 2025 UTC and is due to finish in 60 minutes. The chair is Uggla. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| Uggla | Hello everyone | 16:00 |
| masahito | o/ | 16:01 |
| elodilles | o/ | 16:02 |
| gibi | o/ | 16:02 |
| Uggla | awaiting a moment for people to join. | 16:02 |
| dansmith | o/ | 16:02 |
| bauzas | \o (in another meeting) | 16:03 |
| Uggla | #topic Bugs (stuck/critical) | 16:03 |
| Uggla | #info No Critical bug | 16:03 |
| Uggla | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:04 |
| Uggla | anything about bugs ? | 16:04 |
| Uggla | #topic Gate status | 16:05 |
| Uggla | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:06 |
| Uggla | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | 16:06 |
| Uggla | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&branch=stable%2F*&branch=master&pipeline=periodic-weekly&skip=0 Nova&Placement periodic jobs status | 16:06 |
| Uggla | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:06 |
| Uggla | #info Please try to provide meaningful comment when you recheck | 16:06 |
| Uggla | anything about the gate ? | 16:06 |
| Uggla | seems not. | 16:07 |
| Uggla | #topic Release Planning | 16:07 |
| Uggla | #link https://releases.openstack.org/epoxy/schedule.html | 16:08 |
| Uggla | #info Nova deadlines are set in the above schedule | 16:08 |
| Uggla | #link https://etherpad.opendev.org/p/nova-epoxy-rc-potential | 16:08 |
| Uggla | #link https://releases.openstack.org/flamingo/schedule.html | 16:08 |
| Uggla | #info Remaining post RC1: update min servion version for a SLURP or non-SLURP release : https://review.opendev.org/c/openstack/nova/+/944018/ | 16:08 |
| Uggla | #info Nova Flamingo deadlines will be discussed at the PTG. | 16:09 |
| Uggla | We are almost good for Epoxy release. | 16:09 |
| elodilles | note that this week is the week of Final RC for 2025.1 Epoxy :) | 16:09 |
| Uggla | #topic Review priorities | 16:10 |
| Uggla | #info Flamingo priorities will be discussed at the PTG. | 16:10 |
| Uggla | #topic PTG planning | 16:10 |
| Uggla | #info Next PTG will be held on Apr 7-11 | 16:11 |
| Uggla | #link https://etherpad.opendev.org/p/nova-2025.2-ptg | 16:11 |
| Uggla | Please add the topics you want to discuss in the pad. | 16:11 |
| Uggla | #topic Stable Branches | 16:12 |
| Uggla | #info stable gates seem to be healthy | 16:12 |
| Uggla | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | 16:12 |
| Uggla | elodilles, if you want to add something please go ahead. | 16:13 |
| elodilles | yepp, one thing maybe: | 16:13 |
| elodilles | #info stable/2025.1 branch was cut for nova as well last week | 16:13 |
| elodilles | that's all from me | 16:13 |
| elodilles | back to you Uggla | 16:13 |
| Uggla | thanks elodilles | 16:13 |
| elodilles | np | 16:13 |
| Uggla | I skip the vmwareapi 3rd-party CI efforts Highlights as fwiesel is ooo this week. | 16:15 |
| Uggla | #topic Open discussion | 16:15 |
| Uggla | anything to discuss ? | 16:16 |
| keerthivasan86[m] | Just quick question why https://bugs.launchpad.net/nova/+bug/1766387 ( why tx_queue_size is not getting implemented in vhost , any specific reason ) | 16:16 |
| gibi | just to double check | 16:16 |
| gibi | we do not track anything for RC2 | 16:17 |
| sean-k-mooney | keerthivasan86[m]: if i recall corectly its only supproted in vhost-user but not kernel vhost | 16:17 |
| gibi | is it corr4ect? | 16:17 |
| sean-k-mooney | gibi: not that im aware of | 16:18 |
| keerthivasan86[m] | sean-k-mooney: As part of vhost-net able to see rx_queue_size is implemented at vm level as well. It works for vhost-net as well | 16:19 |
| Uggla | gibi, not yet. Do you think about FUP for vfio pci migration ? | 16:19 |
| sean-k-mooney | keerthivasan86[m]: when it was implented there was a specific reaosn why we didnt enabled it | 16:19 |
| gibi | Uggla: it is going to land on master probably today. I'm not advocating for pusing for an RC2 tomorrow just for that single thing. (it is not an Epoxy regression) | 16:20 |
| sean-k-mooney | keerthivasan86[m]: that may haved chnted but the current code was becasue of how qemu/libivrt worked at the time we implemtned https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/libvirt-virtio-set-queue-sizes.html | 16:20 |
| keerthivasan86[m] | sean-k-mooney: if you are ok , can i work on to implement tx_queue_size for vhost-net ? | 16:21 |
| sean-k-mooney | sure but we need to know when it started to be supported, and we need to take accound of upgrades | 16:22 |
| dansmith | the only PCI FUP that should be backported is doc stuff right? | 16:22 |
| dansmith | Uggla: gibi ^ | 16:22 |
| dansmith | the test and cleanliness things should not be backported, IMHO, but definitely not hold up rc2 | 16:22 |
| Uggla | gibi, this is what I have in mind as well. Waiting for GA as this is the only thing. | 16:22 |
| sean-k-mooney | keerthivasan86[m]: so im not sure fi it needs a spec or if a specless blueprint woudl be enough. | 16:22 |
| sean-k-mooney | dansmith: the doc is the one that really should be | 16:23 |
| sean-k-mooney | dansmith: some of the others are nice to have but not required | 16:23 |
| dansmith | backporting test cleanliness doesn't make sense to me unless it's needed for an actual fix | 16:23 |
| gibi | dansmith: for RC2 prespective only https://review.opendev.org/c/openstack/nova/+/944105/5 in interesting | 16:23 |
| sean-k-mooney | right the doc i could see for an RC2 but at this point i woudl just wait and do that after the release | 16:24 |
| sean-k-mooney | gibi: oh right to make numa work | 16:24 |
| gibi | dansmith: but as I wrote above it is not an E regression and will only land today | 16:24 |
| dansmith | gibi: ah I missed that one.. that's not really a regression i t seems, but.. | 16:24 |
| dansmith | ack yeah okay | 16:24 |
| gibi | yeah I would not spin an RC2 just for thish | 16:25 |
| gibi | but if there are other things forcing an RC2 ... | 16:25 |
| sean-k-mooney | so we can do a stable release basifcally right after the actual release | 16:25 |
| dansmith | right | 16:25 |
| Uggla | +1 | 16:25 |
| sean-k-mooney | so i think RC2 is overkill for any of those | 16:25 |
| gibi | yeah doing stable backports is the way to go if nothing else forces RC2 | 16:25 |
| sean-k-mooney | sounds like we are all in violent agreement | 16:25 |
| gibi | yepp | 16:26 |
| bauzas | yup | 16:26 |
| bauzas | (sorry, meeting was over) | 16:26 |
| Uggla | bauzas, no still in the meeting | 16:27 |
| bauzas | as a reminder, any new release candidate impacts our distros | 16:27 |
| bauzas | so we definitely only need to provide a RC2 if this is a regression | 16:27 |
| elodilles | (with my release core hat on) anyway, let me know if an RC2 is needed (or not needed) as this week is when we can release a final RC as i said o:) | 16:27 |
| elodilles | (i'll generate final RC patches (where there are merged and unreleased stuff on stable/2025.1) somewhere around 2nd half of the week fwiw) | 16:28 |
| bauzas | ack thanks | 16:29 |
| elodilles | np | 16:29 |
| dansmith | are we done? | 16:30 |
| Uggla | keerthivasan86[m], are the answer of sean ok for you ? | 16:30 |
| Uggla | any other topic ? | 16:31 |
| masahito | question for the ptg topic selection and timeslot | 16:31 |
| Uggla | masahito, the current topic are in the etherpad above. | 16:32 |
| masahito | thank. ah, yes. I added some. | 16:32 |
| Uggla | I think not all of them are available. I know new ones will come. | 16:33 |
| Uggla | Then we will organize the timeslot to discuss all of them. | 16:33 |
| sean-k-mooney | Uggla: i assume masahito wanted to ask about when there topics will be scheduled | 16:33 |
| sean-k-mooney | or perhaps suggest when tehy are aviable | 16:33 |
| masahito | my question is the ptg time slot is limited so i don't think the ptg can't cover all topics. so i want to know when and how it's decided. | 16:33 |
| sean-k-mooney | masahito: we dont have many topics currently | 16:34 |
| masahito | s/ptg can't/ptg can/ | 16:34 |
| sean-k-mooney | i woudl be surrprised if we didnt cover them all | 16:34 |
| Uggla | I think for the time slot it will be almost ok next week. | 16:34 |
| keerthivasan86[m] | yes Uggla As sean-k-mooney suggested specless blueprint or fresh blueprint to implement tx_queue_size | 16:35 |
| opendevreview | sean mooney proposed openstack/placement master: improve test logging and replace psycopg2 with psycopg2-binary https://review.opendev.org/c/openstack/placement/+/945487 | 16:35 |
| sean-k-mooney | dansmith: ^ | 16:35 |
| keerthivasan86[m] | can i create fresh blueprint to implement tx_queue_size for vhost-net feature ? | 16:35 |
| dansmith | sean-k-mooney: ack | 16:35 |
| Uggla | Masahito, we’ll try to accommodate your schedule so you can join. | 16:36 |
| masahito | i got it for the next week meeting. | 16:36 |
| Uggla | any other topic ? | 16:36 |
| masahito | thanks. I prefer 14:00UTC or later slot because of family reason as it's remote session :) | 16:36 |
| Uggla | masahito, sure I'll keep that in mind. | 16:37 |
| Uggla | looks we can close the meeting | 16:38 |
| Uggla | thanks all | 16:39 |
| Uggla | #endmeeting | 16:39 |
| opendevmeet | Meeting ended Tue Mar 25 16:39:27 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:39 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-03-25-16.00.html | 16:39 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-03-25-16.00.txt | 16:39 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-03-25-16.00.log.html | 16:39 |
| elodilles | thanks Uggla o/ | 16:39 |
| masahito | thank you o/ | 16:39 |
| Uggla | elodilles, ty | 16:39 |
| bauzas | Uggla: bravo for your first meeting :) | 16:42 |
| Uggla | bauzas, ty | 16:44 |
| gibi | Uggla: thanks | 16:46 |
| simondodsley | elodilles: not sure if you saw the conversation yesterday, but are the stable cores open to backport these 3 XS patches to resolve the SCSI page 83 passthrough issue: https://review.opendev.org/q/topic:%22lun-device-serial%22 | 17:33 |
| gboutry | Hello, I'm currently splitting my hair around https://bugs.launchpad.net/ubuntu/+source/nova/+bug/2103413 | 18:10 |
| gboutry | I'm currently running nova 31.0.0RC1 tag on python3.13, and I can't create a new instance. | 18:10 |
| gboutry | When I'm trying to, I'm getting the error described in the ticket. | 18:10 |
| gboutry | At around nova/nova/network/neutron.py:1233 (allocate_for_instance), admin_client.httpclient.dict seems to get totally wiped | 18:10 |
| gboutry | going from {'session': <keystoneauth1.session.Session object at 0x7e1a2c601950>,... 'endpoint_override': None, ...} to {} (empty dict) | 18:10 |
| gboutry | Thus failing with the error in the lp bug. the line number it happens varies but always in the function, so this looks like a finalizer going off the rails? | 18:10 |
| gboutry | This seems python 3.13 related, but I'm not sure | 18:10 |
| gboutry | I would appreciate any advice :) | 18:11 |
| sean-k-mooney | gboutry: 3.13 is not offically supproted yet | 18:13 |
| sean-k-mooney | it might work but its not actully tested | 18:13 |
| sean-k-mooney | https://github.com/openstack/governance/blob/master/reference/runtimes/2025.1.rst is the supproted python verisons | 18:13 |
| sean-k-mooney | gboutry: initall testing for 3.13 will be part of 2025.2 and it will only become requried in 2026.1 | 18:14 |
| gboutry | sean-k-mooney: thanks for the info, Epoxy is targeted to be released in Ubuntu Plucky (25.04) which is using Python 3.13, we'll most likely need to carry a patch downstream if I can find the root cause of this issue | 18:16 |
| sean-k-mooney | gboutry: i wonder if this could be eventlet related. teh error is coming form keystone authg via neutron client | 18:16 |
| gboutry | I am getting some eventlet assertion errors in the background: https://pastebin.ubuntu.com/p/WhKxDrvQHt/ | 18:17 |
| gboutry | https://github.com/eventlet/eventlet/commit/4d48d10b990910cc87ec6bbeeaea63b295fe314d python 3.13 has been added in this commit | 18:18 |
| gboutry | I'll continue poking in this direction, thanks | 18:18 |
| sean-k-mooney | so that after fork thing has come up before | 18:19 |
| sean-k-mooney | we had to workaround a behvior change i think in 3.11 or 3.12 | 18:19 |
| sean-k-mooney | to make sure we got the right id | 18:19 |
| sean-k-mooney | perhasp that broken on 3.13 again | 18:19 |
| sean-k-mooney | gboutry: https://github.com/eventlet/eventlet/issues/592 | 18:22 |
| sean-k-mooney | we remvoed that workaroudn in epoxy https://github.com/openstack/nova/commit/cd980cdd1ea957a9aa2220014046ea53079a623f | 18:22 |
| sean-k-mooney | because it has been fixed for a long time in eventlet | 18:23 |
| sean-k-mooney | gboutry: you could try reverting that again locally and see if it fixes that issue on 3.13 but its really a eventlet bug and a regression fo something they previosly fixed | 18:23 |
| sean-k-mooney | hum well not quite | 18:24 |
| gboutry | Thanks for the insight! I'll comeback later (need to take care of something), after reverting that, I'll report back! | 18:24 |
| sean-k-mooney | its a slightly diffent error | 18:24 |
| sean-k-mooney | btu realted to the same code | 18:24 |
| sean-k-mooney | cool | 18:24 |
| gboutry | yes, this happens in the new code specific for python 3.13 support haha | 18:25 |
| sean-k-mooney | gboutry: if you figure it out let us know. we will be trying to enable 3.13 this cycle and dont have issue with backproting non invaisive change to supprot it in 2025.1 but it wont officlaly be supproted for a while | 18:30 |
| mikal | sean-k-mooney: sorry I missed the meeting, I was asleep. I am not entirely sure I know what a "specless blueprint" is. Is that just a blueprint in launchpad? There is already one for the SPICE work, are you saying I need a second one for the extra specs stuff? | 19:01 |
| sean-k-mooney | mikal: a specless blueprint is just a launchpad blueprint that we agree does not require a spec so we approch based on the description of the bluepint in the irc meeting | 19:02 |
| sean-k-mooney | i.e. if we belive it not contoverioal enough of big enough to need a spec | 19:02 |
| sean-k-mooney | the current blueprint for the spice direct work will be closed as completed shortly | 19:03 |
| sean-k-mooney | we should have a new blueprint for the vdi part fo 2025.2 | 19:03 |
| mikal | Ahhh, ok. | 19:03 |
| sean-k-mooney | mikal: i dont know if folks will want a mini spec for the vdi part or be fine with a detailed blueprint | 19:03 |
| sean-k-mooney | given it basicaly code complete already and we discussed the design last cycle | 19:04 |
| sean-k-mooney | mikal: regarding sleep you can add the bleuprint to the team meeting wiki or just ask async on the mailing list for it to be dicussed at the next one | 19:05 |
| sean-k-mooney | mikal: i ment to bring it up today i just didnt have a link to the vdi work | 19:06 |
| mikal | Huh. There are a surprisingly large number of SPICE blueprints on launchpad. | 19:07 |
| sean-k-mooney | ya and a couple of video/audio/vdi ones | 19:07 |
| mikal | Yeah, this is news to me. I shall have to read them. | 19:07 |
| sean-k-mooney | like https://blueprints.launchpad.net/nova/+spec/libvirt-audio-device | 19:08 |
| sean-k-mooney | mikal: I dont think any of them are relevnet anymore | 19:08 |
| sean-k-mooney | mikal: im ot aware of anyoen actully working on them | 19:08 |
| mikal | And https://blueprints.launchpad.net/nova/+spec/libvirt-spice-video-sound-driver | 19:08 |
| sean-k-mooney | ya that the really old one | 19:09 |
| sean-k-mooney | there is also a usb redirection one | 19:09 |
| mikal | It would be nice to just close all of these (either as addressed or not) as part of landing the VDI stuff. | 19:09 |
| sean-k-mooney | honestly its proably cleaner if you just file one for libvirt-vdi or similar that covers all yoru usecases | 19:09 |
| sean-k-mooney | ya | 19:10 |
| sean-k-mooney | it would be good to clean them up | 19:10 |
| mikal | Oh for sure, but I should _read_ the ancient ones too | 19:10 |
| mikal | There might be wisdom of the ancients there or something. | 19:10 |
| sean-k-mooney | maybe most look pretty empty but not all | 19:10 |
| sean-k-mooney | mikal: if nothing else it shows there si some interest form other to improve vdi on openstack | 19:13 |
| mikal | Well, there once was. Presumably 2015 guy has gone and done something else now. | 19:13 |
| mikal | I feel like https://blueprints.launchpad.net/nova/+spec/libvirt-spice-vdi is a bit sparse, but I am really not sure what else to put in it. | 19:13 |
| sean-k-mooney | i would copy past some of the old spec that descibe the image properies/extra specs you plan to add | 19:14 |
| sean-k-mooney | mikal: by the way while your usecase is mainly spice the vdi enchnemnt wont sticrtly need spice at least for the audio devce | 19:16 |
| sean-k-mooney | not that it will be very useful with vnc via the proxy | 19:17 |
| mikal | That is true. I presume "direct VNC" would be possible if there was a suitable proxy and would use these features too, but that's forward looking. | 19:18 |
| sean-k-mooney | ya if you ever extend kerbside for that we could add it. i think you can get a better vnc conneciotn if you tunnel websocket back into a tcp connecton and use a better clinet but we dont ereally ahve anythignfor that out of the box | 19:19 |
| mikal | I have this underlying suspicion that I should be feasible to transcode between SPICE and RDP in Kerbside, and I think that might interest some people. However, the RDP specs are at least partially locked down in telco standards which cost money, and its not something I am likely to find the time for anytime soon. | 19:20 |
| mikal | s/that I should be/that it should be/ | 19:20 |
| mikal | I tweaked https://blueprints.launchpad.net/nova/+spec/libvirt-spice-vdi, but its not an essay. | 19:21 |
| sean-k-mooney | ya that looks ok. | 19:22 |
| sean-k-mooney | so the process is bascilly see if peopel think its "simple enough to not need a spec" | 19:22 |
| sean-k-mooney | if the other cores say sure we apprve it for the cycle if not we will ask for a spec and just do that dance | 19:23 |
| mikal | Ok. Should I send an email to the list, or is this conversation sufficient to get it onto an agenda somewhere? | 19:24 |
| sean-k-mooney | ideally you would add it to https://wiki.openstack.org/wiki/Meetings/Nova#Weekly_Nova_team_meeting in open dicssion section | 19:25 |
| sean-k-mooney | if you want to also add a link to the irc logs fo this converstaion that fine. | 19:25 |
| sean-k-mooney | https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html#t2025-03-25T19:01:27 | 19:26 |
| mikal | Done. | 19:39 |
| opendevreview | mitya-eremeev-2 proposed openstack/nova master: Add ability to preserve ephemeral in evacuation https://review.opendev.org/c/openstack/nova/+/938235 | 19:52 |
| opendevreview | mitya-eremeev-2 proposed openstack/nova master: Handle ephemeral root disk in evacuation https://review.opendev.org/c/openstack/nova/+/938235 | 20:00 |
| opendevreview | melanie witt proposed openstack/nova stable/2025.1: unified limits: discover service ID and region ID https://review.opendev.org/c/openstack/nova/+/945525 | 20:37 |
| gboutry | sean-k-mooney: I've tried multiple things, including reverting that patch, without much success. | 20:56 |
| gboutry | I have a small reproducer, but not sure if it is relevant in nova context? | 20:56 |
| gboutry | https://pastebin.ubuntu.com/p/t8pDTvdNqB/ | 20:56 |
| gboutry | Or even if it makes much sense, but it gets the same error, I'll dig in more tomorrow, thanks for your support! | 20:56 |
| sean-k-mooney | so what i think is happening is when we monkeypatch thread its not properly replacing the tread id with the grenthread id | 20:57 |
| sean-k-mooney | which is what casueing the assert to fail | 20:57 |
| sean-k-mooney | or somehting along those lines | 20:57 |
| sean-k-mooney | but ya that seam like an eventlet bug not a nova one | 20:57 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!