| opendevreview | Merged openstack/nova master: doc: Add releases information for SEV-ES and SEV-SNP https://review.opendev.org/c/openstack/nova/+/1002164 | 00:10 |
|---|---|---|
| opendevreview | Merged openstack/nova master: weights: add regression test for JSON log TypeError (bug 2028518) https://review.opendev.org/c/openstack/nova/+/989029 | 00:10 |
| sean-k-mooney | melwitt: ya i had to special case get_endpoint_data in my patch as weel https://review.opendev.org/c/openstack/nova/+/994579/4/nova/accelerator/cyborg.py#148 | 00:11 |
| sean-k-mooney | .get_endpoint() does not incldue the microversion info | 00:12 |
| sean-k-mooney | so you need to extract the info via the session | 00:12 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900 | 00:45 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | 00:45 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | 00:45 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | 00:45 |
| opendevreview | Merged openstack/nova master: Fix PEP-765 syntax warning https://review.opendev.org/c/openstack/nova/+/988636 | 01:05 |
| opendevreview | Merged openstack/nova master: Drop description for ancient horizon https://review.opendev.org/c/openstack/nova/+/989419 | 01:06 |
| opendevreview | Merged openstack/nova master: Update contributor guide for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova/+/961896 | 01:06 |
| opendevreview | Merged openstack/nova master: Update contributor guide for 2026.2 Hibiscus https://review.opendev.org/c/openstack/nova/+/997870 | 01:06 |
| opendevreview | Merged openstack/nova master: Filter out deleted instances when looking for build timouts https://review.opendev.org/c/openstack/nova/+/880125 | 01:31 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix graceful shutdown functional unstable tests https://review.opendev.org/c/openstack/nova/+/1002223 | 02:27 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix graceful shutdown functional unstable tests https://review.opendev.org/c/openstack/nova/+/1002223 | 02:30 |
| gmaan | gibi: melwitt ^^ this is possible fix for the graceful shutdown unstable tests. We can see how it goes, I am not closing bug in case there are other race exist there. | 02:31 |
| gmaan | I added it on top of other fixes | 02:31 |
| opendevreview | Clif Houck proposed openstack/nova master: perf(ironic): eliminate O(N²) ProviderTree deepcopy at startup https://review.opendev.org/c/openstack/nova/+/980676 | 03:13 |
| opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | 03:13 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Add functional test for cleanup_host cancelling queued live migrations https://review.opendev.org/c/openstack/nova/+/986477 | 03:40 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | 05:02 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900 | 05:47 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | 05:47 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | 05:47 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | 05:47 |
| *** benj_5 is now known as benj_ | 07:20 | |
| *** nicolairuckel_ is now known as nicolairuckel | 08:00 | |
| gibi | I'm pretty sure that some of the instability is due to threading allows switching between tasks at any point of execution and then we have the CI slowdown that forces more switches resulting in more race conditions materializing now | 08:43 |
| gibi | I ran a lot of functional-threading tests locally and reasonable amount of it in CI before we made the functional-py313-threading voting and it was all shiny 3 weeks ago. Now I have multiple bugs filed on new race conditions | 08:44 |
| gibi | gmaan: https://review.opendev.org/c/openstack/nova/+/1002223 needs more tweaking according to CI | 08:55 |
| gibi | I assigned two bugs to myself and started looking for a root cause / fix https://bugs.launchpad.net/nova/+bug/2164606 and https://bugs.launchpad.net/nova/+bug/2164893 they are more frequent than the rest of non-graceful hits | 09:04 |
| gibi | btw I can trigger some new race conditions locally if I artificially restric the amount of cpu and io I give to tox via | 09:51 |
| gibi | systemd-run --user --scope -p CPUQuota=400% -p IOReadBandwidthMax="/ 2M" tox -e functional-py313-threading | 09:51 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize TestInstanceNotificationSample.test_instance_action https://review.opendev.org/c/openstack/nova/+/1002246 | 09:55 |
| gibi | OK that is one fix for the many ^^ | 09:56 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Remove unused default argument https://review.opendev.org/c/openstack/nova/+/922073 | 10:07 |
| sean-k-mooney | 'compute.exception' is a notification? | 11:16 |
| sean-k-mooney | is that like any uncuaght excpetion on the compaute agent | 11:16 |
| sean-k-mooney | or something we send at a specific time | 11:16 |
| sean-k-mooney | i mean that definetly a think that exists and we are waitign for it in rebuild filaers https://github.com/openstack/nova/blob/9aa9a54e04fdfec9a8775bf90488558783179d4b/nova/tests/functional/test_servers_resource_request.py#L2230-L2232 | 11:19 |
| sean-k-mooney | so the pattern looks right | 11:19 |
| gibi | sean-k-mooney: yepp compute.exception is a catch all notification | 12:03 |
| sean-k-mooney | i +2'd with a note about doing it for our only other *_with_error helper as well | 12:04 |
| sean-k-mooney | i assume we may have a similar race for share detach as share attach | 12:05 |
| gibi | yepp you are right and I'm updating the patch | 12:06 |
| sean-k-mooney | i dont know if it has a simialr assertion however hence +2, if you think its needed ill happilry re review | 12:06 |
| sean-k-mooney | cool i didnt really want ot dig into that to confirm | 12:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize TestInstanceNotificationSample.test_instance_action https://review.opendev.org/c/openstack/nova/+/1002246 | 12:15 |
| gibi | sean-k-mooney: fixed it up ^^ | 12:15 |
| sean-k-mooney | oh it was listing them but only asserting 2 | 12:32 |
| sean-k-mooney | fun | 12:32 |
| gibi | :) | 12:32 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Stabilize TestInstanceNotificationSampleWithMultipleCompute https://review.opendev.org/c/openstack/nova/+/1002281 | 13:06 |
| gibi | another fix ^^ | 13:06 |
| gibi | I'm taking https://bugs.launchpad.net/nova/+bug/2164888 as well now | 13:10 |
| clif | anyone familiar with a mypy failure around "ExceptionGroup" it complains the name is not defined despite it being a builtin... | 13:25 |
| clif | also are nova tox functional tests able to be run locally? | 13:31 |
| sean-k-mooney | an excpetion group is not a type | 13:31 |
| sean-k-mooney | its a collection of types | 13:32 |
| sean-k-mooney | as far as im aware at least | 13:32 |
| sean-k-mooney | but no i ahve not seen that | 13:32 |
| clif | wdym, it is a type isn't it? https://docs.python.org/3/library/exceptions.html#ExceptionGroup | 13:32 |
| sean-k-mooney | hum ok added in 3.11 | 13:33 |
| sean-k-mooney | it wasnt a type before this is the first release we have been abot to use 3.11 features | 13:33 |
| sean-k-mooney | before expction groups were jsut a tuple of possible expctions | 13:33 |
| clif | it seems to work fine based on local testing but mypy blows up on it in tox pep8 | 13:34 |
| sean-k-mooney | why are you trying to define it in a type signigure? | 13:34 |
| sean-k-mooney | 3.11 was for derive not the full feature | 13:35 |
| clif | I'm not sure what you mean, I'm not using it as a type signature but as a raised exception due to stephenfin's review comment | 13:40 |
| sean-k-mooney | i would not use it here for example https://review.opendev.org/c/openstack/nova/+/980679/12/nova/compute/manager.py#11572 | 13:40 |
| sean-k-mooney | which comment | 13:40 |
| clif | https://review.opendev.org/c/openstack/nova/+/980679/comment/a6f02680_f3646440/ | 13:41 |
| stephenfin | clif: Got the review link to hand? I can take a look | 13:41 |
| stephenfin | nvm :) | 13:41 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/980679/comment/a6f02680_f3646440/ | 13:41 |
| sean-k-mooney | so i dont think we shoudl be raiseing an execption groupu there | 13:41 |
| sean-k-mooney | i intentlaly was raisign only the first excption | 13:41 |
| sean-k-mooney | we coudl do that but we cant assuem the srounding code will handle the group properly | 13:42 |
| stephenfin | clif: look at tool.mypy.python_version in pyproject.toml :( | 13:42 |
| clif | stephenfin: aha :( | 13:43 |
| clif | is it fine to bump that to 3.11? | 13:43 |
| stephenfin | not the first itme I've been bitten by that https://github.com/python/mypy/issues/19349 | 13:43 |
| sean-k-mooney | ah 3.10 so it was a 3.11 related chagne | 13:43 |
| stephenfin | clif: yeah, it should have been done when we bumped project.requires-python but obviously got missed | 13:44 |
| stephenfin | just remember to run `tox -e mypy` locally before submitting since mypy _might_ raise new warnings about e.g. things that were newly deprecated in Python 3.11 | 13:45 |
| stephenfin | sean-k-mooney: ExceptionGroup is new in 3.11, yup | 13:45 |
| sean-k-mooney | will an expction group match an excpt block that only matches on the encloded exeptions? | 13:45 |
| sean-k-mooney | and will it properly handel all of them or just one | 13:45 |
| stephenfin | I believe so, but there's a very easy way to test it ;) | 13:45 |
| opendevreview | Clif Houck proposed openstack/nova master: Bump python_version for mypy to 3.11 https://review.opendev.org/c/openstack/nova/+/1002290 | 13:45 |
| sean-k-mooney | stephenfin: right so the issue i have is your chanign the behvior of the algorthim not just changing the typing | 13:46 |
| sean-k-mooney | again it was intentlly only rasing the first excption not all of them | 13:46 |
| sean-k-mooney | it was loging all of them | 13:46 |
| stephenfin | Sorry, what algorithm was changed? | 13:49 |
| sean-k-mooney | it now raising all the expctions as a group instead of just the first one | 13:50 |
| sean-k-mooney | in the non paralle mode it would have failed after the first expction | 13:51 |
| sean-k-mooney | so the intent was to keep the external behvior the same | 13:51 |
| sean-k-mooney | we can rasie all of them as a group but that a slightly diffenernt semantic | 13:51 |
| stephenfin | (a) that desire isn't captured anywhere so I'm not sure if that was the intent | 13:52 |
| sean-k-mooney | that was part of the code i wote | 13:52 |
| sean-k-mooney | before clif took it over so that was the intent | 13:52 |
| stephenfin | and (b) the behavior has already changed since in the non-parallel mode, only N/M operations would have run (where the Nth operation raised the exception) whereas now M operations will run | 13:52 |
| stephenfin | if operation N raises FooException and N+X raises BarException, raising only FooException is lossy and makes things harder to debug. My suggesion was to try to avoid that | 13:54 |
| sean-k-mooney | both can work but im not sure what we get by raisign all of them | 13:54 |
| sean-k-mooney | stephenfin: that why all of them were logged | 13:55 |
| sean-k-mooney | for debugging | 13:55 |
| sean-k-mooney | stephenfin: we intend to backport this | 13:55 |
| sean-k-mooney | so that why im qustioning using a 3.11 feature | 13:55 |
| stephenfin | okay, backporting does change the equation. I wasn't aware of that | 13:55 |
| sean-k-mooney | well we intned to backprot he prevsiou patch this wone was deabateable | 13:55 |
| sean-k-mooney | we can modify it on backport if requried but ya the hope ofr the O(n2) patch at least was to backport to all stable branches | 13:57 |
| sean-k-mooney | the parallisze i know operator will want it backported but im not sure it fully fits stabel policy | 13:58 |
| stephenfin | okay, if that's the plan then yeah, we'd want to go back to just raising the first exception | 13:58 |
| sean-k-mooney | so that patch may be propsoedn abut not mergd | 13:58 |
| sean-k-mooney | *may be proposed but not merged | 13:59 |
| sean-k-mooney | we can do both we could add a patch to swap to excption groups as a followup | 13:59 |
| stephenfin | clif: âď¸ apologies for the slight waste of your time there đ | 14:00 |
| sean-k-mooney | but if we can keep this 3.10 compatiable it woudl be nice | 14:00 |
| sean-k-mooney | stephenfin: since your about https://review.opendev.org/c/openstack/nova/+/1002246 gibi puhsed htat to stablise a functional test mind taking a look | 14:03 |
| clif | ok, yea I can revert the ExceptionGroup change | 14:11 |
| clif | relatedly, some functional tests are failing on the parallel per-node resource update change, for instance: nova.tests.functional.libvirt.test_pci_in_placement.PlacementPCIInventoryReportingTests.test_device_reconfiguration_with_allocations_config_change_stop fails because the exception raised has changed | 14:12 |
| clif | any advice on that? | 14:14 |
| clif | well now that I think about it some the failures may be related to the exceptiongroup change | 14:19 |
| clif | ah yea it was the exceptiongroup changing how exceptions got raised, makes sense | 14:34 |
| opendevreview | Clif Houck proposed openstack/nova master: Parallelize per-node resource updates https://review.opendev.org/c/openstack/nova/+/980679 | 14:39 |
| gmaan | gibi: ack, will check | 15:05 |
| gibi | gmaan: thanks | 15:19 |
| gmaan | I saw this again, I think this hit twice, I will work on this today https://bugs.launchpad.net/nova/+bug/2164880 | 15:23 |
| gibi | gmaan: thanks | 15:26 |
| melwitt | thank you gmaan for working on these | 15:29 |
| melwitt | sean-k-mooney: gotcha, I see. yeah. really hope we can switch to sdk rather than add those one-off ... the patch is not too ugly (I hope!) | 15:31 |
| stephenfin | melwitt: I'm +2 on https://review.opendev.org/c/openstack/nova/+/1002007 again. The ksa fix is also merged and should be released this week, so we could potentially remove that subclass next week. Just need sean-k-mooney to revisit that now (hint hint) | 16:56 |
| sean-k-mooney[m] | cool we are at the park with frya ill be home in a bit and ill take a look then | 17:00 |
| melwitt | stephenfin: super, thank you | 17:22 |
| opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Use keystoneauth1 Adapter for endpoint discovery https://review.opendev.org/c/openstack/nova/+/1002359 | 17:28 |
| sean-k-mooney | stephenfin: melwitt im pretty sure that https://review.opendev.org/c/openstack/nova/+/1002007/8/nova/accelerator/cyborg.py is incorrect | 18:07 |
| sean-k-mooney | stephenfin: melwitt we do not use an admin client today to talk to cyrbog | 18:07 |
| sean-k-mooney | wew shoudl in the future but we dont today | 18:08 |
| sean-k-mooney | and that cached client is confging it ot an admin client | 18:08 |
| sean-k-mooney | the cachign itself might makes sense in the future but not today when we alwasy need to use the users token | 18:08 |
| sean-k-mooney | i think the problem is that this is folow the same pattern we use to tak to ironic and manilla but the cybrog interaction is not the same as the inronic interaction | 18:09 |
| melwitt | sean-k-mooney: hm ok, so that should be admin=False. I had thought it was but that must have been an earlier iteration | 18:09 |
| sean-k-mooney | ya which means we take the if brnach here https://review.opendev.org/c/openstack/nova/+/1002007/8/nova/utils.py#940 | 18:10 |
| sean-k-mooney | not the else | 18:10 |
| sean-k-mooney | but if we take the if brnach we done need to chagne the signigure to pass in ksa_auth | 18:11 |
| sean-k-mooney | if admin false branch already correctly constuct the service user token and uses the uers token form the context to constuct the sdk connection | 18:12 |
| melwitt | argh dammit | 18:12 |
| melwitt | sorry | 18:13 |
| sean-k-mooney | on a side note i messed up my grub config on one of my dev hosts, i do not know the root password adn signle user mode didnt work so im not reinstalling an os | 18:13 |
| sean-k-mooney | so its one of those days | 18:13 |
| melwitt | I have been there | 18:14 |
| sean-k-mooney | i didnt properly unstack and swift or rather the swift loopback volume we create in devstack is preventing the boot | 18:17 |
| sean-k-mooney | melwitt: have you had a chance to check the auth change on a devstack by the way | 18:18 |
| melwitt | sean-k-mooney: no, but cyborg-tempest-py3 was definitely showing when it fails | 18:19 |
| sean-k-mooney | ack i was plannign to test it on the host im reinstalling before trying to test the nvme serise | 18:20 |
| melwitt | ++ | 18:23 |
| opendevreview | Merged openstack/nova master: Stabilize TestInstanceNotificationSample.test_instance_action https://review.opendev.org/c/openstack/nova/+/1002246 | 18:30 |
| sean-k-mooney | melwitt: my main concern woudl be that all the attachment end up beign owned by nova instead of the end user | 18:32 |
| sean-k-mooney | i.e. the client worked but the ownership was wrong | 18:32 |
| melwitt | yeah, that is an important point | 18:32 |
| sean-k-mooney | hence why i want to check it on a devstack si i can check the db to verify the ownership | 18:33 |
| melwitt | yeah, you are right | 18:36 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix graceful shutdown functional unstable tests https://review.opendev.org/c/openstack/nova/+/1002223 | 19:06 |
| opendevreview | melanie witt proposed openstack/nova master: Switch Cyborg client from keystoneauth1 to openstacksdk https://review.opendev.org/c/openstack/nova/+/1002007 | 20:11 |
| gmaan | melwitt: gibi updated this, matching the service_down_time with what any test can take time to execute, also added the root cause of why service down is happening in between of tests in note/commit msg https://review.opendev.org/c/openstack/nova/+/1002223 | 20:28 |
| opendevreview | Merged openstack/nova master: Stabilize TestInstanceNotificationSampleWithMultipleCompute https://review.opendev.org/c/openstack/nova/+/1002281 | 20:52 |
| opendevreview | melanie witt proposed openstack/nova master: Switch Cyborg client from keystoneauth1 to openstacksdk https://review.opendev.org/c/openstack/nova/+/1002007 | 21:29 |
| opendevreview | melanie witt proposed openstack/nova master: Switch Cyborg client from keystoneauth1 to openstacksdk https://review.opendev.org/c/openstack/nova/+/1002007 | 21:32 |
| melwitt | sean-k-mooney: ok, I think https://review.opendev.org/c/openstack/nova/+/1002007 should be in the right state now. I'm leaving it in WIP mode while we test it more | 22:26 |
| melwitt | sean-k-mooney: seems to work as intended on my devstack, created a server with the fake_driver device and the extended_accelerator_requests table has a record with project_id of the "demo" project I created the server with | 23:47 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!