| *** mhen_ is now known as mhen | 02:11 | |
| *** ykarel_ is now known as ykarel|off | 03:43 | |
| opendevreview | Doug Goldstein proposed openstack/nova-specs master: Expose vlan trunking in metadata/configdrive https://review.opendev.org/c/openstack/nova-specs/+/471815 | 05:35 |
|---|---|---|
| opendevreview | Doug Goldstein proposed openstack/nova-specs master: Expose vlan trunking in metadata/configdrive https://review.opendev.org/c/openstack/nova-specs/+/471815 | 05:39 |
| opendevreview | Thomas Goirand proposed openstack/nova master: Clear the leaked time override in notifications tests https://review.opendev.org/c/openstack/nova/+/1012432 | 08:49 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Cover behavioral changes in microversion guidance https://review.opendev.org/c/openstack/nova/+/1012733 | 10:11 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Move microversion flow chart https://review.opendev.org/c/openstack/nova/+/1014466 | 10:11 |
| bauzas | dansmith: gibi: I have to go errand but I do feel we maybe need to say something about the fact that the API response for instance_name is bad, see https://review.opendev.org/c/openstack/nova/+/1011367/comments/f8cf9908_4b6e5736 | 10:49 |
| gibi | responded there and dropping for food :) | 10:59 |
| zigo | Hi there! These already have +2 from Gibi, all are gate test leaks fix, all improving CI stability. It would be nice if it were merged: | 12:08 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012437 | 12:08 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012434 | 12:08 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012545 | 12:08 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012547 | 12:08 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012436 | 12:08 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012543 | 12:09 |
| zigo | https://review.opendev.org/c/openstack/nova/+/1012432 | 12:09 |
| opendevreview | Merged openstack/nova master: Make libvirt update domain name during live migration https://review.opendev.org/c/openstack/nova/+/1011366 | 12:26 |
| opendevreview | Merged openstack/placement master: Honor sys.argv when parsing the config in the WSGI entry point https://review.opendev.org/c/openstack/placement/+/1010823 | 12:26 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Fix pickling error that forces oslo.service to fall back to fork https://review.opendev.org/c/openstack/nova/+/1000956 | 13:21 |
| opendevreview | Kamil Sambor proposed openstack/nova master: DNM: Allow asking oslo.service for spawn https://review.opendev.org/c/openstack/nova/+/1013893 | 13:46 |
| gmaan | zigo: done. thanks for fixing. | 13:51 |
| zigo | :) | 13:58 |
| sean-k-mooney | zigo: i have opened them, oh just saw gmanns comment | 14:02 |
| sean-k-mooney | ok ill clsoe them again | 14:02 |
| sean-k-mooney | i was gong to say i opened them in tabs but no primises i get to them today | 14:02 |
| opendevreview | Merged openstack/nova master: Clear the leaked time override in notifications tests https://review.opendev.org/c/openstack/nova/+/1012432 | 14:33 |
| opendevreview | Merged openstack/nova master: Mock manager.LOG in the reverts_task_state test https://review.opendev.org/c/openstack/nova/+/1012436 | 14:43 |
| opendevreview | Merged openstack/nova master: Give the quota class sets tests their own shared state https://review.opendev.org/c/openstack/nova/+/1012547 | 14:43 |
| opendevreview | Merged openstack/nova master: Do not mutate the shared fake_vcpumodel in the tests https://review.opendev.org/c/openstack/nova/+/1012545 | 14:43 |
| gibi | I feel that a calming power spread across our gate... | 14:44 |
| sean-k-mooney | perhaps it has taken up meditations or leaned som new breathing exercieses | 14:49 |
| sean-k-mooney | slow breath in, hold, exahle and merge | 14:49 |
| opendevreview | Merged openstack/nova master: Stop CastAsCallFixture leaving a phantom cast entry https://review.opendev.org/c/openstack/nova/+/1012437 | 15:04 |
| opendevreview | Merged openstack/nova master: Clear the guestfs forceTCG flag in the libvirt fixture https://review.opendev.org/c/openstack/nova/+/1012543 | 15:04 |
| opendevreview | Dan Smith proposed openstack/nova master: Warn API users about instance_name inaccuracy https://review.opendev.org/c/openstack/nova/+/1014542 | 15:10 |
| rm_work | so I've been running this in my cloud for a while and it solves some issues for us that we saw with bringing hypervisors back up: https://review.opendev.org/c/openstack/nova/+/1009885 | 15:13 |
| rm_work | would love if anyone had time to glance at it and give thoughts | 15:13 |
| sean-k-mooney | rm_work: so we really shoudl be remvoeing the ablity to skip waitign for vif plugged events | 15:14 |
| sean-k-mooney | they were intoduced becuase without it booting a vm can fail or be unsafe | 15:14 |
| sean-k-mooney | i.e. fiale to intialise the vm networking and bootstrat via cloud init | 15:15 |
| sean-k-mooney | or be unself because for some network abckend espcialy after a reboot | 15:15 |
| sean-k-mooney | neutron may not have configured the tenant isolation | 15:15 |
| dansmith | rm_work: probably want gmaan to look at that has he has been working on shutdown and RPC robustness and what you describe is sort of related (although I'm surprised to read it, tbh) | 15:15 |
| sean-k-mooney | rm_work: by the way wait_for_vif_plugged_event_during_hard_reboot was added to work around some bugs in specific neutorn backends | 15:17 |
| sean-k-mooney | gibi: ^ you might rememebr which backend betwer then i | 15:17 |
| sean-k-mooney | rm_work: in any case the resumetion of the guest on boot shoudl not be happign until after the rpc server is up | 15:18 |
| gmaan | rm_work: dansmith sean-k-mooney its surprise for me that how it worked till now as rpc servers are started after init_host | 15:21 |
| dansmith | right, same, which is why I'm surprised to see it | 15:21 |
| dansmith | wasn't sure if maybe it had changed (either better or worse) with the compute-alt stuff | 15:22 |
| sean-k-mooney | im not sure | 15:22 |
| gmaan | not sure why it was done that way but actual fix should be to bring up the RPC servers up and then init_host | 15:22 |
| sean-k-mooney | but obviosuly without the enpont up we coudl not get the external event | 15:22 |
| gmaan | i do not think so but let me confirm | 15:22 |
| sean-k-mooney | i dont think skiping waiting is the right approch here | 15:22 |
| dansmith | definitely not | 15:23 |
| sean-k-mooney | we shoudl move starting the istances later | 15:23 |
| sean-k-mooney | we do have some level of rpc aviabel already since we have already done the compute service checks before we call _init_instnaces | 15:23 |
| gmaan | it seems same before also https://github.com/openstack/nova/blob/stable/2025.2/nova/service.py#L157-L190 | 15:23 |
| sean-k-mooney | and got the actul instnce object form the db | 15:23 |
| sean-k-mooney | so the indreiction api is workign so we have partial rpc aviabality to at least talk to the conductor at this point | 15:24 |
| dansmith | gmaan: the bug specifically mentions compute-alt | 15:24 |
| dansmith | which smells like a recent regression | 15:25 |
| dansmith | but also the bug and several comments were LLM-generated so I think skepticism is warranted :) | 15:25 |
| gmaan | dansmith: yeah but before that server external events were on 'compute' rpcserver which was also started after init_host. that is why i wondered how it worked before or was not working | 15:25 |
| dansmith | gmaan: yeah, I know | 15:25 |
| dansmith | and given the lack of response here, I wonder if the irc persona is also not a human :D | 15:28 |
| gmaan | or maybe no one set this wait_for_vif_plugged_event_during_hard_reboot ? rm_work did you enabled it recently or it was working before? | 15:29 |
| opendevreview | Dan Smith proposed openstack/nova master: Warn API users about instance_name inaccuracy https://review.opendev.org/c/openstack/nova/+/1014542 | 15:31 |
| gibi | ^^ +2 cc bauzas | 15:33 |
| bauzas | +Wd | 15:35 |
| cardoe | sean-k-mooney: bauzas: since I needed to change the target from 2026.2 to something newer... I've also updated some of the wording from the original spec. https://review.opendev.org/c/openstack/nova-specs/+/471815 | 15:41 |
| gmaan | DB updates etc are ok as conductor service is up with their RPC but I am wondering how any of the other instances repairing (reboot, stop, delete) working? | 15:44 |
| gmaan | it should be easily reproducible as order of host_init and RPC servres start seems clearly wrong | 15:44 |
| sean-k-mooney | cardoe: i assume that is for the metadata for bonding? | 15:44 |
| sean-k-mooney | ah vlan turnking | 15:45 |
| cardoe | For trunking. | 15:45 |
| dansmith | gmaan: right, which is why I don't think it's fundamentally a correct assertion and thought maybe it's a compute-alt regression | 15:45 |
| cardoe | We already do bonding. | 15:45 |
| sean-k-mooney | cardoe: ack ya i could not recall which was the gap | 15:45 |
| sean-k-mooney | im reviewing the cross cell migrtion spec now but ill review that next | 15:46 |
| cardoe | Just hoping to reduce my pile of downstream patches for nova one day | 15:46 |
| cardoe | Half of them are proposed backports by other folks | 15:47 |
| sean-k-mooney | ack im still ok with the idea in genreal ill see what you updated but in princiapl i have no issue with actully making the change if soem else it willing to revew it as well | 15:47 |
| gmaan | dansmith: let me try to push reproducer and manual testing and see if any magic happening somewhere :) or it was a unknown broken things (which i doubt). | 15:49 |
| dansmith | gmaan: ack, but also we can/should push back on rm_work to respond here after dropping that bug and also help provide the repro conditions so we can understand :) | 15:53 |
| gmaan | dansmith: yeah, that i alrady commented on bug and gerrit | 15:57 |
| dansmith | cool | 15:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix parallel evacuations pushing instance to ERROR https://review.opendev.org/c/openstack/nova/+/1003994 | 16:12 |
| rm_work | dansmith: No the IRC persona was just doing other things for a bit 😜 | 16:58 |
| rm_work | gmaan: It was broken likely as long as our clouds were up but we just didn’t notice | 16:58 |
| rm_work | dansmith: 100% true though 😂 I wish I knew enough about nova to be more sure about the solution but I definitely can reproduce the bug | 16:59 |
| rm_work | I’m not married to that fix at all, but these days it feels lazy to report a bug and not at least TRY to fix it | 17:00 |
| rm_work | Happy to abandon for a more correct approach, and happy to test any patch in my cloud to see if it still resolves it | 17:01 |
| rm_work | I can say that the patch I proposed came from solving it internally first, proving it fixed the issue we saw, then cleaning up for upstream | 17:01 |
| rm_work | We currently carry that patch in our downstream deployments (2026.2) | 17:05 |
| rm_work | I’m looking to see if I can get you more details from my original repro notes right now, though I was getting ready for bed 😅 | 17:10 |
| opendevreview | Merged openstack/nova master: Change the instance_name_template default to use UUID https://review.opendev.org/c/openstack/nova/+/1011367 | 17:22 |
| opendevreview | Merged openstack/nova master: Warn API users about instance_name inaccuracy https://review.opendev.org/c/openstack/nova/+/1014542 | 17:22 |
| rm_work | I think the functional tests in the patch show a repro? But if you wanted a separate clean way… maybe I can set up a devstack repro as a followup CR. | 17:26 |
| rm_work | Here’s my original(ish) incident notes: https://paste.opendev.org/show/837024/ | 17:26 |
| rm_work | I had an LLM sanitize it, but I guess I’d be willing to DM an un-sanitized version, my org is just really picky about what I post <_< | 17:26 |
| rm_work | Err I guess it would need to not be a follow up CR, because then I couldn’t repro 😂 | 17:26 |
| rm_work | I need to sleep but I’m going to see if one of my agents can get a devstack repro posted | 17:34 |
| opendevreview | Adam Harwell proposed openstack/nova master: DNM: Reproduce bug 2170422 on DevStack https://review.opendev.org/c/openstack/nova/+/1014572 | 17:35 |
| rm_work | My confidence level in this working the first time is … idk, low? But we’ll see, and I am confident it can revise until it works | 17:35 |
| opendevreview | Adam Harwell proposed openstack/nova master: DNM: Reproduce bug 2170422 on DevStack https://review.opendev.org/c/openstack/nova/+/1014572 | 18:02 |
| rm_work | Haha yep | 18:03 |
| gmaan | rm_work: thanks for reproducer. | 18:52 |
| gmaan | no issue we can talk about it more next week, I will try to dig into the reordering of things and if any issue on that | 18:53 |
| gmaan | but yes its friday and lot of other things to finish so I will check it next week | 18:53 |
| opendevreview | Dan Smith proposed openstack/nova-specs master: Add cross-cell-live-migration spec https://review.opendev.org/c/openstack/nova-specs/+/1008081 | 19:43 |
| opendevreview | Dan Smith proposed openstack/nova-specs master: Add cross-cell-live-migration spec https://review.opendev.org/c/openstack/nova-specs/+/1008081 | 19:45 |
| opendevreview | Dan Smith proposed openstack/nova-specs master: Add cross-cell-live-migration spec https://review.opendev.org/c/openstack/nova-specs/+/1008081 | 20:39 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!