| opendevreview | Merged openstack/nova master: Disable interactive prompt on LVM image creation https://review.opendev.org/c/openstack/nova/+/990576 | 04:30 |
|---|---|---|
| opendevreview | Merged openstack/nova master: Use tempest_concurrency=1 for nova-vtpm job https://review.opendev.org/c/openstack/nova/+/984864 | 04:30 |
| opendevreview | Merged openstack/nova stable/2025.2: Fix resize to unpermitted flavors https://review.opendev.org/c/openstack/nova/+/988153 | 05:14 |
| *** ralonsoh_ is now known as ralonsoh | 07:05 | |
| sean-k-mooney | hi folks are we ok with running my cirros experiemtn for the next 2 weeks https://review.opendev.org/c/openstack/nova/+/990728 i have also created the revert https://review.opendev.org/c/openstack/nova/+/990729/1 im planning to update my pr https://github.com/cirros-dev/cirros/pull/132 to creat a seperate qcow image that is pre populated isntead of updating the existing one as | 11:35 |
| sean-k-mooney | sugested by frickley but before we commit to that and look toward a release i want to actully confirm this fixes the issues | 11:35 |
| sean-k-mooney | in terms of otehr reviews i woudl really love to close out https://review.opendev.org/c/openstack/nova/+/975872 and https://review.opendev.org/c/openstack/nova/+/984540 today if folks have time to reivew | 11:37 |
| sean-k-mooney | gibi: stephenfin if time permits can ye add those to your queue | 11:37 |
| gibi | sean-k-mooney: I trade https://review.opendev.org/c/openstack/nova/+/984540 for https://review.opendev.org/c/openstack/nova/+/986282 | 12:02 |
| gibi | :) | 12:02 |
| sean-k-mooney | sure, oh the eventlet poison ya i can look at that now | 12:03 |
| sean-k-mooney | this still works the way we dicussed right your faking an import expection in the poison so that oslo continues to work but new imporat that dont explcitly guard against that fail | 12:04 |
| gibi | yepp it still raises ImportError | 12:08 |
| sean-k-mooney | i would generally prefer if we didnt have get_eventlet() as that is allowing direct usage outside of the utils file | 12:12 |
| sean-k-mooney | but that prefence is not enought to hold up the patch | 12:12 |
| sean-k-mooney | can we consier if we can prevent leakign that as part of adressign the todo's you have in line | 12:12 |
| sean-k-mooney | gibi: commented and +2w the overall change looks good to me jsut aksed that we consider the feedback inlien as an addtion todo for the followups | 12:19 |
| opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron security-groups https://review.opendev.org/c/openstack/nova/+/981141 | 13:34 |
| *** sambork_ is now known as sambork | 13:47 | |
| opendevreview | Anton Iacobaeus proposed openstack/nova-specs master: Intel TDX support in libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/979608 | 14:02 |
| stephenfin | sean-k-mooney: done | 14:31 |
| stephenfin | (the cirros change: gibi has +W'd the other two) | 14:32 |
| sean-k-mooney | stephenfin: thanks | 14:49 |
| melwitt | gibi: thanks for hitting the unified limits fix | 15:06 |
| sean-k-mooney | from a purely upsteam point of view is that somethign we would want to backport? | 15:11 |
| sean-k-mooney | or master only | 15:11 |
| sean-k-mooney | makeing unified limits work for cybrog/pci-passhtough/neutiron qos is both a bug and a feature | 15:12 |
| sean-k-mooney | so while im happy that tis a bug that that was overlooked on master | 15:12 |
| sean-k-mooney | im not sure it qualifies for the stabel policy as a backportable bug | 15:12 |
| sean-k-mooney | so i was going to not backport premetivly and wait for someoen to ask for it | 15:13 |
| sean-k-mooney | i do know there was at least some operator intest in this so im fine with either approch | 15:14 |
| melwitt | yeah I guess that's true it was a known issue but also I think important for cloud operators, so I would lean toward being reasonable to backport upstream. that's just me though | 15:21 |
| sean-k-mooney | ya im not sure which operaors woudl find more suprising the knwo broken behvior ro that it now works if we backprot it. if elodilles has any opions on the mater i would be happy to hear them but i guess we can wait for it to actully land and then dicuss next steps | 15:25 |
| sean-k-mooney | one of those is i am planing to extend the cyborg tempest plugin to have some quota tests eventually. | 15:26 |
| sean-k-mooney | but before i do that i need to intoduce the idea of serial tests to that plugin and build out some geneic testing infra in it | 15:27 |
| opendevreview | dalekseev proposed openstack/nova master: Skip ironic instances in machine type check https://review.opendev.org/c/openstack/nova/+/991137 | 15:30 |
| elodilles | sean-k-mooney: sorry, i don't have the context o:) though as you say, if something is more like a feature, then it should not be backported upstream. but if it can be considered as a bug fix and the patch is not that invasive and the stable cores agree, then the possibility is there to backport. o:) | 15:37 |
| sean-k-mooney | elodilles: the tl;dr is our unified limits check didnt consider resouces form pci_passhtough, cybrog or neutron when it was computing the resouce requests | 15:38 |
| sean-k-mooney | elodilles: so if you condiufred the limits they were not being enfoced | 15:38 |
| sean-k-mooney | elodilles: i fixed that but it not clear if that is backportable or not | 15:38 |
| sean-k-mooney | these resouce shoudl have been condiserd in the past but its been that way for quite a while | 15:39 |
| sean-k-mooney | what this means in partice is the pci_in_placement spec assuemed/asserted that unified limite woudl provide quota for pci passhtough device when using pci in placment | 15:40 |
| sean-k-mooney | in practice it never has | 15:40 |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/zed/approved/pci-device-tracking-in-placement.html#dependencies """The unified limits feature exists in an opt-in, experimental state and will allow defining limits for the new PCI resources if enabled.""" but we eiarl defiend makign tha twork out of scope of this spec | 15:41 |
| sean-k-mooney | """Device quotas would require unified limits to be implemented. Implementing quotas is out of the scope of this spec beyond enabling the use case by modeling PCI devices in Placement.""" | 15:42 |
| elodilles | hmmm. okay, this really sounds something on the border whether it is OK to backport or not o:) | 15:43 |
| sean-k-mooney | elodilles: so its just one of those things we inteneded it to work but just neer got around to doing it till now | 15:44 |
| sean-k-mooney | ya so that why i was saying i wont premetivly backport | 15:44 |
| sean-k-mooney | and just wait until someone asks and we can deciend then | 15:44 |
| elodilles | so then it depends on the patch and whether the team agree to backport | 15:44 |
| sean-k-mooney | ya this is very much a case of "if operators ask for it and the team agrees im ok to do it" but i dont feel strongly one way or anohter | 15:46 |
| elodilles | sean-k-mooney: yepp, i have the same feeling | 15:48 |
| sean-k-mooney | elodilles: https://review.opendev.org/c/openstack/nova/+/975872 is the change we were dicussing by the way | 15:49 |
| sean-k-mooney | its still in the gate behind a bunch of openstack client changes | 15:50 |
| sean-k-mooney | so it might be a while | 15:50 |
| sean-k-mooney | *openstacksdk | 15:50 |
| melwitt | yeah, that's a fair point bc backporting it could cause "sudden" unexpected quota enforcement if we're thinking of a stable environment so yeah... maybe that might be too much | 15:51 |
| sean-k-mooney | yep | 15:51 |
| sean-k-mooney | with my downstream hat on i was also going to advocate for not backporting this for a similar reason | 15:52 |
| sean-k-mooney | obviously if a customer screams that they must have this we could consider it but i woudl prefer not to | 15:53 |
| elodilles | sean-k-mooney: but it somewhat feels then to me that is not appropriate for upstream backport, rather should be done downstream :/ as melwitt says, we change behavior :/ (even if most of the users were benefit from it...) | 15:58 |
| elodilles | anyway, this can be discussed and evaluated within the team, how safe the team feels regarding having no regression in case of backport, etc. | 16:00 |
| sean-k-mooney | elodilles: ya so im maninly jsut askign is i twork my time and everyone else for me to create teh backports and plan to do it or just punt it for now | 16:01 |
| sean-k-mooney | i think we have setteled on master only for this one | 16:01 |
| elodilles | (and maybe if we do backport, even then it should not happen immediately when the fix lands on master... we should give time this to settle and prove that things work fine) | 16:02 |
| elodilles | sean-k-mooney: so i'd say let's focus now on fixing it on master o:) but it just my opinion o:) | 16:03 |
| melwitt | yeah I'd have to double check but it _might_ be ok bc of the change we made to make only the essential resources (vcpu, ram, disk) required for enforcement and the rest treated as unlimited | 16:03 |
| melwitt | (i.e. the required for enforcement resource list is configurable) | 16:04 |
| melwitt | yeah I think we all agree for now let it bake on master and backport is not an immediate thought | 16:05 |
| elodilles | melwitt: +1 | 16:08 |
| melwitt | CI has seemed so unhappy lately 😩 | 16:17 |
| opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | 16:38 |
| opendevreview | Merged openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | 18:35 |
| opendevreview | Merged openstack/nova master: Poison eventlet import in native threading mode https://review.opendev.org/c/openstack/nova/+/986282 | 18:36 |
| opendevreview | Merged openstack/nova stable/2026.1: Add reproducer test for bug 2105896 https://review.opendev.org/c/openstack/nova/+/989519 | 18:36 |
| opendevreview | Merged openstack/nova stable/2026.1: Fix error when multiple security group in a project with same name https://review.opendev.org/c/openstack/nova/+/989520 | 18:43 |
| opendevreview | Merged openstack/nova master: Add agentic coding guidance and docs https://review.opendev.org/c/openstack/nova/+/984540 | 18:44 |
| frickler | sean-k-mooney: looks like you have uncovered a new failure mode on https://zuul.opendev.org/t/openstack/build/79d870c283264c169171f3a7a332ba3c already (ssh auth fail), maybe that's the next thing that can happen when the first boot is interrupted and the cloud-init process doesn't properly finish? | 19:21 |
| melwitt | I think I have seen that failure mode before but couldn't figure out whether or how it doesn't have the right authorized_keys | 19:25 |
| melwitt | seems like it's there, unless that doesn't mean what I thought it means | 19:28 |
| melwitt | === sshd host keys === | 19:28 |
| melwitt | -----BEGIN SSH HOST KEY KEYS----- | 19:28 |
| melwitt | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCk9qvrsW4suQOSW7BD+7S6v5Fakm6C6e6bxhoWpAi5nH7LRsFbzYj7M+bSvE0mQ64/qwoZaT6lmXFexmdfIkUSevz18TWecvg45bi4nx+sCEK25jtBXaqdGo1Bz7TU44/M4Bsj8rE1yiSHasoZ/qMrEJ4maMFZkSaub0CvqIe1NaRXRR1T/T9JLmA4IXGLN6gOKkw8NXfIRHbVwnQuNpcvCjifJlIKxWJXQq+Ttwq94apHZ0vqyg1/KMygHz6DUCU7kdRGv3/51ROKvJuwmLStjkwB8SyfcvJzQcjwkazX6DYWY7C/VEteP4ng2RfGouIOJRpjQxscsNdTG4aWgs0n root@tempest-serveractionstestothera-server-190417456 | 19:28 |
| melwitt | ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOxWzqADqCihEY+t+o3XDe0hEIagEoU8YZYIjgdXuuJhvdz3NMiyaWJ4iYnFlL1ewByK/CzXuSJCQLzLOQJjtM= root@tempest-serveractionstestothera-server-190417456 | 19:28 |
| melwitt | -----END SSH HOST KEY KEYS----- | 19:28 |
| sean-k-mooney | frickler: that migh tnae sesne if we dont inject ths ssh key or ornly partly confirue dhcp or similar | 19:30 |
| frickler | that's host key, not authorized_keys. there's "Printing cirros user authorized keys" before with no actual key shown, but I'll need to compare against a log for a successful boot | 19:30 |
| sean-k-mooney | frickler: i also found one of the ohter kernel panics the other day but its a very very rare one | 19:31 |
| melwitt | oh. mah bad | 19:31 |
| sean-k-mooney | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_17d/openstack/17d6defdcff648cbbe950cf0137c7906/testr_results.html | 19:32 |
| frickler | "cirros-apply-local already run per instance" might be a hint supporting my theory | 19:32 |
| sean-k-mooney | https://paste.opendev.org/show/bd4gViIvUmb0CjIUcvh4/ | 19:32 |
| sean-k-mooney | frickler: right so if we got far enough to touch the file sayign we are runing or have run our first init script | 19:32 |
| sean-k-mooney | but we have not sync'd the file system or completeded it | 19:33 |
| sean-k-mooney | it possibel the dhcp or other config wont survive the reboot | 19:33 |
| sean-k-mooney | frickler: so the patch to use my custom image is currently in the gate queu | 19:34 |
| sean-k-mooney | so we shoudl start geting data soon on if prepoulating the root fs helps | 19:34 |
| sean-k-mooney | oh no | 19:35 |
| sean-k-mooney | it failed | 19:35 |
| frickler | sean-k-mooney: no, the failure above occured in gate, so you'd need to recheck | 19:35 |
| sean-k-mooney | ah i tought htis was form a diffent patch | 19:35 |
| sean-k-mooney | ok os ya that either the fact that that sometime just fails anyway | 19:36 |
| sean-k-mooney | or that reason athat sometimes fails is also interupted first boot | 19:36 |
| sean-k-mooney | its kind of undortuet we dont caure the console before and after the reboot | 19:37 |
| sean-k-mooney | /dev/root resized successfully [took 0.34s] | 19:40 |
| sean-k-mooney | Printing cirros user authorized keys | 19:40 |
| sean-k-mooney | === system information === | 19:40 |
| sean-k-mooney | so it does not appare to have pritned any authorised keys | 19:40 |
| frickler | well ... we might be able to save the console data before we issue the resize/reboot and only dump it in case of a failure? | 19:40 |
| sean-k-mooney | and since we are not passign a fallback password | 19:40 |
| sean-k-mooney | tempest.lib.exceptions.SSHTimeout: Connection to the 172.24.5.241 via SSH timed out. | 19:41 |
| sean-k-mooney | User: cirros, Password: None | 19:41 |
| sean-k-mooney | its failing | 19:41 |
| frickler | anyway, I'm out for today, will check back tomorrow | 19:41 |
| sean-k-mooney | frickler: i dont see why we could | 19:41 |
| sean-k-mooney | not | 19:41 |
| sean-k-mooney | frickler: no worres ill recheck this for now and ill try and take a look at the cirros-init path so wee where we are markign the first boot as done vs wehre we have actuly sotre the autorisze dkeys ectra | 19:42 |
| sean-k-mooney | we could mask the issue by providing the defautl cirros users pass word in the tempest config but i woudl prefer not too if we can avoid that | 19:43 |
| opendevreview | Dan Smith proposed openstack/nova master: Avoid reading 4GiB of reason text from RFB https://review.opendev.org/c/openstack/nova/+/991220 | 20:43 |
| opendevreview | Dan Smith proposed openstack/nova master: Avoid reading 4GiB of reason text from RFB https://review.opendev.org/c/openstack/nova/+/991220 | 20:50 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!