| opendevreview | Merged openstack/nova-specs master: Repropose flavor-search-by-name spec https://review.opendev.org/c/openstack/nova-specs/+/958866 | 00:30 |
|---|---|---|
| *** mhen_ is now known as mhen | 01:46 | |
| opendevreview | Taketani Ryo proposed openstack/nova-specs master: Adjust the SEV/SEV-ES code for multiple cc support https://review.opendev.org/c/openstack/nova-specs/+/962578 | 05:51 |
| opendevreview | Taketani Ryo proposed openstack/nova-specs master: Adjust the SEV/SEV-ES code for multiple cc support https://review.opendev.org/c/openstack/nova-specs/+/962578 | 05:55 |
| alibasafa | can anyone answer the my question ? . I have two aggregates named general and dedicated. The availability zone of general is nova, and the zone of dedicated is another one I wanted to live migrate an instance from the nova zone to the other zone To do this, I removed the host from the dedicated aggregate and added it to the general aggregate. After the live migration completed, I moved the compute host back to the dedicated aggregate After | 06:12 |
| alibasafa | that, I tried to resize the instance to its own flavor, but I got an error related to FilterScheduler and zone specifications. I found that in the nova_api database, the availability_zone was updated correctly, but in the nova database, the availability_zone was not updated. | 06:12 |
| alibasafa | Why did this happen | 06:12 |
| gibi | It is not allowed to move instances between Availability Zones. If adding a host to an aggregate or removing a host from an aggregate would cause an instance to move between Availability Zones, including moving from or moving to the default AZ, then the operation will be rejected. The administrator should drain the instances from the host first then the host can be moved. | 06:30 |
| gibi | https://docs.openstack.org/nova/latest/admin/availability-zones.html | 06:30 |
| gibi | you might have an older openstack when this is not clearly rejected yet but causing issues down the line | 06:31 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | 07:51 |
| opendevreview | Konrad Gube proposed openstack/nova master: Use Cinder's os-extend_volume_completion volume action. https://review.opendev.org/c/openstack/nova/+/873560 | 08:37 |
| zigo | dansmith: Hi there! I'm running a Dalmatian deployment, and when uploading an image, it fails in the image inspector with the MBR: | 09:07 |
| zigo | Traceback (most recent call last): | 09:07 |
| zigo | File "/usr/lib/python3/dist-packages/nova/virt/images.py", line 162, in do_image_deep_inspection | 09:07 |
| zigo | inspector.safety_check() | 09:07 |
| zigo | File "/usr/lib/python3/dist-packages/oslo_utils/imageutils/format_inspector.py", line 430, in safety_check | 09:07 |
| zigo | raise SafetyCheckFailed(failures) | 09:07 |
| zigo | oslo_utils.imageutils.format_inspector.SafetyCheckFailed: Safety checks failed: mbr | 09:07 |
| zigo | Is this known, and should I backport some patches? | 09:07 |
| zigo | https://review.opendev.org/c/openstack/oslo.utils/+/928448 maybe ? | 09:08 |
| * zigo tries this patch | 09:19 | |
| zigo | This doesn't fix my issue ... :/ | 09:50 |
| sean-k-mooney | zigo: i think there is a glance config option related to funky mbrs | 09:51 |
| zigo | sean-k-mooney: The issue for me is in nova-compute. | 09:51 |
| zigo | When it spawns the VM. | 09:52 |
| sean-k-mooney | oh ok | 09:52 |
| sean-k-mooney | is that the full traceback | 09:52 |
| zigo | Nope. | 09:52 |
| zigo | https://paste.opendev.org/show/bwgoYQ6LeWlUFYcIZ998/ | 09:53 |
| zigo | That's what I have in "server show" | 09:53 |
| sean-k-mooney | ah, and form that i tatke it the image is in raw? format based on the call to fetch_to_raw | 09:55 |
| zigo | Correct. The image is an upload from another cloud's /var/lib/nova/instance/<uuid>/disk | 09:56 |
| zigo | Will it go nicer if I convert to Qcow2 first? | 09:56 |
| sean-k-mooney | it would work in either case | 09:57 |
| sean-k-mooney | i suspect its failing somehwere here https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils/format_inspector.py#L1388-L1421 | 09:58 |
| sean-k-mooney | however i woudl expect a more verbose excetion message in that case | 09:58 |
| zigo | The image is, as much as I can tell, made from the Debian one, with grub-cloud, dual boot (ie: UEFI + mbr). | 09:59 |
| zigo | At least it looks like it, I didn't check much, but it has all of the usual partitionning. | 09:59 |
| sean-k-mooney | is it publicly aviable or would you beable to provide say the first 1mb of the image which has the format info so we coudl inspect it | 10:03 |
| zigo | I'll investigate myself first. It's not a public image, it's customer's data. | 10:04 |
| sean-k-mooney | we dont actully need the full image all the gpu info is in the first few KBs | 10:04 |
| sean-k-mooney | ack | 10:04 |
| sean-k-mooney | on master we provided a cli that you can use to run the inspector https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils/cli.py | 10:05 |
| sean-k-mooney | so you can do python -m oslo_utils.imageutils -i /path/to/image [-v|--verbose] | 10:05 |
| zigo | Ah, nice ! :) I can try it in my Flamingo virtualized test cluster then. | 10:05 |
| sean-k-mooney | its in epoxy too | 10:06 |
| sean-k-mooney | just not 2024.2 | 10:06 |
| sean-k-mooney | https://github.com/openstack/oslo.utils/blob/stable/2025.1/oslo_utils/imageutils/cli.py | 10:06 |
| zigo | It's a shame there's no script entry point then. | 10:07 |
| sean-k-mooney | the master just has type hints | 10:07 |
| sean-k-mooney | well there is a __main__.py for the module | 10:08 |
| zigo | Yeah, but something in /usr/bin would have been much nicer. | 10:08 |
| sean-k-mooney | but you can basiclly just copy paste it somehwere and run it with the older version of oslo | 10:08 |
| zigo | I'll see if I can push such patch ! :) | 10:08 |
| zigo | Oh, I see. Ok, thanks. | 10:09 |
| sean-k-mooney | i think we did that to not pullute /usr/bin with a debug helper | 10:09 |
| sean-k-mooney | but i dont really have a strong opionion either way | 10:09 |
| sean-k-mooney | if we wanted it to be a cli in /bin all that really needed is addting the console_scripts entry point ot setup.cfg to point at the cli module | 10:10 |
| sean-k-mooney | although on master yo uwoudl now do that via pyproject.toml i think | 10:11 |
| zigo | python3 -m oslo_utils.imageutils -i /path/to/disk <--- Shows nothing, just exists. This means validated in Flamingo ? | 10:12 |
| sean-k-mooney | zigo: anyway my hunch is the mbr is subtlly invalid and not fully complient with the gpt spec | 10:13 |
| sean-k-mooney | am thatm mean our error handleing sucks :) | 10:13 |
| sean-k-mooney | try adding -v | 10:13 |
| zigo | # python3 -m oslo_utils.imageutils -v -i /path/to/disk | 10:13 |
| zigo | SAFETY_CHECK_PASSED=False | 10:13 |
| zigo | VIRTUAL_SIZE=262144 | 10:13 |
| zigo | ACTUAL_SIZE=262144 | 10:13 |
| zigo | IMAGE_FORMAT="gpt" | 10:13 |
| zigo | OSLO_UTILS_VERSION="9.1.0" | 10:13 |
| zigo | FAILURE_REASONS='mbr: GPT MBR has invalid start CHS' | 10:14 |
| sean-k-mooney | there you go | 10:14 |
| zigo | Great, that was very helpful sean! | 10:14 |
| sean-k-mooney | lookign at the code it exits 0 on success and 1 on failure by default | 10:14 |
| sean-k-mooney | so that is form here https://github.com/openstack/oslo.utils/blob/stable/2025.2/oslo_utils/imageutils/format_inspector.py#L1267-L1270 | 10:15 |
| sean-k-mooney | so that path means that the os type is 0xEE a protective MBR https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#os-types | 10:19 |
| sean-k-mooney | the check that is failign is "(starth, starts, startt) != (0x00, 0x02, 0x00):" thats StartHead StartSector and StartTrack | 10:25 |
| sean-k-mooney | https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#protective-mbr-partition-record-protecting-the-entire-disk | 10:25 |
| sean-k-mooney | StartingCHS is 3 bytes Set to 0x000200, corresponding to the Starting LBA field. | 10:26 |
| zigo | I probably will propose a patch to accept more valid values. | 10:42 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove legacy placement tables from API DB https://review.opendev.org/c/openstack/nova/+/889086 | 10:43 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove legacy tables from main DB https://review.opendev.org/c/openstack/nova/+/889087 | 10:43 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove legacy columns from main DB https://review.opendev.org/c/openstack/nova/+/889088 | 10:43 |
| opendevreview | Stephen Finucane proposed openstack/nova master: Remove unused 'shutdown_terminate' flag https://review.opendev.org/c/openstack/nova/+/773156 | 10:49 |
| zigo | stephenfin: Gosh, thanks for the placement db cleanup. It's not like if it was a 5 years old technical dept ... :/ | 10:49 |
| *** iurygregory_ is now known as iurygregory | 10:57 | |
| opendevreview | Stephen Finucane proposed openstack/nova master: objects: Remove custom comparison methods https://review.opendev.org/c/openstack/nova/+/472285 | 10:58 |
| sean-k-mooney | zigo: that is unlikely to be accpeted as there arent more _valid_ vlaues | 11:05 |
| zigo | There sure is! | 11:05 |
| zigo | https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html | 11:05 |
| sean-k-mooney | if the os type is declared as 0xEE then it must be that value or its not a valid GPT partion table acrodign to the spec | 11:05 |
| zigo | Search for CHS. | 11:05 |
| sean-k-mooney | no the other value are only valid if the mbr is a legacy mbr not a protective mbr | 11:06 |
| zigo | In the case of "Protective MBR Partition Record" their are some other valid values. | 11:06 |
| zigo | Right, but then why would OpenStack prevent using "Protective MBR" ? | 11:07 |
| sean-k-mooney | we dont protective mbr requeis it to be exactly 0x000200 | 11:07 |
| zigo | My image is perfectly valid and has 0x00, 0xee, 0xfe | 11:08 |
| zigo | (starth, starts, startt) | 11:08 |
| sean-k-mooney | legacy mbr allows other value but you cant mix thet 2 | 11:08 |
| zigo | I believe the Debian image is mixing them to allow both MBR *AND* uefi. | 11:09 |
| sean-k-mooney | this is somtihng that is often done incorectly when trying to create a image that is both bootable as uefi and bios | 11:09 |
| zigo | This is maybe incorrect, but existing in the wild at least. | 11:09 |
| sean-k-mooney | yep we have encounedre that in other cases too. | 11:10 |
| sean-k-mooney | those are not techinially allowed by the gpt spec | 11:10 |
| sean-k-mooney | there is a way to do it | 11:10 |
| sean-k-mooney | but its often not done correctly | 11:10 |
| sean-k-mooney | we had this dicussion with at leat one other disto | 11:11 |
| zigo | Ok, but then, what our format_inspector is trying ot acheive is *not* checking if some images aren't respecting the GPT specs. We're trying to check if nova/cinder/glance is being hacked... | 11:11 |
| zigo | I wouldn't mind fixing the Debian if it's broken, that's probably the case, but that wont fix the "I cannot migrate my VMs from my old cloud" use case... | 11:12 |
| zigo | (and that's what I'm currently trying to fix) | 11:12 |
| sean-k-mooney | we are tryign to require "valid" boot image and currently that includes a valid GPT partion table | 11:12 |
| sean-k-mooney | let me see if i can find the perio bug | 11:12 |
| sean-k-mooney | i have forgotten much of the context ath this point | 11:13 |
| zigo | Right. We aren't trying to prevent border-line wrongly not respecting the spec images. | 11:13 |
| zigo | Just something that Qemu can boot safely. | 11:13 |
| zigo | It'd be ok to be more respecting the spec if we were writing a spec validation tool for qemu, but that's not our goal ... | 11:14 |
| sean-k-mooney | https://bugs.launchpad.net/oslo.utils/+bug/2091114 | 11:14 |
| zigo | Thanks, reading. | 11:15 |
| Uggla | Does someone knows about https://review.opendev.org/c/openstack/nova-specs/+/964444 ? I mean do we have the owners of this patch around ? | 14:36 |
| sean-k-mooney | i just saw the propal come in yesterday | 14:43 |
| sean-k-mooney | so i dont know anything more beyond what is in the review | 14:43 |
| zigo | dansmith: I do not agree with your comment in the review. From your review: | 15:30 |
| zigo | "Removing checks from a security layer because Debian's images have been wrong for a long time is not a good plan, IMHO." | 15:30 |
| zigo | It is wrong that these magic byte checks are doing any security improvements. Plus I'm not sure where this image is comming from, I'm not sure it's from Debian, but it's been there in our older cloud that I'm trying to migrate. | 15:30 |
| dansmith | zigo: understood :) | 15:31 |
| zigo | IMO, we should re-open the topic during the vPTG. | 15:31 |
| zigo | I do agree some kind of checking is good, and thanks for writing these. | 15:31 |
| zigo | Though blocking existing (customer) VMs that's been there for so long isn't helpful. | 15:32 |
| zigo | If I'm being ignored, I'll cary this as Debian specific patches ... :P | 15:32 |
| dansmith | these things are 100% security related btw.. the configuration of the MBR is read and interpreted by the hypervisor, the emulated BIOS/firmware, etc.. so it's entirely possible there's an attack vector there | 15:33 |
| zigo | At what stage the hypervisor reads it?!? | 15:33 |
| zigo | Shouldn't this be done by the virtual BIOS when booting? | 15:34 |
| zigo | (ie: not even in Qemu) | 15:34 |
| dansmith | the hypervisor is running the virtual bios :) | 15:34 |
| zigo | Well in qemu, no? | 15:34 |
| dansmith | yes.. you know qemu is the thing that was hackable with the previous image attack right? | 15:35 |
| zigo | If there was a way to escape when Qemu boots an HDD, that would be a serious security issue in Qemu. | 15:35 |
| dansmith | escape is not the only attack mode, of course | 15:35 |
| zigo | Well, the thing that was hackable, was the image conversion. | 15:35 |
| dansmith | no it wasnt.. you might want to re-read the bug :) | 15:36 |
| zigo | That's very different from executing sea-BIOS instructions. | 15:36 |
| dansmith | it was literally qemu itself | 15:36 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | 15:37 |
| zigo | When reading the bootloader stuff?!? | 15:37 |
| dansmith | when reading the image and executing the guest.. not the bootloader specifically, but again, we're talking about being defensive for the *next* thing here | 15:38 |
| dansmith | anyway, re-open the discussion of supporting spec-violating images in the vPTG if you want, and carry debian patches to remove these matches-the-spec image security checks if you think that's the best plan | 15:39 |
| zigo | IMO, here, we're talking about being deffensive against the sea-BIOS being executed and doing weird thing, which is *not* our job, and which is hopefully harmless. | 15:40 |
| zigo | But fine ... :P | 15:40 |
| dansmith | well, I thought that inspecting the details of even things like qcow2 data structures was completely not our job until qemu told us it was | 15:41 |
| zigo | BTW, my first iteration got things wrong, what I added wasn't the boot signature I've found. | 15:41 |
| zigo | I've found (0x00, 0x01, 0x00) | 15:41 |
| zigo | dansmith: Right. It probably deserves a discussion. :) | 15:42 |
| zigo | Thanks for this begining of a chat already. | 15:42 |
| dansmith | personally, I think we've already had the discussion, and we've already had a prior art bug that shows another distro realized they were wrong and posted the easy "make the image match the spec" commands, which we could put in the docs | 15:43 |
| zigo | I probably can also patch the MBR of these old VMs... | 15:44 |
| zigo | I'll see if that path works and let you know. | 15:44 |
| dansmith | maybe glance could have a feature in image conversion to fix these things specifically with parted.. that's something I'd be on board with and probably would address your migration issue | 15:46 |
| sean-k-mooney | zigo: (0x00, 0x01, 0x00) is because of a bug in parted | 15:50 |
| sean-k-mooney | acording to the the flatcar image fix pr anyway | 15:51 |
| zigo | Ah... thanks a lot! :) | 15:51 |
| zigo | I'll look it up. | 15:51 |
| zigo | FYI, the image I'm talking about is probably running Debian Jessie ... :) | 15:51 |
| sean-k-mooney | https://github.com/flatcar/scripts/pull/2668#issuecomment-2660859901 | 15:52 |
| sean-k-mooney | i have no idea how the flatcar image buildign work by the way | 15:53 |
| sean-k-mooney | they provided a workaroudn for old images https://bugs.launchpad.net/oslo.utils/+bug/2091114/comments/12 | 15:54 |
| sean-k-mooney | by the way on the glance side ther eis https://github.com/openstack/glance/blob/master/glance/common/config.py#L123C1-L138 to allow folks to weaken the security | 15:56 |
| sean-k-mooney | but that is not really a good longterm approch and im not conviced that shoudl be in oslo | 15:56 |
| zigo | Surprisingly, Glance isn't complaining, only Nova, not sure why. | 15:57 |
| dansmith | can't (and shouldn't) be in oslo | 15:57 |
| kevko | hi, i am running nova upgrade check FROM caracal container when all others containers (kolla-ansible deployment) are still antelope ... this should pass , shouldn't ? as it is slurp ...but check failed ... is it ok ? https://paste.openstack.org/show/bXbzW9LwWQwQIEu8vEUV/ | 15:58 |
| dansmith | glance hasn't had upload stream safety checks for very long, so depends on what you're upgrading to | 15:58 |
| sean-k-mooney | dansmith: i was honesstly kind of surpised ot see that in glance if im honest i assuem this is planned to be deprecated and remvoed eventually | 15:58 |
| kevko | point is that i will upgrade control plane to caracal ...and my 300 compute servers will be upgraded one by one during the weeks | 15:58 |
| dansmith | sean-k-mooney: yes, I don't like it either | 15:58 |
| sean-k-mooney | kevko: ya you shoudl be able to upgrade the contolers and do the compute later | 15:59 |
| sean-k-mooney | kevko: that looks like you have not rant either the db migration or somethign similar to that | 16:00 |
| sean-k-mooney | it could also be using the wrogn db | 16:00 |
| sean-k-mooney | the compute_id was added by https://github.com/openstack/nova/commit/a47fdef1bfba8b1e2c3279f2c51eda3d8a985ec1 | 16:01 |
| sean-k-mooney | in bobcat | 16:02 |
| kevko | yeah, but nova-status upgrade check command sounds like "this script will tell you if you can upgrade , which of course including db migration " | 16:02 |
| sean-k-mooney | well your using a nova-status form carical but have not applied a db migration form bobcat | 16:03 |
| kevko | sean-k-mooney: as I said ... antelope is slurp ...so upgrade to caracal and caracal's nova-status upgrade check should work no ? | 16:03 |
| sean-k-mooney | i would need to check our doc for if the status check shoudl be done after db sync or not | 16:04 |
| kevko | sean-k-mooney: yeah, but why I should update db migration from bobcat when antelope -> caracal is supported slurp upgrade ? | 16:04 |
| sean-k-mooney | its not that you do them form bobcat it that you woudl run all the caracal ones | 16:04 |
| kevko | that's the point ... | 16:04 |
| kevko | sean-k-mooney: In my understanding of the world, a DB migration is part of the upgrade... which means the upgrade check should take into account that I’m running it before the upgrade, right? At least that makes more sense to me… | 16:06 |
| sean-k-mooney | kevko: in gerade we run nova stats after you have done the db upgrade | 16:06 |
| sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/projects/60_nova/upgrade.sh#L104 | 16:06 |
| sean-k-mooney | keveko you shoudl run the antelop version before the dbsync and the caracal one after | 16:07 |
| opendevreview | Arnaud Morin proposed openstack/nova master: Fix: add a default dev_type in spec https://review.opendev.org/c/openstack/nova/+/964552 | 16:07 |
| sean-k-mooney | kevko: https://docs.openstack.org/nova/latest/admin/upgrades.html#rolling-upgrade-process | 16:08 |
| sean-k-mooney | that what we docuemtn as well by the way | 16:08 |
| sean-k-mooney | the db sysnc syste the schema which will work with the old code | 16:09 |
| sean-k-mooney | the upgade check expct the new db schema | 16:09 |
| kevko | sean-k-mooney: hmm, thank you.... so upgrade: antelope container -> run upgrade check from antelope container -> exec db migrations to caracal -> replace antelope container to caracal -> run upgrade check again from caracal container | 16:12 |
| sean-k-mooney | basiclly. the first execution fo the status check tells you if you have any online migration or other thing you need to do before upgrading | 16:18 |
| sean-k-mooney | the second tell you the same after you have upgraded | 16:18 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Rally job for eventlet-removal https://review.opendev.org/c/openstack/nova/+/960130 | 16:24 |
| opendevreview | Merged openstack/nova master: pci: Add more detail and examples to pci.alias docs https://review.opendev.org/c/openstack/nova/+/950659 | 18:54 |
| atmark | Hello, I have instances that automatically got shutdown on few computes after restarting nova | 21:47 |
| atmark | `During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.` | 21:47 |
| atmark | Full logs https://paste.openstack.org/show/bZzH7XnMnZut2zZzTvuk/ | 21:47 |
| atmark | OpenStack Version is Yoga | 21:51 |
| atmark | Which table in the db do I need to look at to identify more VMs that has power state mismatch? | 21:52 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!