| opendevreview | Rajat Dhasmana proposed openstack/glance master: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923431 | 05:53 |
|---|---|---|
| frickler | https://review.opendev.org/923433 is failing once again on grenade this time. I'm going to force-merge it once zuul has reported the failures in gerrit | 06:50 |
| frickler | hmm, doing the force-merge now before the cve patches get into gate again. the failures can be seen at https://zuul.opendev.org/t/openstack/buildset/94df5d39c05943db9903df87c0bb9ea4 in case someone wants to take a closer look | 07:13 |
| opendevreview | Merged openstack/glance master: Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923433 | 07:16 |
| frickler | doing the same for 923431 now which as the above pretty obviously has no negative impact that could be discovered by the other tests | 07:17 |
| opendevreview | Merged openstack/glance master: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923431 | 07:17 |
| frickler | ok, enqueued and promoted the stack at 923431 once again. cc fungi | 07:22 |
| frickler | btw. keeping non-voting jobs in the gate pipeline is usually being frowned upon, but I think it is o.k. to gather some additional results in the current situation | 07:30 |
| opendevreview | Abhishek Kekane proposed openstack/glance stable/2024.1: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923468 | 07:48 |
| opendevreview | Abhishek Kekane proposed openstack/glance stable/2023.2: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923470 | 07:52 |
| opendevreview | Abhishek Kekane proposed openstack/glance stable/2023.2: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923470 | 07:55 |
| frickler | abhishekk: gate failure on the last cve patch, maybe you want to have a look https://zuul.opendev.org/t/openstack/build/01bc21c5f614487aa6e941d4c23da635 | 07:56 |
| abhishekk | looking | 07:57 |
| frickler | hmm, weird, that is the change that I merged earlier, not sure why that was running in gate again | 07:58 |
| abhishekk | not related | 07:58 |
| abhishekk | oops | 07:58 |
| frickler | yes, certainly not related. but fixing unstable func tests should be the next priority after the cve | 07:59 |
| abhishekk | can it be removed from the queue | 07:59 |
| abhishekk | ++ | 07:59 |
| frickler | since the gate queue has already been reset, it shouldn't matter much anymore | 07:59 |
| abhishekk | ack | 08:00 |
| opendevreview | Abhishek Kekane proposed openstack/glance stable/2023.1: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923471 | 08:05 |
| gaudenz_ | The change to the image conversion plugin in https://review.opendev.org/c/openstack/glance/+/923251 changes the import API behavior if you previously dependend on the fact that the source format was autodetected. Are there plans to fix this by reintroducing autodetection without calling qemu-img (which would be unsafe)? | 08:48 |
| gaudenz_ | We have code which relies on the old behavior, so this caused some breakage for us. | 08:49 |
| gaudenz_ | Or do you consider the old behavior just a bug and we should have never relied on it? | 08:49 |
| *** gaudenz_ is now known as gaudenz | 08:54 | |
| frickler | integrated-storage run-test: commands[2] | tempest run --combine --serial --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.scenario)|(^tempest\.serial_tests)' --exclude-list ./tools/tempest-integrated-gate-storage-exclude-list.txt --concurrency=3 | 09:12 |
| frickler | doesn't seem useful to combine "--serial" and "--concurrency" | 09:12 |
| opendevreview | Merged openstack/glance master: Reject qcow files with data-file attributes https://review.opendev.org/c/openstack/glance/+/923248 | 09:21 |
| frickler | #1 is done, yay. and the other ones all only waiting on n-v jobs | 09:38 |
| opendevreview | Merged openstack/glance master: Extend format_inspector for QCOW safety https://review.opendev.org/c/openstack/glance/+/923249 | 09:39 |
| opendevreview | Merged openstack/glance master: Add VMDK safety check https://review.opendev.org/c/openstack/glance/+/923250 | 09:46 |
| opendevreview | Merged openstack/glance master: Reject unsafe qcow and vmdk files https://review.opendev.org/c/openstack/glance/+/923251 | 09:46 |
| opendevreview | Merged openstack/glance master: Add QED format detection to format_inspector https://review.opendev.org/c/openstack/glance/+/923252 | 09:46 |
| opendevreview | Merged openstack/glance master: Add file format detection to format_inspector https://review.opendev.org/c/openstack/glance/+/923253 | 09:46 |
| frickler | abhishekk: there was a merge failure and a gate reset for https://review.opendev.org/c/openstack/glance/+/923254 , it also seems weird to merge a change where the commit message starts with "WIP", please have another look | 09:50 |
| frickler | also let me know when we're ready to promote the stable branch patches | 09:50 |
| abhishekk | Ohh, It has a new patch after approval | 09:51 |
| frickler | oh, I see you just approved some of the stable things, will boost them | 09:51 |
| abhishekk | I think we can it's good it is not merged | 09:52 |
| abhishekk | no I was talking about https://review.opendev.org/c/openstack/glance/+/923254 | 09:52 |
| abhishekk | frickler: stable branch will still have timeouts so we need to backport that timeout increment patch first | 09:52 |
| abhishekk | frickler: I will let you know about stable branch promotion | 09:56 |
| abhishekk | thank you so far for your quick support!! | 09:57 |
| opendevreview | Rajat Dhasmana proposed openstack/glance stable/2024.1: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923483 | 10:09 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add releasenote for CVE-2024-32498 fix https://review.opendev.org/c/openstack/glance/+/923485 | 10:11 |
| opendevreview | Rajat Dhasmana proposed openstack/glance stable/2023.2: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923484 | 10:11 |
| opendevreview | Rajat Dhasmana proposed openstack/glance stable/2023.1: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923486 | 10:12 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add releasenote for CVE-2024-32498 fix https://review.opendev.org/c/openstack/glance/+/923485 | 10:15 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add releasenote for CVE-2024-32498 fix https://review.opendev.org/c/openstack/glance/+/923485 | 12:19 |
| fungi | gaudenz: it seems like the general sentiment from the qemu maintainers is that they don't support using any of it on untrusted images, but as there aren't a lot of great alternatives to qemu we have to work with what we've got, and that means trying to filter out potential risks before they ever make it to any qemu tools and being as strict as possible about treating images as the | 12:39 |
| fungi | exact types our services are able to safely support | 12:39 |
| fungi | i don't know of any way to safely recreate qemu's autodetection routines other than to rewrite them from scratch (without side effects) inside openstack | 12:40 |
| gaudenz | fungi: I was more thinking about using the detect_format method from the format_inspector module in Glance. This should be pretty safe as it just tries all inspectors. If one of them is vulnerable with some input this can be triggered by setting the disk_format to this type anyway. | 13:31 |
| gaudenz | Or using python-magic to detect the type would be another option. Should also be pretty safe as it just reads file headers. | 13:31 |
| fungi | yeah, magic number inspection shouldn't have side effects | 13:33 |
| fungi | the main risk is in using qemu-img info because the process loads qemu's drivers to process the file | 13:34 |
| fungi | so anything that a nefarious custom image can be made to do on load is potentially also possible when passing it to qemu-img subcommands like info and convert | 13:35 |
| gaudenz | Yes I totally agree that this is not an option. Was quite surprised what sort of shenanigans one can do with QCOW2 images. | 13:36 |
| *** pdeore_ is now known as pdeore | 13:40 | |
| opendevreview | Merged openstack/glance stable/2024.1: Reject qcow files with data-file attributes https://review.opendev.org/c/openstack/glance/+/923259 | 13:41 |
| Luzi | fungi ? we (and nova) also want to use qemu-img convert in case of encrypted qcow2 images... | 13:42 |
| abhishekk | frickler, these 3 patches needs to be prioritised for stable branches to mitigate timeout, https://review.opendev.org/c/openstack/glance/+/923483, https://review.opendev.org/c/openstack/glance/+/923486, https://review.opendev.org/c/openstack/glance/+/923484?usp=cherry-pick | 13:44 |
| fungi | Luzi: if the dangerous aspects of the image file are outside the encrypted blob we can safely inspect those, i think? i haven't looked closely at the encryption implementation in the qcow2 format specification | 13:45 |
| Luzi | we mainly did encrypted raw images, but nova will need encrypted qcow2 | 13:45 |
| fungi | otherwise we'd have to decrypt before inspecting, if qcow2 encryption covers more than just the block data | 13:46 |
| abhishekk | before encryption can't you inspect the file for malicious contents? | 13:50 |
| Luzi | encryption happens on client side | 13:50 |
| Luzi | glance will only get the already encrypted image | 13:51 |
| pdeore | abhishekk, rosmaita, dansmith, croelandt, mrjoshi glance weekly meeting in 10 minutes at #openstack-meeting | 13:51 |
| abhishekk | yep at client side, won't it be a good idea to introspect? | 13:51 |
| Luzi | what about a malicious client? | 13:52 |
| abhishekk | ack, I was thinking about openstackclient where you are planning to add this mechanism | 13:53 |
| fungi | client-side security measures can only protect users, not the servers | 13:54 |
| abhishekk | otherwise you also can add one config option at glance like introspect_encrypted_image and if that is true glance also perform it before storing (but it will have performance impact) anyway nova or cinder has to do that | 13:55 |
| Luzi | what fungi said | 13:55 |
| fungi | but anyway, like i said, if it's only the block data that's encrypted and not the other image metadata then glance doesn't need the decryption key to check whether options like remote backing files and alternate data locations are set | 13:55 |
| fungi | if the image file is just an encrypted blob though, we'd need to use some other tool without side effects at the decryption stage (wherever that happens) and then inspect the decrypted image prior to passing to qemu tools | 13:56 |
| abhishekk | hmmm, | 13:57 |
| abhishekk | Luzi may be good to bring it in glance meeting as well | 13:57 |
| fungi | i think it will require a read of the encryption format docs for qcow2 to determine what our options are, if nobody's done that yet | 13:58 |
| Luzi | yeah abhishekk | 13:58 |
| abhishekk | ++ | 13:58 |
| Luzi | well there seems to be unencrypted metadata in encrypted qcow2 images | 13:59 |
| Luzi | how do you inspect them in detail? | 13:59 |
| abhishekk | pdeore: no agenda on the meeting etherpad, do we have anything to discuss | 14:00 |
| fungi | Luzi: that's a good sign. check the image inspector code that went into (or is in the process of merging to) glance as part of the ossa-2024-001 fixes | 14:00 |
| pdeore | abhishekk, agenda is there :) | 14:00 |
| abhishekk | Luzi: I think that should be done in nova and glacinder only | 14:00 |
| abhishekk | loding... I still see last week only | 14:01 |
| abhishekk | because we are inspecting it during conversion at the moment | 14:01 |
| abhishekk | and encrypted image is not allowed for conversion as per spec | 14:01 |
| fungi | the image inspector code in glance was a | 14:03 |
| fungi | also copied into cinder and nova, the glance copy was treated as the reference i think | 14:04 |
| fungi | longer-term goal is to split it out into an oslo lib and have cinder/glance/nova all use that | 14:04 |
| fungi | but we couldn't really create a new separate project within the scope of backportable security fixes | 14:05 |
| abhishekk | yes, that's the plan to move it to oslo (utils may be) | 14:06 |
| pdeore | rosmaita, croelandt could you please have a look at this, https://review.opendev.org/c/openstack/glance/+/923485 | 14:09 |
| Luzi | my collegue tested it with an encrypted qcow2 image and the inspect code is able to inspect the metadata | 14:11 |
| Luzi | so that means, before the image is converted in nova or cinder, the inspection will take place and alert, when something is wrong, right? | 14:17 |
| abhishekk | yes | 14:17 |
| Luzi | that is good | 14:19 |
| abhishekk | you can also test it by copying format-inspector from glance and write a little python program | 14:20 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Revert "Make import jobs non-voting" https://review.opendev.org/c/openstack/glance/+/923523 | 14:24 |
| frickler | abhishekk: done now, let's hope stable branches work better after these merge | 14:41 |
| abhishekk | fingres crossed | 14:41 |
| frickler | abhishekk: https://review.opendev.org/c/openstack/glance/+/923263/1 hasn't been approved yet, is that intentional or an oversight? I'm reenqueuing the patches below it now | 15:59 |
| abhishekk | I haven't because gate was failing | 16:05 |
| abhishekk | will do it now | 16:05 |
| abhishekk | approved, last patch workflow -1 as it is WIP in master | 16:06 |
| abhishekk | I was waiting for timeout patch to get in | 16:07 |
| abhishekk | I approve stable/2023.2 once 2024.1 merges | 16:07 |
| opendevreview | Merged openstack/glance stable/2024.1: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923468 | 17:02 |
| opendevreview | Merged openstack/glance stable/2023.2: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923470 | 17:02 |
| abhishek_ | Just FYI, glance CVE patches status | 17:19 |
| abhishek_ | Master all merged except last one which is WIP | 17:19 |
| abhishek_ | stable/2024.1 is in check and gate now | 17:19 |
| abhishek_ | once those merged I will approve 2023.2 and then 2023.1 | 17:19 |
| abhishek_ | Not sure what will be needed to get merged unmaintained zed patches | 17:20 |
| opendevreview | Merged openstack/glance stable/2023.1: [stable only] Make import jobs non-voting https://review.opendev.org/c/openstack/glance/+/923471 | 17:31 |
| opendevreview | Rajat Dhasmana proposed openstack/glance master: Fix: optimized upload volume in Cinder store https://review.opendev.org/c/openstack/glance/+/922316 | 17:39 |
| opendevreview | Merged openstack/glance stable/2024.1: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923483 | 19:16 |
| opendevreview | Merged openstack/glance stable/2023.1: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923486 | 19:16 |
| opendevreview | Merged openstack/glance stable/2023.2: Increase timeout for tempest-integrated-storage-import job https://review.opendev.org/c/openstack/glance/+/923484 | 20:43 |
| opendevreview | Merged openstack/glance stable/2024.1: Extend format_inspector for QCOW safety https://review.opendev.org/c/openstack/glance/+/923260 | 20:43 |
| opendevreview | Merged openstack/glance stable/2024.1: Add VMDK safety check https://review.opendev.org/c/openstack/glance/+/923261 | 20:43 |
| opendevreview | Merged openstack/glance stable/2024.1: Reject unsafe qcow and vmdk files https://review.opendev.org/c/openstack/glance/+/923262 | 20:51 |
| opendevreview | Merged openstack/glance stable/2024.1: Add QED format detection to format_inspector https://review.opendev.org/c/openstack/glance/+/923263 | 21:19 |
| opendevreview | Merged openstack/glance master: Revert "Make import jobs non-voting" https://review.opendev.org/c/openstack/glance/+/923523 | 21:35 |
| fungi | looks like all the ossa-2024-001 cinder patches for master and stable/2024.1 have merged, changes for older maintained branches are still lacking approvals | 22:10 |
| fungi | er, glance patches i mean, not cinder | 22:10 |
| fungi | (cinder's are all merged now, but they had a lot fewer) | 22:11 |
| fungi | oh, actually no, https://review.opendev.org/c/openstack/glance/+/923264 for stable/2024.1 had to be put back into the gate, looks like it failed out but will hopefully merge shortly | 22:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!