Tuesday, 2024-07-02

opendevreviewMerged openstack/nova master: api: Fix bugs in schemas  https://review.opendev.org/c/openstack/nova/+/91584302:24
opendevreviewmelanie witt proposed openstack/nova master: libvirt: Introduce support for rbd with LUKS  https://review.opendev.org/c/openstack/nova/+/88991202:34
*** bauzas_ is now known as bauzas03:36
*** bauzas_ is now known as bauzas05:21
opendevreviewFabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call  https://review.opendev.org/c/openstack/nova/+/91062708:37
*** ykarel__ is now known as ykarel10:37
opendevreviewFabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load  https://review.opendev.org/c/openstack/nova-specs/+/92220111:10
opendevreviewMax proposed openstack/nova master: performance: reduce calls to libvirt / add caching  https://review.opendev.org/c/openstack/nova/+/92285511:49
opendevreviewBence Romsics proposed openstack/os-vif master: Do not add taps in trunk bridges to the dead vlan  https://review.opendev.org/c/openstack/os-vif/+/92303611:51
opendevreviewFabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load  https://review.opendev.org/c/openstack/nova-specs/+/92220112:06
opendevreviewMerged openstack/nova-specs master: Follow-up for openapi spec  https://review.opendev.org/c/openstack/nova-specs/+/91960312:52
opendevreviewMerged openstack/nova master: tests: Ensure API schemas are valid  https://review.opendev.org/c/openstack/nova/+/91584413:44
opendevreviewMichal Arbet proposed openstack/nova-specs master: Add availability zone-aware affinity filter to the scheduler  https://review.opendev.org/c/openstack/nova-specs/+/89077913:56
opendevreviewMichal Arbet proposed openstack/nova-specs master: Add availability zone-aware affinity filter to the scheduler  https://review.opendev.org/c/openstack/nova-specs/+/89077913:58
opendevreviewDan Smith proposed openstack/nova master: Reject qcow files with data-file attributes  https://review.opendev.org/c/openstack/nova/+/92325514:09
opendevreviewDan Smith proposed openstack/nova master: Check images with format_inspector for safety  https://review.opendev.org/c/openstack/nova/+/92325614:09
opendevreviewDan Smith proposed openstack/nova master: Additional qemu safety checking on base images  https://review.opendev.org/c/openstack/nova/+/92325714:09
opendevreviewDan Smith proposed openstack/nova master: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92325814:09
opendevreviewDan Smith proposed openstack/nova stable/2024.1: Reject qcow files with data-file attributes  https://review.opendev.org/c/openstack/nova/+/92327314:10
opendevreviewDan Smith proposed openstack/nova stable/2024.1: Check images with format_inspector for safety  https://review.opendev.org/c/openstack/nova/+/92327414:10
*** bauzas_ is now known as bauzas14:10
opendevreviewDan Smith proposed openstack/nova stable/2024.1: Additional qemu safety checking on base images  https://review.opendev.org/c/openstack/nova/+/92327514:10
opendevreviewDan Smith proposed openstack/nova stable/2024.1: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92327614:10
opendevreviewDan Smith proposed openstack/nova stable/2023.2: Reject qcow files with data-file attributes  https://review.opendev.org/c/openstack/nova/+/92328414:11
opendevreviewDan Smith proposed openstack/nova stable/2023.2: Check images with format_inspector for safety  https://review.opendev.org/c/openstack/nova/+/92328514:11
opendevreviewDan Smith proposed openstack/nova stable/2023.2: Additional qemu safety checking on base images  https://review.opendev.org/c/openstack/nova/+/92328614:11
opendevreviewDan Smith proposed openstack/nova stable/2023.2: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92328714:11
opendevreviewDan Smith proposed openstack/nova stable/2023.1: Reject qcow files with data-file attributes  https://review.opendev.org/c/openstack/nova/+/92328814:11
opendevreviewDan Smith proposed openstack/nova stable/2023.1: Check images with format_inspector for safety  https://review.opendev.org/c/openstack/nova/+/92328914:11
opendevreviewDan Smith proposed openstack/nova stable/2023.1: Additional qemu safety checking on base images  https://review.opendev.org/c/openstack/nova/+/92329014:11
opendevreviewDan Smith proposed openstack/nova stable/2023.1: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92329114:11
mnaserdansmith: sorry I know its a busy morning but I was working on back porting the patches to unmaintained/zed14:35
mnaserhttps://review.opendev.org/c/openstack/nova/+/923291/1/nova/tests/unit/virt/libvirt/test_utils.py14:35
mnaserI think this will fail pep8 because of lack of new line above test_create_image_encryption14:35
dansmithyep, looks like it already did14:37
dansmiththanks14:37
dansmiththat was last minute yesterday14:37
mnaseryeah it's all been last minute all over the place14:37
dansmithit passed pep8 on master when I tried, so maybe the backport trimmed14:37
dansmithsrsly14:38
mnaserI am taking care of zed for all the projects14:38
mnaserhttps://review.opendev.org/c/openstack/nova/+/923297 if you want to eye it over (and it matters to you that is :p)14:38
sean-k-mooneyit passed for me locally too14:38
sean-k-mooneyi think its because in the gate its applied to the tip of the branch14:38
sean-k-mooneywhen we run the jobs14:38
mnaserahh right maybe it ran it over the last commit that was clean14:39
sean-k-mooneyso to see it locally we might need to rebase 14:39
tkajinamI downloaded the chain for master but pep8 fails in my local (even after rebasing the chain on the latest master). zuul hasn't run pep8 year over 923258 but I suspect it may detect the same error.14:45
tkajinams/year/yet/14:45
dansmithoh it does fail for me on master,14:46
dansmithI ran preivously before the tests were added14:46
dansmithI will fix14:46
opendevreviewDan Smith proposed openstack/nova master: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92325814:47
sean-k-mooneyreappoved14:47
opendevreviewDan Smith proposed openstack/nova stable/2024.1: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92327614:50
opendevreviewDan Smith proposed openstack/nova stable/2023.2: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92328714:55
dansmithmnaser: tkajinam sorry about that, thanks for the heads up 2023.1 coming in a sec14:58
opendevreviewDan Smith proposed openstack/nova stable/2023.1: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92329115:00
JayFmelwitt: sean-k-mooney: re: the lock greening errors in python 3.12: https://github.com/eventlet/eventlet/issues/969 is what itamarst tracked it to.15:09
fricklersean-k-mooney: dansmith: do you want us to promote any of the approved patches into gate or do you prefer for all check jobs to finish first?15:12
dansmithfrickler: definitely want to see the tests first15:13
sean-k-mooneyfrickler: based on the quick review we have done we belive the patchs shoudl be correct however we need the wider coverage form check to confirm we are not regressing any edgecases. if we see a timeout ectra or other infa issues we can consier promotiign or requeing15:19
fricklerwell I'm not talking about force-merging (yet), you'll have the same jobs running in the gate pipeline again (except n-v), but anyway, the decision is up to you15:22
opendevreviewFabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load  https://review.opendev.org/c/openstack/nova-specs/+/92220115:27
tkajinamdansmith, no problem, and thank you !15:32
*** bauzas_ is now known as bauzas15:34
dansmithdefinitely want to see all the scenarios run against this since its so much change and it's nearly impossible to do that ahead of time15:54
dansmithif we hit obviously unrelated things I think promote/force is the tool we can use to expedite15:54
dansmiththanks frickler for checking15:54
sean-k-mooneyim checkign now but the nova-live-migration job fails on the second patch15:56
sean-k-mooneyhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7fc/923256/1/check/nova-live-migration/7fc79c6/15:56
opendevreviewFabian Wiesel proposed openstack/nova-specs master: Lazy Metadata Loading in order to Reduce Server Load  https://review.opendev.org/c/openstack/nova-specs/+/92220115:56
sean-k-mooneythe ceph one passed15:56
dansmithhmm15:56
dansmiththat is odd or sure15:57
dansmith*for15:57
dansmithI have to do a medial thing in a few minutes, so I will be out a bit around there15:57
sean-k-mooneylooks like only the blockmigation tests15:57
dansmithright but perhaps something is upset about the disk image?15:57
dansmithJul 02 15:17:20.783035 np0037859393 nova-compute[41928]: ERROR oslo_messaging.rpc.server nova.exception.ImageUnacceptable: Image 02acc766-7c4b-4ffb-a72d-080ea50527d4 is unacceptable: Image not in a supported format15:57
dansmitheff15:57
sean-k-mooneyyep15:58
sean-k-mooneyAttributeError: 'NoneType' object has no attribute 'from_file'15:58
sean-k-mooneybut i tought we caught that15:59
sean-k-mooneyso this is raising https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#15216:01
sean-k-mooneyand we take this else https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#17416:01
sean-k-mooneyoh ok i need to look at the ohte rhost to see it properly16:02
sean-k-mooneyJul 02 15:19:58.679517 np0037859379 nova-compute[85178]: WARNING nova.virt.images [None req-3f27d1d1-b0a1-46a9-ac4b-6b869e49ea15 tempest-LiveMigrationTest-599041496 tempest-LiveMigrationTest-599041496-project-admin] Unable to perform deep image inspection on type 'aki'16:04
sean-k-mooneyso we have special cased for ami image but not aki16:04
sean-k-mooneyso i think its passing on the ami iamge and failing on the kernel image16:05
sean-k-mooneydansmith: and we are only seeing it in that job becasue tis the only one using uec images16:06
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/.zuul.yaml#L120-L12116:06
sean-k-mooneyso "if disk_format == 'ami':" should be "if disk_format in ('ami','aki'):"16:07
sean-k-mooneyhere https://review.opendev.org/c/openstack/nova/+/923256/1/nova/virt/images.py#16616:07
*** bauzas_ is now known as bauzas16:07
dansmithsean-k-mooney: aha, so I think we need to have a serious discussion about AMI in nova because as far as I can tell, the disk image can be anything from a qcow to a raw ext2 or ntfs filesystem and trying to determine safety there is a nightmare16:14
sean-k-mooneydansmith: im carefully going to put a patch on the end to fix 'aki' 16:14
sean-k-mooneyya16:14
dansmithperhaps if we fix our own losing track of the format of the base images in nova we could handle it better, but idk16:14
dansmithsean-k-mooney: ack on the fix thanks ++16:14
sean-k-mooneyi was also wonderign about the other contaienr formats like ova ovf16:14
sean-k-mooneyso we will need to squash the fix into patch 216:15
sean-k-mooneybut i dont want to kick the current set out of the gate16:15
dansmithokay, yeah, let's confirm it fixes it and there's nothing else before we squash but sounds good16:15
sean-k-mooneyhench put it on the end and we can either force merge or pull it back depending on perfernce/urgency16:15
dansmithwe'll have to fix all the backports too16:15
sean-k-mooneyi sould aslo includ ari16:16
sean-k-mooneyhttps://creodias.docs.cloudferro.com/en/latest/cloud/What-Image-Formats-are-available-in-OpenStack-Creodias-Cloud.html16:16
dansmithdo we ever run qemu-img cmmands on the ramdisk? I'd guess not16:20
opendevreviewsean mooney proposed openstack/nova master: add support for aki and ari images  https://review.opendev.org/c/openstack/nova/+/92331716:20
sean-k-mooneydansmith: i dont think we do but if we do its failign before we get to that point16:21
dansmithmaybe we just run it on all three pieces for no reason? but okay, no complaints about the patch as-is16:22
sean-k-mooneyproably realistically if any of the aobve were qcows i would expect it to trigger the vulnerablity16:22
sean-k-mooneyso checking all 3 part is a good thing to do regardless16:23
dansmithah, that's a good point16:23
sean-k-mooneybrb ill keep an eye on the jobs when im back16:25
gibiI'm arond until the top of the hour (already +2 on the Sean's patch)16:26
sean-k-mooneyopenstacksdk-functional-devstack failed on the 4 patch but is passing on the rest 16:46
sean-k-mooneyenstack.exceptions.ConflictException: ConflictException: 40916:46
sean-k-mooneyin the TestAutoAllocatedTopology test16:46
sean-k-mooneyi have seen that fail on othe rpatches and it seams to just be flaky16:46
sean-k-mooneyso for now ill leave the curernt set complete16:46
sean-k-mooneyit will get rerun once i rebase the patches for the ami/aki/ari fix anyway16:47
dansmithack16:47
sean-k-mooneyim assumign that merged reletivly recently because i have only started to see it fail like that in the last 2-3 weeks16:48
sean-k-mooneyi dont recall that job failing previously to that16:48
sean-k-mooneyhum no apprently that was added 8 years ago16:50
sean-k-mooneythat makes me feel old given the autoallocate stuff was added to enable people to move form nova-networks...16:51
sean-k-mooneyofcouse everythign pre covide feels ancient at this point16:51
dansmithsean-k-mooney: going to need a very quick approval on a job change in nova's ceph job to unblock glance in a sec17:40
opendevreviewDan Smith proposed openstack/nova master: Fix disk_formats in ceph job tempest config  https://review.opendev.org/c/openstack/nova/+/92332217:45
dansmithsean-k-mooney: ^17:46
dansmithor any other nova cores17:48
dansmithfrickler: that one will take a long time in the current queue to merge and glance can't merge their stuff without it, so if you have thoughts... :)17:49
gmanndansmith: one question, this is tempest default, that is what you mean in commit msg https://github.com/openstack/tempest/blob/6b8ff89c8fffcc231d9e5b639d57d6e4a8bd47d3/tempest/config.py#L65717:52
dansmithgmann: yes17:52
dansmithabhishek_: ^17:52
dansmithgmann: this is basically the problem: https://github.com/openstack/tempest/blob/master/tempest/api/image/base.py#L10317:52
abhishek_yes17:52
gmanndansmith: ohk, +2 but it will be good if we can also fix test to fetch the required image instead of replying on list order in config option17:56
gmann*relying17:56
dansmithreplied agree of course17:56
gmanndansmith: thanks17:57
abhishek_gmann: agree to fix the test as well17:57
gmann+117:57
fricklerdansmith: the only chance to speed up the CI in the current situation likely would be to temporarily abandon all the stable branch changes that eat up resources but fill anyway, and restore them once all fixes have merged in master18:17
dansmithseems like a bad plan18:17
fricklers/fill/will fail/18:18
fricklerother then that, just wait until zuul has worked through the queue18:18
dansmithroger that18:18
JayFsean-k-mooney: https://github.com/eventlet/eventlet/pull/970/files is the fix for that eventlet "couldn't green locks" issue we saw popup in python 3.12. I'd expect it to merge+be in next eventlet release. TL;DR: eventlet monkey-patching was happening after we greened locks + as a side effect of running the lock greening, we'd create more locks to be greened AIUI18:20
sean-k-mooneyJayF: ack ill try and take a look18:38
JayFsean-k-mooney: consider it just an FYI and notice that I made sure it was taken care of18:39
JayFnot sure you have an AI18:39
sean-k-mooneyJayF: thanks it more i was on calls for the last hour or so and i want to check back in on the status fo the cve18:40
sean-k-mooneyso ill read over that tomorrow18:40
JayFyeah that is more important, only reason I'm letting you know is to get it off your list if it was ever on it18:40
sean-k-mooney:)18:40
fungiwith regard to speeding up fixes, we *can* enqueue the more urgent changes directly into the gate pipeline without waiting for them to clear check. the gate pipeline automatically gets resource priority over the check pipeline19:10
dansmithfungi: up to you really... we have the nova and glance patches that both need to go19:11
fungionce they're in the gate, we can also reorder the queue to put them ahead of the rest19:11
dansmiththey're just tempest/zuul changes and gmann is +2, so... might be good19:11
fungithey'll need workflow +1 votes before i can enqueue them, but they'll still have to pass the same tests in the gate before they can merge anyway so that should be safe19:12
dansmiththe nova one should be good19:12
fungii'm juggling a lot of things, so if someone can explicitly list the more urgent change numbers i need to move to the gate pipeline i'm happy to do that as long as they've gotten approval votes from someone19:13
dansmithhttps://review.opendev.org/c/openstack/glance/+/92332119:13
dansmithand yeah the glance one too19:13
fungithanks19:13
dansmithnova one is https://review.opendev.org/c/openstack/nova/+/923322 in case you didn't et it19:13
fungiyep, i'll enqueue those momentarily19:14
fungithey're both at the top of the gate pipeline now19:19
dansmiththanks fungi 19:20
fungiand already getting nodes assigned19:22
dansmithfungi: we may have spoken too soon19:40
dansmithabhishek_: ^19:40
dansmithlooks like that small change had some knock-on effects in tempest I don't fully understand19:41
dansmithgmann: might could use some help19:41
fungii'm around to re-enqueue a revision if it's worked out19:41
dansmithgmann: does image.disk_formats look right here? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_991/923321/2/check/tempest-integrated-storage-import/991d0c2/controller/logs/tempest_conf.txt19:42
dansmithlooks like the list syntax is not parsed and we grab a chunk of it19:43
dansmithhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_991/923321/2/check/tempest-integrated-storage-import/991d0c2/testr_results.html19:43
dansmithmaybe the inner quotes are not needed?19:44
dansmithabhishek_: ^19:44
dansmithI bet that's the case and oslo.config is splitting on comma only19:44
dansmithso it gets everything up to the first comma19:44
abhishek_ok19:44
abhishek_so without inner '' quotes?19:45
dansmithso maybe should be:19:45
dansmithdisk_formats: qcow2,ari,aki,vhd,vmdk,raw,ami,vdi,iso,vhdx19:45
dansmith?19:45
dansmithtbh I dunno because the example above that one has inner and outer quotes19:47
abhishek_not sure how it is working for image_import_plugins: "['inject_image_metadata', 'image_conversion']"19:47
dansmithbut it sure looks to me like it needs no quotes at all19:47
dansmithright19:47
abhishek_enabled_import_methods: "[\"copy-image\", \"glance-direct\"]"19:48
abhishek_or should do something like this?19:48
dansmithidk19:48
abhishek_available_type_drivers: flat,geneve,vlan,gre,local,vxlan19:50
abhishek_I see somewhwere this19:50
abhishek_in neutron jobs19:51
dansmithack, I think what matters is how the tempest.conf is read, but I dunno19:51
dansmithmight as well change it I guess, it's not going to work as-is19:51
dansmithoh yeah,19:52
dansmithI see in tempest config for neutron19:52
dansmithI think that's right19:52
opendevreviewDan Smith proposed openstack/nova master: Fix disk_formats in ceph job tempest config  https://review.opendev.org/c/openstack/nova/+/92332219:53
abhishek_hmm, ket me do that19:53
fungilmk as soon as it has an approval vote and i can reenqueue it into the gate19:53
dansmithI guess this is why we don't (normally) skip check19:54
abhishek_++19:55
fungiheh19:56
fungiwe're not normally in quite so much of a hurry ;)19:56
dansmithI'm want to get off the merry-go-round19:58
fungiif we get it spinning fast enough, you can take advantage of centripetal force to make a speedy exit19:59
fungiyou'll probably land somewhere over by the cotton candy booth19:59
sean-k-mooneygiven the week in it i expect a lot of people to dissapre after tomorrow one way or another20:01
dansmithfungi: both of those are fixed (we think) and re-approved.. up to you if you want to take another gamble on the roulette wheel20:02
dansmithsean-k-mooney: looks like unless ceph fails, everything passed except for that live migration job, which is fairly effing amazing20:03
dansmithactually ceph already passed on the second one20:04
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/923317/120:04
sean-k-mooneymy fix passed20:04
dansmithsweet, I'll squash here in a few when the ones below are done20:04
sean-k-mooneyand was green on all jobs20:04
dansmithamazing20:04
sean-k-mooneyso ya you could also squash in the vmdk fix fi you wanted to reduce the set of patches20:05
sean-k-mooneybut ill leave that up to your judgement20:05
sean-k-mooneypart of me does not reallly want to mess with them if they are passing even if the end result is the same20:05
fungithe new revisions of both changes are at the top of the gate pipeline again now20:06
abhishek_thank you!20:09
sean-k-mooneywe have resutls for https://review.opendev.org/c/openstack/nova/+/923256 now and the only failure was the aki/ari issue20:23
sean-k-mooneytempest-integrated-compute-enforce-scope-new-defaults https://zuul.opendev.org/t/openstack/build/98a696b69cc844c6bdf9f7cb4b73f54d : SUCCESS in 2h 00m 47s20:23
sean-k-mooneythat job has a 2 hour timeout so it got very close20:23
sean-k-mooneythe last job on the 3rd patch is almost done too20:26
sean-k-mooneyok thats done too20:28
sean-k-mooneyill wait aorund for another 5 minutes otherwise ill check back in an hour or 2 before i go to sleep20:29
opendevreviewDan Smith proposed openstack/nova master: Reject qcow files with data-file attributes  https://review.opendev.org/c/openstack/nova/+/92325520:29
opendevreviewDan Smith proposed openstack/nova master: Check images with format_inspector for safety  https://review.opendev.org/c/openstack/nova/+/92325620:29
opendevreviewDan Smith proposed openstack/nova master: Additional qemu safety checking on base images  https://review.opendev.org/c/openstack/nova/+/92325720:29
opendevreviewDan Smith proposed openstack/nova master: Fix vmdk_allowed_types checking  https://review.opendev.org/c/openstack/nova/+/92325820:29
sean-k-mooneyim not sure you ment to update the first patch..20:30
dansmithoh crap, yeah I rebased my origin master chasing the other fire20:30
dansmithoh well, that sucks20:30
dansmithit was rechecking anyway20:30
sean-k-mooneyi +2w'd the set 20:31
dansmiththanks much20:31
dansmithI'm just about exhausted and fading pretty fast here20:32
sean-k-mooneyok im going to go eat. ill check back in an hour or two before i go to sleep and ill check them in the morning if they have any issues20:32
sean-k-mooneydansmith: go rest20:32
dansmithI have one other thing to do and then I will probably need to take a break for a bit20:32
sean-k-mooneywell theere in the ci so not much more for you do do on the nova side20:33
sean-k-mooneyunless your plannig to respin the backports but we can do that once they start to merge20:33
dansmithI'm going to hold off on those until we make progress yeah20:34
fungiargh, one job timed out on the glance change22:08
fungiduring volume tests, though that may have been coincidence22:09
fungithough the one remaining job on the nova change may be about to do the same22:10
fungithey're both starting over at the top of the gate pipeline again22:14

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!