Thursday, 2024-07-25

*** bauzas_ is now known as bauzas05:41
opendevreviewAlexander Hild proposed openstack/nova master: feat: "vm uuid" in baseBoard asset tag  https://review.opendev.org/c/openstack/nova/+/92490707:34
*** blady is now known as Guest122809:07
*** bauzas_ is now known as bauzas09:18
opendevreviewAlexander Hild proposed openstack/nova master: feat: Include "vm uuid" in baseBoard asset tag in libvirt domain xml  https://review.opendev.org/c/openstack/nova/+/92491610:30
sean-k-mooneygibi: bauzas: can one of ye hit https://review.opendev.org/c/openstack/nova/+/92473311:35
gibilooking11:36
gibi+A11:38
sean-k-mooneygibi: thanks11:39
sean-k-mooneygibi: i left some other comments on https://review.opendev.org/c/openstack/nova/+/924866 by the way in case you want to weigh in there11:40
sean-k-mooneyim +2 on it but want to get dans opion before proceeding11:40
andreykurilinHi folks! I have a custom Neutron plugin that restricts adding&removing specific security groups from ports. It works excellently with Nova's `addSecurityGroup` server action, i.e., Nova propagates an original error from Neutron to the end-user. But 'removeSecurityGroup' behaves differently - it raises 500 if something goes wrong on the Neutron side. Is the behavior difference between similar server actions intentional or not? 11:41
andreykurilinIf not, can this case be considered as a bug? And the last question - if it is a bug, does it require a microversion bump or not, as the 'removeSecurityGroup' API already defines a 400 error code?11:41
gibisean-k-mooney: ack. My limited understanding of the related code tells me that our defaults probably OK and if we start reconfiure things we will hit edge cases. So lets wait for Dan but I think we can land this as is and chalk up one more thing to care about when we rewrite the image code in nova.11:44
sean-k-mooneyyep thats where im at too11:44
sean-k-mooneyim going for an "eyes fully open appoch" just to be transparent and say hay this edge case exist but it existed before so not a regression11:46
sean-k-mooneyit does double down on the fact we need better tempest coverage of the formats we intend to supprot in the nova ci however11:47
sean-k-mooneyeven if those are just in a perodic job11:47
sean-k-mooneyor one that only runs on cahgnes to the image code11:47
gibiI agree. We only caught our regression becuase we were lucky to have a job with ami images11:55
sean-k-mooneyandreykurilin: we do not provide any supprot in nova for third party enturon exteions12:02
sean-k-mooneyandreykurilin: so unless this is in tree in neutron its by defientioan not a nova bug12:03
sean-k-mooneyif there is an aooved neutron api exteion for this beaviro then nova can be enhanced to supprot that when its enabeld12:04
opendevreviewBalazs Gibizer proposed openstack/nova master: DNM: test downstream cross repo testing  https://review.opendev.org/c/openstack/nova/+/92482312:04
sean-k-mooneythe behaivor being differnet is likely not intentional but im not sure if we shoudl fix it12:06
sean-k-mooneyif we were to fix this a micorversoin would not be approiate12:07
*** ykarel_ is now known as ykarel12:07
sean-k-mooneyandreykurilin: your custom neutron plugin is effectivly impelmeting "custom api policy" internaly which we also do not supprot12:07
opendevreviewStephen Finucane proposed openstack/osc-placement master: Remove use of distutils  https://review.opendev.org/c/openstack/osc-placement/+/92492612:12
stephenfinsean-k-mooney: gibi: Can you folks review that when you get a chance? osc-placement is currently broken on Python 3.12, which is the default Python on Fedora 40 (and has been out since October 2023)12:12
sean-k-mooneystephenfin: 3.12 is alos only optional in this release and that was a hard battel to get to even that point12:18
sean-k-mooneybut im very much in favor of fixing any 3.12 issues12:18
sean-k-mooneyso yes ill review it now12:19
sean-k-mooneyhttps://github.com/openstack/governance/blob/master/reference/runtimes/2024.2.rst12:19
sean-k-mooneyfor refernce12:19
sean-k-mooneyah its this12:19
sean-k-mooneyso the other option is to use microversion_parse form oslo?12:19
gibistephenfin: Is it intetional that we ignore the patch version from semver?12:20
sean-k-mooneybut im ok with your in tree version too12:20
sean-k-mooneygibi: depends on the schem12:20
sean-k-mooneypatch only exsits in the 4 . version i.e. major.minor.bugfix.patch12:21
stephenfingibi: It's not semver, it's a microversion. to the best of my knowledge, microversions never have a patch version12:21
sean-k-mooneyand ya microvesion dont use it12:21
stephenfinand won't ever use it12:21
sean-k-mooneyhttps://pypi.org/project/microversion-parse/12:21
sean-k-mooneythat is the shared dep we maintain fo doitng htis by the way12:22
stephenfinyeah, see my commit message. This is small enough that I don't see the point in dragging in a new (and borderline unmaintained library)12:22
sean-k-mooneyright but in genreally i want use to not cargo cult this to every project12:22
sean-k-mooneyso im fine with this for now but either we shoudl pull that in or add this to oslo.utils12:23
stephenfinIf it belongs anywhere, it's probably osc-lib12:23
sean-k-mooneywell we need it for the sdk and in opentack tooo12:23
sean-k-mooneyby openstack i mean nova et al12:23
stephenfinbut this is going away once we switch osc-placement to sdk12:23
sean-k-mooneyoh i just mean i have seen this exact issue in pbr and other projects12:24
stephenfinmicroversion-parse handles things like parsing headers. I think using it in nova makes sense12:24
gibiahh OK I did not realized that we are parsing microversion12:24
sean-k-mooneyso i think we are both ok with the osc_placement patch for now?12:25
gibiand I cannot blame it to lack of coffee as I just had one12:25
gibiI put my +2 ther12:25
gibie12:25
sean-k-mooneyok ill +w it so12:25
gibicool12:25
gibithanks stephenfin for fixing it12:25
stephenfinnp12:26
*** blady is now known as Guest123512:43
*** Guest1235 is now known as gryf12:43
opendevreviewMerged openstack/osc-placement master: Remove use of distutils  https://review.opendev.org/c/openstack/osc-placement/+/92492612:50
andreykurilinsean-k-mooney: thanks for reply. “custom api policy” - yes and no. I can simplify the case - restricting to manipulate security groups for ‘less-privileged’ roles. It does not require a custom policy and reuses everything ‘builtin’. addSecurityGroup action reraises neutron’s policy error which gives end-user enough information what is going on, versus 500 by removeSecurityGroup13:01
andreykurilinNeutron plugin I mentioned previously is entirely on neutron’s project side. It is not nova specific, i.e., it does not modify/extend nova-api. I just tried to give more context about what I’m trying to achieve. it looks like I gave more misleading info rather than useful one :)13:10
sean-k-mooneyandreykurilin: right btu nova does not supprot neutron srbac funcitonaltiy or any non default policy/roles13:12
sean-k-mooneyandreykurilin: so https://docs.openstack.org/neutron/latest/admin/config-rbac.html is not supproted by nova because the netruon folks never created a spec or worked with use to add support13:13
sean-k-mooneywe ocationaly get one of request ot enable subset of it13:13
sean-k-mooneybut there has never been an effort to actully supprot it or update it when neturon adds new fucntionaltiy13:14
sean-k-mooneyandreykurilin: if you want to use any fucntionaly restricte by that then you cannot use nova proxy apis anymore13:14
sean-k-mooneyyou have to do the secrity group config via neutron api only13:15
andreykurilinsean-k-mooney: yes, I understand that nova should not care about neutron rbac. The only problem for me here is that the end user who sees a 500 internal error (with fill out a bug report message) is less happy than someone who sees a propagated error from neutron. As I mentioned, one method already does this.13:16
sean-k-mooneywhat error code does neutron return in this case13:16
andreykurilin40013:16
sean-k-mooney400 bad request is technially not correct for a permission denial13:17
sean-k-mooneydo you have an example of the neutorn error respocne you can post13:17
andreykurilinI could be incorrect, let me recheck quickly13:18
andreykurilinps: I was referencing https://github.com/openstack/nova/blob/master/nova/network/security_group_api.py#L635-L638 versus https://github.com/openstack/nova/blob/master/nova/network/security_group_api.py#L695-L69913:19
sean-k-mooneyso its partly because removing a secrity group is never ment to be somethign that can fail13:19
sean-k-mooneywe may be able to handel the 400 case there safely13:21
sean-k-mooneywe woudl need to knwo if there is any other case that neutron returns a 40013:21
sean-k-mooneybut we are only updatign the securit_group part of the port in that requst13:21
sean-k-mooneyso its proably ok13:21
sean-k-mooneyi dont like that we are returing a 400 form nova in this case by the way13:23
sean-k-mooneyit really should be a 40313:24
andreykurilinsean-k-mooney: rechecked, I was incorrect, sorry. Neutron actually raises 403. I thought that my simplified case for custom neutron logic is similar to policy errors13:24
sean-k-mooneyok so a 403 is what i would expect becuase this is being deigned based on the permission of the roles the request was made with13:25
sean-k-mooneyso nova should be handleling that gracefuly and returning a 403 as well so i think we can catch that and provide a better error message13:26
andreykurilinBut what about 400 ? :) Are you still ok with handling it the same way as in addSecurityGroup ?13:27
sean-k-mooneybut i would also consider this to be skirting what is and is not in scope to fix in nova13:27
sean-k-mooneyandreykurilin: i would prefer that to also be a 403 but that woudl require a microverison update13:28
sean-k-mooneyand im not sure we want that13:28
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/exception.py#L105413:29
sean-k-mooneythat unfortunetly inhrits form invlaide and default to a 400 as a result13:29
sean-k-mooneythat was a bug form when we first added the proxy api https://github.com/openstack/nova/commit/c3ed3dfcf99e9162616960bad5c7b7a36d3b245313:30
sean-k-mooneywe have got better at using sematicly correct http respoce codes since then13:30
andreykurilinYes, it is 400. It is set separately here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/security_groups.py#L43013:30
sean-k-mooneyso if we want to avoid a micor version bump we might be forced to use 400 https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/security_groups.py#L43213:32
sean-k-mooneyalthough i belive we are alwasy allowed to return a 401 and 40313:32
sean-k-mooneybtu we shoudl disucss that with gmann ^13:32
sean-k-mooneyandreykurilin: if you want to create a patch for this do13:32
sean-k-mooneywe can discuss the exacat details of the respoce code and message content as part of the review13:33
andreykurilinsean-k-mooney: great, thanks! I already have a patch for handling 400 as a part of removeSecurityGroup action https://review.opendev.org/c/openstack/nova/+/924399 . I’ll add 403 handling there as well13:34
sean-k-mooneycan you actully file a bug for this too13:35
sean-k-mooneyand add a release note please. 13:35
sean-k-mooneywe would ideally have both if we are going to backport this13:35
andreykurilinsure13:35
dansmithJayF: TheJulia: Someone told me that ironic is using glance's ARI format in some way, presumably to store your ramdisk image, is that right?14:09
opendevreviewMerged openstack/nova stable/2023.2: Change force_format strategy to catch mismatches  https://review.opendev.org/c/openstack/nova/+/92473314:11
TheJuliadansmith: yeah, aki and ari for the raw image artifacts set by an admin to facilitate ops, they never ever get handed off to a conversion operation14:15
dansmithTheJulia: okay, I feel like that's the wrong image type to be using.. what gets stored in the aki?14:16
sean-k-mooneythose are not required as far as i am aware by the way14:17
dansmithoh, the kernel for a pxeboot operation?14:17
TheJuliadansmith: yes14:17
sean-k-mooneywhat i ment by that is using ari and aki14:17
sean-k-mooneyto store the ramdisk and kernel images14:17
dansmiththe reason is that AKI, ARI are not just kernel and ramdisk, they are *amazon* kernel and ramdisk, which at least when it was defined meant "a paravirt kernel" and "a ramdisk with paravirt drivers in it"14:17
dansmiththat said, TheJulia, those things never flow through nova at all, is that right?14:17
sean-k-mooneyi dont think there is any ckecking that the ramdisk image and kernel iamge uuid are in those formats14:18
TheJuliadansmith: well, it is possible to do, given how they can be constructed14:18
dansmithasking because I want to (thermo) nuke AMI,ARI,AKI from nova14:18
TheJuliadansmith: never flow through nova14:18
dansmithokay14:18
fricklerdansmith: sean-k-mooney: I lost track a bit, are there fixes already that should make https://review.opendev.org/c/openstack/nova/+/924734 pass or does that still need further investigation? or just enough rechecks? ;)14:19
sean-k-mooneydansmith: TheJulia nova support suing sepreate ram/kernel disk indepenent of the format of the root disk as far as im aware14:19
sean-k-mooneyso if you took an ami and uploaed all 3 parts as raw14:19
sean-k-mooneyit would work14:19
TheJuliasean-k-mooney: yeah, it *should* and I would expect it to actually14:20
sean-k-mooneyas would uploading a qcow as qcow and refeince seperate kerenl/ram disk14:20
TheJuliaI think this goes back to age old convention14:20
dansmithsean-k-mooney: yeah, I'm sure we're not doing any specific checking on the aki,ari bits14:20
TheJuliaset like 10 years ago because people didn't think of raw14:20
sean-k-mooneyto me the only purposue today is lableing them as not bootable14:20
TheJuliaperfect14:20
sean-k-mooneyfor filtering the images14:20
dansmithTheJulia: no, raw has been around forever, it's because the disk image of an ARI,AKI booted thing is not a disk14:21
TheJuliadansmith: indeed and correct14:21
TheJuliaari/aki has also been in the docs for ages for some human delination purpose I never understood14:21
sean-k-mooneyfrickler: checkign14:21
TheJuliaother than it is an explicit kernel and explicit ramdisk14:22
dansmithTheJulia: it's not for humans specifically, it's for amazon compatibility which has long since gone away in direct form14:22
sean-k-mooneyfrickler: i think there is a requirement repo change to bump packaging to the 2023.2 version to fix the setuptools issues14:22
TheJuliadansmith: wasn't explicitly aware of that, but likely humans just didn't know to update ironic's docs.14:22
dansmithTheJulia: virtual machines booted this way don't use a bootloader on the disk and are sort of warm booted from a pre-crafted memory state14:23
fricklersean-k-mooney: ah, that was merged two days ago, so I'll try to recheck then https://review.opendev.org/c/openstack/requirements/+/92476414:23
sean-k-mooneyyep that the patch i was looking for14:23
sean-k-mooneyperfect14:23
dansmithTheJulia: right, I think the human problem is that people saw "ramdisk" and "kernel" and thought "oh these are what I should use" without considering what the "amazon-" prefix meant in context14:23
dansmithTheJulia: anyway, the thing I wanted to confirm is that if we were to drop all that special behavior from nova it's not going to impact ironic and it sounds like that's true14:24
TheJuliadansmith: yeah, its just a doc issue really14:24
TheJuliadansmith: correct14:24
dansmithTheJulia: you mean a doc issue because you could easily store all those in 'raw' images and it would all work fine right?14:24
TheJuliayeah14:26
dansmithack14:26
TheJuliaat least *should* afaik14:26
TheJuliaI would expect it to in other words14:27
sean-k-mooneyunless ye have also specical cased it for some archane reason im not sure why it woudl not14:28
sean-k-mooneythe format you mark somethign in in glance really does not change the content of the data14:28
dansmithright, glance is going to start using inspector to enforce the content of the stream at some point,14:29
dansmithand if we were to get particular about it, we'd need to decide what we're going to accept for those formats14:29
TheJuliaCan the format be changed post upload?14:32
* TheJulia has never tried, hence the question14:32
dansmithno14:32
TheJuliaoh joy14:34
TheJuliaokay14:34
TheJuliafolks would just have to explicitly correct any incorrect usage14:34
sean-k-mooneyTheJulia: glance does not allow the format to be updated after the fact14:34
dansmithI'm not saying you need to do that, to be clear,14:34
* TheJulia creates a bug to fix the docs so we take care of it sooner rather than later14:34
dansmithI'm saying it's wrong, historically wrong, and I want to remove that specialized support from nova14:35
sean-k-mooneyTheJulia: so yes but by uploading and using new iamges with the corect format14:35
TheJuliayeah, I grok that14:35
dansmithbut you guys can continue being wrong, I just don't want to break ironic if doing so in nova would be a problem14:35
TheJuliasean-k-mooney: yeah, figured that14:35
TheJuliadansmith: no worries, we should fix it anyway just to remove confusion14:35
JayFWe have plenty of other ways to keep being wrong instead ;) 14:35
dansmithJayF: I was being overly dramatic in my use of wrong of course :)14:36
JayFoh, I took it as jovial :D 14:36
dansmithokay good :)14:38
TheJuliaJayF: looks like our devstack plugin uses the wrong convention too, but that also should be an easy/quick fix as far as I can see14:42
JayFyeah we've been telling people aki/ari since forever14:45
JayFI'm probably at least somewhat responsible for that, at least as the last person left from the team that made the agent lol14:45
TheJuliaJayF: https://bugs.launchpad.net/ironic/+bug/207409014:46
dansmithglance probably needs an image format type of "non-bootable-artifact" or something like that14:46
dansmithraw is technically okay for those things, but people think raw means "a MBR/GPT disk that I should be able to boot in nova" too14:47
dansmithperhaps raw but setting hw_arch to be "linux/x86" or something to make it clear it's not a machine-bootable thing or something like that14:47
TheJuliaSimple is likely best here, since raw is just bytes, not a disk image under a certian format14:48
JayFYeah, I think that's likely why we'd have gone with the aki/ari back in the day, like you said above, pattern matching on "kernel" and "ramdisk" and there's no other choice that fits14:48
JayFit's almost like we're trying to combine "purpose" and "image format" in the same field, and it's resisting for good reason :D 14:48
dansmithyeah, and the fact that all three are also options for container_format makes it even less sensical14:49
TheJuliayup14:49
dansmithyou know what we should do14:53
dansmithwe should add a disk_format of 'gpt' to glance,14:54
dansmithand start rejecting raw in nova over time14:54
dansmiththen we can assert that a gpt image actually has a gpt header and we could stop with this "anything that doesn't match another format must be raw" silliness,14:54
dansmithwhich would really help a lot I think14:55
dansmiththen raw can be more like what TheJulia says above (which is correct, just with caveats today)14:55
TheJuliayeah14:57
TheJuliagpt would be nice actually, since there are distinctions but really it should get treated exactly like raw14:58
JayFa partition image, so to speak, would always have to be raw, yeah?14:58
TheJuliathe VM firmware... otoh14:58
TheJuliaJayF: it could be qcow2 as a container14:58
TheJuliaand when unpacked it is just bytes14:58
dansmithTheJulia: right it would be treated like raw inside of nova and for qemu, but we could assert that it's not just random bytes but at least looks like a gpt on the surface,14:59
dansmithrefuse to boot something that is claimed as raw in glance (which could be a kernel or ramdisk or text file)14:59
dansmithand not have the "else: must be raw" behavior everywhere in nova, which is the source of lots of problems14:59
dansmithbecause to qemu, an iso is raw because it doesn't support it15:00
TheJuliayeah15:00
TheJuliamakes sense actually15:00
dansmithand ami/ari/aki images also have to be treated as raw because qemu-img doesn't support them when we're comparing notes15:00
TheJuliayeah15:01
dansmithbasically this sort of hack: https://github.com/openstack/nova/blob/master/nova/virt/images.py#L18615:01
andreykurilinsean-k-mooney: er… I cannot find the actual nova’s code that does it, but 403 is automatically propagated from neutron’s call. So my simplified case was totally irrelevant. Sorry for misleading you. It is only about 400 as a part of removeSecurityGroup now.15:04
*** bauzas_ is now known as bauzas15:04
dansmithralonsoh: does your comment on the ami snapshot patch mean it solves the problem for your neutron stuff?15:38
dansmithah looks like it from the neutron DNM patch, cool15:39
dansmithsean-k-mooney: ^15:39
*** iurygregory__ is now known as iurygregory16:35
priteauLooks like https://review.opendev.org/c/openstack/nova/+/924734 is ready to merge17:07
fricklerdansmith: ^^ do you want to have another look?17:23
gmannsean-k-mooney: andreykurilin: on SG error. I agree to handle both 403 as well 400 in both Add/Remove SG cases which is always better than 500. NOTE: neither 400 or 403 need microversion bump as those are always existing error code from all the APIs so we can just fix this with bug fix with no microversion bump 17:23
dansmithfrickler: +Wd17:23
gmannsean-k-mooney: andreykurilin: 500 -> '400, 403, 404 or 415' is all ok. https://docs.openstack.org/nova/latest/contributor/microversions.html#id3  "The exception to not needing a microversion when returning a previously unspecified error code is the 400, 403, 404 and 415 cases. "17:24
opendevreviewMerged openstack/nova master: Remove AMI snapshot format special case  https://review.opendev.org/c/openstack/nova/+/92486617:29
gmannsean-k-mooney: andreykurilin but for 403 we should define new exception which can clearly have msg about neutron access error otherwise user can confuse on nova access issue. Something inherited from PolicyNotAuthorized or Forbidden but a different error msg -  https://github.com/openstack/nova/blob/master/nova/exception.py#L19117:29
gmannandreykurilin: just read your last msg about 403 is already propagated. can you check what error msg it say?17:31
gmannandreykurilin: bcz I am not sure where we handle the neutron 403, but it is propagated in both cases addSecurityGroup and removeSecurityGroup ? https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/security_groups.py#L421-L430   https://github.com/openstack/nova/blob/eb5e3374bcb5de3b5986b66438a59c73be7fb596/nova/api/openstack/wsgi.py#L66717:49
*** iurygregory__ is now known as iurygregory17:57
andreykurilingmann: `RESP BODY: {"forbidden": {"code": 403, "message": "rule:update_port is disallowed by policy\nNeutron server returns request_ids: ['req-97c54f6e-6876-4cf4-9657-b8b3be322f57’]”}}`. vs `RESP BODY: {"forbidden": {"code": 403, "message": "Policy doesn't allow os_compute_api:os-security-groups:add to be performed.”}}` in case of Nova’s native policy error.18:15
andreykurilinDespite the fact neutron errors have an extra 'suffix' line with request-id, mo, adding some prefix to error's from neutron should give more visibility of "guilty" side.18:16
andreykurilingmann: https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L213-L214 neutronclient’s 403 is handled on ‘globally’18:22
gmannandreykurilin: right. did not check that. ++18:23
andreykurilinYeah, It took me a while to find it18:23
gmannandreykurilin: then you can just add 400 case in Remove SG case18:24
gmannall these handling are so deep in code and hard to find :)18:24
andreykurilingmann: thank you! I’ll update my patch yesterday with a proper release note.18:26
gmannandreykurilin: ++ thanks18:26
andreykurilin*tomorrow (need to relax )18:34
opendevreviewMerged openstack/nova stable/2023.1: Change force_format strategy to catch mismatches  https://review.opendev.org/c/openstack/nova/+/92473419:18
opendevreviewJay Faulkner proposed openstack/nova master: [ironic] Factor out metadata and  send to ironic  https://review.opendev.org/c/openstack/nova/+/92391020:08
JayFsean-k-mooney: https://review.opendev.org/c/openstack/nova/+/923910 + https://review.opendev.org/c/openstack/ironic/+/924887 are tested working in devstack. Still have to write a small migration script and some tempest tests, but wanted to get general agreement on the shape of the change before going further. If you could have a look I'd appreciate it. (/me goes and asks the same of Ironic cores)21:13
*** bauzas_ is now known as bauzas21:16
opendevreviewsean mooney proposed openstack/nova stable/2024.1: Remove AMI snapshot format special case  https://review.opendev.org/c/openstack/nova/+/92498221:46

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!