| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Add comprehensive BMC state validation after firmware upgrades https://review.opendev.org/c/openstack/ironic/+/960230 | 04:45 |
|---|---|---|
| opendevreview | Arnaud Morin proposed openstack/ironic master: Deprecate usage of ironic.api.wsgi:initialize_wsgi_app https://review.opendev.org/c/openstack/ironic/+/960209 | 06:22 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Add comprehensive BMC state validation after firmware upgrades https://review.opendev.org/c/openstack/ironic/+/960230 | 06:33 |
| dtantsur | JayF: IrSO does not enable integration with any OpenStack services. It's not a hard "no", just nobody in the Metal3 world is really interested. | 06:59 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Add comprehensive BMC state validation after firmware upgrades https://review.opendev.org/c/openstack/ironic/+/960230 | 06:59 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Add comprehensive BMC state validation after firmware upgrades https://review.opendev.org/c/openstack/ironic/+/960230 | 10:21 |
| janders | dtantsur I will be at OIS Paris | 10:23 |
| stephenfin | JayF: frickler: Yup, I've been looking at the sdk gate issue since yesterday. Struggling to get an ironic devstack deployment up and running at the moment but once I do, it should be easy to see what SDK is sending that ironic is unhappy with | 10:27 |
| dtantsur | janders: \o/ | 13:04 |
| JayF | stephenfin: you can likely reproduce with tox -elocal-ironic-dev -- we have details on this in the Dev docs. I guess this is assuming that you can reproduce in no auth mode | 13:43 |
| lutimura | https://docs.openstack.org/ironic/latest/contributor/local-dev-guide.html | 13:55 |
| dtantsur | TheJulia: hey, any ideas why my deployment would get stuck in Attempting to resolve tag for: ParseResult(scheme='oci', netloc='quay.io', path='/dtantsur/cirros:0.6.3', params='', query='', fragment='') | 14:05 |
| dtantsur | ` Image oci://quay.io/dtantsur/cirros:0.6.3 could not be found` | 14:09 |
| dtantsur | and this is precisely why I objected to rolling out our own implementation :-/ | 14:09 |
| dtantsur | OCI client got a Non-2xx: status 404, reason NOT FOUND, text {"errors":[{"code":"MANIFEST_UNKNOWN","detail":{},"message":"manifest unknown"}]} | 14:09 |
| stephenfin | JayF: TIL | 14:15 |
| JayF | It's a nugget that was hidden in our docs for a long time, I hooked it up via tox and it's ideal for testing. Sdks | 14:16 |
| stephenfin | Sweet. Also, fwiw, I wasn't able to find a combination of DevStack config options that would allows me to stack without hitting ip route issues. And I tried at least a half dozen times. I suspect there's some docs missing (particularly for PUBLIC_SUBNET_IP). jfyi | 14:18 |
| lutimura | stephenfin: are you also getting errors about nexthop having an invalid gateway? | 14:31 |
| stephenfin | yup | 14:31 |
| TheJulia | dtantsur: Odds are we can't see it or are sending the wrong payload. A manifest unknown means the header is wrong, but it worked in the past so I'm super cuirous what is going on | 14:45 |
| TheJulia | dtantsur: can an unauthenticated user enumerate tags? | 14:46 |
| dtantsur | TheJulia: yep, this is a public repo | 14:47 |
| dtantsur | it works with https://github.com/oras-project/oras-py (but this project seems to have problems with authentication) | 14:48 |
| TheJulia | you *should* be able to enumerate tags with curl if you send the right header. The OCI spec is... frankly awful | 14:48 |
| TheJulia | (not surprising, see prior comment) ;) | 14:48 |
| dtantsur | I cannot do it with curl either. I don't know what oras and oras-py do differently but they do | 14:49 |
| TheJulia | The image servers are super super super touchy about what headers get sent and the headers indicate what kind of response they expect. The error has me wondering if the server behavior has changed slightly | 14:49 |
| TheJulia | ... Interesting | 14:49 |
| TheJulia | Yeah, don't bother digging into oras's depths, you'll cry. | 14:49 |
| TheJulia | hmmm | 14:50 |
| dtantsur | okay, with curl the secret sauce was -H 'Accept: application/vnd.oci.image.manifest.v1+json' | 14:50 |
| dtantsur | I guess Ironic does not do that? | 14:51 |
| TheJulia | it *should* be | 14:51 |
| TheJulia | which is curious | 14:51 |
| dtantsur | I have a feeling it sends the wrong one | 14:52 |
| dtantsur | needs application/vnd.oci.image.manifest.v1+json but sends application/vnd.oci.image.index.v1+json | 14:52 |
| dtantsur | although.. curl works with the index one as well Oo | 14:52 |
| TheJulia | ... I wonder if that somehow got changed accidently | 14:52 |
| TheJulia | ... Okay... | 14:52 |
| TheJulia | That is weird. :\ | 14:53 |
| TheJulia | Does the payload we get back have anything in it from the server? Could we be reformatting the url in some incorrect way? | 14:56 |
| TheJulia | Perhaps worth logging out the full request and interaction to verify we're sending the same | 14:56 |
| JayF | stephenfin: (cc other cores) -- I think we have a race in devstack setup with neutron | 15:02 |
| dtantsur | aha, listing tags works with both headers, getting a manifest - only with the manifest one | 15:02 |
| stephenfin | Oh, now that would make sense | 15:02 |
| JayF | stephenfin: that for some reason gets won by Ironic in our CI, and gets lost by us in some other CI (including the gophercloud CI that adamcarthur5 is working on -- he only got it to pass by disabling spinning up fake bm) | 15:02 |
| JayF | stephenfin: I suggest a similar fix: I doubt you all need IRONIC_BAREMETAL_BASIC_OPS for your tests, and if you do, we can rework them to go without them | 15:03 |
| JayF | and at PTG, I guess, we'll have to engage devstack/neutron teams and figure it out | 15:03 |
| stephenfin | JayF: Thanks. So to be clear, take https://docs.openstack.org/ironic/latest/contributor/devstack-guide.html, and `IRONIC_BAREMETAL_BASIC_OPS=False`, and profit $$$? | 15:09 |
| JayF | stephenfin: lets be explicit about what that does: that disables the entire "ironic fake bare metal VM" stuff | 15:09 |
| stephenfin | For what I want, that should be fine | 15:09 |
| JayF | stephenfin: if you don't need that, 100% do that. If you do need that ... we have a choice between changing your need for that or fixing devstack with our hair on fire and gmaan on leave | 15:09 |
| JayF | lol | 15:09 |
| stephenfin | Yes, best not have hair any more on fire than necessary | 15:11 |
| JayF | I'll note that's what Adam did in the gophercloud PR | 15:11 |
| JayF | so if you're happy with that and wanna approve over there (you are core over there too, right?) | 15:11 |
| stephenfin | just reviewing it now | 15:11 |
| stephenfin | do you have a bug describing the devstack issue that we can link to (so we remember why we did what we did in a few months/years time)? | 15:12 |
| JayF | ever troubleshoot something and never find the smoking gun | 15:13 |
| JayF | and feel like you're a little crazy and/or that there might be something else at play? | 15:13 |
| JayF | I've not put my finger on the exact bug b/c we've been digging it in github actions | 15:13 |
| JayF | so I've only had logs to go by, so there's no bug because I don't know what the bug is or (until now) that it wasn't $githubActionsShenanigans | 15:13 |
| dtantsur | TheJulia: get_manifest seems to be broken entirely since it assumes that digest is a part of the URL.. | 15:14 |
| TheJulia | oh | 15:14 |
| TheJulia | OH | 15:14 |
| TheJulia | your trying to get a manfiest directly from a tag? | 15:14 |
| TheJulia | its always a multi-step interaction. uhhhh hmmm | 15:15 |
| dtantsur | TheJulia: me? Ironic is trying to get a manifest from a digest | 15:15 |
| stephenfin | I reproduced it 6/6 (ish) times in the last 24 hours on VMs from our internal cloud (which both TheJulia and dtantsur have access to also), so we do have _a_ reproducer | 15:15 |
| dtantsur | TheJulia: while I can only get it from a tag with curl | 15:15 |
| TheJulia | dtantsur: yeah, tag has to be resolved to a digest to get it, so tag has to be resolved to a digest pointer | 15:16 |
| JayF | stephenfin: Do you have cycles to pair on this sometime? I think perhaps the intersection of our knowledge bases would have a big multiplicative impact | 15:16 |
| dtantsur | TheJulia: the other way around | 15:16 |
| JayF | stephenfin: assuming the workaround works and hair no longer needs to be on fire :) | 15:16 |
| TheJulia | we *have* to get a digest somehow, in the end, the tag is just a pointer. We have to go ask for it | 15:16 |
| dtantsur | TheJulia: yeah, you can get a digest from the tag manifest | 15:16 |
| dtantsur | not the other way around like you seem to be assuming | 15:16 |
| TheJulia | no, I'm not saying that | 15:17 |
| dtantsur | https://quay.io/v2/dtantsur/cirros/manifests/0.6.3 works | 15:17 |
| dtantsur | https://quay.io/v2/dtantsur/cirros/manifests/sha256:7d6355852aeb6dbcd191bcda7cd74f1536cfe5cbf8a10495a7283a8396e4b75b does not | 15:17 |
| dtantsur | however, Ironic tries the latter | 15:17 |
| TheJulia | oh WOW | 15:17 |
| TheJulia | okay | 15:17 |
| TheJulia | I misunderstood what your saying | 15:17 |
| TheJulia | oh | 15:18 |
| TheJulia | we're creating the wrong url | 15:18 |
| dtantsur | https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests very clearly says that digests should work, so I cannot blame you... | 15:18 |
| stephenfin | JayF: sure, though tbh the extent of my testing has been messing with various CIDR- and IP-related settings on the assumption that is was my environment that was broken | 15:18 |
| JayF | I think anything is possible | 15:19 |
| JayF | and we'll be able to get 100x further by being able to login in the broken state | 15:19 |
| TheJulia | dtantsur: yeah, there are some funky qwarks between server implementations | 15:19 |
| dtantsur | ah | 15:19 |
| dtantsur | https://quay.io/v2/dtantsur/cirros/manifests/sha256:4ed7603f8488ebe12108360bcd8a03f2042ae375ef1a6eb9ae497442f20f2ca6 this one works | 15:19 |
| dtantsur | it's the digest of the manifest itself, not of config or layer. fair enough | 15:20 |
| dtantsur | now I only wonder how I even get this digest in the first place? | 15:20 |
| TheJulia | when we ask for the index, it hands us the pointer if memory serves | 15:20 |
| TheJulia | granted, I think I wrote this last year?! | 15:20 |
| dtantsur | aha, oras tells me that one, so we're cool | 15:21 |
| dtantsur | now back to understanding why tags don't work. maybe it's a simple fix for the accept header.. | 15:21 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/959870/2 could use a semi-urgent review | 15:21 |
| dtantsur | before that: a tricky test: can we authentication to quay | 15:21 |
| TheJulia | Possibly, worth a try, we might need to detect/handle some weirdness. | 15:21 |
| TheJulia | dtantsur: I did some basic tests, but It *might* have been just been digest urls. I can't remember. I tried to test every combo back then | 15:22 |
| dtantsur | I really had some high hopes when I found oras-py.. just to realize it only really works with username/password auth | 15:22 |
| TheJulia | jut when I hit Openshift's image server, I couldn't use tags anymore so maybe I broke something in there | 15:22 |
| TheJulia | Yeah... and if you peel back the hood you'll quite literally cry | 15:23 |
| TheJulia | They may still be using print statements as well in it's base interaction logging | 15:23 |
| dtantsur | get_blob_url returns https:///v2quay.io/dtantsur/cirros/blobs/sha256:7d6355852aeb6dbcd191bcda7cd74f1536cfe5cbf8a10495a7283a8396e4b75b, should I be worried about it? :D | 15:23 |
| TheJulia | That should be correct | 15:24 |
| TheJulia | whoaw | 15:24 |
| dtantsur | I don't think v2quay.io is correct | 15:24 |
| TheJulia | v2quay | 15:24 |
| TheJulia | WUT?! | 15:24 |
| TheJulia | I have literally no idea where that is coming from | 15:24 |
| TheJulia | that is bizzar | 15:25 |
| TheJulia | so... If memory serves | 15:25 |
| dtantsur | that happens when you omit oci://, which cannot be triggered through API | 15:25 |
| TheJulia | yeah, I think I was doing everything with oci:// | 15:26 |
| dtantsur | will fix that too, just for internal consistency | 15:27 |
| dtantsur | TheJulia: good news: our authentication works (take it, oras-py!) | 15:31 |
| dtantsur | (we probably still need to fix oras-py and switch to it eventually) | 15:31 |
| TheJulia | wooohoooo! | 15:31 |
| TheJulia | I have serious doubts if it has the same activity level it had when I last looked at it. | 15:32 |
| dtantsur | I haven't checked parsing and passing the entire token so far, that's the next step (need to revive my bifrost) | 15:32 |
| JayF | We should evaluate if the cost of maintaining a library (think of how annoying it is to backport sushy fixes) is worth it before making that move | 15:33 |
| JayF | It'd be exceedingly annoying to try to backport a fix needed to continue supporting quay.io through a library | 15:34 |
| dtantsur | In any case, I doubt anyone has time for it. If I can fix my OCI issues, I'm good for now. | 15:34 |
| TheJulia | funny thing is, its actually a fairly simple interaction, once you understand the flow and quarks | 15:35 |
| stephenfin | JayF: Just confirming that worked, and I was *finally* able to stack | 15:39 |
| JayF | That makes me wonder if just writing a blog about our implementation would be an idea. Easier to point people to do the simple thing on their own. (Or just ... don't and that's OK too lol) | 15:40 |
| TheJulia | JayF: content is *always* a good idea | 15:40 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: OCI: accept both content types when requesting a manifest https://review.opendev.org/c/openstack/ironic/+/960283 | 15:43 |
| dtantsur | I'll WIP ^^ until I can test it end-to-end | 15:43 |
| TheJulia | ... I so should have thought of that | 15:45 |
| TheJulia | DOH | 15:45 |
| TheJulia | anyway, packing up to hit the road | 15:45 |
| dtantsur | safe travels | 15:45 |
| TheJulia | I'll try! Summer just wound up the corgi | 15:45 |
| JayF | Have a good time o/ pass on my hellos to Summer and the animal entourage | 15:46 |
| dtantsur | funnily, the OCI issue was a side quest for me. now I need to get back to what I was actually testing.. | 15:52 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Fix unbound variable in install-deps.sh https://review.opendev.org/c/openstack/bifrost/+/960286 | 16:03 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Fix off-by-one error https://review.opendev.org/c/openstack/ironic/+/960288 | 16:05 |
| stephenfin | JayF: lutimura: frickler: that's the fix 🤦♂️ | 16:05 |
| stephenfin | I have code in nova to prevent exactly that but I obviously never copied it to ironic | 16:06 |
| * stephenfin really needs to spin up the oslo.wsgi project | 16:06 | |
| dtantsur | "GET /v1/nodes/619a9d1e-66be-4964-8b64-f270dc3eab42/validate HTTP/1.1" 502 | 16:09 |
| dtantsur | don't worry, nothing works today | 16:09 |
| dtantsur | mmmm, aaaaaa, local RPC is so broken here | 16:12 |
| opendevreview | Merged openstack/ironic master: Avoid the wsgi application to start with eventlet https://review.opendev.org/c/openstack/ironic/+/959870 | 16:20 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Prevent overlapping schema versions https://review.opendev.org/c/openstack/ironic/+/960291 | 16:26 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: DNM: Revert "api: Fix off-by-one error" https://review.opendev.org/c/openstack/ironic/+/960292 | 16:26 |
| stephenfin | and there's a patch to prevent this happening again (with a DNM to prove it actually works) | 16:26 |
| stephenfin | JayF: While I have you: Ironic occasionally re-uses controllers as subcontrollers (so fwict `/portgroups/{id}/ports` is almost an alias for `/ports`). What are the microversioning expectations for those? | 16:28 |
| JayF | I'd answer that question by exploring existing behavior tbh | 16:28 |
| stephenfin | Is it expected that if I add a new field to `/ports/{id}` in 2.foo then that will also appear in `GET /portgroups/{id}/ports`? | 16:28 |
| JayF | I think so? | 16:29 |
| stephenfin | Ah, okay. I was hoping this might be written down somewhere but I couldn't find it | 16:29 |
| JayF | but I am not API semantics pro | 16:29 |
| stephenfin | gotcha | 16:29 |
| * stephenfin takes that as freedom to do what he thinks best | 16:29 | |
| stephenfin | :) | 16:30 |
| JayF | well like I said | 16:30 |
| JayF | existing behavior | 16:30 |
| JayF | we are big on not breaking people | 16:30 |
| JayF | so no change in behavior is the A+++++ answer | 16:30 |
| stephenfin | yeah, agreed | 16:30 |
| stephenfin | I ask because Nova reuses has the views module where a lot (but not all) of the response bodies are generated, but there have been a variety of bugs where said code is called by multiple controller methods with subtly different inputs | 16:31 |
| stephenfin | So an aliased API like above doesn't behave exactly like the original API. I'm hoping Ironic's pattern avoids that | 16:31 |
| * stephenfin will document findings | 16:31 | |
| JayF | Honestly I've approved port API changes | 16:32 |
| JayF | and never once thought about the subcontroller | 16:32 |
| JayF | so a little embarassed I never considered it | 16:32 |
| JayF | top two here: https://review.opendev.org/c/openstack/ironic/+/960288 from Stephen need review as they'll need to be backported (or if we get them in; I can update the release request) | 16:53 |
| opendevreview | Luan Utimura proposed openstack/ironic master: Fix nexthop error when adding route to PUBLIC_SUBNET_IP https://review.opendev.org/c/openstack/ironic/+/960299 | 16:56 |
| lutimura | stephenfin: thanks for the fix there! btw, about the nexthop thing, i managed to finish a ./stack.sh run after applying this change: https://review.opendev.org/c/openstack/ironic/+/960299. let me know what you think :) | 16:58 |
| lutimura | (ah, i had to run `ip addr flush dev ens3` to clean up stuff from previous ./stack.sh runs...) | 17:02 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Add schema for inspection rules API (requests) https://review.opendev.org/c/openstack/ironic/+/954039 | 17:13 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: api: Add schema for events API https://review.opendev.org/c/openstack/ironic/+/959880 | 17:13 |
| opendevreview | Stephen Finucane proposed openstack/ironic master: WIP: api: Add schema for portgroups API https://review.opendev.org/c/openstack/ironic/+/959881 | 17:13 |
| JayF | lutimura: where do I need to mail your gold star sticker | 17:40 |
| JayF | What a great find. I'll give it a more in-depth review and hopefully a plus two here in a minute or two | 17:41 |
| JayF | I'll note that while that comment was mostly a joke, I'm always happy to mail people ironic stickers | 17:41 |
| lutimura | JayF: hahaha that'd be awesome! thanks for looking into it; let me know if you have any suggestions, etc. | 17:48 |
| JayF | if the stickers would be awesome DM me an address and I'll assemble you a care package lol | 17:48 |
| JayF | It might be time to refresh my sticker budget anyway | 17:48 |
| opendevreview | Luan Utimura proposed openstack/ironic master: Fix nexthop error when adding route to PUBLIC_SUBNET_IP https://review.opendev.org/c/openstack/ironic/+/960299 | 19:16 |
| opendevreview | Luan Utimura proposed openstack/ironic master: Fix nexthop error when adding route to PUBLIC_SUBNET_IP https://review.opendev.org/c/openstack/ironic/+/960299 | 20:30 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!