| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 00:15 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: oci: Enable embedded authentication passing https://review.opendev.org/c/openstack/ironic-python-agent/+/941300 | 00:25 |
| opendevreview | Julia Kreger proposed openstack/ironic master: OCI: Send the auth header to IPA https://review.opendev.org/c/openstack/ironic/+/941252 | 00:49 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: oci: Enable embedded authentication passing https://review.opendev.org/c/openstack/ironic-python-agent/+/941300 | 00:55 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: oci: permit an 'unknown' but valid image https://review.opendev.org/c/openstack/ironic-python-agent/+/941494 | 01:16 |
| TheJulia | Okay, got it so an oci url user (who supplies a specific digest as well) to be able to do image_download_source=http which can also now handle a remote image registry enforcing authentication. | 01:37 |
| cardoe | Nice | 02:01 |
| cardoe | Will do JayF. I’m just trying to figure out how to do per process / agent / whatever config when you’re on a collapsed box like devstack and using uWSGI or gunicorn. | 02:02 |
| JayF | https://review.opendev.org/c/openstack/ironic-inspector/+/939407 is passing CI now \o/ (to deprecate ironic-lib) | 04:56 |
| frickler | JayF: could you revisit https://review.opendev.org/c/openstack/ironic/+/940635 please? all other backports are merged | 05:37 |
| *** dmellado075539372 is now known as dmellado07553937 | 06:50 | |
| vsaienko | cardoe: AFAIR switch to uwsgi was initiated to mitigate development/devstack deployments issues https://governance.openstack.org/tc/goals/completed/pike/deploy-api-in-wsgi.html We have never switched to uwsgi on production. For development environments I think any kind of wsgi server will be okay. | 07:10 |
| rpittau | good morning ironic! o/ | 08:04 |
| dtantsur | JayF: I use `./bifrost-cli testenv` regularly, the last time a week ago | 09:03 |
| rpittau | JayF: still having issues with bifrost? I used it recently with no issue | 09:27 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Add token validation to command GET endpoints https://review.opendev.org/c/openstack/ironic-python-agent/+/941539 | 09:32 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-inspector master: Temp disable grenade job to let ironic-lib deprecation patches pass https://review.opendev.org/c/openstack/ironic-inspector/+/941341 | 10:48 |
| opendevreview | Merged openstack/ironic-python-agent stable/2023.2: Warn when the provided checksum algorithm does not match the detected https://review.opendev.org/c/openstack/ironic-python-agent/+/934091 | 12:08 |
| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 13:07 |
| hjensas | good morning, I want to run ^ multiple times to see that it is stable. | 13:12 |
| hjensas | I tried to add the same job multiple times in zuul.d/project.yaml, but looks like it still just run one job in zuul status. | 13:13 |
| hjensas | Is there a way I can trick it to run 4 instances of the same job in parallel? | 13:14 |
| JayF | Define the job with four names and make them different by just setting an environment variable that isn't read by anything | 13:51 |
| hjensas | Thansk JayF - I'll try that on the next iteration. | 13:54 |
| opendevreview | Merged openstack/networking-baremetal stable/2023.2: avoid attribute error on bad password or config https://review.opendev.org/c/openstack/networking-baremetal/+/934072 | 14:04 |
| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 14:27 |
| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 14:30 |
| TheJulia | good morning | 14:40 |
| opendevreview | Julia Kreger proposed openstack/ironic master: OCI: Send the auth header to IPA https://review.opendev.org/c/openstack/ironic/+/941252 | 14:48 |
| cardoe | morning all. I dunno how folks feel about the peeling off patches in a series and landing them. Like some of my "make hooks generic" that my redfish using hooks relies on should be good and passes tests. It actually fixes bugs in hooks that affect the agent today. | 14:57 |
| cardoe | Similarly Steve's console bits, his utils patch has been +2'd a bunch and it just changes from rebases. | 14:57 |
| cardoe | As far as the redfish hooks piece, I'm thinking of taking my docs on the shape of inventory a bit farther and actually writing out a jsonschema. A number of the hooks depend on specifics and will just throw KeyError if its not there. So when either implementation goes to save the inventory it gathered but before the hooks are executed, I'd check it against the schema and fail inspection if it didn't validate. | 14:59 |
| JayF | Going to be as blunt as I can be here: I have no interest in landing anything inspector related before ironic-lib-migration and inspector rules migration lands | 15:03 |
| JayF | the bottleneck on those patches has been review bandwidth | 15:03 |
| JayF | well, the rules migration patches :) | 15:03 |
| JayF | speaking of, https://review.opendev.org/c/openstack/ironic-inspector/+/939407 is approvaable | 15:04 |
| cardoe | Well I've +2'd them a couple of times. They have merge conflicts with master right now | 15:04 |
| JayF | and will free us up to land all the perfunctory patches and actually kill the ironic-lib from orbit | 15:04 |
| JayF | ugh, I'm sure cid will rebase them. That tends to happen when stuff is outstanding that long :( | 15:04 |
| cid | On that right now | 15:15 |
| opendevreview | cid proposed openstack/ironic master: DB: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939318 | 15:18 |
| opendevreview | cid proposed openstack/ironic master: Apply Rules: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939218 | 15:18 |
| opendevreview | cid proposed openstack/ironic master: API/Testing: Inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939217 | 15:18 |
| cardoe | JayF: so I +2'd the DB one. Prior to it having conflicts, it had 3 +2's so let's +W that? | 15:23 |
| JayF | looking | 15:24 |
| JayF | I think we were hoping for a +2 from dmitry, but I'm onboard with landing it, that's what followups are for | 15:24 |
| JayF | ty for pointing at that :) | 15:24 |
| cardoe | hjensas: We previously spoke about trunking missing in network_data.json, https://review.opendev.org/c/openstack/nova/+/941227 is a patch series I wanted to add you on. | 15:28 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Remove pre-victoria cycle agent token transition upgrade support https://review.opendev.org/c/openstack/ironic-python-agent/+/941565 | 15:28 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Remove agent_token_required upgrade knob https://review.opendev.org/c/openstack/ironic-python-agent/+/941566 | 15:28 |
| hjensas | cardoe: ok, I added myself on the review. cloud-init is already able to parse/use vlan information from network_data.json JSON schema right? | 15:35 |
| cardoe | Yes. | 15:35 |
| cardoe | With the caveat of https://github.com/canonical/cloud-init/issues/6018 | 15:36 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Ensure IPA is locked down in rescue mode https://review.opendev.org/c/openstack/ironic-python-agent/+/941489 | 15:44 |
| hjensas | cardoe: ah, I wonder if using systemd link files would make sense to give interfaces names. If we could get cloud-init to write the link files. (https://manpages.debian.org/buster/udev/systemd.link.5.en.html) | 15:47 |
| JayF | Is there a good upstream image anywhere that's suitable for using with arm servers? | 15:49 |
| JayF | Trying to prototype a thing and all I can find are installers and images built for raspberry pi. | 15:49 |
| JayF | This is an Ampere-something :D | 15:49 |
| JayF | apparently it's one of these https://www.hpe.com/uk/en/servers/proliant-rl-300.html | 15:52 |
| arne_wiebalck | I see that the Redfish interop profiles made it to the Ironic code base and that they are referenced in DMTF documents (great!). Has anyone experience using them for compliance testing when purchasing new hardware? | 15:54 |
| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 15:58 |
| frickler | JayF: upstream image for what? I did install bookworm on one ampere machine recently | 16:00 |
| JayF | literally anything that Ironic could install | 16:01 |
| JayF | so it needs to be an *image* not an installer | 16:01 |
| JayF | my downstream trying to hello-world their multiarch | 16:02 |
| frickler | ah, I didn't get to trying ironic with it yet. but I'll sure be interested in any experience you come up with ;) | 16:04 |
| JayF | you just did a installer install then, yeah? | 16:04 |
| JayF | that's what we're finding; it's tough to find a prebuilt image that's not specific to hardware (in all cases seemingly: rpi) or cloud usage | 16:05 |
| JayF | but lots and lots of "just install it" | 16:05 |
| shermanm | I've had good luck with the ubuntu cloud arm64 images, and used them as a base to build our own. | 16:07 |
| shermanm | e.g. https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-arm64.img | 16:07 |
| cardoe | arne_wiebalck: I want to. | 16:08 |
| JayF | shermanm: did they contain expected drivers/bootloaders/etc for hardware or did you have to add them? | 16:09 |
| JayF | shermanm: our final state will be something along the lines of building our own, but trying to do a QA run sooner | 16:09 |
| JayF | we're hitting errors about mounting read only when trying to install the bootloader, and when I saw it was a cloud image I kinda immediately blamed it lol | 16:10 |
| shermanm | it all worked basically out of the box, I think I had to add network drivers for some servers, but that wasn't a platform issue? | 16:10 |
| shermanm | caveat, we're running these on cavium thunderx2, and fujitsu a64fx, so not a 1-1 match with yours | 16:10 |
| shermanm | both of these are uefi boot, anything thats u-boot is more of a headache | 16:11 |
| JayF | Thanks, that's really helpful. | 16:14 |
| JayF | It completely blows up my hypothesis that these images might be shaped wrongly for ironic | 16:14 |
| TheJulia | I need way more coffee today | 16:15 |
| JayF | I wouldn't be surprised for a failure mode of "machine won't boot" but it not imaging seemed weird | 16:15 |
| JayF | I went to 7/11 and got a red bull for the first time in a long time, I knew it'd be one of those days lol | 16:15 |
| arne_wiebalck | cardoe: Ok, thanks! The question of what we need to specify in terms of Redfish for upcoming purchases came up today and I remembered we tried to push these profiles a while ago. | 16:15 |
| arne_wiebalck | cardoe: We may have a look and see if/how we could use them in the process. | 16:16 |
| JayF | arne_wiebalck: I believe they were recently revised for accuracy too | 16:16 |
| arne_wiebalck | JayF: Thanks! | 16:16 |
| arne_wiebalck | JayF: Are they still based on the whole potential of sushy? | 16:16 |
| JayF | I don't fully understand what you're asking, but I'm pretty sure I don't know the answer :) | 16:17 |
| arne_wiebalck | JayF: :-D | 16:17 |
| arne_wiebalck | JayF: At the time the calls to a potential endpoint were using all potential calls in sushy. | 16:17 |
| jrosser | JayF: are you doing ironic+ampere? | 16:18 |
| * jrosser reads back | 16:18 | |
| arne_wiebalck | JayF: So, this is mostly likely a superset of what a deployment really needs. | 16:18 |
| JayF | jrosser: we're trying to hello-world a cloud image (of really any distro) onto a https://www.hpe.com/uk/en/servers/proliant-rl-300.html | 16:18 |
| JayF | jrosser: using yoga + some pulled forward patches for the *_by_arch configurations | 16:19 |
| jrosser | ah please ask mossblaser :) | 16:19 |
| arne_wiebalck | JayF: cardoe: We may have a look, but if you have any experience to share, let me know! | 16:19 |
| jrosser | we have this working in our environment on supermicro+ampere | 16:19 |
| JayF | arne_wiebalck: I haven't done purchasing in literally a decade, so I won't have a lot to help. :( | 16:19 |
| JayF | I mainly just wanna know what image | 16:19 |
| shermanm | on this topic, does anyone have nvidia's grace / grace-hopper machines yet? I'll need to make them play nice with ironic eventually | 16:21 |
| TheJulia | NobodyCam: ^ | 16:24 |
| mossblaser | JayF: For our infrastructure we're using images generated by disk image builder using the configuration outlined here: https://docs.openstack.org/openstack-ansible-os_ironic/latest/configure-ironic-multiarch.html#building-an-aarch64-user-image | 16:26 |
| JayF | mossblaser: ack; we're trying to avoid that path right now for a really hilarious simple reason: the bootstrap problem | 16:27 |
| JayF | mossblaser: you need an arm machine to build an arm machine, and $downstream_security makes it tough to navigate that path inside the firewall | 16:27 |
| jrosser | free tier on oracle cloud can fix that | 16:28 |
| mossblaser | ah yes! (we have regular ARM nova VM compute nodes which we run this one) | 16:28 |
| mossblaser | (but as it happens we didn't have those when we first started so as jrosser says, we actually bootstrapped using oracle's free tier systems(!)) | 16:28 |
| JayF | jrosser: oracle has a free tier that'll abduct all the security guys that would say "STOP" when I tried to use a random cloud server ;) | 16:29 |
| shermanm | we've actually been happily building ARM dib images on x86 servers | 16:29 |
| mossblaser | (I think I did manage to run DIB in a qemu soft-emulated ARM to completion once just to see if it could work.... but... just don't do that!) | 16:29 |
| mossblaser | shermanm: does it "just work" or did you need to do something special? (I'm fairly sure I recall this not working when I first tried it) | 16:30 |
| shermanm | jayf: btw, I just confirmed that the exact ubuntu noble image above works via ironic on our arm servers | 16:30 |
| JayF | mossblaser: I have https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Compiling_with_QEMU_user_chroot setup locally and use it a lot, but that's all on my gentoo laptop :D | 16:30 |
| JayF | shermanm: ack, tyvm | 16:30 |
| JayF | shermanm: what ironic version? | 16:30 |
| shermanm | this was via 2023.1, but we've also got it working on xena on a different site | 16:31 |
| JayF | ack | 16:31 |
| JayF | that's very useful, thank you | 16:31 |
| mossblaser | yeah; I think if you're even considering using an emulated VM, a USB stick (or Redfish/IPMI-boot-from-virtual-USB-stick) to manually bootstrap would be a better option! | 16:31 |
| JayF | Confirmed we get the same error on noble :( | 16:31 |
| mossblaser | (i.e. to use the machine to generate the image ;), not as a day-to-day solution(!)) | 16:32 |
| JayF | https://www.irccloud.com/pastebin/ecHB0UAC/ | 16:32 |
| JayF | the /dev/nvme0n1p15 looks suss to me | 16:32 |
| opendevreview | Merged openstack/ironic stable/2024.1: Fix redfish session cache on missing password https://review.opendev.org/c/openstack/ironic/+/940635 | 16:36 |
| shermanm | mossblaser: it "just worked" so long as we had the correct deps installed, here's a dockerfile I've been using | 16:42 |
| shermanm | https://github.com/ChameleonCloud/ipa-builder/blob/main/ipa_builder/Dockerfile | 16:42 |
| shermanm | most important is `qemu-user-static`, debootstrap will use it automatically if the arch flag is set | 16:42 |
| shermanm | same for running docker containers across architectures. it's slow, but not as bad as you'd guess, especially for bootstrapping | 16:42 |
| jrosser | i think a factor for us was that i don't think there is a prebuilt cloud image that uses the 64k-page (largemem) kernel | 16:47 |
| JayF | our end state will be a downstream built image | 16:49 |
| JayF | we're just literally trying to get prototype server #1 past the finish line :) | 16:49 |
| JayF | you all have given us lots of useful data and info | 16:49 |
| shermanm | not that you're hitting it, but I'll note one issue that gave us fits initially is that the arm64 ipxe binaries that ship with ubuntu don't support gzip, and couldn't extract the arm64 IPA initramfs | 16:50 |
| JayF | will keep that in mind | 16:50 |
| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 17:08 |
| rpittau | good night! o/ | 17:11 |
| JayF | shermanm: hilariously enough, someone in #ubuntu-arm is telling me the cloud image won't work lol | 17:16 |
| JayF | I trust the person who actually did the thing | 17:16 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Remove pre-victoria cycle agent token transition upgrade support https://review.opendev.org/c/openstack/ironic-python-agent/+/941565 | 17:36 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Remove agent_token_required upgrade knob https://review.opendev.org/c/openstack/ironic-python-agent/+/941566 | 17:41 |
| TheJulia | I've got a bunch of open small oci patches, fixing minor things and allowing image_download_source=http with registries enforcing authentication, I'll re-test once everything once merged since its just a pain to layer it all together right now :) | 17:58 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Add token validation to command GET endpoints https://review.opendev.org/c/openstack/ironic-python-agent/+/941539 | 18:00 |
| jrosser | JayF: I just deployed an ampere node successfully with the stock noble cloud image | 18:08 |
| opendevreview | Harald Jensås proposed openstack/ironic master: [WIP] OVN UEFI IPv6 https://review.opendev.org/c/openstack/ironic/+/941352 | 18:55 |
| opendevreview | Merged openstack/ironic master: DB: inspection rules migration https://review.opendev.org/c/openstack/ironic/+/939318 | 19:19 |
| TheJulia | ouch | 21:40 |
| TheJulia | https://www.irccloud.com/pastebin/XaCAqUMv/ | 21:40 |
| frickler | codeberg is under massive ddos from what I've heard. related: https://blog.codeberg.org/we-stay-strong-against-hate-and-hatred.html | 21:44 |
| JayF | We could self-mirror that into something like github. | 21:44 |
| TheJulia | dunno, just stinks (and really, sort of annoying we build so many things for tinyipa, but it is what it is) | 21:47 |
| TheJulia | Metal3 also timing out. Fridays are often better for CI :) | 22:15 |
| cardoe | Should we have prebuilt pieces and other loops test building those? | 22:17 |
| opendevreview | cid proposed openstack/ironic master: Pass agent token to get command results https://review.opendev.org/c/openstack/ironic/+/941607 | 22:33 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Ensure IPA is locked down in rescue mode https://review.opendev.org/c/openstack/ironic-python-agent/+/941489 | 22:36 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Add token validation to command GET endpoints https://review.opendev.org/c/openstack/ironic-python-agent/+/941539 | 22:38 |
| opendevreview | cid proposed openstack/ironic-python-agent-builder master: Prevent service restart while in 'rescued' state https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/941610 | 22:39 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Ensure IPA is locked down in rescue mode https://review.opendev.org/c/openstack/ironic-python-agent/+/941489 | 22:47 |
| *** dmellado075539373 is now known as dmellado07553937 | 22:48 | |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Make _by_arch ramdisk settings apply for pxe https://review.opendev.org/c/openstack/ironic/+/941611 | 22:54 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Ensure IPA is locked down in rescue mode https://review.opendev.org/c/openstack/ironic-python-agent/+/941489 | 22:54 |
| opendevreview | cid proposed openstack/ironic master: Pass agent token to get command results https://review.opendev.org/c/openstack/ironic/+/941607 | 23:23 |
| opendevreview | cid proposed openstack/ironic master: Pass agent token to get command results https://review.opendev.org/c/openstack/ironic/+/941607 | 23:24 |
| opendevreview | cid proposed openstack/ironic master: Pass agent token to get command results https://review.opendev.org/c/openstack/ironic/+/941607 | 23:25 |
| opendevreview | cid proposed openstack/ironic-python-agent master: Add token validation to GET command endpoints https://review.opendev.org/c/openstack/ironic-python-agent/+/941539 | 23:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!