| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 00:43 |
|---|---|---|
| zigo | TheJulia: genekuo: Of course, Debian has signed grub + shim + kernel. It's always built, on a special infra dedicated to signing binaries (with only the ftp master team having access to the keys...). | 07:20 |
| zigo | BOOTX64.EFI is supposed to be shim-signed. | 07:25 |
| zigo | cp /usr/lib/grub/x86_64-efi-signed/grubnetx64.efi.signed ${DEST_TFTP_DIR}/grubx64.efi | 07:25 |
| zigo | cp /usr/lib/shim/shimx64.efi.signed ${DEST_TFTP_DIR} | 07:25 |
| zigo | Then get your DHCPd to point to shimx64.efi.signed, or rename that file as BOOTX64.EFI | 07:26 |
| zigo | Also, I strongly suggest adding "-v -v -v" into /etc/default/tftpd-hpa TFTP_OPTIONS, so that it logs in /var/log/syslog and you can see what's going on this way. | 07:27 |
| zigo | I hope this helps ... | 07:27 |
| zigo | On *MY* side, I am having issue with nova yelling it cannot schedule my baremetal host (with "No valid host was found."). How can I fix that ? Where to look ? | 07:29 |
| zigo | Probably my flavors are wrong? | 07:29 |
| dtantsur | zigo: check nova-scheduler logs | 07:29 |
| dtantsur | and possibly nova-compute for any errors or warnings | 07:29 |
| zigo | dtantsur: I'm getting: "Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up." | 07:31 |
| zigo | Nothing in nova-compute.log ... | 07:31 |
| zigo | Oh, shit, my compute service is disabled ! :) | 07:31 |
| zigo | (and forgot to do the discover_hosts, stupid me...) | 07:33 |
| zigo | dtantsur: Now, I'm getting: https://paste.opendev.org/show/bRdOefkwAiPEYTWAjZik/ | 07:37 |
| dtantsur | Not familiar with placement, sorry | 07:37 |
| zigo | Ok, thanks anyways. | 07:39 |
| opendevreview | Huy Mai proposed openstack/sushy-tools master: Add fake_ipa inspection, lookup and heartbeater to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 08:01 |
| Sandzwerg[m] | <zigo> "Ok, thanks anyways." <- In general: check if anything is disabled, check if the flavors match, also placement has resource providers. Check if these exist and are free. We had issues that ironic providers sometimes get lost and we need to recreate them. Could be because of our setup though | 09:59 |
| zigo | Thanks, will try. | 10:09 |
| opendevreview | Verification of a change to openstack/tenks master failed: Fix CI failures https://review.opendev.org/c/openstack/tenks/+/880866 | 10:11 |
| opendevreview | Verification of a change to openstack/tenks master failed: Add retries for get_url and package tasks https://review.opendev.org/c/openstack/tenks/+/880759 | 10:11 |
| iurygregory | good morning Ironic | 11:14 |
| opendevreview | Fabian Wiesel proposed openstack/sushy master: Handle session-uri in body https://review.opendev.org/c/openstack/sushy/+/867969 | 11:40 |
| iurygregory | dtantsur, I noticed the metal3 job is failing in my patch "FAIL - node-1 Baremetalhost introspecting completed expected null to be in ready available", any ideas? .-. https://review.opendev.org/c/openstack/ironic/+/883031 | 11:58 |
| iurygregory | also if anyone knows why grenade is getting stuck when trying to run the upgrade version during ironic-dbsync I appreciate https://zuul.opendev.org/t/openstack/build/e3368f6d7fea470e864953d8f559e7d3/log/controller/logs/grenade.sh_log.txt | 11:59 |
| dtantsur | iurygregory: re metal3: hard to judge from the description. inspection failed. | 13:09 |
| iurygregory | ack, so probably not related to the code I added | 13:10 |
| * TheJulia tries to wake up | 14:01 | |
| genekuo | zigo: When I'm building images with DIB, even /boot/efi/EFI/debian isn't there. | 14:10 |
| genekuo | Looking into the build code to see if I can fix it | 14:11 |
| dtantsur | genekuo: have you added the required element for UEFI? block-device-efi or something like that? | 14:13 |
| genekuo | DIB_RELEASE=bullseye disk-image-create -t raw -o debian-bullseye -p mdadm -p shim-signed -p grub-efi-amd64-signed vm block-device-efi debian | 14:13 |
| genekuo | This is the command I use | 14:13 |
| zigo | genekuo: I can't help, I don't use DIB, however, I can give you my script to build an image with live-build, if you like... | 14:15 |
| zigo | You just need an extra param kernel so the ramdisk knows how to wget the squashfs at boot time, and have the squashfs hosted somewhere ... | 14:16 |
| zigo | genekuo: https://salsa.debian.org/openstack-team/debian/supply/-/blob/master/baremetal/usr/bin/supply-build-live-image | 14:17 |
| zigo | FYI, Supply, my own project, isn't ready (but on good tracks), so I wouldn't advise to use it just yet. | 14:33 |
| dtantsur | Long weekend ahead, see you on Monday folks o/ | 14:55 |
| JayF | o/ | 14:56 |
| * TheJulia needs several long weeks | 15:02 | |
| TheJulia | genekuo: if you post a dib fix, feel free to tag me in gerrit to take a look | 15:38 |
| TheJulia | \o/ shorter path found to cinder issue | 15:39 |
| iurygregory | \o/ | 15:42 |
| iurygregory | yay | 15:42 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Remove Cinder Instance UUID https://review.opendev.org/c/openstack/ironic/+/883412 | 15:42 |
| TheJulia | That might break in fun ways, lets see! | 15:43 |
| iurygregory | what file should I look to see what is happening when running online data migrations in our CI? .-. grenade doesn't like my patch and I have no idea why | 15:43 |
| TheJulia | the path I was heading down was indeed correct, likely something to do with cinderclient... actually. | 15:43 |
| iurygregory | TheJulia, I hope it won't break in fun ways... | 15:43 |
| JayF | the more I look at your cinder changes | 15:43 |
| TheJulia | iurygregory: grenade's log most likely | 15:44 |
| JayF | the less I understand everything | 15:44 |
| TheJulia | JayF: it is a confusing area of the authentication | 15:44 |
| JayF | if you provide an instance_uuid it does $things? | 15:44 |
| TheJulia | nope, stores it in a attachment record body | 15:44 |
| iurygregory | TheJulia, the log ends with 2023-05-16 23:45:22.078 | INFO [alembic.runtime.migration] Running upgrade 4dbec778866e -> 163040c5513f .-. | 15:44 |
| TheJulia | well, aside from that | 15:44 |
| TheJulia | iurygregory: got a link? | 15:44 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/e3368f6d7fea470e864953d8f559e7d3/log/controller/logs/grenade.sh_log.txt | 15:45 |
| mgoddard | Hi Ironic, FYI we're hitting ironic/vbmc issues in kayobe CI, suspect is libvirt 9.0.0 in CentOS stream 9 / Rocky Linux 9.2 | 15:49 |
| iurygregory | yay for changes in libvirt... | 15:49 |
| mgoddard | e.g. https://2be828fdcb6d38ccd8b2-55bc1896b856bdceaa89daf155d7413b.ssl.cf1.rackcdn.com/883368/1/check/kayobe-overcloud-tls-rocky9/6d39c24/primary/system_logs/vbmcd.txt | 15:49 |
| mgoddard | a working run from yesterday, libvirt 8.5.0: https://6c5513d00fc4b6428bbc-5f206d1ac6837606572ca22039b53566.ssl.cf5.rackcdn.com/883263/1/check/kayobe-overcloud-centos9s/b41a94b/primary/kolla/libvirt/libvirtd.txt | 15:52 |
| mgoddard | sorry, vbmcd logs for the working run: https://6c5513d00fc4b6428bbc-5f206d1ac6837606572ca22039b53566.ssl.cf5.rackcdn.com/883263/1/check/kayobe-overcloud-centos9s/b41a94b/primary/system_logs/vbmcd.txt | 15:52 |
| TheJulia | iurygregory: I think you have some issues in your migration, I've left a review with comments | 15:54 |
| iurygregory | TheJulia, much appreciated! o/ | 15:54 |
| TheJulia | I suspect it should be relativelly quick, my guess is different confusion since only one field *can* really be the primary key | 15:56 |
| TheJulia | design wise, you may need to have a separate database incrementing primary key | 15:56 |
| TheJulia | and have that disjointed from node_id | 15:56 |
| TheJulia | and then force node_id *and* compoennt to be a unique constraint | 15:56 |
| TheJulia | node_id internally for nodes, if memory serves, is self incrementing | 15:59 |
| iurygregory | TheJulia, so basically a new column id for PrimaryKey, and create a UniqueConstraint for node_id + component right? | 16:04 |
| TheJulia | yes | 16:05 |
| iurygregory | tks! | 16:05 |
| TheJulia | you can't really, afaik, have a compound primary key | 16:05 |
| iurygregory | yup, you are right | 16:06 |
| opendevreview | Mark Goddard proposed openstack/tenks master: Fix CI failures https://review.opendev.org/c/openstack/tenks/+/880866 | 16:12 |
| opendevreview | Mark Goddard proposed openstack/tenks master: Add retries for get_url and package tasks https://review.opendev.org/c/openstack/tenks/+/880759 | 16:12 |
| TheJulia | mgoddard: could it just be the template payload is no longer compatible? | 16:17 |
| TheJulia | in regards to the libvirt domain xml | 16:17 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware https://review.opendev.org/c/openstack/ironic/+/883031 | 16:43 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 17:19 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 18:01 |
| JayF | dtantsur: I think the problem with your bug dashboard may just be "launchpad is so slow the connection times out before it could ever possibly fetch all the bugs" | 19:37 |
| JayF | dtantsur: at least I know how it works now :D | 19:37 |
| TheJulia | sweet | 20:04 |
| JayF | dtantsur: scratch that; it returns when curl'd but appears to be just hanging inside requests for an unknown reason :| | 20:08 |
| * TheJulia works on fixing unit tests now that she has a fix | 20:09 | |
| JayF | aha, got it! eventlet weirdness | 20:09 |
| JayF | going to just excise eventlet from this, if it needs async I'll do 'real' async | 20:09 |
| * JayF should've jumped from WTF, it's hanging --> blame eventlet monkey patch more quickly | 20:10 | |
| TheJulia | I believe there is a prophecy around that | 20:10 |
| TheJulia | Possibly related to the one who was, the one who is, and the one who will be. | 20:11 |
| TheJulia | We need an AI called Zatheras. | 20:11 |
| JayF | I think eventlet is Zathras | 20:13 |
| JayF | there are many of them, but you can't be 100% sure it's not just pretending (greenthreads) | 20:13 |
| TheJulia | "I wanted that baremetal, not this baremetal" "such is zatharas' lot in life" | 20:13 |
| JayF | barely understandable except by the people who know {him,it} well | 20:13 |
| JayF | and also, how could I forget | 20:16 |
| JayF | EVERYONE BLAME ZATHRAS (eventlet) | 20:17 |
| TheJulia | Only if Tye does not appear tonight ;) | 20:18 |
| JayF | I made https://etherpad.opendev.org/p/ironic-bug-migration-leftovers as I'm finding stuff that needs attention | 20:22 |
| JayF | > ironic-tempest-bfv https://zuul.opendev.org/t/openstack/build/6a204e11b90b4b549e43678be7f00be4 : SUCCESS in 32m 54s | 20:40 |
| JayF | TheJulia: nice work! I assume this is to be believed? | 20:40 |
| TheJulia | looks like it | 20:42 |
| TheJulia | at least, at a glance, I'm fixing the rest of the patch up now | 20:42 |
| TheJulia | and the workload booted, and nothing exploded it seems | 20:44 |
| iurygregory | nice \o/ | 20:44 |
| TheJulia | err, what I have will only work until yoga apparently | 20:52 |
| iurygregory | grenade why you hate me? >.< same failure even after the fix :D, going to try to run ironic-dbsync in a nuc to see the results... | 20:55 |
| JayF | iurygregory: this is for DB firmware model? | 21:05 |
| JayF | iurygregory: still failing on the migration? | 21:06 |
| JayF | I could use a context switch, I'll see if something jumps out in your current patchset | 21:06 |
| iurygregory | JayF, yeah | 21:09 |
| iurygregory | same message in the grenade logs nothing new .-. | 21:09 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/4ba4b5017503403688cf19411adabd08/log/controller/logs/grenade.sh_log.txt | 21:10 |
| JayF | yeah I saw, I'm looking | 21:10 |
| JayF | https://github.com/openstack/ironic/blob/master/ironic/db/sqlalchemy/alembic/versions/dd67b91a1981_add_allocations_table.py calls op.create_foreign_key but yours doesn't | 21:11 |
| JayF | I don't know if that's bad or not, but it's the only real thing I can see in that which screams "broken" to me. | 21:11 |
| iurygregory | hummm | 21:11 |
| iurygregory | interesting | 21:11 |
| iurygregory | based on the docs it seems we can use to create the ForeignKeyConstraint and just pass it in create_table | 21:13 |
| iurygregory | https://alembic.sqlalchemy.org/en/latest/ops.html#alembic.operations.Operations.create_foreign_key | 21:14 |
| TheJulia | could the table creation not be completed yet? | 21:15 |
| iurygregory | I would expect an error or something no? | 21:17 |
| TheJulia | it could be mid-transaction and deadlocked | 21:18 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP bfv service change https://review.opendev.org/c/openstack/ironic/+/882985 | 21:21 |
| JayF | iurygregory: TheJulia: did we ever get A->C grenade jobs running? | 21:25 |
| iurygregory | I'm going to try to upgrade the DB in my bifrost env just to see the result of ironic-dbsync running the upgrade | 21:25 |
| iurygregory | JayF, I had the patch open for that | 21:25 |
| JayF | yeah, I think it bitrotted? | 21:25 |
| iurygregory | yeah | 21:25 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/836966 | 21:26 |
| JayF | we need to get that setup. I found myself wondering "what if all our DB migrations are broken due to $something" and realized that would be a way to test that | 21:26 |
| iurygregory | I totally forgot to update the patch due to downstream priorities basically | 21:26 |
| JayF | makes sense. We should probably aim to get that running before B-1 | 21:27 |
| iurygregory | ++ | 21:27 |
| JayF | if you don't get around to it this week, I might have a look | 21:27 |
| iurygregory | ack | 21:27 |
| iurygregory | I'm going to push an update after my dinner it would be good to see how "grenade" and "grenade-skip" will play together | 21:28 |
| JayF | I just feel like that could be a time sink for us, bigly | 21:28 |
| JayF | given the amount of pain we have with the existing upgrade job | 21:28 |
| JayF | getting something up so we can collab and start seeing how bad it'll be is useful :| | 21:28 |
| iurygregory | yeah, I saw green runs of grenade, so I thought it would be something on my patch only | 21:28 |
| JayF | dtantsur: https://github.com/dtantsur/ironic-bug-dashboard/pull/1 works locally fwiw. Very minor changes but I don't think it's done yet. | 21:42 |
| JayF | TheJulia: you have answers for the two questions at the top of https://etherpad.opendev.org/p/ironic-bug-migration-leftovers perhaps? | 21:43 |
| TheJulia | 1) there is none. 2) i don’t remember. It might be worthwhile reaching out to Diablo rojo | 21:49 |
| JayF | TheJulia: some of these projects don't have an LP at all | 21:55 |
| JayF | e.g. https://bugs.launchpad.net/virtualbmc | 21:56 |
| JayF | is that just "mash the buttons in launchpad" ? | 21:56 |
| TheJulia | Basically yeah | 22:00 |
| TheJulia | Yeah, the radio button needs to be changed setting wise | 22:01 |
| TheJulia | Are there others? | 22:02 |
| JayF | I'm mid-audit, I doubt it but I am still checkint | 22:02 |
| TheJulia | K | 22:02 |
| JayF | tenks doesn't have one at all, but I'm not sure it needs one | 22:03 |
| JayF | Yeah, tenks needs an LP, but I think it's in a different state than virtualbmc | 22:04 |
| JayF | that's the only remaining one | 22:04 |
| JayF | https://bugs.launchpad.net/networking-baremetal I was wrong :( I had this mislabeled in my notes | 22:08 |
| TheJulia | cleaning up virtualbmc | 22:10 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: Bugs are now tracked in launchpad, fix docs https://review.opendev.org/c/openstack/networking-generic-switch/+/883450 | 22:11 |
| TheJulia | https://bugs.launchpad.net/virtualbmc/+bug/1702409 <-- checkout! | 22:14 |
| TheJulia | networking-baremetal re-enabled | 22:15 |
| JayF | lmao | 22:17 |
| JayF | that's not our bug, friendo | 22:17 |
| JayF | and that moment after you comment on a bug and you see it's 6 years old | 22:19 |
| JayF | the bug is in first grade | 22:19 |
| JayF | gg JayF | 22:19 |
| TheJulia | heh | 22:20 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: Bugs are now in launchpad, doc fixes https://review.opendev.org/c/openstack/networking-baremetal/+/883453 | 22:29 |
| opendevreview | Jay Faulkner proposed openstack/virtualbmc master: Bugs are now in Launchpad, update docs to reflect https://review.opendev.org/c/openstack/virtualbmc/+/883454 | 22:34 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: Bugs are now in launchpad, doc fixes https://review.opendev.org/c/openstack/networking-baremetal/+/883453 | 22:35 |
| iurygregory | https://launchpad.net/~ironic-bugs/+mugshots | 22:36 |
| iurygregory | wow I didn't know launchpad had pictures :D | 22:37 |
| opendevreview | Jay Faulkner proposed openstack/networking-generic-switch master: Bugs are now tracked in launchpad, fix docs https://review.opendev.org/c/openstack/networking-generic-switch/+/883450 | 22:37 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Ironic (and IPA) use launchpad now https://review.opendev.org/c/openstack/ironic-python-agent/+/883211 | 22:39 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Update docs: Ironic uses launchpad now https://review.opendev.org/c/openstack/ironic/+/883209 | 22:42 |
| JayF | iurygregory: time to upload a mug then I guess ;) | 22:43 |
| iurygregory | now you gave me the idea to request a mug with the ironic mascot lol | 22:44 |
| JayF | it has to be 192x192 :| | 22:45 |
| JayF | that's annoying | 22:45 |
| iurygregory | they have limitation and they probably don't provide a tool you can select the part of the photo you want (like linkedIn, facebook, twitter...) | 22:46 |
| JayF | I mean, we're a bunch of OSS developers. We can guess why /those/ OSS developers haven't implemented that, I suspect | 22:46 |
| JayF | lol | 22:46 |
| JayF | it's OK they didn't have time, gimp is on flathub, that didn't take long :D | 22:49 |
| *** dmellado95 is now known as dmellado9 | 23:04 | |
| iurygregory | (venv) [iurygregory@blacklotus ironic]$ ironic-dbsync --config-file etc/ironic/ironic.conf.local create_schema | 23:08 |
| iurygregory | 2023-05-17 20:08:04.301 112741 INFO alembic.runtime.migration [-] Context impl MySQLImpl. | 23:08 |
| iurygregory | 2023-05-17 20:08:04.302 112741 INFO alembic.runtime.migration [-] Will assume non-transactional DDL. | 23:08 |
| iurygregory | INFO [alembic.runtime.migration] Context impl MySQLImpl. | 23:08 |
| iurygregory | INFO [alembic.runtime.migration] Will assume non-transactional DDL. | 23:08 |
| iurygregory | INFO [alembic.runtime.migration] Running stamp_revision -> 4dbec778866e | 23:08 |
| iurygregory | ok, at least locally I can create a database, let's see if I can run the upgrade... | 23:08 |
| iurygregory | fingers crossed | 23:08 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Fix Cinder Integration fallout from CVE-2023-2088 https://review.opendev.org/c/openstack/ironic/+/882985 | 23:19 |
| TheJulia | well, the other question is will the db work | 23:21 |
| TheJulia | anyhow, exhausted, going to recharge a littl ebit and go to the game | 23:22 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!