| cbouchar | TheJulia, stevebaker: I too will checkout tomorrow. Thank you for responding. | 00:02 |
|---|---|---|
| opendevreview | Verification of a change to openstack/ironic master failed: Remove netboot jobs from the gate https://review.opendev.org/c/openstack/ironic/+/843937 | 00:06 |
| *** rcastillo_ is now known as rcastill | 00:10 | |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Prevent clear_job_queue failures on older iDRACs https://review.opendev.org/c/openstack/ironic/+/846859 | 00:57 |
| TheJulia | stevebaker[m]: https://cloud.centos.org/centos/9-stream/x86_64/images/ ~780mb, 804 mb, 834mb. The images are growing... (╯°□°)╯︵ ┻━┻ | 01:22 |
| stevebaker[m] | Ahhhhh | 01:23 |
| TheJulia | We needs to allocate more ram I guess :( | 01:24 |
| stevebaker[m] | or get it to stream to disk? | 01:29 |
| TheJulia | stevebaker[m]: VM booting, so actual virtual machines | 03:16 |
| TheJulia | at *least* on the UEFI one most likely | 03:16 |
| TheJulia | our IPA image has only grown a little bit, I'm guessing it is excess firmware | 03:17 |
| stevebaker[m] | TheJulia: oh. If more memory is a constraint we could consider building & publishing a custom test image | 04:25 |
| TheJulia | We likely need to consider it… I’m wondering why both are now timing out with ash as well | 04:26 |
| TheJulia | SSH | 04:26 |
| rpittau | good morning ironic! o/ | 06:49 |
| janders | hey rpittau o/ | 07:10 |
| rpittau | hey janders :) | 07:10 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.6: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/847155 | 07:22 |
| opendevreview | Verification of a change to openstack/ironic master failed: [Minor] Fix misspellings of "insufficient" https://review.opendev.org/c/openstack/ironic/+/845789 | 07:32 |
| opendevreview | Verification of a change to openstack/ironic master failed: Remove netboot jobs from the gate https://review.opendev.org/c/openstack/ironic/+/843937 | 07:33 |
| dtantsur | morning folks. the CI is still in a disastrous state, isn't it? | 07:41 |
| rpittau | dtantsur: well, some random failures in ironic CI, the disastrous part seems to be ironic-python-agent CI, although I saw some patches merge there | 08:03 |
| rpittau | so I wonder if the issue we saw was transient, I just issued a recheck to another patch that was impacted | 08:05 |
| opendevreview | Merged openstack/ironic-python-agent stable/xena: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/847157 | 08:46 |
| snanda | Hi Ironic ! | 08:54 |
| opendevreview | Merged openstack/ironic-python-agent stable/wallaby: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/847159 | 08:54 |
| snanda | Can we please review this one ? Already added priority tag. https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/845577/3 | 08:54 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.1: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/847160 | 08:54 |
| dtantsur | snanda: just to keep the file a bit more manageable, would it be possible to move iDRAC tests to a new file? | 09:01 |
| opendevreview | Merged openstack/ironic-python-agent stable/yoga: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/847156 | 09:05 |
| snanda | dtantsur: Is it really necessary to create new file ? As the other iDRAC deployemnt scenarios are also present in test_basic_ops file. | 09:18 |
| opendevreview | Merged openstack/bifrost master: Exclude ansible collection >1.8.0,<2.0.0 https://review.opendev.org/c/openstack/bifrost/+/843281 | 10:16 |
| dtantsur | snanda: ah, there is prior art, hmm | 10:22 |
| dtantsur | okay then | 10:22 |
| dtantsur | we need to rethink how we compose these tests anyway | 10:23 |
| cbouchar | stevebaker: your selinux fix looks like it will do what I need. TY | 11:05 |
| iurygregory | good morning Ironic | 11:36 |
| *** rcastill is now known as rcastillo | 13:02 | |
| opendevreview | Merged openstack/ironic-python-agent master: Drop support for Python 3.6 and 3.7 https://review.opendev.org/c/openstack/ironic-python-agent/+/844004 | 15:19 |
| TheJulia | oh, I see what is going on with metalsmith jobs | 15:45 |
| TheJulia | hmm. curious | 15:56 |
| rpittau | good night! o/ | 16:28 |
| opendevreview | Julia Kreger proposed openstack/metalsmith master: CI: Extend timeouts for RAX https://review.opendev.org/c/openstack/metalsmith/+/847380 | 16:32 |
| iurygregory | I wish extending timeouts would fix ipa-tempest-bios-ipmi-direct-src XD | 16:36 |
| TheJulia | iurygregory: what is going on there? | 16:43 |
| iurygregory | most of the failures seems to related to ipa loosing connection I'm adding my findings in the Whiteboard | 16:44 |
| iurygregory | https://etherpad.opendev.org/p/IronicWhiteBoard#L57 | 16:45 |
| TheJulia | iurygregory: so... I'm suspecting we're booting back into the ramdisk when we think we're on the instance workload | 16:52 |
| TheJulia | iurygregory: the hint is a cirros user is attempting to be logged into | 16:53 |
| TheJulia | iurygregory: agent token appears to be a red herring. It gets logged after the conductor has wiped the token from memory if memory serves | 16:58 |
| TheJulia | well, if my memory serves | 16:58 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use bifrost on centos9 integration job https://review.opendev.org/c/openstack/ironic/+/846155 | 17:01 |
| iurygregory | hummm | 17:05 |
| iurygregory | I'm wondering what changed in the workflow for this to happen .-. | 17:05 |
| TheJulia | eeek | 17:05 |
| TheJulia | almost 5 minutes before IPa actually starts | 17:05 |
| TheJulia | looks like averaging 247 seconds into boot | 17:06 |
| iurygregory | we only use cirros in the tinyipa scenario right? when we are using DIB the OS is CentOS9 Stream | 17:10 |
| TheJulia | I'm not sure I understand what you mean | 17:18 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Do not require deploy_kernel/ramdisk with the ramdisk deploy https://review.opendev.org/c/openstack/ironic/+/847388 | 17:25 |
| TheJulia | ++ | 17:27 |
| dtantsur | hmmmmm, this has interesting implication for cleaning, isn't it... | 17:28 |
| iurygregory | nvm, I was looking at two things different logs in different jobs... | 17:28 |
| iurygregory | time to close some tabs | 17:29 |
| iurygregory | why I'm having a deja vu feeling " Do not require deploy_kernel/ramdisk" .-. | 17:30 |
| TheJulia | iurygregory: so it is pxe boot retrying | 17:30 |
| TheJulia | dtantsur: it does... actually | 17:31 |
| dtantsur | iurygregory: ikr? | 17:31 |
| TheJulia | iurygregory: so, I need to double check the code | 17:31 |
| dtantsur | TheJulia: yep. I guess I can only do it when cleaning is disabled. | 17:31 |
| TheJulia | but the agent boots, gets a token, pxe retyr timer strikes because the thing is crawling to begin with | 17:31 |
| TheJulia | things reboot, can't get a token | 17:31 |
| TheJulia | because token is already allocated | 17:31 |
| iurygregory | dtantsur, I will take care of backporting since you will be out | 17:31 |
| TheJulia | basically, a fun race I think | 17:32 |
| dtantsur | iurygregory: thanks! I need to fix this cleaning issue, working on it now | 17:35 |
| iurygregory | ack | 17:35 |
| dtantsur | FYI this is a release blocker for us :( | 17:36 |
| iurygregory | oh wow | 17:37 |
| * iurygregory checks downstream channel | 17:38 | |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: No deploy_kernel/ramdisk with the ramdisk deploy and no cleaning https://review.opendev.org/c/openstack/ironic/+/847388 | 17:38 |
| dtantsur | now testing | 17:38 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: No deploy_kernel/ramdisk with the ramdisk deploy and no cleaning https://review.opendev.org/c/openstack/ironic/+/847388 | 17:49 |
| TheJulia | i propose we include ovh in with the rax filter | 17:50 |
| iurygregory | TheJulia, from the failures I saw based on cloud providers | 17:51 |
| iurygregory | Failures | 17:51 |
| iurygregory | - inmotion : iad3 4x | 17:51 |
| iurygregory | - ovh : GRA1 4x / BHS1 6x | 17:51 |
| iurygregory | - iweb : mtl01 x10 | 17:51 |
| iurygregory | in the filter to increase the timeout and/or the one to switch to tinyipa? | 17:53 |
| TheJulia | funky | 17:53 |
| TheJulia | yeah... | 17:53 |
| TheJulia | it feels evil | 17:53 |
| TheJulia | but... | 17:53 |
| TheJulia | anyway, need to write a test and I'll have a fix ready | 17:53 |
| dtantsur | okay, the 3rd patchset passed local testing. | 17:53 |
| dtantsur | see you tomorrow folks o/ | 17:54 |
| iurygregory | evil things ++ | 17:54 |
| iurygregory | dtantsur, bye o/ | 17:54 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Prevent pxe retry when agent token exists https://review.opendev.org/c/openstack/ironic/+/847392 | 18:06 |
| NobodyCam | good morning Ironic folks! Happy almost Friday | 18:13 |
| NobodyCam | any one happen to have in sight in why https://review.opendev.org/c/openstack/ironic/+/840986 didn't merge? rebase maybe? | 18:14 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: CI: Use 2 cpus for IPA jobs https://review.opendev.org/c/openstack/ironic-python-agent/+/847395 | 18:19 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: CI: Use 2 cpus for IPA jobs https://review.opendev.org/c/openstack/ironic-python-agent/+/847395 | 18:19 |
| TheJulia | so yeah, we want treat iweb like we do with rax | 18:21 |
| TheJulia | NobodyCam: lots of CI problems it seems | 18:21 |
| TheJulia | :( | 18:21 |
| TheJulia | actively working some of it | 18:21 |
| iurygregory | NobodyCam, yup! you will need to rebase to remove the dependency in the patch in chain that was Abandoned | 18:23 |
| NobodyCam | +++ | 18:24 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Add iweb to the use tinyipa on list https://review.opendev.org/c/openstack/ironic/+/847396 | 18:24 |
| TheJulia | iurygregory: ^^^ | 18:24 |
| TheJulia | that should clean up ~40% | 18:24 |
| iurygregory | TheJulia, .. related to " Use 2 cpus for IPA jobs " ironic-ipa-base uses ironic-base that has IRONIC_VM_SPECS_CPU: 2 | 18:25 |
| iurygregory | https://opendev.org/openstack/ironic/src/branch/master/zuul.d/ironic-jobs.yaml#L53 | 18:26 |
| iurygregory | so I'm like "WHAT?!" | 18:26 |
| iurygregory | +2 to the iweb | 18:26 |
| *** rcastillo_ is now known as rcastillo | 18:28 | |
| TheJulia | oh, doh | 18:30 |
| TheJulia | nevermind then on that chagne | 18:31 |
| TheJulia | workflowing the iweb change | 18:32 |
| TheJulia | iurygregory: https://review.opendev.org/c/openstack/metalsmith/+/847380/ is passing, if you wouldn't mind so we can get steve's stuff moving forward too | 18:33 |
| TheJulia | well, at least super close to passing on one last job | 18:33 |
| * TheJulia wonders if we could convince nobodycam to be the until 5pm emergency reviewer | 18:34 | |
| TheJulia | stevebaker[m]: https://review.opendev.org/c/openstack/metalsmith/+/847380/ | 18:34 |
| NobodyCam | maybe | 18:34 |
| TheJulia | sweet, metalsmith's last joba ppears to be passing into logs now | 18:36 |
| NobodyCam | However I am only able to give it a +1 | 18:37 |
| iurygregory | TheJulia, +W | 18:38 |
| TheJulia | iurygregory: awesome, thanks | 18:38 |
| TheJulia | now, lets see if CI clears up in the next few hours | 18:38 |
| iurygregory | yeah | 18:38 |
| iurygregory | fingers crossed | 18:38 |
| iurygregory | going to the gym, bbl o/ | 19:06 |
| opendevreview | Merged openstack/metalsmith master: CI: Extend timeouts for RAX https://review.opendev.org/c/openstack/metalsmith/+/847380 | 20:13 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Add iweb to the use tinyipa on list https://review.opendev.org/c/openstack/ironic/+/847396 | 21:04 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Add iweb to the use tinyipa on list https://review.opendev.org/c/openstack/ironic/+/847396 | 21:14 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Default to TinyIPA when nested virt is not possible https://review.opendev.org/c/openstack/ironic/+/847412 | 21:25 |
| iurygregory | back o/ | 21:39 |
| iurygregory | TheJulia, nice one re - "! $(grep -q vmx /proc/cpuinfo)" =) | 21:40 |
| TheJulia | old sysadmin tricks | 21:40 |
| TheJulia | unix whitebeard | 21:43 |
| JayF | `grep -q` is highly, highly underutilized | 21:54 |
| JayF | TheJulia: that could be written as $(grep -qv vmx /proc/cpuinfo) https://gist.github.com/jayofdoom/ebaa4f3c1406cc05dbe5af411b05346e (I was nerd sniped by the potential of playing bash golf) | 21:56 |
| JayF | and it made me wonder if grep -qv worked as expected (it does) | 21:56 |
| TheJulia | nice! | 21:56 |
| TheJulia | I was thinking it was "not enough flags" on grep | 21:56 |
| TheJulia | but then just went "oh, it works", onward! | 21:57 |
| TheJulia | oh joy, centos9 ramdisk is >400megs | 22:27 |
| iurygregory | (╥﹏╥) | 22:30 |
| TheJulia | 780mb uncompressed | 22:30 |
| * TheJulia will interpret that as crying | 22:30 | |
| iurygregory | yes | 22:30 |
| TheJulia | :) | 22:31 |
| JayF | I remember back before we had image streaming | 22:32 |
| JayF | and for agent deploys, your disk image*2 + total size of ramdisk was howm much ram you needed for deploy | 22:33 |
| TheJulia | 21Mlibjavascriptcoregtk-4.0.so.18.20.5 <-- do we *really* need javascript.... | 22:33 |
| TheJulia | for gtk at that.... | 22:33 |
| iurygregory | wow I wouldn't expect we need js | 22:34 |
| TheJulia | gah, the largest firmare offenders are network cards | 22:34 |
| janders | good morning Ironic o/ | 22:34 |
| * JayF in before "sshd needs libXlol for x forwarding, which depends on gtk3-banana which needs javacripe-for-gtk" | 22:35 | |
| TheJulia | "smartnics" | 22:35 |
| iurygregory | morning janders o/ | 22:35 |
| * TheJulia wonders if she can pay bills by making swords... and feels like gtk3-banana could be a useful fuel | 22:35 | |
| JayF | TheJulia: I'm using one of those atlantic nics which have GIGANTIC firmwares. I had to install it with a USB network adapter because install livecds wouldn't package the firmware | 22:35 |
| TheJulia | JayF: so... umm... does your network card run linux too? | 22:36 |
| * TheJulia wonders what needs cracklib | 22:36 | |
| JayF | TheJulia: it's not like one of those mellanox ones that does all the magic which makes it even sadder | 22:37 |
| JayF | TheJulia: afaict it's basically like a winmodem except it's a nic | 22:37 |
| JayF | or really, it probably executes that code on-chip, but like, it's loaded at boot | 22:37 |
| JayF | IDK, it's just obnoxious and makes it super difficult for things like our ramdisk to truly be minimal | 22:38 |
| JayF | time to make our own ironic ramdisk distro, where it somehow detects and pulls network firmware over the network using like, ipxe-snp or something like that | 22:38 |
| JayF | I'm pretty sure that's gtk-bananas | 22:38 |
| TheJulia | so! | 22:38 |
| TheJulia | trammel hudson at lower layer labs has an awesome demo | 22:39 |
| TheJulia | in... UEFI | 22:39 |
| TheJulia | no drivers, no firmware | 22:39 |
| TheJulia | well | 22:39 |
| TheJulia | we would be able to literally rm -r -f all of the firmware files | 22:39 |
| TheJulia | and then... if needed kexec into our full ramdisk | 22:39 |
| JayF | Do you have a link to anything written about that? | 22:40 |
| JayF | My only worry with a pattern like that would be the edges ... e.g. vlan/bonding support and the like | 22:40 |
| JayF | and yet another set of edges to work around | 22:40 |
| * TheJulia looks for the tweets | 22:40 | |
| JayF | TheJulia: oh, I did some interesting science today | 22:41 |
| JayF | guess what github copilot writes if you feed it `def erase_block_device` | 22:41 |
| JayF | I was afraid it'd spit back our code at us, but instead it's the most cursed implementation ever | 22:41 |
| TheJulia | hmm, I'll need to message him, looks like nuked the tweets | 22:41 |
| TheJulia | JayF: how... cursed? | 22:41 |
| JayF | https://gist.github.com/jayofdoom/db94a89d96d5f781e134532f371ded72 | 22:42 |
| * TheJulia twitches | 22:42 | |
| JayF | that's 100% safe, not dangerous at all | 22:42 |
| JayF | and certainly clears off all your [adjusts glasses] megabyte of information | 22:42 |
| TheJulia | ye olde issue of pam requiring cracklib through pw quality... and cockpit requiring it too | 22:45 |
| TheJulia | yeah, we get webkit now | 22:50 |
| TheJulia | rpm -e webkit2gtk3-jsc libproxy-webkitgtk4 saves like 26MB | 22:51 |
| TheJulia | does... ipa... *really* need polkit?!? | 22:53 |
| JayF | Almost certainly not | 22:53 |
| JayF | We don't even create nonroot users in the ramdisk iirc | 22:53 |
| TheJulia | 30+ mb more | 22:54 |
| TheJulia | rpm -e polkit polkit-libs PackageKit polkit-pkla-compat <--- now down to 155MB of libs | 22:55 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent-builder master: Remove some extra Centos Packages https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/847417 | 23:08 |
| TheJulia | ugh, v6 is failing again | 23:09 |
| opendevreview | Merged openstack/metalsmith master: metalsmith list handle missing port https://review.opendev.org/c/openstack/metalsmith/+/846067 | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!