| corvus | clarkb: https://review.opendev.org/962146 | 00:18 |
|---|---|---|
| *** mrunge_ is now known as mrunge | 07:38 | |
| *** dmellado7 is now known as dmellado | 09:23 | |
| *** mtreinish_ is now known as mtreinish | 11:49 | |
| opendevreview | yatin proposed zuul/zuul-jobs master: [WIP] Make fips setup compatible to 10-stream https://review.opendev.org/c/zuul/zuul-jobs/+/961208 | 11:58 |
| mnasiadka | Just wondering about ^^ - wouldn’t it be easier to just build the fips enabled image? | 12:33 |
| *** ykarel_ is now known as ykarel | 12:35 | |
| ykarel | mnasiadka, i don't have full background on building fips images but there were some discussions atleast in tc channel as mentioned by sean-k-mooney https://meetings.opendev.org/irclogs/%23openstack-qa/%23openstack-qa.2025-09-16.log.html#openstack-qa.2025-09-16.log.html#t2025-09-16T11:21:28 | 12:40 |
| mnasiadka | ykarel: if that works and doesn’t prove problematic - sure, but I read the Fedora page about FIPS, and it seems the recommended way is to have a fips enabled image - and if we only need that for CentOS Stream 10, Rocky 10 and maybe Alma 10 in future - it’s not THAT many images to maintain. | 12:42 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [WIP] Make fips setup compatible to 10-stream https://review.opendev.org/c/zuul/zuul-jobs/+/961208 | 12:48 |
| ykarel | yes if thats done these custom roles will also not be needed | 12:48 |
| Clark[m] | mnasiadka: ykarel: I have a strong desire to not build fips specific images for opendev for two reasons. The first is due to the effort of building images. It is already rare all image builds reliably succeed. Every new image has an impact on that. Doubling our image count is much worse. And second because every image is tech debt that the opendev team typically ends up responsible for cleaning up when everyone else has stopped caring enough | 13:20 |
| Clark[m] | to be involved. Removing old test platforms has not been fun lately | 13:20 |
| Clark[m] | Also worth keeping in mind that food is a fairly specific use case and one that is tied to a specific set of government requirements for which there are equivalents from other governments. I don't want to triple out images for another competing standard | 13:22 |
| Clark[m] | Heh "food" yay auto complete. *fips | 13:22 |
| fungi | yeah, for an international collaboration, it seems sort of backwards to go out of our way to support compliance with a government/military standard for one country that represents only 16% of openstack project contributors and 12% of merged changes in the current release | 13:30 |
| fungi | the eu is several times more involved (literally) in developing openstack these days than all of north america | 13:31 |
| mnasiadka | Clark[m]: Actually I thought FIPS is more useful than it is, so agree it doesn’t make any sense ;-) | 13:42 |
| fungi | fips actually forbids a number of newer algorithms simply because the process for getting them added to the standard is so very, very slow | 13:50 |
| fungi | fips compliance doesn't help you run a more secure system, it helps you run a system that meets specific requirements for usa military and government use | 13:51 |
| fungi | it's the usa's "federal information processing standard" | 13:53 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [WIP] Make fips setup compatible to 10-stream https://review.opendev.org/c/zuul/zuul-jobs/+/961208 | 14:05 |
| *** diablo_rojo_phone_ is now known as diablo_rojo_phone | 14:19 | |
| *** diablo_rojo_phone is now known as Guest27275 | 14:20 | |
| *** masayukig_ is now known as masayukig | 14:20 | |
| *** mnasiadka_ is now known as mnasiadka | 14:26 | |
| *** mtreinish_ is now known as mtreinish | 14:26 | |
| *** TheJulia_ is now known as TheJulia | 14:26 | |
| *** prometheanfire is now known as Guest27314 | 14:28 | |
| *** hashar is now known as Guest27315 | 14:28 | |
| *** masayukig_ is now known as masayukig | 14:39 | |
| *** mtreinish_ is now known as mtreinish | 14:39 | |
| *** johnsom_ is now known as johnsom | 14:39 | |
| *** mnaser_ is now known as mnaser | 14:39 | |
| clarkb | as a heads up my ISP was not able to make it out yseterday afternoon and rescheduled to this morning | 14:55 |
| fungi | fun! | 14:59 |
| fungi | i'll be disappearing to run an errand and get an early dinner around 18:30-21:00ish myself | 15:00 |
| opendevreview | yatin proposed zuul/zuul-jobs master: [WIP] Make fips setup compatible to 10-stream https://review.opendev.org/c/zuul/zuul-jobs/+/961208 | 15:23 |
| opendevreview | Jan Gutter proposed zuul/zuul-jobs master: Fix up some EL10 compatibility https://review.opendev.org/c/zuul/zuul-jobs/+/962194 | 15:29 |
| *** cardoe_ is now known as cardoe | 15:32 | |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Initiate retirement of shade https://review.opendev.org/c/openstack/project-config/+/961522 | 15:53 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Retire shade https://review.opendev.org/c/openstack/project-config/+/961524 | 15:53 |
| *** gthiemon1e is now known as gthiemonge | 16:01 | |
| opendevreview | Merged openstack/project-config master: Initiate retirement of shade https://review.opendev.org/c/openstack/project-config/+/961522 | 16:08 |
| *** efoley_ is now known as efoley | 17:05 | |
| fungi | heading out for errands/dinner but will check back in probably around 21:00 utc | 18:33 |
| clarkb | enjoy! I'm still waiting on the isp. Things are happier this morning and they did do things remotely yesterday but still wanted to send someone out. Hopefully I get this resolved soon | 18:34 |
| Clark[m] | ISP is here and I'm on mobile data only for the moment | 20:07 |
| clarkb | ok I think I'm back now | 20:48 |
| corvus | looks like you are | 21:00 |
| clarkb | I disconnected for a short bit again to work on the rats nest of cables. The new ONT is POE which frees up a power outlet in the garage (yay!) but adds more devices to my corner of network doom | 21:09 |
| clarkb | they basically decided that my ONT is so old that its likely it was the source of my problems and they should just replace it so they did | 21:10 |
| clarkb | I need to do more organization work, but I'm reasonably happy with it for now | 21:12 |
| fungi | here's hoping it holds fast tomorrow morning | 21:39 |
| clarkb | I just did a half hour of successful pings to review (after doing cable reorg) with no loss so that is a good sign at least | 21:42 |
| clarkb | I am taking this as an opportunity to do some long over due local device organization and cleanup | 22:17 |
| fungi | cable management is necessary | 22:18 |
| clarkb | ya part of the problem is that my little network corner grew from a small router device + access point to a router + ap + switch to a router + ap + switch + fileserver to a router + ap + switch + fileserver + serverserver and recently I put a printer into the mix and then today there is a new poe injector for my WAN connection | 22:20 |
| clarkb | so I shutdown everything I could get away with and started redoing where things lived rather than them being in whatever space was convenient at the time | 22:20 |
| fungi | don't forget the ups | 22:24 |
| clarkb | oh ya that is on the floor collecting dust. But also because the poe injector is on the ups my home network shoudl still survive power outages for a couple of hours at least (I think the longest outage we've had was just under 4 hours and it made it that long) | 22:25 |
| fungi | i wall-mount all that stuff together to keep the mess out of view and also make it easier to work on | 22:27 |
| corvus | clarkb: the necessary changes for the launcher have merged and promote was successful; it should be okay to pull and restart the launchers, then configure flex in zuul-providers the way we want | 22:27 |
| clarkb | corvus: ++ considering I'm out tomorrow do we want to wait for me to do that Friday? Also do we want to remove the testing config or should we leave that for now to confirm things look good after a restart? | 22:28 |
| clarkb | though I guess I don't know how to lookup the list of label networks via the repl | 22:28 |
| corvus | clarkb: i am 99% confident in this fix so i feel like it's reasonable to confirm it in prod and then just revert if i'm wrong. | 22:29 |
| clarkb | I guess we can leave the clouds.yaml content as is for now (and in fact need to to avoid another outage). The first step is to restart launchers | 22:29 |
| clarkb | once launchers are restarted readd the network config in zuul provider config then remove the clouds.yaml and restart launchers again | 22:30 |
| corvus | ++ | 22:30 |
| clarkb | I can go ahead and and restart launchers now and get a revert of the zuul-provider config updates pushed | 22:30 |
| clarkb | I'll pull images first to make sure I've got the new stuff | 22:30 |
| corvus | that sounds good. if the launchers blow up today we can deal with that now. then you can do the other stuff with known good launchers on friday. | 22:31 |
| clarkb | ya | 22:31 |
| clarkb | zl01 is restarted. Once it looks like it is trying to boot things I'll do zl02 | 22:32 |
| clarkb | corvus: Exception: Unable to find flavor: gp.5.4.4 <- this is for sjc3 so I'm going to check that really quick before doing zl02 | 22:33 |
| clarkb | corvus: oh thats a me bug. I copied the dfw3 config when I made the fake sjc3 config | 22:36 |
| clarkb | corvus: it also appears to be in a tight loop trying to delete images that are in use in openmetal. It isn't clear to me if that is preventing it from proceeding to booting nodes. The logs is so full of these errors that its hard to tell if it is doing real work too or if this is a prestep that is stuck | 22:37 |
| clarkb | corvus: looking at grep -v ERROR I think it may not be doing any real work as a result of this | 22:38 |
| corvus | i'll take a look | 22:38 |
| corvus | yeah it's launching nodes | 22:39 |
| clarkb | corvus: is there an easy way to tell? | 22:40 |
| clarkb | also any concern about this filling the disk? | 22:40 |
| opendevreview | Clark Boylan proposed opendev/zuul-providers master: Fix test sjc3 region flavors https://review.opendev.org/c/opendev/zuul-providers/+/962233 | 22:41 |
| clarkb | that is the fix for the silly flavor issue | 22:41 |
| corvus | definitely a concern about it filling the disk, but that's not a new concern | 22:41 |
| corvus | 2025-09-24 22:38:51,142 DEBUG zuul.Launcher: [e: 4aaa4354e3dc44739ba3f71558423eaa] [req: 8c817375d1ba444e83edc53c2739dd84] Building node <OpenstackProviderNode uuid=2fde142880b8415a85f219d1d12d3852, label=ubuntu-focal, state=requested, provider=opendev.org%2Fopendev%2Fzuul-providers/rax-ord-main> | 22:41 |
| corvus | that log line tells us it's building nodes | 22:41 |
| corvus | it would be worth understanding why we still have nodes using an image from 10 days ago | 22:43 |
| corvus | https://zuul.opendev.org/t/zuul/provider/openmetal-iad3-main/image/ubuntu-noble | 22:43 |
| clarkb | ya I started looking into that and did confirm at least one of the images is from september 14 | 22:44 |
| corvus | 10b4689e46af4cc986dd587163e56e5c is one of the ones it's trying to delete | 22:44 |
| clarkb | corvus: do you think I should proceed with zl02 at this point? eg its safe given the issues I found appaer to be old er and also didn't stop startup ? | 22:44 |
| corvus | yep | 22:44 |
| clarkb | ok doing that now then I'll see that I can learn about the images in openmetal | 22:44 |
| corvus | clarkb: could be the gerrit test node -- https://zuul.opendev.org/t/openstack/nodes | 22:45 |
| clarkb | and whatever 144bc90c6f094faf81f627f84f980e6c is maybe | 22:46 |
| clarkb | I really dislike that glance can't figure out reference counting | 22:46 |
| corvus | well the gerrit test node is from 9 days ago, the other one is newer | 22:46 |
| clarkb | zl02 has been restarted | 22:47 |
| clarkb | on the new image verison | 22:47 |
| corvus | but yes, as a user i do not understand why i can't delete an image that was previously used to launch a server | 22:47 |
| clarkb | e77e2cc4-9966-484a-9c56-062e394e5bcd and 1f68c718-c68f-4613-b32b-4da34eb0192c appear to be the cloud side image ids | 22:48 |
| corvus | i haven't checked but i'd guess the other one is jammy since that's also a gerrit node in openmetal | 22:48 |
| clarkb | e77 is noble and 1f68 is jammy | 22:48 |
| clarkb | there is a good chance that gerrit is noble and bridge is jammy and its that pair of held nodes | 22:49 |
| clarkb | I don't mind delete those two and getting new held nodes if we think that is a good idea to avoid disk filling? though if this has been happenign for ~9 days then we're probably ok? | 22:49 |
| corvus | yeah, we're probably leveled out. :) | 22:51 |
| corvus | since i think our rotation is 7 days | 22:51 |
| opendevreview | Clark Boylan proposed opendev/zuul-providers master: Revert "Remove raxflex networks config" https://review.opendev.org/c/opendev/zuul-providers/+/962235 | 22:52 |
| clarkb | corvus: I wonder if we should slow down the delete requests though? | 22:53 |
| clarkb | corvus: also I guess you should let me know if 962233 is better off being a change that removes the test provider stuff entirely | 22:54 |
| clarkb | not sure how much value there is at this point | 22:54 |
| clarkb | confirmed np0ad4c944a1544 is booted on 1f68c718-c68f-4613-b32b-4da34eb0192c and npca82fb44ea454 is booted on e77e2cc4-9966-484a-9c56-062e394e5bcd so my held nodes are the cause of that error loop | 22:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Revert "Reapply "Select the network to use in raxflex"" https://review.opendev.org/c/opendev/system-config/+/962237 | 23:11 |
| clarkb | and that is the change to clean up clouds.yaml. I'll pcik this back up on Friday and am happy to babysit those changes and ensure we don't get duplicate networks again | 23:12 |
| *** iurygregory_ is now known as iurygregory | 23:41 | |
| clarkb | there is a docker hub outage right now. Just a heads up in case people wonder why jobs mgiht be failing | 23:44 |
| tonyb | Looking at refreshing, https://review.opendev.org/c/opendev/system-config/+/934937 do we really need to use the latest pip? These days isn't the distro packaging for pip+venv adequate? | 23:54 |
| tonyb | If we installed the distro pip+venv then we could update the calls into the pip module with the appropriate executables and virtualenv options, and maybe add a symlink? | 23:55 |
| tonyb | Like docker/podman we could make the jammy->noble transition the point at which things change | 23:56 |
| tonyb | I think that'd be neater than what I did in 934937 | 23:57 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!