| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 984486: Skip local loop device creation for no-final-image builds https://review.opendev.org/c/openstack/diskimage-builder/+/984486 | 03:48 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984800: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 06:07 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984800: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 06:08 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984804: Install pip-check-updates only in jobs requiring that https://review.opendev.org/c/openstack/project-config/+/984804 | 06:08 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: | 06:09 | |
| - [openstack/project-config] 984800: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | ||
| - [openstack/project-config] 984804: Install pip-check-updates only in jobs requiring that https://review.opendev.org/c/openstack/project-config/+/984804 | ||
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984804: Install pip-check-updates only in jobs requiring that https://review.opendev.org/c/openstack/project-config/+/984804 | 06:49 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [openstack/project-config] 984800: Follow up for I170bdb3ffc89bc3307a08da7cd4c7f2793a4e491 https://review.opendev.org/c/openstack/project-config/+/984800 | 07:32 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [openstack/project-config] 984903: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:35 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [openstack/project-config] 984903: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:36 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [openstack/project-config] 984903: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:37 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [openstack/project-config] 984903: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:40 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [openstack/project-config] 984903: DNM: Testing https://review.opendev.org/c/openstack/project-config/+/984903 | 08:41 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984908: Fix pip-check-updates target https://review.opendev.org/c/openstack/project-config/+/984908 | 09:17 | |
| @mnasiadka:matrix.org | static03 load avg: 1824.81, 1866.17, 1766.78 | 09:30 |
|---|---|---|
| @mnasiadka:matrix.org | cpu load was minimal, but load avg was that huge - I'd suppose maybe tarballs download? | 09:41 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [openstack/project-config] 984908: Fix pip-check-updates target https://review.opendev.org/c/openstack/project-config/+/984908 | 09:47 | |
| @fungicide:matrix.org | tarballs downloads wouldn't generally require much load | 11:48 |
| @fungicide:matrix.org | er, generate much load | 11:48 |
| @fungicide:matrix.org | load average on static03 at the moment seems to be ~3.0 | 11:49 |
| @fungicide:matrix.org | and now it's down below 1.0, so whatever was going on a couple hours ago seems to have not been long-lived? | 11:54 |
| @fungicide:matrix.org | interesting that crawlers are still hitting static04 pretty hard even after a couple of days. ones that you wouldn't expect, like `meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)` | 11:56 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/gerritlib] 983476: Update the Gerrit test images https://review.opendev.org/c/opendev/gerritlib/+/983476 | 12:01 | |
| @mnasiadka:matrix.org | fungi: I find that load avg a bit bizarre, given iostat and top showed nearly nothing happening | 12:15 |
| @fungicide:matrix.org | yeah, after seeing the discussion around the same time in #openstack-infra irc, i can see why massively parallel tarball downloads might have been a reasonable guess | 12:17 |
| @mnasiadka:matrix.org | Given that e.g. periodic Kolla build jobs downloads all tarballs - maybe isolating it on a dedicated server or moving data to swift/s3 is a reasonable improvement if that happens often | 12:18 |
| @mnasiadka:matrix.org | We could also think of using local Zuul cached checkouts, but the amount of required projects in job definition would be an overkill | 12:19 |
| @fungicide:matrix.org | would it be any worse than the required-projects list for devstack/tempest jobs? | 12:21 |
| @fungicide:matrix.org | like order of magnitude worse? | 12:21 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/jeepyb] 983482: Test Gerrit 3.11, 3.12, and 3.13 with Gerritlib https://review.opendev.org/c/opendev/jeepyb/+/983482 | 12:21 | |
| @fungicide:matrix.org | even if you just used the existing git cache baked into test nodes without listing them as required-projects, simply fetching the latest tags in those would be a lot faster and less bandwidth than downloading tarballs for everything | 12:24 |
| @fungicide:matrix.org | usually those repos are no more than a couple of days stale, so not much to actually fetch to catch them up anyway | 12:26 |
| @mnasiadka:matrix.org | Actually maybe it wouldn't be that bad, I'll give it a go after we branch 2026.1 | 12:30 |
| -@gerrit:opendev.org- Sylvain Bauza proposed: [openstack/project-config] 984922: Add openstack/agentic-workflows project https://review.opendev.org/c/openstack/project-config/+/984922 | 12:37 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984930: kolla: fix pcu packages and files https://review.opendev.org/c/openstack/project-config/+/984930 | 13:00 | |
| -@gerrit:opendev.org- Sylvain Bauza proposed: [openstack/project-config] 984922: Add openstack/agentic-workflows project https://review.opendev.org/c/openstack/project-config/+/984922 | 13:08 | |
| @clarkb:matrix.org | infra-root just a heads up that 983482 did build a new image but against the same Gerrit tag so we shouldn't need to "upgrade" the running container | 14:54 |
| @clarkb:matrix.org | mnasiadka: fungi re load that is similar to what I was seeing yesterday though a bit higher. The server itself seemed fine. I wonder if that is due to afs waiting on data so we get a bunch of parked threads. its not disk io because its waiting on network traffic? | 14:55 |
| @fungicide:matrix.org | oh could be, yeah | 14:57 |
| @mnasiadka:matrix.org | Oh, that probably explains | 14:58 |
| @mnasiadka:matrix.org | But that might be caused by tarballs traffic I guess | 14:59 |
| @clarkb:matrix.org | yes, I suspect that of all the data flowing through that server tarballs represent the largest files. And that likely means they haev an outsized impact on the local afs caching too | 14:59 |
| @clarkb:matrix.org | its possible we might want to profile the afs cache to see if we need to make it bigger or something | 14:59 |
| @clarkb:matrix.org | fungi: mnasiadka any reason to not proceed with https://review.opendev.org/c/opendev/system-config/+/984702/ this morning to remove mirror02.dfw.rax from our inventory? | 15:00 |
| @clarkb:matrix.org | I got the first two chagnes in yesterday before I had to pop out so this is next on the list | 15:00 |
| @mnasiadka:matrix.org | No reasons, that one I double checked on every angle if that's not used - given I was not the one setting up mirror03 :) | 15:01 |
| @clarkb:matrix.org | great thank you for double checking it. I'll go ahead and approve it now | 15:02 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [openstack/project-config] 984930: pcu: run in project directory https://review.opendev.org/c/openstack/project-config/+/984930 | 15:11 | |
| @mnasiadka:matrix.org | Clark: I assume DNS change is next - and then I could shut down the old mirror instances and if no drama happens - remove them tomorrow or next week? | 15:18 |
| @clarkb:matrix.org | Note that https://review.opendev.org/c/opendev/zuul-providers/+/984866/ failed on what looks like swift connectivity. I've rechecked it to see if that issue persists | 15:18 |
| @clarkb:matrix.org | mnasiadka: yup. I won't be around tomorrow, but happy to keep helping next week as necessary | 15:19 |
| @mnasiadka:matrix.org | I've seen that, and two jobs I checked failed on sjc3 connectivity | 15:19 |
| @clarkb:matrix.org | that is also the region where the mirror went down. I wonder if they are having issues with networking or similar | 15:22 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 984141: Add prepare-repos role https://review.opendev.org/c/zuul/zuul-jobs/+/984141 | 15:34 | |
| @mnasiadka:matrix.org | fungi: I found some issues with the pip-check-updates target for proposal job that I proposed (that sounds a bit funny) - would you be open to have a look at https://review.opendev.org/c/openstack/project-config/+/984930 ? And hopefully that's the only fix I need. | 15:43 |
| @fungicide:matrix.org | yeah, i was sort of watching and waiting for that to stabilize | 15:45 |
| @clarkb:matrix.org | We upgraded Gerrit on Sunday. It is now Thursday. Is it time to drop the Gerrit 3.11 image builds and tests: https://review.opendev.org/c/opendev/system-config/+/984468 ? I think the worst issue we've found so far was the handling of ssh keys and that appeared solvable by removing the -sk pubkey material from the user account | 15:47 |
| @mnasiadka:matrix.org | Clark: https://github.com/apache/mina-sshd/issues/433 has a mention of -sk (fido) keys being problematic - so I doubt it worked before | 15:50 |
| @fungicide:matrix.org | possible it was ignoring those before | 15:51 |
| @mnasiadka:matrix.org | Well there's https://github.com/apache/mina-sshd/commit/bca964c7265c27e415d949e6a7764899012647f4 | 15:52 |
| @mnasiadka:matrix.org | So it's supported from 2.16 | 15:52 |
| @mnasiadka:matrix.org | And OpenDev runs 2.15 | 15:52 |
| @mnasiadka:matrix.org | So anyway - what fungi says probably is right | 15:52 |
| @fungicide:matrix.org | that's the only way i can reconcile the claimed regression in behavior after the upgrade anyway | 15:54 |
| @fungicide:matrix.org | since it sounded like it hadn't been preventing the user from authenticating previously | 15:54 |
| @mnasiadka:matrix.org | Gerrit 3.13 comes with Apache (MINA) sshd 2.16 - so it should work then | 15:54 |
| @mnasiadka:matrix.org | fungi: are you fine with me merging 984930 so I/we can observe tomorrow's periodics if that works? | 15:56 |
| @fungicide:matrix.org | mnasiadka: sure, done | 15:58 |
| @clarkb:matrix.org | yup I didn't check the code I think is at fault, but I suspect they changed how they look up keys and they didn't short circuit before | 16:00 |
| @clarkb:matrix.org | the key lookup happens on the gerrit side fwiw | 16:00 |
| @clarkb:matrix.org | then hands it to mina which raises exception I think | 16:00 |
| @clarkb:matrix.org | I think i have a Gerrit CLA signed as of like yesterday or day before so maybe I'll actually try to fix that | 16:01 |
| @mnasiadka:matrix.org | seems ubuntu mirrors had a hiccup and mirror02.dfw removal failed in gate - what's the drill - recheck or re-enqueue? | 16:01 |
| @clarkb:matrix.org | mnasiadka: typically we would recheck unless there is a time crunch that would make going faster important | 16:03 |
| @mnasiadka:matrix.org | ok, that's what I thought | 16:03 |
| @fungicide:matrix.org | whatever's the lowest effort possible, really | 16:05 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [openstack/project-config] 984930: pcu: run in project directory https://review.opendev.org/c/openstack/project-config/+/984930 | 16:09 | |
| @mnasiadka:matrix.org | Ok then, seems it works https://review.opendev.org/c/openstack/kolla-ansible/+/985003 | 16:19 |
| @fungicide:matrix.org | yay! | 16:19 |
| @fungicide:matrix.org | iterating on jobs that can't run speculatively is always a pain. it's hard to imagine we used to run all our jobs that way | 16:20 |
| @fungicide:matrix.org | i think i repressed those memories | 16:20 |
| @mnasiadka:matrix.org | Well, I think that could be reworked to run speculatively, which would ease management of that. | 16:22 |
| @mnasiadka:matrix.org | But I think I have more interesting things on the radar | 16:22 |
| @mnasiadka:matrix.org | (for now) | 16:22 |
| @clarkb:matrix.org | Gerrit H2 v2 cache files continue to grow, but remain within the bounds of what we saw with H2 v1. | 16:33 |
| @clarkb:matrix.org | (I'm just noting that as I'm trying to check daily in order to catch any unexpected growth early) | 16:33 |
| @fungicide:matrix.org | mnasiadka: parts of it could be run speculatively, though access to the gerrit account authentication credentials for proposing the resulting change would still need to be handled in a config repo | 16:36 |
| @fungicide:matrix.org | i agree with the complexity there, it may make sense to refactor all the earlier steps out into an unprivileged playbook that can be exercised without actually pushing any changes to review | 16:37 |
| @fungicide:matrix.org | and then only the playbook that runs `git review` at the end has to live in project-config | 16:37 |
| @clarkb:matrix.org | oh one thing I wanted to check out of curiousity with Gerrit 3.12 is if we can confirm that PQC ssh algorithms are used. I guess I should rerun some ssh -vvv and see if I can tell | 16:51 |
| @fungicide:matrix.org | seems to be yes, i was previously getting warnings from my openssh client | 16:57 |
| @fungicide:matrix.org | now i no longer get warnings | 16:57 |
| @mnasiadka:matrix.org | fungi: I wasn't thinking of moving anything outside of that repository, just reworking it in a way, that unprivileged playbook would prepare everything and we just add git review in the privileged run in periodic pipeline | 16:58 |
| @clarkb:matrix.org | huh my openssh client is pretty new and I don't recall warnings. Maybe that is a packager configuration option? | 16:58 |
| @fungicide:matrix.org | `OpenSSH_10.2p1 Debian-6, OpenSSL 3.6.1 27 Jan 2026` | 17:00 |
| @fungicide:matrix.org | it used to report this every time i connected to gerrit: `WARNING: connection is not using a post-quantum key exchange algorithm. This session may be vulnerable to "store now, decrypt later" attacks. The server may need to be upgraded. See https://openssh.com/pq.html` | 17:00 |
| @fungicide:matrix.org | and now it no longer does, since the upgrade on sunday | 17:00 |
| @clarkb:matrix.org | I have OpenSSH_10.2p1 as well but never had warnings so ya maybe a compile time option or some distro specific config | 17:01 |
| @fungicide:matrix.org | to be fair, if i were king i would have inverted that warning to something like "your connection is using a supposed post-quantum secure algorithm that may be vulnerable to access by the government agencies pushing these new untested algorithms through baseless and scientifically unsound fearmongering" | 17:05 |
| @clarkb:matrix.org | looking at ssh -vvv output I see my client offer mlkem as first on the list. Gerrit responds with a couple of sntrup algorithms first but then offers a couple mlkem algorithms one of which matches what my client said it can do. I think sntrup algorithms are older? But its good that the server supports them as they are probably better than nothing for older clients | 17:06 |
| @clarkb:matrix.org | anyway I think this confirms what you've already surmised due to the change in your warning behavior | 17:06 |
| @fungicide:matrix.org | but i know my opinions are unpopular outside cliques of traditional cryptographers and cryptanalysts | 17:06 |
| @clarkb:matrix.org | another interesting thing I notice is taht `ssh -Q kex` is a superset of what ssh -vvv seems to indicate it actually offers to the server | 17:08 |
| @clarkb:matrix.org | so it must be using some additional info to trim the list down to what is more likely to be acceptable | 17:08 |
| @clarkb:matrix.org | But cool now Gerrit is less of a concern when it comes to this stuff. Next problem: sha1 and Gerrit not supporting sha256 yet | 17:09 |
| @clarkb:matrix.org | but one thing at a time | 17:09 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 984468: Drop Gerrit 3.11 jobs https://review.opendev.org/c/opendev/system-config/+/984468 | 17:12 | |
| @clarkb:matrix.org | but it is worth calling out that Git 3.0 is expected at the end of the year and that will change the default branch to main and be sha256 by default as well iirc | 17:12 |
| @fungicide:matrix.org | my understanding is that it will create repositories with sha2-256 indices and "main" as the default branch name, but it will still interact fine with repositories that were not created in that manner | 17:14 |
| @fungicide:matrix.org | if gerrit doesn't support it by then, we may run into problems with new project creation importing an external repository that's using the new commit ids | 17:16 |
| -@gerrit:opendev.org- Zuul merged on behalf of Sylvain Bauza: [openstack/project-config] 984922: Add openstack/agentic-workflows project https://review.opendev.org/c/openstack/project-config/+/984922 | 17:18 | |
| @clarkb:matrix.org | fungi: yes it should continue to operate just fine against existing repo aiui. But there is likely work to be done on our side to ensure we integrate with Gerrit and gitea properly | 17:36 |
| @clarkb:matrix.org | And then dealing with whatever end user fallout occurs | 17:36 |
| @fungicide:matrix.org | right | 17:37 |
| @clarkb:matrix.org | and then in theory we'll need to "upgrade" the existing repos we host? | 17:43 |
| @clarkb:matrix.org | which is basically a full conversion and force push I think | 17:44 |
| @fungicide:matrix.org | depends. it's unclear to me how backward-compatible the new repository format is with old clients. i know new clients are meant to handle the old servers/repositories seamlessly | 17:46 |
| @clarkb:matrix.org | it isn't aiui | 17:47 |
| @clarkb:matrix.org | once yuo're sha256 in the repo you need a sha256 capable client | 17:47 |
| @clarkb:matrix.org | you can't have both sha1 and sha256 side by side its all or nothing one way or the other which means the client side must understand the new stuff if you convert I think | 17:47 |
| @fungicide:matrix.org | yeah, so my guess is that we need to keep our repositories on the old format for a while even after gerrit supports the new format | 17:47 |
| @clarkb:matrix.org | ya its possible. I wonder how old of a git can handle sha256 | 17:47 |
| -@gerrit:opendev.org- Sylvain Bauza proposed: [openstack/project-config] 985017: Add Zuul jobs for openstack/agentic-workflows https://review.opendev.org/c/openstack/project-config/+/985017 | 17:52 | |
| @fungicide:matrix.org | i had seen previous discussion about storing both hashes side-by-side and having newer servers generate the new hash format objects when older clients push with the old ones, but i'm not finding any indication that plan ever got implemented | 17:52 |
| @fungicide:matrix.org | looks like github users are becoming increasingly antsy about getting support for the new format: https://github.com/orgs/community/discussions/12490 | 17:56 |
| @fungicide:matrix.org | the most recent comment on that discussion indicates gitea's ready to go, at least | 17:57 |
| @fungicide:matrix.org | but not yet jgit per https://github.com/eclipse-jgit/jgit/issues/73 | 17:58 |
| @clarkb:matrix.org | yes last I checked jgit is likely to be our biggest obstacle | 17:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Sylvain Bauza: [openstack/project-config] 985017: Add Zuul jobs for openstack/agentic-workflows https://review.opendev.org/c/openstack/project-config/+/985017 | 18:08 | |
| @fungicide:matrix.org | still seeing a bunch of docs.openstack.org requests on static04 from meta-externalagent and baidu nabar | 18:18 |
| @fungicide:matrix.org | i'm tempted to stop apache on the server at this point to get them to reevalaute their cached dns responses | 18:19 |
| @clarkb:matrix.org | No objection from me | 18:29 |
| @fungicide:matrix.org | okay, done | 18:34 |
| @fungicide:matrix.org | if we don't see any issues then maybe some time next week we can tear down static04 | 18:35 |
| @clarkb:matrix.org | sounds good to me | 18:43 |
| @fungicide:matrix.org | `E: Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/d/distro-info/python3-distro-info_1.1ubuntu0.2_all.deb Cannot initiate the connection to us.archive.ubuntu.com:80 [...] Network is unreachable` | 18:49 |
| @fungicide:matrix.org | looks like it's still ongoing | 18:49 |
| @fungicide:matrix.org | ran in raxflex-iad3 for reference | 18:50 |
| @clarkb:matrix.org | I can currently reach that name over https I suspect that a name like that is made up of many backends load balanced but I wonder if we should be using https if http is probelmatic now? | 18:51 |
| @clarkb:matrix.org | I guess I don't have any evidence that https is more reliable at this point, but consider http failed maybe it is worth investigating or just using? | 18:52 |
| @fungicide:matrix.org | from my from raxflex-iad3 it seems to be resolving to 91.189.91.82, 91.189.91.81, 91.189.91.83, and 2620:2d:4002:1::103, 2620:2d:4002:1::102, 2620:2d:4002:1::101 as round-robin records | 18:54 |
| @fungicide:matrix.org | it does indeed look like http requests to those time out, while https are working | 18:59 |
| @fungicide:matrix.org | okay, http requests are going through just hitting a lengthy delay | 19:00 |
| @clarkb:matrix.org | ya so this may flip flop when the bots change protocols | 19:00 |
| @clarkb:matrix.org | and we won't actually be better off by changing | 19:00 |
| @fungicide:matrix.org | testing from the iad3 mirror, http://91.189.91.82/ takes me 11 seconds while https://91.189.91.82/ returns in 0.1 seconds | 19:02 |
| @fungicide:matrix.org | i agree this is probably more fallout from out of control crawlers running the whole of the web into the ground | 19:03 |
| @fungicide:matrix.org | i'm going to go grab early dinner, then check back in | 19:05 |
| @clarkb:matrix.org | ya I'm about to eat lunch myself. Also currently getting laptop sorted out with new email connectivity. Something I need to do before traveling in a week and a half | 19:06 |
| @clarkb:matrix.org | fungi: ^ that reminds me that any syncing we want to do of infra root email needs to be done before ~wednesday? Not sure when exactly the access goes away | 19:06 |
| -@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 981925: Allow OpenStack-Ansible Cores to change WIP state https://review.opendev.org/c/openstack/project-config/+/981925 | 19:49 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Stephen Finucane: [openstack/project-config] 983728: Add GitHub mirroring for devstack-plugin-prometheus https://review.opendev.org/c/openstack/project-config/+/983728 | 19:49 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 984702: Remove mirror02.dfw.rax from configuration https://review.opendev.org/c/opendev/system-config/+/984702 | 20:08 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 984738: Remove mirror03.gra1, mirror02.ord and mirror02.dfw from DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/984738 | 20:26 | |
| @fungicide:matrix.org | Clark: oh, thanks, i can try to knock it out tomorrow through the weekend | 20:40 |
| @clarkb:matrix.org | I've finally got around to checking cloud providers for leaked images. I think openmetal and ovh look mostly ok. Then both rax classic and flex have some leaks. Flex far less than classic and the images appear to be "queued" not "active" there so I think the impacts are smaller. In classic we even appear to have images from when nodepool was around | 21:50 |
| @clarkb:matrix.org | I think the next step is to try and delete these images and see if they can be deleted. I'll probably start with flex since there are fewer there | 21:50 |
| @clarkb:matrix.org | but this may end up being a good ptg week cleanup | 21:51 |
| @clarkb:matrix.org | ok the images did clear out of flex without issue. So flex is sorted now as well. I think only rax classic is a concern | 21:56 |
| @clarkb:matrix.org | I suspect that this means that zuul launcher didn't actually know about those images in flex (they were all queued so maybe failed uploads the launcher thought went away) | 21:57 |
| @jim:acmegating.com | any idea of the age? | 21:57 |
| @jim:acmegating.com | wonder if it was from early launcher days -- especially being in flex | 21:57 |
| @clarkb:matrix.org | oh shoot no. The suffixes are uuids now and I didn't bother to show all their properties I just did a listing | 21:58 |
| @jim:acmegating.com | it's okay... i almost think maybe this is a good chance to just reset to zero and then check again later | 21:58 |
| @clarkb:matrix.org | ++ | 21:58 |
| @clarkb:matrix.org | corvus: all of the -17xyzabc sequential like suffixes are from nodepool right? | 21:59 |
| @clarkb:matrix.org | so they can be deleted even for current images like noble | 22:00 |
| @jim:acmegating.com | Clark: should be -- maybe check the metadata on those? see if the tags look like nodepool tags? or check the dates | 22:04 |
| @clarkb:matrix.org | all good ideas thanks | 22:06 |
| @clarkb:matrix.org | spot checking one of them it does have nodepool properties and was built in july so I think in addition to bionic deletions we can also attempt to delete all of these old nodepool images | 22:07 |
| @clarkb:matrix.org | I was able to delete the noble image I checked in rax ord as well. So I'm hopeful that the majority will simply cleanup if we ask to delete them | 22:09 |
| @clarkb:matrix.org | I've put a note on my todo list to go through and do this properly for all of rax classic. But I need to pop out and get some errands done before tomorrow | 22:12 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!