| @mnasiadka:matrix.org | There was a stale lock in debian reprepro job and the jobs did not run for two days - cleared it out and running the sync manually with foreground output (just to check everything is all right) | 09:37 |
|---|---|---|
| @noonedeadpunk:matrix.org | Hey folks! I have stumbled across mirrors for ceph repos: https://review.opendev.org/c/opendev/system-config/+/1005081 | 11:13 |
| @noonedeadpunk:matrix.org | Obviously, there's no AFS path existing, but looking at existing role I wonder if there's an automated way to create them, or they should be pre-created somehow? | 11:13 |
| @clarkb:matrix.org | Dmitriy Rabotyagov: the afs management is largely manual. It makes mirror setups like cephs particularly annoying | 15:34 |
| @clarkb:matrix.org | most mirrors we can set up once and they will automatically mirror new content over time because its all one mirror. Ceph insists on a new mirror for every release | 15:34 |
| @clarkb:matrix.org | we could possibly refactor that to be a single top level volume or something along those lines but that would be a departure from how we've set up reprepro everywhere else | 15:35 |
| @noonedeadpunk:matrix.org | the problem is we're having timeouts for last week or so, as it takes ~30mins to just apt install from thee repos | 15:35 |
| @noonedeadpunk:matrix.org | Here's the log from one of recent jobs: | 15:37 |
| https://zuul.opendev.org/t/openstack/build/f8c9c635ac4b4ec28fc40eac889a6820/log/job-output.txt#14920-14921 | ||
| and ceph-client installation, which is tiny, takes around 5 minutes: | ||
| https://zuul.opendev.org/t/openstack/build/f8c9c635ac4b4ec28fc40eac889a6820/log/job-output.txt#17453-17454 | ||
| @noonedeadpunk:matrix.org | I am really open for proposals here | 15:38 |
| @clarkb:matrix.org | I'm told everyone is expected to use the ceph management tool which is container based now? I don't know how that relates to client tooling though | 15:38 |
| @clarkb:matrix.org | anyway I posted a review to your change. It is missing critical config management to do anything with those reprepro config files | 15:40 |
| @noonedeadpunk:matrix.org | do we have a registry to build containers periodically and publish there? would it be better solution? | 15:40 |
| @clarkb:matrix.org | we use quay.io as does zuul | 15:40 |
| @clarkb:matrix.org | but I think the ceph container stuff is preexisting and you just need to fetch them? | 15:41 |
| @noonedeadpunk:matrix.org | they're running CentOS Stream afaik, which is not what I'd consider doing even in CI... | 15:41 |
| @noonedeadpunk:matrix.org | I got the point about symlinks, but should I drop cron for these old versions? what do you think? | 15:45 |
| @clarkb:matrix.org | yes, cron should also ideally be removed then an admin will have to followup with cleanups on the afs side of things | 15:47 |
| @clarkb:matrix.org | Dmitriy Rabotyagov: note teh comment at the top level was the primary concern | 15:48 |
| -@gerrit:opendev.org- Dmitriy Rabotyagov proposed: [opendev/system-config] 1005081: Update mirrored Ceph versions https://review.opendev.org/c/opendev/system-config/+/1005081 | 15:52 | |
| -@gerrit:opendev.org- Dmitriy Rabotyagov proposed: [opendev/system-config] 1005081: Update mirrored Ceph versions https://review.opendev.org/c/opendev/system-config/+/1005081 | 15:55 | |
| @clarkb:matrix.org | to followup on my earlier note about maybe being able to use a single afs volume for all ceph mirrors: the reason we don't do that today is the reprepro script wants to vos release the volume when the mirror updates. This ensures that each mirror is "updated" at the same time with working data after a reprepro run (we don't sync half working data etc) | 16:23 |
| @clarkb:matrix.org | so changing that would require a bit of thinking and maybe this is the correct setup as is given that functionality | 16:23 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/zone-opendev.org] 1005618: Remove backup01 and backup02 ca-ymq-1.vexxhost records https://review.opendev.org/c/opendev/zone-opendev.org/+/1005618 | 18:37 | |
| @clarkb:matrix.org | infra-root ^ that should be a straightforward cleanup of the work to wind down the old vexxhost backup server that I was doing before the summit | 18:37 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/zone-opendev.org] 1005618: Remove backup01 and backup02 ca-ymq-1.vexxhost records https://review.opendev.org/c/opendev/zone-opendev.org/+/1005618 | 19:18 | |
| @clarkb:matrix.org | I'm going to put together the meeting agenda for tomorrow shortly. I was largely out last week paying attention to the summit event, is there anything that needs to be added to the agenda? | 20:29 |
| @jim:acmegating.com | should we try to about image uploads? | 20:35 |
| @jim:acmegating.com | i feel like we identified a potential problem but i'm not sure it's currently being worked on, and it could escalate | 20:36 |
| @jim:acmegating.com | * should we try to talk about image uploads? | 20:36 |
| @clarkb:matrix.org | ++ I'll add it | 20:56 |
| @clarkb:matrix.org | I'll drop the zuul executor ooms stuff as I think that has been fully resolved at this point | 20:57 |
| @jim:acmegating.com | ++ | 20:57 |
| @clarkb:matrix.org | I removed the anubis topic as well as we seem to be in a happy staedy state there. I'm also removing the zuul alien errors item since its on the backlog and both airship and starlingx have fixed the majority of their errors | 21:06 |
| @clarkb:matrix.org | ok first draft edits are in | 21:08 |
| @clarkb:matrix.org | The next anubis release will ship a rust wasm challenge solver | 21:10 |
| @clarkb:matrix.org | this changes to argon2 rather than sha256 which should shift from cpu to memory as the resource required to solve it? | 21:12 |
| @clarkb:matrix.org | anyway something to keep in mind when we upgrade as the new setup is opt in. May be wortwhile to opt into it early so that any issues are found soon | 21:12 |
| @jim:acmegating.com | Clark: aren't the ai farms the only ones that can afford memory now? | 21:15 |
| @clarkb:matrix.org | corvus: I think the idea is that memory is so precious now that this is a better hurdle? But yes | 21:18 |
| @clarkb:matrix.org | theoretically that memory is more useful for training and inference though | 21:18 |
| @jim:acmegating.com | well maybe i'll learn to close my tabs | 21:19 |
| @jim:acmegating.com | not likely | 21:19 |
| @clarkb:matrix.org | I closed about 200 yseterday | 21:19 |
| @jim:acmegating.com | so like one window? | 21:19 |
| @clarkb:matrix.org | something like that | 21:19 |
| -@gerrit:opendev.org- Anil Shashikumar Belur proposed: [opendev/system-config] 1005631: Upgrade gitea to 1.27.3 https://review.opendev.org/c/opendev/system-config/+/1005631 | 21:57 | |
| @clarkb:matrix.org | Anil Belur: re ^ I think the openinfra foundation (https://openinfra.org/legal/ai-policy) suggests that we use `assisted-by` or `generated-by` rather than `co-authored-by` in commit messages | 21:59 |
| @clarkb:matrix.org | I'm not sure if matters much here, but being consistent is probably a good idea if we can be | 21:59 |
| -@gerrit:opendev.org- Anil Shashikumar Belur proposed: [opendev/system-config] 1005631: Upgrade gitea to 1.27.3 https://review.opendev.org/c/opendev/system-config/+/1005631 | 22:01 | |
| @clarkb:matrix.org | Anil Belur: Not sure if you've seen them before but the screenshots under artifacts for ^ are a good thing to check on | 23:13 |
| @clarkb:matrix.org | and they look good to me so +2 | 23:14 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!