| @clarkb:matrix.org | I'll haev to take a closer look tomorrow morning. Does the base-jobs change half depends on the zuul-jobs change? I guess if you try to use podman it does but can land as is with the current docker default? | 00:28 |
|---|---|---|
| @mordred:waterwanders.com | Clark: thanks! they're independent. the base-jobs one pretty much just doesn't install docker. (it also shifts the iptables rules into a when block, which looking at it again I really could have split into 2 changes and can if you prefer) the base-jobs one is important largely because ensure-docker does not work on resolute (or I think trixie) currently (on my todo list to fix) - and also just because the not doing it is old and outdated | 02:27 |
| @mordred:waterwanders.com | Clark: the zuul-jobs one is mostly cleanup. | 02:29 |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/base-jobs] 994048: Revert "Disable job log uploads to ovh swift" https://review.opendev.org/c/opendev/base-jobs/+/994048 | 05:58 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/base-jobs] 994048: Revert "Disable job log uploads to ovh swift" https://review.opendev.org/c/opendev/base-jobs/+/994048 | 06:01 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 984141: Add prepare-repos role https://review.opendev.org/c/zuul/zuul-jobs/+/984141 | 07:19 | |
| @harbott.osism.tech:regio.chat | usually I think we had set the base-test job to autotest the revert, which hasn't happened in this case, so maybe we'd want to split the revert into two parts? or are we confident enough to just go ahead and do live testing with ^^? | 06:10 |
| @harbott.osism.tech:regio.chat | fyi I added the topic "Dealing with alien zuul config errors in the openstack tenant" to the meeting agenda, but since I mostly can't make it to the meeting, I would actually prefer some async discussion here if possible. or maybe in the TC channel where I'll also mention this as it also relates to some general policy issues IMO | 11:03 |
| @fungicide:matrix.org | #status log Cleaned up stale /sigs content from the project.governance AFS volume now that it's shadowed by a redirect | 14:23 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 14:23 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Mohammed Naser: | 14:37 | |
| - [opendev/system-config] 980840: Add Prometheus monitoring service https://review.opendev.org/c/opendev/system-config/+/980840 | ||
| - [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | ||
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Mohammed Naser: | 14:39 | |
| - [opendev/system-config] 980840: Add Prometheus monitoring service https://review.opendev.org/c/opendev/system-config/+/980840 | ||
| - [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | ||
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 14:39 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 14:43 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Mohammed Naser: [opendev/system-config] 980994: Add node_exporter role for stats reporting https://review.opendev.org/c/opendev/system-config/+/980994 | 14:43 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 14:43 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed wip: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 14:44 | |
| -@gerrit:opendev.org- Michal Nasiadka marked as active: [opendev/system-config] 988310: Add GrepTimeDB long term storage for Prometheus https://review.opendev.org/c/opendev/system-config/+/988310 | 14:46 | |
| @clarkb:matrix.org | mordred: I did have a question on https://review.opendev.org/c/zuul/zuul-jobs/+/990826 | 16:22 |
| @fungicide:matrix.org | okay, one hiccup with adding wiki to the new backup server... | 16:24 |
| @fungicide:matrix.org | `userauth_pubkey: signature algorithm ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]` | 16:24 |
| @fungicide:matrix.org | (that's from auth.log on the new server) | 16:24 |
| @fungicide:matrix.org | there are obvious solutions, just need to decide what's acceptable | 16:25 |
| @clarkb:matrix.org | can we allow that only for wiki? | 16:31 |
| @fungicide:matrix.org | maybe, i'll need to look into that possibility. i've done exclusions in the other direction (new clients authenticating to older servers) so this will require a bit of reading | 16:32 |
| @mordred:waterwanders.com | Clark: yeah ... honestly, I think the construction in base-jobs is the thing that makes more sense to me. But maybe the construct in zuul-jobs makes sense for people just using zuul-jobs roles for places we don't have matching base jobs? My main motivation for that particular bit was just parity with https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-docker/tasks/main.yaml#L85 - so that the two were at general parity. | 16:33 |
| @fungicide:matrix.org | oh, though the wiki server *does* have a `/root/.ssh/id_borg_backup_ed25519` so maybe it's just failing to use that? | 16:33 |
| @mordred:waterwanders.com | that said - I actually think the zuul-jobs role with "use_buildset_registry" would make even more sense to my brain if use_buildset_registry defaulted to true. :) because, I don't know why in a zuul job you'd _ever_ want to install docker/podman but _not_ configure the buildset registry | 16:34 |
| @mordred:waterwanders.com | but - it defaults to false in the docker role, so, you know, probably too much inertia there | 16:34 |
| @clarkb:matrix.org | mordred: oh I see we're already doing this is ensure-docker then aprity makes sense | 16:34 |
| @clarkb:matrix.org | fungi: oh that may be | 16:35 |
| @mordred:waterwanders.com | incidentally, I find it weird that opendev-buildset-registry-consumer doesn't use-buildset-registry | 16:35 |
| @fungicide:matrix.org | yeah, if i `ssh -i .ssh/id_borg_backup_ed25519 ...` i'm able to get it to connect, so maybe borg just needs some sort of openssh config or envvar override to force it to use the newer key | 16:36 |
| @clarkb:matrix.org | ya I'm guessing that rsa is the default tried first on that version? And if we force it to ed25519 it would work | 16:36 |
| @mordred:waterwanders.com | like - seems there should be an easy base job one could use to consume container images built in other jobs of the buildset that doesn't require adding a playbook to the repo. maybe I should add one called "opendev-use-image" ? That honors container_command and is like opendev-build-image but adds the ensure-(docker|podman| and use-buildset-registry roles? | 16:38 |
| @clarkb:matrix.org | I guess that setting the parent and having it auto configure things is likely a better UX | 16:39 |
| @fungicide:matrix.org | okay, progress! i renamed `.ssh/id_borg_backup_ed25519*` to `.ssh/id_ed25519*` and it actually connects, but borg complains that "/opt/backups-202605/borg-wiki-update-test/backup is not a valid repository." so me simply creating an empty directory there was likely the wrong thing | 16:40 |
| @fungicide:matrix.org | is there some sort of `borg init` command i need to initialize that location? | 16:41 |
| @fungicide:matrix.org | i can dig it out of ansible if necessary | 16:42 |
| @fungicide:matrix.org | oh indeed, `playbooks/roles/borg-backup-server/tasks/user.yaml` does run a `borg init` command, good guess | 16:42 |
| @mordred:waterwanders.com | Clark: oooh. I think I see something. I think "opendev-buildset-registry-consumer" might not have fully caught up with newer container jobs. | 16:43 |
| @mordred:waterwanders.com | it uses "playbooks/docker-image/pre.yaml" not "playbooks/container-image/pre.yaml" | 16:43 |
| @mordred:waterwanders.com | playbooks/container-image/pre.yaml also does the ensure- dance. oh. although it defaults container_command to podman. that's confusing | 16:44 |
| @clarkb:matrix.org | ah that may explain it. We did go through a transition period and it wouldn't surprise me that we missed some stuff | 16:44 |
| @mordred:waterwanders.com | yeah. I think I'd be uncomfortable swapping the playbook on that job, just because I'm not sure we've got a general expectation that container_command defaults to podman anywhere else and I'd not want to change a behavior on someone. but I think I can add a second base job that would use that playbook and make sense | 16:46 |
| @clarkb:matrix.org | ya that is probably safest | 16:46 |
| @fungicide:matrix.org | borg-backup to the new server has been running on wiki for a few minutes now, so i think it's finally creating an initial full copy of the files | 16:48 |
| @fungicide:matrix.org | there are longstanding mysql errors from the stream script, not new to the backup server switch, which i'll look into once this part is settled | 16:48 |
| @clarkb:matrix.org | fungi: excellent I guess if taht succeeds we can add a cronjob for it and then think about cleaning up the old server next week | 16:48 |
| @fungicide:matrix.org | well, i already edited the cronjob | 16:48 |
| @fungicide:matrix.org | so when it does fire it'll try to write to the new server from now on | 16:49 |
| @clarkb:matrix.org | ah did you simply replace the target or add a third entry like we have on the other servers? | 16:49 |
| @fungicide:matrix.org | i just replaced it, since we're already backing up to another server anyway | 16:49 |
| @clarkb:matrix.org | sounds good and ya removes a step for later | 16:50 |
| @fungicide:matrix.org | just annoying that it took me several weeks to find a quiet holiday friday to knock this out, sorry! | 16:51 |
| @mordred:waterwanders.com | Clark: ok. it's sort of even sillier, now that I've gone all the way down the rabbit hole | 17:09 |
| @mordred:waterwanders.com | there is TOTALLY a job that does what one would want from a "I want a job to consume but not build images" pov, it's just not "opendev-buildset-registry-consumer" - it's opendev-build-image-base. It's split from the image-build job so for secret protection, but it totally does exactly what a consumer job would want, it's just not clear from the job name or description | 17:14 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 994146: Add opendev-use-container-image job https://review.opendev.org/c/opendev/base-jobs/+/994146 | 17:14 | |
| @mordred:waterwanders.com | strictly speaking not needed - but pushed up as a "maybe a name and doc string are worth it"? | 17:14 |
| @clarkb:matrix.org | I think corvus set up most of that and may have opinions. | 17:24 |
| @fungicide:matrix.org | yeah, there was a bit of a naming scheme, and so for consistency purposes we may want to adjust in multiple places if we do | 17:56 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: | 18:06 | |
| - [zuul/zuul-jobs] 990850: Add resolute and trixie to zuul-jobs test matrix https://review.opendev.org/c/zuul/zuul-jobs/+/990850 | ||
| - [zuul/zuul-jobs] 994148: Remove sphinx version pin https://review.opendev.org/c/zuul/zuul-jobs/+/994148 | ||
| @mordred:waterwanders.com | ++ looking at callsites, I think the existing opendev-buildset-registry-consumer has a solid use case for _not_ installing podman/docker - it's currently being used in integration tests where installing the container system as part of the thing under test is valuable. Like - for system-config jobs ... making sure docker gets installed by system-config ansible is almost certainly a feature not a bug | 18:07 |
| @mordred:waterwanders.com | so _definitely_ don't want to modify that job. :) | 18:08 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: | 18:46 | |
| - [zuul/zuul-jobs] 990850: Add resolute and trixie to zuul-jobs test matrix https://review.opendev.org/c/zuul/zuul-jobs/+/990850 | ||
| - [zuul/zuul-jobs] 994154: Make installing skopeo from source the default https://review.opendev.org/c/zuul/zuul-jobs/+/994154 | ||
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: | 19:10 | |
| - [zuul/zuul-jobs] 990850: Add resolute and trixie to zuul-jobs test matrix https://review.opendev.org/c/zuul/zuul-jobs/+/990850 | ||
| - [zuul/zuul-jobs] 994155: Re-add zuul-jobs-test-ensure-kubernetes-crio-ubuntu-focal https://review.opendev.org/c/zuul/zuul-jobs/+/994155 | ||
| @fungicide:matrix.org | looking at the mysqldump error in the borg backups of wiki, it seems to be non-fatal since the dump still happens regardless | 19:21 |
| @fungicide:matrix.org | `mysqldump: Error: 'Access denied; you need (at least one of) the PROCESS privilege(s) for this operation' when trying to dump tablespaces` | 19:22 |
| @fungicide:matrix.org | this is also showing up when cron runs mysqldump to a local file daily | 19:22 |
| @fungicide:matrix.org | so i'm inclined to just ignore it since we're still getting the mediawiki table, which is what we mainly care about anyway | 19:24 |
| @fungicide:matrix.org | er, mediawiki database i mean | 19:24 |
| @clarkb:matrix.org | We may be able to fix that by limiting the dump to the one DB. It's probably trying to do all schemas | 19:26 |
| @fungicide:matrix.org | yeah, dropping `--all-databases` and specifying the mediwiki db only silences it | 19:27 |
| @fungicide:matrix.org | but i expect the end result is the same | 19:27 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 990850: Add resolute and trixie to zuul-jobs test matrix https://review.opendev.org/c/zuul/zuul-jobs/+/990850 | 19:28 | |
| @mordred:waterwanders.com | ++ to specifying only the mediawiki db | 19:29 |
| @fungicide:matrix.org | backup03.ca-ymq-1.vexxhost.opendev.org:~borg-wiki-update-test/backup contains 43G data now that the initial backup has run | 19:29 |
| @fungicide:matrix.org | well, this will be solved automatically with the new wiki server, so if things are working sufficiently now i'm not inclined to spend a lot of extra time on the old server | 19:30 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!