| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 1001597: WIP Get artifacts from builds via buildset https://review.opendev.org/c/zuul/zuul-jobs/+/1001597 | 00:23 | |
| -@gerrit:opendev.org- Stephen Finucane proposed: | 11:57 | |
| - [opendev/git-review] 987712: Revert "Clean up all references to branchauthor after removal of usage" https://review.opendev.org/c/opendev/git-review/+/987712 | ||
| - [opendev/git-review] 1001646: docs: Add note on removal of auto-topic feature https://review.opendev.org/c/opendev/git-review/+/1001646 | ||
| - [opendev/git-review] 1001647: Re-add author to branch names https://review.opendev.org/c/opendev/git-review/+/1001647 | ||
| -@gerrit:opendev.org- Stephen Finucane proposed: | 12:04 | |
| - [opendev/git-review] 1001646: docs: Add note on removal of auto-topic feature https://review.opendev.org/c/opendev/git-review/+/1001646 | ||
| - [opendev/git-review] 989239: Remove support for global/system git-review config files https://review.opendev.org/c/opendev/git-review/+/989239 | ||
| - [opendev/git-review] 988966: Use configparser utils to parse types https://review.opendev.org/c/opendev/git-review/+/988966 | ||
| - [opendev/git-review] 987712: Revert "Clean up all references to branchauthor after removal of usage" https://review.opendev.org/c/opendev/git-review/+/987712 | ||
| - [opendev/git-review] 1001647: Re-add author to branch names https://review.opendev.org/c/opendev/git-review/+/1001647 | ||
| - [opendev/git-review] 987713: Add gitreview.autotopic git config flag https://review.opendev.org/c/opendev/git-review/+/987713 | ||
| @fungicide:matrix.org | we got some notifications about let's encrypt certs expiring in fewer than 30 days, looks like our daily infra-prod-base runs have been failing since the 15th or 16th because of a hung apt process on mirror02.regionone.osuosl | 12:58 |
|---|---|---|
| @fungicide:matrix.org | yeah, looks like the process has been there since some time on the 15th | 12:59 |
| @fungicide:matrix.org | it was actually a hung gpgv child process so i killed that and apt seems to have retried it which looks like it worked the second time and wrapped up cleanly | 13:00 |
| @fungicide:matrix.org | i was able to `apt update` manually after that without any lock contention errors so i think certs should renew properly in 12-14 hours from now | 13:01 |
| @clarkb:matrix.org | I had to claer out a stuck apt process on that same mirror not that long ago | 14:18 |
| @clarkb:matrix.org | I wonder if there is some underlying issue we need to address | 14:19 |
| @clarkb:matrix.org | Jens Harbott: if you have a moment https://review.opendev.org/c/opendev/system-config/+/1000864 was written to address the haproxy logging issue you discovered | 14:19 |
| @clarkb:matrix.org | fungi: if there are no complaints about anubis this morning we can probably proceed with https://review.opendev.org/c/opendev/system-config/+/1001416 | 14:20 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/1000379 and its parent are CI related anubis improvements too | 14:20 |
| @fungicide:matrix.org | i have to run a bunch of errands at lunch today, so will be gone probably ~15:00-17:00 utc so can approve it after i get back, or let it merge while i'm not around if you're okay monitoring in the meantime | 14:21 |
| @clarkb:matrix.org | fungi: I should be able to monitor it. It is an easy revert if necessary too | 14:21 |
| @jim:acmegating.com | they stuck apt procs on zuul executors are typically curl commands, so different than gpgv. but maybe we should think about a global/universal cleanup? | 14:21 |
| @jim:acmegating.com | s/they/the/ | 14:21 |
| @clarkb:matrix.org | fungi: last time it took a few hours for the scores to build up I think so an earlier start is maybe a good idea | 14:21 |
| @fungicide:matrix.org | agreed, approving 1001416 now | 14:22 |
| @clarkb:matrix.org | corvus: fungi I wonder if we'd get better luck with apt rather than apt-get? I think both are maintained but maybe apt is getting much more attention these days? | 14:22 |
| @jim:acmegating.com | does apt call apt-get? that was the plan from... uh... 28 years ago. but maybe the plan changed. :) | 14:23 |
| @fungicide:matrix.org | apt warns that it doesn't have stable output and is intended for interactive use primarily, while apt-get makes some guarantees about what it prints to stdout/stderr. they still call mostly the same things behind the scenes anyway | 14:23 |
| @fungicide:matrix.org | i'm mostly used to running `apt ...` because it rolls in features from `apt-cache` and `apt-source` and similar previously separate tools | 14:25 |
| @jim:acmegating.com | the stuck processes come from the nightly upgrades anyway right? and that's its own debian package that handles the underlying command? | 14:26 |
| @clarkb:matrix.org | oh ya it may be from unattended upgrades. I think we have an ansible task that does an explicit update too but we're not necessarily the source of the hangs | 14:27 |
| @fungicide:matrix.org | right, i think it would make more sense to turn off the scheduled task for unattended-upgrades and just run it directly from the daily zuul job, because at least for the non-hanging case we don't risk random lockfile collisions | 14:29 |
| @clarkb:matrix.org | the downside to that is if we stick a host in the emergency file for a prolonged period we may not get updates (though we could potentially have those tasks ignore the disabled group) | 14:31 |
| @fungicide:matrix.org | good point | 14:32 |
| @clarkb:matrix.org | one thing I ran into locally recently is that apt-get was using ipv6 mirrors even though I didn't have a global ipv6 address | 14:35 |
| @fungicide:matrix.org | oh that's fun | 14:35 |
| @clarkb:matrix.org | I half wonder if there is some weird network related situation along those lines that we're running into | 14:36 |
| @fungicide:matrix.org | very well could be | 14:37 |
| @fungicide:matrix.org | i'm heading out, back in a couple of hours | 14:44 |
| -@gerrit:opendev.org- Takashi Kajinami proposed: | 14:50 | |
| - [openstack/diskimage-builder] 1001676: Declare Python 3.13 support https://review.opendev.org/c/openstack/diskimage-builder/+/1001676 | ||
| - [openstack/diskimage-builder] 1001677: Add unit tests on Python 3.14 https://review.opendev.org/c/openstack/diskimage-builder/+/1001677 | ||
| @mordred:waterwanders.com | Clark: the dib patches there ^^ reminded me of having disabled dib for resolute in zuul-jobs, which reminded me of the podman stack. if you have some time, would you mind re-looking at https://review.opendev.org/q/hashtag:%22ensure-podman%22 ? You've already reviewed the 1 patch that could maybe impact not-podman, https://review.opendev.org/c/zuul/zuul-jobs/+/995734 but I had to update it since you reviewed due to a hardcoded port. | 15:25 |
| @mordred:waterwanders.com | Yeah - same. I taught myself to stop typing "-get" mostly because of that | 15:26 |
| @mordred:waterwanders.com | still feels slightly weird | 15:26 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 995734: Use net=host for buildset-registry https://review.opendev.org/c/zuul/zuul-jobs/+/995734 | 16:17 | |
| @mordred:waterwanders.com | \o/ | 16:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: | 16:23 | |
| - [zuul/zuul-jobs] 995739: Add podman variant of the registry-buildset-registry job https://review.opendev.org/c/zuul/zuul-jobs/+/995739 | ||
| - [zuul/zuul-jobs] 994297: Add ntftables and aardvark-dns to rootless podman packages https://review.opendev.org/c/zuul/zuul-jobs/+/994297 | ||
| @clarkb:matrix.org | fungi: anubis honeypot change timed out in testinfra test cases. I don't see an obvious issue things were running just slowly. I went ahead and rechecked it as a result | 16:23 |
| @mordred:waterwanders.com | ok - that's the one that has impacts for non-podman use. let me to trigger a container build for something | 16:23 |
| @clarkb:matrix.org | mordred: https://review.opendev.org/c/opendev/system-config/+/1001416 that change's recheck may exercise some of your chagnes that just landed (depends on timing) | 16:23 |
| @clarkb:matrix.org | I'm not sure if my recheck got in ahead of that merge (the matrix gerritbot has some lag so its less obvious than it was with IRC) | 16:24 |
| @mordred:waterwanders.com | cool. I also pushed up a DNM for zuul. | 16:25 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 995585: Align podman packages on resolute and forward https://review.opendev.org/c/zuul/zuul-jobs/+/995585 | 16:27 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 995880: Add test jobs to test podman sockets https://review.opendev.org/c/zuul/zuul-jobs/+/995880 | 16:28 | |
| @mordred:waterwanders.com | Clark: the system-config change did pick up the net=host change - no socat task was run | 16:29 |
| @fungicide:matrix.org | okay, back | 16:40 |
| @mordred:waterwanders.com | Clark: system-config change failed - but it looks like the container parts of it worked | 16:40 |
| @fungicide:matrix.org | guess the test timeout delay bought me some time | 16:40 |
| @clarkb:matrix.org | hrm it shouldn't fail that quickly /me looks | 16:41 |
| @clarkb:matrix.org | mordred: oh its just the ansible-devel job failing. That is non voting because keeping up with ansible is hard | 16:41 |
| @clarkb:matrix.org | system-config-run-gitea is the one that will exercise the registry | 16:42 |
| @mordred:waterwanders.com | Clark: I *think* it got past the buildset registry interaction parts. and zuul-jobs-test-registry-buildset-registry-podman-resolute-8GB passed on the recheck (that was the one that was red in the podman zuul-jobs stack) And the Zuul DNM patch I pushed up ran quickstart successfully, So I'm cautiously optimistic we're good | 16:51 |
| @clarkb:matrix.org | mordred: yes looks like the gitea job has started gitea (which depends on pulling an image and should interact with the buildset registry) | 16:52 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 995740: Add resolute versions of registry jobs https://review.opendev.org/c/zuul/zuul-jobs/+/995740 | 17:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 996725: Add catatonit to podman package list https://review.opendev.org/c/zuul/zuul-jobs/+/996725 | 17:35 | |
| @clarkb:matrix.org | the anubis honeypot change is back in the gate | 17:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 1001416: Reenable Anubis honeypot feature https://review.opendev.org/c/opendev/system-config/+/1001416 | 18:38 | |
| @clarkb:matrix.org | fungi: it has just occurred to me that anubis will not be restarted after ^ lands (gitea09 for example has already updated the file but the containers have not restarted | 18:44 |
| @clarkb:matrix.org | fungi: I don't think anubis will automatically reload its config either (this was why we had problems after removing the honeypot the first time around) | 18:45 |
| @clarkb:matrix.org | once the deployment is done I can work on restarting all of those containers | 18:45 |
| @clarkb:matrix.org | I ran `docker-compose restart anubis` on each of the giteas. This should be sufficient because we just needed anubis to reread its botpolicy.yaml file and didn't make any changes to docker-compose.yaml | 18:55 |
| @clarkb:matrix.org | I am able to reach opendev.org with my browser and I can clone system-config | 18:56 |
| @clarkb:matrix.org | I don't see any weights >0 yet. So this may take some time to start being noticeable if anything were to go sideways | 18:57 |
| @clarkb:matrix.org | I'll eat lunch then check the logs again to see if that has chagned yet | 18:57 |
| @fungicide:matrix.org | Clark: sounds good, i think that's the metric to watch | 19:30 |
| @clarkb:matrix.org | still not seeing any obvious high weight issues. There is a fun "error" anubis rejects clients if they should be able to handle gzip content but don't | 19:31 |
| @clarkb:matrix.org | I don't see any explicit honeypot paths in the log. Not sure that is just due to crawlers not finding it yet or maybe my restart was insufficient (I'm pretty sure it should be sufficient) | 19:34 |
| @clarkb:matrix.org | in any case things don't appear to be sad | 19:34 |
| @fungicide:matrix.org | claiming to handle gzip but then not doing it sounds like a classic poorly-implemented bot heuristic, to be honest | 19:35 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [openstack/diskimage-builder] 1001456: CI: Disable swift in Devstack https://review.opendev.org/c/openstack/diskimage-builder/+/1001456 | 20:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: | 20:37 | |
| - [openstack/diskimage-builder] 1001465: rocky-container: Add support for DIB_DISTRIBUTION_MIRROR https://review.opendev.org/c/openstack/diskimage-builder/+/1001465 | ||
| - [openstack/diskimage-builder] 1001483: CI: Bump dib flavor disk to 6GB https://review.opendev.org/c/openstack/diskimage-builder/+/1001483 | ||
| -@gerrit:opendev.org- Steve Baker proposed: [openstack/diskimage-builder] 1001740: Add dib-init-image tool for creating empty disk images https://review.opendev.org/c/openstack/diskimage-builder/+/1001740 | 21:21 | |
| @clarkb:matrix.org | as a heads up I'll plan to approve https://review.opendev.org/c/opendev/system-config/+/1000864 tomorrow | 21:57 |
| @clarkb:matrix.org | I have more errands to run this evening so don't want to approve it now in case haproxy doesn't restart happily. But tomorrow morning should be fine | 21:58 |
| @clarkb:matrix.org | this is the change to switch to bind mounting a dir for the syslog socket device instead of bind mounting the socket itself to make haproxy logging more resilient to syslogd restarts | 21:58 |
| @fungicide:matrix.org | makes sense | 22:18 |
| @fungicide:matrix.org | i'm basically tapping out of the ring for tonight as well | 22:19 |
| @clarkb:matrix.org | I'm deep in the "its a few days before school starts" woods | 22:24 |
| @clarkb:matrix.org | lots of random errands nad events to go to. Hopefully after next week the new routine will feel familiar and everything will settle down | 22:24 |
| @fungicide:matrix.org | trapper keepers? | 22:27 |
| @clarkb:matrix.org | they want 2" three ring binders | 22:28 |
| @clarkb:matrix.org | because they are not allowed to have bags in the halls so have to carry everythign in one giant binder | 22:28 |
| @fungicide:matrix.org | mead's trapper keeper product line has you covered, no fear | 22:28 |
| @fungicide:matrix.org | hello kitty is still popular no matter where you are | 22:29 |
| @mordred:waterwanders.com | wow. "they are not allowed to have bags in the halls" | 23:19 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!