| *** mrunge_ is now known as mrunge | 07:50 | |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu-jammy-arm64 image https://review.opendev.org/c/opendev/zuul-providers/+/948317 | 08:58 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add ubuntu-jammy-arm64 image builds https://review.opendev.org/c/opendev/zuul-providers/+/948318 | 09:03 |
| opendevreview | sean mooney proposed openstack/project-config master: create openstack/grian-ui git repo https://review.opendev.org/c/openstack/project-config/+/948249 | 10:36 |
| opendevreview | sean mooney proposed openstack/project-config master: Add jobs for openstack/grian-ui repo https://review.opendev.org/c/openstack/project-config/+/948250 | 10:36 |
| opendevreview | daniel.pawlik proposed opendev/system-config master: Add CentOS 10 mirror config https://review.opendev.org/c/opendev/system-config/+/948330 | 14:34 |
| opendevreview | daniel.pawlik proposed opendev/system-config master: Add CentOS 10 Stream mirror config https://review.opendev.org/c/opendev/system-config/+/948330 | 14:36 |
| *** gmaan_pto is now known as gmaan | 14:59 | |
| opendevreview | Dan Smith proposed openstack/project-config master: Make viommu disk image use q35 machine type https://review.opendev.org/c/openstack/project-config/+/948335 | 15:20 |
| clarkb | infra-root I intend on shutting down review02 today. Then we can discuss server cleanup in the meeting tomorrow | 15:24 |
| clarkb | then to followup on last weeks end of week efforts I've got a fix for gitea container image builds here: https://review.opendev.org/c/opendev/system-config/+/948277 which should allow https://review.opendev.org/c/opendev/system-config/+/948247 to land making our image builds more reliable when talking to docker hub | 15:24 |
| clarkb | Probably worthwhile to proceed with 948277 at this point as it should largely be a noop. But would be good for a second reviewer if anyone is around simply because this changse where we source the go modules for gitea when building it from source | 15:25 |
| clarkb | supply chain and all that is probably worth a second look? | 15:25 |
| clarkb | on review02 `docker ps -a` reports only the mariadb container is dfined and running (so we can't accidentally start gerrit on boot if we boot it up again). Then even if gerrit services did start on boot /home/gerrit2/review_site/etc/replication.config does not contain any replication targets so wouldn't be the end of the world to start up on | 15:31 |
| *** elodille1 is now known as elodilles | 15:31 | |
| clarkb | all taht to say I think it is safe for me to issue a `shutdown -h now` on review02. | 15:31 |
| clarkb | let me know if there is any other prep or cleanup work that you think should happen first but I'll aim to do this sometime in the next few hours | 15:32 |
| fungi | i concur | 15:32 |
| clarkb | also the server is in the emergency file but was also removed from our inventory so we won't get ansible failures trying to ssh to the shutdown instance | 15:33 |
| clarkb | re gitea builds I'm realizing that fungi and I may be the only two reviewers until much later today if frickler's day has already ended | 15:38 |
| clarkb | given that GOPROXY's default is to include the global proxy I think this should be safe (its our Dockerfile that we keep in sync with gitea upstream that overrides to force using direct that we then override via our job config...) | 15:39 |
| fungi | eems fine to me | 15:42 |
| fungi | er, seems | 15:42 |
| clarkb | I'm working on breakfast but I'll probably go ahead and approve the gitea build fix afterwards if no objections show up beforehand | 15:43 |
| clarkb | and then after gitea is done shutting down review02 is next | 15:43 |
| opendevreview | Merged openstack/project-config master: Make viommu disk image use q35 machine type https://review.opendev.org/c/openstack/project-config/+/948335 | 15:52 |
| clarkb | re gitea they publish a go.sum file next to go.mod so we should be extra safe in terms of supply chain getting the right thing from the proxy | 16:12 |
| clarkb | I have approved the gitea image build update | 16:12 |
| fungi | does go automatically check the sums then? | 16:19 |
| clarkb | ya I think when you run go mod to install things it does. Unless you're doing an explicit update of the deps and need new hashes | 16:20 |
| clarkb | https://zuul.opendev.org/t/openstack/build/7ca562c530b046ae99ca82057c489792 another post failure here | 16:31 |
| clarkb | I wonder if we just have some background failure rate with swift that wasn't there before? | 16:31 |
| clarkb | there are ~4 cases of this in the zuul tenant at the moment | 16:39 |
| clarkb | *openstack zuul tenant | 16:39 |
| clarkb | the example above ran on ze05 and attempted to upload logs to ovh gra | 16:40 |
| clarkb | ovh system status doesn't show anything super concerning. I guess like last time we can monitor and pull ovh out of the storage backends list if this gets worse | 16:43 |
| clarkb | infra-root ok I'm proceeding with review02 shutdown via `shutdown -h now` momentarily | 16:55 |
| clarkb | review02.opendev.org | SHUTOFF | 16:57 |
| clarkb | systemd kicked me off, ping stopped responding, and ^ nova has noticed it is shutoff | 16:58 |
| fungi | thanks! | 17:01 |
| clarkb | starting to think about meeting agenda stuff: we will discuss finalizing review02 cleanup, the docker hub ipv4 change for image builds, make cancelling the meeting on May 6 official. Anything else? | 17:12 |
| clarkb | I guess wiki's cert should be replaced this week. I will make a note of that on the meeting so that I get called out if it isn't done before friday | 17:12 |
| fungi | i don't have anything | 17:13 |
| clarkb | I can make a note of the very infrequent ovh log upload failures | 17:15 |
| clarkb | I don't see any more after the original set of ~4 today fwiw | 17:16 |
| clarkb | so maybe just a short blip. Seems like last time this happened it was a short blip at a similar time of day | 17:16 |
| opendevreview | Merged openstack/project-config master: create openstack/grian-ui git repo https://review.opendev.org/c/openstack/project-config/+/948249 | 18:00 |
| clarkb | the gitea image build update finally reported so I have rechecked the change (it was the post failure due to log upload issues earlier) | 18:08 |
| clarkb | there is a new burst of post failures on a nova change but they all have logs so caused by job problems not the storage backend | 18:13 |
| clarkb | (just noting that in case anyone sees them and thinks the problem is getting worse) | 18:14 |
| fungi | thanks | 18:14 |
| fungi | i need to go pick something up, so will grab a quick late lunch/early dinner while i'm out, should be back in about an hour | 18:54 |
| fungi | bbiab | 18:55 |
| clarkb | I grabbed lunch too | 19:26 |
| clarkb | the gitea change should land in about an hour and then I expect it to deploy relatively quickly after that | 20:00 |
| fungi | okay, back | 20:11 |
| fungi | i can help keep an eye on it | 20:11 |
| opendevreview | Merged opendev/system-config master: Build gitea with GOPROXY set https://review.opendev.org/c/opendev/system-config/+/948277 | 20:25 |
| fungi | that's a good sign | 20:30 |
| fungi | infra-prod-service-gitea should be wrapping up momentarily | 20:33 |
| fungi | and succeeded | 20:42 |
| clarkb | excellent | 20:42 |
| fungi | gitea still seems to be working | 20:42 |
| clarkb | I neded up stepping away for a thing around the house and it finished much more quickly than I expected | 20:43 |
| fungi | perfect timing, it solved itself while you weren't looking | 20:43 |
| clarkb | I confirmed all of them restarted in the last few minutes and the one I reach is up | 20:44 |
| fungi | same | 20:45 |
| clarkb | git clone works for me too | 20:49 |
| clarkb | I rechecked https://review.opendev.org/c/opendev/system-config/+/948247 since the gitea image build should be fixed now | 20:51 |
| fungi | i missed that a couple of years ago systemd added a replacement for /etc/passwd and /etc/group, because why not? https://www.freedesktop.org/software/systemd/man/latest/sysusers.d.html | 21:39 |
| clarkb | systemd/linux | 21:43 |
| clarkb | ok last call on meeting updates. I'll get that setn out in the next 15 minutes or so | 21:43 |
| fungi | hey! it worked on recheck | 21:57 |
| clarkb | yup I think the only issue was the source hut thing and I dont' blame them. Thats a bug in gitea's dockerfile | 21:58 |
| clarkb | unfortunately i think it is worth being in sync with them os updating the builds to override the override was the right fix | 21:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!