| frickler | it only affects bionic and focal, not jammy, so I assume it must be something inside the image | 06:52 |
|---|---|---|
| frickler | iiuc we install old ca-certificates from release, not using security or updates pockets, and those are too old to contain the LE R3 cert from 2020 | 07:00 |
| frickler | sadly there aren't any older build logs around to compare to a working build | 07:01 |
| frickler | maybe we should tweak nb to keep the build logs for the active images and only cycle the failing build's logs | 07:02 |
| frickler | the cert expired last year, so something else must have changed in the last few days | 07:04 |
| frickler | oh wow, current images are 19d old, maybe we also need a better notification setup. like a periodic job that fails if the image is more than 2 days old | 07:05 |
| frickler | clarkb: oh, you explicitly changed to https https://review.opendev.org/c/openstack/project-config/+/840392/1/nodepool/nodepool.yaml | 07:22 |
| frickler | seems we never verified that this actually works | 07:22 |
| frickler | will revert for bionic+focal, since you left out xenial already | 07:22 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Don't use https in Ubuntu bionic and focal builds https://review.opendev.org/c/openstack/project-config/+/842852 | 07:28 |
| opendevreview | Merged openstack/project-config master: Don't use https in Ubuntu bionic and focal builds https://review.opendev.org/c/openstack/project-config/+/842852 | 07:56 |
| frickler | at least not failing fast any longer, will watch for a bit https://nb02.opendev.org/ubuntu-bionic-0000229939.log | 08:05 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Fix python list for requirements propose-updates job https://review.opendev.org/c/openstack/project-config/+/842853 | 08:23 |
| frickler | infra-root: ^^ this is also still failing, I haven't found a way to test it other than waiting for the next periodic trigger. cf. https://27abb73f6979e2fd6932-3e445c1a5b8a8c84f71b5a61cff5a09b.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/requirements/master/propose-updates/5c99ab0/job-output.txt | 08:24 |
| opendevreview | Merged openstack/project-config master: Migrate neutron-lib-master testing to py39 https://review.opendev.org/c/openstack/project-config/+/842843 | 09:26 |
| frickler | new bionic images are uploading, focal build looking good, too | 09:46 |
| fungi | frickler: good catch on the bionic/focal apt-https issue, sorry for not noticing sooner | 12:00 |
| opendevreview | Dr. Jens Harbott proposed openstack/diskimage-builder master: Fix grub setup on Gentoo https://review.opendev.org/c/openstack/diskimage-builder/+/842856 | 12:21 |
| Clark[m] | Wow how is "up to date ssl certs" not something reprepro grabs? That seems pretty fundamental? | 14:49 |
| Clark[m] | apt supports SSL out of the box on those distros | 14:49 |
| frickler | clarkb: fungi: debian-security mirroring seems broken. is it possible you missed a cleanup when removing source pkgs? | 15:06 |
| frickler | Error: packages database contains unused 'bullseye-security|main|source' database. | 15:06 |
| frickler | noticed through failing kolla build jobs | 15:06 |
| frickler | 00:14 < clarkb> no I think security is separate and never had source | 15:08 |
| frickler | seem it had | 15:08 |
| Clark[m] | frickler: oops I can run the cleanup tomorrow | 15:49 |
| Clark[m] | You have to grab the flock to prevent automated reprepro from running then tun the clear vanished and delete unreferenced commands with some flags. Then I run the normal cron script to ensure it releases properly | 15:50 |
| frickler | Clark[m]: o.k., I'm going to try that now, since it is blocking kolla. I see some commands in the root history, so that should work I hope | 17:31 |
| Clark[m] | frickler there is a screen session on the host still too and the different windows may have commands in them too | 17:34 |
| frickler | Clark[m]: ah, I used tmux now out of old habits, but the critical part seems to be done already and the mirror-update is progressing fine | 17:37 |
| frickler | "Released volume mirror.debian-security successfully" | 17:42 |
| frickler | #status log ran "reprepro clearvanished" manually for debian-security to unbreak mirroring after source removal | 17:43 |
| fungi | thanks frickler! | 17:47 |
| frickler | btw. that volume also seems to be missing in grafana, else I'd likely have noticed this earlier, will make a patch for that | 17:48 |
| frickler | the wheels build should hopefully also be fixed by this | 17:48 |
| frickler | new issue: push-to-intermediate-registry : Push tag to intermediate registry, TIMED_OUT twice in a row, very likely unrelated to my patch https://zuul.opendev.org/t/openstack/build/878a508d4814471bbe631be978fc6915 | 18:12 |
| *** rlandy is now known as rlandy|out | 21:22 | |
| *** prometheanfire is now known as Guest0 | 22:26 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!