Sunday, 2022-05-22

fricklerit only affects bionic and focal, not jammy, so I assume it must be something inside the image06:52
frickleriiuc we install old ca-certificates from release, not using security or updates pockets, and those are too old to contain the LE R3 cert from 202007:00
fricklersadly there aren't any older build logs around to compare to a working build07:01
fricklermaybe we should tweak nb to keep the build logs for the active images and only cycle the failing build's logs07:02
fricklerthe cert expired last year, so something else must have changed in the last few days07:04
frickleroh wow, current images are 19d old, maybe we also need a better notification setup. like a periodic job that fails if the image is more than 2 days old07:05
fricklerclarkb: oh, you explicitly changed to https https://review.opendev.org/c/openstack/project-config/+/840392/1/nodepool/nodepool.yaml07:22
fricklerseems we never verified that this actually works07:22
fricklerwill revert for bionic+focal, since you left out xenial already07:22
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Don't use https in Ubuntu bionic and focal builds  https://review.opendev.org/c/openstack/project-config/+/84285207:28
opendevreviewMerged openstack/project-config master: Don't use https in Ubuntu bionic and focal builds  https://review.opendev.org/c/openstack/project-config/+/84285207:56
fricklerat least not failing fast any longer, will watch for a bit https://nb02.opendev.org/ubuntu-bionic-0000229939.log08:05
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Fix python list for requirements propose-updates job  https://review.opendev.org/c/openstack/project-config/+/84285308:23
fricklerinfra-root: ^^ this is also still failing, I haven't found a way to test it other than waiting for the next periodic trigger. cf. https://27abb73f6979e2fd6932-3e445c1a5b8a8c84f71b5a61cff5a09b.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/requirements/master/propose-updates/5c99ab0/job-output.txt08:24
opendevreviewMerged openstack/project-config master: Migrate neutron-lib-master testing to py39  https://review.opendev.org/c/openstack/project-config/+/84284309:26
fricklernew bionic images are uploading, focal build looking good, too09:46
fungifrickler: good catch on the bionic/focal apt-https issue, sorry for not noticing sooner12:00
opendevreviewDr. Jens Harbott proposed openstack/diskimage-builder master: Fix grub setup on Gentoo  https://review.opendev.org/c/openstack/diskimage-builder/+/84285612:21
Clark[m]Wow how is "up to date ssl certs" not something reprepro grabs? That seems pretty fundamental?14:49
Clark[m]apt supports SSL out of the box on those distros14:49
fricklerclarkb: fungi: debian-security mirroring seems broken. is it possible you missed a cleanup when removing source pkgs?15:06
fricklerError: packages database contains unused 'bullseye-security|main|source' database.15:06
fricklernoticed through failing kolla build jobs15:06
frickler00:14 < clarkb> no I think security is separate and never had source15:08
fricklerseem it had15:08
Clark[m]frickler: oops I can run the cleanup tomorrow15:49
Clark[m]You have to grab the flock to prevent automated reprepro from running then tun the clear vanished and delete unreferenced commands with some flags. Then I run the normal cron script to ensure it releases properly15:50
fricklerClark[m]: o.k., I'm going to try that now, since it is blocking kolla. I see some commands in the root history, so that should work I hope17:31
Clark[m]frickler there is a screen session on the host still too and the different windows may have commands in them too17:34
fricklerClark[m]: ah, I used tmux now out of old habits, but the critical part seems to be done already and the mirror-update is progressing fine17:37
frickler"Released volume mirror.debian-security successfully"17:42
frickler#status log ran "reprepro clearvanished" manually for debian-security to unbreak mirroring after source removal17:43

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!