| *** elodilles is now known as elodilles_pto | 05:55 | |
| bbezak | Hi - rax debian mirror looks not in sync with upstream: | 11:42 |
|---|---|---|
| bbezak | libssl-dev : Depends: libssl3 (= 3.0.16-1~deb12u1) but 3.0.17-1~deb12u1 is to be installed in "http://mirror-int.dfw.rax.opendev.org/debian bookworm InRelease" - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bd5/openstack/bd553b17a1bf4ce28325674909ad7b53/primary/logs/build/000_FAILED_kolla-toolbox.log | 11:42 |
| bbezak | I tried on my bookworm vm upstream and it was ok | 11:43 |
| bbezak | (the same commands) | 11:43 |
| bbezak | I'll double check that on quay.io/openstack.kolla/debian:bookworm | 11:46 |
| bbezak | yeah looks good doing that on raw base quay.io/openstack.kolla/debian:bookworm | 11:51 |
| bbezak | apt-get update \... (full message at <https://matrix.org/oftc/media/v1/media/download/ARbOfRIf5Tfoafwn8k4kugncHv6eucRDgoWi53s0tsF7VGIUL8IzG0X97AIZ7urgl4GdposUXS_WCjzqW3QnwRRCeYvQUJHQAG1hdHJpeC5vcmcva2RjY1FWYlFZUWJNbFFubFh0SE5XaE1I>) | 11:51 |
| bbezak | (sorry for spam) | 11:52 |
| frickler | bbezak: looks like bullseye-backports was removed from the upstream mirrors, more than a year after the announcement was made https://lists.debian.org/debian-backports-announce/2024/07/msg00000.html . guess we need to update our mirror config | 12:06 |
| frickler | fwiw this seems to have been broken since 2025-07-22 | 12:06 |
| frickler | grafana agrees, mirror.debian last released 13d ago https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6h&to=now&timezone=utc | 12:07 |
| opendevreview | Dr. Jens Harbott proposed opendev/system-config master: Drop bullseye-backports from mirror https://review.opendev.org/c/opendev/system-config/+/956463 | 12:09 |
| bbezak | hmmm, here we're failing on bookworm though - which reminds me that we didn't update kolla support matrix :) | 12:11 |
| frickler | bbezak: yes, all debian updates are blocked because of this. | 12:25 |
| bbezak | got it | 12:25 |
| bbezak | thx | 12:25 |
| opendevreview | Merged opendev/system-config master: Drop bullseye-backports from mirror https://review.opendev.org/c/opendev/system-config/+/956463 | 13:04 |
| fungi | debian's own cloud images got caught by surprise too: https://bugs.debian.org/1109866 | 13:11 |
| fungi | the change has deployed now. the next update of debian mirroring is scheduled to run at 14:10 (about 55 minutes from now), but i'll do a manual run in a screen session instead since there are probably a lot of pending updates from the past 1.5 weeks | 13:16 |
| fungi | mmm... | 13:21 |
| fungi | Error: packages database contains unused 'bullseye-backports|main|amd64' database. | 13:21 |
| fungi | This usually means you removed some component, architecture or even a whole distribution from conf/distributions. In that case you most likely want to call reprepro clearvanished to get rid of the databases belonging to those removed parts. (Another reason to get this error is using conf/ and db/ directories belonging to different reprepro repositories). To ignore use | 13:22 |
| fungi | --ignore=undefinedtarget. | 13:22 |
| fungi | i think `reprepro clearvanished` may also result in the indices/packages going away, so any jobs using bullseye-backports will likely start breaking | 13:23 |
| fungi | our examples like https://docs.opendev.org/opendev/system-config/latest/reprepro.html#removing-components do `reprepro --delete clearvanished` so maybe it's the --delete that actually removes files | 13:26 |
| fungi | the reprepro manpage says "clearvanished: Remove all package databases that no longer appear in conf/distributions. If --delete is specified, it will not stop if there are still packages left. Even without --delete it will unreference files still marked as needed by this target. (Use --keep‐unreferenced to not delete them if that was the last reference.) Do not forget to remove all | 13:33 |
| fungi | exported package indices manually." | 13:33 |
| fungi | so it seems like if we just do `reprepro clearvanished` then our later cleanup operations might remove the packages once they're unreferenced | 13:34 |
| fungi | maybe we should do `reprepro clearvanished --keep‐unreferenced` in this case? opinions? | 13:35 |
| fungi | in unrelated news, the wiki server is failing to record auth sessions again, uptime is just shy of 21 days, rebooting now | 13:39 |
| fungi | #status log Rebooted wiki.openstack.org again to get OpenID logins back to a working state | 13:43 |
| opendevstatus | fungi: finished logging | 13:43 |
| NicolaiRuckel | I tried to implement some changes in Nova's libvirt.driver and now some test cases fail with "nova.exception.ObjectActionError: Object action obj_load_attr failed because: attribute system_metadata not lazy-loadable". Does anyone have an idea where to look for debugging? I couldn't figure out where this would be set yet. | 13:46 |
| NicolaiRuckel | I tried to implement some changes in Nova's libvirt.driver and now some test cases fail with "nova.exception.ObjectActionError: Object action obj_load_attr failed because: attribute system_metadata not lazy-loadable". Does anyone have an idea where to look for debugging? I couldn't figure out where this would be set yet. | 13:47 |
| fungi | NicolaiRuckel: have you tried asking in the #openstack-nova channel? | 13:48 |
| NicolaiRuckel | I tried to implement some changes in Nova's libvirt.driver and now some test cases fail with "nova.exception.ObjectActionError: Object action obj_load_attr failed because: attribute system_metadata not lazy-loadable". Does anyone have an idea where to look for debugging? I couldn't figure out where this would be set yet. | 13:48 |
| NicolaiRuckel | sorry for the spam | 13:48 |
| NicolaiRuckel | my irc client was acting up | 13:49 |
| NicolaiRuckel | I didn't try that but I'll do that now. | 13:49 |
| fungi | no worries. i suggested #openstack-nova instead because #opendev is the sysadmins for the collaboration tools run in the opendev collaboratory, people with expertise in the various projects hosted in opendev don't necessarily hang out in here | 13:50 |
| NicolaiRuckel | I see. Thanks! | 13:51 |
| fungi | you're welcome! | 13:51 |
| frickler | fungi: well I don't think we should keep repos that are no longer available upstream. if jobs break because of that, they should get fixed, rather than people wondering years later why things still work in our CI but not in the real world | 13:58 |
| fungi | i agree in principle, though in this case we may be baking that sources list entry into bullseye images, so will start breaking on that? not 100% sure | 14:00 |
| fungi | would need to double-check | 14:00 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: DNM: Test ansible version 11 https://review.opendev.org/c/zuul/zuul-jobs/+/956490 | 14:33 |
| fungi | looking at a recent example https://zuul.opendev.org/t/openstack/build/a579d1c68cc5486c8772eee7b6a69250/console#0/3/20/debian-bullseye is adding /etc/apt/sources.list.d/backports.list explicitly at job runtime | 14:43 |
| fungi | need to see if i can find one that isn't doing that | 14:43 |
| fungi | oh, i could look at the image build logs, d'oh | 14:55 |
| fungi | looks like it's at least not present the last time an apt-get update happens in the chroot: https://zuul.opendev.org/t/opendev/build/fa45c7fd3d584c4aa435331a79cd2373/log/diskimage-debian-bullseye.log#5937-5951 | 15:00 |
| fungi | i can't find any lines matching the substring "backport" in the image build log | 15:02 |
| fungi | so i think any fallout from removing it from our mirrors should be correctable in job configs | 15:04 |
| fungi | ah, right, we set it on by default in the configure-mirrors role though? https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/defaults/main.yaml#L14 | 15:08 |
| fungi | so while in principle it's not included on our images, in practice every single debian-based job is still installing the backports source unless explicitly told not to? | 15:09 |
| fungi | i suppose we could default it to false for debian versions <=bullseye | 15:12 |
| fungi | though perhaps having it default to false for all versions would maker more sense in the long term, per prior discussions, it's likely the sort of behavior change we'd need to announce for zuul/zuul-jobs | 15:13 |
| fungi | any other ideas? | 15:17 |
| fungi | unless we go adjusting the configure-mirrors role, i think our choice is between cleaning up the mirror and expecting every job that uses debian-bullseye nodes to explicitly override configure_mirrors_extra_repos to false, doing nothing in which case debian-bookworm jobs are broken for now due to the stale mirror, or leaving cruft indices/packages behind so that we can update our | 15:20 |
| fungi | mirrors without breaking bullseye-backports users for now | 15:20 |
| fungi | the least impactful options we could do right now would be to make configure-mirrors default to configure_mirrors_extra_repos=false for bullseye and then clean up the mirror, or to leave stale bullseye-backports cruft in the mirror and figure out how to clean that up later | 15:24 |
| fungi | oh, or we could add --ignore=undefinedtarget to our mirror script for now so it will stop choking on the existence of bullseye-backports in the reprepro db | 15:25 |
| fungi | maybe that's the fastest temporary workaround util we can have a discussion about it | 15:25 |
| fungi | "This allows you to temporarily remove some distribution from the config files, without having to remove the packages in it with the clearvanished command." | 15:28 |
| fungi | i'll manually test it with a copy of the script | 15:28 |
| fungi | looks like it's working so far | 15:29 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: reprepro: temporarily ignore undefinedtarget https://review.opendev.org/c/opendev/system-config/+/956497 | 16:09 |
| opendevreview | Corey Leong proposed openstack/project-config master: New Project Cascade https://review.opendev.org/c/openstack/project-config/+/956386 | 16:28 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: reprepro: temporarily ignore undefinedtarget https://review.opendev.org/c/opendev/system-config/+/956497 | 17:03 |
| opendevreview | Merged openstack/project-config master: New Project Cascade https://review.opendev.org/c/openstack/project-config/+/956386 | 17:04 |
| fungi | okay, my manual update of the debian mirror has made it to the vos release stage, so i think we're in the clear with it, shouldn't be much longer now | 17:17 |
| fungi | bbezak: the debian mirror should be up to date again for the moment while we work out how we want to deal with the bullseye-backports situation longer term | 18:00 |
| fungi | let me know if you're still seeing package update errors on any new runs | 18:00 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!