| *** mhen_ is now known as mhen | 01:15 | |
| opendevreview | Abhishek Kekane proposed openstack/glance_store master: Replace assertIn with assertRaisesRegex https://review.opendev.org/c/openstack/glance_store/+/953852 | 06:24 |
|---|---|---|
| opendevreview | Merged openstack/glance_store master: swift: Drop support for v1/v2 auth https://review.opendev.org/c/openstack/glance_store/+/932251 | 07:39 |
| opendevreview | Abhishek Kekane proposed openstack/glance master: Fix py312 and py313 failing unit test https://review.opendev.org/c/openstack/glance/+/953867 | 09:49 |
| abhishekk | tkajinam: o/ any idea about recent failures in glance gate | 10:15 |
| abhishekk | https://zuul.opendev.org/t/openstack/build/8855665562e24b9dbdad4e24f0e4efce | 10:16 |
| tkajinam | qemu-block-extra : Depends: librbd1 (>= 19.2.1-0ubuntu0.24.04.1) but 19.2.0-0ubuntu0.24.04.2 is to be installed | 10:25 |
| tkajinam | abhishekk, I suspect contents of noble repo were synced incompletely | 10:26 |
| abhishekk | yeah, I can see some discussion is going on nova channel about the same | 10:29 |
| croelandt | so, regarding glance_store | 13:12 |
| croelandt | I think we could still merge | 13:12 |
| croelandt | - the regex patch https://review.opendev.org/c/openstack/glance_store/+/953852 | 13:12 |
| croelandt | - the os-client-config patch https://review.opendev.org/c/openstack/glance_store/+/953796 | 13:12 |
| croelandt | and I think I'll go ahead and merge the deprecation of filesystem_store_datadirs https://review.opendev.org/c/openstack/glance_store/+/950568 | 13:12 |
| croelandt | abhishekk: does that make sense? | 13:13 |
| croelandt | dansmith: if you want to do more reviews today :) | 13:13 |
| abhishekk | croelandt: no objection on merging deprecating filesystem_store_datadirs patch | 13:42 |
| opendevreview | Merged openstack/glance_store master: Replace os-client-config https://review.opendev.org/c/openstack/glance_store/+/953796 | 15:29 |
| opendevreview | Merged openstack/glance_store master: Deprecate filesystem_store_datadirs https://review.opendev.org/c/openstack/glance_store/+/950568 | 15:56 |
| croelandt | abhishekk: seems like the gates are working ^ | 15:56 |
| croelandt | oh no ok https://review.opendev.org/c/openstack/glance_store/+/953852 is still failing | 16:32 |
| croelandt | how on Earth did the previous patches merge | 16:33 |
| abhishekk | croelandt: that patch was doc related so it didn't ran the job which is failing | 17:04 |
| croelandt | https://review.opendev.org/c/openstack/glance_store/+/953796 did run the job though | 17:08 |
| abhishekk | no not tempest failing here | 17:17 |
| abhishekk | glance_store-src-ceph-tempest | 17:17 |
| abhishekk | this one is failing for glance_store | 17:17 |
| abhishekk | croelandt: have you got any updates from infra about the gate issue? | 17:37 |
| clarkb | fungi: ^ fyi Ithink this is the problem you were discussing with jrosser | 17:41 |
| clarkb | abhishekk: croelandt: tl;dr is that ubuntu seems to have maybe reverted a broken package update improperly which has anyone who installed that package or mirrored it with reprepro in a stuck situation. We're hoping they fix it by rollign forward with a working package but we may be forced to manually clean up the mirror | 17:42 |
| abhishekk | ack, may I ask eta for the same? | 17:43 |
| abhishekk | clarkb: thank you for the update | 17:43 |
| fungi | abhishekk: you're welcome to ask ubuntu for an eta, absolutely | 17:44 |
| abhishekk | :D | 17:44 |
| fungi | apparently they released a bad package (incorrect dependencies) and then rolled the package list for noble-updates back to an older package version | 17:45 |
| fungi | rather than rolling forward to a newer corrected build | 17:45 |
| abhishekk | :/ | 17:45 |
| fungi | so mirroring software is ignoring the old package because the one we mirrored earlier is newer | 17:45 |
| abhishekk | ack | 17:46 |
| abhishekk | croelandt: this means we need some workaround for our glance_store so that we can get our required patches in | 17:47 |
| fungi | if i get time i'm going to try to figure out how to get the mirroring software (reprepro) to forget the newer package version, though that could end up being disruptive as well | 17:48 |
| abhishekk | ok, thank you for the update | 17:49 |
| fungi | i've made a temporary adjustment to our mirroring routine to explicitly delete the broken newer package, and it looks like that's doing what i'd holed forcing it to pull the older one they rolled back to, but it will still be a bit before that update appears in our mirror network | 18:23 |
| croelandt | thanks! Is there anyway for me to check when the update is available on our mirror network? | 18:24 |
| fungi | croelandt: when https://static.opendev.org/mirror/ubuntu/dists/noble-updates/main/binary-amd64/Packages includes qemu-block-extra 1:8.2.2+ds-0ubuntu1.7 instead of 1:8.2.2+ds-0ubuntu1.8 it will hopefully stop being a problem | 18:27 |
| fungi | apparently they built it against the librbd1 from plucky instead of noble, and published that to the archive without realizing it, then later yanked it, which is why the version should go backwards | 18:28 |
| fungi | croelandt: it should be updated now, i just confirmed, so recheck at will | 19:17 |
| croelandt | fungi: hm, still failing... but differently https://zuul.opendev.org/t/openstack/build/b1068278bf0f48fdba1a5b63db065615 | 22:05 |
| croelandt | The following packages have unmet dependencies: | 22:05 |
| croelandt | qemu-block-extra : Depends: qemu-system-any (= 1:8.2.2+ds-0ubuntu1.7) or | 22:05 |
| croelandt | qemu-utils (= 1:8.2.2+ds-0ubuntu1.7) but 1:8.2.2+ds-0ubuntu1.8 is to be installed | 22:05 |
| clarkb | possible that we needed to delete other packages when updating the mirror | 22:17 |
| clarkb | specifically qemu-utils 1:8.2.2+ds-0ubuntu1.8 | 22:18 |
| fungi | croelandt: okay, so this is going to quickly turn into a game of whack-a-mole if we don't know what packages they removed | 22:18 |
| fungi | i can repeat the earlier process and then see where we end up next | 22:19 |
| croelandt | worst case scenario, we could disable that test, I guess | 22:26 |
| croelandt | This is quite annoying though :/ | 22:27 |
| croelandt | I'm afraid the process will have to be repeated for N packages indeed | 22:27 |
| clarkb | croelandt: you may be able to use apt on a node looking at our mirrors to build a graph of all the related packages | 22:28 |
| clarkb | then fungi can clear those out and let them repopulate in one go | 22:29 |
| clarkb | if we operate under the assumption that all of the updated packages are tied together via dependencies amongst themselves | 22:29 |
| croelandt | clarkb: do I need to ssh a node somehow? | 22:34 |
| clarkb | croelandt: its probably sufficient to start a local ubuntu container and update its mirror config | 22:35 |
| clarkb | our mirrors are publicly accessible. https://mirror.dfw.rax.opendev.org/ubuntu is the one I typically default to checking | 22:36 |
| fungi | my guess is all binary packages built from that same source package need to be rolled back | 22:41 |
| clarkb | oh ya just looking for that one package version might be enough to get an idea of that too | 22:41 |
| fungi | https://packages.ubuntu.com/source/noble-updates/qemu lists them | 22:42 |
| fungi | i'll just tell reprepro to remove the lot of them | 22:42 |
| clarkb | ++ | 22:42 |
| croelandt | ok let's do that and I'll go to bed :p | 22:42 |
| fungi | yes! get some sleep | 22:42 |
| fungi | this will keep | 22:42 |
| fungi | i've got it set up, just need to wait for the present mirror update to finish before i can start it | 22:46 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!