Friday, 2024-02-09

amorinhey team, fungi I received your mail, will do the necessary to extend08:47
frickleramorin: cool, thx for the update, so we can cancel switching off things08:49
fricklerinfra-root: due to $things I won't be around next week. I'll still try to make it to the "early" (UTC afternoon) preptg sessions09:02
amorinfrickler, fungi, clarkb all done on ovh billing side, should be good, let me know10:49
fungithanks amorin!13:06
GhostLyr_Hello! Excuse me, is there any way to contact one of the contributors to openstack/etcd3gw? I was wondering if this is open for community contributions. There are several issues in the associated launchpad project but it seems nobody from the project ever interacted with reported bugs in any way, despite even patches being available for some of them.13:06
GhostLyr_Meanwhile there have been commits to the git repo, so I'm assuming it's not dead or abandoned.13:07
fungiGhostLyr_: openstack's oslo team is responsible for that repository:
GhostLyr_thank you, will move the discussion to their channel. <bows out>13:15
corvusclarkb: i'm thinking of letting zuul restart on the current master state (which will be 9.5.0 when i push the proposed tag); then tomorrow merge the refactor, do the zk state reset, and restart again.  how's that sound?16:00
corvus("letting zuul restart" == the normal scheduled auto restart)16:01
clarkbthat seems reasonable to me. I do plan to review the rest of the refactor stack this morning. The big one was an easy rereview before breakfast but I'm going to sort out food before doing the others16:02
fungii haven't been reviewing the refactor changes, but seems reasonable to me too16:02
fungibased on what i know if the situation anyway16:02
clarkbits a fairly bit change internally, but the test cases should have us well covered on behavior changes16:03
clarkband most of the behavior changes should be limited to circular dependencies which we don't use16:03
clarkbamorin: Thank you!16:07
clarkbcorvus: the main thing would be ensuring that the refactor doesn't merge until the entire restart is complete. Otherwise we could automatically pull in too new zuul halfway through16:14
opendevreviewLajos Katona proposed openstack/project-config master: unmaintained: gerritbot notification for Networking
opendevreviewJeremy Stanley proposed opendev/ master: Switch the keycloak CNAME to the new server
opendevreviewJeremy Stanley proposed opendev/ master: Add DNS entries for another new Keycloak server
opendevreviewJeremy Stanley proposed opendev/system-config master: Update Zuul auth config for new Keycloak images
opendevreviewJeremy Stanley proposed opendev/system-config master: Add backups for the new Keycloak server
opendevreviewJeremy Stanley proposed opendev/system-config master: Clarify testinfra socket name for keycloak rdbms
opendevreviewJeremy Stanley proposed opendev/system-config master: Inventory entry for another new Keycloak server
fungiinfra-root: the new new keycloack replacement's changes are incorporated into the stack now, and i've updated with references to the updated changes17:40
clarkback I'll rereview when I'm through with this zuul refactor stack17:40
fungithanks! getting 908608 and 908609 merged asap will help me work out if there's anything else wrong that wasn't caught in testing17:40
fungibasically once 908609 deploys i should be able to start following the instructions from the zuul docs for configuring the realm17:41
clarkboh I can review those now then while I'm between zuul changes17:46
fungiyeah, it's just dns and inventory17:47
fungishould be very quick to check, hopefully17:47
fungieverything else was just rebased onto those (with a couple of edits to switch from the previous replacement to the newer one)17:48
clarkbyou can probably self approve those if frickler and corvus don't have time this morning. Its just a like for like replacement in existing changes to swap the new unused hosts out17:48
fungiright, and for a server which never saw production (never even successfully deployed)17:48
fungialso has probably been breaking other deploy jobs17:48
funginow that i think about it17:49
corvusi noticed that the centos-9 stream image builds are failing in nodepool and diskimage functional test jobs. i don't plan on digging into that, but thought i would mention it here in case other folks were interested17:49
corvussample failure ^17:50
corvusseems likely to affect prod too17:50
clarkbat first glance that looks like a bug in dnf17:50
fungithanks for the heads up. amoralej and apevec had worked out an updated mirror host for us, so maybe this is from a deferred package update that has now landed with the updated mirror content17:51
clarkbfungi: I'm not sure we use our mirrors in those jobs but I'm checking17:51
clarkboh hrm centos 8 fails with the same error. This might be a dnf bug in bookworm17:52
fungii have a feeling it's a version mismatch, yeah17:53
clarkbthis is the debootstrap like step where we jumpstart things17:53
clarkbso ya I don't think this is related to changes in our centos mirroring as this is a bookworm packge I think17:53
fungimakes sense, i'm looking to see what updated and when17:54
apevecah nodepool builder uses dnf from Debian17:54
fungiright, it's a debian container running diskimage-builder which then builds images for centos, ubuntu, suse, gentoo, whatever17:55
clarkbapevec: sort of. Nodepool is executing diskimage builder. But they are all installed on debian bookworm in the images we use so using dnf from there17:55
clarkbyou can install disk image builder elsewhere and it will use different dnfs in that case. But our installation is set up this way17:55
clarkbdigging through debian package changelogs I don't see any recent updates. But maybe I haven't found the correct package yet17:58
apevecso this is in debian, not centos root:17:59
apevec2024-02-08 14:32:50.604 |     self.command.configure()... (full message at <>)17:59
funginothing recent17:59
clarkbapevec: yes I think so. Because that command is running the download fetch of the initial chroot environment. Once that is done it switches to the chroot and uses the fedora/centos provided dnf18:00
clarkbfetching docker:// should allow us to inspect the bookworm installation directly18:01
fungithe tip of the backtrace is from a file in this package, but also hasn't changed in debian recently:
apevecyeah, on RHEL/CS path would be /usr/lib/python3.*/site-packages/...18:02
fungino relevant bugs reported (yet) in debian against either the dnf nor dnf-plugins-core source or binary packages18:04
funginor the python3-dnf binary package (which is what supplies dnf.util)18:08
clarkbis it possible that dnf is patching itself at runtime via the upstream? (that seems crazy but may be possible)18:11
apevecyeah, crazy it would be, at least in RPM it would break rpm --verify ...18:13
apevecto test the theory, what does dpkg --verify say?18:14
fungijust as a check, i can confirm that the current packaged python3-dnf in debian does not have a _is_file_pattern_present in its dnf.util, just playing around with it in the repl18:18
clarkbI'm struggling to download that container image with docker locally18:19
clarkbsays there is a 500 error somewhere but it isn't clear if dockerd or the remote registry is returning that18:19
fungiinterestingly, the /usr/lib/python3/dist-packages/dnf-plugins/ from dnf-plugins-core doesn't appear to have any calls to dnf.util._is_file_pattern_present() in my version18:20
clarkb you can see it installing dnf in this log18:21
clarkbcorvus: fyi I wasn't able to `docker run -it --rm thaturlabove bash` to fetch the artifact out of the registry. Not sure why yet but that may interest you. POssibly related to the version negotiation that skopeo is broke on too?18:23
clarkbfungi: apevec
fungiit looks like the DownloadCommand.configure() method in my copy doesn't attempt to check for file presence18:23
clarkbwe aren't using the package for dnf-plugins-core. I think we can clean that up. I'll work on a patch18:24
fungiversion mismatch it is18:24
fungigood find18:25
clarkbnow if debian package search would stop 503'ing I can confirm I have the correct package18:27
fungiyes, unfortunately is down, i was noticing that earlier18:27
fungithe database backend anyway18:28
fungii was going to ask about it in #debian-devel but figure people know and it will be up when it's up18:28
apevecahh so I guess dnf-plugins was git cloned b/c it was not packaged in Debian ?18:29
apevecgood catch clarkb++18:29
fungiit's packaged in debian, but maybe once upon a time it wasn't or something we needed wasn't included18:29
apevecif still not packages, we'd need to clone matching tag at least18:30
clarkbfungi: what is the new package name? debian is 503ing for me18:30
fungiclarkb: dnf-plugins-core18:30
fungii linked to the tracker page for it a ways up in scrollback too18:30
fungiwhich was my fallback since pdo wasn't working at the moment18:31
clarkbremote: Use Debian packaged dnf-plugins-core18:31
fungi+1 line, -11 lines. that's my kinda change18:31
fungiaha, so that was indeed a temporary hack 3 years ago while we were waiting for ftp-master to approve the dnf-plugins-core upload18:33
fungiprobably could have cleaned that up when we upgraded the images from bullseye to bookworm18:34
fungioh! actually it's not in bullseye, but it's in bullseye-backports18:36
fungiso we could have also used it with bullseye (not relevant now)18:38
fungiit was available in bullseye-backports as of 2023-01-17, roughly 5 months before bookworm was released18:40
fungibut still a couple of years after we initially needed it18:40
opendevreviewMerged opendev/ master: Add DNS entries for another new Keycloak server
clarkbya and this was missed when we bumped to bookworm18:43
clarkbmight be good to annotate things like that with a "should be fixed in bookworm" or similar to help our eyeballs catch them18:43
fungiagreed, if we came up with a standard we could even make stuff like that greppable so it's easier to spot when we're doing updates18:44
opendevreviewJay Faulkner proposed openstack/project-config master: ironic-unmaintained-core access to all Ironic proj
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Remove command.warn usage
clarkbfungi: does imply zypper and dnf-plugins-core are not coinstallable?19:38
clarkbwow it does thats I don't understand19:40
clarkb`apt show dnf-plugins-core` says Breaks: zypper Replaces: zypper19:40
clarkbthey should be completely independent tools19:41
corvusclarkb: i took one step in debugging registry:
clarkbdnf and zypper use the same dep solver iirc19:41
corvusclarkb: (first part of the paste is the traceback; the second part is a python repr of the contents of that object in storage)19:42
corvus(obtained out of band)19:42
clarkbcorvus: using a new datastructure? I wonder if that is related to the version changes that skopeo hit then19:42
opendevreviewMerged opendev/system-config master: Inventory entry for another new Keycloak server
clarkbI think there may be a way to force docker to use the older version and we can use that to see if it resolves if so19:42
corvusclarkb: could be; i'm unsure and have not paged everything back in19:42
corvusclarkb: though i have a hunch this may relate to multi-arch image formats, and zuul-registry may need an update to handle them19:45
clarkboh i thought it already did?19:46
clarkbmaybe not in the way that docker wants it to though19:46
corvusyes that19:47
clarkbfungi: `apt-file list zypper` and `apt-file list dnf-plugins-core` don't show any overlap in files. I'm not sure how dnf-plugins-core can break zypper. Seems like a bug in the package for dnf-plugins-core?19:47
corvusclarkb: podman run that image works19:47
corvuspodman run --rm -it
clarkbgot it so protocol level issues likely rather than any true problem19:48
clarkbremote: Fetch compatibile dnf download command in container image19:57
clarkbthis makes me sad. I really wonder why the package would replace and break zypper. There is no zypper plugin in the plugins list that I can see.19:57
clarkbthe changelog for dnf-plugins-core says the breaks zypper was added to fix 102808519:59
clarkbok they conflict for the needs restarting manpage20:01
clarkbbut dnf-plugins-core doesn't seem to install that command?20:01
clarkboh it is a dnf plugin20:02
clarkbI feel like there has to be a better way to address this problem. Maybe the manpage should be dnf-needs-restarting.gz.1 ?20:03
clarkbas far as I can tell you can't execute it without the dnf prefix but I'm installing things now to check20:04
corvusthe containerfile element was designed to avoid bootstrapping issues; maybe one or more of these images should switch to using that?20:05
clarkbya `man needs-restarting` works but `needs-restarting` is container not found.20:05
corvus(just brainstorming/reminding; i haven't thought about the particulars)20:06
clarkbcorvus: ya I think that is a longer term posibility. I think the debian package is broken too imo but I don't now what the rules are for man 1 presence for commands that don't actually exist20:06
corvusclarkb: container not found?20:06
clarkbsorry command not found20:06
clarkbbash: needs-restarting: command not found20:06
corvusclarkb: oh you have to docker your docker before you docker20:07
clarkbzypper acutally installs a /usr/bin/needs-restarting and installs a needs-restarting.gz.1 file. dnf installs a plugin you can run as `dnf needs-restarting` then installs a manpage for needs-restarting.gz.120:07
clarkbmy "I don't actually know debian rules for this stuff" opinion is that the dnf-plugins-core package is buggy and it should not install all these manpages for dnf subcommands at the root manpage space20:08
corvusbased on only that sentence, i agree that sounds like the dnf package is not doing something right.20:08
clarkbusing git as an example beacuse it has many subcommands all of those commands are git-foo.1.gz20:10
clarkbfungi: I suspect you know the rules and conventions and I'll defer to you on whether or not we should complain and file a bug20:11
fungiclarkb: i have a feeling it's that red hat's and suse's rpm tools can't coexist21:06
clarkbfungi: the only conflict is a manpage though21:06
clarkbsure they can't exist if the manpages have to be in their current locations. But nothing about the tools themselves conflcit aiui21:06
clarkbyou can even install them together on tumbleweed21:07
clarkbdnf and dnf-plugins-core are valid pacakges and I already have zypper installed21:07
clarkband zypper info --conflicts for both don't show zypper conflicting.21:08
fungioh that's funny21:09
fungigoing to see if i can tell when/why the conflict was added. normally package conflicts shouldn't be used for unrelated tooling21:10
clarkbfungi: the changelog has it21:10
clarkb is related though it seems like they were already making the update before that issue was filed?21:10
clarkbthe tumbleweed package has a deconflicts clause in the specifle
clarkbmv %{buildroot}%{_mandir}/man1/needs-restarting.1 %{buildroot}%{_mandir}/man1/dnfutils-needs-restarting.121:12
clarkbso ya tumbleweed is doing what I suggest debian do above21:13
fungiit's in section 1, so in theory is the manpage for a command called dnfutils-needs-restarting21:17
fungibut if both packages ship that manpage, it implies they both also ship that executable?21:17
fungiexcept it sounds like they don't, so one or the other is probably including that manpage in error21:18
clarkbthe manpage that conflicts is for `needs-restarting`. On debian only zypper installs this in /usr/bin/ to use this command with dnf on debian you have to run `dnf needs-restarting`21:20
clarkbon tumbleweed it looks like with conflicts they both install a binary for needs-restarting and when deconflicted it gets moved to dnfutils-needs-restarting21:20
clarkbthe debian package includes a number of dnf plugin subcommands as top level manpages that don't actualyl appear to get installed as top level binaries21:21
fungioh weird21:21
fungiyeah that's an interesting corner case21:21
clarkbmy "never built a distro" naive take is that none of those manpages in the dnf-plugins-core package should lack dnf- prefixes21:22
clarkbbecause none of them are installed as actual commands and instead they are `dnf foobar` commands21:22
fungigit sets the same precedent for its subcommands though21:23
clarkbah yup tumbleweed will install a shim that checks $0 and acts accordingy for subcommands if you run them directly21:23
clarkbfungi: git does the thing that would fix this21:23
clarkbthey are all git-foo not foo21:23
fungii mean git sets the same precedent for manpages on its subcommands21:24
fungiman git-commit21:24
fungiet cetera21:24
clarkbthere is a /usr/libexec/dnf-utils-3 on debian that I wonder if it does something weird21:24
clarkbliek make needs-restart work somehow but `which` couldn't find it so its not using PATH21:24
clarkbals /usr/share/man/man8/dnf-needs-restarting.8.gz exists21:25
clarkbin any case there shouldn't be anything that actually prevents these two packages from installing next to each other as tumbleweed packaging shows. But it may require moving a manpage out of the way21:26
clarkbtesting again I can't run `needs-restarting` on bookworm directly. So no magic happening as far as I can tell21:27
clarkbyou'd need something like suse's dnf-utils to interpret the command and fork out to the correct subcommand21:28
clarkbfungi: I think I'm caught up on keycloak reviews. Let me know if I missed one21:29
clarkb`mv %{buildroot}%{_libexecdir}/dnf-utils-3 %{buildroot}%{_libexecdir}/dnf-utils` from the tumbleweed spec. So that /usr/libexec/dnf-utils-3 tool is the glue but it isn't installed to do the gluing as far as I can tell21:33
clarkb(I actually installed the packages on the container in case some packaging scripts did it but doesn't appear to)21:33
clarkbthis is likely to be a very lightly used set of packages so probably not urgent upstream if willing to address ti at all. And we've got a reasonable workaround for now I think21:40
fungihot take on #debian-devel is that it's a policy violation to add a conflicts to a package rather than rename or omit the conflicting file21:40
clarkboh itneresting21:41
fungiconflicts is intended to be reserved for packages that absolutely cannot be used on the same system21:41
fungiwhich was also my vague recollection, so seems corroborated at least21:50
fungiclarkb: there's also the cname change, not at all urgent:

Generated by 2.17.3 by Marius Gedminas - find it at!