Friday, 2023-08-11

*** dmellado819 is now known as dmellado8106:55
opendevreviewDr. Jens Harbott proposed zuul/zuul-jobs master: Add testing for Debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89114707:02
opendevreviewDr. Jens Harbott proposed opendev/base-jobs master: Add standard Debian Bookworm nodesets  https://review.opendev.org/c/opendev/base-jobs/+/89114807:20
fricklerinfra-root: ^^ seems we (I) missed some steps when adding bookworm, please check07:21
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Remove ocata related definitions  https://review.opendev.org/c/openstack/project-config/+/82537207:30
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Remove publish-install-guide ocata job  https://review.opendev.org/c/openstack/project-config/+/82537407:30
*** amoralej is now known as amoralej|lunch13:11
*** amoralej|lunch is now known as amoralej13:56
frickleranyone interested in fixing base-jobs tox-docs job? https://zuul.opendev.org/t/opendev/build/87fcfb58c23b4698ab6680bc0c85bb7014:31
fricklerI've bumped the nodepool create_image timeout on nb02, too, and am testing there two parallel uploads now14:37
fricklermy idea would be that we unpause just the most used images if this works and delete the others, to keep some semi-stable situation there until we can get feedback from rax14:38
frickleror we just disable that region. given that 90% of our quota there is permanently filled with stuck-deleting instances, it likely wouldn't matter much either way14:39
fungioh, are there that many stuck instances? i didn't realize it was so bad14:44
fungii hadn't looked and assumed you were talking about only a handful14:44
fricklerabout 120 out of 140 both in dfw and iad, not really 90% but close https://grafana.opendev.org/d/a8667d6647/nodepool-rackspace?orgId=1&from=now-1y&to=now14:47
fungiouch15:00
fungion the base-jobs failure, we're getting "'ImageDraw' object has no attribute 'textsize'" which looks like a problem with pillow>=1015:02
fungihttps://pillow.readthedocs.io/en/stable/deprecations.html#font-size-and-offset-methods15:02
fungiapparently seqdiag needs fixing15:04
fungioh, no it's blockdiag15:09
fungihttps://github.com/blockdiag/sphinxcontrib-blockdiag/issues/2615:10
fungiah, yeah, johnsom proposed https://github.com/blockdiag/blockdiag/pull/171 months ago15:11
fungiwe should probably pin back to pillow<10 for now until we decide on something more thorough15:11
fungii should look at what zuul-jobs did since i think the same diagram exists there too15:11
johnsomYeah, it seems to be an abandoned project. Fedora is likely going to drop the module if they haven't already.15:11
johnsomSo, I have transitioned both designate and octavia off of blockdiag15:12
fungizuul-jobs did the obvious of pinning pillow too apparently: https://review.opendev.org/88802115:12
fungii'll follow suit in base-jobs for the time being and leave some breadcrumbs15:13
fungithanks johnsom!15:13
Clark[m]https://github.com/python-pillow/Pillow/issues/7277 is a good breadcrumb too15:15
opendevreviewJeremy Stanley proposed opendev/base-jobs master: Pin Pillow<10 for blockdiag incompatibility  https://review.opendev.org/c/opendev/base-jobs/+/89120815:17
fungiClark[m]: i linked to the pr, but it links back there too15:17
johnsomEvidently there are other incompatibilities beyond the one I fixed, but given nothing has merged in blockdiag for so long, I didn't think it was worth working through all of the others.15:17
fungifrickler: ^15:17
fungijohnsom: makes sense15:18
clarkbjohnsom: and you said you were prerendering images?15:21
clarkbwe can probably get away with that here too since it is a single image the svg shouldn't be too complicated15:22
johnsomI switched to basic graphvis15:22
johnsomThey aren’t as nice, but work for the ~6 diagrams I had15:23
fungigonna go grab lunch, bbiab15:27
clarkbin this case I think we have a single graph in base-jobs. Not sure how many are in zuul-jobs. But ya replacing them with an inkscape svg might be an easy workaround15:28
clarkblooks like zuul uses a bunch of blockdiag:: sections though15:29
clarkbthey don't seem to be breaking which means maybe the graph content determines the behavior here?15:29
clarkbinfra-root we received a notification from google about their new inactive accounts policy and deletion to occur on December 1. I didn't realize that email addr had an associated google account. Does anyone know what that may be used for? I'm wondering if we need to preserve it via regular logins or something15:32
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Add testing for Debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89114715:37
clarkbapparently ca-certificates-java is broken on bookworm. Specifically the package depends on files produces by the jre/jdk packages but the jre/jdk packages also depend on ca-certificates-java and package installs fails15:45
clarkbthis is impacting the nodepool openshift test platform update because we need to install java for zookeeper there, but it will also impact our planned move of the gerrit container to bookworm.15:46
clarkbThere is a fix in testing but it still hasn't made its way back to stable15:46
clarkbhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1023748 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=103541615:46
clarkbit is interesting to me that they mark bugs as fixed before they actuall fix stable...15:47
clarkbtonyb: ^ fyi15:51
*** ykarel is now known as ykarel|away15:59
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Mark nim jobs unmaintained  https://review.opendev.org/c/zuul/zuul-jobs/+/89121116:12
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121216:16
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Remove fedora-36 jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121516:33
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add testing for Debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89114716:34
fricklerclarkb: oh, I saw that mentioned in the ceph reef release announcement16:35
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121216:38
opendevreviewMerged opendev/base-jobs master: Pin Pillow<10 for blockdiag incompatibility  https://review.opendev.org/c/opendev/base-jobs/+/89120816:41
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121816:41
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122116:49
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122216:53
opendevreviewMerged zuul/zuul-jobs master: Mark nim jobs unmaintained  https://review.opendev.org/c/zuul/zuul-jobs/+/89121117:02
opendevreviewMerged zuul/zuul-jobs master: Remove fedora-36 jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121517:03
fungiclarkb: does zuul not also pin pillow like zuul-jobs does?17:09
clarkboh so it does. My local checkout was out of date when I checked the first time17:10
fungiclarkb: in debian, "fixed" and "fixed in stable" are separate (and bugs remain open while there is still an affected release)17:10
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: Update ensure-ghc to use ghcup  https://review.opendev.org/c/zuul/zuul-jobs/+/84711117:10
clarkbfungi: gotcha. I guess the next round of confusion is why it takes months to backport package fixes for installation of a major language runtime. But iirc they bundle up updates and don't do single package updates17:13
fungiwell, fixes for stable get uploaded to either security or stable-proposed-updates depending on the nature of the patch (or backports, or backports-sloppy...)17:16
fungithings in security and spu do get rolled up into periodic point releases every couple of months, but can be consumed directly from their respective suites before that happens17:16
fungithe batching of proposed updates is arguably a vestigial step from the days when physical media were more prevalent17:18
fungiused to be you'd make media with the stable point releases and then use those to update your machines with lower-priority fixes so you didn't have to always be pulling those over your network17:19
clarkbbut the packages don't land in stable/ until that point release right?17:20
clarkbI'm guessing that is the sort of thing we are waiting on here?17:21
clarkboh maybe we don't have backports enabled in our VM images though17:21
opendevreviewMerged opendev/base-jobs master: Add standard Debian Bookworm nodesets  https://review.opendev.org/c/opendev/base-jobs/+/89114817:22
clarkbthe container image I'm trying to reproduce in has Suites: bookworm bookworm-updates enabled17:22
clarkband everything works there17:23
clarkband we do configure updates in configure-mirrors as well so our VMs should have that too17:23
fungibackports and stable will be different. stable gets patches backported to it, backports gets new versions backported to it17:25
clarkbok in this case it would be a stable update then since the packaging not the upstream is broken I think17:26
clarkboh its an order of operations issue17:28
clarkbin the docker.io/debian:bookworm container I'm trying to reproduce in it runs `Setting up openjdk-17-jre-headless:amd64 (17.0.8+7-1~deb12u1) ...` before `Setting up ca-certificates-java (20230103) ...` and everything works17:28
clarkbin the failed CI jobs they go in the inverse order and it breaks17:28
clarkbwe need the java runtime to be set up first. But is that fixed because the packages are updated somewhere and we don't see that or is it somewhat random depending on the machine?17:29
clarkbthe package versions don't differ which makes me think it is somewhat random rather than package updates. unless they updated the packages without bumping versions (seems unlikely)17:29
clarkbya https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030129 suggested a race or ordering issue and other people indicate they need specific combos of packages to trigger it.17:35
clarkbI'm guessing we don't install suggested packages which causes just ca-certificates-java to fail in the VM but doesn't create that problem on the test container. Let me see if I can reproduce on the container via disabling suggets/recommends17:35
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: Update ensure-ghc to use ghcup  https://review.opendev.org/c/zuul/zuul-jobs/+/84711117:38
clarkbfungi: any ideas on the best way to address this? I haven't been able to come up with a package set in a container that triggers the failure but the bug seems to indicate the specific package list is important to reproducing. I'm wondering if we should just install ca-certficates-java from testing17:48
fungiif that will work for now, it seems like the most straightforward workaround17:49
fungiwith a comment pointing to the request for getting it backported17:50
clarkbok let me try that17:51
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121218:02
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121818:02
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122118:02
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122218:02
clarkbNo package matching 'ca-certificates-java/testing' is available <- I guess I need to explicitly enable that repo18:12
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121218:19
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121818:19
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122118:19
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122218:19
clarkbthis is quickly becoming ugly18:19
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: Update ensure-ghc to use ghcup  https://review.opendev.org/c/zuul/zuul-jobs/+/84711118:28
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121218:33
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121818:33
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122118:33
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122218:33
clarkbthats interesting it almost looks like the python-apt library or is it the apt_repository ansible module can't read bookworm sources.list format?19:08
clarkbapt_pkg.Error: E:Malformed entry 1 in list file /etc/apt/sources.list.d/deb_debian_org_debian.list (Component), E:The list of sources could not be read.19:09
clarkbfungi: it looks like the bookworm container has Components: main in its sources file. Is this a new format?19:11
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121219:12
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121819:12
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122119:12
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122219:12
clarkbmaybe limiting apt_repository to a specific file will keep it from reading those extra files and being upset about them19:13
fungiclarkb: yeah, not technically new (the multi-line sources format has existed for a release or two) but newly default19:14
fungiboth formats should be supported, but it will try to combine everything19:15
clarkbya so python-apt or whatever the lib is must not support the other newer format19:15
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: Update ensure-ghc to use ghcup  https://review.opendev.org/c/zuul/zuul-jobs/+/84711119:18
clarkbhttps://zuul.opendev.org/t/zuul/build/c70315dd014247238f3ca87c4b7fc464/console#1/0/2/debian-bookworm specifying the file did not fix it either. I think the bindings are producing a new style file in the C++ lib and the nwhen it tries to cache it back into python it doesn't understand that format and breaks19:22
clarkbgitlab doesn't sort tags in a way that makes sense to me. Also there is no python-apt tag for 2.6.0 which appears to be the version in bookworm...19:28
clarkbI'm really confused by this19:28
clarkboh maybe the controller aka the bullseye zuul ansible is what is being used not hte bookworm python-apt19:31
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121219:35
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121819:35
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122119:35
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122219:35
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121219:41
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121819:41
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122119:41
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122219:41
clarkbok I think it was actually trying to tell me I hadn't supplied the component...19:41
clarkbbut malformed entry component isn't quite the same as "no componenbt provided" ... lets see if this is happier19:41
fricklercorvus: clarkb: I have a zuul config question in https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/891227/comment/4b8975f4_275fba7d/19:47
Clark[m]frickler: I think you have to define two variants of the job in the template for that. One with voting for branch A and the other non voting for branch B19:51
gmannfrickler: Clark[m] : testing in https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/89123119:54
fricklerClark[m]: but can you do that with the same job name? I think that the result wouldn't be valid yaml19:54
Clark[m]Oh good point. Ya pyyaml would only pick one I guess19:57
opendevreviewTristan Cacqueray proposed zuul/zuul-jobs master: Update ensure-ghc to use ghcup  https://review.opendev.org/c/zuul/zuul-jobs/+/84711120:03
Clark[m]fungi it still can't find the package in 891212. Can you take a look and see if I did anything obviously wrong?20:06
fungilooking20:20
fungiClark[m]: see comment on the change, that's my hunch at least20:27
fungiyou have to update the package cache any time you add new sources, or else they won't actually be indexed20:27
fungiwe explicitly call it in configure-mirrors, for much the same reason20:28
fungii'll push up a change20:45
clarkbfungi: apt_repository is supposed to update caches by default20:48
clarkbbut maybe it isn't we can be explicit about it20:48
fungii didn't see it mentioned in the output20:48
clarkbhttps://docs.ansible.com/ansible/latest/collections/ansible/builtin/apt_repository_module.html#parameter-update_cache20:48
fungiaha!20:48
fungisays default is true?20:49
clarkbyes20:49
fungimaybe it doesn't echo?20:49
clarkbya the module could be eating it. Or the docs could b wrong20:50
fungior it dereferences the release identifier and makes it need ca-certificates-java/trixie20:50
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121220:50
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121820:50
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122120:50
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122220:50
fungiwithout seeing the outpyt from apt-get update in the job, it's hard to know what it actually added20:51
clarkbthat is an attempt at being very explicit about updating caches20:51
clarkbfungi: you think maybe apt_repository is translating the source from testing to trixie? similar to how it translates it form a single line to the multiline deb822 format?20:54
clarkbSeems possible. Also that pacakge does seem to be in main so I don't think I need to add contrib or whatever20:54
fungiright, it may be some odd combination of what apt_repository is adding for configuration and what the package module looks for20:55
clarkbbut I'm trying to double check that now20:55
clarkbya if I go to debian package search and get a deb file download the path it gives me is under main20:56
fungithe error message seems to come from the package module in ansible, since apt-get would say "E: Unable to locate package ..." instead20:56
fungiso it may be that ansible is more picky about use of release aliases20:57
clarkbok, I'll try /trixie next and update the repository specification to match20:57
clarkbassuming it fails20:57
clarkbit failed with the same error. Trying trixie20:58
clarkband if that doesn't work we may need to hold a node...20:59
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121220:59
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121820:59
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122120:59
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122220:59
clarkb"No package matching 'ca-certificates-java/trixie' is available"21:03
fungiwe have used / notation for specifying a release with the ansible package module in the past, right?21:28
clarkbnot sure21:32
clarkbI've rechecked to hold a node21:32
clarkb`apt-get install ca-certificates-java/trixie` just worked on the held node without doing anything else...21:37
clarkbmaybe the apt module is the problem21:37
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121221:39
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121821:39
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122121:39
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122221:39
clarkbI really hope that isnt it21:39
clarkbugh that was it21:49
clarkbI need a break then will clean this up21:49
* fungi sighs21:56
fungisometimes it feels like ansible modules cost us more time than if we just told it to call shell commands21:57
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Fix zookeeper install on debian bookworm  https://review.opendev.org/c/zuul/zuul-jobs/+/89121222:00
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Remove fedora-36 container jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89121822:00
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Auto-generate ensure-podman jobs  https://review.opendev.org/c/zuul/zuul-jobs/+/89122122:00
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Run more ensure-skopeo tests  https://review.opendev.org/c/zuul/zuul-jobs/+/89122222:00
clarkbok I feel like this version is reviewable22:00

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!