| clarkb | corvus: there was a small syntax error in that chnage so I went ahead and fixed ti so we can get test results back | 00:33 |
|---|---|---|
| corvus | oh thx | 00:42 |
| clarkb | corvus: I did have a couple small questions on the change. I think it should work but I'm still trying to wrap my brain around the behavior there | 00:42 |
| clarkb | but I need to eat dinner now so no rush | 00:42 |
| clarkb | also zk01-03 seem to be stable in count of connections and rough znode numbers. I think the next big event is periodic jobs enqueing but I may have to followup on that tomorrow morning | 00:44 |
| corvus | clarkb: replied | 00:45 |
| clarkb | thanks I updated to +2 | 00:47 |
| mnasiadka | corvus: yup, it has depends-on on a DIB change i dug out that should help with retries (of course after some modifications from what I see) | 04:06 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/c/openstack/diskimage-builder/+/721581 | 04:33 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 08:06 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 08:07 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 08:13 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 08:13 |
| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: limit-log-files: allow unlimited files https://review.opendev.org/c/zuul/zuul-jobs/+/953854 | 08:21 |
| opendevreview | Rodolfo Alonso proposed openstack/project-config master: Remove the fullstack job from the master dashboard https://review.opendev.org/c/openstack/project-config/+/953856 | 08:36 |
| priteau | Hello. Do you know what is required to make this patch move out of "Ready to submit" state? https://review.opendev.org/c/openstack/bifrost/+/948245 | 09:35 |
| mnasiadka | clarkb, corvus: I think https://review.opendev.org/c/opendev/zuul-providers/+/953269 should be fine now (with the depends-on), although I see there are some other problems (disk full?) | 09:47 |
| frickler | priteau: I answered in the ironic channel yesterday: it needs a rebase, the status that gerrit shows is wrong | 10:29 |
| frickler | infra-root some review on https://review.opendev.org/c/openstack/diskimage-builder/+/951469 would be nice to allow to build testing/trixie images | 10:30 |
| frickler | mnasiadka: looks like there is another error to check for: "Empty reply from server", I wonder whether we should simply retry any failure? https://zuul.opendev.org/t/opendev/build/b47ca097c4c54a799cf68305a7c27ce5 | 10:32 |
| frickler | and yes, this looks pretty full "/dev/vda1 78062860 73215660 1268820 99% /". maybe a good chance to test the z-l autohold :) | 10:35 |
| frickler | 24G for /opt/git sounds like a lot to me, I had something less than 20G for total image size in memory. | 10:39 |
| mnasiadka | frickler: might make sense, I'll have a look later | 10:53 |
| priteau | frickler: I missed your reply, thanks | 11:13 |
| opendevreview | Merged openstack/project-config master: Remove the fullstack job from the master dashboard https://review.opendev.org/c/openstack/project-config/+/953856 | 11:24 |
| jrosser | when does the ubuntu mirror sync run? there are some broken dependencies on qemu-block-extra in the CI mirrors which are (no longer?) present in the upstream mirror | 12:34 |
| jrosser | (some discussion in #openstack-ansible and #openstack-nova about broken jobs related to that) | 12:35 |
| fungi | jrosser: it updated an hour ago | 12:40 |
| fungi | https://static.opendev.org/mirror/ubuntu/timestamp.txt | 12:40 |
| jrosser | so this http://mirror.iad3.openmetal.opendev.org/ubuntu/dists/noble-updates/main/binary-amd64/Packages is still showing what is different content for qemu-block-extra | 12:41 |
| fungi | "different" compared to... | 12:41 |
| jrosser | different from the equivalent thing at archive.ubuntu.com | 12:42 |
| jrosser | which i think is this https://archive.ubuntu.com/ubuntu/ubuntu/ubuntu/dists/noble-updates/main/binary-amd64/Packages.gz | 12:42 |
| jrosser | the depends on `librbd1 (>= 19.2.1-0ubuntu0.24.04.1)` seems to be wrong | 12:43 |
| fungi | reprepro is pulling the packages from http://us.archive.ubuntu.com/ubuntu per https://opendev.org/opendev/system-config/src/commit/5a27433/playbooks/roles/reprepro/files/ubuntu/config/updates#L2 | 12:44 |
| fungi | i'll check the logs | 12:44 |
| jrosser | us.archive.ubuntu.com shows librbd1 (>= 19.2.0-0ubuntu0.24.04.2) as a depends for qemu-block-extra, so that looks OK | 12:49 |
| fungi | looks like the packages file on us.a.u.c last updated at 11:34 utc, we finished pulling from it about 10 minuts before that | 12:50 |
| jrosser | i think that this has been OK using the upstream repos for at least the last 6 hours | 12:52 |
| fungi | well, it may be that us.archive.ubuntu.com was lagging behind other mirrors | 12:52 |
| jrosser | ah right, ok, that makes sense | 12:52 |
| fungi | so if you weren't pulling from the same one we're mirroring from, you might have gotten different results | 12:52 |
| fungi | it looks like there's another mirror pull in progress, started about 40 minutes ago, so should have the new content shortly | 12:54 |
| fungi | i wonder if ubuntu's doing a point update for noble today or something | 12:58 |
| fungi | because these syncs have been really slow | 12:59 |
| fungi | would also explain the upstream mirrors being somewhat inconsistent | 12:59 |
| jrosser | i think that this is some kind of error | 13:04 |
| fungi | could be, but if so it seems like it may have affected a lot of packages because reprepro mirror pulls are taking over an hour to complete, or at least the last one did | 13:06 |
| fungi | normally they're on the order of a few minutes | 13:06 |
| fungi | maybe they accidentally dumped plucky package data into noble or something | 13:07 |
| opendevreview | Merged openstack/project-config master: Replace OpenStack's CLA enforcement with the DCO https://review.opendev.org/c/openstack/project-config/+/950998 | 13:10 |
| jrosser | there does seem to be some evidence of ceph 19.2.1 being built https://archive.ubuntu.com/ubuntu/pool/main/c/ceph/ | 13:11 |
| fungi | yeah, it's just acting like there's a ton of package churn in the noble packages today | 13:11 |
| stephenfin | fungi: clarkb: (discussing here since clarkb isn't on #openstack-oslo) do we have a list of things pbr does that we want/need to keep written down anywhere? | 13:14 |
| stephenfin | asking before I start one | 13:15 |
| opendevreview | Merged openstack/project-config master: Replace StarlingX's CLA enforcement with the DCO https://review.opendev.org/c/openstack/project-config/+/953819 | 13:16 |
| opendevreview | Merged openstack/project-config master: Replace Airship's CLA enforcement with the DCO https://review.opendev.org/c/openstack/project-config/+/953849 | 13:16 |
| fungi | stephenfin: do pbr's docs count? | 13:18 |
| fungi | or are you asking for a separate document that lists the pbr features we know projects are using? | 13:18 |
| stephenfin | 'ish. I think we're missing docs for things like the custom scriptwriter | 13:19 |
| fungi | and when you say "things pbr does" you mean pbr features or undocumented behaviors? (or both?) | 13:19 |
| fungi | ah | 13:19 |
| fungi | i thought we documented that it creates wsgi scripts and such, but i'll check | 13:20 |
| fungi | you're right, the features chapter of the document is rather short | 13:21 |
| stephenfin | yeah, and it focuses more of the setup.cfg file and its configuration (likely because that's all I knew about when I wrote it 😅) | 13:23 |
| stephenfin | https://etherpad.opendev.org/p/pbr-setuptools-future | 13:23 |
| fungi | the pbr.packaging.generate_script() method is covered in the auto-generated api docs, but that's about it | 13:23 |
| stephenfin | I'll start jotting down stuff there so | 13:23 |
| fungi | thanks! | 13:23 |
| stephenfin | we don't need to get this done today, but I think it would be a good first step is working with the setuptools folks | 13:25 |
| stephenfin | ideally we should learn from each other and either fix things in setuptools (to avoid the need to have pbr provide its own implementation), add a hook point to setuptools, or realise a feature no longer makes sense | 13:26 |
| Clark[m] | stephenfin: off the top of my head the PBR scripts do faster process startup because they avoid pkg_resources/importlib. The setuptools version would work but slower. Then there are the wsgi scripts which setuptools doesn't do at all. PBR's commit message version bump tags are something that Openstack uses. The inclusion of git sha information in package metadata is another important feature that setuptools and others don't do. I think those | 13:36 |
| Clark[m] | may be the big ones. Setuptools-scm will get you close otherwise | 13:36 |
| Clark[m] | PBR has also been 100x better at backward compatibility than setuptools. I think that is more of a culture problem than a technical one. For some reason Pypa doesn't want to keep trivial to maintain code around even if it prevents breaking their users | 13:38 |
| fungi | they see forcing users to drop upstream-eol python versions as a feature, not a bug | 13:38 |
| fungi | doing them a favor by saving them from themselves running code with likely security vulnerabilities | 13:39 |
| Clark[m] | It's not just that though. The whole _ vs - debacle served no purpose other than to break existing packages | 13:41 |
| Clark[m] | And many packages built for old python continue to work for new python. The only exception is if you aren't updating every 6 months to fix the next setuptools breakage | 13:41 |
| fungi | the filename normalization change ostensibly prevents future ambiguous package files from entering the archive (ambiguous in the sense that the dist name and classifiers/version are clearly separated now) | 13:43 |
| fungi | but yes, it's arguable whether the churn was justified | 13:43 |
| Clark[m] | They did it in setup.cfg keys too and broke a bunch of stuff aiui | 13:45 |
| Clark[m] | PBR has a whole translation layer protecting its users (that stephenfin write) | 13:46 |
| Clark[m] | More generally the risk with just using setuptools is you get on their treadmill of breaking changes. PBR insulated its users from much of that | 13:46 |
| Clark[m] | Because we can shim compatibility into PBR rather than into every repo/package | 13:47 |
| Clark[m] | So that's not really a feature of PBR itself more of a shortcoming of setuptools but probably still worth calling out in a discussion of "this is why we have PBR" | 13:47 |
| fungi | a big part of the setuptools maintainers' argument for that sort of churn is that they lack sufficient bandwidth to keep all of the project maintained, so are trying to actively purge older functionality in order to keep the maintenance burden down, with the idea that projects that need the old features can just use old setuptools versions instead | 13:52 |
| fungi | though i think they see it as "projects that are effectively abandanware, or otherwise too lazy to keep updated to the latest packaging standards" | 13:53 |
| fungi | abandonware | 13:53 |
| fungi | jrosser: the latest mirror pull didn't resolve it. one thing i'm noticing is that the version reprepro is holding onto that depends on librbd1 from plucky is qemu-block-extra 1:8.2.2+ds-0ubuntu1.8 while the one with the working dependency on the official mirrors is qemu-block-extra 1:8.2.2+ds-0ubuntu1.7 | 14:13 |
| fungi | i wonder if they "rolled back" by pulling the 0ubuntu1.8 build from the archive, and reprepro won't downgrade to that now because it thinks it's older (since it is) | 14:14 |
| fungi | servers that installed -0ubuntu1.8 are similarly not going to auto-downgrade to -0ubuntu1.7, so it may be that they need to do a new bump of the build version for everything to straighten out | 14:15 |
| fungi | i expect it's possible to force reprepro to forget qemu-block-extra so it refetches the older version, but will take some reading of the docs to figure out | 14:16 |
| fungi | (and i won't have time to get to that right away, since today is "dco day") | 14:17 |
| clarkb | thats a fun package mirror scenario if that is what happened. I wonder why they didn't revert by rolling the version forward with old content | 14:39 |
| clarkb | if we think that is what happened it may be worth pinging ubuntu about it? as you say people who already installed the package won't downgrade this way | 14:41 |
| fungi | it may be that they thought it would be fine because the package was essentially uninstallable, depending on a version of librbd1 that wasn't available | 15:00 |
| clarkb | fungi: when you get a chance https://review.opendev.org/c/opendev/system-config/+/953783 would be a good one for you to look at as it adds a new mailing list | 15:13 |
| fungi | ah, yeah, i saw that but then forgot in my rush to get other stuff done yesterday | 15:17 |
| clarkb | thanks | 15:23 |
| stephenfin | fungi: clarkb: Another pbr question to follow. FYI these are not urgent: I just don't want to waste time if you've already gotten to the bottom of something... | 15:28 |
| stephenfin | I'm trying to fix the remaining issue on https://review.opendev.org/c/openstack/pbr/+/953839 just to unblock that gate. fwict, the issue is that setuptools changed how they normalise their package names between v79.0.0 and v79.0.2 and I'm guessing that breaks this code where we drop a package from the constraints list | 15:29 |
| stephenfin | https://github.com/openstack/pbr/blob/5d4a1815afa920cf20e889be20617105446f7ce2/pbr/tests/test_integration.py#L101-L112 | 15:30 |
| fungi | they changed the dist name normalization in metadata? | 15:31 |
| fungi | what's the actual change in setuptools? | 15:31 |
| fungi | just making sure it's not being conflated with the change to sdist filename normalization | 15:31 |
| stephenfin | sec, lemme grab links | 15:31 |
| stephenfin | I'm looking at 'git diff v75.8.0..v75.8.2', since the job started failing on 2025-02-26 and setuptools cut a release that day (neither virtualenv nor pip did) | 15:32 |
| stephenfin | https://github.com/pypa/setuptools/compare/v75.8.0..v75.8.2 | 15:33 |
| stephenfin | both pkg_resources and setuptools have normalization-related changes there. Nothing else looks like it could be related. The test is failing due to constraints mismatches | 15:34 |
| clarkb | stephenfin: we'er writing too many packages to the constraints file? | 15:35 |
| clarkb | because they don't match? | 15:35 |
| stephenfin | I'm thinking (still working on a fix) that we're using upper-constraints.txt and we're also setting PBRVERSION=0.0, so build packages have version 0.0.0, and constraints say they need X.Y.Z | 15:37 |
| stephenfin | and it only fails on glance-store (again, a guess) because the name in the package's setup.cfg differs from what's in upper-constraints.txt | 15:37 |
| clarkb | stephenfin: and that quoted block of code you linked is there to ensure those projects are not in constraints I think so that 0.0 can be installed? | 15:38 |
| clarkb | so ya if the names don't align we'd keep adding them to constraints then have a conflict | 15:39 |
| stephenfin | yeah, I think that's the idea. I'm just wondering why it broke with v79.0.1 or .2. We didn't have anything else normalization-related like this pop up earlier this year, did we? | 15:39 |
| clarkb | just the sdist names and the setup.cfg entry keys that I can remember | 15:40 |
| fungi | we switched to using pyproject-build instead of running setup.py directly in release jobs around that time, so could maybe be a difference in resulting metadata for that package | 15:40 |
| fungi | stephenfin: glance-store 4.9.1 was released on 2025-02-21... lemme see when that propagated to upper-constraints.txt | 15:42 |
| stephenfin | oh, I didn't think to look at glance-store itself | 15:43 |
| opendevreview | Merged opendev/system-config master: Add a summitspeakers@lists.openinfra.org mailing list https://review.opendev.org/c/opendev/system-config/+/953783 | 15:43 |
| stephenfin | it's only that out of the ~65 packages we test that is failing, which is what's so odd | 15:44 |
| fungi | huh, https://review.opendev.org/c/openstack/requirements/+/942508 removed it on 2025-02-25 | 15:46 |
| fungi | could the problem be that it's not in the constraints list then? | 15:47 |
| fungi | oh, it switched to glance_store | 15:47 |
| stephenfin | bingo | 15:48 |
| stephenfin | so we need to normalize our constraint name also | 15:49 |
| fungi | yeah, 942508 renamed it from glance-store to glance_store | 15:49 |
| fungi | so the output from the generate-constraints script changed around that time, i suppose | 15:50 |
| fungi | frickler: ^ this may be the answer to the baffling failures we were discussing in #openstack-release too? | 15:50 |
| stephenfin | I hope that wasn't me... | 15:51 |
| stephenfin | (I would have been working on it about then iirc) | 15:51 |
| * stephenfin blame tonyb preventatively | 15:51 | |
| noonedeadpunk | hey folks! Do you have any guesses why https://docs.openstack.org/2025.1/deploy/ is not showing any projects, given that https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/artifacts is merged and promoted https://review.opendev.org/c/openstack/openstack-manuals/+/953313 ? | 15:55 |
| noonedeadpunk | is there any sort of server-side cache for html content? | 15:55 |
| fungi | noonedeadpunk: i'll check the log to see if it's failing to update or something, but no normally it should take no more than 5 minutes after the promote job completes | 15:57 |
| clarkb | have you checked that the artifacts contain the data? | 15:58 |
| clarkb | noonedeadpunk: ^ you can do that first | 15:58 |
| clarkb | I think either https://opendev.org/openstack/openstack-manuals/src/branch/master/www/2025.1/deploy/index.html isn't doing what you expect it to do in generating the content (check the build result content to see) or publication is failing somewhere | 15:59 |
| fungi | yeah, volume releases are happening as scheduled at least | 15:59 |
| fungi | https://109254a1749d24a3f999-62593741b623fd737fcd3b17f392bcb8.ssl.cf5.rackcdn.com/openstack/240a3287daba4c029dbc06ed9ca9519d/docs/2025.1/deploy/ seems to be in the preview from the gate job | 16:02 |
| clarkb | and it doesn't have the content. So the issue is in generating the content not publishing it | 16:03 |
| fungi | agreed, i pulled the docs archive tarball from that job and it also only contains an index.html file under ./2025.1/deploy/ | 16:04 |
| fungi | er, from that build | 16:04 |
| clarkb | and that index file doesn't have any guides in it | 16:04 |
| fungi | it has two links in it, one for ansible in docker and one for ansible in lxc | 16:06 |
| clarkb | oh huh I must be blind it does | 16:07 |
| clarkb | ok ignore me the data appears to be there then | 16:07 |
| clarkb | next thing is probably to check if the RW volume has the data in it? | 16:08 |
| fungi | yeah, so the question is did something else race it and deploy an older version of that file? maybe changes merged for different branches around the same time | 16:08 |
| clarkb | oh except you said volume releases are happening so RW and RO should match | 16:09 |
| clarkb | ya rsync races are a good guess | 16:09 |
| noonedeadpunk | yes, it should contain only links | 16:09 |
| fungi | looks like there was another docs change that promoted at almost the same time | 16:10 |
| clarkb | though openstack-manuals seems to be primarily master now (ther are some really old tsable branches but nothign for 2025.1 for example) | 16:10 |
| noonedeadpunk | 2 patches landed eachj after other - that's true | 16:10 |
| noonedeadpunk | but neither of them was promoted | 16:10 |
| noonedeadpunk | (there was a chain) | 16:10 |
| fungi | "Build succeeded (promote pipeline)" | 16:10 |
| fungi | those are promotions | 16:10 |
| noonedeadpunk | but I'd expect there some semaphors then if it's sensetive to races condfition? | 16:11 |
| fungi | builds of jobs running in the promote pipeline | 16:11 |
| noonedeadpunk | and they were run in serial manner, yeah | 16:12 |
| fungi | yeah, promote should be using a supercedent pipeline manager, i think, so as long as those were for the same branch they wouldn't have run concurrently | 16:12 |
| noonedeadpunk | yeah, right | 16:13 |
| clarkb | https://zuul.opendev.org/t/openstack/build/56ea313811794f9999fcce2c5625cdee/log/job-output.txt#133-135 and https://zuul.opendev.org/t/openstack/build/b569b5cc2b454b2593d8e2d982166fdf/log/job-output.txt#133-135 seem to confirm there was a minute and half ish between them | 16:14 |
| noonedeadpunk | Synchronize files to AFS is changed in there as well | 16:15 |
| noonedeadpunk | So I'd expect promote to be successfull | 16:15 |
| clarkb | the executors were replaced with new systems which means new kernels and openafs builds. However, elsewhere afs synchronization seems to be working (I just published a zuul blog post an hour ago for example) | 16:15 |
| clarkb | so I don't think its a fundamental afs problem | 16:15 |
| noonedeadpunk | The only thing I can guess then if AFS wasn;'t mounted in fact | 16:17 |
| clarkb | or the source data was wrong. Or the target path is wrong somehow | 16:18 |
| noonedeadpunk | well artifact does look fine, so unless it fetched wrong artifact... | 16:18 |
| clarkb | https://ef4886d127b9d5e50b2a-46ed996c4c88287cea630d62dd5380de.ssl.cf1.rackcdn.com/openstack/37a9b4537fc9495c99974bf011d09f00/docs-html.tar.gz this is what was fetched | 16:18 |
| clarkb | according to https://zuul.opendev.org/t/openstack/build/56ea313811794f9999fcce2c5625cdee/console#1/0/4/localhost | 16:19 |
| fungi | i tracked back from the last run of that promote job and the artifact it said it pulled in https://zuul.opendev.org/t/openstack/build/b569b5cc2b454b2593d8e2d982166fdf/console#1/0/4/localhost does include those lines in the index.html | 16:19 |
| fungi | is that the most recent run of the job? | 16:19 |
| clarkb | the one I linked to is for https://review.opendev.org/c/openstack/openstack-manuals/+/953312 | 16:19 |
| clarkb | which ran first not second I think | 16:20 |
| noonedeadpunk | yes, right, I just saw that artifact was correct as well | 16:20 |
| fungi | yeah, from there i went to build history to check the most recent build, just in case it had overwritten it with different content | 16:20 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/buildset/a53547d0a00447c2828a57569a53c725 is the last one | 16:21 |
| noonedeadpunk | according to https://zuul.opendev.org/t/openstack/buildsets?project=openstack%2Fopenstack-manuals&pipeline=promote&skip=0 | 16:21 |
| fungi | yeah, that's the buildset containing the build i linked above | 16:21 |
| clarkb | https://109254a1749d24a3f999-62593741b623fd737fcd3b17f392bcb8.ssl.cf5.rackcdn.com/openstack/240a3287daba4c029dbc06ed9ca9519d/docs-html.tar.gz and that build pulled this tarball | 16:22 |
| clarkb | side note: those tarballs are tarbombs :/ | 16:22 |
| noonedeadpunk | does content on AFS match? | 16:22 |
| fungi | last modified time on /afs/.openstack.org/docs/2025.1/deploy/index.html is 2025-05-02T14:27 | 16:25 |
| fungi | so it's not getting overwritten in afs | 16:25 |
| noonedeadpunk | so I am wondering what in promote job does mount afs? | 16:26 |
| fungi | i did at least check from a zuul executor that it does have working afs at that path | 16:26 |
| noonedeadpunk | as right after `aklog` upload hjappens | 16:26 |
| clarkb | openafs is installed on the executors and is part of the bwrap bind mount iirc | 16:27 |
| clarkb | so nothing in the job is mounting it. Its already mounted | 16:27 |
| noonedeadpunk | aha, ok | 16:27 |
| noonedeadpunk | so it just needs auth during runtiome which it gets | 16:28 |
| fungi | yeah, the "upload-afs-roots: Synchronize files to AFS" task should be doing it, and seems to reference the correct unpacked source path | 16:28 |
| fungi | we don't get any output from that task though | 16:28 |
| clarkb | this build is what published my zuul blog post update https://zuul.opendev.org/t/zuul/build/5bee8a15e93d4e219471f064f0debc92 | 16:28 |
| clarkb | that runs in post not promote but I don't think that should matter | 16:29 |
| noonedeadpunk | ok, let me land another patch to the repo then and see what happens | 16:29 |
| clarkb | https://zuul.opendev.org/t/zuul/build/5bee8a15e93d4e219471f064f0debc92/console#3/0/5/localhost its synchronize is far more verbose | 16:30 |
| clarkb | I think that task literally copied no files | 16:30 |
| fungi | yeah | 16:30 |
| fungi | could /var/lib/zuul/builds/b569b5cc2b454b2593d8e2d982166fdf/work/docs/ be empty then? | 16:30 |
| clarkb | in https://zuul.opendev.org/t/openstack/build/b569b5cc2b454b2593d8e2d982166fdf/console#1/0/26/localhost build_roots is empty | 16:31 |
| clarkb | but build_roots is not empty in my working example | 16:31 |
| clarkb | I think this must be the source of the problem | 16:31 |
| fungi | https://zuul.opendev.org/t/openstack/build/b569b5cc2b454b2593d8e2d982166fdf/console#1/0/6/localhost is what should have put the files in place | 16:31 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-afs-roots/library/zuul_afs.py#L109 this is where build_roots comes from. I was initially reading it as an input to th task but it is an output logging what it did | 16:33 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-afs-roots/library/zuul_afs.py#L32-L33 | 16:34 |
| clarkb | is there a .root-marker? | 16:34 |
| clarkb | there isn't one in the tarball. Not sure if the job creates that or it needs it in the tarball | 16:34 |
| fungi | the "Write root_marker file" task was skipped due to a conditional result | 16:35 |
| clarkb | in the zuul example the job writes it here: https://zuul.opendev.org/t/zuul/build/5bee8a15e93d4e219471f064f0debc92/console#2/0/0/localhost | 16:35 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/buildsets?project=openstack%2Fopenstack-manuals&pipeline=promote&skip=0 | 16:35 |
| fungi | https://zuul.opendev.org/t/openstack/build/b569b5cc2b454b2593d8e2d982166fdf/console#1/0/7/localhost | 16:35 |
| noonedeadpunk | this what vccreates .root-marker I think | 16:35 |
| noonedeadpunk | doh | 16:35 |
| noonedeadpunk | https://zuul.opendev.org/t/zuul/build/5bee8a15e93d4e219471f064f0debc92/console#2/0/0/localhost | 16:36 |
| clarkb | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L597-L615 | 16:36 |
| clarkb | write_root_marker: false | 16:36 |
| clarkb | to me that implies the artifact is supposed to contain the root marker and we stopped doing that? | 16:36 |
| fungi | so presumably you'd do that if it should contain one already | 16:36 |
| fungi | yeah | 16:37 |
| clarkb | fungi: ya exactly. I think that flag exists because sometimes the content in the archive is expected to already have the data | 16:37 |
| noonedeadpunk | but what about copy task above? | 16:37 |
| clarkb | noonedeadpunk: that is from the zuul blog post which works | 16:37 |
| noonedeadpunk | ah, doh | 16:37 |
| noonedeadpunk | right | 16:37 |
| noonedeadpunk | ok, yes,. there's no .root-marker task indeed in affected job | 16:39 |
| clarkb | https://opendev.org/openstack/openstack-manuals/src/branch/master/.zuul.yaml#L6-L29 the comments in that job make no sense. Once says no root marker is written and the other say it is written so you need the vars | 16:40 |
| noonedeadpunk | well the job was not touched for a veeeeery long time | 16:41 |
| fungi | looking to see what, if anything has updated in that volume in the past month | 16:41 |
| clarkb | ya but maybe something that tox runs was changed | 16:41 |
| fungi | for comparison purposes | 16:41 |
| fungi | docs for glean and openstack-zuul-jobs updated there recenly | 16:41 |
| fungi | betting they run a different set of jobs | 16:42 |
| clarkb | yes manuals has its own jobs | 16:42 |
| clarkb | openstack-manuals/tools/build-all-rst.sh does write a root marker | 16:43 |
| fungi | 948577,1 and 948425,4 promoted about half an hour apart on may 2, the first one got its content published and the second did not | 16:44 |
| noonedeadpunk | I'm not sure this is actually the job which runs? | 16:45 |
| noonedeadpunk | disregard | 16:45 |
| noonedeadpunk | I mixed myself again with URIs | 16:45 |
| clarkb | we build the artifact tarball with tox -epublishdocs which runs tools/publishdocs.sh which calls tools/build-all-rst.sh --pdf | 16:45 |
| fungi | so it seems like we might have a half-hour window we can narrow the behavior change to | 16:46 |
| fungi | unfortunately the artifacts have all aged out of swift since that was 2 months ago | 16:46 |
| clarkb | I think that should be writing the root marker in that process. Maybe the problem is in the tarball step we aren't including the . file | 16:46 |
| clarkb | oh we only write the root marker if the branch is master | 16:47 |
| clarkb | which it was | 16:47 |
| fungi | which should be irrelevant in the case of openstack-manuals since it doesn't have stable branches i don't think | 16:47 |
| fungi | or rather it stopped having them after stable/ocata | 16:48 |
| noonedeadpunk | right | 16:48 |
| fungi | could the branch test behavior have changed and stopped matching? | 16:49 |
| fungi | (complete shot in the dark) | 16:49 |
| clarkb | maybe its looking at $ZUUL_BRANCH | 16:50 |
| clarkb | https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/console#2/0/13/ubuntu-jammy doesn't seem to record the env vars that were used to run the task | 16:50 |
| clarkb | https://opendev.org/openstack/openstack-manuals/src/branch/master/.zuul.yaml#L23-L29 this is wheer we set it theoretically | 16:51 |
| noonedeadpunk | is it https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/log/tox/publishdocs/3-commands[2].log#13 ? | 16:51 |
| clarkb | noonedeadpunk: ya that looks like it is being set | 16:53 |
| clarkb | noonedeadpunk: I think what I would do to debug furhter is update opensatck-manuals tools/publishdocs and tools/build-all-rst to set -x and add debugging output around root marker creation. Then if we confirm it is being written the next step is checking why it isn't included in the archive | 16:53 |
| noonedeadpunk | so here we run publishdocs.sh https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/log/tox/publishdocs/1-commands[0].log#20 | 16:54 |
| noonedeadpunk | but I don;'t see build-all-rst being sourced | 16:54 |
| noonedeadpunk | oh, it's not having it.... | 16:55 |
| noonedeadpunk | doh | 16:55 |
| clarkb | noonedeadpunk: publishdocs.sh calls it | 16:55 |
| clarkb | but I think we need more logging (set -x would help a lot) | 16:55 |
| noonedeadpunk | I type faster then I think :( | 16:56 |
| clarkb | maybe the publish-docs/html/.root-marker path is not the correct path anymore? | 16:56 |
| clarkb | but I Think first thing is confirm it is being written at all then debug why it isn't getting into the tarball | 16:57 |
| fungi | check jobs should be sufficient for verifying that, so a dnm change ought to do the trick | 16:57 |
| clarkb | yup | 16:57 |
| clarkb | https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/console#3/0/16/ubuntu-jammy this is what creates the tarball | 17:00 |
| clarkb | publish-docs/html does look to be the correct path to me | 17:00 |
| clarkb | noonedeadpunk: you might be able to add a -v to the tar invocation there and do a depends on to get the file listing of things going into the tarball if you confirm the root marker is written | 17:01 |
| noonedeadpunk | ++ | 17:02 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/c/openstack/diskimage-builder/+/721581 | 17:09 |
| clarkb | infra-root when do you think we should pull the old zk servers out of DNS? https://review.opendev.org/c/opendev/zone-opendev.org/+/953844 I can also take that as a signal to cleanup the old servers | 17:10 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/c/openstack/diskimage-builder/+/721581 | 17:10 |
| corvus | clarkb: zk dns any time | 17:23 |
| priteau | I just got some job failures due to: Source /opt/cache/files/cirros-0.5.3-x86_64-disk.img not found. Has the file been removed? | 17:23 |
| fungi | might be something we didn't expect that changed with zuul-built images vs nodepool? | 17:24 |
| fungi | though we're including it in https://opendev.org/opendev/zuul-providers/src/branch/master/dib-elements/cache-devstack/source-repository-images#L4 | 17:25 |
| clarkb | fungi: priteau I see that image listed in both nodepool image build config and zuul launcher built images config | 17:25 |
| clarkb | but also those files are suspected to be treated as a best effort cache. If not present you should download it yourself | 17:25 |
| clarkb | s/suspected/supposed/ I can't type | 17:25 |
| clarkb | priteau: can you link to the job? | 17:26 |
| fungi | yeah, which is devstack's behavior noramlly, at least | 17:26 |
| clarkb | sorry job log | 17:26 |
| fungi | so guessing this isn't devstack-based | 17:26 |
| noonedeadpunk | so it echoing data at least in debug: https://zuul.opendev.org/t/openstack/build/1aa3f63f30434b4baa89af3f7b018145/log/tox/publishdocs/1-commands[0].log#10165 | 17:27 |
| clarkb | noonedeadpunk: I notice there is an rsync -a after the root marker is written that targets the dir that root marker is written to. But that shouldn't overwrite the file | 17:28 |
| clarkb | you have to set extra flags to delete with rsync iirc | 17:28 |
| noonedeadpunk | but it's not implying --delete | 17:28 |
| noonedeadpunk | iirc | 17:29 |
| clarkb | yes. I just watned to call that out | 17:29 |
| clarkb | to amke sure my assumptions made sense and sounds like they do | 17:29 |
| noonedeadpunk | I also was looking at it right now | 17:29 |
| clarkb | noonedeadpunk: maybe add a task after tox that does an ls -al of the dir | 17:29 |
| noonedeadpunk | but it kinda opens door for error | 17:29 |
| clarkb | and maybe even cat the file | 17:29 |
| noonedeadpunk | So I'd rather created it in www/static/ to be on the safe side... | 17:30 |
| clarkb | zuul web node listing seems to only show in use nodes? | 17:30 |
| frickler | fungi: stephenfin: not sure if I missed something in the backlog, but yes, I've been thinking about needing to update the constraints and possibly the requirements.txt in consumers, but that sounds like a difficult/questionable thing to do for stable branches | 17:30 |
| clarkb | and I don't see IP addresses | 17:30 |
| fungi | clarkb: i think because it's tenant-scoped? so only relevant once there's a node request for a build in that tenant? | 17:31 |
| clarkb | fungi: ya. I'm wondering what the equivalent of nodepool list is now so that I as admin can see an overall picture. Maybe that doesn't exist yet? | 17:31 |
| fungi | frickler: what we observed was that generate-constraints renamed glance-store to glance_store in the upper-constraints.txt file at the time when the errors in job began | 17:31 |
| fungi | the change i linked where that happened didn't call it out specifically as happening though | 17:32 |
| corvus | clarkb: ips should be in the json; i think there isn't a column in the javascript ui for them (yet) | 17:33 |
| clarkb | corvus: any idea what the server column with null values and copy buttons is supposed to represent? | 17:34 |
| corvus | i think external id | 17:34 |
| clarkb | oh the first id is the internal zuul id. Got it | 17:35 |
| clarkb | I'm going to step out for a bit. I've got hvac people coming over today and the time window they gave us covers the team meeting. So I need to make sure there is unobstructed access in and out of the garage before then | 17:40 |
| noonedeadpunk | hm, so I actually think it could be that the job was just broken, but it was not failing for some reason before. as until I rebased on top of https://review.opendev.org/c/openstack/openstack-manuals/+/951429 - my DNM was failing: https://zuul.opendev.org/t/openstack/build/f6a4b48ce567463db385a1c6de5f3018 | 17:46 |
| opendevreview | Michal Nasiadka proposed openstack/diskimage-builder master: Retry git clone/fetch on timeout https://review.opendev.org/c/openstack/diskimage-builder/+/721581 | 17:46 |
| noonedeadpunk | (not failing due to missing set -e or smth) | 17:47 |
| noonedeadpunk | as now `Synchronize files to AFS` is already running for 5 minutes | 17:48 |
| stephenfin | clarkb: fungi: wrapping up now, but fwiw it seems setuptools is no longer responsible for generating console scripts. That's instead handled by pip (via the vendored distlib library) and that doesn't use importlib | 17:54 |
| fungi | neat | 17:56 |
| fungi | so maybe we can drop that from pbr and just expect everyone to have a new enough pip? | 17:56 |
| noonedeadpunk | how new these need to be? | 17:57 |
| stephenfin | or at the very minimum put it behind a Python version check, on the assumption that if you're using python3.6 or better, you're good | 17:57 |
| stephenfin | noonedeadpunk: not sure yet, but this is the code that does the generation https://github.com/pypa/pip/blob/main/src/pip/_vendor/distlib/scripts.py | 17:58 |
| stephenfin | just a case of finding out how that's called, and how long/since what release of pip it's been called that way | 17:58 |
| stephenfin | my guess is that it may be a PEP-517 thing but TBD | 17:59 |
| noonedeadpunk | well, according to history, it looks around 1y at least... | 17:59 |
| fungi | scripts.py has been there in one form or another dating back to a vendored copy of distlib that was added to pip in march 2013 | 18:00 |
| fungi | i suppose the bigger question is at what point did it start getting used for this purpose | 18:00 |
| fungi | https://github.com/pypa/pip/commit/9936d6b | 18:01 |
| stephenfin | https://github.com/pypa/pip/commit/22b63127cbccc3476033e54eb72bac8ea932ce3a | 18:02 |
| clarkb | thats only when installing from a wheel but everything installed by pip becomes a wheel first then is installed right? | 18:02 |
| clarkb | I guess that is good news and probably something we can simplify in pbr / remove then | 18:02 |
| fungi | jrosser: i tweaked our mirror script to frontload the update with an explicit `reprepro remove noble-updates qemu-block-extra` and started a manual run of the script. looks like it pulled the older version of the package now, so once this completes the errors will hopefully go away | 18:03 |
| jrosser | fungi: thats great, thank you | 18:04 |
| noonedeadpunk | jrosser: thanks for raising that | 18:04 |
| fungi | clarkb: yes, pip doesn't "install" sdists these days, it builds a wheel and installs that | 18:04 |
| clarkb | noonedeadpunk: so you think 951429 fixes this? | 18:04 |
| clarkb | this == root markers for afs publication? | 18:05 |
| noonedeadpunk | promote jobs already runs for over 20 minutes | 18:05 |
| stephenfin | indeed. even if you give it an sdist directly, it still builds the wheel | 18:05 |
| noonedeadpunk | so either smth went terribly wrong, or it does the job now for real | 18:05 |
| stephenfin | exhibit a https://paste.opendev.org/show/bOq1KbVBPlXRVU33ygAG/ | 18:06 |
| noonedeadpunk | but that dnm which adds set -e failed right away and nothing was merged before - is very suspicious | 18:06 |
| clarkb | stephenfin: I guess the thing to double check is update PBR to drop all scripts handling then pip install a package and see what the scripts look like. Setuptools does have script handlign in it but I'm guessing that largely for setup.py install these days which is largely going away? Basically lets ensure that the normal ip install case does what we expect and if so we can probably | 18:08 |
| fungi | noonedeadpunk: so supposition is that the script was terminating early before the root marker was added, and then the job was proceeding normally thereafter as if it assumed all the content was present? | 18:08 |
| clarkb | drop the handling in PBR entirely | 18:08 |
| clarkb | stephenfin: but also I think it is late for you so don't worry about that now | 18:08 |
| noonedeadpunk | yeah, as some content was generated already, but not all of it | 18:08 |
| noonedeadpunk | so artifacts looked "fine" | 18:08 |
| clarkb | noonedeadpunk: I think adding -e is something that should happen then. To make errors more obvious (rather than dnm it) | 18:09 |
| fungi | heck, it's late enough for me, i'm ready for meetings to be over so i can go get dinner | 18:09 |
| fungi | agreed, sounds like that script needs to become more robust | 18:09 |
| fungi | if this was actually the problem | 18:10 |
| stephenfin | clarkb: I've been checking the behaviour of setuptools by itself (see my paste above) and that looks good. The question will be whether we remove the pbr stuff entirely, or as I said hide it behind a Python version check | 18:10 |
| stephenfin | at some point, releasing pbr2 and leaving pbr as-is for more ancient software might just be easier. not something to be figured out now though | 18:11 |
| noonedeadpunk | but projhect deploy guide was produced in my dnf before the failure: https://zuul.opendev.org/t/openstack/build/f6a4b48ce567463db385a1c6de5f3018/log/job-output.txt#1013 | 18:11 |
| noonedeadpunk | and then it was hitting the warning and failed: https://zuul.opendev.org/t/openstack/build/f6a4b48ce567463db385a1c6de5f3018/log/job-output.txt#1270 | 18:11 |
| clarkb | fungi: I'm trying to pull up lists.opendev.org to get a link to my archived meeting agenda and the page is not loading. Server load seems a bit high but not crazy. Not sure if this is a known problem (ai crawlers possible) | 18:12 |
| clarkb | hrm looks like we may be digging into swap and mariadb is workign hard | 18:13 |
| clarkb | ok page finally loaded for me | 18:13 |
| noonedeadpunk | and warning was also in a kolla/osa jobs: https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/log/job-output.txt#1244 | 18:13 |
| noonedeadpunk | except, it finsihed with `ok: Runtime: 0:01:53.808615` right after that | 18:13 |
| clarkb | fungi: ya there is definitely some suspicious activity in the apache logs | 18:14 |
| noonedeadpunk | but in my DNM which was successfull - it's not even a half: https://zuul.opendev.org/t/openstack/build/1aa3f63f30434b4baa89af3f7b018145/log/job-output.txt#1333 | 18:14 |
| clarkb | if we haven't already we should apply our user agent filter list to that node | 18:14 |
| noonedeadpunk | build finished with problems, 1 warning (with warnings treated as errors). congratulations :) (113.61 seconds) | 18:15 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/240a3287daba4c029dbc06ed9ca9519d/log/job-output.txt#1307-1315 | 18:16 |
| noonedeadpunk | so yeah | 18:16 |
| noonedeadpunk | it should not be DNM then.... | 18:16 |
| clarkb | I need to focus on meeting prep so won't dig into the lists server situation further right now but I suspect step 0 is applyting that filter if we don't already then maybe updating the filter list as necesasry | 18:17 |
| fungi | https://opendev.org/opendev/system-config/src/commit/76233e9/playbooks/roles/mailman3/tasks/main.yaml#L167-L170 has it in there already | 18:18 |
| fungi | so i guess we need to make some additions to the filter list | 18:19 |
| clarkb | agreed | 18:20 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add another weird UA to our general filters https://review.opendev.org/c/opendev/system-config/+/953904 | 18:21 |
| clarkb | fungi: do we want to just add that one? I didn't check how many occurrences there are, but I suspect there may be others if we spend a few minutes checking the logs (usually I try to grep the UAs out then have sort/uniq etc give me a count of the worst offenders then those that are definitely odd get added to our list) | 18:22 |
| clarkb | fungi: I posted a comment to the change for a bug that will need fixing either way | 18:23 |
| noonedeadpunk | and content now promoted: https://docs.openstack.org/2025.1/deploy/ | 18:24 |
| fungi | d'oh, that's what i get for appending a copy of the last line | 18:24 |
| noonedeadpunk | thanks folks for your time - I would never found that .root-marker | 18:25 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add another weird UA to our general filters https://review.opendev.org/c/opendev/system-config/+/953904 | 18:25 |
| corvus | noonedeadpunk: for future reference there's a description of it here: https://zuul-ci.org/docs/zuul-jobs/latest/afs-roles.html#role-upload-afs-roots | 18:33 |
| noonedeadpunk | thanks! | 18:35 |
| mnasiadka | clarkb: Any idea if anything related to disk space changed after switching to niz? 2025-07-01 19:05:30.838 | cp: error copying '/opt/dib_tmp/dib_image.xgYsGicI/image0.raw' to '/opt/dib_tmp/dib-images/ubuntu-focal.vhd-intermediate': No space left on device | 19:09 |
| clarkb | mnasiadka: we're limited to the disk space available on the test nodes. We've made some changes previously to reduce the total amount of space used. But the total space varies by cloud provider and their flavors. We may need to look and see if there is more we can trim out to make things fit. That said I think frickler indicated the total size of the git repos seemed larger than | 19:10 |
| clarkb | expected so maybe we start there? | 19:10 |
| mnasiadka | Ah, right - I'll check | 19:10 |
| fungi | the zuul info logs collected for the build should give some idea of the fs layout and sizes | 19:11 |
| fungi | jrosser: it's updated now and i see qemu-block-extra 1:8.2.2+ds-0ubuntu1.7 depending on librbd1 (>= 19.2.0-0ubuntu0.24.04.2) again | 19:18 |
| mnasiadka | Well, not really seeing anything that is off from the DIB image size report in terms of filenames, but still 23G in total for /opt/dib_tmp/dib_build.9SeIVw8X/built/opt/git/opendev.org and 15G for git/opendev.org/openstack alone | 19:19 |
| fungi | if we're running it on a rax classic node with 40gb rootfs and not mounting the ephemeral disk at /opt then that could easily blow up | 19:21 |
| corvus | (since the image builds happen in the opendev tenant, they've been running on niz nodes for several weeks) | 19:24 |
| mnasiadka | we are using the ephemeral disk, so it's not that | 19:26 |
| mnasiadka | let me remove the depends-on to check if the git retry code is not adding something it shouldn't | 19:27 |
| opendevreview | Michal Nasiadka proposed opendev/zuul-providers master: Add Ubuntu bionic/focal builds, labels and provider config https://review.opendev.org/c/opendev/zuul-providers/+/953269 | 19:27 |
| clarkb | oooh side effects from that would be interesting | 19:27 |
| priteau | clarkb: sorry for the delay responding, this is the job which failed to get the cirros image: https://zuul.opendev.org/t/openstack/build/62ce2e3acdd549f896b24839d88214c6 | 19:34 |
| priteau | Same failure in another one: https://zuul.opendev.org/t/openstack/build/85da9e8b18774b72b865322b54021779 | 19:35 |
| clarkb | can you link to the portion of the log that shows the cirros issue? there are a lot of failures I'm seeing | 19:36 |
| clarkb | also I see zuul logger errors are you rebooting the node possibly? | 19:37 |
| clarkb | (that might impact mounts of /opt in some clouds) | 19:37 |
| priteau | The failure is during the last task of https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_62c/openstack/62ce2e3acdd549f896b24839d88214c6/primary/ansible/seed-deploy | 19:38 |
| priteau | "msg: Source /opt/cache/files/cirros-0.5.3-x86_64-disk.img not found" | 19:38 |
| priteau | I don't believe we are rebooting the node at all | 19:38 |
| clarkb | priteau: if you navigate to that file in the zuul web dashboard you can link to the line | 19:39 |
| clarkb | for example https://zuul.opendev.org/t/openstack/build/62ce2e3acdd549f896b24839d88214c6/log/primary/ansible/seed-deploy#11121 which seems to indicate the file is prsent? | 19:39 |
| clarkb | or maybe the stat: exists: false just below that is that task saying nothing changed because the file isn't there? | 19:42 |
| clarkb | but it doesn't error until later | 19:42 |
| clarkb | priteau: fungi corvus mnasiadka I see the problem. In opendev/zuul-providers we don't put cache-devstack element in all image builds. We should probably move it into the base-elements list so that all images get that file | 19:46 |
| clarkb | priteau: that said we've never promised those files will exist on all images. It is a cache and caches may not have entries. Your jobs should handle the fallback case where they don't exists as well | 19:46 |
| clarkb | in particular that cirros image is quite old and it wouldn't be crazy for us to stop caching it in hopes people use the newer versions instead | 19:48 |
| fungi | we've semi-regularly dropped less-used versions to keep the list from growing massive | 19:49 |
| priteau | For some reason we have issues booting the 0.6.x series. This is the last one in the 0.5.x | 19:50 |
| priteau | But that's another topic to investigate | 19:50 |
| fungi | with the expectation that taking slightly longer to fetch those images in a handful of jobs is an okay tradeoff for keeping our node images smaller | 19:50 |
| fungi | but yeah, 5.3 is probably used in a lot of jobs still | 19:50 |
| opendevreview | Clark Boylan proposed opendev/zuul-providers master: Cache devstack files on all images https://review.opendev.org/c/opendev/zuul-providers/+/953908 | 19:51 |
| clarkb | I think ^ will fix the missing files | 19:51 |
| fungi | gonna go grab dinner, back soon | 19:54 |
| priteau | thanks | 19:55 |
| clarkb | corvus: I updated the zk upgrade etherpad notes. Hopefully those make sense now. | 19:59 |
| clarkb | I'm going to dig up lunch then will look at zookeeper server cleanups after | 20:00 |
| opendevreview | Merged opendev/zone-opendev.org master: Remove zk04-zk06 from DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/953844 | 20:45 |
| clarkb | ok those names don't resolve for me anymore. Last call otherwise I'm deleting the servers at the top of the hour | 20:51 |
| clarkb | #status log Deleted zk04, zk05, and zk06.opendev.org. These zookeeper servers have been replaced by zk01, zk02, and zk03.opendev.org. | 21:03 |
| opendevstatus | clarkb: finished logging | 21:04 |
| clarkb | I event remembered to clean up the emergency file | 21:28 |
| clarkb | took me a few to remember that thoseservers were in there though | 21:28 |
| fungi | thanks! | 21:30 |
| clarkb | corvus: I think both of the launcher changes you wanted to get in place for opendev have landed is the plan still to restart launchers today? | 21:37 |
| clarkb | I'm thinking I can recheck my chagnes that tripped over the provider nodeset assignemtn a few times afterwards if you think that would be helpful | 21:37 |
| corvus | clarkb: i'm almost ready to switch back to that | 21:39 |
| corvus | clarkb: both launchers restarted | 22:00 |
| clarkb | I'll issue a couple rechecks now then | 22:01 |
| clarkb | frickler: I posted a comment on https://review.opendev.org/c/openstack/diskimage-builder/+/951469 asking for clarification on the trixie support change. fungi you may be interested in that too | 22:22 |
| opendevreview | Merged openstack/project-config master: Add zuul-launcher max servers https://review.opendev.org/c/openstack/project-config/+/953803 | 22:25 |
| opendevreview | Merged openstack/project-config master: Grafana: update zuul status page for NIZ https://review.opendev.org/c/openstack/project-config/+/953823 | 22:29 |
| clarkb | fungi: looking at the reprepro hack for ubuntu package cleanup. That script is used by all of the reprepro repos but I guess this is safe because we'll just short circuit on the set -e if it fails in a repo without those packages? | 23:00 |
| clarkb | mostly just wondering what happens here if run against debian mirroring for example | 23:00 |
| clarkb | but thats probably fine to fail a coupel of updates in order to fix ubuntu noble | 23:00 |
| fungi | yeah | 23:04 |
| fungi | if it becomes a problem i can run a copy of the script | 23:04 |
| fungi | but not planning to leave it like this anyway | 23:04 |
| clarkb | yup its temporary | 23:04 |
| clarkb | I'm going to pop out now for a family dinner. Probably won't be back until tomorrow morning | 23:05 |
| fungi | cool, catch up with you then | 23:05 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!