| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 941096: Remove deep copying of config https://review.opendev.org/c/zuul/zuul/+/941096 | 06:25 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 940872: Implement keystore functions for OIDC RS256 https://review.opendev.org/c/zuul/zuul/+/940872 | 10:25 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 940872: Implement keystore functions for OIDC RS256 https://review.opendev.org/c/zuul/zuul/+/940872 | 11:16 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 940971: Manage OIDC signing key rotation https://review.opendev.org/c/zuul/zuul/+/940971 | 11:19 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 941235: Implement command for deleting OIDC signing keys https://review.opendev.org/c/zuul/zuul/+/941235 | 12:39 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 941235: Implement command for deleting OIDC signing keys https://review.opendev.org/c/zuul/zuul/+/941235 | 12:58 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 940703: Fix issue with sparse-checkout of include-vars https://review.opendev.org/c/zuul/zuul/+/940703 | 14:36 | |
| @clarkb:matrix.org | corvus: is there a change to pin google apis for nodepool yet? I'm seeing errors due to `ERROR: No matching distribution found for googleapis-common-protos<2.0.dev0,>=1.56.2` which i think that was the fix for | 18:54 |
|---|---|---|
| @clarkb:matrix.org | corvus: these errors are happening on openstacksdk and dib changes that try to ensure compatibility with nodepool so probably a good ideato keep that working for now | 18:55 |
| @jim:acmegating.com | Clark: i don't think so; we should be able to copy it over. | 19:10 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 941294: Pin googleapis-common-protos https://review.opendev.org/c/zuul/nodepool/+/941294 | 19:44 | |
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/build/12a8b77c0c444a12855b838bbfcfcd87 that fails on ca-certificates not being available in the buildx env. I think there was a zuul-jobs fixup for that | 20:38 |
| @clarkb:matrix.org | https://review.opendev.org/c/zuul/zuul-jobs/+/939823 ianw had a question on this but also this doesn't update the docker roles just the container roles. Do we want a new ps for that or shoudl we do separate changes? | 20:40 |
| @jim:acmegating.com | Clark: i think an answer to the question from ianw would be great. ideally one change i think, but 2 is okay. | 21:00 |
| @clarkb:matrix.org | I wonder if the role predates being able to configure things like that. Or maybe it was simply not a known functionality | 21:06 |
| @clarkb:matrix.org | unfortunately the documentation seems to be pretty sparse so hard to say if there was a deficiency that would've caused it to be overruled | 21:07 |
| @clarkb:matrix.org | I guess the question is if we can drop the new package install and preexisting update-ca-certificates entirely via the config settings | 21:08 |
| @jim:acmegating.com | one thing to consider is that we do have roles that modify buildkitd.toml; hopefully that's robust against similar edits, but it's an extra bit of complexity | 21:08 |
| @jim:acmegating.com | so maybe the apk fix is easier and robust -- at least as long as the underlying system doesn't change again (why is that an implementation detail we care about... nevermind) | 21:09 |
| @jim:acmegating.com | * so maybe the apk fix is easier and robust -- at least as long as the underlying system doesn't change again (why is that an implementation detail we care about... nevermind... rhetorical question) | 21:09 |
| @clarkb:matrix.org | ok let me take a quick look at updating it to include the fix in the docker role too | 21:10 |
| @clarkb:matrix.org | then maybe followups can switch to the config driven cert settings | 21:10 |
| @jim:acmegating.com | Clark: while you're there, maybe add a TODO comment with ianw's comment text/link | 21:11 |
| @clarkb:matrix.org | can do | 21:12 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Yaguang Tang: [zuul/zuul-jobs] 939823: Install ca-certificates in the buildx image https://review.opendev.org/c/zuul/zuul-jobs/+/939823 | 21:16 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/nodepool] 941294: Pin googleapis-common-protos https://review.opendev.org/c/zuul/nodepool/+/941294 | 21:18 | |
| @clarkb:matrix.org | that added a depends on. I'm not sure if that will work in gate (it should in check I think() | 21:18 |
| @clarkb:matrix.org | though maybe it won't work in either | 21:18 |
| @jim:acmegating.com | oh we already had the change in the container path, we just needed it in the docker path? | 21:20 |
| @jim:acmegating.com | oh no i misread, sorry | 21:21 |
| @clarkb:matrix.org | corvus: the change was proposed to the container path but not merged. My update added it to both | 21:21 |
| @clarkb:matrix.org | it neesd to be in both | 21:21 |
| @jim:acmegating.com | yeah i'm caught up :) | 21:21 |
| @clarkb:matrix.org | but maybe this fix isn't working: https://zuul.opendev.org/t/zuul/build/32ed45191c2a42ee9363e866e9013f61 ? | 21:27 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/build/0bda8aa640ca43c79bd3a76373de394d/log/job-output.txt#1034-1038 this is the multiarch job for build-container-image and that skips the tasks to isntall things | 21:29 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/build/32ed45191c2a42ee9363e866e9013f61/log/job-output.txt#1838-1851 this is multiarch with build-docker-image and it doesn't skip but it fails later | 21:30 |
| @jim:acmegating.com | so the build-container-image job doesn't run with a buildset registry? | 21:31 |
| @clarkb:matrix.org | ya that seems to be the case | 21:32 |
| @clarkb:matrix.org | so its a good thing we added the docker side as this fix doesn't seem to work | 21:32 |
| @clarkb:matrix.org | `rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL` is logged but otherwise it seems like we do update certs and add one: https://zuul.opendev.org/t/zuul/build/32ed45191c2a42ee9363e866e9013f61/console#2/2/20/builder | 21:37 |
| @mnaser:matrix.org | Sorry I think we ended up revising the fix locally | 21:37 |
| @mnaser:matrix.org | And didn’t push the final one | 21:37 |
| @mnaser:matrix.org | Let me see what we’re running | 21:37 |
| @mnaser:matrix.org | Clark: sorry, i dont have an easy way to push a change now but https://paste.openstack.org/show/b7RYjXiiXdlf5lUUfN3E/ | 21:41 |
| @clarkb:matrix.org | that looks similar/the same? do you know if it differs? | 21:42 |
| @mnaser:matrix.org | we have a commit that changed the ordering to what yo usee in there | 21:43 |
| @mnaser:matrix.org | which i assume was part of the issue, sorry, commit msg here doesnt have too much info | 21:43 |
| @mnaser:matrix.org | https://paste.openstack.org/show/bXpx54m4WgRxJIakVF9A/ | 21:44 |
| @clarkb:matrix.org | i see let me try that update | 21:44 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 21:46 | |
| - [zuul/zuul] 941298: Fix connection check in skipped stream callback https://review.opendev.org/c/zuul/zuul/+/941298 | ||
| - [zuul/zuul] 941299: Fix busy loop in zuul console https://review.opendev.org/c/zuul/zuul/+/941299 | ||
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Yaguang Tang: [zuul/zuul-jobs] 939823: Install ca-certificates in the buildx image https://review.opendev.org/c/zuul/zuul-jobs/+/939823 | 21:46 | |
| @jim:acmegating.com | that makes sense | 21:56 |
| @clarkb:matrix.org | with that edit the job passes now | 21:57 |
| @jim:acmegating.com | i think it has sufficient review, +3 | 21:58 |
| @mnaser:matrix.org | sorry for not properly coming back to clean it up, we were dealing with a billion fires and trying to land stuff and this came in the way | 21:59 |
| @clarkb:matrix.org | `Unable to start service docker: Job for docker.service failed.\nSee \"systemctl status docker.service\" and \"journalctl -xeu docker.service\" for details.\n` that is a new one for me | 22:06 |
| @mnaser:matrix.org | When I was doing the docker build stuff I’d see this happen from time to time | 22:10 |
| @clarkb:matrix.org | gah and now a recheck is hitting rate limits | 22:41 |
| @clarkb:matrix.org | I'll try again in a few hours I guess | 22:41 |
| @clarkb:matrix.org | and if we continue to fail maybe we consider force merging I dunno | 22:41 |
| @jim:acmegating.com | Clark: are there images we should switch to opendevmirror? | 23:10 |
| @clarkb:matrix.org | corvus: uh maybe we should mirror the buildx image? | 23:20 |
| @clarkb:matrix.org | oh these jobs were failing to pull the registry image | 23:21 |
| @clarkb:matrix.org | and debian:testing | 23:21 |
| @clarkb:matrix.org | so ya maybe we need to look at these jobs and adjust where they pull images from? part of the problem is we need to fix the multiarch issue so bit of a chicken andegg | 23:22 |
| @clarkb:matrix.org | the three failures I pulled up failed on a different image one was registry:2, another debian:testing, and the third moby/buildkit:buildx-stable-1 | 23:22 |
| @jim:acmegating.com | yeah, i think it's worth mirroring those and switching. if you think we've sufficiently lined up the swiss cheese to be confident in the change as-is, i think we can force-merge it. | 23:26 |
| @clarkb:matrix.org | I suspect so the change has been in the gate twice now. The first time only one job failed (on the docker restart thing I pasted above) and the second time different jobs failed on docker raet limits. I feel reasonably confident that this change isn't responsible for those errors and they do succeed otherwise | 23:27 |
| @clarkb:matrix.org | why don't we try again in an hour or so (docker rate limit success rate seems higher in that evening period) and if that fails we can force merge tomorrow monring | 23:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!