| *** mrunge_ is now known as mrunge | 10:59 | |
| clarkb | I'm looking at testing that our speculative gerrit images don't get pruned out of the intermediate registry anymore. I think I can just fetch them using the artifacts links in the job results. I'm not positive it has been long enough to have pruned them so will probably retest tomorrow | 17:44 |
|---|---|---|
| corvus | clarkb: yeah that should do it. i think the bug was actually "it just deletes everything when it runs, and it runs every 24 hours" not "it must be 24 hours". so as long as we have done a prune run since restarting the container, then it should be an effective test | 17:48 |
| corvus | (checking the logs for the prune run would be a good idea i think) | 17:48 |
| clarkb | corvus: ack I'll check that too | 17:53 |
| clarkb | I am able to pull docker://insecure-ci-registry.opendev.org:5000/quay.io/opendevorg/gerrit:fab0f049404140219cf92ea700b5312a_3.11 which was built by https://zuul.opendev.org/t/openstack/build/fab0f049404140219cf92ea700b5312a/artifacts on the 16th. | 18:07 |
| clarkb | Looking at the registry itself I note to things. The first is that I think corvus has a shell with a less on an older log file which maybe keeping it around? The other is that I think we have a new traceback in pruning | 18:08 |
| clarkb | I'm working on a paste. I think what happens is we get a 404 (not sure why) then we try to json loads the result of that 404 and we crash | 18:10 |
| clarkb | oh actually I think I may know why. Because older pruning would've deleted objects we expect to be around | 18:10 |
| clarkb | so we probably just need extra belts and suspenders? | 18:10 |
| corvus | shell should be cleaned up | 18:11 |
| clarkb | mordred: corvus: https://paste.opendev.org/show/b3QUW9yAacW5u97BycFS/ | 18:12 |
| clarkb | basically I think what happened is that our old previously broken pruning has pruned blobs/sha256:12ed6eaae601ebe26ec475149c387e5aec530e7aaf9d35e1127661937f8c0683/data so it is a 404 now | 18:13 |
| clarkb | and then when we got to fetch it we 404 and pass None into json.loads() and boom | 18:13 |
| clarkb | we probably need a try block around some layer of the pruning process to get it to proceed when the work is largely done? | 18:14 |
| corvus | sounds right, we should just treat that manifest is empty | 18:15 |
| clarkb | corvus: ya this is the code path where it wants to keep the manifest but the layers have disappeared under it | 18:18 |
| clarkb | I'll work on a patch | 18:18 |
| clarkb | remote: https://review.opendev.org/c/zuul/zuul-registry/+/971468 Handle missing manifests during pruning | 18:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to 1.25.3 https://review.opendev.org/c/opendev/system-config/+/971469 | 18:38 |
| clarkb | I am around today if that looks good after its first round of testing and we awnt ot upgrade | 18:43 |
| corvus | clarkb: it merged | 20:00 |
| Clark[m] | Thanks (the zuul registry change( | 20:15 |
| corvus | oh haha i thought that's what you meant, but maybe you meant you're around for a gitea upgrade, sorry :) | 20:17 |
| clarkb | corvus: I'm aroubnd for both :) | 20:35 |
| clarkb | I just wanted to clarify | 20:35 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!