| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Ignore errors when deleting tags from dockerhub https://review.opendev.org/c/zuul/zuul-jobs/+/799338 | 00:00 |
|---|---|---|
| corvus | clarkb: ^ maybe we could merge that, then re-run the job? | 00:00 |
| corvus | or i can pull/push manually | 00:00 |
| clarkb | I'm hapyp to land that change and try it that way unless you want to go quicker. | 00:01 |
| corvus | it may be a toss up which is quicker :) | 00:01 |
| clarkb | corvus: we must tag latest before that though? otherwise the tag it wants to move to ltest won't exist? | 00:01 |
| corvus | you know... i am concerned that may not work | 00:02 |
| corvus | the sequence is GET, PUT, DELETE | 00:02 |
| clarkb | ya I'm thinking we may have already tried to update the tag and it is just the delete that failed | 00:02 |
| clarkb | er the delete failing is just what we have noticed | 00:02 |
| corvus | and my theory is that GET and PUT are working. but we're still not seeing 'latest' as being something new | 00:02 |
| clarkb | right | 00:03 |
| corvus | which suggests PUT is not working | 00:03 |
| clarkb | corvus: if you fetch latest locally does it match what dockerhub shows as latest or something else? (I ask this knowing that whenever I try to answer this qusetion it is never easy) | 00:03 |
| corvus | let me see what pulling latest does | 00:03 |
| corvus | we have handy 'tags' in our manifests to help with that :) | 00:04 |
| corvus | or rather 'labels' sorry | 00:05 |
| corvus | "org.zuul-ci.change": "799317", | 00:05 |
| clarkb | if I pull zuul/zuul:latest it says Digest: sha256:4997a8893bf7f583f4862f64975e78dfb73b0e6e7bab89feca2a52c556ccffd5 and that matches https://zuul.opendev.org/t/zuul/build/08da96babad640a29136acdbb2244b49/log/job-output.txt#11400 | 00:05 |
| corvus | clarkb: i think that means zuul/zuul latest really is our change | 00:05 |
| clarkb | yes, and it is just the web ui/api that is struggling | 00:05 |
| corvus | unfortunately, it's only zuul/zuul that updated | 00:06 |
| corvus | so if we merge my zuul-jobs change and re-run, i think we will get all the images to update | 00:06 |
| clarkb | aha because it was first in the iteration. Your failed_when: false change would address that | 00:06 |
| clarkb | ++ | 00:06 |
| clarkb | I've +2'd the z-j change I'll let you decide if you want to try landing and reenqueuing or do it manually | 00:06 |
| corvus | i think it's going to be easiest to land that and re-run | 00:07 |
| clarkb | works for me | 00:07 |
| fungi | makes sense | 00:07 |
| corvus | btw, it did look like the data size eventually dropped, so i think the compression is making a difference | 00:13 |
| clarkb | we are just waiting for a xenial node now I think | 00:17 |
| clarkb | I need to check on family. Will watch the zuul status from my phoen and get back to a proper keyboard when it goes through to continue with things if we want to do that still | 00:25 |
| corvus | clarkb: no worries; i can take care of it; feel free to check in or not as you want :) | 00:33 |
| corvus | clarkb, fungi: thanks for all your help :) | 00:33 |
| opendevreview | Merged zuul/zuul-jobs master: Ignore errors when deleting tags from dockerhub https://review.opendev.org/c/zuul/zuul-jobs/+/799338 | 00:35 |
| clarkb | that took less time than I expected :) | 00:36 |
| clarkb | I'm here again | 00:36 |
| clarkb | we're doing an easy breakfast for dinner so I can get to that once this is done | 00:37 |
| clarkb | corvus: I guess the next step is to reenqueue the promotion job for 799317? | 00:41 |
| * clarkb makes some dinner back in a bit | 00:48 | |
| corvus | yep, re-enqueing now | 01:03 |
| corvus | job completed successfully | 01:05 |
| corvus | zuul-scheduler:latest now says 799317 | 01:05 |
| corvus | i will pull and restart | 01:05 |
| clarkb | huh docker hub reflects the update now too | 01:11 |
| clarkb | I still don't see the 799317 tag but I'm unsure if that is because it is in a weird state or if it got properly cleaned up | 01:11 |
| corvus | me neither; i didn't check to see if the delete succeeded | 01:17 |
| fungi | corvus: and thanks for your help! | 01:19 |
| corvus | i snagged a snapshot in ~corvus/one | 01:21 |
| corvus | once some jobs finish, we can compare to that and see if there are some lines deleted | 01:21 |
| corvus | 3654c3b0e9184334a7c10f3a63ce7af9 exists only in the first snapshot | 01:23 |
| corvus | 2021-07-03 01:22:09,784 DEBUG zuul.Pipeline.openstack.check: Build <Build 3654c3b0e9184334a7c10f3a63ce7af9 of openstack-tox-py37 voting:True on <Worker ze01.opendev.org>> of <Change 0x7fa02cbcce20 openstack/tripleo-common 799203,1> completed | 01:24 |
| clarkb | corvus: is that on zk04? | 01:24 |
| corvus | clarkb, fungi: ^ so far so good | 01:24 |
| corvus | clarkb: yep | 01:24 |
| fungi | nice | 01:24 |
| clarkb | ya grafana shows falling number of watches on zk04 and zk06 whcih we did not see prior to the restart (They only grew) | 01:24 |
| corvus | also, the executor queue graph has real numbers now | 01:25 |
| corvus | okay, i think i'll downgrade my attention; maybe check in tommorow and see how things are looking | 01:39 |
| clarkb | sounds good, thanks for working through this! | 01:40 |
| fungi | i'll be around checking in as well, in case anything falls over in opendev | 01:42 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!