| -@gerrit:opendev.org- Moritz Haase proposed: [zuul/zuul-jobs] 991770: roles: Add 'ensure-pipx' role https://review.opendev.org/c/zuul/zuul-jobs/+/991770 | 08:47 | |
| -@gerrit:opendev.org- Moritz Haase proposed: [zuul/zuul-jobs] 991770: roles: Add 'ensure-pipx' role https://review.opendev.org/c/zuul/zuul-jobs/+/991770 | 08:48 | |
| @priteau:matrix.org | Hello. We have a kayobe-reviewers group now, but there is no member. Would it be possible to add kayobe-core to it, so we can manage members? | 10:15 |
|---|---|---|
| @priteau:matrix.org | https://review.opendev.org/admin/groups/6261f5a84fb9e97a864e7abdaf925b7618db2420 | 10:16 |
| @fungicide:matrix.org | Pierre Riteau: for openstack projects we normally make the ptl or tact sig liaison the initial group member, so i've added mnasiadka to kayobe-reviewers and then he can set the group up however the team needs | 13:12 |
| -@gerrit:opendev.org- Dmitriy Rabotyagov proposed: [openstack/project-config] 981962: Retire Venus Project https://review.opendev.org/c/openstack/project-config/+/981962 | 13:15 | |
| @fungicide:matrix.org | wow, cpython sc is calling for a halt to jit development until it gets a more formal specification: https://discuss.python.org/t/an-announcement-from-the-steering-council-regarding-the-jit-project/107638 | 13:29 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 990847: Fix test jobs to work with buildkit attestations https://review.opendev.org/c/zuul/zuul-jobs/+/990847 | 14:08 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 990825: Add rootless podman support to ensure-podman https://review.opendev.org/c/zuul/zuul-jobs/+/990825 | 14:11 | |
| @clarkb:matrix.org | I will begin the early prep steps for the Gerrit upgrade in about 15 minutes. Plan is documented here: https://etherpad.opendev.org/p/gerrit-upgrade-3.13 | 14:45 |
| @fungicide:matrix.org | cool, i'm around and getting ready as well | 14:51 |
| @clarkb:matrix.org | I made some minor edits to the plan. I realized I didn't update the gerrit 3.13 quay.io image link after we rebuilt for the replication plugin fix so I updated that url. I also realized that when I removed the h2 cache move that we did for 3.11 to 3.12 to handle the h2 version change I didn't keep a step to delete the large h2 cache files we always want to delete so I added that in. Just some small details | 14:52 |
| @fungicide:matrix.org | oh, yep good call | 14:54 |
| @fungicide:matrix.org | status notice Gerrit on review.opendev.org is being upgraded to version 3.13 and will be offline starting at 1600 UTC. We have allocated an hour for the outage window lasting until 1700 UTC. | 14:59 |
| @fungicide:matrix.org | that's what you've got in the pad | 14:59 |
| @fungicide:matrix.org | should we append https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/UIFRQR34WYMF3KDMNQFMO222DIK3HTUD/ to that? | 14:59 |
| @clarkb:matrix.org | sure. Does that fit in the 512 character irc limit? | 15:00 |
| @fungicide:matrix.org | yes, it should | 15:00 |
| @clarkb:matrix.org | yes looks like ~292 bytes | 15:00 |
| @fungicide:matrix.org | want me to send that now? | 15:00 |
| @clarkb:matrix.org | do you want to send that now or should I? | 15:00 |
| @fungicide:matrix.org | i can | 15:00 |
| @fungicide:matrix.org | #status notice Gerrit on review.opendev.org is being upgraded to version 3.13 and will be offline starting at 1600 UTC. We have allocated an hour for the outage window lasting until 1700 UTC. https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/UIFRQR34WYMF3KDMNQFMO222DIK3HTUD/ | 15:00 |
| @clarkb:matrix.org | yup go for it. I'll edit the emergency file on bridge now | 15:00 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 15:00 |
| @fungicide:matrix.org | yeah, the full string came through as a single message in my irc client | 15:01 |
| @fungicide:matrix.org | also helpful for people following along on mastodon | 15:02 |
| @clarkb:matrix.org | I've edited the emregency file now | 15:02 |
| @clarkb:matrix.org | just double checking that I got all the host names correct | 15:02 |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 15:03 |
| -@status:opendev.org- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.13 and will be offline starting at 1600 UTC. We have allocated an hour for the outage window lasting until 1700 UTC. https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/UIFRQR34WYMF3KDMNQFMO222DIK3HTUD/ | 15:03 | |
| @clarkb:matrix.org | I have marked steps 5 and 6 on the etherpad as done now. And now it is time to make tea | 15:04 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 990828: Honor container_command in buildset-registry playbooks https://review.opendev.org/c/opendev/base-jobs/+/990828 | 15:08 | |
| @mordred:waterwanders.com | ^^ I don't super care about _running_ a buildset-registry with podman (I mean, shrug) but since the image build jobs use those playbooks too as part of "run or use a buildset registry" - it means that building an image on a podman node also installs docker. Which normally would be fine - except installing docker on resolute is currently broken. So - that's what got me down that rabbit hole. Should also fix installing docker on resolute, but might as well fix that up | 15:11 |
| @fungicide:matrix.org | Clark: https://review.opendev.org/c/openstack/project-config/+/991671 is a new project creation change that's ready to approve... should we plan to use that to exercise jeepyb once we're done with the upgrade? | 15:11 |
| @clarkb:matrix.org | fungi: I guess we can. The jeepyb gerritlib gerrit integration testing is tested with gerrit 3.13 so it should be covered there arleady | 15:12 |
| @fungicide:matrix.org | well, regardless we shouldn't merge it now because manage-projects won't get run with review.o.o in the disable list | 15:13 |
| @clarkb:matrix.org | correct it should wait for after the upgrade at this point | 15:13 |
| @clarkb:matrix.org | fungi: did you want to create the root screen so that your terminal paramters are respected? | 15:16 |
| @fungicide:matrix.org | done | 15:16 |
| @clarkb:matrix.org | thansk I have attached | 15:17 |
| @fungicide:matrix.org | 30 minutes to go | 15:30 |
| @clarkb:matrix.org | fungi: did you start the screen ^a H recording thing? | 15:36 |
| @clarkb:matrix.org | at 1600 I think you can send the final announcement and I will pause zuul queues. Then we can fight over who wants to drive in the screen to do the actual upgrade | 15:37 |
| @clarkb:matrix.org | ugh looks like backup02's disk has filled. We really need to filter out those emails better with the new infra root inbox | 15:38 |
| @clarkb:matrix.org | I'm going to start pruning on that server now | 15:38 |
| @clarkb:matrix.org | but I think we can use backup03 instead of backup02 for the mariadb backups during the upgrade process | 15:38 |
| @clarkb:matrix.org | heh pruning is failing because we write the pruning logs to the same volume and there is no disk space there to write to. I think we ignore this for the moment and use the new backup server for our backup as part of the upgrade. Then we can sort this out later | 15:41 |
| @clarkb:matrix.org | Looking at emails this started ~june 3 | 15:45 |
| @clarkb:matrix.org | so very recently. Anyway that is a problem for later. We have two other backup servers that should be in good order | 15:45 |
| @fungicide:matrix.org | ah yeah i'd fallen behind checking that inbox recently | 15:53 |
| @fungicide:matrix.org | i started logging in the root screen session now | 15:53 |
| @jim:acmegating.com | yell if you need me | 15:54 |
| @clarkb:matrix.org | thanks I did update the etehrpad to use backup03 in vexxhost instead of backup02 and I spot checked the rax backup server as well. I think the other two servers are good and we can proceed | 15:54 |
| @clarkb:matrix.org | corvus: will do | 15:54 |
| @clarkb:matrix.org | I wonder if we should put a ~500MB space reservation file in place on those volumes | 15:55 |
| @clarkb:matrix.org | better to run out of space and have something we can remove to make things happy again then run out of space and figure it out later. But also I should stop thinking about that for now and get into Gerrit upgrade mode | 15:55 |
| @jim:acmegating.com | do we run backups as a regular user? if so, could we run the prune as root in order to benefit from the root reservation? (assuming there is one) | 15:56 |
| @clarkb:matrix.org | yes backups are regular users dedicated to each backup target iirc | 15:56 |
| @clarkb:matrix.org | I did try to prune as root so maybe we don't have a reservation | 15:56 |
| @jim:acmegating.com | oh :( | 15:57 |
| @clarkb:matrix.org | fungi: you good with sending the final announcement while I pause zuul queues in a few minutes? | 15:57 |
| @fungicide:matrix.org | yep, i have it queued up | 15:59 |
| @fungicide:matrix.org | #status notice Gerrit on review.opendev.org is being upgraded to version 3.13 and will be offline momentarily. We have allocated an hour for the outage window lasting until 1700 UTC. https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/UIFRQR34WYMF3KDMNQFMO222DIK3HTUD/ | 15:59 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 15:59 |
| @clarkb:matrix.org | ok I'm pausing zuul now | 16:00 |
| @clarkb:matrix.org | fungi: once the announcement is done sending I think we can stop gerrit and proceed with the rest of the plan. Do you want to drive in screen for that or should I? | 16:01 |
| @fungicide:matrix.org | yes | 16:01 |
| @fungicide:matrix.org | i can drive | 16:02 |
| @clarkb:matrix.org | ack | 16:02 |
| -@status:opendev.org- NOTICE: Gerrit on review.opendev.org is being upgraded to version 3.13 and will be offline momentarily. We have allocated an hour for the outage window lasting until 1700 UTC. https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/UIFRQR34WYMF3KDMNQFMO222DIK3HTUD/ | 16:02 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 16:02 |
| @clarkb:matrix.org | ready when you are | 16:03 |
| @fungicide:matrix.org | starting now | 16:03 |
| @fungicide:matrix.org | or rather stopping | 16:03 |
| @fungicide:matrix.org | are we taking bets on how long it will be this time? ;) | 16:03 |
| @clarkb:matrix.org | up to ~45 seconds | 16:04 |
| @clarkb:matrix.org | I'll say 120 seconds | 16:04 |
| @fungicide:matrix.org | i'm at 90 | 16:04 |
| @fungicide:matrix.org | whoa 67.1s | 16:04 |
| @fungicide:matrix.org | okay, starting mariadb again | 16:04 |
| @clarkb:matrix.org | ya we restarted recently so that makes it quicker | 16:04 |
| @fungicide:matrix.org | backing up to the new backup server | 16:05 |
| @fungicide:matrix.org | and stopping mariadb again | 16:06 |
| @clarkb:matrix.org | `Fri Jun 5 16:06:35 UTC 2026 Backup finished with warnings.` the warnings are expceted so ya all looking good there | 16:06 |
| @clarkb:matrix.org | copying the indexes takes a bit. We've actually retained ab unch of old index versions as I guess gerritdoesn't really clean them up? One thing on my long todo list is to look into how safe it is to clean those up | 16:08 |
| @fungicide:matrix.org | copying indices is going to take a few minutes, i expect | 16:08 |
| @fungicide:matrix.org | yes | 16:08 |
| @clarkb:matrix.org | maybe keep N, N-1, and N-2 indexes for each index type and remove the others | 16:08 |
| @fungicide:matrix.org | that's probably fine | 16:08 |
| @clarkb:matrix.org | should be near complete | 16:11 |
| @fungicide:matrix.org | and done | 16:12 |
| @fungicide:matrix.org | tasks moved aside, deleting caches next | 16:12 |
| @fungicide:matrix.org | now editing the compose config | 16:12 |
| @fungicide:matrix.org | ready to pull new images? | 16:13 |
| @clarkb:matrix.org | I think so | 16:13 |
| @fungicide:matrix.org | in progress | 16:13 |
| @fungicide:matrix.org | weird, the etherpad ended up with a nonstandard space in the command | 16:14 |
| @clarkb:matrix.org | ya I wonder what I did there | 16:15 |
| @clarkb:matrix.org | the two hashes for the image lgtm though | 16:15 |
| @clarkb:matrix.org | They match the image at https://quay.io/repository/opendevorg/gerrit/manifest/sha256:0df8bf775fb96e77e668013a93305d999c3322b7613e6c32805e263c177685a8 | 16:15 |
| @fungicide:matrix.org | unicodedata.name() says `NO-BREAK SPACE` | 16:15 |
| @fungicide:matrix.org | fun stuff | 16:15 |
| @fungicide:matrix.org | okay, starting mariadb | 16:16 |
| @fungicide:matrix.org | ready for me to gerrit init? | 16:16 |
| @clarkb:matrix.org | I think so | 16:16 |
| @fungicide:matrix.org | it didn't report the expected warning, fwiw | 16:17 |
| @fungicide:matrix.org | but does claim to have initialized | 16:17 |
| @fungicide:matrix.org | and the moment of truth, ready for me to start gerrit? | 16:17 |
| @clarkb:matrix.org | ya the warning has to do with containers and since we down'd maybe its fine | 16:17 |
| @clarkb:matrix.org | oh its because the test env has the selenium container too I think | 16:18 |
| @clarkb:matrix.org | and yes ready to start gerrit | 16:18 |
| @fungicide:matrix.org | aha | 16:18 |
| @fungicide:matrix.org | okay, starting | 16:18 |
| @clarkb:matrix.org | `[2026-06-05T16:18:37.749Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.13.6-dirty ready` from the log | 16:18 |
| @clarkb:matrix.org | web ui loads for me and reports the 3.13.6 version as well. I can also see diffs in https://review.opendev.org/c/opendev/system-config/+/988310/29/playbooks/roles/greptimedb/defaults/main.yaml | 16:19 |
| @fungicide:matrix.org | so i guess we're up to step 24 | 16:19 |
| @fungicide:matrix.org | do we need to unpause zuul before testing further? | 16:19 |
| @jim:acmegating.com | https://review.opendev.org/c/zuul/zuul/+/991899 is a new change | 16:20 |
| @clarkb:matrix.org | fungi: no, I will restart zuul web and schedulers now so that they see the new gerrit version | 16:20 |
| @clarkb:matrix.org | if others can keep testing gerrit with zuul paused that would be great | 16:20 |
| @jim:acmegating.com | that change appears to function normally | 16:20 |
| @fungicide:matrix.org | oh, right | 16:20 |
| @clarkb:matrix.org | once zuul web and scheudlers are restarted I will unpause zuul | 16:20 |
| @clarkb:matrix.org | so check replication for 991899 etc | 16:20 |
| @clarkb:matrix.org | but I'll work on restarting zuul web and schedulers now | 16:20 |
| @fungicide:matrix.org | https://opendev.org/zuul/zuul/commit/114476603b85786a98bc41bd2bc0c1f73e0701a2 | 16:21 |
| @fungicide:matrix.org | the gitea link from that change has content loading for the commit | 16:21 |
| @jim:acmegating.com | yep that lgtm | 16:21 |
| @clarkb:matrix.org | https://zuul.opendev.org/components shows things as STOPPED | 16:22 |
| @clarkb:matrix.org | ah now they say initializing | 16:23 |
| @clarkb:matrix.org | that was what I expected so the STOPPED had me checking logs really quickly but I think that is just a set of state changes that it goes through | 16:23 |
| @jim:acmegating.com | yep. they start stopped. :) | 16:23 |
| @clarkb:matrix.org | corvus: did you want to do any checks between the first set and the second set of zuul udpates? | 16:24 |
| @jim:acmegating.com | i think the only thing we could do is check the log for the version detection; lemme look that up | 16:24 |
| @jim:acmegating.com | other than that, the next thing would be after we unpause and look at comments left | 16:25 |
| @fungicide:matrix.org | the config diff at step 27 is as described | 16:25 |
| @clarkb:matrix.org | fungi: cool can you mark that done on the etherpad? | 16:25 |
| @jim:acmegating.com | `Remote version is: 3.14.0-413-gfa8475c174 (parsed as (3, 14, 0))` | 16:26 |
| @clarkb:matrix.org | uh | 16:26 |
| @jim:acmegating.com | i think that's as good as we get until we try to leave a comment | 16:26 |
| @clarkb:matrix.org | thats weird that it says 3.14 there but in the web ui and the log file it says 3.13.6 | 16:27 |
| @jim:acmegating.com | that is, however, not the expected version is it? | 16:27 |
| @clarkb:matrix.org | corvus: correct. The web ui and the review_site/logs/error_log file both show `Gerrit Code Review 3.13.6-dirty read` | 16:27 |
| @mordred:waterwanders.com | is that showing up that way because it's commits on top of 3.13? | 16:27 |
| @jim:acmegating.com | oh oops | 16:27 |
| @jim:acmegating.com | we talk to 2 gerrit servers | 16:27 |
| @clarkb:matrix.org | corvus: aha | 16:27 |
| @jim:acmegating.com | 2026-06-05 16:22:11,565 INFO zuul.GerritConnection: Remote version is: 3.13.6-dirty (parsed as (3, 13, 6)) | 16:28 |
| @jim:acmegating.com | i think that one is us | 16:28 |
| @clarkb:matrix.org | yup that looks right | 16:28 |
| @fungicide:matrix.org | checked `gerrit show-queue -w -q` and as expected there are only typical tasks, no reindexing in progress | 16:28 |
| @clarkb:matrix.org | the two things I restarted say they are running now. Given the version clarification are we good with me doing the other set now? | 16:28 |
| @fungicide:matrix.org | i think so, yes | 16:28 |
| @jim:acmegating.com | yes | 16:29 |
| @clarkb:matrix.org | proceeding now | 16:29 |
| @clarkb:matrix.org | do you think we should wait for these to finish initializing before unpausing zuul or should we let the pair that is running do the work? | 16:30 |
| @fungicide:matrix.org | can't hurt to wait | 16:32 |
| @fungicide:matrix.org | we have plenty of time left in the window | 16:32 |
| @clarkb:matrix.org | wfm | 16:32 |
| @clarkb:matrix.org | one thing that occurs to me is that since there isn't an automatic reindexing we may consider doing a manual changes reindex like we do during normal gerrit restarts | 16:33 |
| @clarkb:matrix.org | to ensure that anything caught during the shutdown race is up to date when we're done | 16:33 |
| @fungicide:matrix.org | i can do that | 16:34 |
| @fungicide:matrix.org | `gerrit index start changes --force` going now | 16:35 |
| @clarkb:matrix.org | thanks | 16:35 |
| @fungicide:matrix.org | tailing the gerrit log in the screen session looking for `Reindex` lines to get periodic progress reports | 16:36 |
| @clarkb:matrix.org | ok all zuul components are RUNNING again. Ready for me to unpause zuul? | 16:36 |
| @clarkb:matrix.org | done | 16:37 |
| @clarkb:matrix.org | the event queue count is climbing as expected | 16:38 |
| @clarkb:matrix.org | zuul merged some changes: https://review.opendev.org/c/openstack/charm-glance/+/991809 | 16:39 |
| @clarkb:matrix.org | 991899 is queued up too | 16:39 |
| @clarkb:matrix.org | heh I think that caused a system load spike on gerrit (the zuul backup being processed all at once) | 16:40 |
| @clarkb:matrix.org | but trend is back down again | 16:40 |
| @clarkb:matrix.org | anyone have a change to recheck? | 16:41 |
| @clarkb:matrix.org | its the combo of reindexing and zuul having a bunch of work to do all at once creating the load I think | 16:41 |
| @jim:acmegating.com | the robot comments were file comments, so we should look for old and new file comments to check out yeah? | 16:42 |
| @clarkb:matrix.org | corvus: yes that is a good idea | 16:42 |
| @clarkb:matrix.org | I rechecked https://review.opendev.org/c/opendev/system-config/+/991512 | 16:44 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/zuul-providers] 991900: DNM: intentional syntax error https://review.opendev.org/c/opendev/zuul-providers/+/991900 | 16:44 | |
| @clarkb:matrix.org | I don't see my rechecked chagne queued up yet | 16:45 |
| @clarkb:matrix.org | but there are still events counted in the event queue so maybe I need to continue to be patient? | 16:46 |
| @jim:acmegating.com | 900 just got a report | 16:46 |
| @clarkb:matrix.org | https://review.opendev.org/c/openstack/sunbeam-charms/+/990926 this change was rechecekd independently and seems to have enqueued so I suspect patience is what i need | 16:47 |
| @clarkb:matrix.org | corvus: looks like the comments on 900 worked | 16:47 |
| @clarkb:matrix.org | yes 991512 has enqueued | 16:47 |
| @jim:acmegating.com | not sure where to find an old one | 16:47 |
| @clarkb:matrix.org | corvus: one of the i18n changes had a bunch that I was looking at the other day give me a moment | 16:48 |
| @clarkb:matrix.org | corvus: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/991432 here I think. It looks like the comments have basically just disappeared? | 16:48 |
| @clarkb:matrix.org | which makes sense since the web ui client is no longer fetching those comments at all so it never sees them to render them | 16:49 |
| @jim:acmegating.com | cool. cool cool. | 16:49 |
| @clarkb:matrix.org | which is unfortunate but all of the data is in the linter job logs and peopel can recheck to have them reapply as comments on the change | 16:49 |
| @fungicide:matrix.org | yeah, i guess it's just something to be aware of | 16:49 |
| @clarkb:matrix.org | yup and it should be a short term temporary problem as we roll forward with the new behavior | 16:50 |
| @jim:acmegating.com | in that case i wonder if zuul should go ahead and stop using robot comments unconditionally now | 16:50 |
| @jim:acmegating.com | would prepare other gerrit users for the future | 16:50 |
| @fungicide:matrix.org | i'm noticing my gertty has gotten itself upset, i approved https://review.opendev.org/c/openstack/project-config/+/991671 but my vote never got applied to gerrit | 16:50 |
| @clarkb:matrix.org | corvus: that is probably a good idea | 16:51 |
| @fungicide:matrix.org | seeing if restarting gertty gets things into a working state again | 16:51 |
| @clarkb:matrix.org | I have removed my WIP from https://review.opendev.org/c/opendev/system-config/+/989429 as 3.13 seems to be working and I don't think we're considering a rollback at this time | 16:51 |
| @clarkb:matrix.org | this is the change that updates config management. We should land that and remove the hosts from the emergency file so that it can apply | 16:51 |
| @clarkb:matrix.org | fungi: we still have hosts in the emergency file we don't want to approve 991671 yet | 16:52 |
| @fungicide:matrix.org | oh, excellent point | 16:52 |
| @clarkb:matrix.org | fungi: I wip'd it just in case gertty automatically approves it on startup | 16:52 |
| @fungicide:matrix.org | well, since it didn't get approved anyway, no harm done | 16:53 |
| @fungicide:matrix.org | gertty is entirely blocked for me at this point | 16:53 |
| @clarkb:matrix.org | I think the next steps are roughly monitor reindexing, land 989429 and remove hosts from emergency file so that can apply (whcih should noop), then we can approve 991671 | 16:53 |
| @jim:acmegating.com | remote_robot_comments_items = list(remote_revision.get('_gertty_remote_robot_comments_data', {}).items()) | 16:53 |
| @jim:acmegating.com | i think gertty may be broken by this | 16:53 |
| @clarkb:matrix.org | ah yup that api is completely removed in new gerrit | 16:54 |
| @clarkb:matrix.org | https://review.opendev.org/c/opendev/system-config/+/989429 will run the gitea job which takes some time. Do we want to go ahead and approev that now or would we like to get gertty working first? | 16:55 |
| @fungicide:matrix.org | corvus: agreed, that's what's coming up in the tracebacks in my gertty log too | 16:56 |
| @jim:acmegating.com | fungi: fix on the way | 16:56 |
| @fungicide:matrix.org | i'll hold off starting gertty yet anyway so that my approvals don't go through until we're settled on the upgrade tasks | 16:57 |
| @clarkb:matrix.org | reindexing is just over 50% done | 16:57 |
| @clarkb:matrix.org | I'm good with approving 989429 if others are | 16:58 |
| @fungicide:matrix.org | yeah, i already +2'd | 16:58 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [ttygroup/gertty] 991903: Remove robot comment support in later gerrit https://review.opendev.org/c/ttygroup/gertty/+/991903 | 16:58 | |
| @fungicide:matrix.org | (through webui) | 16:58 |
| @jim:acmegating.com | fungi: ^ | 16:58 |
| @fungicide:matrix.org | thanks corvus! | 16:58 |
| @clarkb:matrix.org | ok I'll self approve 989429 now. That will be in the gate for at least half an hour or so | 16:59 |
| @clarkb:matrix.org | but probably more like an hour | 16:59 |
| @fungicide:matrix.org | https://review.opendev.org/c/openstack/security-doc/+/991729 merged | 17:00 |
| @clarkb:matrix.org | ya there were openstack charms changes that merged immediately as they had finished testing during the zuul pause | 17:02 |
| @fungicide:matrix.org | so things are definitely moving | 17:02 |
| @clarkb:matrix.org | these version numbers don't actually mean anything but remember when we were stuck on 2.13? We've come a long way and are now running 3.13 | 17:02 |
| @fungicide:matrix.org | 991729 got a new patchset after we started gerrit 3.13, and then i approved it (through the webui) and then it completed check and went into the gate and merged, so full cycle | 17:03 |
| @clarkb:matrix.org | excellent | 17:03 |
| @clarkb:matrix.org | fungi: once reindexing is complete I think we can close the screen session and copy the screen log file into the upgrade-3.13 dir | 17:04 |
| @fungicide:matrix.org | yep, that was my plan | 17:04 |
| @clarkb:matrix.org | then when 989429 is about to merge I will remove hosts from the emergency file and we can check that applies cleanly | 17:04 |
| @clarkb:matrix.org | then we can land 991671. And in the meantime if anyone is adventurous they can look into pruning on backup02 :) | 17:05 |
| @fungicide:matrix.org | yeah, i can play around with pruning backups there in a few minutes | 17:05 |
| @clarkb:matrix.org | fungi: it might be sufficient to move some of the old backup logs out of /opt onto another fs because once pruning gets going in theory it will start freeing sufficient disk to continue | 17:06 |
| @clarkb:matrix.org | then we can move those logs back again once there is room | 17:06 |
| @fungicide:matrix.org | i was going to vaccuum old journals first and see where that gets us | 17:07 |
| @clarkb:matrix.org | those will be on the wrong gfs | 17:07 |
| @fungicide:matrix.org | oh, though that won't help, it's not the rootfs that filled up, yes | 17:07 |
| @clarkb:matrix.org | right | 17:07 |
| @fungicide:matrix.org | agreed, moving some backup logs to the rootfs temporarily should work | 17:08 |
| @fungicide:matrix.org | gerrit is 80% done reindexing | 17:08 |
| @fungicide:matrix.org | `Failed 3/985024 changes` | 17:15 |
| @fungicide:matrix.org | as expected | 17:15 |
| @clarkb:matrix.org | "perfect" | 17:15 |
| @clarkb:matrix.org | no failures is better but we expect these :) | 17:15 |
| @fungicide:matrix.org | `Reindex changes to version 87 complete` | 17:15 |
| @fungicide:matrix.org | so i'll close down and save the screen log now | 17:15 |
| @clarkb:matrix.org | I detached from the screen I think you can shut it down and put the screenlog in its forever home whenever you're ready | 17:15 |
| @fungicide:matrix.org | done | 17:16 |
| @clarkb:matrix.org | eta on 989429 is 59 minutes | 17:16 |
| @clarkb:matrix.org | which is good that means it should happen after the hourly runs | 17:16 |
| @clarkb:matrix.org | I'll monitor it and pull hosts out of the emergency file as it lands | 17:16 |
| @clarkb:matrix.org | fungi: another thing to check that I'm not super worried about is triggering replication manually. I will run `replication start opendev/bindep` here in a bit | 17:23 |
| @clarkb:matrix.org | we updated our container images for both 3.12 and 3.13 but its good to double check that doesn't fail now that we are on 3.13 | 17:23 |
| @clarkb:matrix.org | yup no failure reported when running that command over ssh | 17:24 |
| @clarkb:matrix.org | now that reindexing is done and zuul is operating at a more typical pace sytem load is back in the expected range for gerrit | 17:27 |
| @fungicide:matrix.org | great | 17:29 |
| @fungicide:matrix.org | pruning did not in fact work, moving `/opt/backups-202010/*.log` only freed up 20mb on the fs, which then got filled immediately before borg tried to create `/opt/backups/borg-gitea09/backup/lock.exclusive` and failed | 17:53 |
| @clarkb:matrix.org | fungi: gitea09 comes after eavesdrop and etherpad. Any idea if it pruned those? | 17:54 |
| @clarkb:matrix.org | nevermind found the logfile and those seem to have had the same lockfile issue | 17:55 |
| @fungicide:matrix.org | ah, no that was the final error it logged | 17:55 |
| @fungicide:matrix.org | while pruning eavesdrop02 is ran out of space | 17:56 |
| @fungicide:matrix.org | `OSError: [Errno 28] No space left on device: '/opt/backups/borg-eavesdrop02/.cache/borg/318ace037f82044d784b15faf1e5666e6a52955a2b4c71c4dfc73a4f07b8d246/chunks' -> '/opt/backups/borg-eavesdrop02/.cache/borg/318ace037f82044d784b15faf1e5666e6a52955a2b4c71c4dfc73a4f07b8d246/txn.tmp/chunks'` | 17:56 |
| @fungicide:matrix.org | anyway, i could move one of the larger backups, like `/opt/backups-202010/borg-wiki-update-test` | 17:57 |
| @fungicide:matrix.org | that should temporarily free a bunch of room | 17:57 |
| @clarkb:matrix.org | looks like there is also a borg compact command that is different to prune | 17:57 |
| @clarkb:matrix.org | maybe it is capable of operating in place in this situation? But ya moving something aside that is much larger should probably help then move it back and rerun again? | 17:58 |
| @fungicide:matrix.org | that's doable | 17:58 |
| @fungicide:matrix.org | trying now | 17:58 |
| @fungicide:matrix.org | oh, wait not that one | 17:59 |
| @fungicide:matrix.org | i was off by a digit | 17:59 |
| @fungicide:matrix.org | i have room to temporarily move `borg-zuul01` to the rootfs | 17:59 |
| @fungicide:matrix.org | so doing that instead | 17:59 |
| @clarkb:matrix.org | hourlies are just starting and zuul estimates ~12 minutes left on the gitea job for 989429 | 18:03 |
| @clarkb:matrix.org | I'll remove the nodes from the emergency file as soon as hourlies complete | 18:04 |
| @fungicide:matrix.org | moving that freed up 25gb (2%) on the volume, which will hopefully be enough headroom | 18:04 |
| @fungicide:matrix.org | then i can move it back and run again | 18:04 |
| @clarkb:matrix.org | sounds good | 18:04 |
| @clarkb:matrix.org | hourlies completed. I have removed the hosts from the emergency file so 989429 should enqueue and apply as soon as it merges | 18:08 |
| @fungicide:matrix.org | yeah, should be merging shortly | 18:21 |
| @clarkb:matrix.org | its behind the estimate but should be running testinfra tests soon if it hasn't started yet since I last checked the stream output | 18:22 |
| @clarkb:matrix.org | nope still waiting for testinfra tests to start | 18:23 |
| @fungicide:matrix.org | yeah, it finished repo creation about 8 minutes ago | 18:23 |
| @fungicide:matrix.org | uploading logs now | 18:23 |
| @fungicide:matrix.org | er, i guess not the final logs, it's on selenium stuff | 18:24 |
| @clarkb:matrix.org | ya this is the testinfra test setup it should be done soon. but that stuff does take a few minutes to run | 18:24 |
| @fungicide:matrix.org | finally doing testinfra tests | 18:26 |
| @clarkb:matrix.org | it passed should merge in just a moment | 18:30 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 989429: Upgrade Gerrit to 3.13 https://review.opendev.org/c/opendev/system-config/+/989429 | 18:31 | |
| @clarkb:matrix.org | fungi: ^ is enqueued now. service-review should run first the manage-projects. Both jobs should effectively noop | 18:32 |
| @clarkb:matrix.org | once that is done I'll check the docker compose file on review03 and if it still looks good I'll remove my wip from that new project creation change | 18:33 |
| @fungicide:matrix.org | perfect | 18:33 |
| @clarkb:matrix.org | then you can use gertty to approve it afterall and I can find lunch | 18:34 |
| @clarkb:matrix.org | service review succeeded. docker-compose.yaml's timestamp is still `Jun 5 16:13` and `gerrit:3.13` still shows up twice in the file and `gerrit:3.12` does not show up at all. That lgtm | 18:37 |
| @clarkb:matrix.org | now for manage-projects to run | 18:37 |
| @clarkb:matrix.org | double success. I have dropped my WIP vote on https://review.opendev.org/c/openstack/project-config/+/991671 | 18:42 |
| @clarkb:matrix.org | fungi: ^ fyi fi you want to try gertty again to approve it. Also looks like backup02 pruning did prune quite a bit of space, but zuul01 still needs to be moved back and pruning rerun? | 18:42 |
| @clarkb:matrix.org | it is just about lunch time here. I'm going to work on making and eating a sandwich. I think the core portion of the gerrit upgrade is no complete | 18:43 |
| @fungicide:matrix.org | thanks! | 18:43 |
| @fungicide:matrix.org | gertty is indeed back to not erroring after cherry-picking in corvus's patch | 18:44 |
| @clarkb:matrix.org | * it is just about lunch time here. I'm going to work on making and eating a sandwich. I think the core portion of the gerrit upgrade is now complete | 18:44 |
| @fungicide:matrix.org | and it did instantly apply my pending approval, as i suspected it might | 18:44 |
| @fungicide:matrix.org | `/opt/backups-202010` is back down to 70% used, so moving the zuul backup back into place and rerunning the prune again | 18:46 |
| @fungicide:matrix.org | i'm also going to try to clean out the trash in our root inbox | 18:47 |
| -@gerrit:opendev.org- Zuul merged on behalf of Andy Ning: [openstack/project-config] 991671: Add repo app-reloader for StarlingX https://review.opendev.org/c/openstack/project-config/+/991671 | 18:53 | |
| @fungicide:matrix.org | we ended at 72% utilization for the `/opt/backups-202010` filesystem on `backup02.ca-ymq-1.vexxhost` after the second prune pass with all moved files back in place | 18:54 |
| @clarkb:matrix.org | steady growth ... but the replacement avoids a lot of old data churn so that is the path forward | 18:55 |
| @clarkb:matrix.org | we just need to finish double checking it and get wiki pointed at it then we can clean up the old server | 18:55 |
| @clarkb:matrix.org | thank you for getting that sorted out | 18:55 |
| @clarkb:matrix.org | manage-projects is just starting for 991671 | 18:55 |
| @fungicide:matrix.org | getting wiki backing up to the new server is still on my to do list for "soon" | 18:58 |
| @clarkb:matrix.org | fungi: one thing to keep in mind is the ssh setup for that is a little weird. Its command specific iirc | 18:59 |
| @clarkb:matrix.org | so you should cross check the contents with other hosts or the existing config on the old one | 18:59 |
| @clarkb:matrix.org | manage-projects reports success | 18:59 |
| @fungicide:matrix.org | yeah, it's embedded in the authorized hosts list | 18:59 |
| @fungicide:matrix.org | i saw the rules, will be duplicating them | 18:59 |
| @clarkb:matrix.org | https://opendev.org/starlingx/app-reloader and https://review.opendev.org/admin/repos/starlingx/app-reloader,general exist | 18:59 |
| @clarkb:matrix.org | and now I will eat my sandwich. Thank you everyone for the help upgrading gerrit | 19:01 |
| @fungicide:matrix.org | i was able to add scott.little to the new `starlingx-app-reloader-core` group via ssh api too, so that seems to be working | 19:01 |
| @fungicide:matrix.org | cleaned up the shared root inbox removing ~6k noise messages, now cleaning up ~60k messages in the spam folder | 19:13 |
| @mordred:waterwanders.com | Huzzah! Thanks everyone for upgrading gerrit!! | 19:15 |
| @fungicide:matrix.org | 3.14 here we come! ;) | 19:17 |
| @clarkb:matrix.org | I will require everyone have pie ready for that upgrade | 19:32 |
| @clarkb:matrix.org | the timing of it should mean I will have apple pie filling from the tree in the backyard so that is probably what I'll end up with | 19:32 |
| @fungicide:matrix.org | good call | 19:34 |
| @fungicide:matrix.org | it'll be fig season here | 19:35 |
| @clarkb:matrix.org | fungi: should we go ahead and approve https://review.opendev.org/c/openstack/project-config/+/990373 at this point? I'm going to work on changes to cleanup 3.12 and add 3.14 this afternoon | 20:26 |
| @clarkb:matrix.org | I think the change to cleanup 3.12 will fail to even run tests until ^ lands hence my question | 20:28 |
| @clarkb:matrix.org | I guess we can just recheck everything next week too. I'm not in a rsuh | 20:29 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 991945: Drop gerrit 3.12 image builds and testing https://review.opendev.org/c/opendev/system-config/+/991945 | 20:30 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 991946: Add Gerrit 3.14 image builds and testing https://review.opendev.org/c/opendev/system-config/+/991946 | 20:38 | |
| @fungicide:matrix.org | looking | 21:05 |
| @fungicide:matrix.org | yeah, approved 990373 | 21:05 |
| @clarkb:matrix.org | Thanks I'll recheck the other two as soon as that lands | 21:12 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [openstack/project-config] 990373: Update jeepyb gerrit build to 3.13 https://review.opendev.org/c/openstack/project-config/+/990373 | 21:12 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 991947: Switch gerrit-3.12 to gerrit-3.13 in one more location https://review.opendev.org/c/openstack/project-config/+/991947 | 21:17 | |
| @clarkb:matrix.org | fungi: ^ arg I missed on more location where the jobs needed updating for jeepyb | 21:17 |
| @clarkb:matrix.org | * fungi: ^ arg I missed one more location where the jobs needed updating for jeepyb | 21:18 |
| @fungicide:matrix.org | https://review.opendev.org/990377 would also be good to get in. i just self-approved its depends-on a moment ago | 21:22 |
| @clarkb:matrix.org | Approved | 21:32 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [openstack/project-config] 991947: Switch gerrit-3.12 to gerrit-3.13 in one more location https://review.opendev.org/c/openstack/project-config/+/991947 | 21:32 | |
| @fungicide:matrix.org | thanks! | 21:33 |
| @clarkb:matrix.org | the gerrit 3.14 builds are queued up so that should be all the jeepyb fixup we need | 21:37 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [zuul/zuul-jobs] 990392: Add a nox-py314 job https://review.opendev.org/c/zuul/zuul-jobs/+/990392 | 21:38 | |
| @clarkb:matrix.org | 3.14 Gerrit builds failed on bezel(isk) issue. I won't debug that on a Friday | 22:11 |
| @clarkb:matrix.org | Presumably I can cross check against upstream builds and figure it out from there | 22:11 |
| @fungicide:matrix.org | we've got a little while until pie | 22:12 |
| @mordred:waterwanders.com | but I like pie | 22:16 |
| @fungicide:matrix.org | okay, you can have pie sooner | 22:18 |
| @fungicide:matrix.org | just make it build | 22:19 |
| @fungicide:matrix.org | to paqraphrase the great and holy prophet sagan, "to make gerrit pie from scratch, you must first fix bazelisk" | 22:20 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/git-review] 990377: Integration testing Gerrit 3.13 and Python 3.14 https://review.opendev.org/c/opendev/git-review/+/990377 | 22:52 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!