| @clarkb:matrix.org | fungi: any chance today is a good day to review those pre gerrit upgrade changes, hopefully land then, then restart Gerrit with the updates? | 14:23 |
|---|---|---|
| @fungicide:matrix.org | yes, i'm just about caught up on e-mail backlog and those are the top of my review priority list | 14:26 |
| @clarkb:matrix.org | excellent. I expect to be around all day and I think this is my main concern to make sure I have time through the rest of the week to retest things with the new images and finalize the TODO prep for the upgrade | 14:26 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 983318: Use Haproxy PROXY Protocol with Gitea https://review.opendev.org/c/opendev/system-config/+/983318 | 14:35 | |
| @fungicide:matrix.org | https://github.com/TecharoHQ/anubis/issues/1208 seems to be the feature request to support that in anubis | 14:47 |
| @fungicide:matrix.org | there's a pr under review to implement since a few months ago | 14:48 |
| @clarkb:matrix.org | fungi: I'm hoping that if we have apache read those headers and set the appropriate logging/http headers then we may not need explicit support for it down the line | 14:51 |
| @clarkb:matrix.org | but its a bit of an experiment still | 14:51 |
| @fungicide:matrix.org | yeah, it was more of a "this would let us trace source/destination all the way through to gitea-web logs" for anubis | 14:53 |
| @fungicide:matrix.org | since the anubis implementation change seems to work, and we're not currently in a place where it's urgent that we turn it on, i'd be fine with the idea of just keeping it in reserve until it becomes necessary | 14:53 |
| @fungicide:matrix.org | and maybe at that point, it will have proxy protocol support | 14:54 |
| @clarkb:matrix.org | looks like all 4 changes are in the gate. Thanks! | 15:11 |
| @fungicide:matrix.org | yeah, i'm going to pop out in a bit for a quick lunch, but should be around the rest of the day after ~16:30z and happy to help/run a restart whenever is convenient | 15:12 |
| @clarkb:matrix.org | Excellent | 15:17 |
| @fungicide:matrix.org | okay, disappearing, back in an hour | 15:20 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 981917: Update Gerrit images to 3.11.10, 3.12.6, and 3.13.5 https://review.opendev.org/c/opendev/system-config/+/981917 | 15:49 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: | 15:54 | |
| - [opendev/system-config] 981922: Disable the Gerrit per request ref cache https://review.opendev.org/c/opendev/system-config/+/981922 | ||
| - [opendev/system-config] 981923: Increase Gerrit's stop grace period to 15 minutes https://review.opendev.org/c/opendev/system-config/+/981923 | ||
| @clarkb:matrix.org | One the last one lands I'll double check the config on review03. The image updates appear to have promoted successfully at least | 15:59 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 982125: Explicitly enable Gerrit robot comments https://review.opendev.org/c/opendev/system-config/+/982125 | 16:12 | |
| @fungicide:matrix.org | we're still waiting on one more, yeah? | 16:22 |
| @fungicide:matrix.org | no, i guess that was the fourth one | 16:23 |
| @clarkb:matrix.org | yup that was the last one | 16:23 |
| @clarkb:matrix.org | Deployment succeeded. I'll check review03 now | 16:24 |
| @clarkb:matrix.org | all three config updates (2 in gerrit.config and one in docker-compose.yaml) are in place | 16:26 |
| @clarkb:matrix.org | I think we can prepare to restart whenever we're ready? | 16:26 |
| @fungicide:matrix.org | i have a root screen session open on review03 and ready to pull | 16:31 |
| @clarkb:matrix.org | cool I've joined the screen and I'm working to find the image on quay now | 16:32 |
| @clarkb:matrix.org | https://quay.io/repository/opendevorg/gerrit/manifest/sha256:020e2191e9a180f24a0f6c559ab220bf9623a8a4bfdf803dd2f559ff56601f22 this appears to be the 3.11 image | 16:32 |
| @fungicide:matrix.org | pulling | 16:32 |
| @fungicide:matrix.org | `quay.io/opendevorg/gerrit 3.11 5e12cf38025e About an hour ago 715MB` | 16:33 |
| @fungicide:matrix.org | `"quay.io/opendevorg/gerrit@sha256:020e2191e9a180f24a0f6c559ab220bf9623a8a4bfdf803dd2f559ff56601f22",` | 16:33 |
| @clarkb:matrix.org | those seem to match | 16:33 |
| @fungicide:matrix.org | based on earlier restarts, i propose to run `docker compose -f /etc/gerrit-compose/docker-compose.yaml down && mv ~gerrit2/review_site/data/replication/ref-updates/waiting ~gerrit2/tmp/waiting_queue_2026-04-06 && rm ~gerrit2/review_site/cache/{gerrit_file_diff,git_file_diff,git_modified_files,modified_files,comment_context}.* && sudo docker compose -f /etc/gerrit-compose/docker-compose.yaml up -d` | 16:34 |
| @clarkb:matrix.org | we could also add the change_kind and diff_summary caches to the cache clearing list | 16:35 |
| @clarkb:matrix.org | those have both grown to > 1GB. But otherwise that command lgtm | 16:36 |
| @fungicide:matrix.org | `docker compose -f /etc/gerrit-compose/docker-compose.yaml down && mv ~gerrit2/review_site/data/replication/ref-updates/waiting ~gerrit2/tmp/waiting_queue_2026-04-06 && rm ~gerrit2/review_site/cache/{gerrit_file_diff,git_file_diff,git_modified_files,modified_files,comment_context,change_kind,diff_summary}.* && sudo docker compose -f /etc/gerrit-compose/docker-compose.yaml up -d` | 16:36 |
| @fungicide:matrix.org | like that? | 16:37 |
| @fungicide:matrix.org | status notice The Gerrit service on review.opendev.org will be offline briefly to upgrade to a newer 3.11 patch release and apply some configuration updates in preparation for an upgrade to 3.12 in the near future | 16:37 |
| @clarkb:matrix.org | yup that looks great | 16:37 |
| @fungicide:matrix.org | also does that notice look good? | 16:37 |
| @clarkb:matrix.org | I was just double checking the strings against what ls reports and I think everything matches | 16:37 |
| @clarkb:matrix.org | yes the notice looks great too | 16:37 |
| @fungicide:matrix.org | okay, i'll send that first | 16:37 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will be offline briefly to upgrade to a newer 3.11 patch release and apply some configuration updates in preparation for an upgrade to 3.12 in the near future | 16:37 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 16:37 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline briefly to upgrade to a newer 3.11 patch release and apply some configuration updates in preparation for an upgrade to 3.12 in the near future | 16:40 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 16:40 |
| @fungicide:matrix.org | ready to restart now if no immediate objections | 16:41 |
| @clarkb:matrix.org | yup I'm here and ready to watch and check afetwards | 16:41 |
| @fungicide:matrix.org | underway | 16:41 |
| @fungicide:matrix.org | the gerrit container has been stopping for one minute so far | 16:42 |
| @clarkb:matrix.org | based on ls against the cache dir it is chewing on git_file_diff | 16:43 |
| @clarkb:matrix.org | every other cache has lost its lock fiel | 16:43 |
| @fungicide:matrix.org | two minutes elapsed now | 16:43 |
| @fungicide:matrix.org | took 135.5s | 16:44 |
| @fungicide:matrix.org | gerrit is on its way back up | 16:44 |
| @clarkb:matrix.org | `[2026-04-06T16:44:37.644Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.11.10-dirty ready` | 16:44 |
| @fungicide:matrix.org | webui footer has `Powered by Gerrit Code Review (3.11.10-dirty)` | 16:44 |
| @clarkb:matrix.org | still waiting for diffs to load, but this is a known slow startup behavior | 16:45 |
| @fungicide:matrix.org | overall this was a rather quick restart | 16:45 |
| @clarkb:matrix.org | yup diffs load for me now | 16:45 |
| @clarkb:matrix.org | next on the check list is pushing a patchset or finding one that someone else has pushed one since the restart and checking replication | 16:46 |
| @clarkb:matrix.org | Nothing has been pushed yet | 16:47 |
| @clarkb:matrix.org | https://review.opendev.org/c/openstack/cinder/+/950546 is a new patchset | 16:49 |
| @clarkb:matrix.org | and seems to have replicated here: https://opendev.org/openstack/cinder/commit/8b5afe677090f9dba3e36a803126a4982d98d940 | 16:50 |
| @clarkb:matrix.org | fungi: ^ anything else you think we should check? This is looking happy I think | 16:50 |
| @fungicide:matrix.org | nope, lgtm | 16:50 |
| @fungicide:matrix.org | i was trolling gerritbot logs for a new upload, but you beat me to kt | 16:50 |
| @fungicide:matrix.org | to it | 16:50 |
| @clarkb:matrix.org | I tend to just spam refresh the open changes list on Gerrit which gives you a mix of new comments and patchsets and eventually you find one | 16:51 |
| @clarkb:matrix.org | Ok my next step is to refresh my held test nodes so that they are all using the new images and then I can rerun the upgrade process on those nodes to double check everything continues to look the way it did previously | 16:51 |
| @clarkb:matrix.org | re the gitea PROXY change it is still unhappy and I've run out of ideas so that may need a held node too. But I probably won't focus on that this week since I need to get back to Gerrit stuff | 16:52 |
| @clarkb:matrix.org | oh also a friendly reminder to let me know if there are meeting agenda edits that should be made. Or make them on the wiki directly. I'll get the agenda sent out later today | 16:53 |
| @fungicide:matrix.org | `sudo tail -F /var/log/containers/docker-gerritbot.log | grep patchset-created | sed s"#^\(.\{19\}\).*'revision': '\([0-9a-f]\{40\}\)', .*, 'project': '\([a-z0-9/]*\)'.*#\1 https://opendev.org/\3/commit/\2#"` | 17:07 |
| @fungicide:matrix.org | for future reference, recipe to return a timestamp and gitea commit url for each new patchset-created event logged by gerritbot on eavesdrop02 | 17:08 |
| @fungicide:matrix.org | i closed out the root screen session on review03 | 17:55 |
| @clarkb:matrix.org | Thanks | 18:05 |
| @clarkb:matrix.org | I'm needing out on the moon flyby | 18:05 |
| @clarkb:matrix.org | * I'm nerding out on the moon flyby | 18:05 |
| @fungicide:matrix.org | there's little better to nerd out on, hard to blame you | 18:08 |
| @clarkb:matrix.org | they just named a couple of craters on the moon live on youtube. Pretty neat | 18:15 |
| @tafkamax:matrix.org | Crater McCraterson | 19:09 |
| @fungicide:matrix.org | always a favorite in public polls | 19:11 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 983474: Bump Gerrit to 3.12 https://review.opendev.org/c/opendev/system-config/+/983474 | 20:40 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 983475: Bump jeepyb gerrit jobs up to gerrit 3.12 https://review.opendev.org/c/openstack/project-config/+/983475 | 20:44 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/gerritlib] 983476: Update the Gerrit test images https://review.opendev.org/c/opendev/gerritlib/+/983476 | 20:46 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 983475: Bump jeepyb gerrit jobs up to gerrit 3.12 https://review.opendev.org/c/openstack/project-config/+/983475 | 20:47 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/jeepyb] 983482: Test Gerrit 3.11, 3.12, and 3.13 with Gerritlib https://review.opendev.org/c/opendev/jeepyb/+/983482 | 20:49 | |
| @clarkb:matrix.org | 983474 and 983475 are both post upgrade changes. Then the other two are things I noticed that would be good to test before the upgrade but don't need to land before the upgrade (the speculative testing is sufficient) | 20:50 |
| @clarkb:matrix.org | but this is all coming out of me working through the remaining TODOs on our upgrade document | 20:51 |
| @clarkb:matrix.org | assuming these test runs all come back happy the next step is to manually work through the upgrade process on the newly held node using the latest images just to be sure nothing changed with these bugfix builds. I'll probably get to taht tomorrow | 20:53 |
| @clarkb:matrix.org | cool those are looking happy so far. I need to run an errand while I wait for them to complete. Then I'll get the meetin agenda out when I get back. I made some edits already. Please let me know if there are more edits that should be applied | 21:01 |
| @clarkb:matrix.org | fungi: one thing on my April todo list is the infra-root email copying. You had mentioned you could do it (which is much appreciated since my thunderbird experience was meh) I don't think that has been done yet. We have about 2 weeks left before the data goes away in the old account iirc. | 21:59 |
| @fungicide:matrix.org | it has not been done yet, i may need my memory refreshed as to which specific folders we wanted to move over and which ones we wanted to just clear out | 22:00 |
| @clarkb:matrix.org | looks like we discussed this on March 19 /me looks for and eavesdrop link | 22:26 |
| @clarkb:matrix.org | fungi: https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2026-03-19.log.html#opendev.2026-03-19.log.html#t2026-03-19T20:47:26 | 22:27 |
| @fungicide:matrix.org | thanks, i've attached that to my to do list entry | 22:30 |
| @clarkb:matrix.org | ok last call on meeting agenda edits. Otherwise this agenda is going out in the next few minutes | 22:34 |
| @clarkb:matrix.org | fungi: (not urgent) do you know if we land https://review.opendev.org/c/opendev/system-config/+/983221 does that impact mirroring of the other distro releases until we clear out the old stuff? or do we just get a warning saying there is unknown data in the repo? I feel like we just get a warning and can land that change then work on the manual cleanup at our own pace? | 22:56 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!