| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/system-config] 974381: Grant fungi access to statusbot in Matrix https://review.opendev.org/c/opendev/system-config/+/974381 | 00:32 | |
| -@gerrit:opendev.org- László Jánosi proposed: [opendev/glean] 974346: Fix vlan and bond config generation https://review.opendev.org/c/opendev/glean/+/974346 | 08:00 | |
| @harbott.osism.tech:regio.chat | seems we were having issues on the docs server yesterday, too, see e.g. https://zuul.opendev.org/t/openstack/build/58c14c070bd749a5943e303cb9c004da will see whether the recheck will work now, manually checking some of the affected urls did work fine for me right now | 09:25 |
|---|---|---|
| @harbott.osism.tech:regio.chat | regarding statusbot auth, I don't think I'd want my current matrix account visible in a repo, do you think we could consider handling that via a secret? | 09:28 |
| @harbott.osism.tech:regio.chat | I'm also noticing that I've had to completely mute this channel in order to avoid being overloaded by notifications, sadly this also seems to result in the `infra-root` trigger to be muted. for now please ping me in IRC if something urgent comes up | 09:29 |
| @fungicide:matrix.org | we could probably set the statusbot nicklist in a private hostvar rather than the public inventory | 14:32 |
| @clarkb:matrix.org | fungi: Jens Harbott yes, that same variable can be set in private vars and I believe the private var version should win due to variable precedence. But that is also why I didn't blindly assume I should add everyone to that list in the repo | 15:45 |
| @clarkb:matrix.org | also for notifications element lets you set the notification to on which is just a visual indicator or noisy which is the audible ping as well as the visual indicator | 15:47 |
| @clarkb:matrix.org | that may be helpful if the problem is the audio portion of the notification | 15:47 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/irc-meetings] 974444: Update irc meetings index page with Matrix https://review.opendev.org/c/opendev/irc-meetings/+/974444 | 15:54 | |
| @clarkb:matrix.org | fungi: I've double checked screenshots for https://review.opendev.org/c/opendev/system-config/+/970451 and did a quick skim of the bazel build logs. I don't see anything concerning with the newer version of nodejs. Is today still a good day for a Gerrit restart on your end? And if so should I approve this change now? | 15:56 |
| @fungicide:matrix.org | yeah, let's go for it | 15:57 |
| @clarkb:matrix.org | change is approved | 15:59 |
| @fungicide:matrix.org | standing by to schedule a restart once it promotes/deploys | 16:00 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [openstack/project-config] 974462: Simplify Zuul tenant config for providers https://review.opendev.org/c/openstack/project-config/+/974462 | 16:07 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/irc-meetings] 974444: Update irc meetings index page with Matrix https://review.opendev.org/c/opendev/irc-meetings/+/974444 | 16:22 | |
| @fungicide:matrix.org | looks like the gerrit change is about to merge | 17:19 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/system-config] 970451: Update node to 24 for Gerrit builds https://review.opendev.org/c/opendev/system-config/+/970451 | 17:19 | |
| @fungicide:matrix.org | deploy jobs are already running | 17:21 |
| @fungicide:matrix.org | and done | 17:22 |
| @fungicide:matrix.org | i guess we can prepare to restart any time now | 17:22 |
| @clarkb:matrix.org | elodilles mentioned trying to get things prepped for some Openstack release stuff and mentioned being done by 1800 UTC. I said that we could wait until then | 17:23 |
| @fungicide:matrix.org | yep, should be fine | 17:23 |
| @clarkb:matrix.org | But yes I think we should start prepping and then restart around then | 17:23 |
| @fungicide:matrix.org | i have a root screen session open on review03 | 17:24 |
| @fungicide:matrix.org | quay.io/opendevorg/gerrit 3.11 f43f988a6736 6 days ago 718MB | 17:25 |
| @clarkb:matrix.org | The thing we didn't do last time that we should be better about is pausing the Gerrit results queue | 17:25 |
| @fungicide:matrix.org | that's what we're running now, i think | 17:25 |
| @clarkb:matrix.org | corvus: do you know if the queue pausing script has been fixed in zuul or do we still ahve to use the special copy on zuul01? | 17:25 |
| @clarkb:matrix.org | fungi: yes that image look correct | 17:26 |
| @clarkb:matrix.org | as the current image I mean | 17:26 |
| @fungicide:matrix.org | i'll go ahead and pull unless there are objections | 17:26 |
| @clarkb:matrix.org | fungi: I think you can pull | 17:27 |
| @jim:acmegating.com | Clark: refresh my memory please, why was it broken? or alternatively, where's the special copy? | 17:27 |
| @clarkb:matrix.org | `/root/zuul-client manage-events --all-tenants --reason "Gerrit restart in progress" pause-result` and `/root/zuul-client manage-events --all-tenants normal` are the two pause commands | 17:27 |
| @clarkb:matrix.org | corvus: ^ that /root/zuul-client special copy is on zuul01. I don't recall what the specific fix was | 17:28 |
| @jim:acmegating.com | that looks like it's just a docker run command to simplify using zuul-client | 17:28 |
| @clarkb:matrix.org | ah ok so maybe it did get fixed | 17:28 |
| @fungicide:matrix.org | quay.io/opendevorg/gerrit 3.11 5bebd6c38d59 51 minutes ago 716MB | 17:28 |
| @fungicide:matrix.org | `"RepoDigests": [ "quay.io/opendevorg/gerrit@sha256:9081c24e5ca0b9f87dcacc038cc9f7a33fdee739ee11f4acdcbce0e62424d9aa", "quay.io/opendevorg/gerrit@sha256:fce878a59a63f1decf78c5d330f6bbaae453354e04ffeae3694221605feb0277"` | 17:29 |
| @clarkb:matrix.org | fungi: the inspect command on 5beb... seems to match the hashes at https://quay.io/repository/opendevorg/gerrit/manifest/sha256:9081c24e5ca0b9f87dcacc038cc9f7a33fdee739ee11f4acdcbce0e62424d9aa | 17:30 |
| @jim:acmegating.com | it doesn't look like there was an update to zuul-client manage-events after the initial merge... | 17:30 |
| @clarkb:matrix.org | huh those are the commands in root's history for `manage-events` on zuul01. Let me see if I can find more info in IRC historiy | 17:31 |
| @jim:acmegating.com | is there a possibility it wasn't broken? :) | 17:31 |
| @fungicide:matrix.org | i've got this queued up in the screen, ready to execute at 18:00 utc: `docker compose -f /etc/gerrit-compose/docker-compose.yaml down && mv ~gerrit2/review_site/data/replication/ref-updates/waiting ~gerrit2/tmp/waiting_queue_2026-01-23 && rm ~gerrit2/review_site/cache/{gerrit_file_diff,git_file_diff,git_modified_files,modified_files,comment_context}.* && sudo docker compose -f /etc/gerrit-compose/docker-compose.yaml up -d` | 17:32 |
| @clarkb:matrix.org | corvus: https://review.opendev.org/c/opendev/system-config/+/967111 this was the fix according to IRC history. The bug was in the helper script not the zuul-client tool | 17:32 |
| @jim:acmegating.com | i do notice that zuul-client is building from dockerhub. and https://review.opendev.org/953231 never merged but i think events have passed that one by and we need a new change now? | 17:32 |
| @fungicide:matrix.org | how's this look for the announcement? `status notice The Gerrit service on review.opendev.org will be offline momentarily while we restart it for an upgrade to NodeJS 24` | 17:34 |
| @clarkb:matrix.org | fungi: that command lgtm. We probably don't need to clear out all of those caches but ti didn't seem to hurt last time so I think we can proceed with that | 17:34 |
| @clarkb:matrix.org | fungi: In the announcement I would say something like "while we restart to pick up new container images after updating the build tooling" or similar. Nodejs isn't a runtime thing onl a build side item | 17:35 |
| @clarkb:matrix.org | mostly I don't want to confuse people and have them think gerrit is now a nodejs application | 17:35 |
| @fungicide:matrix.org | so more like: `status notice The Gerrit service on review.opendev.org will be offline momentarily while we restart it onto new images after an update to the build toolchain` | 17:37 |
| @clarkb:matrix.org | ++ | 17:37 |
| @clarkb:matrix.org | fungi: I can run the zuul-client commands to pause then unpause things | 17:51 |
| @fungicide:matrix.org | thanks | 17:51 |
| @fungicide:matrix.org | ready for me to send the notice? | 17:59 |
| @clarkb:matrix.org | yup | 17:59 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will be offline momentarily while we restart it onto new images after an update to the build toolchain | 17:59 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 17:59 |
| @fungicide:matrix.org | yay, it's working this time | 18:00 |
| @clarkb:matrix.org | let me know when I should pause zuul queues | 18:00 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily while we restart it onto new images after an update to the build toolchain | 18:02 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 18:02 |
| @clarkb:matrix.org | maybe now that the notice is done I should stop queues then we restart gerrit? | 18:03 |
| @fungicide:matrix.org | Clark: i guess go ahead, and then i'll restart gerrit once you're done | 18:03 |
| @clarkb:matrix.org | I'm getting a token has expired error from zuul-client | 18:03 |
| @clarkb:matrix.org | corvus: is this something I can do logged into the web ui? | 18:04 |
| @fungicide:matrix.org | i bet that's fallout from the rotation | 18:04 |
| @clarkb:matrix.org | yes, I'm guessing the old token has no expiration or it was like 10 years in the future and this one doesn't | 18:04 |
| @fungicide:matrix.org | i updated the token for zc | 18:04 |
| @fungicide:matrix.org | oh, maybe that's my fault | 18:04 |
| @fungicide:matrix.org | i think i just used the command in the zuul docs | 18:05 |
| @clarkb:matrix.org | I tried on zuul02 and got the same thing | 18:05 |
| @clarkb:matrix.org | looks like I can do it through the web ui so give me a minute | 18:06 |
| @clarkb:matrix.org | fungi: that is done | 18:08 |
| @clarkb:matrix.org | corvus: looks like with the cli you can pause only result queues but the web ui only lets you do trigger + result queues (so I did that) | 18:08 |
| @clarkb:matrix.org | oh and it is per tenant? | 18:08 |
| @fungicide:matrix.org | https://zuul-ci.org/docs/zuul/latest/client.html#create-auth-token is what i did previously | 18:08 |
| @jim:acmegating.com | Clark: the web ui is per-tenant, the script does all tenants | 18:09 |
| @clarkb:matrix.org | corvus: ack I did openstack and zuul tenants as they are the active ones right now | 18:09 |
| @clarkb:matrix.org | I think that should be sufficient for this restart | 18:09 |
| @fungicide:matrix.org | i wonder if i actually copied `--expires-in 1800` | 18:10 |
| @clarkb:matrix.org | fungi: looks like an extra character snuck into your command in screen | 18:10 |
| @jim:acmegating.com | Clark: pause-result includes pause-trigger | 18:11 |
| @clarkb:matrix.org | I don't knwo if I did that with my xmonad window switching | 18:11 |
| @clarkb:matrix.org | fungi: its -n d instead of -d | 18:11 |
| @fungicide:matrix.org | ah, i did `sudo docker run --rm --volume=/etc/zuul:/etc/zuul -it quay.io/zuul-ci/zuul-scheduler:latest zuul-admin create-auth-token --auth-config local --user localauth` but not specifying `--expires-in` defaulted to 600 (10 minutes i guess) | 18:11 |
| @fungicide:matrix.org | mea culpa! | 18:11 |
| @clarkb:matrix.org | fungi: I think we can proceed if we fix the docker compose up -d command | 18:12 |
| @fungicide:matrix.org | and yeah, fixed the command in screen already | 18:12 |
| @clarkb:matrix.org | oh yup it is fixed now | 18:12 |
| @fungicide:matrix.org | running | 18:12 |
| @fungicide:matrix.org | funny that stopping gerrit is taking longer this time even though it's only been a week | 18:14 |
| @fungicide:matrix.org | 106.3s | 18:14 |
| @clarkb:matrix.org | git_file_diff.h2.db is the last h2 db file with a lock file in place so I think it is processing that one | 18:14 |
| @clarkb:matrix.org | `[2026-01-23T18:14:47.668Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.11.7-22-g66e6095dcb-dirty ready` | 18:15 |
| @fungicide:matrix.org | webui is up | 18:15 |
| @fungicide:matrix.org | shall i trigger reindexing or wait? | 18:16 |
| @clarkb:matrix.org | and diffs are loading for me. Should I go ahead and unpause event queues in openstack and zuul tenants? | 18:16 |
| @jim:acmegating.com | i have a patch to push up, lmk when you want it. | 18:16 |
| @fungicide:matrix.org | yeah should be fine to unpause | 18:16 |
| @clarkb:matrix.org | corvus: let me unpause, then you can push then if that all looks good (replication etc) then we can trigger reindexing | 18:16 |
| @fungicide:matrix.org | wfm | 18:16 |
| @clarkb:matrix.org | corvus: openstack and zuul tenants should be unpaused now | 18:17 |
| @fungicide:matrix.org | i guess we still need a new longer-term token generated and added to hostvars, i can do that when we're done | 18:18 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 974540: Remove statusbot from #opendev irc https://review.opendev.org/c/opendev/system-config/+/974540 | 18:18 | |
| @clarkb:matrix.org | 974540 replicated to gitea and I can see its matching diffs in gitea and gerrit. Zuul is still working through the event queues | 18:19 |
| @clarkb:matrix.org | corvus: I don't see 974540 in openstack's zuul tenant check queue yet | 18:20 |
| @fungicide:matrix.org | are event queues still burning down? | 18:21 |
| @clarkb:matrix.org | oh now it says there are 2 events though (it said 0 before but maybe I caught it while it was still shuffling stuff around) | 18:21 |
| @fungicide:matrix.org | ah | 18:21 |
| @clarkb:matrix.org | it want from 2 -> 6 -> 11 -> 12 so maybe i caught it before it actually started seeing those events | 18:21 |
| @clarkb:matrix.org | * it went from 2 -> 6 -> 11 -> 12 so maybe i caught it before it actually started seeing those events | 18:22 |
| @jim:acmegating.com | it's there now. looks like there was some stuff going on in the openstack tenant. | 18:22 |
| @jim:acmegating.com | (as in, like, a lot of changes to process and query) | 18:22 |
| @fungicide:matrix.org | cool, proceeding with reindexing in that case, unless i should wait longer | 18:22 |
| @clarkb:matrix.org | I think it is probably fine. The gerrit side of things seem to be working (replicating pushes voting) | 18:23 |
| @fungicide:matrix.org | "Reindexer started" | 18:23 |
| @clarkb:matrix.org | if zuul is struggling then it is likely a side effect of pausing then unpausing which could be normal (things didn't process as they arrived so there is a backlog) or there is a bug and we'd have to fix that in zuul anyway | 18:23 |
| @clarkb:matrix.org | in this case I think it falls under normal behavior when we stop letting it chew on things as they arrive as everything looks happy now | 18:24 |
| @jim:acmegating.com | i think we would expect recovery to be slow due to lack of caches in gerrit | 18:24 |
| @clarkb:matrix.org | corvus: oh yup every api request is going to be a little bit slower too | 18:25 |
| @fungicide:matrix.org | does this look right for 10 years? `sudo docker run --rm --volume=/etc/zuul:/etc/zuul -it quay.io/zuul-ci/zuul-scheduler:latest zuul-admin create-auth-token --auth-config local --user localauth --expires-in=315569260` | 18:25 |
| @clarkb:matrix.org | and jgit cachesi n memory that would be wiped out too | 18:25 |
| @clarkb:matrix.org | fungi: your seconds number seems slightly larger than my `60 * 60 * 24 * 356 * 10` value which is probably correct ot handle leap years so ya | 18:26 |
| @jim:acmegating.com | haha i'm over here trying to reverse engineer that with like ... "did he use 365.2425? nope" | 18:27 |
| @fungicide:matrix.org | `units 10years seconds` | 18:27 |
| @fungicide:matrix.org | i rely on the units utility a lot | 18:27 |
| @fungicide:matrix.org | because: lazy | 18:28 |
| @jim:acmegating.com | i'm going to double check something, 1 sec | 18:28 |
| @fungicide:matrix.org | np | 18:28 |
| @jim:acmegating.com | lgtm. wanted to double check the field really is in seconds (because javascript frequently isn't, but this really is) | 18:30 |
| @fungicide:matrix.org | cool, thanks | 18:30 |
| @fungicide:matrix.org | updated the zuul-scheduler group vars with a token that should expire in a decade now | 18:34 |
| @clarkb:matrix.org | thanks the next hourly run should apply it | 18:34 |
| @clarkb:matrix.org | for some reason autohold-list works on zuul02 maybe that doesn't require auth so its fine | 18:35 |
| @fungicide:matrix.org | autoholds are also listed in the webui without authentication so that seems expected | 18:36 |
| @clarkb:matrix.org | yup that must be it then. I was just hoping for an easy test method. Maybe we pause then unpause again or just trust that it will work | 18:36 |
| @clarkb:matrix.org | oh actually we can create an autohold and then delete it that is non impactful | 18:36 |
| @fungicide:matrix.org | yeah | 18:37 |
| @clarkb:matrix.org | reindexing is ~22% done. I also detached from the screen session | 18:37 |
| @fungicide:matrix.org | i'll leave it up reporting the reindex progress until that finishes | 18:38 |
| @clarkb:matrix.org | that was the last chagne in the Gerrit 3.11 upgrade stack. Now we can start looking more closely at the 3.12 upgrade | 18:39 |
| @clarkb:matrix.org | which should get us post quantum algorithm support from MINA for the folks asking about that | 18:40 |
| @jim:acmegating.com | ``` | 18:40 |
| year tropicalyear | ||
| tropicalyear 365.242198781 day | ||
| The time needed for the mean Sun as defined above to increase its | ||
| longitude by 360 degrees. Most references defined the tropical year | ||
| as the interval between vernal equinoxes, but this is misleading. The | ||
| length of the season changes over time because of the eccentricity of | ||
| Earth's orbit. The time between vernal equinoxes is approximately | ||
| 365.24237 days around the year 2000. See "Mathematical Astronomy | ||
| Morsels" for more details. | ||
| ``` | ||
| @jim:acmegating.com | (units source code) | 18:41 |
| @clarkb:matrix.org | `increase its longitude by 360 degrees` this is relative to the sun? | 18:42 |
| @clarkb:matrix.org | it must be | 18:42 |
| @clarkb:matrix.org | oh yes that is what `mean Sun` refers to | 18:43 |
| @jim:acmegating.com | yeah there was a whole other paragraph about the mean sun | 18:43 |
| @jim:acmegating.com | ``` | 18:43 |
| Astronomical time measurement is a complicated matter. The length of the | ||
| true day at a given place can be 21 seconds less than 24 hours or 30 seconds | ||
| over 24 hours. The two main reasons for this are the varying speed of | ||
| Earth in its elliptical orbit and the fact that the Sun moves on the ecliptic | ||
| instead of along the celestial equator. To devise a workable system for time | ||
| measurement, Simon Newcomb (1835-1909) used a fictitious "mean Sun". | ||
| Consider a first fictitious Sun traveling along the ecliptic at a constant | ||
| speed and coinciding with the true Sun at perigee and apogee. Then | ||
| considering a second fictitious Sun traveling along the celestial equator at | ||
| a constant speed and coinciding with the first fictitious Sun at the | ||
| equinoxes. The second fictitious Sun is the "mean Sun". From this equations | ||
| can be written out to determine the length of the mean day, and the tropical | ||
| year. The length of the second was determined based on the tropical year | ||
| from such a calculation and was officially used from 1960-1967 until atomic | ||
| clocks replaced astronomical measurements for a standard of time. All of the | ||
| values below give the mean time for the specified interval. | ||
| See "Mathematical Astronomy Morsels" by Jean Meeus for more details | ||
| and a description of how to compute the correction to mean time. | ||
| ``` | ||
| @clarkb:matrix.org | orbital mechanics are super complicated, something which playing kerbal space program illustrates really well | 18:43 |
| @fungicide:matrix.org | yeah, my hat's off to the units community for being total measurement geeks | 18:54 |
| @fungicide:matrix.org | i rely on it constantly when cooking/baking too, or doing carpentry/cabinetry, not just for computing tasks | 18:55 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 974555: Send statusbot notices to #starlingx-general:opendev.org https://review.opendev.org/c/opendev/system-config/+/974555 | 19:04 | |
| @clarkb:matrix.org | fungi: you should double check I got the room name correct in that change | 19:05 |
| @fungicide:matrix.org | yep, looks right | 19:08 |
| @fungicide:matrix.org | "Reindex changes to version 86 complete" | 19:18 |
| @fungicide:matrix.org | closing out the root screen session on review03 | 19:19 |
| @clarkb:matrix.org | I was just going to report that. Three errors as expected | 19:19 |
| @fungicide:matrix.org | "Failed 3/967957 changes" yes | 19:19 |
| @clarkb:matrix.org | I was able to create a zuul autohold using `zuul-client` on zuul02 then delete it. I think the new token is working | 19:31 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 974555: Send statusbot notices to #starlingx-general:opendev.org https://review.opendev.org/c/opendev/system-config/+/974555 | 20:03 | |
| @fungicide:matrix.org | status has joined that room now | 20:39 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 973689: Validate crc32 checksum for S3 image upload https://review.opendev.org/c/zuul/zuul-jobs/+/973689 | 22:25 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!