| @mnasiadka:matrix.org | fungi: Actually it seems reboot has helped | 06:00 |
|---|---|---|
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zuul-providers] 996627: Revert "Disable openmetal due to errors" https://review.opendev.org/c/opendev/zuul-providers/+/996627 | 11:05 | |
| @fungicide:matrix.org | oh, excellent. thanks mnasiadka! | 13:08 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zuul-providers] 996627: Revert "Disable openmetal due to errors" https://review.opendev.org/c/opendev/zuul-providers/+/996627 | 13:10 | |
| @fungicide:matrix.org | i went ahead and approved it, with a flex region still offline and now an ovh region as well we need all the other capacity we can get | 13:10 |
| @harbott.osism.tech:regio.chat | gra1 should be back online since yesterday evening already? | 14:13 |
| @fungicide:matrix.org | oh right | 14:22 |
| @fungicide:matrix.org | so we're really only down flex sjc3 at the moment, i guess | 14:22 |
| @mnasiadka:matrix.org | I’m puzzled - why https://review.opendev.org/c/openstack/kolla-ansible/+/996096 needs a swift change? | 16:00 |
| @fungicide:matrix.org | mnasiadka: it's the depends-on in the commit message | 16:01 |
| @fungicide:matrix.org | i think somebody mistyped the change number in that url | 16:02 |
| @fungicide:matrix.org | if you try it, the url takes you to an error page in gerrit, zuul is just going by the change number instead which belongs to the openstack/swift repository | 16:02 |
| @fungicide:matrix.org | i'm unsure why zuul is falling back on just looking up the change number instead of the full change url and erroring (change numbers in modern gerrit aren't guaranteed to be unique any more), but that's probably a corner case | 16:05 |
| @fungicide:matrix.org | if you looked in the scheduler's debug log it might explain the choice to go with the swift change in the case of that nonexistent neutron change url | 16:06 |
| @jim:acmegating.com | they're only unique if you import overlapping change ids from other installations; they are unique under normal circumstances. we document the limitation in zuul. | 16:07 |
| @fungicide:matrix.org | i thought that might be the case but didn't see it mentioned in zuul's gerrit driver docs | 16:08 |
| @fungicide:matrix.org | oh! there it is | 16:08 |
| @jim:acmegating.com | https://zuul-ci.org/docs/zuul/latest/drivers/gerrit.html second note box | 16:08 |
| @fungicide:matrix.org | second note in the very first section | 16:08 |
| @fungicide:matrix.org | "Zuul’s Gerrit driver is not currently expected to work with non unique change numbers." | 16:08 |
| @fungicide:matrix.org | yep | 16:08 |
| @fungicide:matrix.org | so that explains the behavior | 16:09 |
| @fungicide:matrix.org | it's really just looking at the hostname and the number at the end of the url, the rest is superfluous | 16:09 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 996680: Push-to-intermediate-registry: don't write empty docker.json https://review.opendev.org/c/zuul/zuul-jobs/+/996680 | 16:42 | |
| @mnasiadka:matrix.org | oh boy, thanks corvus and fungi - if the Zuul message would be that depends-on change does not exist it would be easier, doh :) | 17:02 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [zuul/zuul-jobs] 996684: fetch-subunit-output: Allow configuring artifact name https://review.opendev.org/c/zuul/zuul-jobs/+/996684 | 17:05 | |
| @fungicide:matrix.org | i've added 8 servers to our ansible "emergency" disable list per step #3 in https://etherpad.opendev.org/p/opendev-project-renames-20260709 | 19:55 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will be offline briefly one hour from now, at 21:00 UTC, while we rename a project. | 19:59 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 19:59 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline briefly one hour from now, at 21:00 UTC, while we rename a project. | 20:03 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 20:03 |
| @fungicide:matrix.org | for anyone following along, i've done steps #1-4 in the pad at this point (technically decided to skip step #2), i've already got the root screen session up and recording on bridge, the rename config copied into place and the zuul pause command queued | 20:18 |
| @fungicide:matrix.org | in roughly 40 minutes i'll send another status notice and pause zuul, then once that settles i'll execute the `rename_repos` playbook with the reviewed config | 20:19 |
| @fungicide:matrix.org | that's the config from https://review.opendev.org/c/opendev/project-config/+/995706 to be precise | 20:20 |
| @arnaudm:matrix.org | harbott.osism.tech fungicide we are having hard time with ipv6 at the moment on GRA region, that's the root cause of your issues yesterday. I was pretty busy the full week working on different topics related to production issue so wasn't really available to help, sorry for this | 20:20 |
| @fungicide:matrix.org | arnaudm: no problem, thank you for confirming! | 20:26 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org is going offline momentarily at 21:00 UTC while we rename a project, but will return within a few minutes. | 20:57 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 20:57 |
| @fungicide:matrix.org | that takes a few minutes to hit all the channels/rooms | 20:57 |
| @fungicide:matrix.org | i've paused zuul and https://zuul.opendev.org/t/opendev/status shows the expected banner | 20:58 |
| @fungicide:matrix.org | it's time, so i'm running the playbook as described at the end of step #6 in https://etherpad.opendev.org/p/opendev-project-renames-20260709 | 21:00 |
| @fungicide:matrix.org | gerrit is on its way down | 21:00 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org is going offline momentarily at 21:00 UTC while we rename a project, but will return within a few minutes. | 21:00 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 21:00 |
| @fungicide:matrix.org | the rename steps processed successfully and gerrit is starting again. seems like the gerrit index backup may now be our slowest step in that playbook now, it was easily 2/3 or maybe even 3/4 of the total outage | 21:06 |
| @fungicide:matrix.org | the webui is up for me | 21:07 |
| @fungicide:matrix.org | and the playbook completed with only ok/changed/skipped tasks, no unreachable/failed | 21:08 |
| @fungicide:matrix.org | i'm able to load diffs on changes, will go ahead and unpause zuul | 21:08 |
| @fungicide:matrix.org | the pause banner has cleared from zuul's webui | 21:09 |
| @fungicide:matrix.org | https://opendev.org/x/cursive redirects as expected and https://review.opendev.org/admin/repos/openstack/cursive returns content too | 21:11 |
| @fungicide:matrix.org | we're at step #9 now, i've removed servers from the disable list for ansible again | 21:13 |
| @fungicide:matrix.org | and 990122 has been approved, waiting for it to merge | 21:13 |
| -@gerrit:opendev.org- Zuul merged on behalf of Stephen Finucane: [openstack/project-config] 990122: Rename x/cursive to openstack/cursive https://review.opendev.org/c/openstack/project-config/+/990122 | 21:23 | |
| @fungicide:matrix.org | infra-prod-manage-projects is running in deploy now | 21:25 |
| @fungicide:matrix.org | and it succeeded | 21:31 |
| @fungicide:matrix.org | and with that, the maintenance is complete. i'm shutting down the screen session but the transcription from it will remain in bridge:/root/tmp/rename-20260709 | 21:33 |
| @fungicide:matrix.org | deploy for 990122 succeeded, so approving 995706 to record the rename, and 996370 to readd openstack standard jobs to the renamed repo | 21:37 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/project-config] 995706: Add records for x/cursive -> openstack/cursive rename https://review.opendev.org/c/opendev/project-config/+/995706 | 21:38 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Goutham Pacha Ravi: [openstack/project-config] 996370: Add Zuul jobs for openstack/cursive https://review.opendev.org/c/openstack/project-config/+/996370 | 21:57 | |
| @fungicide:matrix.org | yay, finished on our side. and with that, i'm disappearing for the evening. night all! | 21:57 |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 996725: Add catatonit to podman package list https://review.opendev.org/c/zuul/zuul-jobs/+/996725 | 23:28 | |
| @mordred:waterwanders.com | corvus: this is not an urgent issue - but I thought I'd flag it for your brain. I've seen some cases where the vouched pipeline has run twice on a change for reasons I can't figure out: https://review.opendev.org/c/drizzle/drizzle/+/996565 PS 3 is a good example. I've seen it before - I *Think* - but it's never been important enough to chase. It's still not, honestly. Unless it is. | 23:50 |
| @mordred:waterwanders.com | corvus: hrm. we seem to be running tests for that change again. I wonder if somehow every comment is getting treated as a recheck? | 23:53 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!