| @mnasiadka:matrix.org | So, after the switch to use tox for the job that builds release notes (or maybe it was there forever, I have no clue) - we end up that if a patch is merged on a stable branch - release notes for kolla/kolla-ansible end up not having the newer renos than the stable branch the build/promote was running on | 13:22 |
|---|---|---|
| @mnasiadka:matrix.org | Anybody has an idea what the heck is happening? | 13:22 |
| @fungicide:matrix.org | Clark: yes, i think so | 13:33 |
| @fungicide:matrix.org | mnasiadka: maybe a question for stephenfin as it was his change and it seemed like he had done a lot of local testing with it in different scenarios | 13:34 |
| @fungicide:matrix.org | but if he's also unsure i can try to dig into it once i catch up on a few other things, or we can revert | 13:35 |
| @fungicide:matrix.org | note that was an openstack job change though, not opendev-wide | 13:35 |
| @mnasiadka:matrix.org | Right, but before it used zuul-wide role for building releasenotes - so I thought maybe there are people with better understanding here | 13:36 |
| @mnasiadka:matrix.org | I can ask on #openstack-infra if that's better | 13:36 |
| @mnasiadka:matrix.org | Ah, got it! | 13:45 |
| @mnasiadka:matrix.org | build-reno-releasenotes had an override-checkout to run on master - https://opendev.org/zuul/zuul-jobs/src/commit/79fe3eb1d01f8ac5739b0b7bc4759c407b6e248d/zuul.d/python-jobs.yaml#L636 | 13:45 |
| @mnasiadka:matrix.org | And now we run openstack-tox-docs - https://opendev.org/openstack/openstack-zuul-jobs/commit/b07bf5b4b0d1df18672c9a5c09709d910b2e841e | 13:46 |
| @fungicide:matrix.org | mnasiadka: yeah, i commented to that effect in irc as well | 13:50 |
| -@gerrit:opendev.org- Mauricio Harley proposed: [openstack/project-config] 988770: Add #openstack-pqc IRC channel https://review.opendev.org/c/openstack/project-config/+/988770 | 13:58 | |
| -@gerrit:opendev.org- Mauricio Harley proposed: [opendev/system-config] 988775: Add #openstack-pqc to channel logging and statusbot https://review.opendev.org/c/opendev/system-config/+/988775 | 14:17 | |
| @clarkb:matrix.org | infra-root any thoughts on https://review.opendev.org/c/opendev/system-config/+/987375 to bump gerrit to 3.12.7? | 15:40 |
| @clarkb:matrix.org | if today isn't good we can target Monday | 15:40 |
| @fungicide:matrix.org | i think we can. i've spent my morning battling some wifi meshing instability which has made getting any real work done challenging, but i think i have it ironed out now (downgraded firmware on all my access points) | 15:48 |
| @clarkb:matrix.org | fun. I just upgraded the firmware on my ap and it actually seems more stable thankfully | 15:49 |
| @clarkb:matrix.org | fungi: I guess if you get into a settled state and feel like a gerrit restart is doable I'll let you approve it? I should be around all day myself | 15:55 |
| @fungicide:matrix.org | yeah, i have an errand to run at roughly 17:15-17:45 but otherwise am here all day | 15:57 |
| @clarkb:matrix.org | cool we can aim to upgrade gerrit after you get back then | 15:57 |
| @clarkb:matrix.org | fungi: do we want to review (and possibly approve) https://review.opendev.org/c/opendev/system-config/+/987375 before you head out so that things are hopefully ready by the time you return? | 16:21 |
| @fungicide:matrix.org | yeah, looking | 16:24 |
| @fungicide:matrix.org | lgtm, in it goes | 16:40 |
| @fungicide:matrix.org | okay, heading out briefly, shouldn't be 30 minutes tops | 17:22 |
| @clarkb:matrix.org | cool the change should land soon. I'll check the config update goes in cleanly (should be a noop for 3.12 anyway) | 17:23 |
| @clarkb:matrix.org | then we should be ready when you get back | 17:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 987375: Update Gerrit images to 3.12.7 and 3.13.6 https://review.opendev.org/c/opendev/system-config/+/987375 | 17:24 | |
| @clarkb:matrix.org | looks like the config updated as expected | 17:44 |
| @fungicide:matrix.org | okay, back | 17:57 |
| @fungicide:matrix.org | i've got a root screen session open on review03 | 17:57 |
| @clarkb:matrix.org | joining now | 17:58 |
| @clarkb:matrix.org | fungi: I'm attached | 17:58 |
| @fungicide:matrix.org | quay.io/opendevorg/gerrit 3.12 5648aec9f664 4 weeks ago 718 MB | 17:58 |
| @clarkb:matrix.org | fungi: need to change dir | 17:59 |
| @clarkb:matrix.org | or use -f ya | 17:59 |
| @fungicide:matrix.org | `quay.io/opendevorg/gerrit 3.12 f4812a7aaf95 About an hour ago 720 MB` | 18:00 |
| @fungicide:matrix.org | `quay.io/opendevorg/gerrit@sha256:26db67bbab3c5a707fd6cd688e4ffe7906f9b7edd068ef5ad6ba6f623d0a5436` | 18:00 |
| @fungicide:matrix.org | that's the one we want, right? | 18:00 |
| @clarkb:matrix.org | https://quay.io/repository/opendevorg/gerrit/manifest/sha256:e49bf2365015576c40c4351a2abb7798fe3b05775bcc8dc0e448cdca9ae7d9ed | 18:00 |
| @clarkb:matrix.org | yes that appears to matc | 18:01 |
| @fungicide:matrix.org | oh right, the second one | 18:01 |
| @fungicide:matrix.org | `quay.io/opendevorg/gerrit@sha256:e49bf2365015576c40c4351a2abb7798fe3b05775bcc8dc0e448cdca9ae7d9ed` | 18:01 |
| @clarkb:matrix.org | and the first is present if you open that quay url as the amd64 image | 18:01 |
| @clarkb:matrix.org | so ya lgtm | 18:01 |
| @fungicide:matrix.org | status notice The Gerrit service on review.opendev.org will be offline momentarily while we restart it onto a new patch release | 18:01 |
| @fungicide:matrix.org | that look good? | 18:02 |
| @clarkb:matrix.org | yup | 18:02 |
| @fungicide:matrix.org | #status notice The Gerrit service on review.opendev.org will be offline momentarily while we restart it onto a new patch release | 18:02 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 18:02 |
| @clarkb:matrix.org | these are the cache db's that are large: comment_context-v2.mv.db, gerrit_file_diff-v2.mv.db, git_file_diff-v2.mv.db, git_modified_files-v2.mv.db, modified_files-v2.mv.db | 18:03 |
| @clarkb:matrix.org | I don't think it hurts to clear out others like diff summary | 18:03 |
| @clarkb:matrix.org | I just want to amke sure we get all of the big ones | 18:03 |
| -@status:opendev.org- NOTICE: The Gerrit service on review.opendev.org will be offline momentarily while we restart it onto a new patch release | 18:05 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 18:05 |
| @fungicide:matrix.org | `docker-compose -f /etc/gerrit-compose/docker-compose.yaml down && mv ~gerrit2/review_site/data/replication/ref-updates/waiting ~gerrit2/tmp/waiting_queue_2026-05-15 && rm ~gerrit2/review_site/cache/{gerrit_file_diff,git_file_diff,git_modified_files,modified_files,comment_context}-v2.* && docker-compose -f /etc/gerrit-compose/docker-compose.yaml up -d` | 18:05 |
| @fungicide:matrix.org | that look right? | 18:05 |
| @clarkb:matrix.org | yes I think so | 18:06 |
| @fungicide:matrix.org | notably, the db files all got a -v2 when we upgraded last time | 18:06 |
| @fungicide:matrix.org | okay, running now | 18:06 |
| @clarkb:matrix.org | yup because the old file paths couldn't be reused due to h2 v2 vs h2 v1 incompatibilities | 18:06 |
| @fungicide:matrix.org | stopping the container took only 72.5s this time, and gerrit should be on its way back up now | 18:08 |
| @fungicide:matrix.org | webui loads for me | 18:08 |
| @clarkb:matrix.org | `[2026-05-15T18:08:43.827Z] [main] INFO com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.12.7-dirty ready` has been logged | 18:09 |
| @fungicide:matrix.org | `Powered by Gerrit Code Review (3.12.7-dirty)` | 18:09 |
| @fungicide:matrix.org | time to reindex? | 18:09 |
| @clarkb:matrix.org | lets check the queues first | 18:09 |
| @clarkb:matrix.org | ya show-queue -w against the ssh service looks largely empty | 18:09 |
| @clarkb:matrix.org | do diffs load? I think that is a good indicator that caches are working again too | 18:10 |
| @clarkb:matrix.org | yes diffs load for me | 18:10 |
| @clarkb:matrix.org | https://review.opendev.org/c/airship/armada-go/+/988793 was pushed after the restart | 18:11 |
| @clarkb:matrix.org | https://opendev.org/airship/armada-go/commit/6f17d4b7fe6fd7cec6d8e234c892b75adf67592e appears to have replicated. Its a bit of an anticlimactic commit to replicate but it did replicate | 18:11 |
| @clarkb:matrix.org | fungi: so ya I think we can trigger reindexing | 18:12 |
| @fungicide:matrix.org | okay, starting in a sec | 18:12 |
| @fungicide:matrix.org | did `gerrit index start changes --force` just now | 18:14 |
| @clarkb:matrix.org | it is about 10% done | 18:18 |
| @clarkb:matrix.org | reidnexing is now complete. The expected 3 changes failed to index | 18:49 |
| @clarkb:matrix.org | and show-queue output looks about right. I think this is done now? | 18:49 |
| @clarkb:matrix.org | fungi: I detached from the screen if you want to shut it down | 18:50 |
| @clarkb:matrix.org | assuming you're happy with the end results here too | 18:50 |
| @fungicide:matrix.org | yep, closing it out now | 18:53 |
| @clarkb:matrix.org | that was weird my local machine crashed or maybe just USB crashed? | 20:14 |
| @clarkb:matrix.org | no panic in journalctl output. Not sure what happened | 20:14 |
| @clarkb:matrix.org | I took that as an opportunity to do package updates since I ended up rebooting anyway and now my networking is not working | 20:40 |
| @fungicide:matrix.org | oh fun | 20:40 |
| @fungicide:matrix.org | and yeah, since i got my home network stable again finally, i too am catching up on package upgrades for my machines now | 20:41 |
| @fungicide:matrix.org | particularly since kernel upgrades are going to need reboots, even if they were automatically applied previously | 20:42 |
| @fungicide:matrix.org | up-side is i'm on linux 7 now | 20:42 |
| @fungicide:matrix.org | (at least i hope that's going to be an up side) | 20:42 |
| @clarkb:matrix.org | I'm on 7 too and it looks like maybe dbus problems. Wpa supplicant can't talk to /run/dbus/system_bus_socket due to some permissions issue | 20:43 |
| @fungicide:matrix.org | ick | 20:44 |
| @fungicide:matrix.org | did dropping back to your previous kernel help? | 20:45 |
| @clarkb:matrix.org | But network manager is running so I don't know if this is expected or not | 20:45 |
| @clarkb:matrix.org | I haven't tried that yet. Was trying to diagnose the issue on the running system | 20:45 |
| @clarkb:matrix.org | I guess I should try that next. I'm half worried it's a hardware problem which could explain the previous crash maybe | 20:45 |
| @fungicide:matrix.org | oh that's no good | 20:49 |
| @fungicide:matrix.org | i was selfishly asking the kernel question in case i run into some similar issue | 20:50 |
| @clarkb:matrix.org | Booting on the old kernel doesn't change things. The issue is wpa supplicant I think | 20:53 |
| @clarkb:matrix.org | Network manager uses it as the backend for managing things for wifi | 20:53 |
| @fungicide:matrix.org | ah, at least my main workstation has wired networking, but yeah my mobile devices are using a fragile mix of networkmanager and systemd-networkd | 20:55 |
| @clarkb:matrix.org | Ya network manager says it can't initialize wpa supplicant and wpa supplicant says it has permission issues so I'm tracking that down next | 20:56 |
| @clarkb:matrix.org | Polkit-default-privs just updated which seems a likely culprit | 20:57 |
| @fungicide:matrix.org | agreed, given the symptoms | 20:58 |
| @fungicide:matrix.org | polkitd upgraded for me, but was basically just a cve backport not a new upstream release | 21:00 |
| @clarkb:matrix.org | this is getting awkward, I'm using my phone connected over usbc to bridge my wireless connection because that works fine as no wpa supplicant is needed | 21:25 |
| @clarkb:matrix.org | I've reinstalled all the dbus and wpa_supplicant and networkmanager packages just to make sure something config wise didn't go sideways. I also compared to another host and things seem to line up. I'm not seeing why permission is denied in the first place but I'm guessing I need to figure that out | 21:25 |
| @clarkb:matrix.org | ok apparmor is doing it and apparmor-profiles updated | 21:38 |
| @fungicide:matrix.org | fwiw, i reverted back to an old kernel because my bluetooth controller wasn't discovered at all | 21:42 |
| @fungicide:matrix.org | made it hard to type on a bt keyboard | 21:42 |
| @clarkb:matrix.org | https://build.opensuse.org/package/revisions/openSUSE:Factory/apparmor looks like they did a major apparmor update | 21:42 |
| @clarkb:matrix.org | `aa-complain wpa_supplicant` transformed my audit log denials into alloweds but now wpa_supplicant itslef logs that it cannot open dbus | 21:50 |
| @clarkb:matrix.org | ok fixed it with the classic flags=(attach_disconnected) I can see in the audit log that apparmor is preventing wpa_supplicant from logging and from reading /etc/libnl/classid but I have network again | 22:18 |
| @clarkb:matrix.org | now to go file a bug I guess | 22:18 |
| @clarkb:matrix.org | and then cry | 22:18 |
| @clarkb:matrix.org | looks like the previous version of apparmor didn't have a wpa_supplicant profile so that explains why this started with 5.0 | 22:52 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!