| @paladox:matrix.org | ianw: lmao | 13:14 |
|---|---|---|
| @paladox:matrix.org | corvus: I'm not sure if you use the zuul plugin, but i've migrated that as well https://gerrit-review.googlesource.com/c/plugins/zuul/+/485641 (i saw the WMF used it) | 13:14 |
| @paladox:matrix.org | seems zuul for gerrit-review is down? https://ci.gerritcodereview.com/t/gerrit/buildset/8523b9b8065444c19010f1a3263977f0 | 13:15 |
| @paladox:matrix.org | I faked some data but the change works for me: https://imgur.com/a/lFyE75G | 13:19 |
| @jim:acmegating.com | Paladox: Warning FailedMount 46s (x7 over 102s) kubelet (combined from similar events): MountVolume.SetUp failed for volume "kube-api-access-6dnv4" : write /var/lib/kubelet/pods/cdd554a8-50a1-4e21-956c-b43fb815037a/volumes/kubernetes.io~projected/kube-api-access-6dnv4/..2025_06_20_13_52_46.1734204385/namespace: no space left on device | 13:54 |
| @jim:acmegating.com | looks like the k8s node is out of disk space... | 13:55 |
| @jim:acmegating.com | or... | 14:00 |
| @jim:acmegating.com | ``` | 14:00 |
| > kubectl get events | ||
| LAST SEEN TYPE REASON OBJECT MESSAGE | ||
| 2m3s Warning ReadOnlyLocalSSDDetected node/gke-zuul-control-plane-default-pool-ede28458-4cmq | ||
| 7m2s Warning ReadOnlyFileSystemDetected node/gke-zuul-control-plane-default-pool-ede28458-4cmq | ||
| 37s Warning ReadOnlyLocalSSDDetected node/gke-zuul-control-plane-default-pool-ede28458-7ygi | ||
| 4m8s Warning ReadOnlyLocalSSDDetected node/gke-zuul-control-plane-default-pool-ede28458-q55p | ||
| 4m1s Warning ReadOnlyLocalSSDDetected node/gke-zuul-control-plane-default-pool-ede28458-zjjr | ||
| ``` | ||
| @jim:acmegating.com | it's otherwise having some filesystem problems | 14:00 |
| @jim:acmegating.com | those started on june 14, during the most recent automatic upgrade | 14:02 |
| @jim:acmegating.com | Paladox: okay, i'm not entirely sure what's going on there, but i scaled the cluster down and up to get a fresh node, and it seems happier now. | 14:20 |
| @jim:acmegating.com | Paladox: we also may want to try merging this change: https://gerrit-review.googlesource.com/c/zuul/ops/+/485701 | 14:21 |
| @jim:acmegating.com | and then we should really see if we can find someone who can increase the size of those cluster nodes :/ | 14:22 |
| @paladox:matrix.org | Oh thanks! | 14:25 |
| @paladox:matrix.org | +2'd | 14:25 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 952856: Update upload before releasing lock https://review.opendev.org/c/zuul/zuul/+/952856 | 16:17 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 952857: Refresh launcher objects after locking https://review.opendev.org/c/zuul/zuul/+/952857 | 16:24 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 952858: Wake launcher when component registry changes https://review.opendev.org/c/zuul/zuul/+/952858 | 16:36 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 952799: Add NIZ autohold support https://review.opendev.org/c/zuul/zuul/+/952799 | 17:07 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 952943: Add node icons to nodeset requests page https://review.opendev.org/c/zuul/zuul/+/952943 | 17:17 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!