| -@gerrit:opendev.org- Zuul merged on behalf of Benedikt Löffler: [zuul/nodepool] 925760: Return size of EBS snapshot in GiB https://review.opendev.org/c/zuul/nodepool/+/925760 | 08:10 | |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 926013: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 09:38 | |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 926013: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 10:13 | |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 926013: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 10:52 | |
| @sdodsley:matrix.org | Hi - looking for some assistance. I have deployed zuul with nodepool using softwarefactory. Nodepool is creating the nodes correctly, but they are not showing in the nodes tab of zuul. Any idea why that might be? I am also seeing some strange zull config errors, such as `Will not fetch project branches as read-only is set` and `Decryption failed`. Not sure what is causing those, so advise would be appreciated. | 13:06 |
|---|---|---|
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 926013: [wip] Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 13:27 | |
| @jangutter:matrix.org | Did you set nodepool to pre-boot a few nodes? | 13:50 |
| If `nodepool list` shows nodes, and the zuul web ui doesn't show them, then I _think_ zuul cannot speak to nodepool via zookeeper. | ||
| _however_ softwarefactory is a distinct project (you should ask them) and it should be managing the connection between Zuul and Nodepool. It's possible that the source of the problem is with your sf config. | ||
| @jangutter:matrix.org | Also, it's entirely possible that there are no nodes because there are no node-requests, so sending up a job to Zuul might trigger more info. | 13:51 |
| @jangutter:matrix.org | Hi folks, I've experimented a bit on various combinations of the following: | 14:12 |
| * Ubuntu | ||
| * minikube | ||
| * cri-o | ||
| * podman (and/or the absence of docker) | ||
| The current jobs for minikube+cri-o seem to be in various states of flakiness/brokenness. This is likely because kubic (the previous home of cri-o) has been sunsetted, and some k8s registries have undergone migration: | ||
| https://kubernetes.io/blog/2023/10/10/cri-o-community-package-infrastructure/ | ||
| https://kubernetes.io/blog/2023/03/10/image-registry-redirect/ | ||
| Additionally, in Noble, podman is now part of the distro, and cri-o uninstalls docker-ce... | ||
| So, I _think_ I worked around the big problems, it's not quite clean yet. I'm going to try to add a podman based minikube option next. This is all yak-shaving to try to get the zuul-operator CI up and running (with publishing to quay.io) again. | ||
| I'll send up another review, with the commit message updated. (but no fundamental code changes). I'm not familiar with how load-bearing these jobs and roles are, and I'd appreciate it if folks could point me to things that I can try to test them with. | ||
| @jim:acmegating.com | jangutter: we tried to make the tests cover all the known functional use cases completely; that's certainly true for the registry roles and the non-k8s docker/podman roles; probably less so for the k8s roles. and we probably need docker and quay variants, which we may not have. i think the main users of the k8s roles are probably the nodepool functional tests (which only use minikube and not the registry) and the zuul-operator. | 14:18 |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 926139: Update doc regarding merge mode https://review.opendev.org/c/zuul/zuul/+/926139 | 14:20 | |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 926013: Fix k8s-crio buildset registry test https://review.opendev.org/c/zuul/zuul-jobs/+/926013 | 14:50 | |
| @sdodsley:matrix.org | Yes - nodepool is pre-booting nodes. I'll ask on the SF channel | 15:12 |
| @jim:acmegating.com | Simon Dodsley: nodes may not show up in the nodes tab of zuul unless they're associated with that tenant (which only happens once they're in use); so ready nodes generally shouldn't show up there. | 15:43 |
| @jangutter:matrix.org | @corvus looks like the nodepool jobs use microk8s, not minikube - so hopefully those codepaths are not touched by the cri-o fix. | 16:11 |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/nodepool] 926148: [DNM] check ensure-kubernetes jobs for regressions https://review.opendev.org/c/zuul/nodepool/+/926148 | 16:13 | |
| @clarkb:matrix.org | jangutter: the issues with minikube are why things transitioned to microk8s I think | 16:14 |
| @clarkb:matrix.org | we then had to pin microk8s due to some update that seemed not well tested, but maybe that can be unpinned now | 16:14 |
| @jangutter:matrix.org | Clark: The way minikube is run seems to be in a very unsupported way (likely because back then there wasn't an easy way to run it without VM mode). However, it looks like the podman driver seems to have progressed to a point where it might be usable in all distros. | 16:17 |
| @clarkb:matrix.org | cool. I don't think the microk8s stuff has a direct impact on making minikube work I just wanted to call out why thigns are that way in case it make sense to just use microk8s instead | 16:18 |
| @jangutter:matrix.org | (our setup is CentOS based, so there's also a bit of an incentive on my side to have a solution that works for both deb and rpm based distros) | 16:20 |
| Yeah, sadly the small-scale testing k8s solutions are fragmented. Minikube looked nice because it supports storage and ingress - two things that a nontrivial k8s app like the zuul-operator needs. | ||
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 924970: [wip] Update ensure-kubernetes with podman support https://review.opendev.org/c/zuul/zuul-jobs/+/924970 | 16:37 | |
| -@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul] 925434: Combine reserved variable name lists in check_varnames https://review.opendev.org/c/zuul/zuul/+/925434 | 16:39 | |
| @mordred:waterwanders.com | corvus: I went ahead and updated the grammar like you suggested - because why not ^^ | 16:40 |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 924970: [wip] Update ensure-kubernetes with podman support https://review.opendev.org/c/zuul/zuul-jobs/+/924970 | 16:41 | |
| -@gerrit:opendev.org- Jan Gutter proposed: [zuul/zuul-jobs] 924970: [wip] Update ensure-kubernetes with podman support https://review.opendev.org/c/zuul/zuul-jobs/+/924970 | 17:36 | |
| -@gerrit:opendev.org- Jan Gutter proposed: | 18:06 | |
| - [zuul/zuul-operator] 924748: Minor fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 925916: ensure-podman: add tasks to configure socket group https://review.opendev.org/c/zuul/zuul-jobs/+/925916 | 18:19 | |
| -@gerrit:opendev.org- Jan Gutter proposed: | 18:26 | |
| - [zuul/zuul-operator] 924748: CI Fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- Jan Gutter proposed: | 18:39 | |
| - [zuul/zuul-operator] 924748: CI Fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- Jan Gutter proposed: | 18:41 | |
| - [zuul/zuul-operator] 924748: CI Fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 925916: ensure-podman: add tasks to configure socket group https://review.opendev.org/c/zuul/zuul-jobs/+/925916 | 18:44 | |
| -@gerrit:opendev.org- Jan Gutter proposed: | 18:53 | |
| - [zuul/zuul-operator] 924748: CI Fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 925353: Get files via merger for >=2048 pushed commits https://review.opendev.org/c/zuul/zuul/+/925353 | 19:00 | |
| -@gerrit:opendev.org- Jan Gutter proposed: | 19:13 | |
| - [zuul/zuul-operator] 924748: CI Fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- Jan Gutter proposed: | 19:14 | |
| - [zuul/zuul-operator] 924748: CI Fixes and dependency updates https://review.opendev.org/c/zuul/zuul-operator/+/924748 | ||
| - [zuul/zuul-operator] 924872: Change references from docker.io to quay.io https://review.opendev.org/c/zuul/zuul-operator/+/924872 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 925916: ensure-podman: add tasks to configure socket group https://review.opendev.org/c/zuul/zuul-jobs/+/925916 | 19:43 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 20:13 | |
| - [zuul/zuul-jobs] 925916: ensure-podman: add tasks to configure socket group https://review.opendev.org/c/zuul/zuul-jobs/+/925916 | ||
| - [zuul/zuul-jobs] 926164: Add ability to exclude a specific platform https://review.opendev.org/c/zuul/zuul-jobs/+/926164 | ||
| @sdodsley:matrix.org | Hi team - is there a way to debug the zuul pipeline? I see jobs going into the pipeline, but then they just vanish. The config files all look correct, but there must be something wrong, like a typo I can't see, so any debugging messages would be useful. | 21:51 |
| @clarkb:matrix.org | > <@sdodsley:matrix.org> Hi team - is there a way to debug the zuul pipeline? I see jobs going into the pipeline, but then they just vanish. The config files all look correct, but there must be something wrong, like a typo I can't see, so any debugging messages would be useful. | 21:56 |
| The way I usually approach this is to find the event id in the zuul debug log that matches with the thing I'm trying to debug (change is approved, new patchset, etc). Then you can grep for that id value in the debug log and get a fairly complete play by play on what choices were made by zuul | ||
| @sdodsley:matrix.org | Clark: thanks for that - which log specifically? | 21:58 |
| @clarkb:matrix.org | the zuul scheduler debug log is the main one, but depending on where things go wrong you may need to look at the executors and merger logs too | 21:59 |
| @sdodsley:matrix.org | thanks | 21:59 |
| @sdodsley:matrix.org | / | 21:59 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 926167: Handle zuul_return format errors https://review.opendev.org/c/zuul/zuul/+/926167 | 22:26 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!