| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 02:14 | |
| - [zuul/zuul] 960875: Launcher: don't use the main node with slots https://review.opendev.org/c/zuul/zuul/+/960875 | ||
| - [zuul/zuul] 960921: Handle launch failures with subnodes https://review.opendev.org/c/zuul/zuul/+/960921 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 960924: Always associate nodes with providers https://review.opendev.org/c/zuul/zuul/+/960924 | 02:14 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 962561: Pin PostgreSQL in Docker test setup to v17 https://review.opendev.org/c/zuul/zuul/+/962561 | 04:59 | |
| @dacofr:matrix.org | Hello 👋 | 13:20 |
|---|---|---|
| I'm a new joiner and I have a question (maybe a dummy one) :) In my company, we are using Zuul and I'm wondering if it's possible for Zuul to load dynamically it's configuration from a repository ? I mean, rather than reading files in .zuul.d directory (or .zuul.yaml), calling an external tool and grabing configuration from this external tool. | ||
| Thanks for your help | ||
| @jangutter:matrix.org | Daco: I _think_ that would be an antipattern, specifically for Zuul. One of Zuul's advantages is that the commit triggering the pipeline is a major source of config. | 13:28 |
| External config to the project _is possible_ using trusted projects, but, for multiple reasons that's always in the post-review path. | ||
| @jangutter:matrix.org | What you're describing is more useful in the sense of a job runner. Zuul is a CI system foremost, and has been triggering things via gitops since long before it was hip. | 13:30 |
| @jangutter:matrix.org | Of course, nothing prevents a Zuul _job_ from fetching some things in the middle of its run, but that's not really Zuul config (won't affect things like dependencies, etc.) | 13:41 |
| @fungicide:matrix.org | yes, if the mechanism by which the configuration is generated is not managed in the repository such that it can be determined speculatively for changes proposed to that mechanism, it guts a substantial amount of zuul's usefulness. the fact that trusted config content isn't allowed to run speculatively is not so much a benefit as a limitation necessitated by security, and most zuul users i'm aware of do their best to avoid putting anything they don't have to in a trusted config project because it becomes too easy to break your own configuration when it can't be tested in advance | 13:41 |
| @jangutter:matrix.org | Our setups don't allow Zuul to do the "CD" part, so we usually use special purpose CD tooling to handle those kinds of rollouts. First time I encountered Zuul's opinions (and software gating) I definitely did not understand their usefulness. They are _extremely_ well thought out, battle-tested and encourage a "virtuous feedback cycle" that's very hard to deny. Part of the reason Zuul's so good is that it's intentionally very hard to commit many antipatterns. | 13:49 |
| @jangutter:matrix.org | (clarification: I work for a company that's not opendev ) | 13:50 |
| @jim:acmegating.com | [opendev isn't a company either, it's a volunteer effort among several projects to collaboratively run a development environment, and anyone is welcome to help out. i volunteer some of my time to opendev for the benefit of the community, but acme gating's work with other companies on zuul is how i'm able to work on both zuul development and the community commons] | 13:53 |
| @jim:acmegating.com | [but i get your main point: opendev does use zuul for CD] | 13:54 |
| @jim:acmegating.com | Daco: one of the reasons people sometimes ask that quostion (I don't know if this is your motivation or not) is they want to avoid having too much CI configuration in their software repos. opinions vary on whether that's a good thing (i and many zuul devs think that software testing and gating is extremely closely tied to the development of any particular project, and therefore they belong as close together as possible, but even so, sometimes for organizational purposes, it's useful to move it out of a repo). no matter anyone's opinion on that subject, there are still many options (particularly using job inheritance) for separating a project's code and zuul config, so zuul can still load its config directly from git repos and speculatively execute changes, etc, but the actual config is in a different git repo. | 14:01 |
| @fajfer:reszka.org | I had a draft post for my blog regarding that on shifting how much of power the devs actually have over their repositories. For me a lot of repos have templates ready where you can basically only manipulate job vars and the rest is marked as final | 14:03 |
| @fungicide:matrix.org | a lot of the projects in opendev do that (put common shared job configuration in one or more dedicated untrusted projects), and that's essentially how zuul's "standard-library" (zuul-jobs) is implemented too | 14:04 |
| @jim:acmegating.com | Daco: here's a more complex public example i set up: https://gerrit.googlesource.com/gerrit/+/refs/heads/master/.zuul.yaml the gerrit project wanted to minimize the .zuul.yaml in their repo, so all the jobs are defined in other repos. we convinced them to put just a tiny amount of .zuul.yaml in the main repo so that we could have easy branch-context-aware configuration. if they had wanted to take a harder stance, we could have avoided that file too, but it would have been more complex to maintain. | 14:05 |
| @jangutter:matrix.org | Daco: Would you have an example of a problem you want to solve? Perhaps I've been trying to indoctrinate you into the cult and there's a simple answer to the thing you want to do! | 14:06 |
| @fajfer:reszka.org | Point is you can set it up so people dont hurt themselves even if they do manipulate zuul.yaml :) I am also just a mere user outside of OpenDev | 14:06 |
| @dacofr:matrix.org | We mainly use Zuul for CI (not for CD). And I'm particular case it's for terraform jobs. Most of our repositories used a template, which is defined in trusted config project. The straigthforward is one repo with one terraform folder with one zuul.d/project.yaml pointing to terraform template. | 14:07 |
| But more more, we see repositories where Zuul configuration is overridden by duplicating jobs configuration to match their needs : several terraform folder in same repos, and because they want to parallelise jobs they need to create one jobs per terraform folder. | ||
| @dacofr:matrix.org | Thanks for all answers. | 14:07 |
| @jim:acmegating.com | Daco: it should be possible to structure the job to accept the folder as a job variable, and then create many simple jobs that inherit from it, and just change that variable, like this example in zuul's own config: https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L45-L55 | 14:10 |
| @dacofr:matrix.org | Yes, my idea is to reduce Zuul configuration, and use somehow convention over configuration. I can use somehow pre-commit, but most of the time some users skip it :) | 14:11 |
| @jim:acmegating.com | Daco: if the number of such jobs is very large, then you can make a script to generate the config, which zuul does in the "zuul-jobs" repo, where we have many near-identical jobs for different platforms | 14:11 |
| @jim:acmegating.com | > <@dacofr:matrix.org> Yes, my idea is to reduce Zuul configuration, and use somehow convention over configuration. I can use somehow pre-commit, but most of the time some users skip it :) | 14:11 |
| if you run the same checks in your zuul gate pipeline, then that will enforce them | ||
| @jim:acmegating.com | the zuul-jobs repo has a job that runs to generate the zuul config for that repo, and if its output doesn't match the repo contents, then it fails. that ensures that no one can make changes to the job configuration without running the script that generates all the jobs | 14:12 |
| @dacofr:matrix.org | That's an idea I've in mind, indeed. | 14:13 |
| @jangutter:matrix.org | We've copied so many tricks from zuul-jobs.... | 14:13 |
| @dacofr:matrix.org | But if for any reason, with my platform we want to change/adjust configuration for all repositories, we need to have a way to reconfigure all repositories | 14:14 |
| @dacofr:matrix.org | * But if for any reason, with my platform we want to change/adjust configuration for some terraform jobs, we need to have a way to reconfigure all repositories | 14:14 |
| @jim:acmegating.com | that's where inheritance comes in, if you can put that part of the job/configuration in a central repo, then you can change it globally. | 14:15 |
| @dacofr:matrix.org | I will create an archive file with an example | 14:16 |
| @jangutter:matrix.org | Daco: Hah, we have somewhat similar cases -> it's not terraform but it tends to be things like containers and helm charts. | 14:20 |
| @jangutter:matrix.org | It's one of those difficult problems we haven't quite solved yet: overcomplicating the Zuul job can be an antipattern too. There's real value in keeping Zuul config and jobs lean, while improving test runners. | 14:28 |
| If you could refactor the tester to do the parallelization in the job itself, and pass it multiple nodes, then a project could just run the job with a custom nodeset, for example. | ||
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 962304: Clean up tox remnants https://review.opendev.org/c/zuul/zuul/+/962304 | 14:58 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 962595: Fix TestMinReadyTenantVariant https://review.opendev.org/c/zuul/zuul/+/962595 | 15:04 | |
| @fajfer:reszka.org | Daco: if you ping me in 2 days i have plenty of terraform jobs in zuul myself, maybe we can discuss this and see if it works for you too? | 15:47 |
| @fajfer:reszka.org | I wont have time tomorrow but otherwise we can give it a go | 15:47 |
| @fajfer:reszka.org | We also generate them because of sheer laziness | 15:49 |
| -@gerrit:opendev.org- Dan Duvall proposed: [zuul/nodepool] 952859: Declare base images as standalone/overridable stages https://review.opendev.org/c/zuul/nodepool/+/952859 | 16:45 | |
| @dduvallwmf:matrix.org | corvus: I went ahead and rebased this change since WMF is currently building our nodepool image from this ref. I'm curious, do you anticipate this merging before the next release? Obviously it would be great for us to start building from an actual release tag at some point, but I understand you all have your own upstream priorities | 16:53 |
| @dduvallwmf:matrix.org | * corvus: I went ahead and rebased this change since WMF is currently building our nodepool image from this ref. I'm curious, do you anticipate this merging before the next nodepool release? Obviously it would be great for us to start building from an actual release tag at some point, but I understand you all have your own upstream priorities | 16:54 |
| @jim:acmegating.com | Dan Duvall: let's ask swest for a review of that since he revewied the zuul change; should be an easy review | 16:58 |
| regarding the release: my guess is that we might make one in a couple of months. things are really slowing down with nodepool development (since the effort is going into nodepool-in-zuul), so there's not a lot of urgency to make a release. but the nice thing about that is there isn't much of a reason to move your base since upstream isn't moving, so hopefully that's not too much of a burden. | ||
| so yes, i think that is likely to make it into the next release, but that might be a while. | ||
| @dduvallwmf:matrix.org | Sounds good, and thanks for the update. Shouldn't be too burdensome at all really. We just tweak a Git URL and ref in our build. I'm excited about the upcoming nodepool-in-zuul as well! | 17:01 |
| @jim:acmegating.com | me too. we're || this close to having the static driver. opendev has been using the openstack driver for 3 months now. | 17:02 |
| @fungicide:matrix.org | it's shaping up nicely. i find views like https://zuul.opendev.org/t/zuul/image/debian-trixie super useful, for example | 17:02 |
| @jim:acmegating.com | (but, to be clear, it's not ready for others to use yet -- we're still doing things like breaking the configuration syntax) | 17:02 |
| @fungicide:matrix.org | and sometimes ending up with more network interfaces than we expected ;) | 17:04 |
| @dduvallwmf:matrix.org | sounds like progress to me :) | 17:04 |
| @fungicide:matrix.org | you can't make omelette without breaking a few oeufs | 17:05 |
| @dduvallwmf:matrix.org | we're moving very slowly on our end with Zuul deployment (too many other things to do) but i think the pace will pick back up soon | 17:05 |
| @clarkb:matrix.org | Dan Duvall: I'm curious does that mean you're using the same python assemble system that opendev built but on a different base platform? | 17:22 |
| @clarkb:matrix.org | Mostly just thinking those base container images assume the assemble process works (but you can build that on different platforms I guess) | 17:23 |
| @dduvallwmf:matrix.org | we're overriding the base image to install some extra WMF specific things (and to adhere to our internal policy about base images being from our registry) but otherwise relying on all of your build processes | 17:26 |
| @dduvallwmf:matrix.org | we do that by using a `docker-bake.hcl` file and defining a named buildkit context that matches the stage used for the base image | 17:26 |
| @fungicide:matrix.org | very cool | 17:26 |
| @clarkb:matrix.org | got it | 17:27 |
| @dduvallwmf:matrix.org | https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/blob/a27c05cb810ef747f6272cad5e69bc9be69343da/bake.hcl#L38 | 17:27 |
| @dduvallwmf:matrix.org | yeah i like the setup so far. it means we can get our own dependencies into the image but not have to reinvent your build process (and deal with the inherent maintenance burden that comes with that) | 17:28 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!