| @clarkb:matrix.org | hwangbro the Jenkins gearman plugin was originally written to support zuul < 3.0. Zuul acted like Jenkins brain and ran the jobs in Jenkins via that plugin. Zuul v3 completely dropped Jenkins as it was a major source of scaling issues for us (we had to run 8 large Jenkins masters and they could only deal with about 100 test nodes and we had to restart then weekly because it leaked test node threads) | 00:07 |
|---|---|---|
| @clarkb:matrix.org | It is theoretically possible to write a zuul job that acts as a proxy for Jenkins jobs but I'm not sure anyone is doing that currently | 00:08 |
| @clarkb:matrix.org | Is there some problem with having zuul run the jobs directly? | 00:08 |
| @hwangbro:matrix.org | Jenkins is pretty ingrained in our workflow, so we're trying to make that work first before deciding whether or not we want to rework everything and ditch Jenkins | 00:09 |
| @clarkb:matrix.org | Just thinking out loud about using zuul to run Jenkins jobs. You'll need to somehow coordinate the git repos contents between the two as zuul constructs speculative future states. | 00:11 |
| @clarkb:matrix.org | Nodepool also won't be able to manage the resources for the jobs as it cannot talk to Jenkins anymore | 00:11 |
| @clarkb:matrix.org | But if you solve those two things some some of wrapper job that triggers the Jenkins job and then bubbles results back up might work | 00:12 |
| @hwangbro:matrix.org | Yeah, figuring out how to get the git repos into Jenkins seems like the challenging bit. Thanks for clearing up about Gearman <-> Jenkins not being supported | 00:16 |
| @clarkb:matrix.org | It might be easier to run CI for a single repo or job to evaluate rather than try and integrate Jenkins. Then you can decide if it is worth moving everything into zuul | 00:22 |
| @jim:acmegating.com | hwangbro: when openstack/opendev moved from jenkins+zuul to plain zuul, we started by simplifying our jobs so they were more or less simple shell scripts instead of relying on lots of jenkins plugins. that made it easier to make the transition, and also reduces the vendor lock-in (and that still applies with zuul too :). that's obviously not the only concern, but i think it may be relevant. | 01:42 |
| @pabelanger:matrix.org | hwangbro: a while back ovirt was using zuul to run jenkins: https://softwarefactory-project.io/zuul/t/ovirt/projects but I have no idea how it ended up for them | 02:18 |
| @pabelanger:matrix.org | they would first run zuul job, then use jenkins CLI to trigger the job run, and async for results | 02:19 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] Fix race involving job request locks https://review.opendev.org/c/zuul/zuul/+/808994 | 06:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] Fix waitUntilSettled nodepool race https://review.opendev.org/c/zuul/zuul/+/809013 | 06:33 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 08:52 | |
| - [zuul/zuul] Implement ABC for caching changes in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805835 | ||
| - [zuul/zuul] Cache Gerrit refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805837 | ||
| - [zuul/zuul] Cache Github refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805838 | ||
| - [zuul/zuul] Cache Pagure refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806556 | ||
| - [zuul/zuul] Cache Gitlab refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806557 | ||
| - [zuul/zuul] Cache Git refs (driver) in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806755 | ||
| - [zuul/zuul] Periodically maintain connection caches https://review.opendev.org/c/zuul/zuul/+/806756 | ||
| - [zuul/zuul] Clean up dangling cache data nodes more often https://review.opendev.org/c/zuul/zuul/+/807102 | ||
| -@gerrit:opendev.org- Simon Westphahl proposed: | 09:06 | |
| - [zuul/zuul] Implement ABC for caching changes in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805835 | ||
| - [zuul/zuul] Cache Gerrit refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805837 | ||
| - [zuul/zuul] Cache Github refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805838 | ||
| - [zuul/zuul] Cache Pagure refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806556 | ||
| - [zuul/zuul] Cache Gitlab refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806557 | ||
| - [zuul/zuul] Cache Git refs (driver) in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806755 | ||
| - [zuul/zuul] Periodically maintain connection caches https://review.opendev.org/c/zuul/zuul/+/806756 | ||
| - [zuul/zuul] Clean up dangling cache data nodes more often https://review.opendev.org/c/zuul/zuul/+/807102 | ||
| -@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] Remove the local builds list from the executor client https://review.opendev.org/c/zuul/zuul/+/809175 | 12:06 | |
| -@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] Remove the local builds list from the executor client https://review.opendev.org/c/zuul/zuul/+/809175 | 12:22 | |
| @tobias.henkel:matrix.org | Clark: for when you have time, https://review.opendev.org/c/zuul/zuul/+/775329 would be a small change that requires a second review (gracefully handle unlabel if the label is not existing) | 14:50 |
| @clarkb:matrix.org | tobiash: yup should be able to look at that in a bit. Was just skimming the updated ABC cache change above too | 14:51 |
| @tobias.henkel:matrix.org | Clark: thanks! | 14:51 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] Only report dequeue if we have reported start https://review.opendev.org/c/zuul/zuul/+/761520 | 14:54 | |
| @tobias.henkel:matrix.org | corvus: regarding the time database change. Do you know what penalty we should expect per build that started? As a reference, we're seeing peaks of ~200 starting jobs per minute | 16:04 |
| -@gerrit:opendev.org- Zuul merged on behalf of Tobias Henkel: [zuul/nodepool] Allow empty string in leaked port cleanup https://review.opendev.org/c/zuul/nodepool/+/799046 | 16:05 | |
| @jim:acmegating.com | tobiash: i left a thought on that ^ | 16:08 |
| @clarkb:matrix.org | swest: https://review.opendev.org/c/zuul/zuul/+/805835/ looks good to me. Thanks for the update | 16:08 |
| @tobias.henkel:matrix.org | corvus: ack | 16:12 |
| @tobias.henkel:matrix.org | corvus: I quickly took some timings and the getBuilds call in the time db change takes ~400-500ms which I think is a problem on our heavy loaded system | 16:16 |
| @tobias.henkel:matrix.org | at least as long as we don't have multiple schedulers | 16:17 |
| @jim:acmegating.com | tobiash: i can take a look at whether we are using indexes appropriately | 16:23 |
| @tobias.henkel:matrix.org | corvus: responded on 761520 | 16:24 |
| -@gerrit:opendev.org- Guillaume Chauvel proposed: | 16:34 | |
| - [zuul/zuul] Update tests/base.py to use proper git data https://review.opendev.org/c/zuul/zuul/+/742746 | ||
| - [zuul/zuul] Fix gerrit merge commit change with zuul configuration https://review.opendev.org/c/zuul/zuul/+/762886 | ||
| @jim:acmegating.com | tobiash: thatks, it's becoming more clear to me now. i do think we need a change there. | 16:43 |
| @gchauvel:matrix.org | corvus: I will update the commit message, but the -1 part is required, a change which is a merge commit can be a merge commit on the target branch, but it can also be a merge commit which isn't merged to the target branch. In the latter case changes/******/revisions/*****/files?parent=1 is not sufficent and it would require additional complex crawl in /MERGE_LIST which could result in more /MERGE_LIST , etc... and to identify when to stop | 16:46 |
| @gchauvel:matrix.org | corvus: thus using merger getFilesChanges is more suited | 16:47 |
| @clarkb:matrix.org | guillaumec: under what situations does that make sense as a gerrit user? I'm wondering if there is a use case there or we can just say zuul doesn't need to support that? | 16:47 |
| @clarkb:matrix.org | is a change like that even mergable? | 16:48 |
| @clarkb:matrix.org | Anyway I ask beacuse my first thought on that is that gerrit wouldn't let a change like that merge | 16:50 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] Only report dequeue if we have reported start https://review.opendev.org/c/zuul/zuul/+/761520 | 16:50 | |
| @tobias.henkel:matrix.org | corvus: how about something like that? ^ | 16:50 |
| @jim:acmegating.com | Clark, guillaumec: yeah those are good questions. i'm not opposed to using the merger if we have to, but i'd like to avoid if it it's not necessary, and understanding this would help | 16:51 |
| @jim:acmegating.com | tobiash: lgtm; btw if we are reporting non-started buildset ends, then there will be "Unable to find buildset {uuid} in DB" at error log in the scheduler (but it's harmless) | 16:54 |
| @westphahl:matrix.org | Clark: thanks! | 16:54 |
| @jim:acmegating.com | Clark: if you have a minute for https://review.opendev.org/805743 -- it's zk backup commands for nodepool, similar to what we did for zuul | 16:56 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] Only report dequeue if we have reported start https://review.opendev.org/c/zuul/zuul/+/761520 | 16:58 | |
| @tobias.henkel:matrix.org | corvus: I think I also need to protect from item.job_graph being None | 16:58 |
| @clarkb:matrix.org | corvus: ya I can take a look at that | 16:59 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] Only report dequeue if we have reported start https://review.opendev.org/c/zuul/zuul/+/761520 | 17:06 | |
| @clarkb:matrix.org | corvus: I've approved the nodepool image data change. Do you think that is something opendev should be running in cron? It seems not necessarily super critical but could help speed recovery up if you end up in a bad spot | 17:14 |
| @foodster:matrix.org | Hello! new to zuul gating..so apologies if this is a stupid question.. | 17:15 |
| I am trying to setup a gating pipeline with gitlab and when we have 2 MRs that trigger the pipeline one after another, merge operation fails for the MR that comes later as it needs a rebase by gitlab..is it possible to rebase from the pipeline itself..I could not find anything related to it in gitlab driver documentation? | ||
| @jim:acmegating.com | Clark: yeah, i don't think it's critical, but maybe a good idea if not too much trouble | 17:18 |
| @jim:acmegating.com | @foodster:matrix.org: it sounds like gitlab may be doing a rebase workflow. if you can configure it not to do that, and instead fast-forward or merge when necessary, then i would expect it to work. | 17:19 |
| @jim:acmegating.com | the github driver has a cherry-pick mode that may be similar to a rebase workflow | 17:21 |
| @jim:acmegating.com | anyway, it's a good question, and the answer might be to tweak config setting in either gitlab or zuul, or it might be to enhance the gitlab driver to better support that case. i don't know off the top of my head. | 17:25 |
| @foodster:matrix.org | thanks..lemme see if there is some setting in gitlab that can be changed | 17:26 |
| @gtema:matrix.org | https://docs.gitlab.com/ee/user/project/merge_requests/fast_forward_merge.html should be helping, but I would be also interested | 17:47 |
| @fungicide:matrix.org | it looks like the desired behavior would actually involve turning off the fast-forward merge requests option, since --ff-only behavior is more like the problem description | 17:52 |
| @fungicide:matrix.org | you would turn on the fast-forward option if you wanted to avoid having merge commits added to stitch the unchanged merge request into the branch history | 17:53 |
| @foodster:matrix.org | yeah..we need ff merge | 17:53 |
| @fungicide:matrix.org | but forcing fast-forward only merges means you must rebase changes on each other in the order you want them to merge. i.e. you're choosing the sequence in which changes merge instead of letting zuul choose | 17:54 |
| @fungicide:matrix.org | the end result is the developers on the project must coordinate to rebase their merge requests on one another in the order they will merge, if you want them to be gated in parallel | 17:57 |
| @fungicide:matrix.org | zuul usually performs best in situations where merge commits are acceptable, because that allows it to decide for you what order the merge requests will merge in | 17:58 |
| @clarkb:matrix.org | also I'm not sure github or gitlab have a way to stack PRs/MRs | 17:59 |
| @clarkb:matrix.org | That sort of workflow could potentially work with Gerrit where you can explicitly build a tree | 17:59 |
| @fungicide:matrix.org | the alternative which might work is https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html since that probably also rebases the squashed commit at merge time | 18:01 |
| @fungicide:matrix.org | reviews of https://review.opendev.org/q/topic:tox-role would be appreciated, since there are some opendev users who would like to make use of the fixes/options there | 18:36 |
| @gtema:matrix.org | > <@fungicide:matrix.org> the alternative which might work is https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html since that probably also rebases the squashed commit at merge time | 18:41 |
| Squash support for gitlab is not yet merged in Zuul. I recently created change for that, but no reviews so far | ||
| @fungicide:matrix.org | gtema: aha, thanks for clarifying | 18:47 |
| @foodster:matrix.org | gtema: can you share your change request for squash? want to refer it and see if I can update the driver to add rebase option | 19:16 |
| @gtema:matrix.org | https://review.opendev.org/c/zuul/zuul/+/806231/3 | 19:18 |
| @clarkb:matrix.org | corvus: fyi https://review.opendev.org/c/zuul/nodepool/+/805743 failed tested. I don't have time to look at it right this moment though | 20:38 |
| @jim:acmegating.com | doesn't seem related, i suspect it's a race in the testnodeassignmentattenantquotaram test (which is relatively new) | 20:53 |
| @jim:acmegating.com | (and i think that failure borked the test suite to cause the others) | 20:53 |
| @jim:acmegating.com | i think i'll recheck 743 but we should take a look at that test | 20:54 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] Add commands to export/import image data from ZK https://review.opendev.org/c/zuul/nodepool/+/805743 | 22:01 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] WIP: Add an API for ZK-backed objects https://review.opendev.org/c/zuul/zuul/+/809293 | 22:34 | |
| @jim:acmegating.com | swest: ^ i think if we start with something like that, and then layer part of your pipeline state restore change on top of it, we can load the pipeline state when we start processing a pipeline, then update it in ZK as we go | 22:35 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] WIP: Add an API for ZK-backed objects https://review.opendev.org/c/zuul/zuul/+/809293 | 22:36 | |
| @mordred:inaugust.com | corvus: element does a really great job of making method names look super fancy | 22:43 |
| @jim:acmegating.com | mordred: element? aroo? what do you see? | 22:44 |
| @jim:acmegating.com | mordred: oh, you're seeing the name i typed into matrix-weechat. yeah i haven't fixed that bug yet | 22:46 |
| @jim:acmegating.com | that's not element's fault; that's matrix-weechat being overly agressive at dealing with underscores | 22:46 |
| @mordred:inaugust.com | ah - nod. well - it's looks neat | 23:04 |
| @jim:acmegating.com | mordred: the emphassis is on the wrong syllable | 23:05 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] WIP: Add a zookeeper map to developer docs https://review.opendev.org/c/zuul/zuul/+/809300 | 23:30 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!