| @iwienand:matrix.org | hrm, still seeing issues building nodepool containers from upstream repos | 00:49 |
|---|---|---|
| @iwienand:matrix.org | 2021-10-27 00:44:35.422342 | ubuntu-focal | [91mE: Failed to fetch https://provo-mirror.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_11/amd64/crun_1.2-1_amd64.deb File has unexpected size (253024 != 252748). Mirror sync in progress? [IP: 91.193.113.70 443] | 00:49 |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 811955: Pass ZK context to deserialize method of ZKObjects https://review.opendev.org/c/zuul/zuul/+/811955 | 01:35 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 812450: Move ZuulMark from configloader to model https://review.opendev.org/c/zuul/zuul/+/812450 | 01:39 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Matthieu Huin https://matrix.to/#/@mhuin:matrix.org: [zuul/zuul] 735586: Zuul-web: Add authentication-realm attribute to tenants https://review.opendev.org/c/zuul/zuul/+/735586 | 01:50 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Matthieu Huin https://matrix.to/#/@mhuin:matrix.org: [zuul/zuul] 735586: Zuul-web: Add authentication-realm attribute to tenants https://review.opendev.org/c/zuul/zuul/+/735586 | 02:09 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 815578: DNM: testing https://review.opendev.org/c/zuul/nodepool/+/815578 | 02:11 | |
| @iwienand:matrix.org | i'm not really sure if it will help but i filed an issue @ https://github.com/kubic-project/issues/issues/15 | 02:43 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815207: Document (and warn about) tag job role checkouts https://review.opendev.org/c/zuul/zuul/+/815207 | 03:08 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815196: Use an election for scheduler periodic stats https://review.opendev.org/c/zuul/zuul/+/815196 | 04:11 | |
| -@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 814996: Make the ConfigLoader work independently of the Scheduler https://review.opendev.org/c/zuul/zuul/+/814996 | 07:19 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [zuul/zuul] 815529: Use all but one CPU when unittesting https://review.opendev.org/c/zuul/zuul/+/815529 | 07:50 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 10:42 | |
| - [zuul/zuul] 815616: Only reset the pipeline state if needed https://review.opendev.org/c/zuul/zuul/+/815616 | ||
| - [zuul/zuul] 815617: Ensure same layout UUID across schedulers https://review.opendev.org/c/zuul/zuul/+/815617 | ||
| -@gerrit:opendev.org- Simon Westphahl proposed: | 10:52 | |
| - [zuul/zuul] 815616: Only reset the pipeline state if needed https://review.opendev.org/c/zuul/zuul/+/815616 | ||
| - [zuul/zuul] 815617: Ensure same layout UUID across schedulers https://review.opendev.org/c/zuul/zuul/+/815617 | ||
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 815617: Ensure same layout UUID across schedulers https://review.opendev.org/c/zuul/zuul/+/815617 | 10:56 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: | 12:16 | |
| - [zuul/zuul] 815450: Create bundle items during queue deserialization https://review.opendev.org/c/zuul/zuul/+/815450 | ||
| - [zuul/zuul] 815495: Fix Gerrit change (de-)serialization https://review.opendev.org/c/zuul/zuul/+/815495 | ||
| - [zuul/zuul] 815616: Only reset the pipeline state if needed https://review.opendev.org/c/zuul/zuul/+/815616 | ||
| - [zuul/zuul] 815617: Ensure same layout UUID across schedulers https://review.opendev.org/c/zuul/zuul/+/815617 | ||
| -@gerrit:opendev.org- Simon Westphahl proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815565: Remove unecessary assignment in re-enqueue https://review.opendev.org/c/zuul/zuul/+/815565 | 12:16 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815553: Fix test race with paused jobs https://review.opendev.org/c/zuul/zuul/+/815553 | 12:19 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: | 12:23 | |
| - [zuul/zuul] 812451: Recursively delete all sub-nodes of ZKObjects https://review.opendev.org/c/zuul/zuul/+/812451 | ||
| - [zuul/zuul] 812466: Only retry ZK operations for Kazoo exceptions https://review.opendev.org/c/zuul/zuul/+/812466 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: | 12:26 | |
| - [zuul/zuul] 812452: Store build sets in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812452 | ||
| - [zuul/zuul] 812467: Add support for sharded ZKObjects https://review.opendev.org/c/zuul/zuul/+/812467 | ||
| -@gerrit:opendev.org- Tobias Henkel proposed on behalf of Simon Westphahl: [zuul/zuul] 815278: DNM: execute tests with two schedulers https://review.opendev.org/c/zuul/zuul/+/815278 | 12:28 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 812750: Add LocalZKContext for job freezing https://review.opendev.org/c/zuul/zuul/+/812750 | 12:35 | |
| -@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: | 13:55 | |
| - [zuul/zuul] 735586: Zuul-web: Add authentication-realm attribute to tenants https://review.opendev.org/c/zuul/zuul/+/735586 | ||
| - [zuul/zuul] 736968: zuul-web: add pipelines' manager, triggers data in status https://review.opendev.org/c/zuul/zuul/+/736968 | ||
| - [zuul/zuul] 734082: web UI: user login with OpenID Connect https://review.opendev.org/c/zuul/zuul/+/734082 | ||
| - [zuul/zuul] 734850: web UI: allow a privileged user to dequeue a change https://review.opendev.org/c/zuul/zuul/+/734850 | ||
| - [zuul/zuul] 736772: web UI: allow a privileged user to re-enqueue a change https://review.opendev.org/c/zuul/zuul/+/736772 | ||
| - [zuul/zuul] 768115: Web UI: allow a privileged user to request autohold https://review.opendev.org/c/zuul/zuul/+/768115 | ||
| - [zuul/zuul] 768199: Web UI: add Autoholds, Autohold page https://review.opendev.org/c/zuul/zuul/+/768199 | ||
| - [zuul/zuul] 810699: Web UI: Show pipeline types as icons https://review.opendev.org/c/zuul/zuul/+/810699 | ||
| - [zuul/zuul] 781858: web UI: allow a privileged user to promote a change https://review.opendev.org/c/zuul/zuul/+/781858 | ||
| - [zuul/zuul] 802559: Web UI: Add "Create Autohold Request" form, improve API error messages https://review.opendev.org/c/zuul/zuul/+/802559 | ||
| - [zuul/zuul] 769943: Example Docker compose: keycloak integration https://review.opendev.org/c/zuul/zuul/+/769943 | ||
| -@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 814996: Make the ConfigLoader work independently of the Scheduler https://review.opendev.org/c/zuul/zuul/+/814996 | 14:11 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 815527: Add some debug logging to change cache https://review.opendev.org/c/zuul/zuul/+/815527 | 14:37 | |
| @bridgefan:matrix.org | I was wondering if anyone here knows whether Zuul handles nested Depends-On statements? (eg. patch1 depends on patch2 and patch3 depends on patch1, will patch3 also use patch2?) | 15:27 |
| @clarkb:matrix.org | bridgefan: what you describe there is a dependency cycle. By default Zuul doesn't allow these but there is a configuration toggle to allow it and then it should be properly supported: https://zuul-ci.org/docs/zuul/reference/queue_def.html#attr-queue.allow-circular-dependencies | 15:30 |
| @clarkb:matrix.org | ianw: corvus neat both nodepool and zuul depend on the same repo for libcontainers' tools but zuul only installs skopeo and seems to not have problems and nodepool installs podman and breaks due to that image not matching the index | 15:38 |
| @clarkb:matrix.org | I was curiuos why zuul wasn't affected and it seems to be due to the specific packages | 15:38 |
| @clarkb:matrix.org | hrm it failed again. I guess the mirroring of that is load balanced and I see consistent stuff from home? | 16:49 |
| @jim:acmegating.com | Clark: it looks like we got through the first batch of sos changes; i think https://review.opendev.org/812760 is up next and waiting for your review | 16:50 |
| @clarkb:matrix.org | corvus: yup I'll try to get to that soon | 16:50 |
| @jim:acmegating.com | Clark: so you're thinking single bad load-balanced mirror? | 16:50 |
| @clarkb:matrix.org | corvus: ya at least looking at the repo from my desktop Packages and Packages.gz matches what I get when I download the crun package | 16:51 |
| @clarkb:matrix.org | file size and sha1 | 16:51 |
| @clarkb:matrix.org | but the sha is different when the job fails | 16:51 |
| @jim:acmegating.com | Clark: i jut did a wget of https://provo-mirror.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_11/amd64/crun_1.2-1_amd64.deb and got 253024 -- that's the bad size right? | 16:53 |
| @clarkb:matrix.org | no that is what shows up in the Packages file I got | 16:53 |
| @jim:acmegating.com | oh, i guess i don't understand "Hashes of expected file: Filesize:252748" | 16:54 |
| @jim:acmegating.com | i would think that would mean the expected hash of the file. but i guess that means unexpected hash of the file as received? | 16:55 |
| @clarkb:matrix.org | The problem would be getting a different Packages content which expects a different package file or a different package file for a stale Packages | 16:55 |
| @jim:acmegating.com | i confirm that the .deb i got appears okay. | 16:55 |
| @clarkb:matrix.org | its hard to know but maybe fungi can infer based on the error messages? | 16:55 |
| @clarkb:matrix.org | oh interesting the latest error complains about the packages file itself not the crun package | 17:00 |
| @clarkb:matrix.org | previous failures were for the crun package. Maybe they are updating the Packages file to match and some hash somewhere hasn't been updated? Though I don't know how hashes of the Packages file are verified | 17:01 |
| @clarkb:matrix.org | is that via the Release file? | 17:01 |
| @clarkb:matrix.org | yup, if I pull the Releases file and the Packages file they match up | 17:02 |
| @clarkb:matrix.org | This feels like some sort of subtle inconsistency that we hit via the test nodes more than local (load balancer etc) | 17:03 |
| @clarkb:matrix.org | But now I would expoect zuul to see issues too since Release and Packages will affect skopeo | 17:08 |
| @bridgefan:matrix.org | Clark: thanks | 17:08 |
| @clarkb:matrix.org | corvus: question on https://review.opendev.org/c/zuul/zuul/+/812760 I'm happy to approve it as is if that is a non issue | 17:15 |
| @pearcetyler:matrix.org | Hello 👋 My dependent gate pipeline is running tests one at a time instead of testing in parallel (subsequent items in the pipeline remain queued until the previous one completes). I used the quickstart guide to get Zuul running -- Do I need to change any configuration to allow Zuul to test in parallel? | 17:15 |
| @jim:acmegating.com | Tyler Pearce: assuming you have enough nodes, that should be enough. you may have provides/requires or dependencies which cause jobs to wait though. | 17:16 |
| @jim:acmegating.com | or semaphores | 17:16 |
| @jim:acmegating.com | Tyler Pearce: if the status page says 'queued' its probably waiting on a node; if it says 'waiting' then it's, well, waiting for something else. | 17:17 |
| @jim:acmegating.com | mouseover 'waiting' should tell you what | 17:17 |
| @pearcetyler:matrix.org | It does say queued. How do I provision additional nodes? | 17:17 |
| @jim:acmegating.com | Tyler Pearce: configure the nodepool launcher for whatever cloud/k8s cluser/set of static nodes you have. by default, the quickstart configures a single static node as a docker container for demonstration purposes. you could probably make more of those, but it would be better to use a cloud, or a k8s cluster, or manage a set of static nodes separately from the quickstart docker-compose (that doesn't scale well). | 17:19 |
| @pearcetyler:matrix.org | Ahh I see. Thank you! | 17:20 |
| @jim:acmegating.com | Tyler Pearce: here are the docs for nodepool, and you can see the different drivers listed there: https://zuul-ci.org/docs/nodepool/ | 17:20 |
| @pearcetyler:matrix.org | Perfect, thank you! | 17:20 |
| @clarkb:matrix.org | corvus: is https://zuul.opendev.org/t/zuul/build/ca5ba7800e5345cc9c6cdd3ca84f3944/log/job-output.txt#10957-10990 the error you were adding debugging to debug? | 17:29 |
| @clarkb:matrix.org | corvus: also thought on https://review.opendev.org/c/zuul/zuul/+/813552 but nothing critical | 17:31 |
| @jim:acmegating.com | Clark: yep that's the one, i'll take a look thanks | 17:32 |
| @jim:acmegating.com | Clark: replied on 812760 | 17:36 |
| @jim:acmegating.com | Clark: oh that test failure was from before the debug log change merged isn't it? oh well | 17:38 |
| @clarkb:matrix.org | ah unfortunate | 17:43 |
| @clarkb:matrix.org | Left a comment on https://review.opendev.org/c/zuul/zuul/+/813809/ I don't intend on +2/+Aing that but if others feel that is the best way to address that problem I won't -1 either | 17:43 |
| @jim:acmegating.com | Clark: yeah, i don't love that change either. why don't i go ahead and work on a followup to remove that for the end of the stack. that way we don't have to worry about rearranging the furniture in the middle of that stack, but can have something explicit when the dust settles? | 17:49 |
| @fungicide:matrix.org | just catching up, yeah that package checksum and size differences would be an indication that the folks maintaining that package repository aren't careful to upload new packages and then regenerate the indices after that | 17:50 |
| @clarkb:matrix.org | > <@jim:acmegating.com> Clark: yeah, i don't love that change either. why don't i go ahead and work on a followup to remove that for the end of the stack. that way we don't have to worry about rearranging the furniture in the middle of that stack, but can have something explicit when the dust settles? | 17:50 |
| That seems reasonable. | ||
| @clarkb:matrix.org | fungi: we are also seeing mismatch between Release content and Package file contents | 17:51 |
| @fungicide:matrix.org | probably they do something naive like generate indices elsewhere and then copy them and the files to the site at roughly the same time but possibly out of order, at which point you race getting an old index with a new package | 17:51 |
| @fungicide:matrix.org | or they forgot to copy a file, if it's persisting | 17:51 |
| @fungicide:matrix.org | also they could be overwriting packages in-place rather than incrementing versions, which is a pretty big no-no | 17:52 |
| @clarkb:matrix.org | it does seem to be persistent | 17:56 |
| @clarkb:matrix.org | though dpeending on how often they sync may not have crossed multiple sync periods yet | 17:56 |
| @ologinov:synapse.sardinasystems.com | Hi folks! | 19:38 |
| I have a Zuul (docker-compose) quick installation installed by default. I noticed that the standard installation (where used only gerrit), by changing zuul-config, scheduler can add a tenant, pipelines and jobs for zuul (following the step-by-step guide).It works as expected.Scheduler log after merge to zuul-config(Gerrit) here: https://pastebin.com/uht0hcZn | ||
| If I'll add a gitlab connection, projects to etc_zuul/main.yml, do the same with zuul-config, then I will see that the tenant has been created, projects have been created, but not pipelines and jobs.Git diff changes for default zuul and modified for Gitlab connection here: https://pastebin.com/iWdbyCi0 | ||
| Gitlab: scheduler log after push into zuu-config (master) | ||
| https://pastebin.com/zqbLZwXj | ||
| I compared the output in debug mode and came to the conclusion that the PipelineManager from zuul/manager/__init__.py not performed in the case of gitlab. | ||
| @ologinov:synapse.sardinasystems.com | I have no idea at this point what I can do next. Do you have any idea guys? | 19:40 |
| @clarkb:matrix.org | Oleg Loginov: my first hunch is that zuul isn't able to connect to gitlab and pull configs which prevents it from loading any pipelines or jobs | 20:20 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815744: Use a metaclass to deserialize event objects https://review.opendev.org/c/zuul/zuul/+/815744 | 20:25 | |
| @jim:acmegating.com | Clark, tobiash, swest: ^ there's an alternative to globals() | 20:26 |
| @jim:acmegating.com | in all honesty, if folks would prefer an explicit list, we can still do that. this actually seems less error-prone though. so as much as i normally figure metaclasses are the answer to a question no one asked, maybe this is a reasonable use? | 20:30 |
| @ologinov:synapse.sardinasystems.com | Clark: i see that GitLab can send message via webhook to scheduler, and scheduler can leave comment for GitLab MR | 20:32 |
| @ologinov:synapse.sardinasystems.com | it's possibly if zuul-config will be located on Gerrit repo | 20:33 |
| @ologinov:synapse.sardinasystems.com | I mean it tested when pipelines was created from Gerrit repo | 20:35 |
| @clarkb:matrix.org | Oleg Loginov: right that is why I'm wondering if it isn't loading config from gitlab | 20:35 |
| @ologinov:synapse.sardinasystems.com | > <@ologinov:synapse.sardinasystems.com> Clark: i see that GitLab can send message via webhook to scheduler, and scheduler can leave comment for GitLab MR | 20:36 |
| it's not enough for zuul-config,right? | ||
| @ologinov:synapse.sardinasystems.com | I saw that scheduler gives messages about each commit/MR/etc, which Gitlab did there | 20:39 |
| @clarkb:matrix.org | Oleg Loginov: the scheduler needs to be able to fetch the repos when it starts up to load their configs | 20:49 |
| @clarkb:matrix.org | if there are projects in gitlab with the zuul configs and zuul cannot clone/fetch them this could explain the problem | 20:49 |
| @clarkb:matrix.org | corvus: re metaclass, I think my only concern with it is that the use of `__init__` seems very underdocumented and typically you'd implement `__new__`, in this case you just want to add an attribute not actually create a new class object so `__init__` is probably the right choice, but I can't find anywhere that indicates `__new__` will call that `__init__` with those args | 20:50 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815747: Fix race is test_zookeeper_disconnect https://review.opendev.org/c/zuul/zuul/+/815747 | 20:52 | |
| @clarkb:matrix.org | corvus: I guess in this case s/cls/self/ might be more accurate btu it just happens to be that the object created by the metaclass' `__new__` method is the new class | 20:54 |
| @clarkb:matrix.org | because we're writing a class that emits a class so the first arg to `__init__` is what we think of classically as self it just happens to be a class | 20:55 |
| @ologinov:synapse.sardinasystems.com | > <@clarkb:matrix.org> if there are projects in gitlab with the zuul configs and zuul cannot clone/fetch them this could explain the problem | 20:55 |
| correctly, but i can fetch repo (project is public) without username/password via http. How can I check it from zuul side that connection is correct, or may be another way for check it? | ||
| @clarkb:matrix.org | Oleg Loginov: the zuul scheduler log should record stuff about asking for merges for the various projects at startup. Then you can check the merger and executor logs for any failed merges as well as confirming that the scheduler saw those merges complete | 20:56 |
| @jim:acmegating.com | Clark: yeah, i'd be happy to s/init/new/ or s/cls/self/. neither is critical for this usage. | 20:57 |
| @clarkb:matrix.org | corvus: I think I have a slight prefernece to s/cls/self/ as its a bit more clear what is going on though it might first create confusion. But not strongly enough to -1 over that | 20:58 |
| @jim:acmegating.com | it's 32 changes away, so we have time | 20:59 |
| @clarkb:matrix.org | ha yes. fwiw it does seem that zuul is having the issues with the skopeo install on the docker images | 21:00 |
| @clarkb:matrix.org | I wonder if we should be reaching out to suse folks directly | 21:01 |
| @clarkb:matrix.org | (I believe that repo is populated by their OBS service) | 21:01 |
| @jim:acmegating.com | more than https://github.com/kubic-project/issues/issues/15 ? | 21:01 |
| @clarkb:matrix.org | Ya like maybe email Andreas or Colleen directly | 21:02 |
| @clarkb:matrix.org | or Dirk | 21:02 |
| @ologinov:synapse.sardinasystems.com | Clark: thanks for help! | 21:03 |
| @jim:acmegating.com | where does provo-mirror come from? | 21:03 |
| @clarkb:matrix.org | corvus: I think that the repo url we give apt is getting http redirected to the provo mirror | 21:04 |
| @clarkb:matrix.org | and that may redirect to different locations depending on locality? But I haven't checked that yet | 21:04 |
| @jim:acmegating.com | hrm. if i just visit that in a browser i don't get redirected | 21:04 |
| @clarkb:matrix.org | huh | 21:04 |
| @jim:acmegating.com | https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_11/ | 21:04 |
| @clarkb:matrix.org | we definitely don't put provo-mirror in the apt config we use https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_11/ | 21:05 |
| @clarkb:matrix.org | oh neat podman is available in Bullseye | 21:05 |
| @clarkb:matrix.org | maybe skopeo is too? THere is possibiltiy we could drop the special repo entirely | 21:06 |
| @clarkb:matrix.org | https://packages.debian.org/bullseye/skopeo Boom I'll get patches up for nodepool and zuul | 21:06 |
| @jim:acmegating.com | apt-get install skopeo works in a debian:bullseye container | 21:06 |
| @clarkb:matrix.org | Then I need to do a school run | 21:06 |
| @jim:acmegating.com | Clark: i can do the patches if you want to review...? :) | 21:07 |
| @clarkb:matrix.org | sure | 21:07 |
| @clarkb:matrix.org | either way | 21:07 |
| @jim:acmegating.com | on it | 21:07 |
| @clarkb:matrix.org | ianw: ^ fyi | 21:07 |
| @iwienand:matrix.org | yeah, i was thinking we could probably drop the kubic installs | 21:09 |
| @jim:acmegating.com | btw, if i wget packgaes, i do eventually end up with redirects to provo-mirror. so i think it's the actual files that have the magic redirect | 21:09 |
| @jim:acmegating.com | bullseye has skopeo 1.2.2 rather than 1.3.0 | 21:10 |
| @clarkb:matrix.org | I don't expect that will cause problems for our docker image pull/push stuff | 21:10 |
| @clarkb:matrix.org | but I've not kept up with their release notes. podman + dib + container element might be the one more likely to break, but I think we try it and this will be far more relaible if it works | 21:11 |
| @jim:acmegating.com | ++ | 21:11 |
| @iwienand:matrix.org | we did get some good dib runs out of that mirror overnight, has it broken again? | 21:13 |
| @clarkb:matrix.org | ianw: yes and now it is a problem with the Packages file so both zuul and nodepool are sad | 21:14 |
| @clarkb:matrix.org | previously it was package specific but now it is the packages index file so all packages are affected | 21:14 |
| @clarkb:matrix.org | its also not a 100% failure like the redirect is load balancing or the provo-mirror has multiple backends | 21:14 |
| @iwienand:matrix.org | Clark: ahh, ok; so something more systematic on the whole mirror than just one corrupt file. yesterday everything i saw was that one crun .deb | 21:15 |
| @ologinov:synapse.sardinasystems.com | checked scheduler and executor logs - no messages about failed merge something.I will be glad to any direction of search. | 21:15 |
| @jim:acmegating.com | remote: https://review.opendev.org/c/zuul/zuul/+/815752 Rely on bullseye image for skopeo in container image [NEW] | 21:17 |
| @jim:acmegating.com | if folks can see that ^ i wonder if we are missing the bot | 21:19 |
| @jim:acmegating.com | bot logs say it just reconnected | 21:20 |
| @clarkb:matrix.org | I've seen it miss a small number of events. The change loads for me but I'll have to review it after the school run. Back soon | 21:20 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 812760: Add RepoState object https://review.opendev.org/c/zuul/zuul/+/812760 | 21:21 | |
| @jim:acmegating.com | yay its back | 21:21 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 815753: Use bullseye podman in container https://review.opendev.org/c/zuul/nodepool/+/815753 | 21:22 | |
| @iwienand:matrix.org | https://review.opendev.org/c/openstack/diskimage-builder/+/815574 [dnm] trying fedora 35 will try ^ out | 21:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 812673: Store RepoFiles for a build set in Zookeeper https://review.opendev.org/c/zuul/zuul/+/812673 | 21:23 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 815754: Rely on bullseye image for podman in container image https://review.opendev.org/c/zuul/nodepool/+/815754 | 21:25 | |
| @jim:acmegating.com | ianw: ^ want to take a look at that? (and also https://review.opendev.org/815752 ?) | 21:25 |
| @jim:acmegating.com | oh you wrote a change while i was writing it. | 21:26 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 815754: Rely on bullseye image for podman in container image https://review.opendev.org/c/zuul/nodepool/+/815754 | 21:27 | |
| @jim:acmegating.com | ianw: ^ i updated mine to include some bits from yours. main thing is i have omitted install-recommends. i don't think it's necessary with bullseye. | 21:27 |
| @jim:acmegating.com | ianw: so if that works, i think that's preferable. | 21:27 |
| @iwienand:matrix.org | ok, istr that being there for a specific reason to do with some tun/tap thing | 21:28 |
| @jim:acmegating.com | ianw: can you take a look at my commit message and see if you disagree? | 21:28 |
| @jim:acmegating.com | based on what was written in the comment, i don't think it should be necessary. but maybe you have more context. | 21:29 |
| @jim:acmegating.com | ``` | 21:29 |
| Depends: init-system-helpers (>= 1.52), libc6 (>= 2.15), libdevmapper1.02.1 (>= 2:1.02.97), libgpgme11 (>= 1.4.1), libseccomp2 (>= 2.4.1), conmon (>= 2.0.18~), containernetworking-plugins (>= 0.8.7), golang-github-containers-common, crun | runc (>= 1.0.0~rc92~), iptables | ||
| Recommends: buildah (>= 1.10.1-6~), fuse-overlayfs (>= 1.0.0~), slirp4netns (>= 0.4.1~), catatonit | tini | dumb-init, uidmap, golang-github-containernetworking-plugin-dnsname | ||
| ``` | ||
| @iwienand:matrix.org | https://packages.debian.org/stable/podman | 21:30 |
| @iwienand:matrix.org | the rec: slirp4netns was the one i was thinking of | 21:30 |
| @jim:acmegating.com | ianw: okay, if you update the comment then i'll abandon mine and +2 yours :) | 21:31 |
| @iwienand:matrix.org | and possibly uidmap was well. i think it was both about running as non-root | 21:31 |
| @jim:acmegating.com | something like the phrase "several programs such as slirp4netns and uidmap which are necessary for running as non-root" would have stopped me from optimizing | 21:32 |
| @jim:acmegating.com | i just assumed that containernetwork-plugins was it, and was in recommends, not requires in kubic | 21:33 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 815753: Use bullseye podman in container https://review.opendev.org/c/zuul/nodepool/+/815753 | 21:35 | |
| @iwienand:matrix.org | corvus: yes, it's been a dynamic area ... i'll try a follow-on when things stabilise to double check | 21:37 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 815752: Rely on bullseye image for skopeo in container image https://review.opendev.org/c/zuul/zuul/+/815752 | 21:39 | |
| @jim:acmegating.com | ianw: thanks! +2 nodepool and updated ^ | 21:40 |
| @clarkb:matrix.org | ianw: I +2'd https://review.opendev.org/c/zuul/nodepool/+/815753 but didn't approve it as you indicated youwere testing it with another change. Can you approve it when you are happy with the results? | 21:57 |
| @clarkb:matrix.org | corvus: the skopeo change lgtm too | 21:58 |
| @iwienand:matrix.org | Clark: yep, will test with the f35 change in dib | 21:59 |
| @jim:acmegating.com | i'm going to +3 and promote the zuul change | 22:02 |
| @clarkb:matrix.org | corvus: ++ | 22:03 |
| @iwienand:matrix.org | Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 | 22:31 |
| @iwienand:matrix.org | that's annoying, it seems container networking is broken with the debian podman | 22:32 |
| @clarkb:matrix.org | Do we need to bind mount in an /etc/hosts? | 22:34 |
| @clarkb:matrix.org | Er no /etc/resolv.conf | 22:34 |
| @iwienand:matrix.org | we never used to have to, but i am conflating two things here -- the f35 update and the debian podman. it could be either | 22:36 |
| @iwienand:matrix.org | "Several files will be automatically created within the container. These include /etc/hosts, /etc/hostname, and /etc/resolv.conf to manage networking. These will be based on the host’s version of the files, though they can be customized with options" https://docs.podman.io/en/latest/markdown/podman-run.1.html | 22:40 |
| @clarkb:matrix.org | corvus: in https://review.opendev.org/c/zuul/zuul/+/813913/10/zuul/model.py why are we maintaining jobs and job_graph separately in the buildset? | 23:06 |
| @clarkb:matrix.org | Doesn't the job_graph have a list of jobs that we can query | 23:06 |
| @clarkb:matrix.org | Its also not clear to me how we populate that new attribute of the buildset. Is that jobs dict empty? | 23:07 |
| @clarkb:matrix.org | ianw: on our test nodes we set up a 127.0.0.1 resolver which won't be accessible from the container unless they use host networking | 23:13 |
| @jim:acmegating.com | Clark: looking | 23:14 |
| @clarkb:matrix.org | ianw: I wonder if the non debian version is able to detect that and replace stuff appropriately but debian isn't because they don't want people talking to google dns or some such | 23:14 |
| @jim:acmegating.com | Clark: a job graph is basically a list of job names and the dependencies between them. the jobs attribute of the buildset are the actual Job() instances (which contain tons of data) | 23:17 |
| @jim:acmegating.com | (so each has information the other lacks) | 23:18 |
| @clarkb:matrix.org | hrm, but that new attribute isn't really managed at all on the buildset from what I see? | 23:20 |
| @jim:acmegating.com | Clark: re 2nd question -- BuildSet.jobs is passed by reference to JobGraph, and JobGraph.addJob mutatates it | 23:20 |
| @clarkb:matrix.org | but the change updates to read jobs from it. Just not seeing where we write them | 23:20 |
| @clarkb:matrix.org | ok /me will need to look more closely at addJob | 23:20 |
| @iwienand:matrix.org | Clark: yep, that's what i was thinking too :) https://review.opendev.org/c/openstack/diskimage-builder/+/815763 should dump the resolv.conf, but i'm also trying to get a node on hold | 23:21 |
| @clarkb:matrix.org | corvus: right so addJob does a dict update of the jobgraph but not the buildset jobs? | 23:21 |
| @clarkb:matrix.org | oh I think I may understand now we're serializing and deserializing the jobs and job graph at the buildset level | 23:22 |
| @clarkb:matrix.org | we just haven't done that for jobs yet | 23:22 |
| @clarkb:matrix.org | the attribute is added but not really been updated yet? | 23:22 |
| @jim:acmegating.com | Clark: JobGraph.jobmap is the BuildSet.jobs object, so that's where the mutation happens | 23:22 |
| @clarkb:matrix.org | corvus: right but serialize and deserialize completely ignore it so it appears unused. Its confusing due to that I think | 23:23 |
| @jim:acmegating.com | so this is the old "serialize this object as much as we can, then move on to its child" pattern | 23:24 |
| @jim:acmegating.com | yep, serializing a job is not straightforward, the next few changes set us up for that. | 23:24 |
| @clarkb:matrix.org | right I think what is confusing is that we actually add the attribute | 23:24 |
| @clarkb:matrix.org | rather than adding it when we do the serialize/deserialize work | 23:24 |
| @clarkb:matrix.org | In previous changes we've left things commented out until they get the actionable work | 23:25 |
| @jim:acmegating.com | i think they're all used? | 23:25 |
| @jim:acmegating.com | BuildSet.jobs is commented out in serialize | 23:26 |
| @jim:acmegating.com | but the actual variable is used; it's JobGraph._job_map | 23:26 |
| @clarkb:matrix.org | I think I must be very confused. We add the jobs attribute, we don't deserialize/seralize it. And nothing seems to write to it | 23:27 |
| @clarkb:matrix.org | we read from it | 23:27 |
| @clarkb:matrix.org | but wouldn't it always be an empty dict do dict.get() will return None for those lookups? | 23:27 |
| @jim:acmegating.com | JobGraph.addJob updates BuildSet.jobs | 23:27 |
| @jim:acmegating.com | (and JobGraph.getJobs() reads from it) | 23:28 |
| @clarkb:matrix.org | It updates JobGraph._job_map. Where does that become Buildset.jobs? | 23:28 |
| @jim:acmegating.com | in the initializer; it's passed in when created (see comment on JobGraph.__init__) | 23:29 |
| @jim:acmegating.com | * in the initializer; it's passed in when created (see comment on `JobGraph.__init__`) | 23:29 |
| @clarkb:matrix.org | I might need a break, I just don't see how that happens | 23:30 |
| @clarkb:matrix.org | in `__init__` we set it to an empty list `[]` | 23:30 |
| @clarkb:matrix.org | and in BuildSet init we set it to an empty dict `{}` | 23:30 |
| @jim:acmegating.com | it's `_job_map` not `jobs` on JobGraph | 23:31 |
| @jim:acmegating.com | line 2246 right after the comment | 23:31 |
| @clarkb:matrix.org | ok the comment says jobs | 23:31 |
| @jim:acmegating.com | the word "jobs" in the comment wasn't intended to refer to an attribute name, it was intended to refer to the concept of a job | 23:32 |
| @jim:acmegating.com | i guess we could s/job/Job/ | 23:32 |
| @clarkb:matrix.org | ok, I'm still not seeing where we map Buildset.jobs -> JobGraph._job_map | 23:32 |
| @jim:acmegating.com | line 2900 | 23:33 |
| @clarkb:matrix.org | aha in buildest deserialize | 23:33 |
| @clarkb:matrix.org | I guess I find this setup of having an attribute that is a class and having an attribute that is a reference to something in that lcass to be ocnfusing | 23:34 |
| @clarkb:matrix.org | Why not access everything through the jobgraph? | 23:34 |
| @jim:acmegating.com | yeah, it's not straightforward. it's there because jobgraph isn't an actual zk object. it's just a helper class with all the logic encapsulation of handling job dependencies, etc. | 23:35 |
| @clarkb:matrix.org | Ya I guess in my head we could still serialize that to Zk under the job_graph attribute | 23:35 |
| @jim:acmegating.com | it would fit the new pattern better to put all of that on the buildset object, but that would require a cognitive shift and probably a big change in everything that touched 'job_graph' | 23:36 |
| @clarkb:matrix.org | left a comment about a different comment that seems to be old/unneeded | 23:36 |
| @jim:acmegating.com | so this is a more minimal change | 23:36 |
| @clarkb:matrix.org | I didn't approve it in case you want to fix it in that change but suspect you'll tack that on to the end instead and feel free to approve this change if so | 23:37 |
| @jim:acmegating.com | i'll go ahead and fix it in that change | 23:38 |
| @clarkb:matrix.org | It would be nice if gerrit had a next change in stack button when the stack is too big and you hvae to click show more to see the next change | 23:39 |
| @clarkb:matrix.org | 815752 got bumped out of the gate. I've reapproved it but will let you decide if we should promote it again | 23:45 |
| @clarkb:matrix.org | Oh we don't need to promote because the changes ahead are all failing due to the skopeo install :( | 23:46 |
| @jim:acmegating.com | i'll recheck and promote it | 23:46 |
| @clarkb:matrix.org | corvus: I think we may not need to promote | 23:46 |
| @clarkb:matrix.org | well I guess promotion in this case is for the other changes | 23:46 |
| @clarkb:matrix.org | so still helps | 23:47 |
| @jim:acmegating.com | oh i see you already re-enqueued it? | 23:47 |
| @iwienand:matrix.org | https://github.com/containers/podman/issues/3277 does not fill me with hope | 23:47 |
| @clarkb:matrix.org | I approved it which stuck it in the gate | 23:47 |
| @jim:acmegating.com | cool, i promotedi t | 23:47 |
| @jim:acmegating.com | and yeah, it was a severed head which means it would have eventually become the main head without intervention; promoting it restarted the jobs, but at least all the others are behind it now, so overall slightly more efficient) | 23:48 |
| @clarkb:matrix.org | ianw: ya docker gets this stuff wrong too. Its really annoying and I think one of the reasons that systemd wants to do dns over dbus? | 23:48 |
| @iwienand:matrix.org | at this point even "echo 'nameserver 8.8.8.8' > /etc/resolv.conf" in the container isn't helping | 23:49 |
| @clarkb:matrix.org | ianw: huh that would point at resolv.conf not being the main issue? | 23:50 |
| @clarkb:matrix.org | corvus: I've made it up to https://review.opendev.org/c/zuul/zuul/+/814070/ but think I need to pause there to help with dinner stuff. I'll try to review that one and children in the morning | 23:52 |
| @clarkb:matrix.org | ianw: do we think newer podman did fix this? | 23:52 |
| @clarkb:matrix.org | ianw: otherwise I wonder why it isn't working with the debian version. Local debian patches? Might be worth checking the deb packaging for it? | 23:53 |
| @iwienand:matrix.org | Clark: that, or something about the suse packaging and network setup ... | 23:53 |
| @iwienand:matrix.org | i'm not sure containerfile is turning out to be much more stable than -minimal builds, unfortunately | 23:54 |
| @clarkb:matrix.org | oh right could be suse fixing it on the other side too. Fun | 23:54 |
| @jim:acmegating.com | Clark: thanks! | 23:57 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 23:57 | |
| - [zuul/zuul] 813913: Serialize JobGraph objects to ZK https://review.opendev.org/c/zuul/zuul/+/813913 | ||
| - [zuul/zuul] 814065: Serialize ProjectMetadata on JobGraph https://review.opendev.org/c/zuul/zuul/+/814065 | ||
| - [zuul/zuul] 814071: Add test_freeze_noop_job https://review.opendev.org/c/zuul/zuul/+/814071 | ||
| - [zuul/zuul] 814069: Remove setBase from job freeze API https://review.opendev.org/c/zuul/zuul/+/814069 | ||
| - [zuul/zuul] 814070: Create Abstract and FrozenJob classes https://review.opendev.org/c/zuul/zuul/+/814070 | ||
| - [zuul/zuul] 814242: Make FrozenJob.updateParentData a static method https://review.opendev.org/c/zuul/zuul/+/814242 | ||
| - [zuul/zuul] 814281: Remove toDict from FrozenJob https://review.opendev.org/c/zuul/zuul/+/814281 | ||
| - [zuul/zuul] 814243: Make FrozenJob a ZKObject https://review.opendev.org/c/zuul/zuul/+/814243 | ||
| - [zuul/zuul] 814329: Implement frozen job serialization/deserialization https://review.opendev.org/c/zuul/zuul/+/814329 | ||
| - [zuul/zuul] 814679: Store FrozenJob data in separate znodes https://review.opendev.org/c/zuul/zuul/+/814679 | ||
| - [zuul/zuul] 815154: Update test_inventory to be ZK-friendly https://review.opendev.org/c/zuul/zuul/+/815154 | ||
| - [zuul/zuul] 815565: Remove unecessary assignment in re-enqueue https://review.opendev.org/c/zuul/zuul/+/815565 | ||
| - [zuul/zuul] 815744: Use a metaclass to deserialize event objects https://review.opendev.org/c/zuul/zuul/+/815744 | ||
| - [zuul/zuul] 815764: Add a pipeline change list object to ZK https://review.opendev.org/c/zuul/zuul/+/815764 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: | 23:57 | |
| - [zuul/zuul] 814544: Cleanup stale items after refreshing a pipeline https://review.opendev.org/c/zuul/zuul/+/814544 | ||
| - [zuul/zuul] 814570: Reference active change queues in pipeline state https://review.opendev.org/c/zuul/zuul/+/814570 | ||
| - [zuul/zuul] 814571: Update pipeline state when modifying attributes https://review.opendev.org/c/zuul/zuul/+/814571 | ||
| - [zuul/zuul] 814772: Allow passing extra attributes to ZKObject.fromZK https://review.opendev.org/c/zuul/zuul/+/814772 | ||
| - [zuul/zuul] 814862: Bail out when a project moves between connections https://review.opendev.org/c/zuul/zuul/+/814862 | ||
| - [zuul/zuul] 814773: Move re-enqueue to pipeline processing https://review.opendev.org/c/zuul/zuul/+/814773 | ||
| - [zuul/zuul] 814899: Delete old build sets immediately https://review.opendev.org/c/zuul/zuul/+/814899 | ||
| - [zuul/zuul] 815309: Cancel jobs before resetting builds https://review.opendev.org/c/zuul/zuul/+/815309 | ||
| - [zuul/zuul] 815111: Store builds in Zookeeper https://review.opendev.org/c/zuul/zuul/+/815111 | ||
| - [zuul/zuul] 815276: Add change queues to change queue managers https://review.opendev.org/c/zuul/zuul/+/815276 | ||
| - [zuul/zuul] 815277: Refresh pipelines before checking for empty queues https://review.opendev.org/c/zuul/zuul/+/815277 | ||
| - [zuul/zuul] 815428: Fix GitHub PR (de-)serialization https://review.opendev.org/c/zuul/zuul/+/815428 | ||
| - [zuul/zuul] 815429: Add missing logger to Build and BuildSet classes https://review.opendev.org/c/zuul/zuul/+/815429 | ||
| - [zuul/zuul] 815450: Create bundle items during queue deserialization https://review.opendev.org/c/zuul/zuul/+/815450 | ||
| - [zuul/zuul] 815495: Fix Gerrit change (de-)serialization https://review.opendev.org/c/zuul/zuul/+/815495 | ||
| - [zuul/zuul] 815616: Only reset the pipeline state if needed https://review.opendev.org/c/zuul/zuul/+/815616 | ||
| - [zuul/zuul] 815617: Ensure same layout UUID across schedulers https://review.opendev.org/c/zuul/zuul/+/815617 | ||
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Simon Westphahl: [zuul/zuul] 815278: DNM: execute tests with two schedulers https://review.opendev.org/c/zuul/zuul/+/815278 | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!