Tuesday, 2026-06-30

« Monday, 2026-06-29 Index Wednesday, 2026-07-01 »
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [openstack/diskimage-builder] 995426: Replace uses of sudo -E with calls to sudo.ws https://review.opendev.org/c/openstack/diskimage-builder/+/99542600:20
@mordred:waterwanders.comfungi, Clark ^^ there's a quick stab00:21
@abelur:matrix.orgfungi: Clark  The Matrix spec lists a "meetbot replacement" as TBD. I've got a working POC - that reuses opendev/meetbot's engine unchanged (byte-identical minutes), thin matrix-nio transport, plus a gate that spins up a throwaway Synapse to replay a meeting.  A few quick questions before I write anything up formally:02:39
1. Is there appetite to land a native Matrix meetbot now, or is it still parked behind statusbot/eavesdrop on the Matrix roadmap?
2. Repo placement: would you prefer a Matrix transport inside opendev/meetbot (shared engine, one codebase) or a new repo like the Matrix gerritbot?
3. Is there a preferred Matrix client library I should standardize on for consistency with gerritbot/the other bots, or is matrix-nio (Python) fine?
4. For deployment - is running it as a container on eavesdrop02 via system-config (compose, like gerritbot) the right shape, or is there a different home you'd want?
5. What's the homeserver client URL the bot should connect to? opendev.org delegates via .well-known, so I assume not opendev.org itself - matrix.opendev.org? something Element-hosted (EMS)?
6. Is a @meetbot:opendev.org bot account acceptable, and who provisions it / generates the access token?
7. Should the container build + publish via a Zuul image job (like gerritbot), and to which registry/name?
8. For minutes: should output land on the existing meetings.opendev.org volume so Matrix meetings sit beside the IRC ones, same format/path?
@fungicide:matrix.orgAnil Belur: 1. now is probably fine, the rest of the bots are working with matrix now, i think12:20
@fungicide:matrix.org2. is up for debate but there's a third option (my preference), to start with something lightweight like https://opendev.org/opendev/system-config/src/branch/master/docker/matrix-eavesdrop12:21
@fungicide:matrix.org3. we used matrix-nio for the matrix-eavesdrop example above, i think that's fine12:22
@fungicide:matrix.org4. yes i think that's the best plan still12:22
@fungicide:matrix.org5. `https://opendev.ems.host` per https://opendev.org/opendev/system-config/src/commit/543a185/playbooks/roles/matrix-eavesdrop/templates/config.yaml#L312:24
@fungicide:matrix.org6. that's probably fine, or even just `@meet` or something similarly short, i can create the account when it's closer to time to try deploying12:25
@fungicide:matrix.org7. yes, to something like `quay.io/opendevorg/meetbot` similar to our others12:27
@fungicide:matrix.org8. yes definitely12:27
@jim:acmegating.comfungi: Clark https://review.opendev.org/993695 is a housekeeping change we should not forget14:13
-@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 995203: Redirect skopeo container cache to zuul_user_dir https://review.opendev.org/c/zuul/zuul-jobs/+/99520314:36
@clarkb:matrix.orgfungi: Anil Belur yes for 1) I think meetbot is the only functionality that is missing. I think matrix-nio is fine as matrix-eavesdrop uses it. One thing that we might consider is putting the functionality in matrix-eavesdrop so that we can reduce the number of bot accounts we need to manage (and image builds etc)14:36
@clarkb:matrix.orgcorvus: ^ do you have concerns with combining the functionality like that? I know zuul uses eavesdrop for logging and also doesn't intend to do meetings so maybe there is a reason to not do that?14:37
@jim:acmegating.comClark fungi Anil Belur yes, in our earlier discussions i was also imagining that it would be added to matrix-eavesdrop since the functionality is so similar and related.  so just extending that existing program to support meetings sounds good.  that also takes care of several of the other questions (library, account, server, operation, etc).14:41
@clarkb:matrix.orggreat in that case I think the thing that makes the most sense to me is to port the work you've already done into matrix-eavesdrop here: https://opendev.org/opendev/system-config/src/branch/master/docker/matrix-eavesdrop/src which should hopefully be straightforward as both use matrix-nio and then let that handle container image builds etc14:49
@clarkb:matrix.org* great in that case I think the thing that makes the most sense to me is to port the work you've already done into matrix-eavesdrop here: https://opendev.org/opendev/system-config/src/branch/master/docker/matrix-eavesdrop/src which should hopefully be straightforward as both use matrix-nio and then let that handle container image builds etc cc Anil Belur 14:49
@clarkb:matrix.orgwiki seems happier this morning so I've syned up the agenda page with the email I sent yesterday14:54
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/base-jobs] 993695: Update base-minimal post-logs https://review.opendev.org/c/opendev/base-jobs/+/99369514:54
@clarkb:matrix.orgJens Harbott: ^ I noted on the change that I Think we can fix up the commented out providers that are enabled now the next time we do a provider enable/disable for an outage. Or we can fix it in a followup14:55
@clarkb:matrix.orgbut I figured the change as is was an improvement over what we were doing before so went ahead and approved it14:55
@mordred:waterwanders.comcorvus, Clark speaking of base-jobs changes, next time you're in the mood, https://review.opendev.org/c/opendev/base-jobs/+/994539 is the fix for the honor_container_command change we tried landing last week. 16:50
@mordred:waterwanders.comcorvus: Also, use-buildset-registry is still keeping a ~/.docker/config.json around triggering a preserve-home-paths. Should we fix that in base-jobs rather than zuul-jobs? (I'm honestly not sure how we'd fix it in zuul-jobs, this isn't a secrets file we're shredding)16:51
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 995544: Add .docker/config.json to preserve-home-paths https://review.opendev.org/c/opendev/base-jobs/+/99554416:53
@mordred:waterwanders.comcorvus: like that ^^ ?16:53
@jim:acmegating.commordred: we could think about whether we should be doing that on the executor...16:55
@jim:acmegating.commordred: do you have a simple failure i can look at?16:56
@jim:acmegating.comi mean "job with warning" not real failure of course16:56
@mordred:waterwanders.comcorvus: https://review.opendev.org/c/drizzle/drizzle/+/99542316:56
@clarkb:matrix.orgthat is a good point. There is a good chance we don't need those creds there unless we're using the executor to shuffle the data around?16:56
@mordred:waterwanders.comthe "funny" part is that in that particular change I don't actually need docker at all, but given the honor_container_command patch isnt' in yet, it's helpfully surfacing the docker path :)16:58
@jim:acmegating.comhrm, i feel like we probably should have included the job name in that warning....17:00
@jim:acmegating.combecause i'm not 100% sure which job is actually causing those... do you know?17:00
@mordred:waterwanders.comI was actually going to say the same thing. I'd guess it's the build-image job - but then I would have expected our intermediate registry fix to have fixed that17:03
@mordred:waterwanders.comthe others are using opendev-buildset-registry-consumer17:04
@mordred:waterwanders.comand they're all calling "use-buildset-registry" explcitly in a pre-run17:05
@jim:acmegating.comthat *should* only be running on the remote node (unless there's a 'delegate' on them?)17:07
@jim:acmegating.comi still feel like i'm missing something17:07
@mordred:waterwanders.comoh! I think it's going to be the image build job17:07
@mordred:waterwanders.combecause of push to intermediate registry17:07
@mordred:waterwanders.comwhich we fix here: https://review.opendev.org/c/zuul/zuul-jobs/+/99534617:07
@jim:acmegating.comyeah -- it should be only *one* job since we only see the warnings once17:08
@mordred:waterwanders.comoh, or, rather the buildset-registry job17:08
@mordred:waterwanders.comso - my money is on opendev-buildset-registry being the job, and the file persisting from push-to-intermediate-registry17:09
@jim:acmegating.comi'll buy that.  i think we should merge that now so i'll upgrade to a +317:09
@mordred:waterwanders.com++17:09
@mordred:waterwanders.comI will abandon my preserve-home-paths patch17:09
@jim:acmegating.commordred: meanwhile https://review.opendev.org/995547 should help with diagnosis17:09
@mordred:waterwanders.comYES17:10
@jim:acmegating.commordred: +3 on 994539 i think we'll be around to deal with fallout17:10
@mordred:waterwanders.comthanks! I'll go look for an easy patch to recheck when it lands17:11
@mordred:waterwanders.comok - I have a small easy patch I can recheck17:13
@mordred:waterwanders.comhttps://review.opendev.org/c/wandertracks/wandertracks/+/995549 (also the patch itself removes the  local podman workarounds, so should be a good trial ballon either way)17:14
-@gerrit:opendev.org- Zuul merged on behalf of Dmitriy Rabotyagov: [openstack/project-config] 982878: Deprecate Vitrage project https://review.opendev.org/c/openstack/project-config/+/98287817:18
-@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/base-jobs] 994539: Reapply "Honor container_command in buildset-registry playbooks" https://review.opendev.org/c/opendev/base-jobs/+/99453917:19
@mordred:waterwanders.comok - rechecked - watching the container job17:33
-@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [zuul/zuul-jobs] 995346: Update uses of shred to actually remove the files https://review.opendev.org/c/zuul/zuul-jobs/+/99534617:37
@clarkb:matrix.orgstill seeing gitea slowness impacting manage-projects. Its not the end of the world right now, but something to be aware of and keep an eye on17:41
@mordred:waterwanders.comuhoh. my image build job is on 2nd attempt. 17:51
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 995552: Revert "Reapply "Honor container_command in buildset-registry playbooks"" https://review.opendev.org/c/opendev/base-jobs/+/99555217:52
@mordred:waterwanders.comI didn't catch the live console logs this time around, but I'm pretty sure we're going to need that ^^ :( 17:53
@mordred:waterwanders.comok - caught the live console17:54
@mordred:waterwanders.comhttps://zuul.opendev.org/t/opendev/stream/a2553eccb4114ead8c111ab3ff3bf2ca?logfile=console.log17:54
@clarkb:matrix.org+2 from me once we confirm it is needed I can approve17:54
@mordred:waterwanders.comyeah. go ahead and do it17:56
@mordred:waterwanders.com```17:56
2026-06-30 17:55:53.210102 | TASK [pull-from-intermediate-registry : Load information from zuul_return]
2026-06-30 17:55:53.302536 | localhost | Output suppressed because no_log was given
failure
2026-06-30 17:55:53.303962 |
```
@clarkb:matrix.org```17:56
2026-06-30 17:55:53.210102 | TASK [pull-from-intermediate-registry : Load information from zuul_return]
2026-06-30 17:55:53.302536 | localhost | Output suppressed because no_log was given
failure
```
@mordred:waterwanders.comso - that'll probably take more debugging17:57
@clarkb:matrix.orgbut that is happening in pre run right?17:57
@mordred:waterwanders.comyeah17:57
@clarkb:matrix.orgyour chagne is only changing post run?17:57
@clarkb:matrix.orgoh you affect pre.yaml too sorry my brain is in a million places today17:57
@clarkb:matrix.orgI've approved it. Probably some weird side effect from the pre.yaml changes17:58
@mordred:waterwanders.commy change consumes the opendev-build-image jobs with container_command: podman ... in theory this should work - I'm doing use ...17:59
@mordred:waterwanders.comOH17:59
@mordred:waterwanders.comoh17:59
@mordred:waterwanders.comI totally get it17:59
@mordred:waterwanders.comoh, nevermind.18:00
-@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/base-jobs] 995552: Revert "Reapply "Honor container_command in buildset-registry playbooks"" https://review.opendev.org/c/opendev/base-jobs/+/99555218:03
@clarkb:matrix.orgAirship says they will start looking at Zuul config errors now18:25
@mordred:waterwanders.comClark, corvus : 18:46
```
2026-06-30 17:55:05.096852 | TASK [Run buildset registry (if not already running)]
2026-06-30 17:55:05.141674 | ubuntu-noble | ok
2026-06-30 17:55:05.169201 | ubuntu-noble | included: run-buildset-registry
2026-06-30 17:55:05.183790 |
```
Looking back further at that failed job run from earlier - we didn't detect the running buildset registry even earlier in the playbook. So for some reason I don't yet understand, we're not detecting the previously zuul_returned buildset registry info
@mordred:waterwanders.comI cannot see anything in the patch that even touches that.18:51
@clarkb:matrix.orgwas it just a coincidence maybe?18:52
@clarkb:matrix.orgsomethign else caused the lookup failure potentially18:52
@mordred:waterwanders.comyeah - maybe? I'm at a loss. (I'm still trying to poke, but I'm running out of places to look)18:54
@mordred:waterwanders.comhttps://zuul.opendev.org/t/opendev/build/1d7ee404e8464298b5717dd32c88a405/log/job-output.txt#333 <-- another container build job during the new patch timeframe in a different repo also decided to try to run the buildset registry - so not grokking the zuul_return (it ran twice, failed both times) - that job is now correctly working. so SOMETHING about that patch is breaking the zuul_return processing and skipping of running a registry on host19:21
@clarkb:matrix.orgweird19:22
@mordred:waterwanders.comyeah. I _cannot_ fathom anything in that patch that would have that impact19:22
@clarkb:matrix.orgmordred: is it `Error: could not find pasta, the network namespace can't be configured: exec: "pasta": executable file not found in $PATH`19:24
@clarkb:matrix.orgmaybe the podman installation is insufficient19:24
@mordred:waterwanders.comthat would be a different bug - an actual  issue with running a buildset registry on podman19:24
@clarkb:matrix.orgya I guess that failure doesn't match what we saw in the other failure19:25
@mordred:waterwanders.comlike - that's a real break, but that job should never have gotten to that line19:25
@mordred:waterwanders.comthey both fail in that neither one should have attempted to run a buildset registry in the first place19:25
@clarkb:matrix.orgwell it says when buildset registry is not defined then run a buildset registry19:28
@clarkb:matrix.orgso it seems ok that it ran that?19:28
@mordred:waterwanders.comit shouldn't - because there is an opendev-buildset-registry job in that project pipeline that this job depends on. In other runs it will load the existing buildset registry information that was zuul_return'd by opendev-buildset-registry and skip running it19:29
@clarkb:matrix.orggot it19:30
@mordred:waterwanders.comso runing it is a symptom of "something broke processing the returned data"19:30
@mordred:waterwanders.com(which we then saw in the fetch-from-intermediate failure too)19:30
@clarkb:matrix.orgI wonder is ensure-docker loading that info19:32
@mordred:waterwanders.comincidentally - the wandertracks change successfully ran the buildset registry, but it's running on default nodes (noble I think) whereas the drizzle change, which runs on resolute nodes, failed on the pasta thing. so _Second_ bug is that ensure-podman on resolute is missing a package even in the non-rootless case19:32
@clarkb:matrix.orgmordred: I think that you have to explicitly load it in a role/playbook. If ensure-docker was doing that and we aren't in ensure-podman then maybe that is the cause19:32
@mordred:waterwanders.comlooking19:33
@clarkb:matrix.orgpush/pull-to/from-intermediate-registry do that loading19:33
@clarkb:matrix.orghttps://codesearch.opendev.org/?q=result_data_file&i=nope&literal=nope&files=&excludeFiles=&repos=19:34
@clarkb:matrix.orgso maybe this is an order of operations thing. I wonder if we pull-from-intermediate-registry earlier in other jobs and this is just an interaction around that in your jobs?19:35
@mordred:waterwanders.comno - it works properly without the patch - same order.(and I'm just running opendev-build-container-image really) ...but, we reutrn it in the opendev-buildset-registry job with:19:38
```
- name: Return registry information to Zuul
zuul_return:
secret_data:
buildset_registry: "{{ buildset_registry }}"
```
@clarkb:matrix.orgmordred: it looks like the successful run is running a buildset registry with docker. This is because nothing is defining that buildset registry value before we do the checks in the chagne we've reverted/reapplied19:43
@clarkb:matrix.orgmordred: so maybe the first thing to address is that?19:43
@clarkb:matrix.organd figure out what if any errors exist from there19:43
@mordred:waterwanders.comClark: it does this: https://zuul.opendev.org/t/opendev/build/d988388430c5435d9edd2e4c2ea20694/log/job-output.txt#459 in a successful run19:54
@mordred:waterwanders.com(that's a recent run after our revert - but same behavior before the revert)19:55
@clarkb:matrix.orgthats weird because it seems like we don't load the info until https://zuul.opendev.org/t/opendev/build/d988388430c5435d9edd2e4c2ea20694/log/job-output.txt#68519:58
@mordred:waterwanders.comI agree. 20:00
@mordred:waterwanders.comClark: there's a note in front of one instance of that that says: "# This can be removed if we add this functionality to Zuul directly"20:00
@mordred:waterwanders.comClark: also - fwiw, that load is what failed on the original failures we both  posted in20:04
@clarkb:matrix.orgyes I'm half operating under the assumption that that load is maybe a one off or some state problem and not a fundamental issue in the jobs20:05
@mordred:waterwanders.comso, I'd say that construct failing is the thing that was broken in all the failure cases - although it manifested slightly differently in a couple of them20:05
@clarkb:matrix.organd then the podman can't run buildset registry showing that buildset_registry was expected to already be loaded is the actual bug20:05
@clarkb:matrix.orgbut I don't understand how buildset_registry is ever set that early at all20:05
@mordred:waterwanders.comthing is - podman _can_ run buildset registry (and did successfully) on noble. that was the original wandertracks job20:06
@clarkb:matrix.orgit isn't set in the inventory20:06
@mordred:waterwanders.comno, it isn't - because it has secret info in it20:06
@mordred:waterwanders.com(it has the buildset registry password)20:07
@clarkb:matrix.orgya so I would expect it to always run the buildset registry locally, but your example clearly doesn't so now I'm digging through logs and ansible to figure out why20:07
@mordred:waterwanders.comso we showed 2 bugs:20:07
1. whatever is going on with zuul_return and buildset_registry data not being able to be read by pull from intermediate registry
@mordred:waterwanders.com2. podman + buildset registry is broken on resolute (I'll fix that independently)20:08
@mordred:waterwanders.comFor tracking purposes:20:11
- Failing job 1: https://zuul.opendev.org/t/opendev/build/5f50c032dacf4c90b637d604b5325370 (on resolute, will show as podman error)
- Successful recheck of failing job 1: https://zuul.opendev.org/t/opendev/build/d988388430c5435d9edd2e4c2ea20694
- Failing job 2: https://zuul.opendev.org/t/opendev/build/62891e2ddd034690b3e274ee16c2bc1f (on noble, will show at pull from intermediate)
- Successful recheck of failing job 2: https://zuul.opendev.org/t/opendev/build/d2c526e423cd4f82b18bf33b87d0addf
@clarkb:matrix.orgI'm basically reading https://zuul.opendev.org/t/opendev/build/d988388430c5435d9edd2e4c2ea20694/console and working backward from the skipped run-buildset-registry role inclusion20:12
@clarkb:matrix.orgsince at that point we should have buildset_registry defined, btu I'm not seeing where20:12
@clarkb:matrix.orgmyhunch is in ensure-docker somewhere20:12
@clarkb:matrix.orgsince that would explain why podman tries to run it20:13
@clarkb:matrix.orgbut I really don't see it in there20:13
@mordred:waterwanders.comClark: 20:14
```
def getResultData(self):
data = {}
secret_data = {}
try:
with _safe_open(self.jobdir.result_data_file) as f:
file_data = f.read()
if file_data:
file_data = json.loads(file_data)
data = file_data.get('data', {})
secret_data = file_data.get('secret_data', {})
# Check the variable names for safety, but zuul is allowed.
data_copy = data.copy()
data_copy.pop('zuul', None)
check_varnames(data_copy)
secret_data_copy = data.copy()
secret_data_copy.pop('zuul', None)
check_varnames(secret_data_copy)
```
@mordred:waterwanders.comthat's in zuul now, so it _should_ be being injected into secret data for the job20:14
@mordred:waterwanders.comsorr, there's a ```return data, secret_data``` at the end of that20:14
@clarkb:matrix.orgohhh that would explain why the docker based run is skipped I guess20:14
@mordred:waterwanders.comit would?20:15
@clarkb:matrix.orgit doesn't explain why the podman based run does not skip. I checked the diffs and the var doesn't get renamed or anything liek that so I don't think its a typo mismatching the values20:15
@mordred:waterwanders.comah - yes. explains why it works when it works20:15
@mordred:waterwanders.comdoes NOT explain why it's broken where it's broken :) 20:15
@mordred:waterwanders.comClark: I've tried to bring a fun weird bug for our debugging pleasure20:16
@clarkb:matrix.orgmordred: hrm that code you linked is used around pausing to load the info out of the job context for zuul to use20:16
@clarkb:matrix.orgwe aren't doing a job pause in the successful runs as far as I can tell so maybe it doesn't explain that20:16
@mordred:waterwanders.comwe are though20:16
@mordred:waterwanders.comthe opendev-buildset-registry _job_ pauses20:16
@clarkb:matrix.orgoh are we?20:16
@mordred:waterwanders.comyeah20:17
@clarkb:matrix.orgright but that is a different job20:17
@clarkb:matrix.orgin this particular job we would need to load the result data somewhere to know about the paused buildset registry running in the other paused job20:17
@clarkb:matrix.organd its that load that I'm not finding (but seems to exist when using docker)20:17
@mordred:waterwanders.comhttps://zuul.opendev.org/t/opendev/buildset/4b2e54d4a95544bb9319f0f40eef18fd20:17
@mordred:waterwanders.comI'm _pretty_ sure zuul auto-injects it these days20:18
@mordred:waterwanders.comcorvus: ^^ ?20:18
@clarkb:matrix.orgmordred: I think the next debugging step may be to add debugging around whether or not buildset_registry is defined and what the contents are (can we safely print bits of info like the ip address without exposing the password? but even then exposing the password is probably fine if it is an ephemeral registry and we don't rely on it in jobs toat actually publish images)20:21
@mordred:waterwanders.com* For tracking purposes:20:27
- Failing job 1: https://zuul.opendev.org/t/opendev/build/1d7ee404e8464298b5717dd32c88a405 (on resolute, will show as podman error)
- Successful recheck of failing job 1: https://zuul.opendev.org/t/opendev/build/578a4ea230534329976da07c42f486a0
- Failing job 2: https://zuul.opendev.org/t/opendev/build/62891e2ddd034690b3e274ee16c2bc1f (on noble, will show at pull from intermediate)
- Successful recheck of failing job 2: https://zuul.opendev.org/t/opendev/build/d2c526e423cd4f82b18bf33b87d0addf
@jim:acmegating.comi think the question is: if opendev-buildset-registry returns secret data, will it be present for dependent jobs?  and my answer is yes it should be20:31
@clarkb:matrix.orgcorvus: but is it present as an ansible var without an explicit load of the data? Some roles like pull-from-intermediate-registry load it explicitly20:32
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [zuul/zuul-jobs] 995585: Align podman packages on resolute and forward https://review.opendev.org/c/zuul/zuul-jobs/+/99558520:32
@jim:acmegating.comClark: i expect the explicit load to be to handle the case where we want to use it in the same job20:33
@clarkb:matrix.orgcorvus: ah got it20:33
@mordred:waterwanders.comnod. so - the expectation is that with opendev-buildset-registry paused, we should see buildset_registry set as a varaible in other jobs in that buildset20:33
@jim:acmegating.comyep20:33
@clarkb:matrix.orgok so something about the change is causing that info to not be present to ansible20:33
@mordred:waterwanders.comnow - why that got broken by that change is ... yeah20:34
@mordred:waterwanders.coma _very_ mystery20:34
@mordred:waterwanders.com(that zuul-jobs patch above is a fix for the incidental "podman+registry+resolute" bug we accidentally uncovered which is not really related)20:34
@clarkb:matrix.orgit occurs to me that ensure-podman could unset the var, but I don't see evidence ofthat either20:38
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 995586: Reapply "Reapply "Honor container_command in buildset-registry playbooks"" https://review.opendev.org/c/opendev/base-jobs/+/99558620:42
@mordred:waterwanders.comClark: I've got nothing - but there is an updated version of the patch, but I reverted out the block and the iptables changes to make it as small as humanly possible20:43
@clarkb:matrix.org+2 from me seems like reducing the possible scope and trying again may give us more info and might get things working as expected20:45
@mordred:waterwanders.comI wish I had even a theory20:45
@mordred:waterwanders.comdo you want me to put in a debug print of one of the non-secret fields in buildset_registry? Like buildset_registry.username?20:47
@clarkb:matrix.orgmaybe just a buildset_registry is defined check at the very beginning of that?20:48
@mordred:waterwanders.com++20:48
@clarkb:matrix.orgI dunno I don't think it is necessary without a theory your simplification chagne may be just as good20:48
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 995586: Reapply "Reapply "Honor container_command in buildset-registry playbooks"" https://review.opendev.org/c/opendev/base-jobs/+/99558620:51
@mordred:waterwanders.comoh, piddle. sorry - didn't see we already had the +A :(20:51
@jim:acmegating.comi think it's the block20:52
@jim:acmegating.comit must be something about the interaction of block and include_role, like it's applying the "when" to every task in the role20:53
@jim:acmegating.comhttps://zuul.opendev.org/t/opendev/build/f092094b4b774e44b95e4411e326681b/console#1/0/60/ubuntu-noble20:53
@jim:acmegating.comthe zuul_return task was skipped in the opendev-buildset-registry job in the failing buildset20:54
@jim:acmegating.comi'm guessing that's because the "when buildset_registry not defined" block condition got propagated all the way down to the individual task in the role20:54
@clarkb:matrix.orgin that case the latest version should be happier as it drops the block20:54
@mordred:waterwanders.comoh wild20:55
@clarkb:matrix.orgdo we want to go back to the non debug version of that and reapprove or keep the debugging anda pprove?20:55
@jim:acmegating.comyeah should be a good indication that's the right track20:55
@mordred:waterwanders.comyeah. let me strip that back out20:55
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 995586: Reapply "Reapply "Honor container_command in buildset-registry playbooks"" https://review.opendev.org/c/opendev/base-jobs/+/99558620:55
@clarkb:matrix.org+2 from me on ps320:56
@jim:acmegating.com+320:56
@mordred:waterwanders.comso - just saying this back so my incredulous mind can grok it. our theory is that the block propagated the when to every line inside of the role, so in the actual opendev-buildset-registry job, where that variable is not set and we expect it to run, it ran the role but added a set of "when: not set" to each task - which then caused it to not zuul_return the data20:59
@jim:acmegating.comyes, or something like that, i haven't tested/verified/researched that exact thing, but that's my starting theory.  i know something like that happens with include_tasks or similar, i am (or will be) surprised that it's true for include_role too21:00
@mordred:waterwanders.com"When using when on a block that contains include_role , the when condition is evaluated before every single task inside the included role . This often causes early skips. The best practice is to place the when condition on the include_role task itself to evaluate the role as a whole."21:00
@jim:acmegating.comseems like the best practice may be "never use block"21:01
@jim:acmegating.com(if you can help it)21:01
@clarkb:matrix.orgmordred: is that chatgpt explaining the situation?21:02
@mordred:waterwanders.comClark: gemini :)21:03
@mordred:waterwanders.combut here's the link to the ansible docs on it: https://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_blocks.html21:04
-@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/base-jobs] 995586: Reapply "Reapply "Honor container_command in buildset-registry playbooks"" https://review.opendev.org/c/opendev/base-jobs/+/99558621:04
@mordred:waterwanders.com"All tasks in a block inherit directives applied at the block level. Most of what you can apply to a single task (with the exception of loops) can be applied at the block level, so blocks make it much easier to set data or directives common to the tasks. The directive does not affect the block itself, it is only inherited by the tasks enclosed by a block. For example, a when statement is applied to the tasks within a block, not to the block itself."21:04
@mordred:waterwanders.comI've rechecked https://review.opendev.org/c/wandertracks/wandertracks/+/995549 and https://review.opendev.org/c/drizzle/drizzle/+/995540 which are our two fail/success cases from earlier21:05
@jim:acmegating.comthat excerpt doesn't seem to say that it would then recursively apply to the tasks in the included role21:07
@mordred:waterwanders.comah - here's the more relevant link: https://github.com/ansible/ansible/issues/7438221:08
@mordred:waterwanders.comhttps://github.com/ansible/ansible/issues/74382#issuecomment-82490112121:08
@jim:acmegating.comthat's more clear; looks like they added docs for that to the "playbook guide"21:11
@mordred:waterwanders.comhttps://zuul.opendev.org/t/opendev/stream/b0c5aaf42571417eb2bf38bfdbc97fec?logfile=console.log 21:11
@mordred:waterwanders.comthere's the stream of one of the test changes for anyone who wants to follow along21:11
@jim:acmegating.comhttps://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_blocks.html#grouping-tasks-with-blocks21:12
@jim:acmegating.comthere it is in docs!21:12
@jim:acmegating.comsorry the anchor isn't good there in my link21:12
@jim:acmegating.comi'm trying to point to the "Note" a bit below that21:12
@jim:acmegating.com`All tasks in a block, including the ones included through include_role, inherit directives applied at the block level.`21:13
@clarkb:matrix.orgI'm not sure I like that21:13
@jim:acmegating.comi'm sure i don't21:13
@clarkb:matrix.orgbut I guess it is documented and we can be careful about it in the future21:13
@mordred:waterwanders.comwe have gotten past previous failure points21:13
@jim:acmegating.comwe've gotten very similar things wrong in the past, we'll mess it up again21:13
@mordred:waterwanders.comit's SO counterintuitive21:14
@mordred:waterwanders.comas a human what I clearly mean is "if this is not set, include this role"21:14
@mordred:waterwanders.comthe github issue used the argument that you'd expect this behavior for become, which I agree with. If only there was a way for a program to say "these keys are applied transitively and these keys aren't"21:15
@mordred:waterwanders.com(please read that with maximum snark)21:16
@clarkb:matrix.orgya particularlyfor a when condition21:16
@mordred:waterwanders.comId' say when+include_role is common enough that special-casing it in code would be entirely justified21:16
@mordred:waterwanders.combut, hey, what do I know21:17
-@gerrit:opendev.org- Monty Taylor https://matrix.to/#/@mordred:inaugust.com proposed: [opendev/base-jobs] 995591: Only open iptables ports when running a buildset registry https://review.opendev.org/c/opendev/base-jobs/+/99559121:21
@mordred:waterwanders.comhttps://zuul.opendev.org/t/opendev/stream/50f4ccb3218745bd93ef3e0fa37baa7d?logfile=console.log <-- otherjob also past the point. looking good overall21:22
@mordred:waterwanders.comimage build jobs succeeded \o/21:29
@abelur:matrix.orgThanks - that makes sense, and it's actually less work. I'll port the meeting handling into matrix-eavesdrop rather than stand up a separate bot, reusing the existing image/account/role. The POC already uses matrix-nio, so it's mostly lifting the command parsing + the OpenDev minute-rendering engine into bot.py and adding a reply path (eavesdrop currently only logs, never sends).21:41
@abelur:matrix.orgClark: fungi  corvus Two minor things before I push a change up:21:43
1. Engine reuse — the OpenDev minute/log formatting comes from the ircmeeting engine in opendev/meetbot, which isn't on PyPI. Easiest for a first pass is to vendor it into the eavesdrop src/ package. Any objection to vendoring, or would you prefer I pull it in as a build-time dependency from opendev/meetbot?
@abelur:matrix.org* Clark: fungi  corvus Two minor things before I push a change up:21:43
1. Engine reuse - the OpenDev minute/log formatting comes from the ircmeeting engine in opendev/meetbot, which isn't on PyPI. Easiest for a first pass is to vendor it into the eavesdrop src/ package. Any objection to vendoring, or would you prefer I pull it in as a build-time dependency from opendev/meetbot?
@abelur:matrix.org2.  Account UX - happy to reuse @logs:opendev.org to keep account count down. Only thing is it'll be the sender of meeting replies ("Meeting started", minutes links), which reads a little oddly under "logs". Fine as-is, or would you prefer a friendlier display name on the same account?21:43
@abelur:matrix.orgI'll target a single change against system-config under topic matrix. 21:44
Will test locally against a throwaway Synapse first (that's how the POC gate works today).
@jim:acmegating.comAnil Belur: I'm going to reply in the main room instead of the thread -- we generally try to avoid threads in these public rooms because for a lot of matrix clients (including, weirdly, element) they can be hard to discover and follow.  (non-threaded replies work pretty well though)21:55
To your questions: 1) vendor is fine with me, but i don't feel strongly about that; happy to yield to clark or fungi if they have stronger opinions.
2) agree that logs as a display name is weird... i could live with it (it's still... a log), but if we can find something better that would be great. that is of course the hardest problem in computer science, so... good luck to us all in picking a name.
@jim:acmegating.com * Anil Belur: I'm going to reply in the main room instead of the thread -- we generally try to avoid threads in these public rooms because for a lot of matrix clients (including, weirdly, element) they can be hard to discover and follow.  (non-threaded replies work pretty well though)21:55
To your questions:
1) vendor is fine with me, but i don't feel strongly about that; happy to yield to clark or fungi if they have stronger opinions.
2. agree that logs as a display name is weird... i could live with it (it's still... a log), but if we can find something better that would be great. that is of course the hardest problem in computer science, so... good luck to us all in picking a name.
@clarkb:matrix.orgvendoring/forking that ircmeeting stuff seems fine. For 2. do we think we want to change the display name of that account and share the account? it will still be `@logs:opendev.org` but we can have it display as meetbot or something?22:10
@clarkb:matrix.orgon the irc side the nick is `opendevmeet` so maybe we change the display name to something like that?22:11
@clarkb:matrix.organd yes element makes threads really hard to discover, I had no idea a thread had been started22:11
@jim:acmegating.com`opendevmeet` sounds good, especially with the web hostname being 'meetings' these days.  `OpenDevMeet`  `OpenDev Meeting`  `meet` would all work for me22:14
@jim:acmegating.combut then i'd be okay with `meat` so yeah don't listen to me either22:15
@mordred:waterwanders.commmm meat22:17
@clarkb:matrix.orgour local butcher is "The Meating Place"22:18
@clarkb:matrix.orgI'm popping out for a bike ride22:33
@mordred:waterwanders.comcorvus: so - funny story. image builds on podman with container_command set to podman ... and then fails on push to buidset registry23:02
@jim:acmegating.comcool what will it be this time?  ipv6?  docker/podman protocol mismatch?  ansible weird?  i can't wait to find out23:10
@mordred:waterwanders.comcorvus: leading candidate is "podman does chunked PATCH uploads concurrently, then ignores one of them for no apparent reason until the registry times out" https://zuul.opendev.org/t/opendev/build/d99ec2106c85465db2d867f76ac48c42/log/podman/buildset_registry.txt#1004 23:17
@jim:acmegating.comso door #2 23:19
@mordred:waterwanders.comyah23:20
@mordred:waterwanders.comI wonder if skopeo would do any better23:20
@mordred:waterwanders.com(I'm going to try to come up with a reproducer that doesn't involve an end to end zuul job :) )23:23
« Monday, 2026-06-29 Index Wednesday, 2026-07-01 »

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!