Tuesday, 2025-04-08

« Monday, 2025-04-07 Index Wednesday, 2025-04-09 »
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 942532: Protect connection event forwarder with timeout https://review.opendev.org/c/zuul/zuul/+/94253206:00
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 946240: Always clear wake event of TPE event connectors https://review.opendev.org/c/zuul/zuul/+/94624007:18
-@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 946613: test check run https://review.opendev.org/c/zuul/zuul/+/94661310:37
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 946619: Allow boto3 session to configure itself https://review.opendev.org/c/zuul/zuul/+/94661911:48
-@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul] 946629: Store resources of pod type nodes in host inventory https://review.opendev.org/c/zuul/zuul/+/94662913:48
-@gerrit:opendev.org- Luca Milanesio proposed: [zuul/zuul] 946632: Always include project name in change key https://review.opendev.org/c/zuul/zuul/+/94663214:00
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 946619: Allow boto3 session to configure itself https://review.opendev.org/c/zuul/zuul/+/94661914:04
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 946619: Allow boto3 session to configure itself https://review.opendev.org/c/zuul/zuul/+/94661914:07
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 946595: Omit succssful jobs on very long gerrit reports https://review.opendev.org/c/zuul/zuul/+/94659514:52
@clarkb:matrix.orgzuulians https://gerrit-review.googlesource.com/c/homepage/+/464287 is something we should consider from a zuul perspective. I think this may impact zuul's gerrit change id depends-on handling16:31
@clarkb:matrix.orgbut there may be other implications I haven't considered yet16:31
@clarkb:matrix.orgis there any reason to not approve https://review.opendev.org/c/zuul/zuul-jobs/+/941490/ at this point?16:56
@clarkb:matrix.orgit seems well tested so I don't expect problems but the impact might be broad so figured I'd ask before I do so16:56
@jim:acmegating.comClark: +3 from you sgtm16:57
@clarkb:matrix.orgdone17:04
-@gerrit:opendev.org- Zuul merged on behalf of Aurelio Jargas: [zuul/zuul-jobs] 941490: Add role: `ensure-python-command`, refactor similar roles https://review.opendev.org/c/zuul/zuul-jobs/+/94149017:38
@f2ked:matrix.orgdumb question.. when I start the scheduler a bunch of MergeRequests are created.. what do they represet?17:48
@f2ked:matrix.org* dumb question.. when I start the scheduler a bunch of MergeRequests are created.. what do they represent?17:48
@clarkb:matrix.orgZuul configuration is loaded out of git repos. When a scheduler starts it may have to ask the mergers to provide it with the config data if it isn't already cached in zookeeper17:49
@clarkb:matrix.orgthey should all (or mostly) be of the cat job type17:49
@f2ked:matrix.orgyes.. but there are so many, the number seems to be related to the number of branches but then multiplied by something?17:51
@clarkb:matrix.orgyes the scheduler needs the zuul config files from every branch in every repo17:51
@f2ked:matrix.orgmy project causes almost 1600 of these but I only have like 500 branches17:52
@f2ked:matrix.org(which I am pruning)17:52
@f2ked:matrix.orgahh but this is gerrit so I need all the `refs/changes/*` too?17:54
@clarkb:matrix.orgI think change configs should be dynamic and not on startup17:56
@clarkb:matrix.orgat startup it should only care about the baseline config17:56
@f2ked:matrix.orgwhere is this information kept? if filesystem then should I create a sharable volume for it?18:07
@fungicide:matrix.orgit's stored in zookeeper18:08
@fungicide:matrix.orgso cached persistently across scheduler restarts18:09
@clarkb:matrix.orgthe git repos are on disk though and you can speed startup by keeping the cached repos around18:12
@f2ked:matrix.orgAny tips on figuring out why my trigger isn't making it out to zuul?18:32
```
- pipeline:
name: check
post-review: false
require:
XXX-gerrit:
open: true
current-patchset: true
trigger:
XXX-gerrit:
# - event: patchset-created
# - event: change-restored
- event: comment-added
comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*zuul-check
success:
XXX-gerrit:
Verified: 1
failure:
XXX-gerrit:
# we do not -1 vote yet
Verified: 0
```
and debug logs show
```
zuul-scheduler-1 | 2025-04-08 18:28:56,744 DEBUG zuul.Scheduler: Processing tenant trigger events in XXX-tenant
zuul-scheduler-1 | 2025-04-08 18:28:56,746 DEBUG zuul.Scheduler: [e: 5e2bb7327c084a05944f2554e4dc7d8b] Forwarding trigger event <GerritTriggerEvent comment-added XXX-gerrit.ZZZ.com/MMM master 44880,1 Verified:0, Verified-B:0, Submodule-Dependencies-Merged:0, Code-Review:0>
zuul-scheduler-1 | 2025-04-08 18:28:56,749 DEBUG zuul.Scheduler: Released XXX-tenant read tenant lock
```
and it seems to stop there :(
@clarkb:matrix.orgI think if you have zuul log in debug mode it will capture the entire event in its log. Then you can compare your regex against the comment that was added there18:36
@fungicide:matrix.orgrunning your scheduler(s) with the -d command-line option should result in debug level logging18:41
@f2ked:matrix.orgI am.. these are debug messages18:47
@f2ked:matrix.organd the process is identical to any comment :(18:47
@fungicide:matrix.orgah, yes, you should see messags about "GerritTriggerEvent comment-added ... does not match ... <your regex>"18:52
@clarkb:matrix.orgright so in your logs prior to those messages should be the actual gerrit event content for comment-added.18:52
@clarkb:matrix.organd afterwards you should see the filtering. I think if you grep by the 5e2bb737... event id you'll get the info that is assoicated together here18:53
@f2ked:matrix.orgcorrect.. so in my case that is `5e2bb7327c084a05944f2554e4dc7d8b`18:54
@f2ked:matrix.orglooking for the "does not match"18:55
@f2ked:matrix.orgI don't see anything about "match"ing a regex19:00
@fungicide:matrix.orgi'm digging in our debug logs for a good example19:00
@fungicide:matrix.orgfirst, this is what a successful match looks like in the log:19:01
@fungicide:matrix.org2025-04-08 00:22:18,235 DEBUG zuul.Pipeline.openstack.check: [e: 9502fbb5fb71408fac0956fcd0999bca] Event <GerritTriggerEvent comment-added opendev.org/openstack/cinder master 913217,13 Verified:0, Code-Review:0, Workflow:0, Review-Priority:0> for change <Change 0x7f9874f83150 openstack/cinder 913217,13> matched <GerritEventFilter connection: gerrit types: comment-added ignore_deletes: True comments: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck> in pipeline <IndependentPipelineManager check>19:02
@f2ked:matrix.orgthank you.19:05
@f2ked:matrix.organother dump question.. I'm using ssh only with gerrit.. would using http as well be more helpfull?19:05
@fungicide:matrix.orgi've got a lot of non-match examples due to different event types, empty comment strings, et cetera, still looking for one that failed a match on the comment string itself19:05
@clarkb:matrix.org> <@f2ked:matrix.org> another dump question.. I'm using ssh only with gerrit.. would using http as well be more helpfull?19:07
Using http enables more features. not sure it is required for this. I would use http if you can
@fungicide:matrix.orgzuul can't do inline comments without rest api access to gerrit, for example19:07
@fungicide:matrix.orghttps://zuul-ci.org/docs/zuul/latest/drivers/gerrit.html#http-configuration makes a mention19:08
@fungicide:matrix.orgf2ked: okay, i've managed to confirm this is what a non-match debug message looks like when someone makes a comment that doesn't match the defined regex...19:20
@fungicide:matrix.org2025-04-02 02:00:18,809 DEBUG zuul.Pipeline.openstack.check: [e: 0702fd64209e40ea9654fa0ca5c16332] Event <GerritTriggerEvent comment-added opendev.org/openstack/nova stable/2024.2 943924,1 Verified:0, Code-Review:0, Workflow:0, Review-Priority:0> for change <Change 0x7f0f3fa03f90 openstack/nova 943924,1> does not match <GerritEventFilter connection: gerrit types: comment-added ignore_deletes: True comments: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck> in pipeline <IndependentPipelineManager check> because Comments [<zuul.lib.re2util.ZuulRegex object at 0x7f0f75d28550>] do not match None19:21
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 899767: Add runtime graphs to web https://review.opendev.org/c/zuul/zuul/+/89976720:07
-@gerrit:opendev.org- Aurelio Jargas proposed: [zuul/zuul-jobs] 945277: ensure-python-command: Install venv in Zuul-scoped path https://review.opendev.org/c/zuul/zuul-jobs/+/94527721:20
-@gerrit:opendev.org- Aurelio Jargas proposed:21:26
- [zuul/zuul-jobs] 945278: ensure-python-command: Optimize pip install https://review.opendev.org/c/zuul/zuul-jobs/+/945278
- [zuul/zuul-jobs] 945276: ensure-nox: Add support for global symlink https://review.opendev.org/c/zuul/zuul-jobs/+/945276
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 946596: Fix job display in web ui https://review.opendev.org/c/zuul/zuul/+/94659621:30
-@gerrit:opendev.org- Zuul merged on behalf of Felix Edel: [zuul/zuul] 946148: Annotate "reporting start|enqueue|dequeue" log messages with event UUID https://review.opendev.org/c/zuul/zuul/+/94614821:41
-@gerrit:opendev.org- Aurelio Jargas proposed: [zuul/zuul-jobs] 946718: Fix deprecated cleanup-run https://review.opendev.org/c/zuul/zuul-jobs/+/94671821:47
@f2ked:matrix.orghi.. trying to build a docker which worked for me on `release 11.0.0` but any newer release I keep getting22:34
```
#16 91.52 × python setup.py bdist_wheel did not run successfully.
#16 91.52 │ exit code: 1
#16 91.52 ╰─> [6 lines of output]
#16 91.52 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
#16 91.52 or: setup.py --help [cmd1 cmd2 ...]
#16 91.52 or: setup.py --help-commands
#16 91.52 or: setup.py cmd --help
#16 91.52
#16 91.52 error: invalid command 'bdist_wheel'
#16 91.52 [end of output]
```
any tips?
@fungicide:matrix.orgprobably has to do with the version of setuptools you're ending up with. i expect we should replace the "build wheel" task in playbooks/zuul-stream/pre.yaml with a call to pyproject-build22:36
@f2ked:matrix.orgthis is the the Dockerfile from the repo22:36
@fungicide:matrix.orgdirectly invoking setup.py has been deprecated for a while, also a recent release of the wheel library may have broken the bdist_wheel entrypoint, so it could be newer wheel as well22:37
@fungicide:matrix.orgthe only bdist_wheel reference i seee in the zuul/zuul repo at this point is in playbooks/zuul-stream/pre.yaml22:39
@f2ked:matrix.orgI see the same, but it seems to be "nested", here is the whole thing (sorry)22:41
```
#16 89.90 Building wheel for paho-mqtt (setup.py): started
#16 90.05 Building wheel for paho-mqtt (setup.py): finished with status 'error'
#16 90.05 error: subprocess-exited-with-error
#16 90.05
#16 90.05 × python setup.py bdist_wheel did not run successfully.
#16 90.05 │ exit code: 1
#16 90.05 ╰─> [6 lines of output]
#16 90.05 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
#16 90.05 or: setup.py --help [cmd1 cmd2 ...]
#16 90.05 or: setup.py --help-commands
#16 90.05 or: setup.py cmd --help
#16 90.05
#16 90.05 error: invalid command 'bdist_wheel'
#16 90.05 [end of output]
```
@fungicide:matrix.orghttps://github.com/pypa/wheel/issues/660 is the recent bug i was remembering22:41
@f2ked:matrix.org* I see the same, but it seems to be "nested", here is the whole thing (sorry)22:41
```
#16 89.90 Building wheels for collected packages: paho-mqtt, jsonpath-rw, ibm-vpc, ibm_cloud_sdk_core, ibm-platform-services, ibm-cos-sdk, ibm-cos-sdk-core, ibm-cos-sdk-s3transfer
#16 89.90 Building wheel for paho-mqtt (setup.py): started
#16 90.05 Building wheel for paho-mqtt (setup.py): finished with status 'error'
#16 90.05 error: subprocess-exited-with-error
#16 90.05
#16 90.05 × python setup.py bdist_wheel did not run successfully.
#16 90.05 │ exit code: 1
#16 90.05 ╰─> [6 lines of output]
#16 90.05 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
#16 90.05 or: setup.py --help [cmd1 cmd2 ...]
#16 90.05 or: setup.py --help-commands
#16 90.05 or: setup.py cmd --help
#16 90.05
#16 90.05 error: invalid command 'bdist_wheel'
#16 90.05 [end of output]
```
@fungicide:matrix.orgah, okay, so the error showed up when building paho-mqtt on your platform22:42
@fungicide:matrix.orglooks like we currently pin paho-mqtt<2.0.0 in requirements.txt22:42
@fungicide:matrix.orgnewer versions publish pre-built wheels, but the latest version prior to 2.0.0 does not, so builds from their sdist source22:43
@fungicide:matrix.orghttps://pypi.org/project/paho-mqtt/1.6.1/#files only has the sdist22:43
@f2ked:matrix.orgFWIW there are a bunch of them22:45
```
#16 90.05 Building wheel for paho-mqtt (setup.py): finished with status 'error'
#16 90.35 Building wheel for jsonpath-rw (setup.py): finished with status 'error'
#16 90.68 Building wheel for ibm-vpc (setup.py): finished with status 'error'
#16 91.41 Building wheel for ibm-cos-sdk (setup.py): finished with status 'error'
#16 91.71 Building wheel for ibm-cos-sdk-core (setup.py): finished with status 'error'
#16 92.01 Building wheel for ibm-cos-sdk-s3transfer (setup.py): finished with status 'error'
```
@fungicide:matrix.orglooks like this is going to be happening in the context of the python-builder:3.11-bookworm image, i guess?22:46
@f2ked:matrix.orgthose are all "nodepool" "launcher-ish" things right?22:47
@fungicide:matrix.orgif this is fallout from the regression i linked above in wheel 0.46.0 then it may imply that we're building with new wheel but a setuptools prior to 70.122:48
@fungicide:matrix.orgjsonpath-rw is another example of a package which only distributes an sdist: https://pypi.org/project/jsonpath-rw/#files22:50
@fungicide:matrix.orgi don't suppose there's any output indicating which version of wheel and setuptools were used when building those?22:50
@fungicide:matrix.orgwheel 0.46.0 was yanked from pypi, but a 0.46.1 appeared just 2 hours ago22:52
@f2ked:matrix.org```22:52
#15 4.344 Setting up python3-wheel (0.38.4-2) ...
#15 4.462 Processing triggers for libc-bin (2.36-9+deb12u10) ...
#15 4.490 + pip install wheel
#15 4.809 Requirement already satisfied: wheel in /usr/local/lib/python3.11/site-packages (0.45.1)
```
@fungicide:matrix.orgmmm, okay, so looks like it was using 0.45.1 as of that last message22:53
@f2ked:matrix.orgso this came from22:53
```
FROM quay.io/opendevmirror/node:16-bookworm as js-builder
```
?
@fungicide:matrix.orgthough if it ended up doing build isolation it may have quietly downloaded wheel 0.46.1 in a separate build venv22:53
@f2ked:matrix.orgoh wait22:54
```FROM docker.io/opendevorg/python-builder:3.11-bookworm as builder
```
@f2ked:matrix.org* FROM docker.io/opendevorg/python-builder:3.11-bookworm as builder22:54
@fungicide:matrix.orgthe python-builder:3.11-bookworm image i mentioned earlier, i think22:54
@fungicide:matrix.orgyeah22:54
@fungicide:matrix.orgi'm struggling to find where in the dockerfile zuul and/or its python dependencies are actually being installed22:56
@fungicide:matrix.orgcontainers are still a weak spot for me22:56
@f2ked:matrix.orgme too :( I appreciate the help22:57
@jim:acmegating.comi just approved a change to get a refreshed build, and it is now failing in opendev's zuul with the error f2ked cited; that's new within the past 2 hours: https://zuul.opendev.org/t/zuul/builds?job_name=zuul-upload-image&project=zuul/zuul22:59
@jim:acmegating.comso perhaps that error can now be diagnosed with opendev builds22:59
@fungicide:matrix.orgyeah, i suspect this is a reemergence of the wheel 0.46.0 regression in the just-released wheel 0.46.122:59
@fungicide:matrix.orgnote that the wheel package readme/description now declares: "This project used to contain the implementation of the setuptools bdist_wheel command, but as of setuptools v70.1, it no longer needs wheel installed for that to work."23:00
@fungicide:matrix.orgi suspect that we're ending up with a version of setuptools contemporary with debian/bookworm or cpython 3.11.x and latest wheel23:00
@jim:acmegating.comthe wheel versions in the 2 most recent builds are 0.45.1 (working) and 0.46.1 (failing)23:00
@jim:acmegating.comhttps://zuul.opendev.org/t/zuul/build/a0f2d54ba5684215b58c9f3d77c34601/log/job-output.txt#221623:01
https://zuul.opendev.org/t/zuul/build/9fea139101c14afab2a79fd14d28eded/log/job-output.txt#2245
@fungicide:matrix.orgthe similar-looking problem described in https://github.com/pypa/wheel/issues/660 resulted in wheel 0.46.0 being yanked from pypi in a matter of hours23:02
@clarkb:matrix.orgdo we explicitly update wheel? I don't think we've rebuilt the base images that recently23:02
@clarkb:matrix.orgwe did semi recently but before wheel updated23:02
@jim:acmegating.comyeah, assemble does it explicitly23:02
@jim:acmegating.comhere: https://opendev.org/opendev/system-config/src/branch/master/docker/python-builder/scripts/assemble#L9323:03
@f2ked:matrix.orghttps://hub.docker.com/r/opendevorg/python-base/tags says 3.11-bookworm was updated 20 days ago?23:05
@fungicide:matrix.orgif we have a step that explicitly upgrades wheel, then we should maybe do the same for setuptools23:05
@jim:acmegating.comfungi: i was wondering the same thing; i'm trying that locally23:06
@fungicide:matrix.orgi have a strong suspicion that will solve it23:08
@f2ked:matrix.orgare there older docker images that can be "FROM"ed from?23:10
@jim:acmegating.comf2ked: that won't fix it (if we're correct about the cause)23:11
@jim:acmegating.comyes, adding setuptools does appear to fix the issue; i pushed remote:   https://review.opendev.org/c/opendev/system-config/+/946722 Add setuptools to python-builder assemble [NEW]        23:16
@fungicide:matrix.orgof course, it may start working sooner if the wheel maintainers yank 0.46.1 from pypi over similar concerns to why they yanked 0.46.023:20
@jim:acmegating.comyeah... but maybe it's an okay change regardless?  it seems aligned with what we were trying to do there...  i'm not sure if there was a particular issue we were avoiding by not including it, or just being conservative23:22
@fungicide:matrix.orgyes, i think it's a good choice either way23:23
@fungicide:matrix.orgclarkb was the one who switched that script to using wheel several years ago in https://review.opendev.org/c/opendev/system-config/+/86215223:24
@clarkb:matrix.orgThat was feedback form pypa that pip install doesn't side effect the way we want in all cases iirc23:27
@fungicide:matrix.orgyeah, seems that was the most recent change to the script23:27
@clarkb:matrix.orgshould I approve the change?23:29
@fungicide:matrix.orgoh, but actually the pip install -U pip wheel was there from mordred's very first iteration of the script in https://review.openstack.org/63187823:29
@clarkb:matrix.orgwhy isn't wheel pulling in newer setuptools if it needs it?23:30
@clarkb:matrix.orgI guess that is just an oversighto n their part23:30
@fungicide:matrix.organd yeah, i find no rationale stated for not upgrading setuptools there, so probably just an oversight/happened to work out (until today)23:30
@clarkb:matrix.orgI +2'd but didn't approve in case we want to check check results first or something23:30
@fungicide:matrix.orgclarkb: that repo is in a tenant that won't enqueue until check completes anyway, so there's time to look over the results regardless23:31
@clarkb:matrix.orgya but will we do that at this point in the day :)23:31
@fungicide:matrix.orghttps://zuul.opendev.org/t/openstack/build/940798011644410db652761a49b21c4a has preliminary results23:32
@clarkb:matrix.orgDoing a depends on from lodgeit might be a good idea 23:34
@clarkb:matrix.orgOr something else that uses assemble..jeepyb maybe?23:34
@fungicide:matrix.orgah, yeah i guess we don't call the assemble script in the base image build?23:34
@fungicide:matrix.orgdo we in the uwsgi-base build?23:35
@fungicide:matrix.orgsurvey says yes: https://zuul.opendev.org/t/openstack/build/cbe349c477d048a9b25b54941d072d6a/log/job-output.txt#178023:36
@f2ked:matrix.orgFWIW changing `assemble` in the `Dockerfile` "solves" this for me23:36
```
RUN set -x \
&& sed -i -e 's;/tmp/venv/bin/pip install -U pip wheel build;/tmp/venv/bin/pip install -U pip wheel build setuptools;' `which assemble`
RUN assemble
```
@fungicide:matrix.org"Successfully installed build-1.2.2.post1 packaging-24.2 pip-25.0.1 pyproject_hooks-1.2.0 setuptools-78.1.0 wheel-0.46.1"23:36
@f2ked:matrix.orgwhich bodes well for your ultimate fix23:37
@clarkb:matrix.orgfungi: ya we run assemble in the builder iirc which puts everything together and uwsgi uses the builder too23:37
@fungicide:matrix.orgso i think that's a successful exercise of it23:37
@fungicide:matrix.orgno need for a depends-on from a lodgeit change unless you want a second confirmation23:38
@fungicide:matrix.organd i need to knock off for the night, getting dark here23:40
@clarkb:matrix.orgI guess we can approve this as only new image builds will be affected which makes impact small23:42
@clarkb:matrix.orgeven if it doesn't work its not like existing images would be affected23:42
-@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 945907: WIP Make Gerrit Connection Replication Aware https://review.opendev.org/c/zuul/zuul/+/94590723:49
« Monday, 2025-04-07 Index Wednesday, 2025-04-09 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!