| opendevreview | Stephen Finucane proposed openstack/pbr master: Pin zuul ansible, pyperclip versions https://review.opendev.org/c/openstack/pbr/+/960133 | 11:50 |
|---|---|---|
| opendevreview | Stephen Finucane proposed openstack/pbr master: trivial: Inline LocalInstallScripts._make_wsgi_scripts_only https://review.opendev.org/c/openstack/pbr/+/960028 | 11:50 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Remove use of easy_install https://review.opendev.org/c/openstack/pbr/+/960029 | 11:50 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Inline overridden script writer https://review.opendev.org/c/openstack/pbr/+/960030 | 11:50 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: WIP: Remove remaining user of pkg_resources https://review.opendev.org/c/openstack/pbr/+/960031 | 11:50 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: WIP: Remove remaining user of pkg_resources https://review.opendev.org/c/openstack/pbr/+/960031 | 14:09 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Remove remaining user of pkg_resources https://review.opendev.org/c/openstack/pbr/+/960031 | 14:15 |
| rosmaita | hello ... i need some help with an openstack-tox based job that is failing at the "Install any sibling python packages" phase | 16:08 |
| rosmaita | it's installing glance_store as a sibling, but then is failing because the built package 4.8.3.dev1 violates the constraint of glance-store===4.8.2 | 16:08 |
| rosmaita | it's only failing on stable/2024.2, and the job is identical on the other branches but passing | 16:08 |
| rosmaita | here's the job: https://zuul.opendev.org/t/openstack/build/15fa157c73ec4efca138eb933e2d8ca6/logs | 16:08 |
| rosmaita | job name is "cross-glance-tox-functional" | 16:08 |
| rosmaita | this one passed yesterday in stable/2024.1: https://zuul.opendev.org/t/openstack/build/0454dba0787243f1b54a31a3fdf4f294 | 16:08 |
| rosmaita | it installs glance_store-4.7.1.dev6 with no problem even though the u-c is glance-store===4.7.0 | 16:08 |
| rosmaita | tox version same on both, both running py310, both on ubuntu-jammy | 16:08 |
| rosmaita | i am out of ideas about what could be causing this | 16:08 |
| fungi | is glance_store (or glance-store) listed in the required projects for the job? | 16:11 |
| fungi | i thought tox-siblings stripped out constraints for required projects | 16:11 |
| rosmaita | yes https://opendev.org/openstack/glance_store/src/branch/stable/2024.2/.zuul.yaml#L138 | 16:13 |
| rosmaita | though the requirement is 'glance-store' in u-c ... but then that would affect the other branches, too | 16:13 |
| fungi | this could be related to the unpatched setuptools regression related to dist name normalization, i suppose | 16:19 |
| rosmaita | does sound plausible ... i have not been paying attention to that issue | 16:21 |
| rosmaita | both the passing and failing jobs are using the same version, though: Requirement already satisfied: setuptools>=40.8.0 in ./.tox/.pkg/lib/python3.10/site-packages (80.9.0) | 16:25 |
| fungi | i can't find where in the logs it records altering the constraints file, but it clearly did something because it's passing /tmp/tmpwai21ri0 as the constraints file to all the pip install commands | 16:38 |
| rosmaita | so your theory is that glance-store shouldn't even be in there | 16:40 |
| rosmaita | i can't figure out what would be making that branch act differently | 16:40 |
| fungi | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml is where that happens | 16:41 |
| fungi | is tox.ini on that branch doing something strange with constraints management? | 16:43 |
| fungi | oh, even more strange, this is a job running *for* glance_store | 16:44 |
| rosmaita | i don't think so ... i diffed caracal and dalmatian, and the only diff in the tox.ini file is the TOX_CONSTRAINT_FILE value | 16:44 |
| rosmaita | yeah, it installs both glance and glance_store, and then runs tox out of the glance directory | 16:44 |
| rosmaita | (don't ask me why we did it like that, i don't remember) | 16:44 |
| croelandt | +1 the only difference between these branches is that they point to their respective upper constraints file | 16:45 |
| rosmaita | fungi: croelandt is going to take over for a while, i need to walk the dog | 16:45 |
| fungi | i think it may have something to do with glance-store vs glance_store in the upper-constraints.txt file in the requirements repo | 16:47 |
| fungi | stable/2024.1 and stable/2024.2 use glance-store, while later branches use glance_store instead | 16:49 |
| fungi | looks like that changed when generate-constraints ran on 2025-02-22 for https://review.opendev.org/c/openstack/requirements/+/942508 | 16:51 |
| croelandt | Interesting, we merged https://review.opendev.org/c/openstack/glance_store/+/940529 around that time in stable/2024.2 | 16:52 |
| fungi | runs for stable/2024.1 may be using an older setuptools where glance-store still works, due to running on an older python version? | 16:52 |
| fungi | pure speculation | 16:52 |
| croelandt | So maybe 1) I send a dummy patch to stable/2024.2 to make sure this triggers the issue 2) send a patch that s/glance-store/glance_store/ in .zuul.yaml to make sure it fixes the issue | 16:53 |
| croelandt | how does that sound? | 16:53 |
| croelandt | well no actually sorry, I misunderstood | 16:55 |
| fungi | the required-projects list in the job definition uses the git repository names, not python dist package names, so no | 16:56 |
| croelandt | hm, both stable/2024.1 and stable/2024.2 use Python 3.10 | 16:56 |
| fungi | yeah, rules that out | 16:57 |
| fungi | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_install_sibling_packages.py is the actual code that the name normalization when figuring out what to strip out of the constraints file | 16:58 |
| fungi | the write_new_constraints_file() function in particular | 16:58 |
| clarkb | fungi: croelandt I think the way it works is we let tox install everything. Then the zuul role iterates over all listed packages that were installed and if one of them is a sibling it installs that sibling from source updating the version | 16:58 |
| clarkb | fungi: croelandt if you then have followup steps that validate the versions against constraints you may fail. I suspect that is what may be happening here but I haven't pulled the logs up | 16:58 |
| fungi | clarkb: yes, it's the second phase where things are breaking | 16:59 |
| fungi | for some reason the sibling isn't getting stripped from the constraints file | 16:59 |
| fungi | so it tries to install from source and then the resulting version conflicts with what's in constraints | 16:59 |
| clarkb | could it be the normalized name is used on one side and not the other so the lines don't match and the file isn't updated? | 16:59 |
| fungi | that's my primary suspicion, the question is why | 17:00 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_install_sibling_packages.py#L150 it does seem to be a naive equality check | 17:00 |
| croelandt | and also why only on 2024.2 and not 2024.1 :/ | 17:00 |
| clarkb | not equivalence check | 17:00 |
| clarkb | the sibling package name is glance_store which != https://opendev.org/openstack/requirements/src/branch/stable/2024.2/upper-constraints.txt#L570 so the line is not updated in the tmp constraints | 17:05 |
| croelandt | but I also see glance-store https://opendev.org/openstack/requirements/src/branch/stable/2024.1/upper-constraints.txt#L569 here | 17:07 |
| clarkb | I guess we don't update the line we simply drop it entirely | 17:07 |
| clarkb | 2024.1 says: 'Package glance-store (glance_store) on system in src/opendev.org/openstack/glance_store' in that task. But the failing job for 2024.2 says 'Package glance_store on system in src/opendev.org/openstack/glance_store' | 17:09 |
| clarkb | so the 2024.1 job almost seems to understand the equivalence but 2024.2 does not for some reason | 17:09 |
| clarkb | ah 2024.1 also says 'Found glance-store python package installed' but 2024.2 says 'Found glance_store python package installed' | 17:10 |
| clarkb | that logging comes from https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_install_sibling_packages.py#L177-L207 | 17:11 |
| clarkb | we initially grab the name from here: https://opendev.org/openstack/glance_store/src/branch/stable/2024.2/setup.cfg#L2 (its the same value for 2024.1 and 2024.2) | 17:13 |
| clarkb | fungi: croelandt https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_install_sibling_packages.py#L123-L141 in 2024.1 this method returns glance-store and in 2024.2 it returns glance_store | 17:15 |
| croelandt | hm so it comes from pip freeze | 17:16 |
| clarkb | but as you mention they are both python 3.10 jobs. Tox.ini for glance in those branches both have the same min version so I'm not sure why 2024.2 is using a different tox env. Maybe something has a hard dep on setuptools and is updating it in 2024.2? | 17:17 |
| clarkb | we might have evidenceo f that in the logs | 17:17 |
| clarkb | fungi: glance_store is the new version of the name right? So newer tools that normalize more would potentially do it? | 17:18 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Remove dead code https://review.opendev.org/c/openstack/pbr/+/964712 | 17:18 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Remove unused code https://review.opendev.org/c/openstack/pbr/+/964713 | 17:18 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Rename setup.cfg parsing utilities https://review.opendev.org/c/openstack/pbr/+/964714 | 17:18 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: Merge pbr.core, pbr.utils into pbr.setupcfg https://review.opendev.org/c/openstack/pbr/+/964715 | 17:18 |
| opendevreview | Stephen Finucane proposed openstack/pbr master: WIP: Add initial pyproject.toml hook https://review.opendev.org/c/openstack/pbr/+/964716 | 17:18 |
| clarkb | it looks like the same version of setuptools and pip at first glance. Quite a mystery. | 17:19 |
| clarkb | could probably hold a node and inspect the tox env directly | 17:19 |
| croelandt | so glance itself has glance-store in its requirements.txt | 17:21 |
| clarkb | fungi: I guess tox/nox siblings should probably avoid being the normalization police so updating the role to compared normalized forms would be an appropriate "fix" here but it doesn't explain why things differe between the branches | 17:21 |
| frickler | 4.8.2 was release in Mar 2025, so with the new release tooling, 4.7.0 is from 2024, so old format. I think thats the essential difference | 17:32 |
| clarkb | frickler: oh the package itself self reports the name differently when running pip freeze | 17:34 |
| clarkb | so ya in that case the correct fix here is probably to update siblings checks from equality to equivalence when modifying constraints | 17:34 |
| rosmaita | i wish we had called the thing glancestore ! | 17:36 |
| fungi | python packaging standards consider any sequence of ".", "-" or "_" tp be equivalent to a single "_" so we could probably use a regular expression to look for any match that fits those rules | 17:37 |
| rosmaita | clarkb: frickler: fungi: croelandt: it sounds like this has to be fixed on the zuul side? | 17:37 |
| fungi | rosmaita: probably in zuul/zuul-jobs:roles/tox/library/tox_install_sibling_packages.py yes | 17:38 |
| fungi | at least that seems like the most sensible place to do it, to me | 17:38 |
| clarkb | here https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_install_sibling_packages.py#L150 and here https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/nox/library/nox_install_sibling_packages.py#L145 | 17:39 |
| clarkb | remote: https://review.opendev.org/c/zuul/zuul-jobs/+/964724 Find tox/nox siblings with normalized package names | 18:17 |
| clarkb | I think something like that may fix it | 18:17 |
| opendevreview | Merged openstack/pbr master: Pin zuul ansible, pyperclip versions https://review.opendev.org/c/openstack/pbr/+/960133 | 18:24 |
| clarkb | fungi: https://review.opendev.org/c/zuul/zuul-jobs/+/964724 should be reviewable now. rosmaita even confirmed with a depends on on top of my updated unit test case | 19:00 |
| fungi | oh excellent! | 19:07 |
| fungi | zuul rides again | 19:07 |
| fungi | yep, lgtm, thanks! | 19:09 |
| rosmaita | thank you! | 19:12 |
| JayF | OK, I'm working with clif to figure out a weirdness in CI. | 19:13 |
| JayF | https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/955799 depends on https://review.opendev.org/c/openstack/ironic/+/964664 (a change that raises the ram for the ironic-standalone-anaconda job) | 19:13 |
| JayF | the stable-branch version of those jobs just uses override-checkout: https://opendev.org/openstack/ironic-tempest-plugin/src/branch/master/zuul.d/stable-jobs.yaml#L27 | 19:13 |
| JayF | in this setup, is the zuul config for ironic-standalone-anaconda being used as the parent the one from HEAD of master *or* the one from HEAD of stable/whatever | 19:14 |
| JayF | I think this has rubber ducked me into landing on clif's original hypothesis: we have to backport the ram bump | 19:15 |
| fungi | https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.override-checkout "This value is also used to help select which variants of a job to run. If override-checkout is set, then Zuul will use this value instead of the branch of the item being tested when collecting jobs to run." | 19:17 |
| fungi | is that what you're asking? | 19:17 |
| JayF | well, mainly, I'm wondering if override-checkout also impacts job config as well | 19:18 |
| JayF | e.g. for https://opendev.org/openstack/ironic-tempest-plugin/src/branch/master/zuul.d/stable-jobs.yaml#L27 -- is the override-checkout branch version of ironic-standalone-anaconda being used? | 19:18 |
| fungi | if you set it at the job level, then it selects the job variant configured on that branch if it exists | 19:20 |
| fungi | including for the parent job definition | 19:21 |
| JayF | OK, cool, so we have to backport the fix thru ironic | 19:21 |
| JayF | hooray :| | 19:21 |
| fungi | the build details should show which variants were used in the inheritance list | 19:22 |
| fungi | it will list the project and selected branch for each ancestor | 19:23 |
| fungi | JayF: https://zuul.opendev.org/t/openstack/build/1b0883969fe646499c242ce8a50aa94a/log/zuul-info/inventory.yaml#417-418 "source: openstack/ironic/zuul.d/ironic-jobs.yaml@stable/2025.1#294" | 19:26 |
| JayF | ah, nice. I'll keep that in mind. | 19:26 |
| clarkb | JayF: fungi: not to confuse things any more than necessary but the ironic repo will have master on your depends on content and stable/2025.1 or whatever on the latest version for that branch. What override checkout does is checks out a specific branch it doesn't impact the speculative git state if that makes sense | 19:33 |
| clarkb | the specualtive state is still there. it isn't checked out | 19:33 |
| JayF | I think the intersection of those two things is what confused me | 19:34 |
| clarkb | and then the job config is automatically loaded based on the branch names | 19:34 |
| clarkb | since you can only have one job config | 19:34 |
| JayF | just an edge case but fungi's info seems to indicate to me that backporting the fix in ironic repo is the path to make it work | 19:34 |
| JayF | even if the behavior is difficult to understand | 19:34 |
| clarkb | yes I think the conclusion to backport is the correct appraoch here | 19:35 |
| clarkb | I just want to call that out as it often becomes important if testing upgrades for example | 19:35 |
| clarkb | you can acutally dependso n a change in master and a change in stable/2025.1 and both sepcualtive git states will be present in the git repo. Then you can install stable/2025.1 specualtively, test things, upgrade to/install master speculatively and test things work post upgrade | 19:35 |
| clarkb | its really powerful | 19:36 |
| fungi | right, if you depends-on to a change that alters job configuration on one branch but then override-checkout in the job to a different branch, you can end up with a variant of the job that doesn't include your change because it's to a different branch than the one being selected | 19:37 |
| JayF | but it is correct to say even after the ironic change merges (removing the depends-on complication) into master | 19:37 |
| fungi | one of the reasons not to take the complexity of branch overrides lightly | 19:37 |
| JayF | the stable/2025.1 version of the job will use the stable/2025.1 zuul config from ironic/ | 19:37 |
| clarkb | JayF: yes | 19:37 |
| clarkb | also functionality like this is where zuul quickly becomes more powerful than merge trains in gitlab/github aiui | 19:39 |
| clarkb | those are limited to same branch same project and gitlab has a limit of like 10 total changes | 19:39 |
| clarkb | whereas zuul lets you compose abritrary branches and projects with as many changse as your hardware can handle before anything merges into the target branches | 19:40 |
| fungi | or as many as your brain can handle before the complexity causes it to melt | 19:41 |
| rosmaita | fungi: clarkb: is it the infra-core group that can merge https://review.opendev.org/c/zuul/zuul-jobs/+/964724 ? | 20:09 |
| fungi | rosmaita: it's the zuul community's maintainers who take care of that, and clarkb has already mentioned it in their matrix room | 20:10 |
| rosmaita | oh, ok, thanks | 20:10 |
| fungi | and now it's merged | 20:32 |
| fungi | as of about 5 minutes ago | 20:33 |
| clarkb | fungi: not yet, that was a different change of mine that merged | 20:42 |
| tonyb | Using `Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/964724` should work right? or is zuul-jobs trusted? | 21:30 |
| clarkb | tonyb: zuul-jobs is not trusted but roles in zuul-jobs may be used in trusted contexts | 21:38 |
| clarkb | tonyb: rosmaita already indicated that this was done via a change in the comments of that change | 21:38 |
| rosmaita | tonyb: clarkb: i was kind of surprised that it worked, but it did | 21:40 |
| tonyb | clarkb: Cool. I was thinking for me rather than rosmaita :) https://review.opendev.org/c/openstack/requirements/+/957587 | 21:42 |
| fungi | ah, never mind, i misread yeah | 21:57 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!