Tuesday, 2026-08-25

opendevreviewMerged openstack/pbr master: Set setuptools min in requirements.txt  https://review.opendev.org/c/openstack/pbr/+/100214002:16
bbezakHi. We’re ready to make a release now as patch above merged ?06:20
bbezak(Pbr)06:20
mnasiadkastephenfin: can you help with ^^?08:09
stephenfinmnasiadka: https://review.opendev.org/c/openstack/releases/+/100226411:48
mnasiadkaAh, I was not aware pbr is tracked in releases eepo12:00
fricklerstephenfin: fungi: the tomli thing is still failing with 7.1.2: https://zuul.opendev.org/t/openstack/build/af09bd0a1c5a45069edf0f97d55cef09 :(12:51
fricklermaybe we should pull both 7.1.1 and 7.1.2 at this point?12:52
stephenfinI'm trying and failing to reproduce that locally :(13:23
opendevreviewStephen Finucane proposed openstack/pbr master: Duplicate check for tomli dependency  https://review.opendev.org/c/openstack/pbr/+/100228613:31
opendevreviewStephen Finucane proposed openstack/pbr master: Add tomli as dependency on Python 3.7 - 3.10  https://review.opendev.org/c/openstack/pbr/+/100228713:31
fricklerstephenfin: would it help if I hold a CI node for you to access?13:31
stephenfinfrickler: The issue with just yanking is that it doesn't help us progress13:34
bbezakpython3.10 -m venv /tmp/v && /tmp/v/bin/pip install -q pbr==7.1.1 setuptools==59.6.0 && /tmp/v/bin/python -c "import setuptools.dist as d; d.Distribution({'name':'test'})"13:34
bbezakTested on laptop13:35
stephenfinbbezak++ thank you πŸ™ So the version of setuptools is too old, not too new πŸ˜…13:35
fricklerstephenfin: I didn't say "just yanking", I'm all for providing a fixed 7.1.3 asap, but does it help you or anyone to keep 7.1.1 and 7.1.2 around until then?13:36
stephenfinYes that would make sense. Sorry I misunderstood what you were suggesting13:38
bbezakAnd to reproduce it for 7.1.2 one needs to work on the repo https://paste.opendev.org/show/bIULVCgqsSGthc4eyhSP/13:46
fungii can yank both new versions for now14:03
fungionce i can get to the e-mail to log into the pypi account, because after username+password+2fa it also wants to send a confirmation token by e-mail14:08
fungithey've gone way overboard with their authentication roadblocks14:09
bbezakI can see we’re on 7.0.3 now, thx14:17
fungiyes, all done14:19
opendevreviewAdam Harwell proposed openstack/tooz master: Add Consul coordination driver using native services  https://review.opendev.org/c/openstack/tooz/+/99714514:58
opendevreviewAdam Harwell proposed openstack/oslo.messaging master: HTTP-RPC driver: core transport, server, and dedup  https://review.opendev.org/c/openstack/oslo.messaging/+/99747014:58
opendevreviewAdam Harwell proposed openstack/oslo.messaging master: HTTP-RPC driver: tooz service discovery and fanout broadcaster  https://review.opendev.org/c/openstack/oslo.messaging/+/99747114:58
opendevreviewAdam Harwell proposed openstack/oslo.messaging master: HTTP-RPC driver: DevStack plugin, CI jobs, and documentation  https://review.opendev.org/c/openstack/oslo.messaging/+/99747214:58
stephenfinfungi: clarkb: I'm not really sure how to proceed with the pbr stuff /o\15:04
stephenfinhttps://review.opendev.org/c/openstack/pbr/+/1002286 should fix the main issue seen in kolla but while testing I'm seeing a new issue:15:04
stephenfinModuleNotFoundError: No module named 'setuptools.command.build'15:04
stephenfinI dug around long enough without getting anywhere before deciding to pull in an LLM, which gave me the following https://paste.opendev.org/show/bStAmqTzyBQw9bxDSuNn/15:05
stephenfinI'm still verifying that but the reasoning checks out superficially at least. That means the fix in 7.1.2 is fixing stuff in one environment while breaking it elsewhere15:06
clarkbthe pre summer release of pbr works right?15:07
clarkband that builds things?15:07
fungiyes, 7.0.315:07
clarkbok so the problem is in PBR itself not the environment changing around us15:08
fungiit just requires all packages to have a setup.py file activating pbr and setup.cfg file declaring the package name15:08
clarkbI just wanted ot make sure that setuptools hasn't also updated in a way that is creating problems for us15:08
fungiso goes through different (potentiallt deprecated in some places) parts of setuptools15:08
stephenfinno, the issue is that the matrix of possible package combos and installer mechanisms is very large, and I have no good way to test against all of them15:09
stephenfinI don't even know many of the combos exist/are possible until we push a release out and someone complains πŸ˜…15:09
stephenfinunknown unknowns and all that15:10
clarkbstephenfin: right so the approach we've used to avoid problems like this historically has been to be very careful about what things we require in PBR and what things we drop support for etc15:10
clarkbwe've sort of had this conversation before but you're taking a very different approach to PBR maintenance than the one fungi and I have applied for years whcih si fine but is far more prone to these problems15:10
clarkbmaybe we can reset the approach and make it much more opt in based on known factors. eg you have a pyproject.toml otherwise don't even process things with this new code path15:11
clarkbif we want to do major refactors of the entire code base holistically and drop support for features then we probably need to start with more robust testing?15:12
clarkbIf 7.0.3 works today then I think going down one of those two paths is what I would do. (Figure out testing to make big refactors less error prone or resume small incremental changes that drop drop support for features or add hard requirements on new ones)15:13
stephenfinthe only thing we removed in 7.1.x is dead code and testing for Jammy (due to infra issues moreso than anything)15:14
stephenfinand pyproject.toml file check already exists15:14
clarkbstephenfin: if the pyproject.toml file checkl exists why were we unconditionally importing tomli and failing in stable branches that don't do anything with toml?15:14
clarkb(that was the original break aiui)15:15
clarkbI haven't seen the new failure yet so haven't looked into it beyond skimming the notes above15:15
stephenfinbecause it's registered via an entry point15:15
stephenfinand unlike Python code, we can't make the entry point conditional on Python/setuptools versions15:16
clarkbright, the entry point itself would have to guard against this stuff / dispatch the correct underlying routines bsaed on the package setup (and I guess the platform itself)15:17
stephenfinexactly15:17
fungiclarkb: unfortunately the "you have a pyproject.toml" test is also potentially misleading since a lot of tools have recently co-opted it as a registry to combine all their configuration, so people may stick tool settings in one even if their packaging is still via other methods15:18
clarkbI should note a third approach that may be worth considering (but potentially problematic for end users) is forking pbr into legacy and modern pbr libraries15:18
clarkbfungi: we could just say that is unsupported by pbr15:19
clarkbeither you use pyproject.toml for packaging and everything else or you don't use it at all15:19
fungii agree, we can take that angle, for sure15:19
clarkbI think for openstack there are unlikely many cases of that situation15:19
stephenfinclarkb: I'm leaning that way too now15:19
stephenfinif we only had to care about pip, it would be likely be far more manageable, but the old easy_install stuff is a debugging nightmare15:21
clarkbstephenfin: the upside to forking is we can set a new baseline of supported functionality in the modern version. Then theoretically the difficult to modify legacy stuff can sit as is with minimal updates15:21
clarkbstephenfin: do you haev a link to the new build command error?15:21
stephenfinsure. I'm working on bbezak's reproducer from above. repasting:15:22
stephenfinhttps://paste.opendev.org/show/bIULVCgqsSGthc4eyhSP/15:22
clarkbreading well I'm less interested in the artifical reproducer. I'm curious to know where this hapepns in reality15:23
clarkb(in particular it is relatively trivial to install random setuptools versions that a broken/bad and find they don't work with $version of PBR)15:23
stephenfinif you run that you'll get this exception https://paste.opendev.org/show/bZqgf6gNJkoZiCVUfKWi/15:23
clarkbbut what we should be avoiding is breaking PBR with setuptools on various distros and latest versions that run on python 3.8 etc15:23
stephenfinthe setuptools version in that reproducer is the version that ubuntu 22.04 shipped with15:24
stephenfinso it's an genuine issue15:25
clarkback that is the context I was missing thanks15:25
clarkbso that shows it failign to import tomli15:26
clarkbkolla has a pyproject.toml15:26
clarkbso it is correct for pbr to try and process the toml file and import tomli. But that version of setuptools does not include tomli? If that is the case then kolla simply doesn't support that platform imo15:27
clarkbI don't think PBR needs to add support where there never was support15:27
clarkbit only needs to support things where the support was there. I'm failing at english but hopefully that makes sense15:27
clarkbbut lets see maybe that version of setuptools has tomli in a different location?15:28
stephenfinYep. I knew setuptools.extern.tomli was removed in v71.0.0 but I'd ignored that since that requires Python 3.8 and by Python 3.8 we already had PEP-517 and all the build isolation stuff15:28
stephenfinbut it turns out it was only added in v61.0.0 which is obviously > v59.6015:29
stephenfin*59.6.015:29
clarkbstephenfin: oh actually I think thee may be a bug here. The failing command is `setup.py egg_info` that is a legacy command and shouldn't touch pyproject.toml at all right?15:29
clarkbanyway I think the main issue is that kolla is attempting to use pyproject.toml packaging with a platform that never supported pyproject.toml packaging and pbr shouldn't be responsible for backporting that functionality15:30
clarkbkolla needs to upgrade their platform to get working pyproject.toml support15:30
stephenfinright, it shouldn't15:30
clarkbor use legacy packaging?15:30
clarkbthen separately there may be a smaller bug that involves PBR trying to do pyproject.toml things when executing setup.py directly15:31
stephenfinbut it also shouldn't be loading the `setuptools.finalize_distribution_options` entrypoints (which I believe is how that code is being loaded and executed) pbr.pbr entrpoint15:31
clarkbbut in this case that is just an easy way to expose the issue kolla is hitting when they try to pip install15:31
stephenfin*via the pbr.pbr entry15:31
stephenfinmy currently plan is to do the following15:32
stephenfina) revert https://review.opendev.org/c/openstack/pbr/+/1002140 since it apparently breaks easy_install (per the initial LLM report I provided) and instead guard the PEP-660 hook points with setuptools versions checks15:34
stephenfin(TODO: figure out how to do the version checks, since I can't rely on packaging or importlib.metadata or setuptools not removing a __version__ marker (if they haven't done so already)15:34
stephenfin)15:34
clarkboh I didn't realize we were doing anything with requirements.txt15:35
clarkbsorry I was halfway around the pacific when most of this stuff got updated and then asleep when things broke and still haven't fully caught up15:35
stephenfinb) proceed with https://review.opendev.org/c/openstack/pbr/+/1002286 and hope that returning early from that function is enough to avoid any other calls into the pyprojecttoml.py file15:36
clarkbstephenfin: what is the expected source of tomli in newer setuptools? Is it just part of python stdlib now or something?15:37
stephenfinThey vendor it before Python 3.11. From 3.11 they use tomllib from the stdlib15:37
clarkbthere is both a python version check and a setuptools check in that diff. And they imply that tomli is magically available in newer things and setuptools vendors it in older things15:38
clarkbaha got it thanks15:38
clarkbstephenfin: fwiw the docstring on that method implies that this will help since setuptools is running every finalize_distribution_options method which in theory means some compatibiltiy layer/legacy version will run if ours doesnt't?15:39
clarkball that to say I agree that your plan should improve things. Maybe we want a stronger notice/comment in the requirements.txt file about how this cannot be used to upgrade setuptools only include it?15:39
stephenfinthat's my thinking yes, but I suspect we may be the first people to have tried this path πŸ˜‡15:40
clarkb*cannot be used to reliably upgrade setuptools.15:40
clarkbas a funny side note: if they had chosen json or ini then most of these specific problems would simply not exist15:41
clarkbwe'd probably have more exciting different problems, but the decision to use a new serialization format with poor support in stdlib seems to have created this specific set of problems15:42
fungior rather the rush to depend on it prematurely in various tooling15:44
fungipeople are no longer content to add something to the stdlib and then wait 10 years to say it's safe to assume15:44
fungiit seems like a culture of instant gratification has started to consume the python ecosystem overall15:45
clarkbfwiw I'm not sure that https://review.opendev.org/c/openstack/pbr/+/1002287 will do what is expected of it for the same reason that old setuptools doesn't upgrade to new setuptools15:46
clarkbhowever, that should only happen in the "legacy" path and if we're skipping this method entirely along that path then maybe it is ok15:46
clarkbI'll leave a comment real quick once I can get my thoughts together15:47
stephenfinsorry, I think https://review.opendev.org/c/openstack/pbr/+/1002287 isn't needed. I've abandoned it.15:47
stephenfin*I now think15:47
stephenfinWe need to avoid loading anything to do with pyproject.toml parsing unless we can be sure we can support it15:48
clarkback no need for a comment then15:48
clarkbstephenfin: ya I think if we simply avoid that path unless it is supported by the platform then we should be ok (and that means toml parsing would be available)15:49
clarkbstephenfin: and the rough criteria for that are setuptools>=64.0.0 and pyproject.toml file present in the package repo? (I'm mostly thinking ahead to what documentation updates may be needed too)15:51
fungiwe already have that requirement somewhat documented15:52
clarkbI guess python3.7 or newer as well15:52
clarkbbut importantly PBR can't be expected to magic a working environment. You either start with one or have the build-system requires in pyproject.toml handle it for you15:53
clarkbwhich is maybe not super well documented? I'm not sure15:53
clarkbI need to head out on the school run now. One last thought before I go: for testing maybe we try to add a test job/framework that has a small number of representative packages and it cross multiplies them against specific python versions using pyenv or whatever and specific setuptools versions15:56
clarkbthere is something along these lines in the unittests just with setuptools versions but I think the stdlib stuff matters here based on these results15:56
clarkbso have a job that iterates through python3.7 + setuptools foo and bar then 3.8 etc15:56
clarkbor different jobs for each. But if we keep the representative set of packages small that should give us good coverage while making debugging not terrible15:56
fricklerkolla stable/2025.1 doesn't have pyproject.toml. so not sure what they (we) were doing wrong there?16:02
opendevreviewTakashi Kajinami proposed openstack/tooz master: Use etcd from distribution  https://review.opendev.org/c/openstack/tooz/+/99791916:14
clarkbfrickler: oh yup I missed the branch selection in the git clone earlier16:37
clarkbfrickler: so I think the only issue is that pyproject.toml code was running when it shouldn't have. https://review.opendev.org/c/openstack/pbr/+/1002286 should hopefully fix that path16:38
opendevreviewTakashi Kajinami proposed openstack/oslo.db master: Add job to run tests with real database backends  https://review.opendev.org/c/openstack/oslo.db/+/100235016:44
opendevreviewTakashi Kajinami proposed openstack/oslo.db master: Drop test with invalid flush_on_nested option  https://review.opendev.org/c/openstack/oslo.db/+/100235216:48
stephenfinclarkb: I'm wondering if we could simplify https://review.opendev.org/c/openstack/pbr/+/1002286 to check for Python 3.11 and skip everything if not present? πŸ€”17:00
stephenfinmy rationale being that OpenStack only supports Python 3.11 or later, so we can insist that anyone going down the pyproject.toml-only route must also drop support for Python 3.10 or earlier (which they should already have done)17:02
stephenfinthings like git-review will need to keep their setup.cfg / setup.py files but I think that's okay17:03
cardoeSo then pbr supports Python 2.6/2.7 and then 3.11+?17:03
clarkbmore things than just git review use pbr though17:04
clarkbI think ansible does too? or maybe some specific ansible packages?17:04
clarkbcardoe: the new releases of PBR are attempting to drop python older than 3.7 iirc17:04
stephenfinno, it continues to support all versions. You just need to keep setup.cfg / setup.py around if you want to support Python < 3.1117:04
stephenfinclarkb: also no: mordred figured a way around that. We're still testing on 2.717:05
clarkbstephenfin: we also have to think about projects that are on new python without pyproject.toml but those should be trivially supported since we're just failing on imports17:05
clarkbstephenfin: oh ok17:05
cardoeoh I think that makes sense17:05
clarkbstephenfin: what is the oldest version of python that setuptools 64 supports? (I don't know that we need to align but we're already requiring that for pyproject.toml support so maybe that is informative)17:05
stephenfinpython 3.7 https://pypi.org/project/setuptools/64.0.0/17:06
stephenfinbut as I noted earlier, those old versions of setuptools vendor tomli17:07
stephenfin(in fact current versions still do for Python 3.10 support)17:07
stephenfinI don't think we want to go down that path?17:07
clarkbI'm mostly wondering if it is worth supporting pyproject.toml where setuptools can support it (down to 3.7 I guess)17:10
clarkbbut I guess not doing that simplifies things and the only projects I can think of that might have issues are those you already mentioned and they can use the old packaging17:11
stephenfinIf we do that, we need to figure out how to get toml support in there17:20
stephenfinthe current tomli fallback only works on certain setuptools versions or requires users pre-install that package themselves17:21
opendevreviewStephen Finucane proposed openstack/pbr master: Revert "Set setuptools min in requirements.txt"  https://review.opendev.org/c/openstack/pbr/+/100235517:25
opendevreviewStephen Finucane proposed openstack/pbr master: Only provide PEP-660 entry points on setuptools >= 64.0.0  https://review.opendev.org/c/openstack/pbr/+/100235617:25
opendevreviewStephen Finucane proposed openstack/pbr master: Insist on Python 3.11 for pyproject.toml-only projects  https://review.opendev.org/c/openstack/pbr/+/100235717:25
stephenfinclarkb: fungi: ☝️17:25
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235817:27
clarkband that is a half sketched out "what woudl it look like to try and create test scenarios that explode the python, setuptools, pip, etc matrix17:28
stephenfinnice17:29
clarkbI sort of left the hard part out (which is managing isolated envs for setuptools/pip versions), but for a first pass of do we even want to pursue this I think it shows the rough idea17:32
fungii'll note that if we go that route, it will probably be a long time until we can drop the setup.cfg and setup.py from bindep, which had been my test case for pbr's pyproject.toml support17:33
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235817:38
clarkbfungi: https://zuul.opendev.org/t/openstack/build/4ccfd1c0525c489e906874c6c4b22263 this is hitting the pbr changes from stephenfin any idea why that package is not aroudn anymore?17:46
clarkbspecifically `E: Unable to locate package ksmtuned` from apt-get 17:46
fungilooks like that build was on ubuntu-noble17:48
fungihttps://packages.ubuntu.com/ksmtuned claims it's included on noble17:48
fungihttps://zuul.opendev.org/t/openstack/build/4d2c1ce338d54931be188b0f13ec5da4/ has an apt update failure17:50
fungimaybe one of our mirror servers is broken?17:50
clarkbthey both ran in rax flex dfw3 so ya maybe17:51
clarkbalso is it weird that the devstack issue didn't fail on the ap-tget update?17:51
tkajinamI suspect devstack does not set +e17:57
tkajinamand no explicit return code check is implemented in its install_package func17:57
tkajinamI found a few job failures caused by unstable mirror so yeah I suspect that. ex. https://review.opendev.org/c/openstack/python-glanceclient/+/999283/comments/ca93c0ae_761ff87317:58
* tkajinam noticed the same is being discussed in matrix18:01
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235818:28
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235821:24
clarkbfungi: stephenfin ^ the biggest gotcha I've run into doing that has to do with build isolation in pip and build. In particular we want to test using PBR in a speculative state. So I have a whole install PBR into the venv step from the zuul checkout. Then use no build isolation flags. Thats fine except that if a package has extra build depends other than setuptools and pbr we21:25
clarkbwon't be handling those cases21:26
clarkbI'm wondering if anyone knows of a good way to modify these isolated builds specifically to test libs like pbr. I mean setuptools and flit etc must have a solution to this? Or maybe they are just doing what I'm doing and not testing with isolation when they want to test themselves21:26
clarkbanyway I don't think that is a huge deal because PBR implies setuptools (for now anwyay) and I am not aware of any projects with additional build deps21:27
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235821:31
fungii really don't know since i've not closely observed the development process in those projects21:34
clarkbya the docs don't seem to give any clues, but maybe if I dig into their test frameworks something will pop up21:35
clarkbanyway I think if we ignore that problem then for any given python version we can curate a set of scenarios against real packages using various setuptools and pip versions21:36
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235821:49
clarkbok I think that is working now. Its not super different from the existing testingthat we have. I think what sets it apart is that its very explicitly controlling the python, setuptools, and pip versions where we try to run tests on platforms we care about and jsut accept what they give us in other jobs22:05
clarkbif we think this is valuable I can clean it up and we can work on a list of scenarios. I do think what this gives us for the future is the ability to say "X broke after this release" capture that as a new scenario and ensure that it does fail then test against that22:05
clarkbwhich is more difficutl with the existing test jobs22:05
clarkbbasically I'm hopeful that this gives us a way to capture an ongoing set of regression corner cases and ensure they are happy going forward22:06
clarkbthough the test case for jammy with setuptools 59.6.0 succeeded and I'm not basing this on top of the fixes so maybe we need extra installation checks? Currently it uses build to make an sdist, make a wheel, and then does a pip install22:11
clarkbI see the problem. Installing build installs tomli which amkes it work22:12
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235822:16
opendevreviewTakashi Kajinami proposed openstack/oslo.log master: Remove duplicated optional dependencies from test requirements  https://review.opendev.org/c/openstack/oslo.log/+/100200222:37
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235822:42
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235823:08
opendevreviewClark Boylan proposed openstack/pbr master: WIP Sketch out a more generic integration test for PBR  https://review.opendev.org/c/openstack/pbr/+/100235823:23
clarkbfungi: stephenfin  it is interesting how many tools end up relying on other tools without explicit dependencies even with the modern toolchains. Like pip needs wheel installed to build a wheel and will apparently fail in some situations if it can't do that23:24
clarkbseparately build pulls in tomli making all these problems go away23:24
clarkbits amazing any of this ever works from one computer to another23:24
fungiyes, also all of them at one point or another work around those problems by just vendoring in copies of some of their dependencies23:43
opendevreviewMerged openstack/oslo.db master: Do not install code to build release notes  https://review.opendev.org/c/openstack/oslo.db/+/100218023:55

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!