| *** darmach5 is now known as darmach | 12:55 | |
| sean-k-mooney | fungi: clarkb did i see talk about using spdx license specfics instead of classifer recently | 17:12 |
|---|---|---|
| sean-k-mooney | since we have had all the issue with not using pyproject.toml recently | 17:12 |
| clarkb | sean-k-mooney: https://opendev.org/opendev/bindep/src/branch/master/pyproject.toml#L47-L51 | 17:13 |
| sean-k-mooney | as im setting up the new grian-ui repo im takign the time to do so with pyproject.toml form day one | 17:13 |
| sean-k-mooney | ya it might have been that that i saw | 17:13 |
| clarkb | bindep is acting as our pyproject.toml methods test case. And yes, but we haven't changed yet because apparently older python can't understand it | 17:13 |
| sean-k-mooney | so i have a workign version for the new repo using setuptools-scm instead of pbr currently | 17:14 |
| clarkb | keep in mind that you lose a number of features when you do that. I think in many cases that is probably fine but its worth being aware of | 17:15 |
| sean-k-mooney | https://paste.opendev.org/show/bGwBlwDlf69UT4eNIGpY/ | 17:15 |
| sean-k-mooney | yes i had it using pbr first | 17:15 |
| sean-k-mooney | but i was having issue also makeing it work with src layout | 17:15 |
| sean-k-mooney | so i said i woudl try this | 17:15 |
| sean-k-mooney | what im really wondering is do we see openstack movign away form setuptools any time soon | 17:16 |
| sean-k-mooney | i.e. to hatchling or uv | 17:16 |
| clarkb | things like version number generation being aware of commit message content labels (something that has created problems in the past and not something I personally think is ciritical), wsgi scripts installation goes away (but is half broken anyway), autogeneration of authors files?, sphinx integration (something that goes away when you use pyproject.toml anyway), and probably | 17:16 |
| clarkb | more | 17:16 |
| fungi | sean-k-mooney: there's also a debate raging on discuss.python.org right now about the "correct" use of that metadata, and i suspect the end result will be even more churn | 17:17 |
| sean-k-mooney | well setuptools-scm can do dynmic version number | 17:17 |
| sean-k-mooney | based on git tags | 17:17 |
| clarkb | sean-k-mooney: it can do dynamic versions but not incrementing based on commit messages iirc | 17:17 |
| clarkb | sean-k-mooney: pbr will look at your most recent tag and then increase the version appropriately based on commit message content | 17:17 |
| sean-k-mooney | right we cant force a bump | 17:17 |
| sean-k-mooney | yep | 17:17 |
| sean-k-mooney | clarkb: by the way i have not settled on not using pbr | 17:18 |
| sean-k-mooney | i started with it | 17:18 |
| clarkb | sean-k-mooney: oh I think its fine. And I've been hoping someone would try it. I think most of the pbr functionality you lose isn't critical and being able to not maintain pbr forever would be nice | 17:18 |
| fungi | it's been pointed out that project information != package information, and so for example projects that embed/vendor dependencies in the packages they distribute on pypi will want to deeclare different (or at least additional) package licenses from whatever the project license is | 17:19 |
| sean-k-mooney | well that why i was debating starting without it | 17:19 |
| clarkb | I doubt openstack would move to uv or hatchling any time soon. setuptools-scm is a much smaller leap and any changes are going to be small and incremental if history is an indiciation | 17:19 |
| sean-k-mooney | i have been talkign to stephenfin about this on and off for a year or two | 17:19 |
| clarkb | but I think a lot of people are not aware of the various pbr features taht do exist so its good to remind people | 17:19 |
| sean-k-mooney | well bacially since the whole wsgi_srcript problem was found with pbr 6 | 17:19 |
| clarkb | sean-k-mooney: the primary feature that I think would need to remain is the ability to record the git hash of the commit the package was built from | 17:20 |
| clarkb | pbr does this by writing out a metadata file and if you do pbr list instead of pip list it will emit that info alongside the package version number | 17:20 |
| clarkb | in addition to git based versions and git based file inclusion I mean | 17:20 |
| clarkb | its a small but important feature for tracing the source of a package that pep440 broke by not providing a mechanism for this in the version itself | 17:21 |
| fungi | well, also pbr's version calculation functionality is more feature-rich than setuptools-scm's | 17:21 |
| sean-k-mooney | https://paste.opendev.org/show/bF8bnULgoJLL7WwRsVFB/ | 17:21 |
| sean-k-mooney | im not sure if that is all of it but that a reasonabel output | 17:21 |
| sean-k-mooney | ah ok | 17:22 |
| clarkb | sean-k-mooney: that doesn't include the source commit hash | 17:22 |
| sean-k-mooney | i was not aware of pbr list | 17:22 |
| clarkb | so you can't know what commit the package was built from | 17:22 |
| sean-k-mooney | let me see if just swapign back to pbr will work now that i have filed out the rest of the file | 17:22 |
| clarkb | fungi: ya but as long as you explicitly tag versions (like openstack does for every release now) that is basically a non issue I think | 17:23 |
| fungi | more that some projects rely on hinting dev version prediction using commit message footers | 17:23 |
| clarkb | oh are projects doing that these days? | 17:28 |
| sean-k-mooney | ya so its a little more work to get pbr enabeld again because i will have ot undo moving everythign form setup.cfg to pyproject.toml | 17:28 |
| clarkb | sean-k-mooney: you don't have to undo that | 17:28 |
| sean-k-mooney | fungi: clarkb not any i have worked on | 17:28 |
| sean-k-mooney | clarkb: if i dont define a setup.cfg it does not add the sha info | 17:29 |
| clarkb | sean-k-mooney: https://opendev.org/opendev/bindep/src/branch/master/setup.cfg you need a very minimal setup.cfg | 17:29 |
| clarkb | right you need a setup.cfg but its 2 lines | 17:29 |
| clarkb | not a wholesale undo | 17:29 |
| sean-k-mooney | oh ok | 17:29 |
| clarkb | like needing a minimal setup.py | 17:29 |
| fungi | this should all be clear in the latest pbr docs too, i tried to make sure of that | 17:30 |
| sean-k-mooney | so it build but still no sha | 17:32 |
| sean-k-mooney | anyway this is not a blocker | 17:33 |
| sean-k-mooney | oh do i alos need the minimal setup.py | 17:34 |
| sean-k-mooney | ok ya | 17:34 |
| sean-k-mooney | so it works | 17:34 |
| fungi | sean-k-mooney: the automated release changes use sem-ver headers btw, not sure how many humans do though... https://review.opendev.org/c/openstack/nova/+/944890 | 17:36 |
| fungi | s/headers/footers/ | 17:36 |
| sean-k-mooney | that about the only place i have seen them ya | 17:37 |
| sean-k-mooney | i may have done it once | 17:37 |
| fungi | that helps ensure that .dev versions will target the next minor release version instead of the last version | 17:37 |
| fungi | so that they'll sort later than contemporary dev versions on stable branches | 17:39 |
| sean-k-mooney | ack, typically i us a tool to boostrap repos like this, but since many peropel are off today i wanted to do it by hand following the python packaging guide since its been a while | 17:40 |
| clarkb | sean-k-mooney: yes you need a setup.py beacuse if you don't supply one setuptools generates a default one that doesn't have pbr=true set. This is true even if you set pbr=true in setup.cfg/pyproject.toml for setuptools | 17:40 |
| sean-k-mooney | yep | 17:40 |
| clarkb | I think it has to do with the way pbr hooks into setup() is actually an old distutils thing and setuptools ignores it when running out of pyproject.toml | 17:41 |
| clarkb | but even if you use pyproject.toml you're still running a setup.py one way or another so its not a big deal | 17:41 |
| clarkb | I should say pyproject.toml with setuptools | 17:41 |
| sean-k-mooney | well bar any funkyness with the versions | 17:41 |
| sean-k-mooney | i now have a repo that works with or without pbr and generage something reasonable | 17:41 |
| sean-k-mooney | so if we ever discontinue pbr | 17:41 |
| sean-k-mooney | moving to setuptool-scm shoudl be trivial | 17:42 |
| sean-k-mooney | release tooling aside. | 17:42 |
| fungi | yes, the idea with the state and direction of our example in bindep's packaging is that it's relying on very little of pbr | 17:43 |
| fungi | so weaning off of it from that state would be fairly painless | 17:43 |
| fungi | opendev/engagement is simpler still since it doesn't need older python compatibility | 17:44 |
| sean-k-mooney | well new project just need 3.10+ | 17:45 |
| sean-k-mooney | 3.9 for oslo kindof | 17:46 |
| sean-k-mooney | sure we need to supprot older things for stable | 17:46 |
| sean-k-mooney | so tools like bindep are a bit more constriaed | 17:46 |
| sean-k-mooney | by the way i need ot evntually teach it that debian testing is just debian... | 17:47 |
| fungi | right, and even there it's mostly "modern" except for a few things that need versions of pip and setuptools that only work on python 3.9 and later | 17:47 |
| fungi | sean-k-mooney: i thought i fixed that? | 17:48 |
| sean-k-mooney | maybe i havenet tried it in a while | 17:48 |
| sean-k-mooney | https://review.opendev.org/c/opendev/bindep/+/932192 | 17:49 |
| sean-k-mooney | i wonder if that is released | 17:49 |
| sean-k-mooney | 2.12.0 | 17:49 |
| sean-k-mooney | hum | 17:49 |
| sean-k-mooney | oh it might be my fault | 17:50 |
| fungi | it might also be my fault if the fix was incomplete | 17:50 |
| fungi | i mostly use sid/unstable, and noticed the problem there, but in theory testing should be more or less the same strings | 17:51 |
| sean-k-mooney | so it works if i use a venv with system pyton on sid | 17:51 |
| fungi | do you have an /etc/os-release file? | 17:52 |
| sean-k-mooney | i do im wondering if it was broken in the past because i was using other python verison form nix | 17:53 |
| sean-k-mooney | via nix home-manager | 17:54 |
| sean-k-mooney | it seam to be workign perfectly fine for me now | 17:54 |
| fungi | oh, nix sandboxes into a chroot so yeah, your outer /etc may be shadowed by it | 17:54 |
| sean-k-mooney | it seam to be working with 3.11 form nix too | 17:54 |
| fungi | okay cool | 17:54 |
| sean-k-mooney | my guess is i just had an old verion in some of my venvs | 17:55 |
| sean-k-mooney | on of the reasons i was thinking about uv | 17:55 |
| sean-k-mooney | is like pyenv it can provide many pyton versoin but unliek it it will pull prebuilt ones | 17:56 |
| clarkb | I guess the alternative to changes like https://review.opendev.org/c/openstack/nova/+/944890 would be to make more tags | 17:56 |
| sean-k-mooney | i was wonderign if we could use that in our jobs at some point where we need to test newer version then are in ubuntu | 17:56 |
| clarkb | but ya I don't think there is any easy way around that pbr version bumping if we're relying on it. May require setuptools-scm to become smarter or find alternatives (like more tags) | 17:56 |
| clarkb | sean-k-mooney: you already can for python with pyenv | 17:57 |
| clarkb | sean-k-mooney: https://review.opendev.org/c/zuul/zuul/+/945521 is an example | 17:57 |
| sean-k-mooney | i tought pyenv alwasy compiled | 17:57 |
| clarkb | it does but it only takes a couple minutes for a non optimized build | 17:57 |
| sean-k-mooney | clarkb: on i know we can do it in ci | 17:57 |
| clarkb | and the runtime for non optimized builds are reasonable | 17:58 |
| fungi | sean-k-mooney: yeah, the challenge with non-lts distros is the constant churn and breakage, particularly for image building. once upon a time we had suse tumbleweed for example and it was similarly problematic. even fedora and non-lts intermediate ubuntu versions were too hard to keep up | 17:58 |
| sean-k-mooney | i was suggestign we coudl maybe speed up those jobs by using uv to provide the 3.13 build | 17:58 |
| clarkb | ya I mean you can do that too. There is uv support in zuul-jobs | 17:58 |
| clarkb | I just don't think it is critical as pyenv seems to work. But maybe uv will work a bit faster and people will be happier | 17:59 |
| clarkb | fungi: ya and now we're struggling with the major releases too (centos 10 stream) | 17:59 |
| sean-k-mooney | i got devstack to mostly work with "uv pip" last year but at the tiem it dint really like our constratis files | 17:59 |
| clarkb | it still isn't clear to me if uv pip supports constraints | 17:59 |
| fungi | i think they prefer their custom lockfile format | 18:00 |
| sean-k-mooney | i havent treid sicne then | 18:00 |
| sean-k-mooney | for us the issue iw we need a consitent set across repo | 18:00 |
| sean-k-mooney | which unfortuenly most of the modeern tools dont cater for | 18:01 |
| fungi | you could in theory preprocess the dependencies with pip install -c and then pip freeze and feed the resolved freeze file into uv | 18:01 |
| clarkb | re optimized python builds vs not having similar performance my hunch is that when building unoptmized from scratch the compile is able to make use of local features that the generic optmized builds can't use for wide applicability and the result is performance that comes out in a wash | 18:01 |
| sean-k-mooney | oh you mean like -macrch native | 18:01 |
| clarkb | sean-k-mooney: and things like sse | 18:01 |
| clarkb | I suspect the generic builds like distros have to be more cautious. Whereas pyenv is running ./configure and it should configure those bits | 18:02 |
| sean-k-mooney | i havn't really been following any of the jit workin in 3.13+ but i suspect doign taht at runtim is goign to be more common goign forward | 18:02 |
| fungi | cpython has a build optimization mechanism where it compiles once, runs a simplified version of its `make test` under profiling, then compiles everything again optimized based on the profiling results | 18:02 |
| clarkb | ya thats the slow process. The first compile takes like 2 minutes on our test nodes. Then more than an hour to do the extra steps | 18:03 |
| sean-k-mooney | ah so the same approch as autotools | 18:03 |
| sean-k-mooney | or cmake | 18:03 |
| fungi | correct | 18:03 |
| fungi | so the added time spent doing an optimized build outweighs the performance gains for the job run with it | 18:04 |
| sean-k-mooney | right that why i was suggetign that perhaps using uv to provide pre built binaries might not be terribel. | 18:06 |
| clarkb | sure. I'm just saying that in practice building it seems fine | 18:07 |
| clarkb | you get a python that is basically the same performance as a prebuilt python | 18:07 |
| clarkb | and the cost to build is relative small | 18:07 |
| clarkb | zuul-jobs already knows how to manage uv. You would just need to expand ensure-python to use it to fetch python as a pyenv alternative to test it | 18:08 |
| sean-k-mooney | ack. i wasnt wored about the perforamce of the resultant python, more the time/reliablity in a job | 18:08 |
| sean-k-mooney | to be fair | 18:08 |
| sean-k-mooney | i dont think i have ever seen the pyenv based jobs fail to build | 18:08 |
| sean-k-mooney | so im not that concerned | 18:09 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!