| opendevreview | melanie witt proposed openstack/devstack master: DNM testing https://review.opendev.org/c/openstack/devstack/+/900262 | 00:50 |
|---|---|---|
| stephenfin | clarkb: fungi: Giving the increasing cadence of packaging related changes in the Python ecosystem, at what point do we think about migrating OpenStack to one of the other installers that has widespread adoption? | 10:36 |
| frickler | stephenfin: given that the current situation seems to still be broken, IMO "now" would be a sensible answer | 11:07 |
| opendevreview | Luigi Dino Tamagnone proposed openstack/tempest master: Adding a new config option in tempest.conf as the prefix for the random name resources https://review.opendev.org/c/openstack/tempest/+/897231 | 11:33 |
| fungi | stephenfin: frickler: i'm not sure what you mean by installer, the installer with the widest adoption is pip, then probably conda, followed by distribution package managers | 13:46 |
| fungi | if you mean build backend, the build backend with the widest adoption is still setuptools, by far | 13:47 |
| fungi | if you mean migrating to a new build backend that lots of people talk about even if it's not as widely used, i don't know that any of them solve wsgi scripts in editable mode, that would require some vetting and someone putting together a poc i think | 13:49 |
| sean-k-mooney | fungi: the installation of the package scripts is doen by pip not the backend | 14:35 |
| sean-k-mooney | fungi: what i was chatting to stephenfin about this morning is just goign to setuptools_scm and removing our use of the wsgi_scripts extenion by adding console_scripts and wsigi files in each repo | 14:36 |
| sean-k-mooney | fungi: effectivly limiting oursele to the funcitonlity purly form setuptools | 14:37 |
| sean-k-mooney | we could looke at alreternives too | 14:37 |
| sean-k-mooney | but at this point this is much larger then "just make centos work in a venv" | 14:38 |
| fungi | sean-k-mooney: yes, that's why i said stephenfin's question was confusing, we already use the most popular installer (pip). keep in mind that pbr currently does more than just version number calculation, so we would either need to find alternative plugins for those other features too (or stop relying on them) | 14:43 |
| stephenfin | Yup, my question was poorly put. A better question would be should we investigate using alternative build backends and migrating OpenStack away from pbr | 14:44 |
| fungi | also erplacing pbr would have non-negligible impact on downstream consumers of our projects, particularly distribution package maintainers, who may be relying on pbr-specific means of performing version injection and the like | 14:44 |
| fungi | i'm not opposed to it, just wary that it may be more work than people realize | 14:45 |
| stephenfin | which would involve figuring out what pbr does and what the equivalent functionality, in any, is in other build backends | 14:45 |
| sean-k-mooney | fungi: the problem is without doign somithign like have installer invoke "pbr install_scripts" or similar | 14:48 |
| fungi | the main ones i'm aware we rely on are authors and changelog file generation. also the last time i looked, setuptools-scm doesn't version things the same way pbr does, lacks support for the semver hinting some projects put in their commit messages, and doesn't separately embed commit id metadata | 14:48 |
| sean-k-mooney | im not sure how we can reconsile editbale installes with pyproject.toml and everything that impleis | 14:48 |
| sean-k-mooney | fungi: ya so that si also partly why stephenfin was suggestign if we stayed with pbr for all that goodness | 14:49 |
| fungi | on a more philosophical level, i consider "editable" installation to be a terrible idea, and it's unfortunate that we rely so heavily on it | 14:49 |
| sean-k-mooney | we might need to implemnte our own suprrrot for pep 660 | 14:49 |
| sean-k-mooney | fungi: without it i dont see how devstack could continue to be useful | 14:50 |
| sean-k-mooney | fundementaly to me editabel installs are what make that tool useful | 14:50 |
| fungi | devstack would just need a "commit phase" where edits are reinstalled into its venvs (also devstack should be using venvs) | 14:50 |
| sean-k-mooney | well it is using venvs now | 14:50 |
| fungi | oh, that got implemented finally? | 14:51 |
| sean-k-mooney | and no i mean as a use fo devstack if i need to do more then just restart the systemd service | 14:51 |
| sean-k-mooney | that is a deal breakre for me | 14:51 |
| sean-k-mooney | fungi: yep for debian based distors | 14:51 |
| sean-k-mooney | fungi: that what started this it does not work for centos9stream/rhel and presumable fedora | 14:51 |
| fungi | what if the service restart reran pip install? pip caches just about everything so that shouldn't be all that slow | 14:52 |
| sean-k-mooney | fungi: that woudl be less discrtive to the workflow so we added a prestart service entry to do pip install then sure | 14:52 |
| fungi | anyway, i haven't caught up on e-mail yet this morning, so i suspect i'm missing context on why the pep 660 support isn't working | 14:53 |
| sean-k-mooney | to me the public interface for "get new code to work" is systemctl restart devstack@* | 14:53 |
| fungi | guess i'll switch to catching up on e-mail | 14:53 |
| sean-k-mooney | fungi: simple answer is pep 660 support is working but pep-660 is for pure libs and explcit does not cover datafiles or scripts | 14:53 |
| sean-k-mooney | fungi: they did nto provide a standardise way to support scrtips other then console_scripts or gui_scripts | 14:54 |
| fungi | oh, well sure, python packaging isn't "supposed" to cover any of that, according to the people writing the specifications | 14:54 |
| sean-k-mooney | fungi: so this is why its not a problem with the build system | 14:54 |
| fungi | again, i don't think replacing pbr will solve that, the standards authors have intentionally said it's out of scope | 14:54 |
| sean-k-mooney | its a probelm with the installer i.e. pip | 14:54 |
| sean-k-mooney | the changed the interface form "setup.py devleop" to importing setup tools and invokeing moduels | 14:55 |
| sean-k-mooney | without provideing any hookpoint we can overload | 14:55 |
| fungi | the official answer from the python packaging community is that if you need to install anything that isn't a basic library, use a different packaging solution. wheels/sdists pyproject.yaml and pip target library use cases. application use cases are poorly served by them | 14:56 |
| sean-k-mooney | yep thats the crux of the issue | 14:57 |
| sean-k-mooney | openstack is a set of applciations and libs | 14:57 |
| fungi | they basically don't want to have anything to do with files that need to be installed outside the module path | 14:57 |
| sean-k-mooney | so we can solve the wsgi_script issue by ebeding the wsgi_script in the lib and just installing it as a console_script | 14:57 |
| frickler | maybe devstack could place the pip install into the systemd service definition if doing that separately is what bothers you? | 14:58 |
| fungi | so from that perspective, yes we could for example switch to conda which does support installing more than just libraries | 14:58 |
| sean-k-mooney | frickler: it would have to yes | 14:58 |
| sean-k-mooney | frickler: editabel installs are a fundemental part of how devstack works | 14:58 |
| sean-k-mooney | provided we can keep the edit and continue workflow workign without reqiuring you to run pip install -c </path/to/constriats> <my repo> | 14:59 |
| sean-k-mooney | that woudl be ok | 15:00 |
| frickler | sean-k-mooney: well ... yes and no, you still need to know which service to restart ... and if you know to do that, one could also argue you know which pip install to run | 15:00 |
| frickler | and 99% of devstack is running in CI, where that does not matter | 15:00 |
| sean-k-mooney | frickler: supporting editbale installs for this usecase was one of the requirement for proceeding with global_venv | 15:01 |
| frickler | but if it turns out that they are infeasible, maybe some alternative approach can be found | 15:01 |
| sean-k-mooney | the alternitive would be to revert the global_env i supprot for most peopel | 15:02 |
| frickler | like create an update.sh script that finds changed repos, does pip reinstalls and service restarts | 15:02 |
| sean-k-mooney | no because you dont nessiarly want to restart all services | 15:02 |
| sean-k-mooney | i would be fine with just doing the pip install in a pre_start action in the systemd unit | 15:03 |
| sean-k-mooney | then we can use non editbael installs and keep the gloabl_venv | 15:03 |
| sean-k-mooney | if peopel didn not hate that idea i could try creating a patch for that | 15:03 |
| sean-k-mooney | if we do that we dont need to do any of the pyproject.toml stuff either although we still can | 15:04 |
| frickler | I thought that that was still needed for future compatibility? | 15:06 |
| fungi | keep in mind that pep 660 was sort of created under duress, the python packaging community mostly considers editable installs an ugly hack and an unfortunate misfeature that they're now stuck figuring out how to support going forward | 15:06 |
| fungi | as such, relying on editable installs is likely to become increasingly painful regardless of what solution we do or don't adopt | 15:07 |
| stephenfin | Do we know for sure that wsgi scripts will continue to work for non-editable installs also? | 15:12 |
| stephenfin | I mean, we're hitting it with editable installs but that's likely because we're going down wholly new codepaths where distutils is not a thing. Won't the non-editable codepaths also change before long, given distutils is gone in 3.12? | 15:12 |
| fungi | well, distutils isn't gone, it has moved into setuptools | 15:15 |
| fungi | ot | 15:15 |
| fungi | it's just no longer in the stdlib | 15:15 |
| stephenfin | Not yet, fwict. I don't see distutils vendored into the main branch, though I do see a feature/distutils-sync branch | 15:17 |
| fungi | pretty sure it's been in there for a while. there's even been a switch to tell setuptools not to use its embedded version of distutils | 15:17 |
| fungi | which we had to turn on at one point, if memory serves | 15:18 |
| stephenfin | Ah, indeed. It's 'setuptools/_distutils'. Apologies | 15:18 |
| stephenfin | Are you aware whether the setuptools maintainers have indicated when, if ever, they'll remove support for the 'install' command? | 15:19 |
| fungi | i haven't seen any discussion of timelines for that, no | 15:20 |
| stephenfin | ack | 15:21 |
| fungi | stephenfin: pfmoore replied to your d.p.o post asking why we'd changed our minds about just overloading console_scripts and pointing wsgi to those. did we run into a fundamental problem trying that approach? | 15:31 |
| clarkb | fwiw you could also have uwsgi invoke scripts out of the library path and not do anything special right? | 15:48 |
| clarkb | console_scripts are special because we want things to end up in users' PATH entries | 15:48 |
| sean-k-mooney | the scripts are not generated now | 15:49 |
| clarkb | but uwsgi doesn't really care about that we feed it a file path | 15:49 |
| sean-k-mooney | clarkb: the issue is we dont have scripts for wsgi in nova | 15:49 |
| sean-k-mooney | we use pbr to generate it | 15:49 |
| sean-k-mooney | so in the ediabel case there is nothing in the lib path to execute | 15:50 |
| clarkb | right so whether or not you write out those files instead of generating them is a bit orthogonal to PBR | 15:50 |
| clarkb | just add the file to nova, then it ends up in the nova install path and uwsgi loads that directly | 15:50 |
| clarkb | that is independent of the build backend | 15:50 |
| sean-k-mooney | yep we could | 15:50 |
| sean-k-mooney | thats why we were wondieinf if we shoudl stop using wsgi_scripts and do that | 15:50 |
| sean-k-mooney | then that leed to the question what else do we need pbr to do | 15:51 |
| sean-k-mooney | adn is it doing it properly now | 15:51 |
| clarkb | I think fungi listed those. Mostly the versioning stuff | 15:52 |
| sean-k-mooney | yep | 15:53 |
| sean-k-mooney | so i think we have a few paths forward | 15:53 |
| sean-k-mooney | first we can propse moving the wsgi script to files in teh repos | 15:53 |
| sean-k-mooney | second we can make devestack to pip install on start of the sytemd service | 15:54 |
| sean-k-mooney | thid we can proceed with pyproject.toml using pbr | 15:54 |
| kopecmartin | does anyone know if there is a way how to get/generate OpenAPI/swagger definition of the Openstack API? I found only this abandoned patch: https://review.opendev.org/c/openstack/api-site/+/286659 | 17:04 |
| kopecmartin | gmann maybe? ^ | 17:04 |
| clarkb | kopecmartin: gtema has emails about it on the openstack-discuss list | 17:05 |
| * kopecmartin is gonna check | 17:05 | |
| kopecmartin | thanks clarkb | 17:05 |
| stephenfin | clarkb: the console scripts stuff generates a files with an if __name__ == '__main__' block, inside which you've a call to whatever callable you pointed at from you build config file | 17:37 |
| stephenfin | We need a file that has an 'application' callable inside it, not wrapped in '__main__' | 17:37 |
| clarkb | stephenfin: yes and then writes that into the bin/ dir associated with teh current environment iirc | 17:37 |
| stephenfin | yup | 17:37 |
| clarkb | the idea being you can isntall python commands easily | 17:37 |
| stephenfin | indeed | 17:37 |
| clarkb | (and in PBRs case it overrides the default implementation becuse the default is slow) | 17:38 |
| clarkb | I guess I was trying to say for an application callable you don't need it in your PATH or a bin/ dir | 17:38 |
| clarkb | so you can just have the file install like any other file into the library path and have uwsgi call into that | 17:38 |
| stephenfin | right, but it appears at some point we will no longer have the mechanism to place arbitrary files into the library path from setuptools | 17:39 |
| stephenfin | and that this is already the case if using the pep-660 editable wheel stuff | 17:39 |
| clarkb | onyl if you are autogenerating them at package build time | 17:39 |
| clarkb | if they already exist on disk (autogenerated and committed to git for example) then that should be reliable over time | 17:40 |
| clarkb | otherwise setuptools would stop installing anything | 17:40 |
| clarkb | another option would be to supply it externally. I just checked our django mailman3 setup and that is what they do | 17:40 |
| clarkb | that seems far more error prone for users though | 17:41 |
| stephenfin | You sure? What's the mechanism for that? If we had e.g. an bin/nova-api-wsgi script in nova, how does one indicate that in setup.cfg / pyproject.toml | 17:41 |
| stephenfin | Historically you could use the distutils 'scripts' option but that's not supported by PEP-621 | 17:42 |
| clarkb | stephenfin: pbr (and setuptools_scm) install all the things checked into git by default. I don't know what the end result path would be for that specific bin prefiex | 17:42 |
| clarkb | but it doesn't have to be a script it could be nova/wsgi/wsgi.py then your wsgi application would be in venv/lib/python3.10/site_packages/nova/wsgi/wsgi.py or whatever the end result is | 17:42 |
| stephenfin | are we talking about separate things? I suspect you're talking about libraries installed into /lib rather than executables installed in /bin | 17:43 |
| stephenfin | ? | 17:43 |
| clarkb | correct becuase you don't need the wsgi application glue code to be in bin/ | 17:43 |
| stephenfin | at the moment we (and most deployment tooling) is expecting an executable | 17:43 |
| stephenfin | absolutely, but all our deployment tooling is using that | 17:43 |
| stephenfin | at least based on a codesearch.o.o. search for nova-api-wsgi | 17:44 |
| clarkb | right it would require the deployment tooling to update their uwsgi.ini to point at the new location | 17:44 |
| clarkb | I'm not saying it is a drop in replacement. I'm saying its a replacement that would be viable with changes to setuptools over time | 17:44 |
| stephenfin | right, we're on the same page now | 17:44 |
| stephenfin | so what we're saying is it's likely that pbr will need to lose this functionality | 17:45 |
| stephenfin | unless we fancy reimplementing swathes of pbr-specific code to support this | 17:45 |
| clarkb | if we treat editable installs as a top tier feature I guess (goes back to fungi indicating that seems to be where a lot of the upstream contention is) | 17:45 |
| stephenfin | repeating what I said to fungi earlier | 17:46 |
| stephenfin | Do we know for sure that wsgi scripts will continue to work for non-editable installs also? | 17:46 |
| stephenfin | I mean, we're hitting it with editable installs but that's likely because we're going down wholly new codepaths where distutils is not a thing. Won't the non-editable codepaths also change before long, given distutils is gone in 3.12? | 17:46 |
| clarkb | I haven't tested it myself, but supposedly those hooks still exist? | 17:46 |
| stephenfin | fungi corrected me that distutils is being vendored by setuptools now | 17:46 |
| clarkb | the specific problem here is they redid how editable installs work and this is fallout from that | 17:47 |
| clarkb | I guess setuptools could stop vendoring distutils | 17:47 |
| stephenfin | but given the deprecation warnings on the install command stuff, I suspect that will disappear from setuptools before too long also | 17:47 |
| stephenfin | it seems clear they want to simplify it and make it easier to maintain, which is understandable (though obviously a pain for us) | 17:47 |
| clarkb | in theory you can use pyproject.toml to pin setuptools until you end up on a python interpreter that can't run that code. Which seeing how things have gone would probably be 3.13 :) | 17:49 |
| clarkb | as far as porting to setuptools_scm goes I think it could be made to work. Its not straightforward even for trivial problems (ssbarnea converted some project and it took like 3 releases to get it working right) | 17:50 |
| clarkb | s/problems/projects/ | 17:50 |
| clarkb | The main thing would be acceptance of lost functionality and acceptance of following the upstream direction (whcih we already largely have to do) | 17:51 |
| fungi | clarkb: stephenfin: wheels can still contain non-python files, for example i use pbr as a pep-517 build backend and put sample configuration for test fixtures in the wheel so that running tests from a wheeled installation still works | 17:51 |
| clarkb | another approach would be to use peotry | 17:51 |
| clarkb | I think poetry has its own builder with a lot of rich functionality that pypa intentionally avoids | 17:51 |
| stephenfin | fungi: where does it place those files? | 17:54 |
| stephenfin | once installed in a venv, for example | 17:54 |
| clarkb | unfortunately I suspect whichever path we take we'll find something changes direction and forces us to go through a similar process again. Thats kinda how we ended up with pbr in the first place because distutils2 got killed and then setuptools implemented big chunks of it that made it into PBR later after telling us that PBR was a mistake but they made sure to do it | 17:55 |
| clarkb | differntly... | 17:55 |
| fungi | stephenfin: relative to the path in the wheel itself. wheels just get unpacked in-place | 17:57 |
| fungi | myvenv/lib/pythonX.Y/site-packages/mypackage/tests/fixtures/testconfig.yaml | 17:58 |
| stephenfin | Gotcha. Per above discussion with clarkb though, /lib is not /bin and all installers are currently expecting WSGI scripts placed /bin | 18:04 |
| stephenfin | *placed in /bin | 18:04 |
| *** tosky_ is now known as tosky | 18:43 | |
| sean-k-mooney | im finisinsh not but just to recap i cna confirm that for non editiabel installs the weehsl end up with the wsgi script in a scripts folder at teh root of the wheel | 19:21 |
| sean-k-mooney | adding 'nova-28.1.0.dev39.data/scripts/nova-api-wsgi' | 19:24 |
| sean-k-mooney | those cscipt get put in /bin in the venv | 19:26 |
| sean-k-mooney | it is only in thge bin dir | 19:27 |
| sean-k-mooney | [19:25:19]??? find .venv-test/ -name nova-metadata-wsgi | 19:28 |
| sean-k-mooney | .venv-test/bin/nova-metadata-wsgi | 19:28 |
| sean-k-mooney | what we likely could try to do | 19:28 |
| sean-k-mooney | is put the generated script in the editabel wheel in the same way | 19:28 |
| sean-k-mooney | i.e try to explicty wrihte to nova-28.1.0.dev39.data/scripts/ | 19:29 |
| sean-k-mooney | form the build_editiable hook | 19:29 |
| sean-k-mooney | in pbr | 19:29 |
| sean-k-mooney | specifically here https://github.com/openstack/pbr/blob/master/pbr/build.py#L82-L92 | 19:30 |
| sean-k-mooney | i am on pto tomorrow but i might try hacking this to at least see if that works. | 19:33 |
| sean-k-mooney | this is using https://peps.python.org/pep-0491/#the-data-directory | 19:46 |
| sean-k-mooney | based on https://peps.python.org/pep-0491/#recommended-installer-features | 19:48 |
| sean-k-mooney | installers shoudl also implemnnt #!python rewrites for any script in {distribution}-{version}.data/scripts/ | 19:49 |
| sean-k-mooney | menaing pip shoudl rewrite the #! line to the correct one for the venv for us | 19:49 |
| sean-k-mooney | these sepcific feature were aslo suported in wheel format 1.0 https://peps.python.org/pep-0427/ | 19:51 |
| sean-k-mooney | clarkb: fungi frickler is we just modify the editable wheel and append the wsgi_scripts into it like that and follow the wheel spec i think that will support our current usecases | 19:53 |
| fungi | sounds worth a try, thanks for lookig into it (again)! | 19:54 |
| sean-k-mooney | at the very least i want to see fi we cna ease the transition path | 20:03 |
| sean-k-mooney | regardelss fo what we end up doing | 20:03 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!