| opendevreview | Brian Haley proposed openstack/devstack master: Add support for IPv6 tunnel endpoints https://review.opendev.org/c/openstack/devstack/+/710519 | 01:27 |
|---|---|---|
| *** ykarel_ is now known as ykarel | 04:51 | |
| *** pojadhav|afk is now known as pojadhav | 05:10 | |
| opendevreview | Rajat Dhasmana proposed openstack/tempest master: Add test to rebuild volume backed instance https://review.opendev.org/c/openstack/tempest/+/831018 | 05:58 |
| *** akekane_ is now known as abhishekk | 06:46 | |
| *** jpena|off is now known as jpena | 07:36 | |
| opendevreview | Merged openstack/devstack master: Add Ubuntu 22.04 LTS (jammy) platform job https://review.opendev.org/c/openstack/devstack/+/839359 | 08:16 |
| opendevreview | mitya-eremeev-2 proposed openstack/tempest master: Tempest can be aware of pre-provisioned networks https://review.opendev.org/c/openstack/tempest/+/831784 | 09:59 |
| ykarel | frickler, jammy job devstack broken with https://review.opendev.org/c/openstack/devstack/+/839820 | 10:44 |
| frickler | ykarel: meh :( hopefully dansmith can fix that | 11:01 |
| ykarel | yes let's wait for him | 11:06 |
| lajoskatona | kopecmartin: Hi, just a quick question regarding https://review.opendev.org/c/openstack/tempest/+/821732 , is it something that can fit into current tempest to have serial test group? | 11:17 |
| frickler | kopecmartin: gmann: actually, seeing the results in https://review.opendev.org/c/openstack/devstack/+/842127/ , it seems we made a mistake by making the jammy jobs voting but not run them in gate. would you prefer to make them non-voting for now or revert the perfdata patches until they can be fixed for jammy? | 11:31 |
| kopecmartin | lajoskatona: it looks interesting, i can imagine it could help also optimize the test execution (because not all tests would have to be run serially if some of the tests require so) | 11:43 |
| kopecmartin | but i need to have a closer look, test it a bit | 11:43 |
| kopecmartin | frickler: i think we can make the jammy jobs non-voting for now | 11:44 |
| lajoskatona | kopecmartin: ok, thanks for checking | 11:47 |
| frickler | kopecmartin: o.k., I'll build a patch | 11:53 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Make jammy platform jobs non-voting https://review.opendev.org/c/openstack/devstack/+/842532 | 11:56 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Drop openEuler support https://review.opendev.org/c/openstack/devstack/+/842534 | 12:07 |
| frickler | kopecmartin: if you could have a quick look at https://review.opendev.org/c/openstack/devstack/+/842532 I would move it to gate directly | 12:10 |
| frickler | also I didn't see your mail about openeuler before submitting my patch, I'll wip it for now | 12:12 |
| kopecmartin | frickler: no problem, i sent it quite late .. we can wait a day to see if anyone reached out and then let's go ahead with the patch | 12:14 |
| ykarel | frickler, just curios how https://zuul.openstack.org/status#842532 running in both check and gate together | 12:35 |
| frickler | ykarel: I enqueued it manually into gate to skip waiting a bit | 12:41 |
| ykarel | frickler, okk got it | 12:42 |
| ykarel | frickler, also noticed too many jobs running unnecessary there | 12:42 |
| ykarel | even if jammy jobs modified | 12:42 |
| ykarel | it can be improved to run only jobs which modified but would need some refactor | 12:43 |
| frickler | ykarel: that's always a difficult thing, in particular when jobs are also being run/inherited by other projects, since file-matchers then become mostly useless | 12:45 |
| ykarel | frickler, i think that shouldn't be an issue, as modified job is auto included irrespective of file matchers | 12:48 |
| ykarel | we doing it in neutron and it worked fine | 12:49 |
| ykarel | frickler, https://review.opendev.org/c/openstack/neutron/+/827615 | 12:50 |
| ykarel | can you check if we missed some case with that | 12:50 |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration https://review.opendev.org/c/openstack/tempest/+/828686 | 13:08 |
| dansmith | ykarel: no glob module in jammy? that seems wrong... | 13:21 |
| ykarel | dansmith, i checked normal import works fine, but not with pip build env | 13:22 |
| ykarel | i mean python3 -c "from glob import glob" works fine | 13:22 |
| dansmith | ykarel: that blows my mind a little.. glob is distributed with python, AFAIK | 13:23 |
| dansmith | I'll get a jammy vm to poke at | 13:23 |
| ykarel | dansmith, one workaround that worked for my local tests was pip install --no-build-isolation | 13:23 |
| ykarel | that may help in tracing it further | 13:23 |
| dansmith | huh, okay | 13:24 |
| dansmith | how urgent? should we just throw that flag on there? | 13:25 |
| ykarel | dansmith, currently those jobs are being moved to non-voting | 13:40 |
| ykarel | so not that urgent | 13:40 |
| dansmith | ack | 13:40 |
| ykarel | but will continue fail until it's fixed and local deployments should also fail | 13:40 |
| ykarel | have a workaround though MYSQL_GATHER_PERFORMANCE=False | 13:41 |
| opendevreview | Merged openstack/devstack master: Make jammy platform jobs non-voting https://review.opendev.org/c/openstack/devstack/+/842532 | 14:08 |
| dansmith | ykarel: pip install of that module does not fail for me on jammy locally | 14:23 |
| ykarel | dansmith, when i reproduced, i tried same what was failing in devstack | 14:25 |
| ykarel | sudo -H LC_ALL=en_US.UTF-8 SETUPTOOLS_USE_DISTUTILS=stdlib http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite python3.10 -m pip install -c /opt/stack/requirements/upper-constraints.txt /opt/stack/devstack/tools/dbcounter | 14:25 |
| ykarel | with ^ it reproduced for me, u tried also same? | 14:25 |
| dansmith | dan@jammy:~/devstack/tools$ pip install dbcounter/ | 14:25 |
| dansmith | but not sure how that's different | 14:26 |
| dansmith | still works as root | 14:26 |
| ykarel | good to confirm if with above command it reproduces for you | 14:26 |
| dansmith | I'll have to stack on this new vm before I have all the other stuff like u-c | 14:27 |
| ykarel | u-c can use web url | 14:27 |
| dansmith | the above works without the -c, but yeah, hang on | 14:28 |
| dansmith | ykarel: https://paste.opendev.org/show/bwhMhQuC6TKl7GOm95FT/ | 14:30 |
| dansmith | completely fresh jammy install, still warm, just "apt-get install python3-pip" before I ran that | 14:31 |
| dansmith | so I'm puzzled | 14:31 |
| ykarel | strange then likely devstack doing something, i have devstack installed where i tried this | 14:32 |
| dansmith | ack, I'm in a meeting, then another meeting after, but will stack | 14:33 |
| dansmith | but I mean, glob is built into python so I can't even imagine what the deal is | 14:33 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Add wait for server SSH-able in base attach_volume method https://review.opendev.org/c/openstack/tempest/+/842240 | 14:53 |
| opendevreview | Caique Mello proposed openstack/tempest master: WIP Fix compare volume stats for storage_protocols https://review.opendev.org/c/openstack/tempest/+/842432 | 15:29 |
| *** jpena is now known as jpena|off | 16:31 | |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration https://review.opendev.org/c/openstack/tempest/+/828686 | 16:41 |
| dansmith | clarkb: you often know about these things -- does this problem (last comment) ring a bell? | 17:00 |
| dansmith | in jammy I can install that no problem, but after devstack runs on jammy, pip fails to install that tiny module | 17:01 |
| dansmith | like, pip seems different after the install | 17:01 |
| dansmith | but glob is a python builtin so I have no idea why it's complaining | 17:01 |
| clarkb | devstack will install latest pip and setuptools iirc | 17:01 |
| clarkb | could be that jammy system pip is just old enough to not have problems compared to latest? | 17:01 |
| dansmith | it doesn't fail on focal after devstack tho | 17:02 |
| clarkb | hrm that is odd | 17:02 |
| dansmith | but seriously.. glob is built-in, why would it be complaining anyway? | 17:02 |
| clarkb | is the glob built in maybe split out into a python-something package and jammy doesn't install it by default but older distros do? | 17:02 |
| clarkb | for example python3-venv is separated by the distro into a different package | 17:03 |
| dansmith | but glob is so fundamental, and I can import it in python itself no problem | 17:03 |
| clarkb | do you have a link to the failure? | 17:03 |
| dansmith | oh, sorry I forgot to paste :D | 17:04 |
| dansmith | https://review.opendev.org/c/openstack/devstack/+/839820 | 17:04 |
| dansmith | hence the "last comment" above | 17:04 |
| dansmith | actually | 17:05 |
| dansmith | that's complaining about some installed thing, not my code | 17:05 |
| dansmith | in /usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py which does import glob (like I do, so it seemed like it was my problem) | 17:05 |
| dansmith | wtaf | 17:06 |
| clarkb | https://github.com/pypa/pip/issues/8060 is similar but for python2 | 17:07 |
| clarkb | but seems like maybe related to isolation stuff whcih might break finding glob and may apply here? | 17:08 |
| dansmith | but but.. it's GLOB :) | 17:08 |
| dansmith | ykrel said --no-build-isolation fixes it | 17:09 |
| dansmith | which I mean, I can do, but I don't understand why I should need that, especially since it seems to be something that pip libs are breaking on | 17:09 |
| clarkb | I think the build isolation flag may isolate even the std stuff. Either beacuse it is packaged separately by the distros or ends up in a path that isolated python envs can't find or because they want a limited set of tools smaller than stdlib python | 17:13 |
| clarkb | fungi: ^ you may know? | 17:14 |
| dansmith | but it's some pip library that is using glob, so I have to add glob to my deps because pip requires it, just so pip can install me? | 17:14 |
| fungi | trying to page in all this context, just a moment | 17:15 |
| clarkb | well glob is in stdlib python so I'm not sure how you would request that | 17:15 |
| clarkb | but ya I think there has been efforts towards isolating builds so that they express their needs. And maybe they got overzealous? | 17:15 |
| dansmith | right, there's a glob2 which does more, but even adding that doesn't fix it, so I don't think it's related to my deps | 17:15 |
| clarkb | I s uspect it is an interaction with debian/ubuntu's python packaging and that isolation | 17:16 |
| fungi | what's the exact error message about glob missing? | 17:16 |
| fungi | link to a stack trace maybe? | 17:16 |
| fungi | or log? | 17:16 |
| dansmith | fungi: https://paste.opendev.org/show/bnKGPHSiMReyuqohrb9a/ | 17:17 |
| fungi | thanks! | 17:17 |
| dansmith | works in focal, works in jammy, fails in jammy after devstack updates pip (or something) | 17:17 |
| fungi | i need to dig into how dbcounter is packaged just to be sure i'm not overlooking something | 17:18 |
| dansmith | clarkb: fwiw, pip freeze does not show pip installed and pip is /usr/bin/pip | 17:18 |
| fungi | pip list should show pip version info | 17:18 |
| dansmith | fungi: I went the toml approach thinking I was being modern | 17:19 |
| fungi | pip freeze backward-incompatibly omits things | 17:19 |
| clarkb | https://zuul.opendev.org/t/openstack/build/2279572f42a24728bd556ef10d106697/log/job-output.txt#11245-11262 | 17:19 |
| clarkb | linking to lines works now :) | 17:19 |
| dansmith | https://paste.opendev.org/show/bkQjr4kWSZ3o25bMVdOs/ | 17:19 |
| dansmith | not much difference | 17:19 |
| fungi | dansmith: you're in luck, i wasted hours last night trying to move a pbr-based package from setup.cfg to the new setuptools support for metadata in pyproject.toml | 17:19 |
| dansmith | fungi: pretty much having trouble thinking of my situation as being lucky today, but.. good? :) | 17:21 |
| fungi | well, at least i'm used to looking at the new pip and setuptools error messages about it, since i was endeavoring to do it with all warnings raised as errors | 17:22 |
| fungi | my own personal flavor of masochism, i guess | 17:23 |
| dansmith | heh | 17:23 |
| opendevreview | Merged openstack/devstack master: Configure placement section in neutron conf https://review.opendev.org/c/openstack/devstack/+/842127 | 17:23 |
| fungi | dansmith: so... i'm able to install it into a venv with python 3.10 on debian/unstable | 17:25 |
| dansmith | fungi: I am in jammy too, until I have run devstack, then not | 17:25 |
| fungi | oh, neat | 17:25 |
| clarkb | and the only thing that I can think of that devstack does is update pip and setuptools and all that | 17:26 |
| clarkb | oh! | 17:27 |
| clarkb | local sudo_pip="sudo -H LC_ALL=en_US.UTF-8 SETUPTOOLS_USE_DISTUTILS=stdlib " | 17:27 |
| clarkb | maybe that is why | 17:27 |
| dansmith | clarkb: did you see that my pip seems unupdated? | 17:27 |
| dansmith | clarkb: nope, bare pip fails too | 17:27 |
| clarkb | speficailly we're overriding SETUPTOOLS_USE_DISTUTILS=stdlib | 17:27 |
| dansmith | clarkb: see my paste | 17:27 |
| dansmith | https://paste.opendev.org/show/bnKGPHSiMReyuqohrb9a/ | 17:27 |
| clarkb | ah ok | 17:27 |
| fungi | if you make a venv and install it into the venv in that state instead of globally, does it work? | 17:31 |
| fungi | just wondering if it's a behavior specific to running pip as root | 17:32 |
| clarkb | apparently we don't update pip to latest on devstack anymore with focal and jammy | 17:32 |
| clarkb | (I think devstack shouldn't do that and should always consistentyl install the same version of pip from upstream but hey what do I know) | 17:32 |
| dansmith | fungi: yes | 17:33 |
| dansmith | fungi: so must be something else we've installed on the system? | 17:33 |
| clarkb | looks like frickler changes that beacuse running get-pip.sh twice in a stack/unstack/stack scenario broke | 17:34 |
| fungi | dansmith: either something (un?)installed in the system, or pip &co behave differently when run as root for some reason | 17:35 |
| clarkb | we install setuptools globally with devsatck and that uses the constraints version | 17:36 |
| clarkb | a new virtualenv would use the bundled setuptools in the virtualenv (which would be the distro version if using python3 -m venv iirc) | 17:36 |
| clarkb | maybe something along those lines | 17:36 |
| dansmith | fungi: fails as normal user as well | 17:36 |
| fungi | with pip install --user you mean? | 17:37 |
| dansmith | fungi: defaults to that if non-root now | 17:37 |
| dansmith | clarkb: installed setuptools==59.6.0 into my venv, still works | 17:38 |
| fungi | oh, i'd missed that, but yeah that's a useful data point | 17:38 |
| fungi | setuptools 59.6.0 is what you see qith sudo pip list? | 17:38 |
| dansmith | pip list, but yeah | 17:38 |
| fungi | s/qith/with/ | 17:38 |
| fungi | in the system installation i mean (not in the venv). okay so rules out the setuptools version then | 17:39 |
| dansmith | yar | 17:39 |
| dansmith | this is some hot garbage, huh? | 17:39 |
| fungi | it's just another day in troubleshooting the python packaging ecosystem, unfortunately | 17:39 |
| dansmith | isn't that what I said? :) | 17:40 |
| clarkb | if you do `python3 then import glob from glob` does that work? | 17:40 |
| clarkb | without a venv. basically can we reproduce without pip | 17:40 |
| dansmith | clarkb: yes, that's what I was saying earlier.. glob is clearly there | 17:40 |
| dansmith | dan@jammy:~/devstack$ python3 -c 'from glob import glob' | 17:40 |
| clarkb | ok so it is somethign to do with how pip does isolation. Maybe we need to understand that then compare against what devstack changes | 17:40 |
| dansmith | ^ no error | 17:40 |
| fungi | just not when system pip is using build isolation | 17:40 |
| dansmith | and --no-build-isolation works | 17:40 |
| fungi | and python3 on jammy is 3.9? | 17:43 |
| fungi | libpython3.9-minimal seems to ship /usr/lib/python3.9/glob.py so i guess we need to work out why /usr/lib/python3.9 isn't in sys.path during the build | 17:43 |
| dansmith | 3.10 | 17:44 |
| dansmith | dan@jammy:~/devstack$ python3 -V | 17:44 |
| dansmith | Python 3.10.4 | 17:44 |
| fungi | s/9/10/ in that case, i checked both since i have the packages for them | 17:44 |
| dansmith | it has to be something related to python3.10 because this does not fail on focal | 17:44 |
| fungi | `/usr/bin/python3.10 -c 'import sys;print(sys.path)'` definitely indicates that '/usr/lib/python3.10' would normally be in there | 17:45 |
| clarkb | right thats why I wonder about their siolation method | 17:46 |
| clarkb | are they manipulating sys.path? seems likely and we need to figure out how? | 17:46 |
| opendevreview | Merged openstack/devstack stable/yoga: lib/tempest: add wait for Glance image import https://review.opendev.org/c/openstack/devstack/+/842390 | 17:51 |
| fungi | i put this in tools/dbcounter/setup.py: | 17:54 |
| fungi | import setuptools, sys | 17:54 |
| fungi | setuptools.setup() | 17:54 |
| fungi | print(sys.path) | 17:54 |
| fungi | then use pip install --verbose | 17:54 |
| fungi | and it does echo a copy of the sys.path list for the build | 17:55 |
| fungi | includes things like site-packages/pip/_vendor/pep517/in_process and /tmp/pip-build-env-q6wvor67/site | 17:55 |
| fungi | i don't happen to have a post-devstack jammy system handy to be able to reproduce the install error and see what the system path is at that point | 18:03 |
| fungi | might swizzle around the order of the print and setup calls if the problem arises during the latter | 18:04 |
| dansmith | fungi: did you delete the setup.cfg and toml? because putting that in setup.py doesn't seem to get executed | 18:06 |
| fungi | i did not delete the setup.cfg and pyproject.toml, no | 18:07 |
| dansmith | durnent seem to do anything for me | 18:08 |
| dansmith | https://paste.opendev.org/show/bxXWZoFaeeJJn7FLJRZK/ | 18:09 |
| fungi | https://paste.opendev.org/show/814496/ | 18:10 |
| fungi | that's what `pip install --verbose tools/dbcounter` into a venv looks like for me, at least | 18:10 |
| dansmith | so much with the different | 18:11 |
| fungi | so under those circumstances it does seem to call setup.py | 18:11 |
| clarkb | I wonder if there is a way to tell pip to keep the isolated environment for debugging | 18:12 |
| dansmith | I want to know what it is about that package and not anything else that breaks | 18:25 |
| dansmith | can't be because it's local, because the projects are local | 18:26 |
| dansmith | removing setup.cfg and the toml makes it (a) print the path from setup.py and (b) install just fine (albeit unnamed) | 18:27 |
| dansmith | must be triggering something different in pip if you go the toml route or something | 18:27 |
| clarkb | I think the toml path is the pep517 path. otherwise you are legacy | 18:30 |
| clarkb | my hunch is that everything else being installed is legacy and bypasses this entirely | 18:30 |
| dansmith | once again, punished for trying to do the right thing :P | 18:31 |
| clarkb | looking at pip source it also only does that when doing the wheel deps not the install deps | 18:33 |
| opendevreview | Dan Smith proposed openstack/devstack master: Convert dbcounter to legacy setup.py https://review.opendev.org/c/openstack/devstack/+/842609 | 18:34 |
| dansmith | fehxd ^ | 18:35 |
| clarkb | dansmith: do you know what version of pip is breaking (its the jammy version which is 22.0.2)? | 18:42 |
| dansmith | pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10) | 18:43 |
| clarkb | https://github.com/pypa/pip/blob/22.0.2/src/pip/_internal/build_env.py should be what creates the isolated env then | 18:43 |
| dansmith | I guess I could try updating to upstream pip and see if that changes anything | 18:44 |
| clarkb | the diff between version 22.1 and 20.0.2 in that file is fairly large. That seems like a reasonable next thing to test | 18:45 |
| dansmith | same deal | 18:46 |
| clarkb | https://github.com/pypa/pip/blob/22.0.2/src/pip/_internal/build_env.py#L89 jumps out to me | 18:47 |
| dansmith | I purge'd python3-pip, ran install_pip.sh, which got me 22.1 | 18:47 |
| clarkb | my hunch looking at that code is whatever location glob is at is being explicitly removed by that sitecustomize script | 18:50 |
| clarkb | dansmith: if you want to be really hacky you can modify build_env.py to write out a sitecustomize.py that prints out info as it goes? (I think getting pdb attached might be weird since that is a script invoked by a subprocess) | 18:51 |
| clarkb | specifically modify https://github.com/pypa/pip/blob/22.0.2/src/pip/_internal/build_env.py#L101-L124 to print out info about itself so that as it runs in a sub python we get info? | 18:52 |
| dansmith | but glob should be right next to lots of other site things, and it's the first thing in that script so I would assume everything else is going to fail too.. let me try something | 18:53 |
| clarkb | ya its possible that glob is just a canary and that any stdlib entry would break because its path was removed | 18:53 |
| dansmith | so, sys doesn't fail | 18:54 |
| clarkb | weird | 18:54 |
| * dansmith and this is sys.path right before import glob: ['/usr/local/lib/python3.10/dist-packages/pip/_vendor/pep517/in_process', '/opt/stack/keystone', '/opt/stack/glance', '/opt/stack/cinder', '/opt/stack/neutron', '/opt/stack/nova', '/opt/stack/placement', '/tmp/pip-build-env-q2zcbsqp/overlay/local/lib/python3.10/dist-packages', '/tmp/pip-build-env-q2zcbsqp/normal/local/lib/python3.10/dist-packages'] | 18:54 | |
| clarkb | dansmith: I think the /tmp paths would provide both sys and glob | 18:55 |
| clarkb | so maybe the issue is furhter up the stack and sys.path is appropriate but the venv is missing stuff | 18:55 |
| dansmith | https://paste.opendev.org/show/bACLo1FJ8PVLmEKasYlk/ | 18:55 |
| dansmith | first line is sys.path. subsequent lines are os.listdir(d) for d in sys.path | 18:55 |
| dansmith | none of those have things like os or sys in them, yet they were importablt | 18:57 |
| dansmith | *importable | 18:57 |
| clarkb | I wonder if python knows how to find some of those stdlib entries based on where the python executable is running from just to bootstrap istelf | 18:58 |
| dansmith | dan@jammy:~/devstack$ ls /usr/lib/python3.10/{sys,os,glob}* | 18:58 |
| dansmith | /usr/lib/python3.10/glob.py /usr/lib/python3.10/os.py /usr/lib/python3.10/sysconfig.py | 18:58 |
| dansmith | no sys, but os and glob are right next to each other | 18:58 |
| dansmith | sys is special, but os is not: | 18:59 |
| dansmith | https://paste.opendev.org/show/bHIM3HSGhttVPKDSDqdL/ | 18:59 |
| dansmith | but still, I can import os, so.. | 18:59 |
| dansmith | preventing the listdir of the missing dir shows me that none of them contain os.py | 19:01 |
| dansmith | so I don't see why it would be find-able but glob would not | 19:01 |
| clarkb | my hunch is that stdlib things that python itself may rely on get special treatment and those end up ebing importable with path setting them (or something like that) | 19:03 |
| clarkb | but glob isn't in that class so the isolated env pip uses without a sys.path including a path to it breaks | 19:03 |
| clarkb | this would potentially explain the focal vs jammy difference if python3.8 and 3.10 handle this differently | 19:03 |
| clarkb | I've got a python3.8 locally on tumbleweed. Let me try unsetting sys.path and importing glob | 19:05 |
| clarkb | dansmith: if I do import sys; sys.path = []; import sys ; import os ; import glob it doesn't fail until import glob | 19:05 |
| clarkb | this is with python3.8 but also maybe more up to date python3.8 than focal? It does seem like os is special | 19:06 |
| dansmith | bbut, os.__file__ is a real file | 19:06 |
| clarkb | ya but my hunch is the interpreter itself is relying on it | 19:06 |
| clarkb | so it gets loaded early before we manipulate path to be empty | 19:07 |
| clarkb | or there is some bypass etc | 19:07 |
| clarkb | I need to pause for lunch. Back in a bit | 19:07 |
| clarkb | but my next thing is to try doing that pip install locally with tumbleweed 3.8 to see if it fails for me too | 19:08 |
| dansmith | even if you're right, I don't know how this is the fault of the package being installed, or what we're supposed to do | 19:21 |
| dansmith | I can't put "glob" in the build requirements and it would be rather smelly to have to list pip's own requirements in there | 19:21 |
| dansmith | so if you're just saying "this is why it's not working but it's clearly a bug in pip's isolation" then .. I guess but I'm surprised their issue tracker isn't blowing up with that level of broken | 19:22 |
| opendevreview | Brian Rosmaita proposed openstack/devstack stable/xena: lib/tempest: add wait for Glance image import https://review.opendev.org/c/openstack/devstack/+/842404 | 19:23 |
| clarkb | dansmith: yes Ithink that is what I'm saying. I haven't been able to prove that is the case yet but that is my hunch | 19:26 |
| clarkb | but I suspect it may only happen if you somehow convince that script to remove more than it should? something like that | 19:27 |
| clarkb | dansmith: ok hacking up pip locally I made pip not delete the temp dir so that I could inspect it directly | 19:44 |
| clarkb | dansmith: that gave me access to the sitecustomize.py file | 19:44 |
| clarkb | what this line https://github.com/pypa/pip/blob/22.0.2/src/pip/_internal/build_env.py#L107 expands to is removed from the path. On my local machine it keeps ['/tmp/pip-build-env-swd1xnmt/site', '/usr/lib/python38.zip', '/usr/lib64/python3.8', '/usr/lib64/python3.8/lib-dynload'] in the path but I suspect on jammy for whatever reason we're removing the /usr/lib stuff because they | 19:45 |
| clarkb | end up in that list of removals | 19:45 |
| clarkb | dansmith: if you modify https://github.com/pypa/pip/blob/22.0.2/src/pip/_internal/build_env.py#L74 to call TempDirectory with delete=False passed in then the dir isn't deleted | 19:46 |
| clarkb | Maybe do that and check what sitecustomize.py looks like to see if we're adding the system paths to the removal list and ultimately removing them? | 19:46 |
| clarkb | I guess I can spin up a jammy container and give it a go too | 19:48 |
| clarkb | oh except it seems like devstack is a necessary side effect to cause it and we don't know what it does to be a rpbolem yet? fun | 19:49 |
| dansmith | yeah which is so bizarre | 19:51 |
| clarkb | in my jammy container install completes, and sitecustomize.py removes /usr/local/lib/python3.10/dist-packages from path | 19:55 |
| clarkb | _USE_SYSCONFIG_DEFAULT = sys.version_info >= (3, 10) that might be why python3.8 is fine (I don't know what it means yet though) | 20:00 |
| clarkb | that is from pip/src/pip/_internal/locations/__init__.py and affects get_purelib() | 20:01 |
| opendevreview | Dan Smith proposed openstack/devstack master: Convert dbcounter to legacy setup.py https://review.opendev.org/c/openstack/devstack/+/842609 | 20:03 |
| clarkb | dansmith: what does python3 -c 'import sysconfig; print(sysconfig.get_paths())' return on the broken setup? | 20:05 |
| clarkb | I think pip uses the platlib and purelib values of that result to pass to sitecustomize.py | 20:06 |
| clarkb | if we see the regular stdlib locations in those paths we would know the mechanism at least | 20:06 |
| dansmith | {'stdlib': '/usr/lib/python3.10', 'platstdlib': '/usr/lib/python3.10', 'purelib': '/usr/local/lib/python3.10/dist-packages', 'platlib': '/usr/local/lib/python3.10/dist-packages', 'include': '/usr/include/python3.10', 'platinclude': '/usr/include/python3.10', 'scripts': '/usr/local/bin', 'data': '/usr/local'} | 20:16 |
| clarkb | ok that looks fine. I'm so confused | 20:18 |
| clarkb | dansmith: it does seem like your sys.path is being manipulated to not have stuff like /usr/lib/python3.10 in it based on your printing of sys.path earlier. But seems like sitecustomize.py may not be the reason for it | 20:25 |
| dansmith | yeah | 20:31 |
| opendevreview | Clark Boylan proposed openstack/devstack master: DNM debugging jammy pip install stuff https://review.opendev.org/c/openstack/devstack/+/842620 | 20:36 |
| clarkb | I'm holding the jammy node so I can see this in action a bit better | 20:39 |
| dansmith | I have to pick this up later, but will do some more hacking of my local libs and maybe try to spawn myself a shell or something | 20:40 |
| clarkb | ok I think at least part of the problem is that sys.path order is not stable and they assume it is with https://github.com/pypa/pip/blob/22.0.2/src/pip/_internal/build_env.py#L111 | 21:27 |
| clarkb | its not stable after the addsitedir call just prior | 21:27 |
| clarkb | dansmith: I think the devstack side effect is doing an editable install | 21:31 |
| dansmith | editable install of what? | 21:31 |
| clarkb | doing editable installs inserts an easy-install.pth in /usr/local/lib/python3.10/dist-packages/ which gets parsed by addsitedir() in sitecustomize.py | 21:31 |
| clarkb | dansmith: of all the openstack packages (that is how keystone swift nova etc end up in the path) | 21:31 |
| dansmith | ack, I see them there | 21:31 |
| clarkb | the addsitedir of that .pth updates the sys.path and I think it is doing it in a non order preserving way which may break the later path mangling in sitecustomize.py. But I don't quite understand the mechanism yet if that is the cause | 21:32 |
| clarkb | import sys; new = sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p = getattr(sys, '__egginsert', 0); sys.path[p:p] = new; sys.__egginsert = p + len(new) <- that monstrosity in the .pth file might be to blame? | 21:32 |
| clarkb | its possible that pip here is relying on an assumption that python3.10 on jammy breaks when you have editable installs | 21:33 |
| dansmith | that's a pretty cynical view and I hope you're wrong | 21:35 |
| dansmith | this sitecustomize monkeybusiness also seems really hacky to me, so I'm really surprised to see it embedded in pip like this | 21:36 |
| dansmith | let me try to slowly uninstall the editables to see if the last one makes it work | 21:37 |
| clarkb | ok I think I understand the assumption better. I have no idea if it is a good one yet. The assumption seems to be that addsitedir will only append to sys.path and that is what the slice on len(original_sys_path) is doing | 21:39 |
| dansmith | so.I uninstalled everything except the last editable (placement) and it works | 21:39 |
| dansmith | I should have one them one at a time and checked, but I think that means it's not just *any* editable (although maybe a bunch increases the likelihood if it's an ordering thing) | 21:40 |
| clarkb | er I may have that backwards but ya it assumes that entries are being determinstically added on one end or the other | 21:40 |
| dansmith | but what about that breaks finding glob | 21:40 |
| dansmith | ? | 21:40 |
| dansmith | I assume the edtiables are supposed to show up earlier in the path (and do for me) to take precedence, because it's for "development" | 21:41 |
| clarkb | it is removing the regular stdlib location in sitecustomize.py via their path mangling | 21:41 |
| clarkb | and it seems to do with how editable installs affect sys.path via the .pth file (and sitecustomize.py maybe having bad assumptions) | 21:41 |
| dansmith | so you mean it's lossy or buggy but not just that the order is different | 21:41 |
| dansmith | sitecustomize is the python blob embedded in the pip library right? | 21:42 |
| clarkb | ya I think different orders of sys.path during sitecustomization.py mangling causes it to ultimately remove entries ti shouldn't | 21:42 |
| dansmith | that's the thing that seems .. hacky to me | 21:42 |
| clarkb | yup its that embedded blog | 21:42 |
| clarkb | *blob | 21:42 |
| * dansmith shakes his head | 21:42 | |
| clarkb | ['/tmp/pip-build-env-3zogqb0w/site', '/opt/stack/keystone', '/opt/stack/swift', '/opt/stack/glance', '/opt/stack/cinder', '/opt/stack/neutron', '/opt/stack/nova', '/opt/stack/placement', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages'] <- that is sys.path when | 21:43 |
| clarkb | sitecustomization.py starts. | 21:43 |
| clarkb | actually let me start an etherpad | 21:43 |
| dansmith | actually, if it's trying to be such a clean room, why is it even including most of what is in sys.path anyway? | 21:44 |
| dansmith | why isn't it saying "this is what your sys.path is, not $PATH - $some stuff" | 21:44 |
| clarkb | dansmith: https://etherpad.opendev.org/p/kG2bTkne05iyqNmxeacJ | 21:45 |
| clarkb | I added the print statements then put their outputs below them in comments | 21:45 |
| clarkb | to get to that point I modified pip to not delete the tempdir whcih kept the script around then I can run the script manually to see the results | 21:46 |
| dansmith | so you think it's this assumption? sys.path[len(original_sys_path):] | 21:51 |
| clarkb | ya. That seems to assume addsitedir's side effect is appending to the path? Or maybe prepending depending on what info they want | 21:52 |
| clarkb | but instead we seem to get an insert in the middle | 21:52 |
| dansmith | well, they're using an iterating sets | 21:53 |
| dansmith | sys.path should be stable, but the order they're iterating them isn't, but I'm not quite seeing the actual bug yet | 21:53 |
| dansmith | I really have to finish something and get out of here, but I'll be fresh-er on monday to really dig in | 21:53 |
| clarkb | I think they expect all the addsitedir contents to be appended | 21:53 |
| clarkb | dansmith: ya I don't think this is a rush but likely something to run down before we rely on jammy more properly | 21:54 |
| clarkb | they call addsitedir on the specifically identified pure lib and plat lib locations | 21:54 |
| dansmith | yeah, especially if it's going to require a patch to the embedded code in pip :/ | 21:54 |
| clarkb | and then later collect just that set. Then they look at the original sys.path and remove anything that may have been added from the site paths | 21:55 |
| clarkb | so ya I am pretty sure they expect this to be appending | 21:55 |
| clarkb | but it isn't | 21:55 |
| clarkb | on my local python3.8 it does appear to append | 22:01 |
| clarkb | there is an interesting thing though where running sitecustomize out of a venv and running it out of the root of my install produces two different sys.paths as the end | 22:01 |
| clarkb | on python3.8 the easy-install.pth is quite simple and doesn't have that monstrosity I pasted before. It just lists the paths | 22:05 |
| clarkb | removing the extra content from easy-install.pth fixes it on the jammy node | 22:06 |
| clarkb | and they append as expected | 22:07 |
| clarkb | hrm that extra content has been in setuptools for forever | 22:10 |
| clarkb | I wonder why I didn't get it when I did a python3.8 pip install -e. Maybe pip install -e isn't doing a setup.py develop. here we go | 22:10 |
| opendevreview | Clark Boylan proposed openstack/devstack master: DNM debugging jammy pip install stuff https://review.opendev.org/c/openstack/devstack/+/842620 | 22:20 |
| clarkb | this is turning into quite the thread | 22:20 |
| clarkb | that extra content in the .pth file only shows up if we use the rewrite method that ^ removes: https://github.com/pypa/setuptools/blob/main/setuptools/command/easy_install.py#L1679-L1702 | 22:21 |
| clarkb | and this only seems to be an issue if you have that content in the .pth file | 22:21 |
| clarkb | and the reason it doesn't append is the sys.__egginsert value | 22:32 |
| clarkb | I think the bug is something like "Pip's isolated install mode uses a sitecustomize.py value that does not handle legacy easy-install.pth files properly as those legacy files do not append to sys.path. The legacy behavior can be set via SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite" | 22:34 |
| clarkb | I have a hackypatch for pip that seems to fix things. fungi any chance you're willing to push it upstream? | 22:48 |
| clarkb | pip is on my no fly zone list and we share an employer so shouldn't be a prolbme for you to push my work? | 22:48 |
| clarkb | I may be overthinking it | 22:48 |
| opendevreview | Brian Haley proposed openstack/devstack master: Add support for IPv6 tunnel endpoints https://review.opendev.org/c/openstack/devstack/+/710519 | 23:03 |
| clarkb | dansmith: fungi: https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/ | 23:06 |
| clarkb | to be clear I do not plan to make a PR upstream. I don't midn if someone else does. | 23:06 |
| clarkb | also my change that removes the rewrite config seems to get further but still fails on something related to dbcounter module not being found | 23:08 |
| fungi | oof, i get sidetracked by something and there's 150 new lines of discussion about this problem. that can't be a good sign | 23:14 |
| clarkb | fungi: tl;dr is I haven't had a rabbit hole this good in a long time | 23:15 |
| clarkb | I may hold a rebuild of my latest patchset to devstack to see why dbcounter isn't getting hooked up as a sqlaclchemy plugin but I've had enough brain meilting for today | 23:17 |
| fungi | still catching up, but editable installs still need a minimal setup.py per the setuptools docs on moving to pyproject.toml | 23:19 |
| fungi | editable installs aren't a solved problem yet (and the debate on how to solve them wages on in the python discourse) | 23:20 |
| clarkb | I'm not sure if dbcoutner is getting an editable install or not | 23:20 |
| clarkb | I think it isn't | 23:21 |
| fungi | i would have mentioned it earlier but i didn't think editable installs were coming into play (i avoid them like the plague because they're so increasingly poorly supported by the python packaging ecosystem over the past decade) | 23:21 |
| clarkb | the problem is editable installs of otherpackages breaking your ability to install new packages editable or not | 23:21 |
| fungi | my most recent pr for pip has been getting ignored by maintainers for weeks, so i'm not sure i'll have much more luck pushing this fix | 23:24 |
| clarkb | on the node I had held I installed dbcounter by removing the extra info from easy-install.pth and then pkg_resources.iter_entry_points('sqlalchemy.plugins') shows dbcounter, but I'm not sure how to check that better | 23:29 |
| clarkb | I suppose it is possible that we rely on the rewrite flag to properly write out the entrypoints or something | 23:32 |
| clarkb | thus extending the thread. But ya I'm running out of time to eb able to dig into this today. | 23:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!