| opendevreview | Ghanshyam proposed openstack/tempest master: Add new tempest job enable the rbac scope checks and new defaults https://review.opendev.org/c/openstack/tempest/+/614484 | 01:19 |
|---|---|---|
| *** tkajinam is now known as Guest2162 | 02:04 | |
| *** tkajinam is now known as Guest2174 | 03:28 | |
| *** yadnesh|away is now known as yadnesh | 04:03 | |
| *** pojadhav- is now known as pojadhav | 05:03 | |
| *** tkajinam is now known as Guest2179 | 05:36 | |
| *** pojadhav- is now known as pojadhav | 05:57 | |
| opendevreview | Nobuhiro MIKI proposed openstack/devstack master: Skip installing and starting dstat service if it's disabled https://review.opendev.org/c/openstack/devstack/+/865237 | 08:02 |
| *** yadnesh is now known as yadnesh|afk | 08:20 | |
| *** jpena|off is now known as jpena | 08:22 | |
| *** yadnesh|afk is now known as yadnesh | 08:51 | |
| *** jpena is now known as jpena|off | 08:56 | |
| *** jpena|off is now known as jpena | 08:57 | |
| opendevreview | Merged openstack/devstack stable/yoga: Mark centos-9-stream as n-v on stable/yoga https://review.opendev.org/c/openstack/devstack/+/864716 | 09:52 |
| yadnesh | hey folks, can someone help me figure out a ssl issue. I see this while running telemetry tempest tests https://paste.openstack.org/show/bnvuGM6Hi0N9Fd4STSfS/ | 10:58 |
| kopecmartin | #startmeeting qa | 15:00 |
| opendevmeet | Meeting started Tue Nov 22 15:00:14 2022 UTC and is due to finish in 60 minutes. The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'qa' | 15:00 |
| kopecmartin | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 15:00 |
| kopecmartin | agenda ^ | 15:00 |
| lpiwowar | o/ | 15:00 |
| kopecmartin | \o | 15:00 |
| kopecmartin | let's quickly go through the usual | 15:03 |
| kopecmartin | #topic Announcement and Action Item (Optional) | 15:03 |
| kopecmartin | nothing here | 15:03 |
| kopecmartin | #topic Antelope Priority Items progress | 15:03 |
| kopecmartin | #link https://etherpad.opendev.org/p/qa-antelope-priority | 15:03 |
| kopecmartin | i've seen some progress on "Clean up deprecated lib/neutron code" | 15:04 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/1996748 | 15:04 |
| kopecmartin | thank you slaweq for working on that | 15:04 |
| kopecmartin | regarding s-rbac, there has been some progress too , the s-rbac job seems to be passing now \o/ | 15:05 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/614484 | 15:05 |
| kopecmartin | thanks gmann for that patch ^^ | 15:05 |
| kopecmartin | "Use admin clients *only if* admin access needed" | 15:06 |
| kopecmartin | anything new here ^^ arxcruz | 15:06 |
| arxcruz | kopecmartin: waiting feedback from gmann | 15:06 |
| kopecmartin | #link https://review.opendev.org/c/openstack/tempest/+/852377 | 15:07 |
| kopecmartin | but didn't we concluded that we don't need to add those parameters? | 15:07 |
| arxcruz | basically, the code that enable us to use different client is there, once i get a green light, i can start working replacing tests that calls the manager to use non-admin client | 15:07 |
| arxcruz | depending on the test/requirements | 15:07 |
| kopecmartin | well, we know that situation when one test uses demo and the other one admin can't happen, it's either one or the other and then considering the amount of jobs (which is either voting on tempest patches or in general where tempest is used) we can switch only those methods to demo which will work/perform their action successfully all the time, in any case | 15:10 |
| * frickler sneaks in late, reads backlog | 15:11 | |
| arxcruz | yes, that's the idea | 15:11 |
| arxcruz | the actual patch doesn't change anything, but enable us to do that | 15:11 |
| kopecmartin | so that just brings question whether we need that parameter because at the end, the method is using either admin client or demo client | 15:12 |
| kopecmartin | i'm not sure we need to be able to that, that's all i'm saying, but the patch won't hurt | 15:12 |
| kopecmartin | it will add an option to change the client, whether it's needed / or will be needed is tough to answer | 15:13 |
| kopecmartin | moving on | 15:14 |
| kopecmartin | #topic OpenStack Events Updates and Planning | 15:14 |
| kopecmartin | skipping for now | 15:14 |
| kopecmartin | #topic Gate Status Checks | 15:14 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:14 |
| kopecmartin | nothing there, any urgent / gate blocking patch to review? | 15:14 |
| kopecmartin | #topic Bare rechecks | 15:15 |
| kopecmartin | #link https://etherpad.opendev.org/p/recheck-weekly-summary | 15:15 |
| frickler | there might have been https://review.opendev.org/c/openstack/devstack/+/864848 to discuss but I think we've circumvented that now | 15:15 |
| * kopecmartin checking | 15:15 | |
| kopecmartin | btw, recheck status looks good | 15:16 |
| frickler | it won't look so good next time | 15:17 |
| frickler | did an awful lot of rechecks on the devstack-plugin-ceph patch to pin to focal | 15:17 |
| frickler | until I gave up and made it non-voting | 15:17 |
| kopecmartin | frickler: i saw, thank you for rechecking and getting the patches merged | 15:19 |
| kopecmartin | #link | 15:19 |
| kopecmartin | #link https://review.opendev.org/c/openstack/devstack/+/864848 | 15:19 |
| kopecmartin | can it cause any harm? | 15:19 |
| kopecmartin | the project is archived under openstack namespace | 15:20 |
| frickler | well "harm" I'm not sure, it just doesn't match our policy. | 15:20 |
| kopecmartin | i need to check that, i | 15:21 |
| frickler | but also a new version of cursive is now on pypi and https://review.opendev.org/c/openstack/barbican/+/864226 + its dep should solve that urgent issue | 15:21 |
| kopecmartin | i'm not very familiar with that | 15:21 |
| kopecmartin | oh, yeah, released yesterday | 15:21 |
| kopecmartin | so it seems, barbican is no longer blocked by that patch then? | 15:22 |
| frickler | still waiting for CI but I hope so | 15:22 |
| kopecmartin | ok, then it's ok, better to implement it according the policies | 15:23 |
| kopecmartin | thanks for pointing that out | 15:23 |
| kopecmartin | #topic Periodic jobs Status Checks | 15:23 |
| kopecmartin | stable | 15:24 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-wallaby-py3&job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&job_name=tempest-full-zed&pipeline=periodic-stable | 15:24 |
| kopecmartin | master | 15:24 |
| kopecmartin | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 15:24 |
| kopecmartin | looks good, although latest tempest-all timeouted | 15:24 |
| * kopecmartin checking the logs | 15:24 | |
| kopecmartin | i wonder whether that can be related to jammy, i've seen similar timeouts on cs-9 fips jobs where the reason was that certain operations just took longer on that distro | 15:25 |
| kopecmartin | let's keep an eye on that | 15:26 |
| kopecmartin | #topic Distros check | 15:27 |
| kopecmartin | cs-9 | 15:27 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0 | 15:27 |
| kopecmartin | fedora | 15:27 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0 | 15:27 |
| kopecmartin | debian | 15:27 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0 | 15:27 |
| kopecmartin | jammy | 15:27 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-jammy&job_name=devstack-platform-ubuntu-jammy-ovn-source&job_name=devstack-platform-ubuntu-jammy-ovs&skip=0 | 15:27 |
| kopecmartin | rocky | 15:27 |
| kopecmartin | #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx | 15:27 |
| kopecmartin | there have been some failures, but the latest builds look good | 15:28 |
| frickler | cs-9 seems broken on stable branches. do you intend to look into fixing it or should we drop it instead of having it non-voting? | 15:28 |
| kopecmartin | i'm not planning to fix that :/ | 15:29 |
| kopecmartin | last week i proposed a patch to make it n-v on yoga | 15:29 |
| kopecmartin | the next logical step will be probably dropping the jobs on stable branches entirely | 15:29 |
| frickler | ack | 15:30 |
| kopecmartin | there isn't that much of a traffic there (on stable branches) and seeing the failure rates .. it will be better to save some resources instead | 15:30 |
| frickler | we also need to check platform-focal job now instead of -jammy | 15:30 |
| kopecmartin | good point | 15:31 |
| frickler | https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&project=openstack/devstack | 15:31 |
| *** pojadhav is now known as pojadhav|afk | 15:32 | |
| frickler | looking good so far | 15:32 |
| kopecmartin | perfect, let me update the agenda right away | 15:32 |
| kopecmartin | #topic Sub Teams highlights | 15:34 |
| kopecmartin | Changes with Review-Priority == +1 | 15:34 |
| kopecmartin | #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade) | 15:34 |
| kopecmartin | nothing there | 15:34 |
| kopecmartin | #topic Open Discussion | 15:34 |
| kopecmartin | anything for the open discussion? | 15:35 |
| kopecmartin | #topic Bug Triage | 15:36 |
| kopecmartin | #link https://etherpad.openstack.org/p/qa-bug-triage-antelope | 15:36 |
| kopecmartin | this relatively new bug in devstack has caught my attention | 15:36 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/1997017 | 15:36 |
| kopecmartin | it looks like the user set a wrong ip in HOST_IP which got me thinking, is it recommended to set that value? | 15:37 |
| opendevreview | Elod Illes proposed openstack/tempest master: Set back to use Bionic for Ussuri, Train and Stein https://review.opendev.org/c/openstack/tempest/+/865300 | 15:37 |
| kopecmartin | well, the fact i don't set it doesn't mean it's not an option .. | 15:37 |
| frickler | I think the issue is with py3.9 | 15:38 |
| frickler | uwsgi won't use that | 15:38 |
| frickler | on 20.04 one should use py3.8 | 15:38 |
| kopecmartin | oh, i see | 15:38 |
| frickler | although also I don't see where in the config the switched to py3.9 | 15:39 |
| frickler | but the logs show it being used | 15:39 |
| kopecmartin | py3.9 is the default python3 on the env maybe/ | 15:40 |
| kopecmartin | ? | 15:40 |
| kopecmartin | they wanted to use that version of python specifically based on the first sentence | 15:40 |
| frickler | yes and we don't support that | 15:41 |
| kopecmartin | will you comment there please? | 15:42 |
| frickler | sure | 15:42 |
| kopecmartin | then there's another new bug | 15:42 |
| kopecmartin | #link https://bugs.launchpad.net/devstack/+bug/1996465 | 15:43 |
| kopecmartin | seems like devstack didn't configure nova-cpu.conf correctly | 15:43 |
| kopecmartin | i'm not sure i've ever tried devstack on multinode setup | 15:43 |
| frickler | I wonder what this devstack-vagrant is meant to be | 15:43 |
| frickler | one needs to prepare local.conf specifically for the second node | 15:44 |
| frickler | our docs for that are likely outdated | 15:44 |
| frickler | so the reference would be the multinode job in our .zuul.yaml | 15:44 |
| frickler | I will comment and ask for more data | 15:45 |
| kopecmartin | thank you frickler | 15:45 |
| kopecmartin | this is all from my side | 15:45 |
| kopecmartin | if there isn't anything else | 15:46 |
| frickler | so one thing about the ceph plugin | 15:46 |
| kopecmartin | sure | 15:46 |
| frickler | it is working with jammy now in general | 15:46 |
| frickler | but there is a failure during the tempest deployment | 15:46 |
| frickler | so that would best need some tempest expert to look at | 15:46 |
| kopecmartin | i can take a look | 15:47 |
| kopecmartin | did anything change? | 15:47 |
| frickler | no, it never ran on jammy before. so something about python-ceph plugin dependencies confliction with tempest venv | 15:49 |
| frickler | I can't find the proper zuul result right now | 15:49 |
| frickler | maybe create a revert of https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/864948 and that should show the failure now that devstack default has changed | 15:50 |
| kopecmartin | ack, i'll have a look | 15:51 |
| frickler | https://zuul.opendev.org/t/openstack/build/8c70f6060c3141b890a7f32d6594a62c has the issue | 15:51 |
| frickler | note that the "ERROR: cowardly refusing to delete `envdir` (it does not look like a virtualenv): /opt/stack/tempest/.tox/tempest" is not the original failure | 15:52 |
| kopecmartin | ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/future' | 15:52 |
| frickler | right | 15:52 |
| kopecmartin | very weird, i'll have a look | 15:53 |
| kopecmartin | anything else? | 15:53 |
| frickler | maybe also something for clarkb who has been successful in similar issues | 15:53 |
| frickler | that's it from me then | 15:53 |
| kopecmartin | thank you frickler | 15:54 |
| kopecmartin | see you around | 15:54 |
| kopecmartin | #endmeeting | 15:54 |
| opendevmeet | Meeting ended Tue Nov 22 15:54:48 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:54 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2022/qa.2022-11-22-15.00.html | 15:54 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2022/qa.2022-11-22-15.00.txt | 15:54 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2022/qa.2022-11-22-15.00.log.html | 15:54 |
| frickler | \o | 15:54 |
| *** yadnesh is now known as yadnesh|away | 16:12 | |
| opendevreview | Merged openstack/devstack stable/yoga: Use proper sed separator for paths https://review.opendev.org/c/openstack/devstack/+/863451 | 16:36 |
| clarkb | frickler: kopecmartin: the first thing I would try is pip intsall future in a jammy container | 16:38 |
| clarkb | just to check if this is a bug with the packaging somehow on jammy | 16:38 |
| clarkb | and work backward from there | 16:38 |
| clarkb | oh wait its trying to install globally but this is a tox venv install | 16:45 |
| clarkb | tox updated on the 13th | 16:47 |
| clarkb | frickler: kopecmartin: is this tox install issue only affecting ceph jobs? If so that adds even more mystery | 16:50 |
| kopecmartin | clarkb: thanks, i'll try it | 16:52 |
| kopecmartin | yeah, it's just a tox install and it affects only that one job o.O | 16:52 |
| clarkb | `pip install {opts} {packages}` is set as the install command. I wonder if that is not running relative to the tox env dir anymore for some reason | 16:54 |
| clarkb | however, I would expect that to affect all tempest jobs if that was a problem | 16:54 |
| * clarkb looks for a recent jammy tempest run that succeeded | 16:54 | |
| clarkb | side note: I see nova and neutron in the gate right now and they don't have a shared tempest job (this is what broke things with the ovn changes) | 16:55 |
| clarkb | https://zuul.opendev.org/t/openstack/build/8c70f6060c3141b890a7f32d6594a62c/log/job-output.txt#20438-20556 that is the context for the failure. The last line in that block does seem to show it using global pip | 17:01 |
| clarkb | compared to https://zuul.opendev.org/t/openstack/build/819962de74584064890d1e25d010f65e/log/job-output.txt#21057-21174 which pretty clearly uses the venv pip | 17:02 |
| clarkb | fungi: ^ have you seen this before? tox commands installing globally and not into a venv? | 17:13 |
| fungi | clarkb: new pip doesn't need to be installed in a venv to run, so if there's a pip entrypoint wrapper in $PATH that might do it | 17:16 |
| fungi | how long ago did this start? | 17:16 |
| clarkb | aiui this is specific to the ceph jobs on jammy which were never on jammy before and other jammy jobs don't do this | 17:17 |
| fungi | okay, so seems likely to be a jammy vs focal thing? | 17:17 |
| clarkb | I don't know if this ran on focal either | 17:17 |
| clarkb | it might be easiset to hold a node and interact with tox directly | 17:18 |
| clarkb | and check things like pip in paths and so on | 17:18 |
| clarkb | comparing the two tox install steps between a succeeding job on jammy and this failure they look identical though. Makes me suspect some sort of side effect earlier in the devstack run | 17:19 |
| fungi | where is the install command being set? tox.ini in what project? | 17:19 |
| clarkb | fungi: tempest | 17:19 |
| clarkb | this is running `tox -r -e full --notest` in tempest | 17:19 |
| clarkb | allowlist_externals = * is also set | 17:20 |
| fungi | any idea why that install_command is even being set there? what is it trying to accomplish? | 17:20 |
| clarkb | maybe removing that line so that tempest records external commands is a good place to start | 17:20 |
| clarkb | fungi: I have no clue | 17:20 |
| clarkb | andreaf added it when they switched from tox_install.sh to using just pip | 17:21 |
| clarkb | I agree that seems unnecessary here since that looks like a very default string? | 17:22 |
| fungi | i bet the proper thing to do would have been to remove it | 17:22 |
| fungi | entirely | 17:22 |
| fungi | rather than changing it to something that wants to be default behavior but actually isn't | 17:22 |
| clarkb | python -m pip install {opts} {packages} is the default | 17:23 |
| clarkb | which is slightly different | 17:23 |
| clarkb | I can push up a change that drops the allowlist and install_command and set up a depends on to see if that gives us better info | 17:23 |
| clarkb | wait that job succeeded later? | 17:24 |
| fungi | my guess is that pip is no longer being installed into the venv (because pip can now manage a venv without being installed in it), and as such the execution path is falling back to a globally-installed one | 17:24 |
| clarkb | kopecmartin: frickler: I'm really confsused as to whether or not this is still a problem? can you clarify? | 17:24 |
| frickler | currently the problem isn't there because we pinned the devstack-plugin-ceph jobs to focal. I'll push up a partial revert and then we can hold a node there | 17:26 |
| clarkb | frickler: oh let me push that and I'll make the edits to tox.ini too | 17:26 |
| clarkb | frickler: the job is defined in nova? | 17:28 |
| clarkb | oh it must be the parent that sets the nodeset | 17:29 |
| opendevreview | Clark Boylan proposed openstack/tempest master: Cleanup tox.ini https://review.opendev.org/c/openstack/tempest/+/865314 | 17:31 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack-plugin-ceph master: Revert "Temporary pin the ceph jobs nodeset to Focal" https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | 17:32 |
| frickler | clarkb: ^^ | 17:32 |
| frickler | nova-multinode-ceph inherits from that | 17:33 |
| *** jpena is now known as jpena|off | 17:33 | |
| clarkb | frickler: that should depends on 865314 | 17:34 |
| clarkb | frickler: or would you prefer the tempest change depend on that? | 17:35 |
| frickler | clarkb: lets wait for initial results for both patches to verify the failure is still there, then I can add depends-on to the ceph patch | 17:37 |
| clarkb | ok | 17:37 |
| clarkb | if this fixes things we may need to send an email calling it out. https://codesearch.opendev.org/?q=install_command&i=nope&literal=nope&files=tox.ini&excludeFiles=&repos= shows a fair bit of this going on | 17:41 |
| fungi | yes, i looked for the same and saw lots of cases of it. cargo cult copy-pasta | 17:43 |
| fungi | but longer term, i doubt we can rely on pip inside venvs with the direction the packaging ecosystem is headed. pip development is focused on being usable as a stand-alone utility outside your venv now (can even be downloaded and run directly as a zipapp) | 17:45 |
| clarkb | but thats a tox problem to solve | 17:45 |
| fungi | so whether and what pip you've installed in a venv may not implicitly mean anything, and we might need to get a lot more specific about where we're using pip from | 17:45 |
| clarkb | assuming we aren't doing any crazy overrides tox has an interface they need to maintain | 17:45 |
| fungi | yeah, it may manifest as new tox options | 17:45 |
| fungi | ideally backward-compatible defaults at least with some deprecation period before changing behavior | 17:46 |
| fungi | a lot of these new behaviors are coming about because python has been notoriously hard for newcomers to use on windows, and so there's a push within the packaging community to both make things easier on windows and make all other platforms work more like windows | 17:49 |
| clarkb | sure, I just think that we should be able to ignore those problems as long as we are using tox (or possibly fix tox if necessary). Tox is a tool that abstracts this from us already and we shouldn't need to think too hard about how to ensure tox actually installs into a venv | 17:50 |
| clarkb | of course if we're overriding the commands to do this then we are on our own so we hsoul dstop doing that if we can | 17:50 |
| fungi | exactly | 17:52 |
| *** dmellado_ is now known as dmellado | 18:01 | |
| *** dmellado_ is now known as dmellado | 18:04 | |
| frickler | clarkb: https://zuul.opendev.org/t/openstack/build/f6fb61d3ca6049b1b51311154f903122 looks different yet again | 18:15 |
| frickler | I'll dep on your tempest patch anyway | 18:15 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack-plugin-ceph master: Revert "Temporary pin the ceph jobs nodeset to Focal" https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | 18:16 |
| clarkb | frickler: that looks like a failure setting up the venv. Whereas before we seemed to not even try to steup a venv | 18:16 |
| clarkb | interesting | 18:16 |
| clarkb | frickler: Ithink it is trying to use python2 and not finding it | 18:17 |
| clarkb | why are we recreating the env though that seems like a bug | 18:19 |
| clarkb | we just went through all this effort to instll the env with constraints and all that, then we delete it and start over ... | 18:20 |
| opendevreview | Dr. Jens Harbott proposed openstack/devstack master: Bump cirros version to 0.6.1 https://review.opendev.org/c/openstack/devstack/+/859773 | 18:20 |
| clarkb | basepython is set to python3 so thats weird | 18:21 |
| clarkb | ya looks like devstack/lib/tempest install_tempest does all that effort to intsal lthe venv. Then configure_tempest says "just kidding" and deletes it all | 18:22 |
| clarkb | and does it again. | 18:22 |
| clarkb | can probbaly speed up tempest jobs by not doing that | 18:22 |
| clarkb | we also install tempest globally. I think we end up installing tempest 3 times | 18:27 |
| clarkb | and only one of those installs happens in install_tempest | 18:28 |
| clarkb | looks like it is still failing to find python, this is the weirdest thing. its like something is mangling path and that leads to pip and python problmes | 19:01 |
| clarkb | I'm not familiar enough with tempest and dvestack at this point or that plugin to have a hunch as to what might be doing it | 19:01 |
| clarkb | but maybe simplifying things like my change and also perhaps not installing tempest 3 times each run would help reduce the places it could break | 19:01 |
| clarkb | but also lets hold a node. I can work on getting that setup after my meeting | 19:14 |
| clarkb | frickler: I put a hold in place and rechecked your change | 20:04 |
| clarkb | frickler: fungi: interestingly it seems that the venv being created by tox isn't a valid venv | 21:17 |
| clarkb | and that is what causes it to try and use global `python` which doens't exist on the node | 21:18 |
| clarkb | frickler: fungi: ok the issue is that the virtualenv installed by `virtualenv` is creating fooenv/local/bin not fooenv/bin | 21:23 |
| clarkb | we had this before and it had to do with things I don't recall but I've got a pip branch somewhere with details | 21:23 |
| clarkb | hrm no the pip branch I'm thinking of is for a different problem | 21:24 |
| clarkb | upgrading virtualenv fixes it in the held node I think https://github.com/pypa/virtualenv/pull/2415 was the fix. However, python3-distutils is alredy installed on the node when it is broken and they indicated that would fix it so I dunno | 21:32 |
| clarkb | I also cannot reproduce this in a jammy container | 21:36 |
| clarkb | I've mangaed to reproduce it finally. It breaks when latest setuptools is installed which vendors distutils and virtualenv finds that | 22:53 |
| clarkb | the devstack plugin for ceph pip install -U --force PasteDeploy which for some reason also upgrades setuptools from the distro version to latest | 22:53 |
| clarkb | this is why other jobs are fine, they use distro setuptools | 22:53 |
| clarkb | I believe this happens because pastedeploy==2.1.1 has an install_requires on setuptools | 22:58 |
| clarkb | also the pip --force flag seems to be undocumented but definitely changes the behavior here | 22:58 |
| clarkb | fungi: ^ do you know what pip instal --force is meant to do? I'm having a hard time finding it in the source and git history | 23:02 |
| clarkb | but it definitely seems to force setuptools to be upgraded here. Dropping it only updates PasteDeploy | 23:02 |
| clarkb | --ignore-installed and --force-reinstall produce similar behavior so I guess its something along those lines | 23:04 |
| fungi | clarkb: yeah, i was about to ask if it was like --force-reinstall | 23:07 |
| fungi | clarkb: --upgrade-strategy=eager might have a similar effect | 23:08 |
| fungi | used to be pip's default behavior when specifying -U but now it does only-if-needed upgrades by default | 23:09 |
| opendevreview | Clark Boylan proposed openstack/devstack-plugin-ceph master: Revert "Temporary pin the ceph jobs nodeset to Focal" https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | 23:10 |
| clarkb | fungi: ^ I've worked around it by using --no-deps | 23:10 |
| clarkb | I think that should work | 23:10 |
| fungi | and yes, with python 3.12 completely removing distutils, setuptools is getting more aggressive about using its own copy | 23:10 |
| clarkb | well in this case its any import distutils finding the setuptools copy | 23:11 |
| clarkb | no setuptools involved | 23:11 |
| clarkb | I hvae to say after spending most of a day debugging this people who say python environments are too complicated are correct | 23:11 |
| clarkb | the job is running tempest but that is unhappy | 23:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!