| opendevreview | Jeremy Stanley proposed openstack/releases master: Release PBR 6.0.0 https://review.opendev.org/c/openstack/releases/+/900041 | 13:19 |
|---|---|---|
| fungi | hberaud: damani: stephenfin: clarkb: ^ | 13:20 |
| hberaud | ack | 13:20 |
| hberaud | fungi: yesterday I proposed https://review.opendev.org/c/openstack/releases/+/899965 | 13:21 |
| hberaud | (a minor bump= | 13:21 |
| hberaud | ) | 13:21 |
| fungi | hberaud: oh, the docs in master talk about 6.0.0 | 13:21 |
| hberaud | but I think you are right, so let's abandon my patch | 13:22 |
| hberaud | +2'd | 13:23 |
| fungi | looks like we'll first need to work out why pbr is breaking the validate job | 13:38 |
| fungi | https://zuul.opendev.org/t/openstack/build/919ffe90095849f8aa8a3a211d20c829/log/tox/validate/validate-request-results.log#4820 | 13:38 |
| fungi | No such file or directory: '.tox/venv/bin/python3' | 13:38 |
| fungi | i guess more context is at https://zuul.opendev.org/t/openstack/build/919ffe90095849f8aa8a3a211d20c829/log/tox/validate/validate-request-results.log#341-344 | 13:40 |
| fungi | it does a `tox -e venv --notest` and then tries to `.tox/venv/bin/python3 setup.py --name` but can't find .tox/venv/bin/python3 | 13:41 |
| fungi | both the pbr release changes hit the same error, but there was a tooz release proposed between them which didn't, so it's almost certainly specific to pbr | 13:43 |
| fungi | oh, also problematic is that we're directly invoking setup.py in that script in order to determine the project name | 14:13 |
| fungi | the recommended replacement seems to be python -c 'from setuptools.config.setupcfg import read_configuration as c; print(c("setup.cfg")["metadata"]["name"])' | 14:14 |
| fungi | or in setuptools<61 you need to python -c 'from setuptools.config import read_configuration as c; print(c("setup.cfg")["metadata"]["name"])' | 14:15 |
| fungi | still, none of those help if we can't find where tox created the venv | 14:18 |
| fungi | i can't reproduce locally with python 3.11 and latest tox/setuptools | 14:46 |
| fungi | the last successful pbr release was 2023-01-07 so about 11 months worth of things that could have changed to break this in the meantime | 15:03 |
| fungi | er, 2023-01-11 | 15:03 |
| clarkb | fungi: is it possibly using python2 in the venv env for pbr? | 15:05 |
| fungi | that would be odd, but it's possible i suppose... seeing if i can tell from the logs | 15:05 |
| fungi | pbr's tox.ini doesn't specify a default python, so i suppose it's possible | 15:06 |
| fungi | the job ran on a jammy node though, so getting python 2.7 installed would be a trick i think? | 15:08 |
| clarkb | ya if it ran on jammy I think that is far less likely | 15:08 |
| fungi | i don't see any indication in the log that python 2.7 packages were installed either | 15:10 |
| fungi | looks like it's all 3.10 | 15:10 |
| fungi | i'll try to reproduce the behavior with 3.10 instead of 3.11, though i doubt it will make any difference | 15:11 |
| clarkb | repping for .tox/venv in job-output.txt there is no mention of it until the file not found errors | 15:11 |
| clarkb | I suspect that the job isn't creating the venv given that | 15:11 |
| clarkb | tox should fairly verbosely log that it is creating the venv I think? Happy for log pointers indicating it did create the venv if anyone knows where that is | 15:12 |
| fungi | the command immediately prior to the failure should be creating it | 15:12 |
| fungi | it does a `tox -e venv --notest` | 15:12 |
| clarkb | oh we aren't using the role for tox here so we don't get the debugging... | 15:12 |
| clarkb | thats where I would start. make the tox invocations debug | 15:12 |
| fungi | the only other thing i can figure is maybe it's creating that .tox directory somewhere other than the current path | 15:13 |
| clarkb | DEBUG: cwd = /home/zuul/src/opendev.org/openstack/releases/.tox/validate/tmp/releases-rzsbjdvc/openstack/pbr | 15:14 |
| clarkb | maybe it is discovering that there is a .tox/ dir above it and creating it there. That would be weird though because it is for a different tox.ini | 15:14 |
| clarkb | but ya tox --whatever-debug-outout-flag-is-just-like-the-zuul-role would help a lot | 15:14 |
| fungi | i have a feeling this is behavior triggered by the presence of pyproject.toml | 15:15 |
| clarkb | pbr doesn't have one though | 15:15 |
| fungi | mmm, though pbr doesn't have a pyproject.toml | 15:15 |
| fungi | right | 15:15 |
| clarkb | intentionally because its a chicken and egg thing | 15:15 |
| fungi | do we just normally add -v to tox commands for additional debugging? | 15:18 |
| *** diablo_rojo is now known as Guest5670 | 15:18 | |
| clarkb | -vv --skip-missing-interpreters=false <- is what the tox role sets by default | 15:18 |
| fungi | aha, i'll inject those | 15:19 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: DNM: Get more verbose logging for PBR release fail https://review.opendev.org/c/openstack/releases/+/900053 | 15:22 |
| clarkb | fungi: in ^ do the args to tox-log-command.sh get passed through to the nested tox -evenv invocation? | 15:27 |
| fungi | i guess i'll find out | 15:28 |
| fungi | i don't think they do though | 15:29 |
| fungi | i misread what that was doing | 15:29 |
| fungi | wow, i always forget there's pbr-specific workarounds in these scripts | 15:34 |
| fungi | use_tox = repo.endswith('/pbr') | 15:34 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: DNM: Get more verbose logging for PBR release fail https://review.opendev.org/c/openstack/releases/+/900053 | 15:36 |
| fungi | that ^ should get it | 15:36 |
| fungi | so basically, the command we're seeing fail only ever gets run when there's a pbr release | 15:37 |
| clarkb | that explains why only pbr is affected :) | 15:38 |
| fungi | it's probably something that changed early in the year while everyone was still scrambling with tox v4 impact | 15:40 |
| fungi | ugh, of course that job inspects the commit being tested, so i need to reverse the order of those changes | 15:53 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: DNM: Get more verbose logging for PBR release fail https://review.opendev.org/c/openstack/releases/+/900053 | 15:54 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: Release PBR 6.0.0 https://review.opendev.org/c/openstack/releases/+/900041 | 15:54 |
| clarkb | that didn't add a lot of debugging. The validate venv does. Very weird | 16:10 |
| clarkb | perhaps the python check_output() function is eating it? | 16:11 |
| clarkb | yup it does | 16:12 |
| clarkb | fungi: check_output captures stdout and returns it to the caller. Which is then discareded where you added the -vv... | 16:12 |
| opendevreview | Clark Boylan proposed openstack/releases master: DNM: Get more verbose logging for PBR release fail https://review.opendev.org/c/openstack/releases/+/900053 | 16:22 |
| opendevreview | Clark Boylan proposed openstack/releases master: Release PBR 6.0.0 https://review.opendev.org/c/openstack/releases/+/900041 | 16:22 |
| clarkb | I think that might actually be a correct change which can be moved out of DNM if others agree | 16:22 |
| fungi | aha, that is probably less hacky than what i was going to try | 16:23 |
| fungi | seems to have helped | 16:29 |
| clarkb | yup and i think its doing the thing I guessed. Its going into .tox of the releases dir not the pbr dir | 16:30 |
| * clarkb adds this to the list of reasons to use nox | 16:31 | |
| fungi | basically, the behavior when calling tox from tox has changed | 16:31 |
| clarkb | DEBUG: venv: 344 I create virtual environment via CPython3Posix(dest=/home/zuul/src/opendev.org/openstack/releases/.tox/venv, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:50] | 16:32 |
| fungi | thanks, i was trying to copy text from the stream output and failing. i don't know why it's impossible to copy from zuul's log streaming widget, probably something broken with the javascript | 16:33 |
| fungi | ah, okay, i can right-click copy right-click paste, i just can't highlight into the primary x11 selection buffer and middle-click | 16:34 |
| fungi | so probably js trying to be smart about overriding text selection | 16:35 |
| fungi | anyway, i guess we can override the path to the interpreter. i'll give that a shot | 16:35 |
| clarkb | either that or figure out how to drop nested tox. I don't know/remember why pbr is special here | 16:37 |
| fungi | we needed pbr installed from source in order for things to work, i think | 16:37 |
| fungi | at build time rather than just run time | 16:38 |
| fungi | there are almost certainly other ways to go about that | 16:38 |
| clarkb | aha | 16:38 |
| clarkb | fungi: fwiw I think you can probably keep the check_output -> check_call change. Then it will fail early and include stdout + stderr. Far easier to debug that script imo | 16:39 |
| fungi | yeah, i was just going to stuff my proposed fix under that debug change | 16:39 |
| fungi | it looks like the tox devenv subcommand might do this these days | 16:42 |
| clarkb | `tox --devenv pbrvenv -e venv` ? | 16:45 |
| clarkb | and that seems to ignore hte .tox directory entirely? ya that may be the way to go | 16:45 |
| clarkb | oh except it always sets usedevelop to true | 16:45 |
| fungi | tox devenv -c tox.ini pbrenv | 16:45 |
| fungi | well, pbr also sets usedevelop=true in its tox.ini anyway | 16:45 |
| clarkb | ah. fungi I think you have to set -e venv otherwise it uses -e py by efault which is different? | 16:46 |
| clarkb | but ya I agree this seems like it could be a standin if we don't want to use the parents .tox dir path | 16:46 |
| fungi | tested locally and `pbrenv/bin/python3 setup.py --name` worked afterward | 16:46 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: Fix PBR name check with new tox https://review.opendev.org/c/openstack/releases/+/900053 | 16:53 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: Release PBR 6.0.0 https://review.opendev.org/c/openstack/releases/+/900041 | 16:53 |
| fungi | i'll remove the zuul config edits once we're sure this is working | 16:55 |
| clarkb | deliverables/_independent/pbr.yaml: validate_tarball_base: Could not get the name of openstack/pbr for version 6.0.0: 'int' object has no attribute 'decode' | 17:02 |
| clarkb | progress but now we hit this | 17:02 |
| clarkb | heh this is another case of the tool supressing the error... I think thats fine for end user facing tools but for dev tools it just makes things more difficult | 17:03 |
| clarkb | ah that check_call does need to be a check_output | 17:05 |
| clarkb | but only that one | 17:05 |
| fungi | i'm increasingly glad i temporarily stripped out all the other jobs while we iterate on this | 17:06 |
| opendevreview | Clark Boylan proposed openstack/releases master: Fix PBR name check with new tox https://review.opendev.org/c/openstack/releases/+/900053 | 17:07 |
| opendevreview | Clark Boylan proposed openstack/releases master: Release PBR 6.0.0 https://review.opendev.org/c/openstack/releases/+/900041 | 17:07 |
| clarkb | fungi: that passes now if you want ot restore the zuul config | 17:18 |
| fungi | thanks! on it | 17:18 |
| clarkb | side note: is a pbr release the sort of thing to wait for Monday on anyway? | 17:19 |
| clarkb | or just send it bceause we've got testing :) | 17:19 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: Fix PBR name check with new tox https://review.opendev.org/c/openstack/releases/+/900053 | 17:19 |
| opendevreview | Jeremy Stanley proposed openstack/releases master: Release PBR 6.0.0 https://review.opendev.org/c/openstack/releases/+/900041 | 17:19 |
| fungi | clarkb: monday may make more sense. most of the urgency seemed to be that adding pyproject.toml files to projects with usedevelop on is blocked because current py39 jobs are running on focal which has python3.8 as its python3, so build isolation is building the packages with a different python version than it's being installed on which seems to break without proper pep 660 support | 17:22 |
| fungi | which is apparently what prompted stephenfin to implement it | 17:23 |
| clarkb | It seems unlikely I'll have time to debug if anything goes wrong over the weekend fwiw. But I'm happy to risk it and fix on Monday | 17:25 |
| fungi | fwiw, the release managers generally don't approve release requests on fridays anyway unless given sufficient rationale, so i don't think we need to worry about it | 17:26 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!