| ianw | i've noted it in the etherpad and added a checklist point at the end. i'll turn it back *on* for https://173.231.255.71/ so we don't have confusion | 00:00 |
|---|---|---|
| Clark[m] | That seems reasonable | 00:09 |
| opendevreview | Merged opendev/system-config master: install-ansible: overhaul install ansible requirements https://review.opendev.org/c/opendev/system-config/+/866542 | 01:34 |
| ianw | https://173.231.255.173 has been cycled from 3.5 -> 3.6 -> 3.5 with downgrade process. | 01:53 |
| ianw | output is https://paste.opendev.org/show/b6QBme2ellt5nQ5FYtNT/ | 01:53 |
| ianw | the illegal reflective access looks exactly like https://bugs.chromium.org/p/gerrit/issues/detail?id=12639, but that was marked as fixed | 01:54 |
| ianw | i'll investigate | 01:54 |
| ianw | note it also happens on 3.5.4, so not a regression as such | 01:55 |
| Clark[m] | I think there is a known thing with guice and java 11 | 02:19 |
| ianw | yeah everything sort of says it's fixed in 5.0.1, which should be included ... | 02:28 |
| ianw | tools/nongoogle.bzl: GUICE_VERS = "5.0.1" | 02:28 |
| ianw | I've filed https://bugs.chromium.org/p/gerrit/issues/detail?id=16506 ... it looks the same as the original bug but it's not fixed, and the other is closed | 02:54 |
| ianw | i've sent the notification but it's hit moderation | 03:20 |
| Clark[m] | I can send it through tomorrow morning but my secrets content isn't currently accessible | 03:24 |
| opendevreview | Frikin Evgenii proposed openstack/diskimage-builder master: Add variable for check installing python3 in yum element https://review.opendev.org/c/openstack/diskimage-builder/+/856577 | 03:25 |
| ianw | Clark[m]: no rush, just ticking things off the checklist :) | 03:25 |
| ianw | ansible-playbook [core 2.14.1] ... that lg on bridge. hourly run is going and seems fine so far | 03:26 |
| ianw | https://gitlab.com/mailman/hyperkitty/-/merge_requests/485 is for the no-margin on hyperkitty i mentioned yesterday | 03:56 |
| *** yadnesh|away is now known as yadnesh | 04:43 | |
| opendevreview | Merged opendev/system-config master: install-ansible: update venv once a day https://review.opendev.org/c/opendev/system-config/+/866633 | 04:59 |
| *** ysandeep is now known as ysandeep|brb | 05:07 | |
| *** ysandeep__ is now known as ysandeep | 05:51 | |
| *** marios is now known as marios|ruck | 06:19 | |
| *** yadnesh is now known as yadnesh|afk | 07:46 | |
| *** jpena|off is now known as jpena | 08:41 | |
| *** yadnesh|afk is now known as yadnesh | 08:49 | |
| opendevreview | Jiri Podivin proposed openstack/project-config master: Releasing tripleo-ansible as PyPi deliverable https://review.opendev.org/c/openstack/project-config/+/866839 | 09:48 |
| *** dviroel|biab is now known as dviroel | 11:14 | |
| *** rlandy|out is now known as rlandy|rover | 11:14 | |
| *** yadnesh is now known as yadnesh|afk | 11:50 | |
| *** yadnesh|afk is now known as yadnesh | 12:34 | |
| *** frenzy_friday is now known as frenzy_friday|food | 13:09 | |
| *** pojadhav is now known as pojadhav|afk | 13:48 | |
| *** ysandeep is now known as ysandeep|out | 13:59 | |
| *** dasm|off is now known as dasm | 14:43 | |
| *** pojadhav|afk is now known as pojadhav | 14:46 | |
| opendevreview | Thierry Carrez proposed opendev/irc-meetings master: Move Large Scale SIG meeting on even weeks in 2023 https://review.opendev.org/c/opendev/irc-meetings/+/866873 | 15:24 |
| fungi | ttx: frickler has a comment/question on ^ | 15:29 |
| *** frenzy_friday|food is now known as frenzy_friday | 15:34 | |
| opendevreview | Thierry Carrez proposed opendev/irc-meetings master: Move Large Scale SIG meeting on even weeks in 2023 https://review.opendev.org/c/opendev/irc-meetings/+/866873 | 15:38 |
| ttx | good point, fixed ! | 15:39 |
| opendevreview | Merged opendev/irc-meetings master: Move Large Scale SIG meeting on even weeks in 2023 https://review.opendev.org/c/opendev/irc-meetings/+/866873 | 15:46 |
| clarkb | fungi: looks like you got the gerrit upgrade announcement? I was just logging in to check that | 15:56 |
| fungi | yep | 15:56 |
| clarkb | thanks! (this new ui is much nicer) | 15:57 |
| fungi | i wanted to double-check that the moderation functionality was working as expected | 15:57 |
| fungi | i'll let you get the next one | 15:57 |
| clarkb | no worries. I was just making sure i got to it | 15:57 |
| clarkb | my todo list is starting to build a collection of writing activities for when things slow down a bit more. Today though I need to find breakfast and then look at nodepool test failures and lack of openstack api statsd info | 16:00 |
| *** dviroel is now known as dviroel|lunch | 16:15 | |
| clarkb | fungi: are we restoring list server dns record ttls todayish? I'm not in a rush just don't want to forget | 16:20 |
| fungi | yeah, and merging the config removal for the old server too i guess | 16:22 |
| fungi | lemme un-wip | 16:22 |
| clarkb | fungi: oh and we should restart things for the site_owner update and double check /etc/mailman.cfg is properly updated in the container? | 16:22 |
| fungi | yes | 16:23 |
| fungi | so the remaining non-dnm changes for topic:mailman3 should be safe to merge now | 16:23 |
| *** marios|ruck is now known as marios|out | 16:40 | |
| opendevreview | Ade Lee proposed zuul/zuul-jobs master: Add ubuntu to enable-fips role https://review.opendev.org/c/zuul/zuul-jobs/+/866881 | 17:11 |
| *** yadnesh is now known as yadnesh|away | 17:14 | |
| *** dviroel|lunch is now known as dviroel | 17:16 | |
| *** jpena is now known as jpena|off | 17:31 | |
| clarkb | fungi: https://zuul.opendev.org/t/zuul/build/47d0916427b14372bfedbbd7a4e91a61/log/job-output.txt#869-933 at first I thought this might be something we're doing wrong with apt-key but digging more it seems there are a ton of bugs around ansible just getting this wrong :/ but thought I'd share it because I think a workaround is to just drop the Release.key file into | 18:23 |
| clarkb | /etc/apt/something/something.gpg instead? will that work on focal and newer? | 18:23 |
| clarkb | apparently it does a check after adding the key to see if it is present and if they don't process that correct we get this error which I expect is a bug in ansible and not with the key itself | 18:28 |
| clarkb | oh I think apt_key module wants us to pass in an id to cross check against and we are not doing that | 18:31 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Specify crio repo key ids for apt_key https://review.opendev.org/c/zuul/zuul-jobs/+/866920 | 18:35 |
| fungi | clarkb: yes, well i think raw .gpg files will work there for even older releases but .asc needs at least focal? | 18:43 |
| clarkb | ah | 18:45 |
| clarkb | I'm shifting gears now beacuse I think tox siblings is completely broken possily due to a tox release | 18:45 |
| fungi | ooh, ouch | 18:47 |
| fungi | 4.0.0... looking through the changelog now | 18:48 |
| frickler | ah, was just about to mention it. https://tox.wiki/en/latest/faq.html#breaking-changes-in-tox-4 | 18:50 |
| fungi | yup | 18:51 |
| fungi | "breaking" is no understatement | 18:51 |
| frickler | I did some sporadic testing earlier and got one small issue fixed from rc1, but likely there's more | 18:51 |
| fungi | mentioned in #openstack-infra but lots of projects are going to need to replace whitelist_externals | 18:51 |
| fungi | it's been deprecated for a while but removed in 4.0.0 | 18:52 |
| clarkb | I wonder if it could possibly be the color output confusing things | 18:54 |
| clarkb | I think a change to set the no color flag will be self testing in zuul-jobs so I'll give that a go | 18:54 |
| fungi | ooh maybe | 18:57 |
| clarkb | problem is I dunno if old tox will accept --colored no | 18:58 |
| clarkb | I guess I can set NO_COLORS=0 in the env instead and it should ignore that | 18:58 |
| fungi | yeah, that ought to be safe | 19:02 |
| clarkb | NO_COLORS=0 isn't working for me on the command line | 19:05 |
| fungi | ouch, also all existing tox plugins need a rewrite? | 19:05 |
| clarkb | neither is NO_COLORS=1 | 19:05 |
| clarkb | TERM=dumb does work | 19:05 |
| clarkb | but that is a bit more side effecting since other things may look for the TERM value | 19:06 |
| clarkb | arg | 19:06 |
| fungi | yeah, just reached that part of the faq | 19:06 |
| clarkb | https://github.com/tox-dev/tox/search?q=NO_COLORS | 19:07 |
| clarkb | so they aren't doing anything explicitly with that flag in the code? this is bonghits | 19:07 |
| clarkb | hahahahahahahahaha | 19:07 |
| fungi | that's what i was about to verify too | 19:07 |
| clarkb | NO_COLOR=1 works | 19:07 |
| clarkb | I'm going to need a drink and it isn't even noon yet | 19:07 |
| fungi | okay, so faq double-typo | 19:07 |
| * clarkb edits the change that is basically done to fix it | 19:08 | |
| fungi | clearly you're not the only one drinking on the job if this is any indication ;) | 19:08 |
| fungi | i'm willing to bet the implementation started out expecting COLORS=0 and then got changed to NO_COLOR=1 somewhere along the way, and whoever was writing up the faq got them swizzled around | 19:09 |
| fungi | i replied to the discord thread where it was announced, pointing that out | 19:11 |
| fungi | er, discuss | 19:12 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Set tox NO_COLOR=1 flag https://review.opendev.org/c/zuul/zuul-jobs/+/866926 | 19:12 |
| clarkb | I believe ^ will be self testing | 19:12 |
| clarkb | note I'm not sure if this will break tox siblings for target envs that don't show up in --showconfig | 19:13 |
| clarkb | I'm not sure what we are doing with --showconfig and how it might be affected by the changed output yet. Its weird because running tox -e docs seems to work and do what we have set but then yo udon't actually get the showconfig output you expect | 19:14 |
| clarkb | I think if 866926 isn't a clear win the next thing to try is pinning tox < 4.0 for now in zuul-jobs then have a change to remove the cap be self testing of things | 19:15 |
| clarkb | honestly that might be better anyway? | 19:15 |
| clarkb | I'll work on that change now too | 19:15 |
| clarkb | NO_COLOR didn't help and its the same error. Pin change should be up momentarily | 19:17 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Pin tox to <4 in ensure-tox installation https://review.opendev.org/c/zuul/zuul-jobs/+/866928 | 19:19 |
| fungi | i've sent something to the openstack-discuss ml on behalf of the tact sig, but other project liaisons may want to follow suit | 19:20 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Pin tox to <4 in ensure-tox installation https://review.opendev.org/c/zuul/zuul-jobs/+/866928 | 19:32 |
| clarkb | fungi: do you have a link to the discuss discussion around tox 4? Might be good so that we can follow problems others have | 19:33 |
| fungi | clarkb: https://discuss.python.org/t/tool-tox-4-on-the-horizon/6537/26 | 19:34 |
| clarkb | thank you | 19:34 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Make zuul-tox-docs voting again https://review.opendev.org/c/zuul/zuul-jobs/+/866929 | 19:36 |
| clarkb | fungi: '# !!! unused: whitelist_externals' gets emitted to --showconfig output when you use the invalid flags | 19:43 |
| clarkb | basically tox is ignoring it and not breaking on it. Which should be fine for that one in particular since all it does is warn if you escape the venv | 19:43 |
| clarkb | usedevelop is also ignored and I expect that to be more problematic | 19:43 |
| clarkb | in zuul/zuul-jobs --showconfig only emits envs for [tox] and [tox:linters} | 19:44 |
| clarkb | oh beacuse it doesn't emit the whole thing only what is in your envlist? so maybe this is ok | 19:44 |
| clarkb | ya ok | 19:45 |
| rlandy|rover | fungi++ clarkb++ - thanks for quick response on the tox 4.0.0 bloodbath | 19:47 |
| frickler | also the quoting seems to have gotten messed up, try tox -e linters in zuul-jobs after fixing the allowlist_externals | 19:49 |
| opendevreview | Merged zuul/zuul-jobs master: Pin tox to <4 in ensure-tox installation https://review.opendev.org/c/zuul/zuul-jobs/+/866928 | 19:51 |
| clarkb | doing a naive `tox -edocs --showconfig > newshownconfig` and then loading it into https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_install_sibling_packages.py#L312-L330 produces no errors under my local python3.10 install | 19:56 |
| fungi | have you diffed 3.x output against it? | 19:58 |
| fungi | maybe that would yield clues | 19:58 |
| clarkb | ya its very different | 19:59 |
| clarkb | I'm not sure that will behelpful since tox seems tohave changed it so dramatically | 19:59 |
| fungi | i guess it's a question of whether the bits tox_install_sibling_packages.py tries to parse out of it are different | 20:00 |
| fungi | i half wonder if it's changed from old ini format to toml | 20:01 |
| clarkb | I don't think so as it seems to be valid ini when I do the config parsing locally | 20:02 |
| clarkb | basically I'm using tox 4 -e docs --showconfig > file. Then loading that file into the block of code I linked to above and it works without complaint | 20:02 |
| fungi | valid as in successfully parsed by configparser, but with the same actual data? | 20:02 |
| clarkb | fungi: yes successfully parses without error as far as I can tell | 20:03 |
| opendevreview | Merged zuul/zuul-jobs master: Make zuul-tox-docs voting again https://review.opendev.org/c/zuul/zuul-jobs/+/866929 | 20:03 |
| clarkb | now whether or not it is identical input data is harder to say. There are probably some hacks we can do | 20:03 |
| clarkb | ya let me push a DNM change up to try and capture that info | 20:03 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: DNM Cat the tox --showconfig output under tox v4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 20:06 |
| clarkb | that should cat the file that is being read so we can see what exactly is being processed | 20:06 |
| frickler | that's the quoting thing, can be workarounded by s/#/\#/ https://github.com/tox-dev/tox/issues/2617 | 20:08 |
| * frickler has enough for today | 20:09 | |
| clarkb | frickler: I'm not sure I understand the quoting thing? | 20:09 |
| clarkb | I haven't been abel to reproduce parsing errors locally | 20:09 |
| frickler | the # is parsed as comment character even when it is within the bash command, so tox tries to execute only the half of the command up to the # | 20:10 |
| clarkb | I see and maybe that is breaking the output of --showconfig? | 20:10 |
| clarkb | I guess we'll see data soon enough | 20:11 |
| clarkb | and ya enjoy your evening I think we've sufficiently worked around this that we can take our time to solve the underlying issues with v4 | 20:11 |
| frickler | I think those may be two independent issues | 20:12 |
| clarkb | got it | 20:12 |
| *** dviroel is now known as dviroel|brb | 20:14 | |
| clarkb | heh I can't yaml. new ps for the debugging change momentarily | 20:17 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: DNM Cat the tox --showconfig output under tox v4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 20:18 |
| clarkb | fungi: I think you can probably followup to your openstack-discuss list email saying things should be pinned to latest v3 and working? | 20:19 |
| fungi | yeah, i wanted to be sure we didn't spot anything else problematic but seems stabilized now | 20:22 |
| clarkb | tox v4 is prefixing lines with stuff like ROOT: 238 I but it doesn't do that for me locally | 20:23 |
| clarkb | you end up with multiple lines with ROOT: at th estart which breaks the ini | 20:23 |
| clarkb | Ithink this is a tox bug since showconfig should emit valid config, but we should probably handle it anyway, file a bug and hope they fix it | 20:24 |
| clarkb | but if they don't we've handled it and can take it from there | 20:24 |
| clarkb | I'm going to find lunch now, but thats the next thing to figure out, why I don't get that output locally and how to turn it off in zuul like it seems to eb turned off here | 20:24 |
| frickler | the main diff I see is that tox4 has multiline output for e.g. "pass_env", for tox3 that is shown as a list in a single line | 20:27 |
| frickler | and now I'm really gone ;) | 20:28 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: DNM Cat the tox --showconfig output under tox v4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 20:41 |
| clarkb | ok its due to add -v or -vv (we add -vv by default). ^ hasbeen updated to remove the extra args from the show config command which I think is what we want to control the ini content more directly | 20:41 |
| clarkb | I still think that is a bug in tox though. They should prefix the -v output with ini comments | 20:42 |
| clarkb | I didn't actually find lunch yet so I should actually do that now I guess | 20:42 |
| fungi | yeah, i think in tox v3 it still produced valid ini content | 20:44 |
| fungi | and used comments | 20:44 |
| fungi | i recall that being a bug in older tox which they fixed, but i guess they regressed it in v4 | 20:45 |
| clarkb | that updated patch fixed the configparser error and now we've got real errors due to the delta between versions I think | 20:46 |
| fungi | (verbose output originally broke the ability to parse it) | 20:46 |
| clarkb | others should feel free to push updates to that chagne I won't mind and I really need to sort out food now | 20:46 |
| fungi | i too need to go for food, my dinner order should be ready for pickup shortly, but i'll bbiab | 20:46 |
| clarkb | envdir is now env_dir so we need to check for both I guess | 20:47 |
| clarkb | and map any of the other things we look for that might have moved :( | 20:47 |
| *** dviroel|brb is now known as dviroel|afk | 20:47 | |
| ianw | sorry, running a little late today ... i saw some toots about tox 4 being a "complete rewrite" | 21:11 |
| ianw | i'm guessing that pretty much means we have complete failures | 21:12 |
| clarkb | ya it appears to be quite different | 21:12 |
| clarkb | tox v3 --showconfig would show you the args you are executing under so you could determine i -e was set or not. Tox v4 does not do this and tox siblings uses this info to determine what environments to do siblings installs in | 21:12 |
| clarkb | I'm not seeing an obvious method for getting that info we culd use instead | 21:13 |
| clarkb | oh it does print the default envs as testenvs though so I think that is what we have to is only look at testenvs | 21:13 |
| JayF | How do new groups get created in Gerrit? | 21:18 |
| JayF | If it's in git; I can't find it. If it's in gerrit; I can't find it :D | 21:18 |
| clarkb | JayF: you put them in the gerrit acl file that you want to use them with and it gets created | 21:18 |
| clarkb | JayF: via openstack/project-config/gerrit/acls | 21:18 |
| JayF | ack; so just reference the group and it gets created | 21:18 |
| JayF | and then I assume I need to nudge someone with admin access to get membership set? | 21:19 |
| fungi | okay, i'm back and typing with my non-chopstick hand for now | 21:19 |
| fungi | JayF: correct. refer to a nonexistent group name in an acl and automation will create the missing group as empty when that change merges, the one of the gerrit admins can add an initial user for you (if it's an openstack repo, then generally it will be the ptl or tact sig liaison) | 21:22 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: DNM Cat the tox --showconfig output under tox v4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 21:22 |
| JayF | okay; I'm going to start work on https://lists.openstack.org/pipermail/openstack-discuss/2022-December/031372.html | 21:22 |
| clarkb | looks like env_dir is canonical now but tox will still honor envdir in your tox.ini which is good because things like tempest use envdir | 21:25 |
| clarkb | I suspect that 866935 is close to working for zuul-jobs under tox v4 | 21:26 |
| clarkb | the quoting issue that frickler found seems like an important one so maybe we should plan for switching back to tox v4 by default once they have a release that fixes that? | 21:27 |
| fungi | probably, and maybe let projects know to test it with depends-on to spot anything else | 21:28 |
| clarkb | ya that sounds like a good plan. Once I get CI results back assuming it runs cleanly I'll update the change to be mergeable | 21:28 |
| clarkb | then I can WIP it and we can make a plan? | 21:28 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: ironic-release group for releasing+maint of bugfix https://review.opendev.org/c/openstack/project-config/+/866937 | 21:29 |
| fungi | wfm | 21:30 |
| clarkb | looks like there may be a few more failures I need to address | 21:31 |
| clarkb | oh also we should file a bug upstream about --showconfig emitting debug lines without comments | 21:31 |
| clarkb | it looks like old tox doesn't emit them at all even when called with -vv | 21:32 |
| fungi | i think it used to, but not sure i can find the point at which they fixed that without a lot of fiddling | 21:32 |
| clarkb | let me check if its on a different fd too | 21:33 |
| fungi | i know there was one point at which combining verbose and showconfig resulted in extra lines you needed to strip off before you could parse the output | 21:33 |
| clarkb | its not its all stdout | 21:33 |
| clarkb | fungi: yup there are a couple of prefix lines but they ren't part of the problem here | 21:33 |
| clarkb | as they aren't interleaved | 21:33 |
| clarkb | so you just ignore the first couple of lines basically (not great but much better than interleaved content) | 21:34 |
| clarkb | wait when did whitelist/allowlist become and error instaed of a warning | 21:34 |
| clarkb | ugh | 21:34 |
| clarkb | thats going to be the biggest issue I bet | 21:34 |
| corvus | if tox is breaking backwards compat now, then we probably need a version specifier in the ensure-tox role | 21:36 |
| corvus | (like an ensure_tox_version argument) | 21:37 |
| clarkb | corvus: ack I can incorporate that into what I'm writing here | 21:37 |
| clarkb | corvus: do we think default to latest and let people override? thats probably best for preventing things from becoming ancient | 21:37 |
| fungi | corvus: elodilles has a change proposed for that | 21:38 |
| fungi | it's been up for a while and may need some comments addressed, but may be a good place to continue from | 21:38 |
| clarkb | fungi: I'll try to find it and rebase on that I guess | 21:38 |
| fungi | it showed up in conflicts for the tox<4 pin change is the only reason i remembered it | 21:38 |
| corvus | i think the resulting patch will have zero lines in common with 766441 | 21:38 |
| fungi | ahh | 21:39 |
| fungi | but will obsolete it at least | 21:40 |
| corvus | as for defaults... perhaps it should default to latest... but given the catastrophic breakage, maybe we should start by defaulting it to 3 and then moving to latest in 2 weeks? | 21:41 |
| clarkb | there is a not small part of me that wonders if it wouldn't just be easier to move to nox. But I know that isn't the case | 21:41 |
| clarkb | corvus: ++ | 21:41 |
| clarkb | corvus: Ithink we don't want latest until tox fix the string quoting issue frickler found. That is a critical issue to me | 21:41 |
| corvus | even better reason to intentionally pin for now | 21:42 |
| clarkb | they no longer prefix tox logs with the env name in the file name so we have to do that when fetching logs... | 21:47 |
| clarkb | ugh | 21:49 |
| fungi | at what point do we decide that tox4 is so different than tox<4 that it's a separate tool which needs its own jobs/roles? i guess we're not quite to that point yet... | 21:51 |
| corvus | i think some of us may be close | 21:53 |
| * corvus is reading nox manual | 21:54 | |
| corvus | https://github.com/wntrblm/nox/pull/677 is amusing ("Constrain tox to <4.0.0 and minor fixes") -- it's for their tox_to_nox tool | 21:56 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Update zuul-jobs to handle tox3 and tox4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 22:04 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Move tox logs into env specific dirs https://review.opendev.org/c/zuul/zuul-jobs/+/866939 | 22:04 |
| clarkb | 866939 move tox logs into venv specific dirs under tox v3 as that should be fine to do before v4. Then the followup change does a number of things to try and make tox v4 work under zuul-jobs dogfood testing | 22:05 |
| corvus | clarkb: q's on 935 | 22:09 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: ironic-release group for releasing+maint of bugfix https://review.opendev.org/c/openstack/project-config/+/866937 | 22:09 |
| *** dasm is now known as dasm|off | 22:11 | |
| corvus | clarkb: oh and of course we'll need docs in the readme | 22:11 |
| corvus | for the new var | 22:11 |
| corvus | and remove the zuul-tox-docs comment | 22:12 |
| clarkb | corvus: ++ your comments are good and need fixing. I'll hold off on pushing an update until I can see more test results just to be sure nothing extra breaks | 22:12 |
| clarkb | corvus: zuul-tox-docs comment? | 22:13 |
| clarkb | corvus: looks like the linters jobs are failing on the string problem frickler found. I'm not going to try and fix that now intead I'll leave a comment pointing to the upstream issue and the resulting -1s will prevent us from landing this change too soon | 22:17 |
| clarkb | oh nevermind tox says you have to manually escape everything now which is wat | 22:18 |
| clarkb | thats just amazing | 22:18 |
| corvus | clarkb: in zuul-tests.d/project.yaml you have zuul-tox-docs commented out | 22:19 |
| clarkb | corvus: oh right thanks | 22:19 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Update zuul-jobs to handle tox3 and tox4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 22:21 |
| corvus | clarkb: update looks good -- but did we want to default that to '<4' now and then send an announcement that we're going to change it to ''? | 22:22 |
| corvus | or do we think that the issues of allowlist and escaping are small enough that we should go ahead and default it to '' and just tell people about the new option? | 22:23 |
| clarkb | corvus: I didn't do that because the log file checks would be wrong, but I think everything else is forward and backward compatible. I guess I can stub out what the >=4 version would look like in that test and we can swap them around when we update the dfeault? | 22:23 |
| clarkb | the escaped \# are something I'm not sure if they will be backward compatible either. I think allowlist is fine | 22:24 |
| corvus | i kinda like that approach, i think it's friendliest | 22:24 |
| clarkb | corvus: ok let me see preliminary results from this ps and then I'll update to try that | 22:24 |
| corvus | (basically, any zuul-job user will need to make updates for allowlist + escaping before they upgrade their usage of this role to v4) | 22:24 |
| clarkb | yes | 22:26 |
| clarkb | well the scaping might only be a problem for ini comment start chars | 22:26 |
| corvus | (and i like the idea of the zuul project not being responsible for tox changes, but this is a super old and widely used role that's making trouble for everyone so providing a good on-ramp as a one-time thing seems nice) | 22:26 |
| clarkb | so if you don't have ; or # you'd be ok? but ya basically those things need addressing | 22:26 |
| corvus | i'll draft a message | 22:26 |
| clarkb | corvus: in the readme for that role should I say this is likely tochange to '' in the future to default to latest? | 22:27 |
| clarkb | or do you not want to go that far yet? | 22:27 |
| corvus | i think it's fine (but not required) to mention that | 22:27 |
| *** rlandy|rover is now known as rlandy|out | 22:29 | |
| clarkb | linters: 50070 E failed with /home/zuul/src/opendev.org/zuul/zuul-jobs/tools/check_jobs_documented.py (resolves to /home/zuul/src/opendev.org/zuul/zuul-jobs/tools/check_jobs_documented.py) is not allowed, use allowlist_externals to allow it [tox/session/cmd/run/single.py:54] | 22:30 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Update zuul-jobs to handle tox3 and tox4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 22:35 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 22:35 |
| clarkb | corvus: something like that maybe? | 22:35 |
| corvus | clarkb: looks great | 22:38 |
| corvus | clarkb: fungi how's this? https://etherpad.opendev.org/p/6wky6mkVu4uo2c3vaLpJ | 22:39 |
| fungi | corvus: lgtm! | 22:41 |
| fungi | thanks | 22:42 |
| clarkb | corvus: the email looks good. My only concern is that with holidays approaching many people may ignore this until next year? I'm waffling on whether or not december 21 is too early | 22:42 |
| clarkb | but otherwise looks good to me. I guess if openstack really wants to they can set that value which is the outlet so the 21st is fine? | 22:42 |
| corvus | yeah, timing isn't great. i think it would be fine to defer that until jan 3 or 10 if you like that better | 22:43 |
| JayF | corvus: one question I'd have as a PTL reading that: how can I get tox 4 in my project for testing in CI and/or to opt-in once we're working | 22:44 |
| clarkb | JayF: one my change lands you can set ensure_tox_version: '' as a job var | 22:45 |
| clarkb | alternatively do that now and depends on | 22:45 |
| JayF | clarkb: that's what I sorta gleaned from the email; but it might be useful to make that more explicit | 22:45 |
| clarkb | adding that to the email would be good | 22:45 |
| clarkb | ya | 22:45 |
| JayF | none of this stuff is hard; but it's all unfamiliar | 22:45 |
| opendevreview | Merged opendev/system-config master: create-venv: make upgrade venv once per day https://review.opendev.org/c/opendev/system-config/+/866644 | 22:47 |
| opendevreview | Merged opendev/system-config master: pip: use latest instead of upgrade https://review.opendev.org/c/opendev/system-config/+/866645 | 22:48 |
| corvus | clarkb: https://zuul.opendev.org/t/zuul/build/452d9393f0ce41429f114941344e83d4 | 22:54 |
| corvus | oO | 22:54 |
| corvus | fungi: ^ you have a history with that | 22:59 |
| clarkb | I think this may be a quoting issue too | 23:00 |
| clarkb | https://github.com/tox-dev/tox/issues/2622 for --showconfig problems | 23:01 |
| clarkb | corvus: is it just tox_extra_args set to that path? that is weird | 23:03 |
| corvus | clarkb: JayF i'm at a loss on how to improve the last sentence of the email to address JayF's point while keeping in mind the audience of the list the subscribers to zuul-announce (ie, not just openstack). if you have any suggestions, please feel free to update the etherpad! | 23:03 |
| fungi | looking | 23:03 |
| JayF | that's an improvement for sure; ty | 23:04 |
| corvus | clarkb: i, er, don't understand what's going on here. i may be getting distracted by "ROOT: tox-bindep disabled itself as it does not support tox4 yet." is that just a red herring? | 23:05 |
| clarkb | oh I need to drop the show config output so will need a new ps anyway | 23:05 |
| clarkb | corvus: no I think that may be part of it since we start with tox v3 when we ensure-tox (I checked that) | 23:05 |
| clarkb | corvus: so maybe it is upgrading tox as part of that job specially? I'll look at that idea | 23:05 |
| corvus | clarkb: i thought that should be running with 3 though: https://zuul.opendev.org/t/zuul/build/452d9393f0ce41429f114941344e83d4/console#1/0/18/ubuntu-jammy | 23:06 |
| fungi | corvus: i think tox-bindep is something sbarnea made. tox v4 does not support plugins made for tox<4 according to the breaking changes faq | 23:06 |
| clarkb | corvus: it seems to be runnin tox multiple times though and I'm wondering if it is updating tox in between somehow? | 23:07 |
| fungi | https://github.com/tox-dev/tox-bindep | 23:07 |
| corvus | fungi: i agree -- and i think (1) it's confusing why it's running at all in an environment that should be tox3 but as clarkb says maybe it's getting upgraded during the test run... but also (2) we may need to change that if we want this test job to run tox4 | 23:08 |
| corvus | fungi: re #2 -- is there something else we can add to requirements that is compatible with tox4? | 23:08 |
| fungi | it does say `.tox installdeps: tox-bindep, tox >= 3.2` | 23:08 |
| clarkb | the child changes which flips to tox v4 globally does pass this test | 23:09 |
| corvus | clarkb: that only adds to my confusion! | 23:09 |
| fungi | is the goal to test that we can install a tox plugin? trying to figure out what tox-bindep is used for there and why | 23:09 |
| clarkb | oh its because it depends on tox >= 3.2 | 23:09 |
| clarkb | fungi: where is that installdeps line? in tox-bindep? | 23:10 |
| clarkb | I think we just stop doing anyhting with tox-bindep since its broken | 23:11 |
| corvus | clarkb: i think fungi was referring to https://zuul.opendev.org/t/zuul/build/452d9393f0ce41429f114941344e83d4/log/job-output.txt#1866 | 23:11 |
| corvus | the job claims to have installed tox 3.27.1 which seems to match that requirement to me, so i don't know why it would self-upgrade... | 23:11 |
| clarkb | https://github.com/tox-dev/tox-bindep/blob/main/setup.cfg#L50 and that comes from here I think | 23:12 |
| clarkb | corvus: its installing tox-bindep which depends on tox>=3.18 or whatever which pulls in a nested tox 4.0.0 in the end which breaks because tox-bindep isn't a compatible plugin | 23:12 |
| clarkb | I think I have a fix and if it doesn't work we can just delete this test | 23:12 |
| fungi | ahh, commit message which introduced it says we're testing the "requires" feature of tox.ini | 23:12 |
| corvus | clarkb: but if it's starting with 3.27.1 why would it decide to upgrade for 3.18? | 23:13 |
| clarkb | because this is a tox being installed in a tox env | 23:13 |
| clarkb | tox all the way down | 23:14 |
| corvus | clarkb: ah gotcha | 23:14 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Update zuul-jobs to handle tox3 and tox4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 23:14 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 23:14 |
| corvus | clarkb: any idea why the followup change passes this test? | 23:14 |
| clarkb | corvus: what is confusing to me is how the v4 flip job passed | 23:14 |
| clarkb | corvus: ya that I'm confused about. | 23:14 |
| fungi | yeah, looks like tox is being told to install another tox in a venv, so new install completely | 23:14 |
| clarkb | corvus: oh that test doesn't run on the child change for some reason | 23:15 |
| corvus | clarkb: i was just about to say that | 23:15 |
| clarkb | I skipped and thought it did, but it does not. I'll update the child change file matcher to match ensure-tox role | 23:15 |
| corvus | sounds good and that sounds like a plausible explanation | 23:15 |
| fungi | https://github.com/tox-dev/tox-bindep/issues/25 | 23:15 |
| fungi | lovely | 23:16 |
| corvus | so i think the remaining question is -- is there something other than tox-bindep we should use to address the original issue for https://review.opendev.org/812004 ? | 23:16 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 23:16 |
| clarkb | corvus: any small pip installable might work? | 23:16 |
| corvus | requestsexceptions is small | 23:17 |
| fungi | yeah. seems fine | 23:18 |
| clarkb | I need to read this test job a bit more to undersatnd how it is checking things | 23:18 |
| clarkb | its doing https://opendev.org/zuul/zuul-jobs/src/branch/master/test-playbooks/python/tox.yaml#L54-L77 | 23:19 |
| clarkb | and it is passing in a tempfile as an argument and that is simply echoing to files with bash then confirming the outputs? Yes I think we just need any small package to see that it installs something | 23:20 |
| corvus | i'm writing a change for requestsexceptions | 23:20 |
| clarkb | cool | 23:20 |
| corvus | (well, specifically, i'm writing a commit message) | 23:20 |
| fungi | thinking back, the reason i added tox-bindep there was because that was the specific addition which was failing tox jobs in some project | 23:20 |
| fungi | trying to find the discussion to confirm | 23:20 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 23:24 |
| fungi | okay, it was related to this revert: https://review.opendev.org/c/zuul/zuul-jobs/+/812001 | 23:24 |
| clarkb | fungi: https://github.com/tox-dev/tox/commit/554bd0a84391f6648136fe747b77bf22e408b02b they fixed the faq around disabling colors | 23:24 |
| fungi | nice! | 23:24 |
| clarkb | corvus: oh wait requires is special | 23:25 |
| clarkb | corvus: https://tox.wiki/en/latest/config.html#core | 23:25 |
| clarkb | corvus: a better package may be virtualenv? | 23:25 |
| clarkb | oh it will provision a new env though so ya requestexceptions is fine | 23:26 |
| clarkb | sorry for the noise I just had to digest this a bit | 23:26 |
| corvus | yeah, i didn't know about it being special, but i agree, after learning that, requestsexceptions should still work for us | 23:26 |
| clarkb | I understand the issue on the v4 flip too | 23:28 |
| fungi | https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2021-09-30.log.html#t2021-09-30T17:42:29 confirms the example was based on one found "in the wild" in openstack/tripleo-quickstart-extras | 23:28 |
| fungi | so i'm quite sure it was just chosen for convenience | 23:29 |
| clarkb | tox v4 is enforcing that posargs come after -- and the test there doesn't do that | 23:29 |
| fungi | no special reason not to swap it out | 23:29 |
| clarkb | this is anyhting thing that users will need to address | 23:29 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 23:29 |
| clarkb | that patchset illustrates what I mean re -- | 23:30 |
| clarkb | hopefully it makes sense | 23:30 |
| clarkb | I meant to say the -- requirement is *another thing* not anything users will need to address | 23:32 |
| corvus | clarkb: now that we understand that requires does a nested env -- i think my change may need to be squashed with your v3-compat change | 23:32 |
| corvus | (because it fails zuul-job-test-tox with the command line argument failure) | 23:33 |
| corvus | er, your v4-compat change | 23:33 |
| corvus | i dunno, one of your changes :) | 23:33 |
| clarkb | corvus: I think my v4 change may work because I'm pinning tox to the old compat version in the nested env | 23:34 |
| clarkb | corvus: if you wnt to squash them together or put them in order I don't think that hurts either | 23:34 |
| corvus | oh yeah, i think we should try to avoid pinning that, it's probably a time bomb | 23:35 |
| corvus | i'll rebase my change onto the end of your stack though | 23:35 |
| clarkb | sounds good | 23:35 |
| fungi | but yeah, for example tripleo-quickstart-extras won't be able to switch to tox v4 without either dropping their use of tox-bindep or getting tox v4 support implemented in tox-bindep | 23:35 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 23:36 |
| clarkb | fungi: ya | 23:36 |
| corvus | clarkb: https://zuul.opendev.org/t/zuul/build/e04140c2864e4a88bbd0efe1e87ac07d waah | 23:43 |
| corvus | clarkb: i feel like i'm not seeing the "--" i expect to see in the logs | 23:44 |
| clarkb | ya maybe I didn't identify all the places we do that? | 23:44 |
| clarkb | ya there are two tempfiles... | 23:45 |
| ianw | anyone who's not a newcomer like me know what https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/gerrit/files/static/system-cla.html is for? | 23:45 |
| fungi | ianw: so that our machine accounts don't need a signed cla to push changes | 23:47 |
| fungi | e.g. the proposal bot | 23:47 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 23:48 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 23:48 |
| clarkb | corvus: ^ hopefully that is it | 23:48 |
| fungi | ianw: once upon a time the icla had to match up to foundation memberships and it was a way around needing to have fake people associated with such accounts | 23:49 |
| clarkb | wow https://github.com/tox-dev/tox/issues/2622#issuecomment-1341761453 | 23:49 |
| clarkb | many tools consider stderr a failure. I just I don't ok | 23:51 |
| clarkb | I give up | 23:51 |
| corvus | clarkb: what do you know about ci tools? | 23:51 |
| corvus | i'm happy to review nox implementation changes | 23:52 |
| clarkb | assuming the dust has settled tomorrow I do intend on trying to track down those nodepool issues | 23:54 |
| clarkb | corvus: I WIP'd https://review.opendev.org/c/zuul/zuul-jobs/+/866943 os that it doesn't land early | 23:55 |
| ianw | fungi: thanks. just thinking about the submit-requirements, and how we might test it | 23:57 |
| clarkb | ianw: my day started by looking into the crio failures for this nodepool change's functional k8s job https://review.opendev.org/c/zuul/nodepool/+/862630 | 23:57 |
| clarkb | ianw: that led me to https://review.opendev.org/c/zuul/zuul-jobs/+/866920 but things had exploded by then | 23:57 |
| clarkb | ianw: any chance this is familiar to you as you set up the crio on jammy stuff? | 23:57 |
| ianw | Email: devel\x3akubic@build.opensuse.org | 23:58 |
| ianw | This key has expired | 23:58 |
| ianw | i think maybe the root of it is the kubic repo key has ... expired | 23:59 |
| clarkb | oh! | 23:59 |
| ianw | not that https://zuul.opendev.org/t/zuul/build/0e4d02d655e54f9b85e44e0a7cc9c619/console#1/0/111/ubuntu-jammy really says that | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!