| opendevreview | Tony Breeds proposed opendev/system-config master: Add inventory entries for new Vexxhost mirrors https://review.opendev.org/c/opendev/system-config/+/925438 | 00:55 |
|---|---|---|
| opendevreview | Tony Breeds proposed opendev/zone-opendev.org master: Remove stray whitespace https://review.opendev.org/c/opendev/zone-opendev.org/+/926690 | 00:55 |
| tonyb | I've taken the WiP flag off of: https://review.opendev.org/q/topic:add_mediawiki+is:open | 02:22 |
| tonyb | they're still very much in progress but I've asked a bunch of questions around how we bring them up a mergeable state. | 02:23 |
| tonyb | AFAICT the testing on the held node shows the only thing that doesn't work is the skin | 02:23 |
| tonyb | switching to the OpenDev logo looks a little silly as it's cropped (the .svg is 170(ish)px and the vector skin expects 155px I can fix that with some custom CSS but I think that will need to be a "when we're live" task | 02:25 |
| *** jroll05 is now known as jroll0 | 06:04 | |
| *** elodilles_ooo is now known as elodilles | 06:44 | |
| frickler | sadly openmetal had the same LE error tonight. responded to the mail thread | 06:55 |
| opendevreview | Tyler proposed openstack/diskimage-builder master: Fall back to extract-image on ubuntu build https://review.opendev.org/c/openstack/diskimage-builder/+/926748 | 12:28 |
| opendevreview | yatin proposed opendev/irc-meetings master: Update Neutron CI Weekly meeting time https://review.opendev.org/c/opendev/irc-meetings/+/926763 | 13:15 |
| opendevreview | Merged opendev/irc-meetings master: Update Neutron CI Weekly meeting time https://review.opendev.org/c/opendev/irc-meetings/+/926763 | 14:10 |
| opendevreview | James Page proposed openstack/project-config master: Retire the mistral charm repository https://review.opendev.org/c/openstack/project-config/+/926791 | 15:09 |
| clarkb | any reason to not approve the default nodeset update now? /me is catching up to make sure nothing is more important today | 15:11 |
| frickler | clarkb: I'm not aware of any blockers | 15:17 |
| tonyb | nope I think it should be fine | 15:17 |
| clarkb | cool I was just finishing up a scan of email and irc looks good. I'll approve it now | 15:18 |
| frickler | clarkb: fungi: can you please have a look at https://review.opendev.org/c/openstack/project-config/+/926405 ? I have a bunch of release patches to approve that would make for good testing of this now | 15:20 |
| * frickler forget to mention that in the meeting yesterday | 15:20 | |
| fungi | on it | 15:20 |
| clarkb | I had a couple of super minor suggestions but +2'd as I think getting some test feedback sooner is probably more important here. I didn't approve to give fungi a chance ot look too | 15:24 |
| fungi | i left some comments at ~ the same time, but also approved it | 15:25 |
| opendevreview | Merged opendev/base-jobs master: Change the default nodest to ubuntu-noble https://review.opendev.org/c/opendev/base-jobs/+/926360 | 15:26 |
| opendevreview | Merged openstack/project-config master: Update update-constraints job and script https://review.opendev.org/c/openstack/project-config/+/926405 | 15:43 |
| clarkb | spot checking openstack jobs they do appaer to be explicit using jammy as expected | 15:49 |
| fungi | excellent | 15:50 |
| frickler | thx, commented even though it is already merged now | 15:52 |
| fungi | heading out to grab a quick lunch now, but will bbiab | 15:59 |
| frickler | this looks like the first noble related gate failure, if someone has a time to look deeper that would be nice https://zuul.opendev.org/t/openstack/build/3f89f826fa334fb4a359c14961f67495 | 16:27 |
| clarkb | frickler: looking | 16:27 |
| frickler | ah, that's Pillow not having a wheel for py3.12 it seems | 16:28 |
| frickler | and we pin pillow to some old version because of other issues | 16:28 |
| clarkb | frickler: yup looks like pillow 9.5.0 only has wheels up to 3.11 | 16:28 |
| clarkb | so either the job gets pinned to jammy or we bump pillow or we include pillow build reqs in bindep.txt? | 16:28 |
| clarkb | latest pillow has wheels for 3.13 even | 16:29 |
| frickler | bumping pillow will also fail doc builds due to some sphinx stuff iirc, so 1 or 3 more likely | 16:29 |
| frickler | tag-releases has also failed https://zuul.opendev.org/t/openstack/build/c282702c8e5f47af8cf6f6ec9e5ef8c8 | 16:29 |
| frickler | sadly I need to do some downstream maintenance now | 16:30 |
| clarkb | frickler: for that second one you need to use yaml.safe_load() or yaml.load(safe=true) or whatever it is now | 16:30 |
| clarkb | I can push up a patch for the second thing | 16:30 |
| frickler | ty | 16:30 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Use yaml.safe_load in release jobs https://review.opendev.org/c/openstack/project-config/+/926821 | 16:37 |
| clarkb | this change should fix the second issue | 16:37 |
| clarkb | thinking about the first one I guess technically on openstack master projects don't need to support python3.12 yet so maybe we should pin that to jammy? | 16:38 |
| clarkb | I think as a general rule working with newer stuff is a good thing though if the project is open to having a bindep rule they may need to clear out one day | 16:38 |
| frickler | clarkb: fyi https://bugs.launchpad.net/ironic/+bug/2026345 is the bug related to the stuck pillow version, see also https://review.opendev.org/c/openstack/requirements/+/897537 | 16:50 |
| frickler | pinning the doc job to jammy for now sounds reasonable I'd say | 16:51 |
| clarkb | https://zuul.opendev.org/t/openstack/build/fe772397a4dd4facbfcd9d31667723ff is another failure (that was triggered by the fix I just pushed). I think what is happening there is that python3.12 setuptools/importlib entrypoints support must be weird | 16:53 |
| frickler | also worth pinning to jammy I'd say. but maybe also a good idea to collect these issues in an etherpad as a todo list? | 16:56 |
| clarkb | ya I'm having a hard tiem understanding how that .get() ever worked in flake8 skimming through older versions of the code. Seems like __getitem__ is implemented for usind foo[bar] but not foo.get() | 16:59 |
| clarkb | oh maybe we need to unpin hacking | 17:00 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Bump the version of hacking to latest https://review.opendev.org/c/openstack/project-config/+/926825 | 17:02 |
| clarkb | I'll rebase the other change on ^ if that works | 17:02 |
| fungi | oh, is that the cause of the linters failure on the yaml.safe_load() change? | 17:04 |
| clarkb | yes | 17:04 |
| clarkb | we're using really old flake8 via old hacking | 17:04 |
| clarkb | I don't know yet that latest hacking fixes things, but I'm hoping it does :) | 17:04 |
| fungi | fingers crossed | 17:05 |
| clarkb | haha and now the irc access job has failed | 17:06 |
| clarkb | no ssl wrap socket... | 17:06 |
| clarkb | that might be a harder one to fix properly | 17:06 |
| fungi | pew pew lasers | 17:06 |
| fungi | looking | 17:06 |
| clarkb | ah looks like we can use SSLContext.wrap_socket | 17:07 |
| fungi | https://github.com/eventlet/eventlet/issues/795 | 17:08 |
| fungi | yeah | 17:08 |
| clarkb | I've got a patch now but will wait for results from linting as I may have to fix errors from that too | 17:11 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 17:17 |
| clarkb | surprisingly only one rule violation. That change fixes the hacking version, the new hacking rule violation, and the ssl.wrap_context issue | 17:18 |
| clarkb | or it should, zuul will confirm | 17:18 |
| fungi | clarkb: close but no cigar | 17:31 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 17:31 |
| clarkb | fungi: yup PROTOCOL_TLS_CLIENT implies some extra checking that I'm just goign to disable per the comment in there | 17:31 |
| fungi | wfm | 17:32 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 17:40 |
| gtema | So what is the solution for the pillow failures? Pin own jobs to jammy? | 17:43 |
| tonyb | gtema: I'm not sure we have a solution yet. but a temporary pin would be a quick bandaid | 17:44 |
| gtema | Ok, thks for info | 17:45 |
| fungi | gtema: or unpin pillow may work | 17:45 |
| gtema | Fungi, will not work for me since I install openstack services in my jobs | 17:46 |
| fungi | yeah, in that case using the nodeset corresponding to a valid openstack pti target for that release is your best bet | 17:47 |
| gtema | Ok | 17:47 |
| clarkb | you can also theopretically add libjpeg-dev to your bindep file | 17:48 |
| clarkb | then pip can build a wheel for you | 17:48 |
| gtema | Will try | 17:48 |
| fungi | ah, yeah if the only problem with pillow is lack of py312 wheels, that probably will get around it | 17:48 |
| clarkb | 926825 passed the linter job multiple times and now fails on some random ansible thing with the fourth patchset and that patchset didn't even touch anything ansible realted... | 17:49 |
| clarkb | ERROR! Unexpected Exception, this is probably a bug: '_AnsiblePathHookFinder' object has no attribute 'find_spec' | 17:49 |
| clarkb | I guess recheck to see if it is consistent | 17:49 |
| fungi | agreed | 17:50 |
| clarkb | the irc access job passed though so this is in theory very close | 17:51 |
| clarkb | https://superuser.com/questions/1847344/how-to-solve-ansible-venv-runtime-exception-in-specific-versions this may be an ansible compatibility issue | 17:54 |
| clarkb | odd that it worked other times | 17:54 |
| clarkb | I'm working to bump up ansible and ansible-lint now | 17:55 |
| clarkb | if anyone is wondering the version matrix between ansible, ansible core, ansible-lint, and ansible-compat is incredibly confusing and not really documented at all | 18:02 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 18:03 |
| clarkb | that is a minimal bump to just ansible. We'll see if ansible-lint and ansible-compat explode with those versions | 18:04 |
| clarkb | https://zuul.opendev.org/t/openstack/build/4c9705925ea04d5caa4b1c8bcbc824d1/log/job-output.txt does anyone else see the error in there? I cannot see it | 18:15 |
| fungi | "an AnsibleCollectionFinder has not been installed in this process" | 18:16 |
| fungi | could it be that? | 18:16 |
| fungi | but regardless, the "error" seems to be the nonzero exit of the command tox called | 18:17 |
| clarkb | right xargs returned 123 bceause one of the ansible-lint commands returned non zero | 18:17 |
| clarkb | but none of the output indicates a failure/error from ansible-lint | 18:17 |
| fungi | stderr was also empty | 18:18 |
| clarkb | the ansible collection finder issue is due to older lint and newer ansible and apparently the compat library has a fix in it. So maybe I unpin the compat lib? | 18:19 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 18:19 |
| fungi | yeah, i was just looking at https://github.com/ansible/ansible-lint/issues/2945 and see that was from 1.5 years ago | 18:19 |
| fungi | looks like the fix was included as far back as v6 | 18:21 |
| clarkb | in ansible-compat v6 you mean/ | 18:21 |
| fungi | oh, i was looking at the "add support for ansible 2.15" commit in ansible-lint | 18:21 |
| clarkb | I'm about to pin that job back to jammy and leave the hacking bump in place. Then honestly maybe we should stop running ansible-lint because its getting a bit ridiculous that the version matrix involves like 5 different things that aren't documented | 18:23 |
| fungi | i have to agree, ansible-lint doesn't generally catch errors beyond what basic yaml parsing would | 18:23 |
| clarkb | not sure if yall saw my email to openmetal but I have confirmed that a new cert appears to have been issued | 18:25 |
| fungi | but yeah, the fix seems to be in ansible-compat v3.0.0 which is pulled in by ansible-lint v6 | 18:25 |
| clarkb | fungi: possible that this is a similar but different issue related to even enwer ansible? because we should already be using those particular versions | 18:26 |
| fungi | based on the tags indicated for https://github.com/ansible/ansible-compat/commit/6f3a36d anyway | 18:26 |
| clarkb | maybe the failure is related to all the warnings if we are treating warnings as errors? | 18:26 |
| clarkb | I don't see evidence of that in our .ansible-lint config file though | 18:27 |
| clarkb | maybe I need to unpin ansible-lint and just fix all the resulting errors | 18:27 |
| fungi | i guess newer ansible-lint wants you to tell it whether to treat specific yaml files as playbooks or as roles? | 18:28 |
| clarkb | fungi: well I didn't change the ansible-lint version | 18:28 |
| fungi | mmm, right | 18:28 |
| clarkb | AttributeError: 'str' object has no attribute 'resolve' is the latest error | 18:29 |
| clarkb | I'm going to try unpinning ansible-lint and if that doesn't work I don't know what else to do | 18:29 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 18:30 |
| clarkb | like at the very least I would expect a tool like that to either explode at install saying "this combo isn't compatibile and pip can't resolve the requirements" or actually execute when run and provide an error for what broke | 18:33 |
| clarkb | ok that managed to get it to run and report errors but there are a lot of errors. I'm going to pin the job to jammy to get this landed and fix the release job then followup with an unpin on jammy that fixes the lint issues | 18:40 |
| fungi | sounds good. i'd also be fine with just skipping the new error classes while we discuss whether to keep the job at all | 18:41 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 18:42 |
| clarkb | fungi: most of them are straighforward name your tasks/plays errors which I'm fine with. Then maybe a few cases will be ignored. We'll see | 18:42 |
| clarkb | in theory that latest patchset should work since it pins the ansible stuff back to jammy and the hacking stuff should work there too | 18:43 |
| clarkb | the last patchset ran on noble even though I told it to use jammy because project-config is not speculatively tested | 18:50 |
| clarkb | so I dunno fi we want to force merge that or what | 18:50 |
| clarkb | maybe force merge a change that only pins to jammy then have this change run on top of it? | 18:50 |
| fungi | that sounds safest | 18:51 |
| clarkb | ok I'll split it up now | 18:51 |
| fungi | i'm set up to do the zuul bypassing | 18:52 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 18:53 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Run openstack-zuul-jobs-linters on Jammy https://review.opendev.org/c/openstack/project-config/+/926843 | 18:53 |
| opendevreview | Merged openstack/project-config master: Run openstack-zuul-jobs-linters on Jammy https://review.opendev.org/c/openstack/project-config/+/926843 | 18:55 |
| fungi | i think we'll need to recheck 926825 since it probably got enqueued without the parent applied? | 18:56 |
| clarkb | ya | 18:57 |
| clarkb | fungi: I rechecked and it is going to pass this time if you want to rereview and approve | 19:23 |
| fungi | yup | 19:23 |
| clarkb | I'll rebase the fix for release jobs on it now | 19:23 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Use yaml.safe_load in release jobs https://review.opendev.org/c/openstack/project-config/+/926821 | 19:24 |
| clarkb | that should pass now. | 19:24 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Run ansible-lint on Ubuntu Noble https://review.opendev.org/c/openstack/project-config/+/926848 | 19:26 |
| clarkb | and this change is about half complete but I want to see it run and double check I'm on the right path now | 19:26 |
| clarkb | oh ha now the unpin of ansible-lint is running on jammy again beacuse of the lack of speculative execution. That said I think if we get that working on jammy it should run on noble | 19:28 |
| opendevreview | Merged openstack/project-config master: Omnibus fixes for running jobs on Noble with Python3.12 https://review.opendev.org/c/openstack/project-config/+/926825 | 19:39 |
| opendevreview | Merged openstack/project-config master: Use yaml.safe_load in release jobs https://review.opendev.org/c/openstack/project-config/+/926821 | 19:42 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Run ansible-lint on Ubuntu Noble https://review.opendev.org/c/openstack/project-config/+/926848 | 19:50 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Run ansible-lint on Ubuntu Noble https://review.opendev.org/c/openstack/project-config/+/926848 | 19:56 |
| clarkb | we should probably push changes to system-config and ozj and similar to see if noble blow them up for any reason. Actually reviewing tonyb's wiki changes is on my todo list maybe that will generate new patchsets for system-config | 20:52 |
| clarkb | tonyb: ok reviews posted. | 21:35 |
| clarkb | let me know if you have any questions about what I wrote | 21:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Noble packages to Docker mirror https://review.opendev.org/c/opendev/system-config/+/926861 | 22:42 |
| clarkb | infra-root ^ fyi I've just discovered that we'll need that to more fully enable noble nodes | 22:42 |
| corvus | +3 | 22:47 |
| clarkb | fungi: thank you for offering to check on the deployment of that | 22:50 |
| fungi | no problem, i'll be up for a while still | 22:50 |
| fungi | waiting for the production db import to finish on the held etherpad node, it's taking... a while | 22:51 |
| fungi | but less labor intensive than trying creating a bespoke test payload on a separate held node with the earlier version or something | 22:51 |
| clarkb | I've got a family dinner I have to pop out for soonish. Fwiw I'm totally not offended if people pin the nodeset on jobs that are struggling. It would be helpful if a small amount of debugging is done first though to catch issues like the docker deb mirror problem that need to be fixed more centrally | 22:52 |
| fungi | looks like it's about 2/3 done loading the dump into mariadb now | 22:52 |
| fungi | i've loaded yesterday's database backup from production into the held node at 104.130.140.169 and see that https://etherpad.opendev.org/p/clarkb-test looks broken | 23:27 |
| clarkb | fungi: yes that pad has been broken for years on production. Not sure why | 23:28 |
| fungi | oh! good ;) | 23:28 |
| clarkb | I would try something lese like gerrit-upgrade-3.7 etc | 23:28 |
| clarkb | er I meant to say 3.8, 3.9 etc | 23:28 |
| fungi | okay, https://etherpad.opendev.org/p/gerrit-upgrade-3.9 looks like it works | 23:29 |
| clarkb | we should probably delete clarkb-test on prod and start over with a new one I guess | 23:29 |
| fungi | including its headings | 23:29 |
| clarkb | oh neat | 23:29 |
| clarkb | so maybe this will mostly just work. We probably need to find a pad with headings that ep_headings2 doesn't support | 23:30 |
| clarkb | and check if that explodes things or not and if not we're good to proceed? | 23:30 |
| fungi | https://etherpad.opendev.org/p/mm3migration used several levels of headings too | 23:30 |
| fungi | but yeah, i think the old ep_headings supported 6 levels and ep_headings2 only has 4 | 23:31 |
| clarkb | fungi: does the monospace for code format work in the new one | 23:31 |
| fungi | yes | 23:31 |
| clarkb | there should be monospaced content in the gerrit upgrade pads | 23:31 |
| clarkb | cool | 23:32 |
| fungi | the mm3migration pad has lots too | 23:32 |
| fungi | in all honesty, level 5 and 6 headings were kinda useless because they were smaller than normal text, so odds that we'll find pads using them at random are small (though maybe a db query could) | 23:33 |
| clarkb | ya and probably worst case we export the raw pad data and just reenter it | 23:33 |
| fungi | in less fun news, https://review.opendev.org/926861 has multiple failures i haven't looked into yet. tox-linters failed but also opendev-buildset-registry hit a retry_limit | 23:34 |
| clarkb | if it generally works we're also probably generally good to proceed :) | 23:34 |
| clarkb | oh shoot probably more noble things | 23:34 |
| clarkb | the buildset registry problem is a chicken and egg issue. so we can pin the nodeset on that job | 23:34 |
| clarkb | the flake8 issue is the same one project-config hit. | 23:35 |
| clarkb | I'll write a couple of changes | 23:35 |
| fungi | Failed to update apt cache: W:The repository 'http://mirror-int.dfw.rax.opendev.org/deb-docker/noble noble Release' does not have a Release file., W:Data from such a repository can't be authenticated and is therefore potentially dangerous to use., W:See apt-secure(8) manpage for repository creation and user configuration details., E:Failed to fetch | 23:35 |
| fungi | http://mirror-int.dfw.rax.opendev.org/deb-docker/noble/dists/noble/stable/binary-amd64/Packages 404 Not Found [IP: 10.208.224.195 80], E:Some index files failed to download. They have been ignored, or old ones used instead. | 23:35 |
| fungi | looks like we may have a chicken-n-egg problem there? | 23:35 |
| clarkb | ya that is the chicken and egg issue | 23:36 |
| clarkb | hrm we don't pin flake8 in system-config so why is that failing | 23:36 |
| clarkb | oh nevermind we do pin it somewhere | 23:37 |
| clarkb | via hacking | 23:37 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Noble packages to Docker mirror https://review.opendev.org/c/opendev/system-config/+/926861 | 23:40 |
| clarkb | fungi: corvus ^ that pins things as the first step | 23:41 |
| fungi | and then we'll need to bypass gating and merge it, i suppose | 23:41 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run tox-linters on Noble https://review.opendev.org/c/opendev/system-config/+/926863 | 23:42 |
| clarkb | fungi: no we shouldn't need to because system-config can run speculatively | 23:42 |
| fungi | yeah, but it can't speculatively add the package mirror it's trying to access, can it? | 23:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run opendev-buildset-regitry on Noble https://review.opendev.org/c/opendev/system-config/+/926864 | 23:44 |
| clarkb | fungi: right that is why I pinned it to jammy since jammy has the packages | 23:44 |
| fungi | ah, got it | 23:44 |
| corvus | (in theory.....;) | 23:44 |
| fungi | heh | 23:44 |
| clarkb | fungi: 925864 unpins and we'll need to recheck that when the mirror content is available | 23:44 |
| fungi | yep, makes sense now | 23:44 |
| fungi | i should have looked at the revision | 23:44 |
| clarkb | landing the base change in that stack should get most things ahppy again. Then I can followup with the unpins (hacking in particular because I'm sure there are a bunch of rules to fix or ignore) | 23:45 |
| fungi | yeah | 23:45 |
| clarkb | the linters job and the buildset registry job both passed with the pin in place | 23:49 |
| clarkb | I expect the change to be mergeable now but I need to pop out for dinner. Feel free to upadte if necessary | 23:49 |
| fungi | will do, thanks!!! | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!