| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 02:43 |
|---|---|---|
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 02:43 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 02:43 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Add new openstack/devstack based functional testing https://review.opendev.org/c/openstack/diskimage-builder/+/949942 | 02:51 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove nodepool based testing https://review.opendev.org/c/openstack/diskimage-builder/+/952953 | 02:51 |
| opendevreview | Tony Breeds proposed openstack/diskimage-builder master: Remove testing for f37 https://review.opendev.org/c/openstack/diskimage-builder/+/952954 | 02:51 |
| ianw | tonyb: ^ so what was the wrap up on the private network not being seen by the devstack user? | 03:01 |
| tonyb | ianw: I need to open a bug against openstack-client/devstack but for now disabling the openstack-cli-server is a reasonable work around | 04:40 |
| tonyb | ^^^ are ready for reviews. fc37 & gentoo fail devstack, but passes nodepool. opensuse-15 fails both. I'm inclined to go with what's there and fix gentoo and opensuse (as they're both non-voting) as additional fixes | 04:42 |
| tonyb | I'll also rebase the {Rocky,CentOS-Stream}-10 changes onto the end of the series | 04:43 |
| ianw | oh i see, in the job, cool | 04:58 |
| tonyb | actually I remembered I need to reduce the retries on the jobs as well #ooops | 05:06 |
| frickler | ianw: the bug is that the openstack-cli-server daemon in devstack ignores the --os-cloud option and always uses the credentials it was started with, which happens to be --os-cloud=devstack-admin | 07:32 |
| ianw | ++ | 07:55 |
| opendevreview | Merged openstack/project-config master: Update neutron grafana dashboard with new FWaaS jobs https://review.opendev.org/c/openstack/project-config/+/950660 | 08:41 |
| *** jroll09 is now known as jroll0 | 08:51 | |
| opendevreview | Merged opendev/system-config master: Update rabbitmq puppet module https://review.opendev.org/c/opendev/system-config/+/952946 | 13:14 |
| fungi | that ^ failed the same way in deploy... i've gone through and tested cloning every repo in our current modules.env (except for the ones we're hosting in opendev) and was able to pull them all. i wonder if we need to clear out the old checkout on bridge? | 13:27 |
| corvus | oh maybe so...maybe it can't recover from the url change? | 13:32 |
| fungi | i'm just guessing that it will only reclone if the rep is not there, and won't update the remotes in an existing clone | 13:35 |
| Clark[m] | I wonder if we need rabbitmq at all for any existing services..maybe just comment it out? | 14:09 |
| corvus | it's for storyboard | 14:13 |
| Clark[m] | ah | 14:14 |
| Clark[m] | and yes I suspect that the remote doesn't get updated. Maybe move the repo aside in case we need to inspect the old repo state post new clone | 14:14 |
| corvus | i'll rm -fr /etc/puppet/modules/rabbitmq any objections? | 14:14 |
| corvus | ok i'll mv it then | 14:14 |
| fungi | sounds good to me, thanks! | 14:15 |
| corvus | issued mv rabbitmq/ rabbitmq-2025-06-2 | 14:15 |
| corvus | s//`/, s/2/20`/ | 14:15 |
| clarkb | tonyb: frickler reviewed the dib devstack testing change and I've done a quick followup too. I think there are a small number of cleanups to do then probably a number of followup we can consider that aren't necessary for this change (like sorting out the use of openstack cli server for devstack proper and doing something different in our tests and adding ipv6 checks) | 14:34 |
| clarkb | tonyb: frickler: but I think we don't need to scope creep that change any further just clean up its existing implementation then proceed with what we've got and add followup improvements as we are able | 14:35 |
| clarkb | tonyb: do we know what it was that fixed almalinux testing? was it the switch to using fips? | 14:39 |
| frickler | I don't remember any almalinux specific issue, do you mean in the current patch? iiuc that was all variations of the cli-server issue | 14:52 |
| clarkb | frickler: early iterations of the change all failed against almalinux. When I added noble testing too as a comparison it just worked | 15:08 |
| clarkb | one idea was that there were dhcp problems using the public net and that was why I suggested switching to fips | 15:08 |
| clarkb | not sure if that ended up being the case or if there was some other subtle fix I missed in my review | 15:08 |
| frickler | clarkb: ah, I think that was in the iteration where the instances were started on the public network, then, where there is no DHCP server. glean does work with config-driver network config and static addresses in that case, but almalinux uses networkmanager and that always requires dhcp afaict? | 15:12 |
| frickler | s/driver/drive/ | 15:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to Gitea 1.24.1 https://review.opendev.org/c/opendev/system-config/+/948560 | 15:13 |
| clarkb | frickler: got it that would explain it | 15:13 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 15:16 |
| clarkb | as expected the gitea .1 release didn't take long. I've udpated my change to deploy the 1.24 update and also set up an autohold followup so that we can check the held instance interactively | 15:17 |
| fungi | the new .1 changes are fairly minimal | 15:42 |
| clarkb | yup which is probably a good sign that early adopters haven't found any major issues | 15:43 |
| fungi | i concur | 15:45 |
| fungi | today is probably not a great day of the week to do upgrades, but might be nice to try for early next week? | 15:45 |
| clarkb | yup early next week sounds great | 15:46 |
| clarkb | once the held node is up we can check for anything testing misses too | 15:46 |
| clarkb | with dib and gitea out of the way the other things on my mind recently are the openstack dco switch which I think fungi has under control and the setuptools deprecations that will eventually hit pbr. Do we know what if anything pbr is planning to do at this point to mitigate those deprecations? | 15:57 |
| fungi | i don't remember whether anyone even figured out if they're going to impact pbr, at least one thing was an unintentional regression in setuptools | 16:05 |
| clarkb | fungi: the ones planned for october will iirc | 16:06 |
| clarkb | specifically the easy_install cleanup will be completed and the bits pbr uses in there will be gone. Also there is the pkg_resources cleanup which will also affect pbr. Not sure if that was given a timeline though | 16:07 |
| clarkb | https://github.com/pypa/setuptools/blob/main/setuptools/command/easy_install.py | 16:08 |
| clarkb | I think we did confirm that the pip written scripts are still "bad" eg they do the slow lookups of entrypoints | 16:10 |
| clarkb | however with pkg_resources also being deprecated maybe that won't be the case forever. That said I suspect we'll have to do something to make pbr keep working into the future even if that is to let setuptools write scripts for us and accept their slowness | 16:11 |
| clarkb | I've got to pop out momentarily, but this isn't urgent either. We have until october I just didn't want it to fall off the radar | 16:12 |
| clarkb | for opendev we can probably use not pbr though and we'll be ok. | 16:12 |
| fungi | from a pip development perspective, i think it already uses importlib in most places, only falling back on pkg_resources when necessary | 16:16 |
| fungi | https://review.opendev.org/c/openstack/pbr/+/842410 from a few years ago and your earlier https://review.opendev.org/c/openstack/pbr/+/662035 did it for dist version purposes, i guess the other uses of pkg_resources haven't been excised yet | 16:19 |
| fungi | so i was wrong about "most places" | 16:23 |
| fungi | er, i meant s/pip/pbr/ earlier | 16:29 |
| frickler | any pbr work will likely need a good deal of CI fixing getting done first. I pinged damani about that some time ago but I haven't seen any update so far. stephenfin did start some of the work but iiuc ran out of funding | 16:38 |
| frickler | probably a good idea to discuss this at the TC level by now, seems to be similar to the eventlet situation to me | 16:40 |
| frickler | gouthamr: ^^ FYI, not sure whether to just add this to the agenda | 16:40 |
| gouthamr | frickler: ack, we could.. i don't have the full context, but would inviting damani and/or stephenfin help? | 16:48 |
| gouthamr | or we could add it to the oslo team's agenda if it has fallen out of their radar given all the eventlet work | 16:48 |
| Clark[m] | gouthamr basically setuptools broke PBR semi recently and they reverted the change and set a date in October to reapply it. That way PBR could have time to update and avoid the problem | 17:19 |
| Clark[m] | The exact date is encoded in the GitHub link I shared earlier. Before that date PBR needs to be updated to avoid breaking again | 17:19 |
| gouthamr | ah thank you Clark[m] | 17:21 |
| gouthamr | https://github.com/pypa/setuptools/blob/1fe0c5d2f46e141d6097910d2905a3fce5e30c52/setuptools/command/easy_install.py#L27 | 17:21 |
| * gouthamr sends a note to stephenfin and damani | 17:22 | |
| fungi | one of the setuptools maintainers opened https://bugs.launchpad.net/pbr/+bug/2107732 about it | 17:50 |
| clarkb | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_348/openstack/3482ab7f8eb5450f9cbb1c3858b98e29/bridge99.opendev.org/screenshots/ gitea 1.24.1 screenshots | 17:54 |
| clarkb | https://200.225.47.60:3081/opendev/system-config held node on the child change | 17:54 |
| clarkb | at first glance this looks like it is working | 17:55 |
| clarkb | so ya I think we can aim for a monday update | 17:55 |
| clarkb | https://200.225.47.60:3081/opendev/system-config/src/branch/master/playbooks/roles/gerrit/tasks/main.yaml new gitea has grown a file browser navigation panel | 18:16 |
| clarkb | on the left side there (that is new) | 18:16 |
| clarkb | text search seems to work as well as it always has (eg mostly but not entirely the way we would prefer). `GIT_SSL_NO_VERIFY=1 git clone https://200.225.47.60:3081/opendev/system-config` worked for me as well and seems to produce a repo I can interact with locally | 18:19 |
| clarkb | I think I'm happy with this. let me know if there are other checks we want to perform before upgrading | 18:19 |
| clarkb | looking at setuptools' ScriptWriter it does use importlib first then falls back to pkg_resources. Its possible that we can just say pbr stops manaing those scripts and accept if there is any slowness? | 18:21 |
| clarkb | or maybe setuptools would be open to avoiding either library and doing more what pbr does | 18:22 |
| clarkb | which is direct importation because you don't need the fancy stuff (there may be corner cases though where you do, but pbr has done this for a long time and it has never been an issue) | 18:22 |
| clarkb | fungi: do you know if pip and pypi negotiate some sort of rate limit process? I'm looking at https://zuul.opendev.org/t/zuul/build/2920f617b1d0482fb2990b9c8ade3c8b/log/job-output.txt#9068 which eventually times out and it seems to be very slowly requesting data from pypi | 18:28 |
| clarkb | another explanation may be that dependency resolution is processing the results of each downloaded version before requesting the next? | 18:28 |
| clarkb | it is possible we need to set package version floors for more deps | 18:29 |
| clarkb | to aid the dependency resolution system | 18:29 |
| fungi | yeah, occam says we're just giving it too hard of a problem to solve | 18:43 |
| fungi | though if they do any rate-limiting it would be in fastly's cdn endpoints | 18:43 |
| clarkb | heh tehre is a gitea 1.24.2 now | 20:50 |
| clarkb | its about 9 minutes old. I'll update my change and rotate the node hold | 20:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to Gitea 1.24.2 https://review.opendev.org/c/opendev/system-config/+/948560 | 20:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 20:54 |
| fungi | wow | 20:55 |
| fungi | very minimal changelog at least | 20:56 |
| fungi | must have been a new regression | 20:56 |
| clarkb | ya this seems pretty typical of their release process | 20:58 |
| clarkb | whcih is why I'm never in a rush to deploy .0 | 20:58 |
| clarkb | https://200.225.47.52:3081/opendev/system-config is the newly held gitea 1.24.2 server | 23:22 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!