| clarkb | that works for me. Agenda sent | 00:02 |
|---|---|---|
| tonyb | clarkb: Ahh okay, that makes sense. I guess I can work on that as well. | 00:10 |
| tonyb | Question about the wiki switch/update. How do we want to handle logs from the apache process inside the container? At the moment they're being written to stdout/stderr which means they end up in docker logs / syslog. Should I switch that so they're written to something like /var/log/containers/apache2/* ? I guess similar question for the mariadb/memcached/elasticsearch containers. | 00:19 |
| tonyb | How do we test/upgrade the bridge host? It looks like it's running Focal and therefore is on the list to update. | 00:28 |
| tonyb | I noticed this because we can no longer install ansible from git | 00:29 |
| opendevreview | Tony Breeds proposed openstack/project-config master: Build a stow of python 3.13. https://review.opendev.org/c/openstack/project-config/+/918482 | 00:44 |
| Clark[m] | tonyb: re wiki if you look at other container setups we redirect stdout docker logs to syslog which I think would be fine here | 00:46 |
| tonyb | Clark[m]: Oh cool. | 00:46 |
| Clark[m] | tonyb: re bridge all of our system-config jobs deploy a test bridge and it is tested somewhat implicitly by being able to run the config management for stuff. I think you can propose a change to see if that works on jammy/noble. Then it's a matter of deploying it, syncing stuff and switching zuul over | 00:47 |
| Clark[m] | tonyb: what do you mean we can no longer install Ansible from git. Do you mean in the Ansible devel job? | 00:47 |
| Clark[m] | I think we install from packages elsewhere | 00:48 |
| tonyb | Clark[m]: Yeah the Ansible devel job fails because python is too old | 00:48 |
| Clark[m] | Oh and we need to update ssh rules on nodes to allow ssh from the new host as well as the current one | 00:48 |
| Clark[m] | Ack | 00:49 |
| tonyb | I get that we use packages elsewhere but that's an early warning. | 00:49 |
| tonyb | Clark[m]: okay so bridge is special but not as special as I was thinking. | 00:50 |
| Clark[m] | Yup. I just wanted to make sure we weren't breaking in production unexpectedly | 00:50 |
| tonyb | Okay. I might try and get the ansible-devel job working again as an early step | 00:51 |
| Clark[m] | Ya it's special due to its position in the world but the testing of it works just like everything else for the most part | 00:51 |
| tonyb | I guess we have the file matchers set so if we update system-config-run we also run all the services? | 00:52 |
| Clark[m] | Maybe? If we're missing matchers we should have then we can always add them | 00:54 |
| tonyb | Oh it looks like bridge99 is already jammy Hmm https://opendev.org/opendev/system-config/src/branch/master/zuul.d/system-config-run.yaml#L62 | 00:55 |
| tonyb | I'll do more reading/digging | 00:55 |
| opendevreview | Merged opendev/glean master: testing: remove centos7 and 8 https://review.opendev.org/c/opendev/glean/+/921911 | 00:58 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: dib-functests: run on bookworm https://review.opendev.org/c/openstack/diskimage-builder/+/922697 | 01:07 |
| opendevreview | Takashi Kajinami proposed opendev/storyboard master: Adopt to recent tox https://review.opendev.org/c/opendev/storyboard/+/922699 | 02:25 |
| opendevreview | Takashi Kajinami proposed opendev/storyboard master: Fix test executions https://review.opendev.org/c/opendev/storyboard/+/922699 | 02:31 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Rebalance Mailman's and Exim's outgoing batch size https://review.opendev.org/c/opendev/system-config/+/922703 | 03:23 |
| opendevreview | Takashi Kajinami proposed opendev/storyboard master: Fix test executions https://review.opendev.org/c/opendev/storyboard/+/922699 | 03:27 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Run ansible-devel under python-3.11 https://review.opendev.org/c/opendev/system-config/+/922704 | 04:23 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Run ansible-devel under python-3.11 https://review.opendev.org/c/opendev/system-config/+/922704 | 04:48 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Run ansible-devel under python-3.11 https://review.opendev.org/c/opendev/system-config/+/922704 | 07:43 |
| frickler | infra-root: websites on static.o.o seem to be very slow, taking a closer look now | 08:27 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Run ansible-devel under python-3.11 https://review.opendev.org/c/opendev/system-config/+/922704 | 08:41 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Run ansible-devel under python-3.11 https://review.opendev.org/c/opendev/system-config/+/922704 | 09:25 |
| *** diablo_rojo_phone is now known as Guest10734 | 11:21 | |
| fungi | frickler: did you manage to spot the cause? i see a slow afs warning in dmesg from ~30 minutes after you mentioned the issue | 13:12 |
| fungi | aha, never mind. i didn't finish reading everywhere | 13:13 |
| corvus | where can i read about slow static.o.o? | 14:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Delete centos 8-stream mirror content https://review.opendev.org/c/opendev/system-config/+/922749 | 16:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove centos rsync mirroring tooling https://review.opendev.org/c/opendev/system-config/+/922750 | 16:18 |
| corvus | i'm going to restart the schedulers and web servers again for another regression fix (that probably doesn't impact opendev much if at all, but just in case) | 16:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove centos rsync mirroring tooling https://review.opendev.org/c/opendev/system-config/+/922750 | 17:03 |
| opendevreview | Merged openstack/diskimage-builder master: Disable Stream-8, non-vote FC 37, and assume yes on debian functest https://review.opendev.org/c/openstack/diskimage-builder/+/921823 | 17:03 |
| corvus | #status log restarted zuul schedulers/web | 17:10 |
| opendevstatus | corvus: finished logging | 17:10 |
| fungi | thanks! | 17:21 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: Provide an ability to disable serial console injection https://review.opendev.org/c/openstack/diskimage-builder/+/922441 | 17:27 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: remove console entries when console is disabled https://review.opendev.org/c/openstack/diskimage-builder/+/922442 | 17:27 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: minor ci: quick cleanup and remove centos8 https://review.opendev.org/c/openstack/diskimage-builder/+/922763 | 17:29 |
| dpanech | Hi we are getting infrastructure-related errors in Zuul jobs, eg here: https://zuul.opendev.org/t/openstack/build/b168543cf18f49b592967ee6b81a3aa6 (https://mirror-int.dfw.rax.opendev.org/ not reachable) | 17:38 |
| clarkb | dpanech: looks like the mirror returned a 404 response | 17:41 |
| fungi | that looks like an ansible change where it has started looking for debian-11.9 instead of debian-11 | 17:45 |
| clarkb | ya it seems like it can't find a pyzmq wheel on pypi that is valid for the current platform so falls back to the wheel cache and can't find anything there either | 17:46 |
| clarkb | but in all cases the mirror appears to be accessible and returns appropriate responses | 17:46 |
| clarkb | and they must be doing binary only installs because the sdist ins't fetched? | 17:47 |
| dpanech | Sorry, is there anything wrong with our scripts? | 17:50 |
| clarkb | dpanech: I don't know enough about what the job is trying to do there to say one way or another. I'm just describing what I see in the lgos and I don't think it is a mirror problem. | 17:52 |
| dpanech | clarkb: I'll ask one of our developers to join here | 17:53 |
| clarkb | https://zuul.opendev.org/t/openstack/build/b168543cf18f49b592967ee6b81a3aa6/console#0/4/23/debian-bullseye is where the wheel cache is configured so ya at least part of the issue here is the one fungi points out. Need to fix the config for that so that it points at 11 not 11.0 | 17:55 |
| clarkb | *11.9 | 17:55 |
| clarkb | https://mirror.dfw.rax.opendev.org/wheel/debian-11-x86_64/pyzmq/ and there is a wheel there | 17:56 |
| clarkb | hrm it does actually seem to find the sdist and wheels from pypi upstream. So why isn't it simply using one of those | 18:00 |
| dpanech | clarkb: >>Need to fix the config for that so that it points at 11<< -- I don't think we changed anything in Zuul config recently. What config would that be? | 18:01 |
| clarkb | dpanech: thats a separate issue. I don't think it explains why your job failed | 18:02 |
| fungi | most likely a behavior change in ansible, for how it populates distribution version variables | 18:02 |
| fungi | as for the error itself, pip requires all index urls to be valid even if they don't contain matching packages | 18:02 |
| clarkb | it does this: Downloading https://mirror-int.dfw.rax.opendev.org/pypifiles/packages/58/9d/d26c8808cfc5a033d2fcb724767cc2e183af1c2af1440865776a113cc6f9/pyzmq-20.0.0-cp39-cp39-manylinux1_x86_64.whl.metadata | 18:02 |
| clarkb | then throws an exception reading that metadata | 18:03 |
| fungi | and that's through the caching proxy, not from the wheel mirror | 18:05 |
| clarkb | correct | 18:05 |
| clarkb | to be clear I think the wheel mirror thing is a problem and one that should be addressed. But I think if fixing that fixes this problem we're going to mask the actual problem which is that you can't install the wheel from pypi proper which should be the preferred method | 18:06 |
| fungi | the last successful run for that job was 2024-06-21 20:28:48 (~4 days ago), so whatever changed did so after that | 18:08 |
| fungi | that sort of coincides with the pip 24.1 update, which happened the day before | 18:09 |
| fungi | and did include changes to dependency solving | 18:09 |
| clarkb | https://github.com/pypa/pip/blob/24.1/src/pip/_vendor/pkg_resources/__init__.py#L2863 ya this is the line that is exploding | 18:09 |
| clarkb | I wonder if latest pip is unable to parse that metadata for some reason | 18:09 |
| fungi | i'm struggling to view the build output in a browser because it's so massive, which file did you find the traceback in? | 18:10 |
| clarkb | https://zuul.opendev.org/t/openstack/build/b168543cf18f49b592967ee6b81a3aa6/log/job-output.txt#63829-63840 its there | 18:10 |
| clarkb | fungi: you may need to fetch the file and view it lcoally in a text editor | 18:10 |
| clarkb | my browser is struggling too but just managing | 18:10 |
| clarkb | I'll make a paste actually that seems friendliest | 18:11 |
| fungi | thanks | 18:11 |
| clarkb | fungi: dpanech https://paste.opendev.org/show/bYdq2ryZhiRAJemQDnFp/ | 18:13 |
| fungi | interestingly i don't see "cpython" appearing in any of the metadata fields (other than a passing mention in the markdown description | 18:14 |
| clarkb | ya but also that exception occurs after the initial exception so it may just be in a bad state and trying to look up invalid stuff | 18:15 |
| clarkb | dpanech: fungi: probably the next step is to try and install pyzmq of that version with python3.9 on debian 11 and see if you can reproduce the explosion | 18:15 |
| clarkb | and if so file a bug with pip / debug further depending on your interest level | 18:15 |
| clarkb | ya https://github.com/pypa/pip/blob/24.1/src/pip/_vendor/pkg_resources/__init__.py#L3070 explodes then https://github.com/pypa/pip/blob/24.1/src/pip/_vendor/pkg_resources/__init__.py#L3072 is called which explodes furhter | 18:17 |
| clarkb | reading that code I suppose the idea is that its ok for the first explosion to occur but not the second as it seems dep map is populated by _compute_dependencies so maybe the cpython version string is the problem | 18:18 |
| clarkb | fungi: dpanech: I think it is tripping over the Requires-Dist lines in the metadata file | 18:19 |
| fungi | https://github.com/pdm-project/pdm/issues/1675 | 18:20 |
| fungi | looks similarish | 18:20 |
| clarkb | Requires-Dist: py ; implementation_name === "pypy" and or Requires-Dist: cffi ; implementation_name === "pypy" specifically | 18:20 |
| clarkb | yup that issue seems to have reached the same conclusion | 18:20 |
| clarkb | the problem is the === ? | 18:21 |
| clarkb | oh wait no its due to not listing require-dist for cpython? | 18:21 |
| fungi | looks like pip 24.0 used packaging 21.3 while pip 24.1 upgrades its vendored copy of packaging to 24.1 | 18:21 |
| clarkb | oh wait no they say you can cahnge it to == and its fine | 18:21 |
| clarkb | so ya not an infrastructure problem. Its a pip problem | 18:21 |
| dpanech | clarkb: fungi: ok thank you, I'll pass this on | 18:23 |
| clarkb | also based on that I'm not sure that changing the wheel mirror for debian path will help as presumably those wheels will have the similarly "corrupt" metadata | 18:23 |
| clarkb | dpanech: so ya I think your options are to file a bug with pip and see if they can fix it, or downgrade pip, or see if pyzmq every changed their metadata in newer versions of their packages and possibly upgrade to a newer version of the package | 18:23 |
| clarkb | then separately we can see about fixing access to the wheels built for debian 11 specifically ,but I don't think that will help in this instance | 18:27 |
| fungi | looks like it's being pulled in as a dependency of a git install of https://github.com/0rpc/zerorpc-python but that doesn't seem to be what's setting the upper bound on it | 18:28 |
| dpanech | igor-soares: I'll send the chat transcript in a bit | 18:29 |
| igor-soares | Alright. Thank you. | 18:30 |
| fungi | dpanech: igor-soares: we publish the channel log at https://meetings.opendev.org/irclogs/%23opendev/latest.log.html#t2024-06-25T17:38:49 | 18:30 |
| fungi | so pyzmq 20.0.0 was released almost 4 years ago, but seems to maybe be the last version which published manylinux1 wheels for cp39 | 18:32 |
| clarkb | oh that would explain why that version is being selected | 18:32 |
| clarkb | still doesn't garuntee newer versiosn don't have the same issue though | 18:32 |
| clarkb | https://github.com/zeromq/pyzmq/blob/main/pyproject.toml#L42 I suspect that the latest stuff would work actually | 18:33 |
| clarkb | https://github.com/zeromq/pyzmq/blob/v21.0.0/setup.py#L1446-L1447 and even version 21 I guess | 18:35 |
| clarkb | so fixing the debian wheel mirror path may actualyl fix the CI jobs but then anyone trying to install in the real world would be broken | 18:35 |
| fungi | looks like that was the point where they switched to making manylinux2010 wheels for cp39 | 18:36 |
| clarkb | and bullseye isn't manylinux2010 compatibile? | 18:37 |
| fungi | that's what's baffling, i think it should be? | 18:37 |
| clarkb | I would expect it to be. Bullseye is what 4 yaers old now? thats ~2020 not 2010 | 18:38 |
| clarkb | tonyb: following up on qusetions about brdige. Bridge in prod appears to be jammy | 18:41 |
| clarkb | tonyb: I think we can/should just correct the node type in our jobs | 18:41 |
| fungi | https://github.com/0rpc/zerorpc-python/blob/99ee6e47c8baf909b97eec94f184a19405f392a2/setup.py#L40 is where the pyzmq>=13.1.0 dependency is taken from in the failing starlingx build | 18:44 |
| clarkb | manylinux2010 requires pip 19 or newer, but the job seems to say it is using pip 24.1 which aligns with what we seein the tracebacks | 18:47 |
| dpanech | fungi: clarkb: thanks for your help, we are looking into it on our side | 18:50 |
| fungi | dpanech: igor-soares: in summary, hopefully you can reproduce by trying to install pyzmq with pip 24.1 on debian 11 | 18:51 |
| clarkb | dpanech: igor-soares as a side note our rackspace hosted mirrors have internal and external network interfaces. We set up the jobs to use the internal interfaces with the mirror-int.* names for reliability and throughput purposes. If you want to test things you can drop the -int portion of the string and hit the publicly accessible interface and all the services are teh same | 18:51 |
| clarkb | but I suspect this is reproduceable talkign to pypi directly and isn't directly related to our mirrors | 18:52 |
| igor-soares_ | fungi: clarkb: thanks for your input. This is good info to keep digging on our end. | 18:56 |
| fungi | i would try to do it myself but don't have a debian-11 vm or chroot handy and would need to boot/make one | 18:58 |
| clarkb | docker run debian:bullseye would probably work | 18:59 |
| fungi | i also don't have docker installed on my workstation (tried once but it made an utter mess) | 19:00 |
| fungi | i think it's still picking remaining traces of docker out of its teeth | 19:01 |
| clarkb | fungi: I have approved the mailman + exim change | 19:57 |
| fungi | thanks, i'll try to check on it after deployment | 19:57 |
| opendevreview | Monty Taylor proposed zuul/zuul-jobs master: Support .python-version files in ensure-python https://review.opendev.org/c/zuul/zuul-jobs/+/922515 | 20:10 |
| opendevreview | Monty Taylor proposed zuul/zuul-jobs master: Support .python-version files in ensure-python https://review.opendev.org/c/zuul/zuul-jobs/+/922515 | 20:28 |
| opendevreview | Merged opendev/system-config master: Rebalance Mailman's and Exim's outgoing batch size https://review.opendev.org/c/opendev/system-config/+/922703 | 20:47 |
| opendevreview | Monty Taylor proposed zuul/zuul-jobs master: Support .python-version files in ensure-python https://review.opendev.org/c/zuul/zuul-jobs/+/922515 | 20:56 |
| clarkb | I think the mailman update deployed about 20 minutes ago | 21:22 |
| opendevreview | Monty Taylor proposed zuul/zuul-jobs master: Support .python-version files in ensure-python https://review.opendev.org/c/zuul/zuul-jobs/+/922515 | 21:30 |
| clarkb | now need someone to send mail to openstack-discuss and see if the time deltas shrink | 21:33 |
| fungi | uwsgi processes last started 21:07 so that sounds about right | 21:40 |
| clarkb | this is interesting: if I boot nomodeset with tumbleweed's kernel I only get 1920x1080 (naive is 2560x1440), but if I boot ubuntu noble's kernel nomodeset I get only native resolution | 22:53 |
| clarkb | both are running X and not wayland which I thought maybe could be related. I suspect ubuntu's kernel is simply better about heuristicing the display when operating with the mesa driver instead of amdgpu | 22:53 |
| clarkb | er I guess it is likely vesa not mesa | 22:54 |
| clarkb | anyway I'm still able to reproduce the problems when booting without nomodeset which loads the amdgpu driver under noble as well as jammy and tumbleweed and was not able to do so with similar hardware (my brother has almost the same laptop) so ya I think I have to conclude its hardware and get on the phone tomorrow | 22:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!