Wednesday, 2023-06-28

tonybCan anyone point me at issues we've seen with CentOS stream (like ping, and the removal of the algo that was used for signing rpms?01:29
tonybnm I found a enough context for my needs01:34
Clark[m] and fwiw 02:02
Clark[m] is a problem tripleo attributed to CentOS kernel updates 02:05
Clark[m] is one where network manager was overwriting resolv.conf. I seem to recall we made a change to more explicitly tell NM to not manage that file but it seems that it also affected buildah02:12
Clark[m]I seem to recall there were problems specifically with openstack stuff when setting up fips jobs but I wasn't following super closely 02:17
opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
opendevreviewMerged openstack/project-config master: Normalize projects.yaml
opendevreviewCaio Cesar Ferreira proposed openstack/project-config master: Add Power Metrics app to StarlingX
opendevreviewMerged openstack/project-config master: Add Power Metrics app to StarlingX
SvenKieskehey! so I understand we mirror different stuff already, correct? would it be possible to also mirror some (all?) used packages from I don't know how much storage that would need and how much is available.16:07
SvenKieskereason being we are seeing many CI failures due to pypi mirror problems16:07
SvenKieskein kolla-ansible, that is16:07
Clark[m]No. We did mirror way back when but it grew too large (on the rate of a gigabyte a day)16:08
SvenKieskee.g. here:
Clark[m]As far as I know that growth rate has not changed and may have even gotten worse as AI/ML tools have gotten more popular16:09
SvenKieskeah. good to know! what about mirroring only some packages? or does openstack tend to pull in all of it? :D16:09
Clark[m]The proxy cache system we have in place works reasonably well even though it isn't oerfect16:10
Clark[m]No before we mirrored all of we mirrored only the subset from openstack requirements and that creates a bunch of problems16:10
Clark[m]Basically it made this weird dance that had to be done to add any dep (or update a dep whose deps have changed)16:10
SvenKieskeI figured it's good to ask, and your answer shows it's really good to ask first :D16:10
SvenKieskeyeah, sure, you can't really specify in a sane way which mirror to use for which dependency, I think there's still an open issue for that for pip16:11
Clark[m]Also it always helps to link to job logs when discussing job failures16:11
Clark[m]I'm guessing in this case the issue was pypi failed over to it's stale backend and the index was incomplete16:11
SvenKieskemhm, okay, I don't know enough about pypi internals to comment on that16:13
SvenKieskethanks anyway!16:14
Clark[m]Can you link to the failure?16:14
SvenKieskeI'd have to dig that up, I just trusted frickler's comment :) let me look..16:15
Clark[m]I trust it to but it doesn't characterize the pypi failure just that it had to do with pypi16:15
SvenKieskeyeah, well, our error output is not helpful at all, I fear. I guess it is this line:
SvenKieskebut that's already a for loop installing requirements and the error code is just "non zero exit code" because of bash wrappers -.-16:19
Clark[m]That looks like the stale backend for pypi serving old index for pbr that doesn't include the constrained version16:19
SvenKieskethe for loop was added because of unrealiable mirrors, but it seems, 5 tries with 120 seconds timeout each are not enough :(16:19
Clark[m]I have encouraged openstack before to take this up with pypi since the use of constraints means openstack hits it often whereas other pypi users just install old software from the old index16:20
SvenKieskecan I read up on this stale index stuff somewhere? sounds like a bug to me?16:20
Clark[m]It is a security concern for those other users. For openstack it causes failures due to constraints16:20
SvenKieskebut nobody bothered to open an upstream bug, right? :)16:21
Clark[m]There is an old cheese shop/warehouse/pypa bug on GitHub somewhere (pretty sure it's closed because they "fixed" it)16:21
SvenKieske this one?16:21
Clark[m]The tldr is pypi is fronted by a cdn. The cdn requests data from the primary backend when it needs data. If that fails it requests data from a secondary backend. The problem is the secondary backend is often months out of date so you get stale info when this situation occurs16:22
Clark[m]Yes that appears to be the one16:22
SvenKieskeI'll try to reopen it :P :D I can be stubborn if it helps..16:23
Clark[m]That pvr version is about 5 months old implying the secondary backend is at least that stale16:25
Clark[m]However it may be package specific. I don't know enough implementation details for that system 16:26
SvenKieskeI managed to get a response, but I fear I will run in circles with this.16:33
SvenKieskesee the github issue for details16:33
SvenKieskeClark[m]: thanks for chiming in on the github issue, maybe we can work out a fix now with pypi :)16:54
SvenKieskeI also pinged them again over on libera chat #pypa-dev; I think they won't receive notifications over closed issues otherwise.16:57
Clark[m]I got emailed for the updates to the issue at least. But who knows if they watch them (I get so much email I can't blame anyone for ignoring it)16:58
opendevreviewMerged opendev/ master: Replace ze01-ze03
opendevreviewMerged opendev/system-config master: Replace ze01-ze03
corvusi deleted the fact cache for those 3 hosts18:03
opendevreviewMerged zuul/zuul-jobs master: Use zuul_workspace_root for prepare/mirror workspace test roles
opendevreviewElod Illes proposed openstack/project-config master: Fix release-id fetching for release job
opendevreviewMerged openstack/project-config master: Fix release-id fetching for release job
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: DNM: Test base-test
corvusi rebooted ze01-ze03 and they now have apparently working afs trees20:13
corvusand the most recent playbooks runs are green20:16
corvusso i think i'm going to bring up one of those20:16
clarkbsounds good20:16
corvus#log started ze01 on new jammy host20:17
corvusit's running jobs...20:18
clarkband the afs failure previously was on executor startup20:18
clarkbso that is a really good sign20:18
corvusyep; and a job i just spot checked was successful20:20
corvus#status log started ze02 and ze03 on new jammy hosts20:25
opendevstatuscorvus: finished logging20:25
fungiawesome, thanks!20:39

Generated by 2.17.3 by Marius Gedminas - find it at!