| tkajinam | Failed to download packages: kernel-devel-5.14.0-432.el9.x86_64: Cannot download, all mirrors were already tried without success | 09:20 |
|---|---|---|
| tkajinam | hmm | 09:20 |
| tkajinam | I guess it takes some time until the new kernel version is pulled to base cs9 image ? | 09:20 |
| jcapitao[m] | RDO is also facing issues when pulling packages from opendev AFS | 09:21 |
| frickler | the centos upstream we have been syncing from seems to be pretty unstable, had a similar issue yesterday already | 09:25 |
| apevec | hmm it was supposed to be primary source, if we have logs let's report to https://pagure.io/centos-infra/issues | 09:29 |
| jcapitao[m] | I see it syncs every 6h | 09:29 |
| tkajinam | I can't open the package list of c9s appstream in mirror but I'm suspecting that a few packages are not synced properly there | 09:32 |
| jcapitao[m] | next iteration will be at 12 noon UTC | 09:33 |
| apevec | log is https://static.opendev.org/mirror/logs/rsync-mirrors/centos-stream.log right? | 09:34 |
| jcapitao[m] | I'm using https://mirror.dfw.rax.opendev.org/logs/ which is another mirror | 09:36 |
| jcapitao[m] | but it's the same right | 09:36 |
| jcapitao[m] | there is also https://files.openstack.org/mirror/logs/ | 09:36 |
| jcapitao[m] | it's all the same content | 09:37 |
| tkajinam | ok so -432 of kernel-devel does not exist in https://mirror.dfw.rax.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/ | 09:42 |
| frickler | yes, the logs are also stored in AFS, so should be consistent everywhere | 09:42 |
| tkajinam | while we have -432 of kernel exists in https://mirror.dfw.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ | 09:43 |
| tkajinam | hmm. the log indicates that package was in the incremental diff list detected by rsync... | 09:46 |
| jcapitao[m] | so few packages got lost in last sync. I have same issue with podman-5.0.0-1.el9.x86_64.rpm | 09:47 |
| apevec | yeah, seems to be incomplete sync, repodata in AFS references RPM packages which are not in AFS e.g. crun-1.14.4 | 09:47 |
| apevec | oh you mean they were there then deleted ? | 09:47 |
| tkajinam | deleting AppStream/x86_64/os/Packages/kernel-devel-5.14.0-432.el9.x86_64.rpm | 09:49 |
| tkajinam | I'm still confused by the log (mainly because it's huge) but it might be possible that the package was missing from the upstream mirror so was removed | 09:50 |
| tkajinam | AppStream/x86_64/os/Packages/.~tmp~/kernel-devel-5.14.0-432.el9.x86_64.rpm | 09:51 |
| tkajinam | though this one comes later. | 09:51 |
| jcapitao[m] | same with deleting AppStream/x86_64/os/Packages/podman-5.0.0-1.el9.x86_64.rpm | 09:53 |
| frickler | the ".~tmp~" indicates that the sync happened while upstream was itself syncing from some other source? | 09:54 |
| jcapitao[m] | I think you're right tkajinam it seems those files were not available on sender side during the rsync operation | 09:54 |
| tkajinam | ok... if these packages haven't been yanked then we can probably wait for the next sync | 10:00 |
| apevec | amoralej_: jcapitao: might be still worth to open centos-infra ticket to check what happened on centos primary rsync ? | 10:55 |
| jcapitao[m] | yes FTR I'll open one after next iteration and will put some logs | 11:04 |
| jcapitao[m] | apevec tkajinam : FYI I opened an issue on centos-infra side https://pagure.io/centos-infra/issue/1384 | 14:08 |
| apevec | thanks Joel! | 14:12 |
| Adam_king | hello | 14:26 |
| Adam_king | i'm new here. I don't really understand how this site works | 14:26 |
| Adam_king | I'm trying to learn how to contribute | 14:27 |
| corvus1 | i have approved https://review.opendev.org/914412 which should cause us to delete some unused nodepool image files. i'll check on it later to make sure it works as expected. | 15:01 |
| clarkb | corvus1: thanks! | 15:07 |
| clarkb | looks like Adam_king only stuck around for a handful fo minutes :/ | 15:07 |
| clarkb | jcapitao[m]: tkajinam apevec as discussed the other day the primary options available to us are either to have upstream update the mirror in a safe order (generally you add new packages then update indexes then remove packages so that regardless of when you sync you're get a working state), we update our sync script to either use some expected tooling we are using or update it to | 15:09 |
| clarkb | verify state before publishing within afs (this is how reprepro works with debuntu, it does a verification step and if that fails we don't publish), or switch to a different upstream that might be more stable | 15:09 |
| opendevreview | Merged openstack/project-config master: Enable nodepool delete after upload option https://review.opendev.org/c/openstack/project-config/+/914412 | 15:10 |
| corvus1 | the good news is that i don't think the builders are deleting too many files; the bad news is i don't think they're deleting any. | 16:07 |
| corvus1 | clarkb: oh i think the diskimage parent system is a little more manual than we would like. i'm pretty sure that value didn't get propagated to children | 16:10 |
| corvus1 | rather than update the config to be explicit, i'll fix that in the builder | 16:11 |
| clarkb | corvus1: oh interesting so the config wasn't applied to any of the images because it is only on an abstract definition | 16:23 |
| corvus1 | yep; found/fixed the issue in https://review.opendev.org/914658 | 16:25 |
| corvus1 | the format list was likely inherited, but not the enable flag | 16:26 |
| fungi | just checking in for a few minutes while i have working internet access, i'm still going to be mostly afk through the weekend | 16:28 |
| clarkb | fungi: not a whole lot going on. We could do the gitea 1.21.10 upgrade if people are brave but I don't think it is urgent. Probably not a bad thing to be somewhat slushy right now anyway with the openstack release coming up | 16:30 |
| clarkb | I've got paperwork things I should be doing too. The universe is conspiring to make that happen | 16:30 |
| clarkb | fungi: I think the only thing that came up with centos stream synced a bad mirror state again | 16:30 |
| clarkb | twice in ~2-3 days | 16:30 |
| fungi | clarkb: TheJulia: JayF: if it's not clear, devstack doesn't take its target branch directly from a zuul var because certain jobs may want to run devstack with a different branch than what triggered it (grenade for example) | 16:32 |
| clarkb | fungi: yup (though we just discovered grenade doesn't do that properly for reasons) | 16:33 |
| fungi | oh wow, even better | 16:33 |
| clarkb | fungi: https://opendev.org/openstack/grenade/src/branch/stable/zed/.zuul.yaml#L381-L407 this zed job config applies to all the stable/202* branches and then sets wallaby as the old branch | 16:34 |
| * TheJulia tries to come up with a nice and appropriately sarcastic reply | 16:34 | |
| fungi | devstack backward-compat jobs for some non-branching libs/clients may also run devstack for stable branches, can't recall if we still have any of those | 16:35 |
| fungi | though that may be done with branch overrides these days too | 16:36 |
| TheJulia | fungi: I think I got what I was looking at yesterday sorted, but it is good to know it is never simple :) | 16:41 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Temporarily remove release docs semaphores https://review.opendev.org/c/openstack/project-config/+/914688 | 17:48 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Revert "Temporarily remove release docs semaphores" https://review.opendev.org/c/openstack/project-config/+/914689 | 17:48 |
| corvus1 | fungi: out of curiosity, what is the critical section for a release job semaphore? is it just the copy, or is it the whole build? | 17:53 |
| corvus1 | (i mean, in theory, not in implementation) | 17:53 |
| corvus1 | (obvs the current implementation is the whole job) | 17:53 |
| corvus1 | clarkb: fix merged; and builder just restarted | 18:07 |
| corvus1 | https://paste.opendev.org/show/bGQp4jJHenbNlpO3MTUK/ | 18:08 |
| corvus1 | that looks correct to me | 18:08 |
| corvus1 | still see qcow2 files for everything | 18:08 |
| corvus1 | and i don't see any associated error messages; so i think everything looks good | 18:09 |
| frickler | corvus1: iiuc it should be only the copy step | 18:18 |
| corvus1 | then it may be worth looking into a playbook semaphore (and ensuring the copy is in its own playbook to minimize the critical section time). might be fast enough you could leave the semaphores in place during release. | 18:20 |
| clarkb | ya it is the copy step and it doesn't actually result in true issues but job errors can occur that are benign | 18:29 |
| clarkb | basically resync gets mad that tmp files move around under it and reports failure but everything copies properly | 18:29 |
| opendevreview | Alan Pevec proposed opendev/system-config master: Add AFS mirror update logs location https://review.opendev.org/c/opendev/system-config/+/911869 | 20:59 |
| fungi | corvus1: to clarify, if changes merge for or releases are tagged for different branches of the same project, release notes publication rsyncs occurring in parallel for them can either step on one another (one trying to remove the other's tempfiles) or complete out of order (leading to missing content until the next run for any branch of that project) | 23:39 |
| fungi | for the releases site, it's multiple release requests merging at the same time causing similar issues for publication of the releases.openstack.org site | 23:40 |
| fungi | it would be good to look into alternative solutions to avoid those collisions without resorting to serializing across every project (ephemeral per-project semaphores would do the trick in the release notes case, while a supercedent pipeline manager might be good for release docs) | 23:43 |
| fungi | but nobody's found the time yet | 23:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!