| *** elodilles_pto is now known as elodilles | 06:41 | |
| dpawlik | clarkb, fungi hey, https://grafana.opendev.org/d/9871b26303/afs?orgId=1 - could you check why the centos mirror is outdated please? | 11:45 |
|---|---|---|
| frickler | rsync: rename "/afs/.openstack.org/mirror/centos/8-stream/AppStream/x86_64/os/Packages/.yajl-2.1.0-12.el8.x86_64.rpm.vmbB5H" -> "AppStream/x86_64/os/Packages/yajl-2.1.0-12.el8.x86_64.rpm": File too large (27) | 12:18 |
| frickler | seems that is afs's way of telling us the volume quota is reached, although grafana claims that there are still 5 GB free | 12:19 |
| frickler | also I'm confused, why is /afs/.openstack.org/mirror/centos/8-stream not within the centos-stream volume? | 12:21 |
| frickler | seems the latter only holds 9-stream for confusion and fun | 12:21 |
| fungi | dpawlik: the "centos" mirror for centos 7 and centos stream 8 is 5 days stale, the "centos-stream" mirror for centos stream 9 updated 6 hours ago | 12:25 |
| fungi | which one are you asking about? | 12:26 |
| frickler | fungi: did you read what I wrote earlier about this? my suggestion would be to increase the mirror.centos quota from 500G to 550G | 13:54 |
| fungi | frickler: yes, i just wanted to get clarification on which mirror dpawlik was asking about | 13:58 |
| fungi | since we mirror 7/stream 8 separately from stream 9 | 13:58 |
| fungi | also it wasn't clear that the "file too large" error meant insufficient quota | 13:59 |
| fungi | seems unlikely yajl-2.1.0-12.el8.x86_64.rpm is that big, but maybe rsync's tempfiles are also taking up some space temporarily during the rsync | 14:00 |
| frickler | that error is only the last one in a long list, nothing special about that file | 14:00 |
| frickler | guess I should be more verbose in reporting my debugging findings | 14:01 |
| fungi | and yeah, i have no concerns with raising the quota on the centos mirror, especially since we'll be regaining 400gb of space when we drop the fedora mirror | 14:06 |
| fungi | looks like we ought to raise the quota on centos-stream as well (or see if we're mirroring additional things in those we don't actually need) | 14:07 |
| *** Guest535 is now known as starkis | 14:49 | |
| *** JasonF is now known as JaqyF | 14:54 | |
| *** JaqyF is now known as JayF | 14:54 | |
| dpawlik | fungi: sorry for delay, so Centos 8 stream has some package | 15:01 |
| dpawlik | that is outdated and needs to be updated for TripleO | 15:02 |
| dpawlik | cc fserucas__ | 15:02 |
| fungi | dpawlik: thanks, so that's the one that hasn't updated for a few days. i agree with frickler we should increase the quota on the volume (and also on the other volume that has stream 9) | 15:03 |
| apevec | I checked both sources we use for CS8 ( mirror.facebook.net/centos/8-stream/ ) and CS9 ( mirror.rackspace.com/centos-stream/9-stream/ ) are now up to date with latest published CentOS Stream content | 15:04 |
| fungi | it does seem like it grew too large for the amount of space we allowed for it, so the rsync isn't completing any longer and afs is (safely) holding back updates to the read-only volume as a result | 15:04 |
| fungi | rather than publish an incomplete/corrupt mirror | 15:05 |
| apevec | yep, atomic updates are a good thing! | 15:05 |
| fungi | once more of our sysadmins are awake and we can get consensus on increasing those quotas we can force an update so we don't have to wait for the cronjob to try again | 15:06 |
| fungi | i've pinged our sysadmin highlight in #opendev to see if there are any concerns before we go ahead with the quota increase | 15:08 |
| dpawlik | thanks fungi++ and frickler++ | 15:17 |
| clarkb | I think part of the issue is centos isn't pruning old package versions so the space required grows consistently. Let me see if I can find an example of that. I think thunderbird was a particularly bad case | 15:19 |
| fungi | also things like dotnet sources in powertools are pretty hige | 15:19 |
| fungi | huge | 15:19 |
| mmalchuk | merged | 15:22 |
| mmalchuk | sorry... wrong channel | 15:23 |
| dpawlik | fungi: what is the biggest one :D ? | 15:25 |
| fungi | dpawlik: dotnet-sdk-6.0-source-built-artifacts-6.0.0-0.6.28be3e9a006d90d8c6e87d4353b77882829df718.el8.x86_64.rpm is about 3.7gb | 15:26 |
| fungi | dpawlik: but all the dotnet* packages together are roughly 10gb | 15:28 |
| dpawlik | uuu | 15:31 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Convert all zuul tenants except openstack to Ansible 8 https://review.opendev.org/c/openstack/project-config/+/892976 | 15:31 |
| dpawlik | I hope users are using it, not it just created and that's it | 15:31 |
| clarkb | looking at http://mirror.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ it appears we end up with ~4 versions of many packages | 15:32 |
| clarkb | some have more some have less. But in theory I think that means centos-9 is ~4x larger than it needs to be. You probably need some headroom though to swap out packages to be fair. But 4x is a lot | 15:32 |
| clarkb | oh though I've just noticed we're mirroring i686 | 15:33 |
| clarkb | we should stop doing that entirely | 15:33 |
| clarkb | side note: why are i686 packages in the x86_64 directory? Is that not a bug? | 15:34 |
| dpawlik | good question | 15:35 |
| fungi | are they duplicates of x86_64 packages, or just usable by both architectures? | 15:35 |
| clarkb | fungi: there are also no arch packages | 15:36 |
| clarkb | I think those are for both | 15:36 |
| dpawlik | both | 15:36 |
| clarkb | and yes they appear to all be duplicates | 15:36 |
| fungi | we should be able to pattern match and filter the i686 packages in that case | 15:37 |
| clarkb | yup I'm working ona a change | 15:37 |
| clarkb | just waiting for the centos 7 package index to load so I can check if we need the same exclude there | 15:40 |
| clarkb | remote: https://review.opendev.org/c/opendev/system-config/+/892978 Exclude i686 rpms in our centos mirrors [NEW] | 15:47 |
| opendevreview | Merged openstack/project-config master: Convert all zuul tenants except openstack to Ansible 8 https://review.opendev.org/c/openstack/project-config/+/892976 | 17:35 |
| fungi | dpawlik: the mirror is up to date now | 17:37 |
| fungi | see if things are working any better for you | 17:37 |
| clarkb | without i686 packages | 17:37 |
| fungi | right | 17:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!