Friday, 2026-10-02

-@gerrit:opendev.org- Steve Baker proposed:00:27
- [openstack/diskimage-builder] 1007795: Add DIB_BIND_MOUNTS option for container environments https://review.opendev.org/c/openstack/diskimage-builder/+/1007795
- [openstack/diskimage-builder] 983815: Add tarball element for unprivileged container builds https://review.opendev.org/c/openstack/diskimage-builder/+/983815
- [openstack/diskimage-builder] 1007796: Add dnf-assert element https://review.opendev.org/c/openstack/diskimage-builder/+/1007796
-@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 1008396: WIP: Introduce barbican-ui-core group https://review.opendev.org/c/openstack/project-config/+/100839604:29
-@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 1008396: WIP: Introduce barbican-ui-core group https://review.opendev.org/c/openstack/project-config/+/100839604:31
-@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 1008396: Introduce barbican-ui-core group https://review.opendev.org/c/openstack/project-config/+/100839604:38
-@gerrit:opendev.org- Ivan Anfimov marked as active: [openstack/project-config] 1008396: Introduce barbican-ui-core group https://review.opendev.org/c/openstack/project-config/+/100839604:39
-@gerrit:opendev.org- Ivan Anfimov proposed wip: [openstack/project-config] 1008396: Introduce barbican-ui-core group https://review.opendev.org/c/openstack/project-config/+/100839604:48
-@gerrit:opendev.org- Ivan Anfimov marked as active: [openstack/project-config] 1008396: Introduce barbican-ui-core group https://review.opendev.org/c/openstack/project-config/+/100839604:53
-@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 1008415: Revert "Temporarily remove IPv6 for review and ymq mirror" https://review.opendev.org/c/opendev/zone-opendev.org/+/100841506:35
-@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 1008415: Revert "Temporarily remove IPv6 for review and ymq mirror" https://review.opendev.org/c/opendev/zone-opendev.org/+/100841508:26
-@gerrit:opendev.org- Juan Larriba proposed: [openstack/project-config] 1008432: Add repository for devstack-plugin-jaeger https://review.opendev.org/c/openstack/project-config/+/100843210:22
-@gerrit:opendev.org- Juan Larriba proposed: [openstack/project-config] 1008435: Add repository for devstack-plugin-jaeger https://review.opendev.org/c/openstack/project-config/+/100843510:54
-@gerrit:opendev.org- Juan Larriba proposed: [openstack/project-config] 1008435: Add repository for devstack-plugin-jaeger https://review.opendev.org/c/openstack/project-config/+/100843510:58
-@gerrit:opendev.org- Juan Larriba proposed: [openstack/project-config] 1008435: Add repository for devstack-plugin-jaeger https://review.opendev.org/c/openstack/project-config/+/100843511:33
-@gerrit:opendev.org- Juan Larriba proposed: [openstack/project-config] 1008435: Add repository for devstack-plugin-jaeger https://review.opendev.org/c/openstack/project-config/+/100843511:34
@priteau:matrix.orgIs anyone else having issues loading https://zuul.opendev.org/t/openstack/status in full?13:35
@fungicide:matrix.orgPierre Riteau: my browser did briefly complain about being unable to reach the zuul.opendev.org site, i wonder if somethings going on, will check14:00
@priteau:matrix.orgSame thing on Safari, Chrome, Firefox. They all go to 100% CPU usage. I know there are lots of jobs queued right now but I don't usually see this issue14:06
@harbott.osism.tech:regio.chat157 patches in check, thats ... a lot14:07
@fungicide:matrix.orgload average on the zuul-web servers is reasonable/low14:08
@harbott.osism.tech:regio.chatlooks like starlingx pushed a stack of 127 changes14:08
@harbott.osism.tech:regio.chatyeah, it seems it is the zuul UI code in the browser that has issues handling this14:09
@fungicide:matrix.orggot it, so just a large status blob and maybe apache is busy, looking at that14:11
@scott.little:matrix.orgSorry guys, that was me.  I've been carrying a major feature as a series of code reviews.  Just rebased the whole thing.  Didn't think about the impact that would have on zuul.  Going forward I'll squash it all.  probably start a feature branch.   Not sure what can be done about the backlog I just created. 14:18
@fungicide:matrix.orgscott.little: it will burn down on its own as the jobs complete14:26
@clarkb:matrix.orgfungi: https://review.opendev.org/c/opendev/system-config/+/1007977 should be a quick review if you have a moment and make the new dashboard a bit more helpful. I'm going to work on cleaning up the dashboard momentarily too (remove the systemd panel as we're not exporting that info)15:26
@clarkb:matrix.orgThen another thing I was thinking about was cleaning up the wheel mirrors for xenial etc in afs. We can't delete the top level directory for each distro without a bit more work because that is the volume mount target right?15:27
@clarkb:matrix.orgso we need to delete each of the subdirs and then remove the mount? and then we can remove the top level dir?15:27
@fungicide:matrix.orgyes we would empty the directory, then delete the volumes in afs (or maybe we can just delete the volumes and it will reclaim the space their contents occupied?)15:28
@clarkb:matrix.orgoh that is a good question15:30
-@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 1008493: Drop node exporter dashboard panels for systemd and ntp https://review.opendev.org/c/openstack/project-config/+/100849315:36
@clarkb:matrix.orgfungi: https://docs.openafs.org/AdminGuide/HDRWQ235.html these are the steps to remove a volume. It says `it automatically removes a volume's VLDB entry and both the volume header and all data from the partition.`15:38
@clarkb:matrix.orgso I think that may actually be the easiest option15:38
@clarkb:matrix.orgThe main consideration is that we're not trying to vos release the volumes before we start doing that I'm guessing? Though I think the vos release should start to fail rather than breaking anything else15:43
@clarkb:matrix.organd nothing should be doing that for the old wheel mirrors as we should have dropped the jobs associated with publishing them15:43
@fungicide:matrix.orgcorrect, though we are still updating and vos releasing the wheel volumes for more recent distros prior to our cutoff, just not the ones that we don't have the platforms to run the update jobs for any longer15:48
@clarkb:matrix.orgyup so we can do a first pass cleanup against those older volumes first. Then need to do more work to clean up the others15:49
@clarkb:matrix.org(if we decide to clean them up too)15:49
@clarkb:matrix.orgI think we would run something like this on afs01.dfw:15:54
```
vos remove -server afs01.dfw.openstack.org -id mirror.wheel.xenialx64.readonly -localauth
vos remove -server afs02.dfw.openstack.org -id mirror.wheel.xenialx64.readonly -localauth
vos remove -server afs01.dfw.openstack.org -id mirror.wheel.xenialx64 -localauth
fs rmmount /afs/openstack.org/mirror/wheel/ubuntu-16.04-x86_64 -localauth
```
@clarkb:matrix.orgI don't actually think fs rmmount will take -localauth though so that may require proper kerberos auth. Also do we rmmount on the .openstack.org path instead of the openstack.org path?15:55
@fungicide:matrix.orgyeah, fs doesn't support that option like vos does15:58
@fungicide:matrix.orgi think we may need to rmmount both yes, or at least the rw path rather than the ro one15:58
@fungicide:matrix.orgcorvus probably remembers better than i do15:59
@clarkb:matrix.orgack. Do we want to proceed with that now?15:59
@fungicide:matrix.orggo for it15:59
@clarkb:matrix.orgI'll get started just as soon as I've logged in so the fs command can be run15:59
@fungicide:matrix.orggenerally it should report an error if it can't do something, and we can check behind it with `listvldb` et cetera16:01
@fungicide:matrix.orgwe may also need to `vos release ...` the parent volume after the rmmount16:01
@jim:acmegating.comhuh i thought there was a localauth for everything16:02
@fungicide:matrix.orgor at least you need to do that after adding mounts inside other volumes, so i assume removal is the same in that regard16:02
@jim:acmegating.comi also don't know if the release is necessary, or if mount points are like immediate metadata changes16:02
@jim:acmegating.comoh, then yeah, probably so :)16:03
@fungicide:matrix.orgcorvus: i've never been able to figure out how to get fs subcommands that require privs to do localauth16:03
@fungicide:matrix.orgmaybe there's some trick or the option is spelled differently16:03
@clarkb:matrix.orgok I'm logged in. Proceeding with the vos remove commands now16:05
@clarkb:matrix.org`vos: server 'afs01.dfw.openstack.org' not found in host table` that string `afs01.dfw.openstack.org` is what the listvldb output shows as the server16:06
@clarkb:matrix.orghit this on the first command trying to remove the readonly volume not the third one to remove the readwrite volume16:06
@clarkb:matrix.org`vos listaddrs` will apparently provide more info and then I can use the ip address16:07
-@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 1007977: Add host pid namespace to node exporter docker config https://review.opendev.org/c/opendev/system-config/+/100797716:07
@fungicide:matrix.orgClark: oh! right, i always forget you have to use the ip address instead of the hostname for some of that because of the local vs global mapping for the hostnames16:08
@clarkb:matrix.orghrm that just spits the names out including the one I am trying to use16:08
@clarkb:matrix.orgfungi: just use afs01.dfw's public ip addr instead?16:08
@fungicide:matrix.orgyes16:08
@clarkb:matrix.orgack trying that now16:08
@fungicide:matrix.orgper the warning at https://docs.opendev.org/opendev/system-config/latest/afs.html#creating-a-volume16:09
@fungicide:matrix.organd like how we document in https://docs.opendev.org/opendev/system-config/latest/afs.html#recovering-a-failed-fileserver16:10
@clarkb:matrix.orgnow mirror.wheel.xenialx64.readonly is not a volume that is found so I'm misinterpreting how to address the RO volume from that document16:10
@clarkb:matrix.orgI can just use the ID which is the number reported in listvldb output I think16:10
@fungicide:matrix.orghttps://docs.opendev.org/opendev/system-config/latest/afs.html#deleting-a-volume says we only need to rmmount the rw path16:11
@clarkb:matrix.org`VLDB: Volume 'mirror.wheel.xenialx64.readonly' no match` <- this is the error and `mirror.wheel.xenialx64` is the name per listvldb and the documentation says you add the .readonly extension so I'm not sure what I'm doing wrong there16:12
@clarkb:matrix.orgfungi: if you read the doc above it says that if you rm the RW it will keep the ROs around16:12
@clarkb:matrix.organd they have to be claened up separately16:12
@fungicide:matrix.orgfor vos remove yes, not fs rmmount16:13
@clarkb:matrix.orgoh got it16:13
@clarkb:matrix.orgalso the upstream doc says `Do not remove the mount point if copies of the read-only volume remain.`16:13
@clarkb:matrix.orgso we may need to update our doc to do the rmmount last16:13
@fungicide:matrix.orgalso our doc indicates including the `-partition $PARTITION` option when removing the ro version16:14
@clarkb:matrix.orgyup I'll try that now16:14
@clarkb:matrix.orgvalue looks like `/vicepa` ?16:14
@clarkb:matrix.orgthat has not returned immediatelywith an error so I think it is doing stuff in the background while I wait for the command to exit16:16
@clarkb:matrix.org`Volume 536870974 on partition /vicepa server afs01.dfw.openstack.org deleted` yup that looks better16:16
@fungicide:matrix.org`sudo vos examine -id mirror.wheel.xenialx64` reports "partition /vicepa16:16
@fungicide:matrix.orgso i think so16:16
@clarkb:matrix.organd now listvldb shows two entries. The second RO and the RW16:17
@clarkb:matrix.orgI'll keep going to claen those up and figure out the rmmount then post an updated list of commands here and then update our docs16:17
@clarkb:matrix.orgthe issue with the ip addrss is that I'm running the commands on the same server. Looks like specifying afs02 works fine16:19
@fungicide:matrix.orgright16:19
@fungicide:matrix.orgbecause of how /etc/hosts is setup16:19
@clarkb:matrix.orgok `mirror.wheel.xenialx64` is no longer in the listvldb output16:20
@clarkb:matrix.orgnow to figure out the rmmount16:20
@clarkb:matrix.org`fs lsmount /afs/.openstack.org/mirror/wheel` says `'/afs/.openstack.org/mirror/wheel' is not a mount point.` So I need to vos release the dir above: `'/afs/.openstack.org/mirror' is a mount point for volume '#mirror'` which means I need to `vos release mirror -localauth`?16:23
@clarkb:matrix.orgI should say the rmmount is gone and ls in /afs/.openstack.org/mirror/wheel shows that it is gone16:24
@clarkb:matrix.org* I should say the rmmount is done and ls in /afs/.openstack.org/mirror/wheel shows that it is gone16:24
@clarkb:matrix.org`vos release -id mirror -localauth`16:25
@clarkb:matrix.orgmaybe the -id is optional looks like our docs show it without -id. Upstream docs use -id16:26
@clarkb:matrix.organyway does that seem safe? I think it won't conflict with any of the work mirror update is doing because it is a distinct volume16:26
@clarkb:matrix.orghttps://mirror.dfw.rax.opendev.org/wheel/ doesn't show it anymore either16:30
@clarkb:matrix.orgweird but ls /afs/.openstack.org/mirror/wheel does show it16:30
@clarkb:matrix.orgas a regular dir now but it isn't even a regular dir in the RW side so I think a vos release is necessary16:31
@clarkb:matrix.orgI'll do that shortly if no one objects16:31
@clarkb:matrix.orgthat seems to ahve done it. Ls is consistent between RW and RO now16:35
@fungicide:matrix.orgthat matches my recollection as well16:37
@clarkb:matrix.org```16:38
vos listvldb -localauth | grep -A 10 mirror.wheel.xenialx64
vos remove -server 104.130.138.161 -partition /vicepa -id mirror.wheel.xenialx64.readonly -localauth
vos remove -server afs02.dfw.openstack.org -partition /vicepa -id mirror.wheel.xenialx64.readonly -localauth
vos remove -server 104.130.138.161 -partition /vicepa -id mirror.wheel.xenialx64 -localauth
vos listvldb -localauth | grep -A 10 mirror.wheel.xenialx64
fs lsmount /afs/openstack.org/mirror/wheel/ubuntu-16.04-x86_64 # Confirm mount point matches volume
fs rmmount /afs/.openstack.org/mirror/wheel/ubuntu-16.04-x86_64
```
@clarkb:matrix.org* ```16:39
vos listvldb -localauth | grep -A 10 mirror.wheel.xenialx64
vos remove -server 104.130.138.161 -partition /vicepa -id mirror.wheel.xenialx64.readonly -localauth
vos remove -server afs02.dfw.openstack.org -partition /vicepa -id mirror.wheel.xenialx64.readonly -localauth
vos remove -server 104.130.138.161 -partition /vicepa -id mirror.wheel.xenialx64 -localauth
vos listvldb -localauth | grep -A 10 mirror.wheel.xenialx64
fs lsmount /afs/openstack.org/mirror/wheel/ubuntu-16.04-x86_64 # Confirm mount point matches volume
fs rmmount /afs/.openstack.org/mirror/wheel/ubuntu-16.04-x86_64
fs lsmount /afs/.openstack.org/mirror
vos release -localauth mirror
```
@clarkb:matrix.orgThat is a fairly complete set of steps including some sanity checking along the way16:39
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008499: Document fs rmmount happening after vos remove https://review.opendev.org/c/opendev/system-config/+/100849916:44
@clarkb:matrix.organd that is the doc update showing the order as vos remove then rmmount due to the warning in the upstream doc16:44
@clarkb:matrix.orgI believe that these additional wheel mirrors can be safely removed: centos-7-x86_64, centos-8-aarch64, centos-8-x86_64, ubuntu-18.04-aarch64, ubuntu-18.04-x86_64, debian-10-aarch64, and debian-10-x86_6416:46
@clarkb:matrix.orgif we're happy with the xenial results and that list looks good I can keep working through that list after I find breakfast and catch up on my morning16:47
@clarkb:matrix.orgyou can see a tiny dip in disk usage on the afs graph for total use16:49
@fungicide:matrix.orgthat list sounds safe to me, yes16:55
@clarkb:matrix.orgI cross checekd against zuul-provider node types16:55
@clarkb:matrix.organd we don't have any ci nodes with those operating systems anymore as far as I can tell16:56
@clarkb:matrix.orgok I'm back now. Any concerns with me proceeding to clean up that list with the process summarized above (and used to clean the xenial wheels?)17:50
@fungicide:matrix.orgno concerns18:03
@clarkb:matrix.orgcool proceeding now18:04
@fungicide:matrix.orgi'm gonna pop out for a late lunch/early dinner, back shortly18:22
@clarkb:matrix.orgok that round of wheel mirror cleanups is done18:28
@clarkb:matrix.orgI think this got us from 95.5% disk consumption to 94.2%. At least it is measurable18:32
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008520: Cleanup unused mirror content from mirrors and mirror-update https://review.opendev.org/c/opendev/system-config/+/100852018:52
@clarkb:matrix.orgAfter ^ that lands I think we can do volume cleanups for fedora, centos, debian-openstack, opensuse, and openeuler (openeuler is good to go right now as there didn't seem to be any cleanup necessary for it)18:52
@clarkb:matrix.orgof course its a bigger question of whether or not we want ot drop the top level distro entries. I think it is probably fine at this point since it seems unlikely we would add them back given the space constraints we are now dealing with18:53
@clarkb:matrix.orgalso debian-openstack doesn't really haev a lot of info/config management. I think it was done then removed and we never cleaned it up?18:53
@clarkb:matrix.orgbut after this set I think we're into making hard decisions territory. Aka removing content that is mirrored today that people are likely using to some extent18:55
@clarkb:matrix.orgat least if we're doing it at an afs volume level. Its possible we can trim within a volume if there is extraneous data being mirrored18:56
@clarkb:matrix.orghttps://mirror.dfw.rax.opendev.org/centos-stream/9-stream/BaseOS/aarch64/os/EFI/BOOT/ like this could be cleaned up but its also tiny18:57
@clarkb:matrix.orglooking at centos stream more closely we get multiple versions of each package. It may make sense to say only keep the most recent 2 pacakges?18:58
@clarkb:matrix.orgsomeone that understand centos and yum/dnf better than I would probably want to weigh in on that, but we could theoretically halve? the size of the centos stream mirror by doing that. Lots of big packages with many old versions18:59
@clarkb:matrix.orgI think debuntu already clear out old packges18:59
@clarkb:matrix.orgso can't do that with them18:59
@clarkb:matrix.organyway I think the next step is cleaning up those volumes identified above after making that change or something like it to make doing so safe(r). Then we can start making hard decisions afterwards19:01
@fungicide:matrix.orgokay, back and catching up19:34
@fungicide:matrix.orgregarding the debian-openstack mirror, yes i think that hasn't been used in many years19:35
@fungicide:matrix.orghttp://mirror.dfw.rax.opendev.org/wheel/ looks much shorter already20:13
-@gerrit:opendev.org- Clark Boylan proposed: [openstack/project-config] 1008533: Cleanup afs grafana dashboard wheel graphs https://review.opendev.org/c/openstack/project-config/+/100853320:23
@clarkb:matrix.orgfungi: ^ that catches the afs dashboard up (there is also the node exporter change to drop systemd and ntp related panels/graphs)20:23
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008520: Cleanup unused mirror content from mirrors and mirror-update https://review.opendev.org/c/opendev/system-config/+/100852020:24
@clarkb:matrix.organd that should hopefullyfix the silly test failures20:24
@fungicide:matrix.orgi also had https://review.opendev.org/999021 to drop fedora mirroring, which may overlap or be related20:24
@clarkb:matrix.orgfungi: oh cool. Yours is a bit more complete at removing the content from system-config but misses some of the cleanups I have in my change. DO you want me to fix your change then rebase mine on top for the opensuse etc stuff?20:26
@clarkb:matrix.orgspecifically you're not removing the fedora cron job or scripts or keytab from the mirror node I don't think. We're just not creating them anew20:27
@fungicide:matrix.orgi'm fine if you want to just incorporate the missing bits in your change and i'll abandon mine20:27
@clarkb:matrix.orgack20:27
-@gerrit:opendev.org- Clark Boylan proposed: [opendev/system-config] 1008520: Cleanup unused mirror content from mirrors and mirror-update https://review.opendev.org/c/opendev/system-config/+/100852020:30
@clarkb:matrix.organd done^ that drops the testing keytab values that aren't needed and removes the fedora rsync script20:30
@fungicide:matrix.orgchecking, thanks!20:31
@clarkb:matrix.orgI'm going to take advantage of some early fall weather and go for a bike ride20:35
@fungicide:matrix.orggreat idea20:36
@clarkb:matrix.orgits like perfect outside right now20:36
@fungicide:matrix.orgtomorrow morning we're planning to do a long (maybe 6-8mi) walk now that the weather's gotten nice again20:36
@fungicide:matrix.orgClark: one thing to note is that Dmitriy Rabotyagov has https://review.opendev.org/1005081 proposed to mirror new ceph versions (as of 2-3 weeks ago), so it might be good to confirm what the utility is for those20:51
@fungicide:matrix.orgapparently jobs are installing new versions from not-our-mirrors and it's taking a long time20:52
@fungicide:matrix.orgthough i see you also reviewed it a couple of weeks ago20:53
@fungicide:matrix.orgah, though i guess your change is not removing ceph mirroring20:55
@fungicide:matrix.orgbut it does seem like we have an opportunity to clean up a bunch of that too20:56
@clarkb:matrix.orgYes I think we may need to push back on that proposal given the state of things currently 22:23
@clarkb:matrix.orgWe just don't have the space to mirror what we already have let alone anything new 22:23
@clarkb:matrix.orgbike rides are good for thinking and I think I've convinced myself that the best bang for buck hard decision is dropping arm64 packages. So I'll work on a series of changes to propose that across centos stream, debian, and ubuntu22:49
@clarkb:matrix.orgfungi: looks like we're mirroring source packages for debian too. I think we can drop those. I'll propose that at the bottom of the stack since that shouldn't be impactful22:54
-@gerrit:opendev.org- Clark Boylan proposed:23:24
- [opendev/system-config] 1008547: Stop mirroring debian source packages https://review.opendev.org/c/opendev/system-config/+/1008547
- [opendev/system-config] 1008548: Drop centos stream arm64 pacakges https://review.opendev.org/c/opendev/system-config/+/1008548
- [opendev/system-config] 1008549: Drop debian arm64 packge mirrors https://review.opendev.org/c/opendev/system-config/+/1008549
- [opendev/system-config] 1008550: Drop UCA arm64 package mirrors https://review.opendev.org/c/opendev/system-config/+/1008550
- [opendev/system-config] 1008551: Drop Ubuntu ports arm64 package mirrors https://review.opendev.org/c/opendev/system-config/+/1008551
@clarkb:matrix.orgI think 1008547 and its parent are safe to go at any time. THough I think we need to manually run the reprepro command to delete orphaned packages afterwards?23:24
@clarkb:matrix.orgwe will probably need to do that for most thigns except ubuntu-ports since it has its own volume. We can just delete that volume23:24
@clarkb:matrix.organyway thats my proposal. I expect that will safe a couple of terabytes23:25
@clarkb:matrix.org* anyway thats my proposal. I expect that will save a couple of terabytes23:25

Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!