| @abelur:matrix.org | thank you! TIL the screenshots seems very useful for a quick review. | 01:56 |
|---|---|---|
| @mnasiadka:matrix.org | The reprepro export for Debian seems to be running very long, I think we’ll need to fix the cache, looking at the logs there was some failure and export process got killed on 12th Sep | 04:19 |
| @mnasiadka:matrix.org | Moved contents cache aside and running reprepro export in a tmux on mirror-update host | 08:38 |
| @noonedeadpunk:matrix.org | mnasiadka: regarding https://review.opendev.org/c/opendev/system-config/+/1005081 - do you think I should just drop Reef as well? | 08:56 |
| @harbott.osism.tech:regio.chat | I think we may be getting these issues pretty consistently due to timeouts when there is a new point release, which happened just on that date https://www.debian.org/News/2026/20260912 | 09:16 |
| not sure if we should increase the timeouts in general | ||
| @mnasiadka:matrix.org | I have a feeling that reprepro does not like getting SIGKILL anyways... | 14:18 |
| @clarkb:matrix.org | do we think our script needs to be updated (either more generous timeouts or trying to sigterm and give reprepro more time to close gracefully?) | 14:56 |
| -@gerrit:opendev.org- Zuul merged on behalf of Stephen Finucane: [openstack/project-config] 1003838: Initiate retirement of molteniron https://review.opendev.org/c/openstack/project-config/+/1003838 | 14:59 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 1005747: reprepro: Disable Contents generation for Debian https://review.opendev.org/c/opendev/system-config/+/1005747 | 15:03 | |
| @mnasiadka:matrix.org | Clark: Actually we can also think about ^^ | 15:03 |
| @clarkb:matrix.org | mnasiadka: I think I'm willing to try that. Its limited to debian in that change so we can start small before rolling it out more widely. It is the sort of thing that would be good to get fungi's opinion but he is out this week and if we think this is urgent I am fine with proceeding | 15:05 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 1005747: reprepro: Disable Contents generation for Debian https://review.opendev.org/c/opendev/system-config/+/1005747 | 15:06 | |
| @mnasiadka:matrix.org | Well, currently I'm running reprepro export on tmux on mirror-update, it's been running for at least 8 hours - and it's probably not even halfway done (it's fully rebuilding the contents.cache.db) | 15:06 |
| @mnasiadka:matrix.org | I'm happy with killing it, cleaning up and implementing the change to see if that works. | 15:07 |
| @mnasiadka:matrix.org | I can even do that manually first to check if that brings us to something working - if that's an option we want to pursue | 15:07 |
| @clarkb:matrix.org | I know in the past fungi has worked through day long updates so the timing doesn't surprise me. But if we think this is a potential fix then it may be worth trying? maybe Jens Harbott and corvus have thoughts? | 15:08 |
| @jim:acmegating.com | regarding removing the "Contents" item: i yield to others with more definitive knowledge of whether that's necessary. but in general, i agree, less is better here, so if we can do less, great. :) | 15:13 |
| regarding the locking -- it's only byte-range locking that should be a problem with openafs; i thought reprepro only used file-locking which should be fine. so i do wonder if afs is a contributing factor here. perhaps it's just "killing reprepro is bad" no matter the filesystem? | ||
| @clarkb:matrix.org | it could also be the construction of contents does a full scan which thrashes the cache? | 15:15 |
| @jim:acmegating.com | (i have medium confidence in that afs stuff i said, btw; it's been a minute) | 15:15 |
| @jim:acmegating.com | yeah, sounds like removing contents may be a lot less work which sounds good | 15:16 |
| @mnasiadka:matrix.org | I think "killing reprepro is bad" is a general thing | 15:16 |
| @mnasiadka:matrix.org | Contents DB is only used when searching (e.g. apt-file search something) and is purely optional | 15:18 |
| @mnasiadka:matrix.org | And that operation should not happen on CI nodes in general (I'd assume) :-) | 15:18 |
| @clarkb:matrix.org | ya similar to us not mirroring source packages | 15:19 |
| @clarkb:matrix.org | its not a typical CI use case so we can trim things down by avoiding it | 15:19 |
| @mnasiadka:matrix.org | If we want to merge the patch, I'm happy to monitor that and kick off reprepro export when the debian mirror config files get updated | 15:25 |
| @jim:acmegating.com | sounds good to me | 15:26 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 1005755: Switch minio/minio to pigsty/silo https://review.opendev.org/c/zuul/zuul-jobs/+/1005755 | 15:36 | |
| @clarkb:matrix.org | mnasiadka: I would say go ahead and self approve the change when you have time to babysit the update without contents | 15:37 |
| @mnasiadka:matrix.org | On other front ipv6 on Gerrit is still not in a good shape, do we have any other channels to VexxHost than just bugging mnaser ? ;-) | 15:55 |
| @clarkb:matrix.org | We can file a ticket in the vexxhost issue tracker. Others like rico lin have also helped in the past if we want to track someone down | 16:05 |
| -@gerrit:opendev.org- Clark Boylan proposed: | 16:53 | |
| - [zuul/zuul-jobs] 1005755: Switch minio/minio to pigsty/silo https://review.opendev.org/c/zuul/zuul-jobs/+/1005755 | ||
| - [zuul/zuul-jobs] 1005764: Convert mc to mcli in s3 tests https://review.opendev.org/c/zuul/zuul-jobs/+/1005764 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 1005747: reprepro: Disable Contents generation for Debian https://review.opendev.org/c/opendev/system-config/+/1005747 | 17:00 | |
| @mnasiadka:matrix.org | Ok, and fantastically export happened in less than 3 minutes, check passed and now vos release is running | 17:03 |
| @clarkb:matrix.org | nice. Just need to double check our expectations about usage are correct too (monitor jobs using the debian mirrors) | 17:05 |
| @mnasiadka:matrix.org | I've left the old db/contents.cache.db (with a unixtime extension) which is ~1GB - but we could remove it (since it's corrupted anyway) | 17:08 |
| @mnasiadka:matrix.org | And not needed anymore. | 17:08 |
| @mnasiadka:matrix.org | But the volume quota seems to be close | 17:08 |
| @clarkb:matrix.org | removing it makes sense if it is non functional | 17:10 |
| @clarkb:matrix.org | unless rebuilding it is easier with the broken content in place. In that case keeping it until we know we won't rebuild it may be a good idea | 17:10 |
| @clarkb:matrix.org | (but I suspect that isn't the case since rebuilding it was already very slow) | 17:11 |
| @mnasiadka:matrix.org | Yeah, I'll remove it, rebuild is basically remove + reprepro export | 17:13 |
| @mnasiadka:matrix.org | Anyway, Kolla Debian build jobs seem to be happy now, so we should be good | 17:14 |
| @mnasiadka:matrix.org | mirror.debian quota usage is 91% | 17:14 |
| @clarkb:matrix.org | Dan With: fungi is out now, but I wanted to followup on the rackspace flex auth issue we ran into that you thought you might know the cause of. Is there something we should be doing on our end to try and address that? eg would generating a new token reset things or is this the sort of situation where we can't address it from our end and need to file a ticket or similar? | 17:19 |
| @dan_with:matrix.org | Clark: We found an errant keystone setting, and it will be remediated. 90% sure it is related to the cause. If you have accounts that are locked out, contact me DM and I will unlock them for you. | 20:09 |
| @dan_with:matrix.org | I Just need the account number DDI | 20:10 |
| @clarkb:matrix.org | thanks. I've started a matrix DM message with you to share that info | 20:23 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!