| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 999383: Add infra-prod-prometheus https://review.opendev.org/c/opendev/system-config/+/999383 | 09:12 | |
| @fungicide:matrix.org | disappearing for lunch, should be around again by 16:00 utc and will switch to opendev sysadmin reviewing and upgrades (if nothing explodes between now and then) | 14:46 |
|---|---|---|
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/grafyaml] 999344: Use copies of data when creating datasources https://review.opendev.org/c/opendev/grafyaml/+/999344 | 16:20 | |
| @fungicide:matrix.org | i tried pruning /opt/backups-202605 on backup03.ca-ymq-1.vexxhost because it's already at 92% but that didn't reduce utilization at all so i guess there's nothing to prune yet? | 18:36 |
| @fungicide:matrix.org | i'm worried at the rate it's going that the volume will be full by the end of the weekend | 18:36 |
| @fungicide:matrix.org | well, not "nothing" but it only freed 0.6% of the volume, so insignificant at least | 18:43 |
| @fungicide:matrix.org | borg-review03 on backup03 is 41% larger than on backup02 | 18:48 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed on behalf of Clark Boylan: [opendev/system-config] 998741: Update anubis to 1.26.2 https://review.opendev.org/c/opendev/system-config/+/998741 | 21:08 | |
| @fungicide:matrix.org | i've manually upgraded the anubis package on wiki.openstack.org to 1.26.2 | 21:14 |
| @fungicide:matrix.org | seems to still be working normally, though a soft refresh left my browsers on the anubis challenge page (hard refresh acted as expected) | 21:17 |
| @fungicide:matrix.org | i may merge 998741 over the weekend if i find time, since it's an opportunity to make sure things are functioning without as much risk of impact to users | 21:20 |
| @clarkb:matrix.org | fungi: yes backup pruning is based on keeping backups over certain intervals. I think that is described in the prune script. It is possible the new server has run for long enough to be prunable. Could also be that the slightly newer borg version has different behavior. I think it is possible that we need to run the borg compact command | 21:22 |
| @clarkb:matrix.org | fungi: maybe you can try running that against one of the backup repos to see if it frees up space? And if so then we probably need to add it to the prune script | 21:22 |
| @fungicide:matrix.org | i will try to compact | 21:23 |
| @fungicide:matrix.org | i've got `/opt/borg/bin/borg compact /opt/backups/borg-review03/backup` running in a root screen session, checking `df -h /opt/backups-202605` before and after to see what impact it makes | 21:27 |
| @fungicide:matrix.org | that's our largest backup repository, so if it doesn't make a dent there then it's probably not the solution | 21:28 |
| @clarkb:matrix.org | Ack | 21:28 |
| @fungicide:matrix.org | also isn't it saturday morning for you right now? go do weekend stuff | 21:35 |
| @clarkb:matrix.org | It is. But it's quiet so I thought I'd check in. I should go find something to do | 21:35 |
| @clarkb:matrix.org | Thanks for looking at the backups | 21:35 |
| @fungicide:matrix.org | of course! | 21:39 |
| @fungicide:matrix.org | i mean, i was prompted by a warning e-mail so it's not like i went hunting | 21:39 |
| @fungicide:matrix.org | Clark: good call! compacting the review03 backup freed 300gb | 21:53 |
| @fungicide:matrix.org | i'll look at shimming thay into the script | 21:53 |
| @clarkb:matrix.org | Nice. I would double check the command exists on the other backup server. It's possible this is a version specific difference | 21:56 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 999465: Compact borg repos after pruning https://review.opendev.org/c/opendev/system-config/+/999465 | 22:02 | |
| @fungicide:matrix.org | Clark: oh! good call, it's not an available subcommand on backup01.ord.rax, i'll wip for now | 22:04 |
| @fungicide:matrix.org | anyway, in the meantime i temporarily applied that one-line addition on backup03.ca-ymq-1.vexxhost and am rerunning the prune script to see how much we free by compacting all the repositories | 22:07 |
| @fungicide:matrix.org | yeah, compacting took the volume down from 92% utilization to 40% | 22:09 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!