| fungi | clarkb: thanks for working through the paste server replacement! i'll try to take a look at those followup changes tomorrow | 00:08 |
|---|---|---|
| clarkb | sounds good | 00:14 |
| clarkb | fungi: then depending on your availability we may want ot proceed with gitea upgrades and/or gerrit updates | 00:34 |
| fungi | yeah, i expect to be around for a good chunk of tomorrow | 00:36 |
| clarkb | may also be worth noting that I didn't hit any hiccups with podman | 00:37 |
| clarkb | the compatibility with docker ocmpose and our docker-compose shim seems to be working as testing indicated | 00:38 |
| fungi | excellent | 00:52 |
| corvus | huzzah | 00:55 |
| opendevreview | Merged openstack/project-config master: oslo: Remove irc notification from oslo-incubator https://review.opendev.org/c/openstack/project-config/+/938897 | 10:10 |
| opendevreview | Merged openstack/project-config master: Remove infrawatch/sg-core from zuul repos https://review.opendev.org/c/openstack/project-config/+/938537 | 10:10 |
| *** dviroel is now known as dviroel_brb | 12:47 | |
| *** dviroel_brb is now known as dviroel | 13:00 | |
| opendevreview | Merged openstack/project-config master: Add Zuul jobs for ansible-role-httpd project https://review.opendev.org/c/openstack/project-config/+/935695 | 13:00 |
| opendevreview | Merged openstack/project-config master: Add Zuul jobs for frrouting role https://review.opendev.org/c/openstack/project-config/+/935696 | 13:05 |
| corvus | clarkb: looks like the jaegertracing mirror worked, which means the updated version of the job with the repo auto-creation worked. | 14:26 |
| Clark[m] | Nice and in theory zuul ci should be a lot happier now | 14:55 |
| Clark[m] | Just need to keep an eye on how often the mirror jobs fail I guess | 14:56 |
| Clark[m] | corvus: I have a change up to mirror uwsgi-base so that I can move all of lodgeit/paste off of docker hub now that it runs under podman | 14:56 |
| corvus | yeah i just re-approved the change in zuul to use the mirror, so once that lands we'll see it in practice there... | 15:13 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: fix(packer): prevent task failure when packer_variables is not defined https://review.opendev.org/c/zuul/zuul-jobs/+/836744 | 15:19 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: [upload-logs-base] add ipa extension to known mime types https://review.opendev.org/c/zuul/zuul-jobs/+/834045 | 15:21 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:22 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:23 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: [upload-logs-base] add public url attribute https://review.opendev.org/c/zuul/zuul-jobs/+/834043 | 15:27 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: [upload-logs-base] add public url attribute https://review.opendev.org/c/zuul/zuul-jobs/+/834043 | 15:30 |
| corvus | Clark: fungi thinking more about the mirror jobs on-demand... i don't think my idea of running them on job config changes in deploy/promote would work because the config should have changed by then and there should be no delta to trigger the job... :/ | 15:32 |
| corvus | i guess we could use file matchers if we split them into individual files, but yuck. | 15:33 |
| opendevreview | Andy Ladjadj proposed zuul/zuul-jobs master: [upload-logs-base] add android mime-type https://review.opendev.org/c/zuul/zuul-jobs/+/834046 | 15:36 |
| opendevreview | Merged opendev/system-config master: Mirror uwsgi base image https://review.opendev.org/c/opendev/system-config/+/939383 | 15:36 |
| fungi | ah well, it was worth pondering anyway | 15:37 |
| *** jhorstmann is now known as Guest6084 | 15:38 | |
| fungi | and at least a reminder that we have to wait for a daily run to occur (and succeed) before merging changes that use it | 15:38 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/939126 is probably the best place to start with new paste | 15:52 |
| clarkb | the other changes followon and/or change some image stuff which is less important. I think if we get backups going then remove the old server we're in good shape | 15:52 |
| clarkb | fungi: also I don't know that the bindep changes have landed yet. Should we go ahead and land the first series and possibly make a release? | 15:53 |
| clarkb | I think my prioritiy today is getting paste backups sorted, reviewing niz, and landing bindep changes that we are comfortable with. Then if time allows gitea and gerrit updates | 15:54 |
| fungi | i'll take a look in a bit, trying to wrap up something else real quick | 15:54 |
| clarkb | sounds good I need breakfast and can catch up on email | 15:55 |
| amorin | hey fungi clarkb for you info, it seems there are some inconsistencies on OVH BHS1 region with some of your instances (np*). We are deleting them on our side, I believe it's not a big deal for openinfra automation, but better letting you know that | 16:00 |
| clarkb | amorin: our nodepool server should attempt to delete instances that it no longer needs, but if that fails then yes deleting them on the cloud side should be fine. If you accidentally delete something zuul and nodepool were using it should detect that as a network failure and reschedule the job | 16:01 |
| clarkb | amorin: maybe let us know when you think you are done so that we can avoid attributing any unrelated oddities to this? | 16:01 |
| clarkb | and thank you for the heads up | 16:02 |
| amorin | the team told me the instances were in building and deleted state | 16:02 |
| amorin | :) | 16:02 |
| amorin | the clean is ongoing, will let you know when done | 16:02 |
| clarkb | fungi: I think I'll go ahead and approve the bindep changes that I believe are safe | 16:11 |
| fungi | wfm, thanks | 16:12 |
| clarkb | basically everything before the pyproject.toml addition | 16:13 |
| fungi | yeah, i mean in my estimation the entire series is "safe" from an "it works and should have no user-facing impact) standpoint, but some of the later changes in there are more likely to be controversial and could stand some additional opinions | 16:15 |
| fungi | i did try to sequence them in order from least to most questionable, as much as possible | 16:16 |
| clarkb | yup. Also I think doing at least two releases with a checkopint before pyproject.toml is a good idea | 16:17 |
| clarkb | we can do that without waiting to land changes though | 16:17 |
| clarkb | just tag a commit older than HEAD | 16:17 |
| clarkb | I set up all the changes with topic:noble-paste | 16:21 |
| fungi | thanks! going through them now | 16:27 |
| fungi | all of those lgtm | 16:32 |
| clarkb | cool once we've confirmed paste02 backups are running I can approve https://review.opendev.org/c/opendev/system-config/+/939128 too | 16:35 |
| clarkb | and at some point need to delete the old server too | 16:35 |
| clarkb | other things that came out of this: we can switch to the mariadb mirror image pretty easily but it does restart the container (at least it did for paste I don't think it iwll for review because we don't directly manage the containers that way on that server) | 16:38 |
| clarkb | wondering if maybe we want to bundle a switch of the image into the gitea upgrade? any opinions on that? | 16:38 |
| opendevreview | Merged opendev/zone-opendev.org master: Reset paste.o.o's TTL to the default (1 hour) https://review.opendev.org/c/opendev/zone-opendev.org/+/939379 | 16:39 |
| clarkb | I can update that change. It should be a noop functionality wise so won't require a bunch of careful rereview | 16:39 |
| clarkb | fungi: ^ any opinion on that? | 16:43 |
| fungi | just a sec, first looking into the block storage trouble ticket rackspace opened | 16:45 |
| fungi | The server which hosts your Cloud Block Storage device, System disk for 2aa2768d-567e-4685-9f99-df7d0dda8ffb, '7c7ee2bc-dd37-4916-9db2-3048e4c30e70' requires an emergency maintenance. | 16:46 |
| fungi | i'll see if i can work out which server that is | 16:46 |
| clarkb | we don't usually bfv | 16:47 |
| opendevreview | Merged opendev/bindep master: Update for newer Python, Tox and Hacking https://review.opendev.org/c/opendev/bindep/+/932190 | 17:05 |
| opendevreview | Merged opendev/bindep master: Support and prefer unittest.mock https://review.opendev.org/c/opendev/bindep/+/878733 | 17:05 |
| fungi | i'm having no luck finding the impacted server instance, starting to suspect it might be a trove server | 17:15 |
| clarkb | that could also explain why it is bfv if they use that in trove | 17:15 |
| fungi | yeah, unfortunately i'm not in a good spot to get in through the rackspace dashboard nor do i think we have the right plugins on bridge to use the openstack cli for trove stuff | 17:18 |
| fungi | but on the other hand we have very little in production relying on trove at this point | 17:18 |
| fungi | if someone else gets a chance to dig deeper, i've left the ticket notifications unread in our shared inbox for now | 17:22 |
| fungi | clarkb: yeah, probably fewer outages is good as long as we don't anticipate too many independent modifications conflating debugging of a problem. i doubt that's a risk this time | 17:30 |
| clarkb | ok let me update the change | 17:30 |
| fungi | i'll need to disappear shortly for lunch, but can help with gitea/gerrit upgrades later i think | 17:31 |
| opendevreview | Merged opendev/bindep master: Fix Python version guessing logic for Debian Sid https://review.opendev.org/c/opendev/bindep/+/932191 | 17:33 |
| clarkb | cool oh also gerrit upstream noted someone else had the cache problem we saw | 17:33 |
| clarkb | I'm hoping this leads to better cache management and in the meantime we need to be careful with that. Considering the cache problems I'm thinking maybe gerrit is a good oen to defer if we don't have time/ability to debug if necessary | 17:34 |
| fungi | awesome | 17:35 |
| fungi | headed out, i'll bbiaw | 17:37 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to Gitea 1.23 https://review.opendev.org/c/opendev/system-config/+/938826 | 17:44 |
| clarkb | I won't bother to rehold the node in the child change. I think if ci works for that patchset we should be fine | 17:44 |
| opendevreview | Merged opendev/bindep master: Add testing for Debian testing and unstable https://review.opendev.org/c/opendev/bindep/+/932192 | 17:45 |
| opendevreview | Merged opendev/bindep master: Move stray release note into the correct location https://review.opendev.org/c/opendev/bindep/+/938522 | 17:45 |
| opendevreview | Merged opendev/system-config master: Backup paste02 to the backup servers https://review.opendev.org/c/opendev/system-config/+/939126 | 18:06 |
| opendevreview | Merged opendev/system-config master: Remove paste01 from config management https://review.opendev.org/c/opendev/system-config/+/939378 | 18:06 |
| clarkb | those didn't enqueue the backup job. I'm working on a fix for that now | 18:09 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix backup job file triggers https://review.opendev.org/c/opendev/system-config/+/939477 | 18:14 |
| clarkb | I think landing ^ should correct that for us | 18:14 |
| clarkb | https://zadzmo.org/code/nepenthes/ | 18:18 |
| clarkb | when you fight fire with fire you get ^ | 18:19 |
| clarkb | https://groups.google.com/g/repo-discuss/c/0GYHpMxGTT0 <- other people having problms with gerrit caches | 18:24 |
| clarkb | git_file_diff and gerrit_file_diff caches on our install are at 24GB large right now. Still much smaller than when we had problems | 18:25 |
| clarkb | but also quite a bit bigger than any other cache on the system | 18:25 |
| clarkb | If I'm reading Martin correctly they seem to attribute this to bloomfilter creation for the database being slow on large DBs. This was one of the potential problems that can happen when I originally raised but and they seem to confirm it here observing it on their own server. | 18:33 |
| clarkb | The issue is all db operations are synchronous (something we inferred) and slow anything (in thise case bloomfilter generation) locks that db and nothing can use it. This db is used to generate pages for changes and eventually you get timeouts and sadness from the webserver because things queue up behind the lock | 18:33 |
| clarkb | there are a couple of efforts to make this particular problem better. The first is toe update h2 which would allow concurrent db access. The other | 18:34 |
| clarkb | er | 18:34 |
| clarkb | the other has to do with avoiding unnecessary cache growth during reindexing (so the db's stay small) | 18:34 |
| clarkb | or maybe smaller :) | 18:35 |
| clarkb | re paste02 backups I think the change to remove paste01 from inventory may actually trigger the backup jobs so landing the file trigger match fix isn't super urgent. But probably best to land things like that hwne we notice them so we don't have problems later | 18:36 |
| opendevreview | Ghanshyam proposed opendev/irc-meetings master: Updating the QA office hours https://review.opendev.org/c/opendev/irc-meetings/+/939480 | 18:54 |
| clarkb | ya the backup job is enqueued now so hopefully soon we can confirm we have backups running | 18:59 |
| opendevreview | Merged opendev/system-config master: Fix backup job file triggers https://review.opendev.org/c/opendev/system-config/+/939477 | 19:12 |
| opendevreview | Ghanshyam proposed opendev/irc-meetings master: Updating the QA office hours https://review.opendev.org/c/opendev/irc-meetings/+/939480 | 19:14 |
| clarkb | and the job failed. Looking into it now | 19:38 |
| clarkb | borg failed to build against python3.12 due to ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’ | 19:44 |
| fungi | huh | 19:46 |
| opendevreview | Merged opendev/irc-meetings master: Updating the QA office hours https://review.opendev.org/c/opendev/irc-meetings/+/939480 | 19:46 |
| clarkb | latest borgbackup won't run on the server side (need newer python I think) but the internet seems to say that borg is tolerant of different versions between client and server | 19:47 |
| clarkb | Ithink we may half step this and install a newer version of borg on noble and keep everything else the same. I've already got borg upgrade discussions on the meetup agenda for next week and we can dig into that futher | 19:47 |
| clarkb | I'll whip up a change shortly | 19:48 |
| clarkb | actually the latest 1.2 series release was from march 2024 which might be new enough for python3.12 (released october 2023) | 19:49 |
| fungi | looks like borgbackup 1.2.7 officially added python 3.12 support | 19:49 |
| fungi | according to their changelog | 19:49 |
| clarkb | ok cool so I think we half step noble to 1.2.8 and then plan to move everything else up to 1.2.x maybe (1.2 requires python3.8 or newer so some old stuff might stay on old versiosn) | 19:49 |
| fungi | https://github.com/borgbackup/borg/blob/1.4-maint/docs/changes.rst#version-127-2023-12-02 | 19:49 |
| clarkb | working on the initial noble bump up change now | 19:50 |
| clarkb | now that i think of it tonyb may have idscoverd this but I can't find any fixes from tonyb doing some quick seearches | 20:03 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use newer borgbackup on Ubuntu Noble https://review.opendev.org/c/opendev/system-config/+/939491 | 20:06 |
| clarkb | I'm really glad we're slowly stepping through this with paste. This is a good exercise for finding the problems without a ton of risk | 20:07 |
| clarkb | fungi: ^ 939491 should hopefully pass CI (indicating there are no major incompatibilities between 1.1.18 on the server side and 1.2.8 on the client) and then we can get that deployed to paste02 and see if it works in practice | 20:07 |
| clarkb | while I wait for that I'm going to eat lunch | 20:08 |
| fungi | cool, i need to step back out for about 30 minutes myself | 20:19 |
| clarkb | https://borgbackup.readthedocs.io/en/1.2-maint/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811 may be a problem for us | 20:27 |
| clarkb | we run borg check on the backup servers (so the servers act as a client) | 20:28 |
| clarkb | except for a couple things. First this would be an entirely new archive for a new host so we aren't worried about losing any archives improperly and second we don't run borg check with borg > 1.2.4 against the old archives | 20:30 |
| clarkb | I suspect that this is probably ok in the interim while we figure out the upgrade coordination | 20:30 |
| clarkb | this is goign to be rough for older backup hosts/clients though. Not sure how we handle that yet. I think for paste though since it will be a new archive the risks are very low worst case we only impact it | 20:34 |
| clarkb | so I think the main risk here is that if you update without following the upgrade procedures borg check could complain about archives in repos in a state it doesn't like. borg check --repair would then unconditionally delete those archives | 20:43 |
| clarkb | since this is a new repo with no archives yet I still think the risk is low for paste02. I also suspect the risk is low for the other servers too. It seems that with our archive pruning it is unlikely we've got archives old enough to not have TAMs but I can try to check that after I get the change apssing | 20:44 |
| clarkb | " Archives created by old borg versions < 1.0.9 do not have TAMs. Archives created by newer borg version should have TAMs already." | 20:45 |
| clarkb | from https://borgbackup.readthedocs.io/en/stable/usage/upgrade.html#borg-1-x-y-upgrades | 20:46 |
| clarkb | so basically my read on this is we can get problems with mixed versions if everything doesn't already have a tam yet | 20:46 |
| clarkb | and while we still have older versions in use new archives could be added without TAMs which would cause problems with newer borg but that would indicate an attack so we probably do want it to raise the alarm and alert us | 20:47 |
| clarkb | I think I can't use facts when defining role defaults because the facts haven't been loaded yet | 20:54 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use newer borgbackup on Ubuntu Noble https://review.opendev.org/c/opendev/system-config/+/939491 | 21:01 |
| clarkb | "all valid archives just need to get a valid TAM, all malicious archives need to get removed and we'll be done with that problem." from https://github.com/borgbackup/borg/issues/7802 | 21:21 |
| clarkb | I'm having a hell of a time validating that our existing archives have TAMs :/ | 21:21 |
| clarkb | but I'm feeling a lot more confident this is a safeish update | 21:21 |
| clarkb | I think worst case we might have some older archives (perhaps on the server that didn't get things retired and pruned) that will accelerate our plan to retire and prune things there | 21:22 |
| clarkb | all other archives should be newer and should have valid TAMs | 21:22 |
| clarkb | (even though I haven't actually be able to confirm that yet). But I also believe taht running with newer borg on paste02 shoudl be fine and only affect archives for that host | 21:22 |
| clarkb | ianw: I know it is Friday for you and you're not working on this stuff regularly anymore. But if by chance you haev a moment to sanity check all of ^ that would be more than welcome | 21:23 |
| fungi | we so rarely restore from backups that the window for exploiting it is likely nonexistent | 21:23 |
| clarkb | `borg info --debug` against a repo shows TAM-verified manifest | 21:26 |
| clarkb | fungi: ya I'm more concerned with us losing valid backups than being exploited (though I want to be cautious about both things) | 21:27 |
| clarkb | I suspect that any backups that would be uncategorically deleted by borg check --repair are also so old that we don't really care about them though | 21:27 |
| clarkb | anything current should be tam verified bceause we're running with a new enough borg to tam by default | 21:27 |
| clarkb | I think the easiest way to debug this is to install a newer borg client (like 1.2.6 or newer) and then run commands against the archives for that host | 21:29 |
| clarkb | but we don't have to switch that host to using that client to check yet? | 21:29 |
| clarkb | https://github.com/nextcloud/all-in-one/discussions/3338#discussioncomment-6993215 essentially that is what this recommends | 21:30 |
| clarkb | you use a newer client with a command that checks the tam state while bypassing the default requirement for the tam to verify | 21:30 |
| clarkb | basically all 1.2.5 did was require that everything be tam verified (unless you pass that special flag) | 21:31 |
| clarkb | so I think worst case we may discover we have archives that are not tam verified and in those cases we can simply delete the archives and move on. | 21:31 |
| clarkb | and I think I'm reasonably confident we can proceed with newer client on noble and then plan to upgrade everything else in the short but not immediate term | 21:32 |
| fungi | well, we also have redundant backup servers. if the risk is low we could just upgrade one at first? | 21:32 |
| clarkb | fungi: ya I think the reason I discounted that was this page: https://borgbackup.readthedocs.io/en/1.2-maint/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811 says upgrade all clients first | 21:33 |
| clarkb | but the more I read about this I think we can get away with not doing that and addressing any fallout that might occur so ya that could be a good option | 21:33 |
| clarkb | that document is giving you the difficult but most reliable path forward. We're looking at a easier but potentially less reliable path forward and I think I'm ok with that | 21:34 |
| clarkb | also I don't think we run borg check --repair anywhere automatically | 21:34 |
| clarkb | so this seems even safer. They were just so strongly anti running borg check until everyone updated first that I was concerned about it. | 21:34 |
| clarkb | I was initially concerned that if we upgraded without following the steps we could lose all of our archives but I don't think that is likely and it would only occur if we ran borg check --repair? | 21:35 |
| clarkb | "The check command is a readonly task by default. If any corruption is found, Borg will report the issue and proceed with checking. To actually repair the issues found, pass --repair." yup I think that is safe | 21:36 |
| clarkb | I don't like the way they've written the upgrade notice. I think it is misleading and makes things much scarier than they really are | 21:37 |
| fungi | did we still want to try upgrading gitea and/or gerrit today? | 21:45 |
| clarkb | I'm leaning towards no. I'm now trying to sort out why noble with borg 1.2.8 exits rc 1 when backing up to 1.1.18 on the server | 21:49 |
| clarkb | there is no obvious error in the log that I see yet but it is a big log so I'm still looking | 21:50 |
| clarkb | it also seems to generally work because the second pass with random data doesn't fail | 21:51 |
| fungi | no worries, i'm semi-distracted here at the moment anyway, but should have some available time tomorrow if you're still interested in trying to knock it out before next week | 21:53 |
| clarkb | I think the new goal is probably to get paste backing up so that we can see what the borg check behavior is over the weekend | 21:54 |
| clarkb | I'm beginning to suspect that the error here is due to the warning about a previously unknown repo despite our flag saying that is ok being bubbled up as a warning == rc 1 error | 21:55 |
| fungi | paste02 being the previously unknown repo? | 21:57 |
| clarkb | no in CI its a made up test specific one | 21:58 |
| clarkb | https://github.com/borgbackup/borg/blob/4832364c1d602896c636a5330516a16ef47c4025/src/borg/archiver/__init__.py#L677-L692 this code does seem to be trying to use different rc codes for varying levels of success | 21:59 |
| fungi | ah | 22:00 |
| clarkb | it classifies the exit code into: success, warning, error, or signal. Only success appears to be 0. So my hunch here is that since we get a warning for backing up to a previously unknown repo for the first time it treats that as rc 1. The next pass doesn't fail because its already there and gets rc0 | 22:00 |
| clarkb | I think my solution to this may be to have the script run the backups twice and only check the rc code the second time | 22:00 |
| clarkb | this ensures we coalesce to where we want to be | 22:00 |
| fungi | that makes sense | 22:00 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use newer borgbackup on Ubuntu Noble https://review.opendev.org/c/opendev/system-config/+/939491 | 22:07 |
| clarkb | fungi: I'm wary of landing ^ at this point if we're not able to watch it. Mostly from a risk assessment perspective I'd rather roll back paste contents to pre upgrade state if necessary than potentially break all of our other backups if we're not in a position to debug and address it | 22:10 |
| clarkb | thats probably better for an early tomorrow land and monitor appraoch? | 22:11 |
| clarkb | the current state of the job is that it will continue to fail but its failing gracefully (just not adding paste02 to the backup servers and not backing up paste02 backups for everything else should contineu to run) | 22:11 |
| fungi | works for me, sure | 22:12 |
| clarkb | ianw: also thank you for the excellent set of test cases here | 22:13 |
| clarkb | they have been quite useful in building confidence this will work in some capacity and in a way we expect it to | 22:14 |
| clarkb | corvus: ^ I'm not sure if you've followed along but this is the sort of thing that you may have an interest in too just because it is at the intersection of python3.12, noble, and important background services like backups | 22:16 |
| clarkb | related: I feel like bup died for similar reasons though they were harder to address (python3 was the problem iirc?) | 22:16 |
| fungi | just wait till we get to python 3.13, there's a lot of new excitement with stdlib removals | 22:18 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Deprecate ironic-lib https://review.opendev.org/c/openstack/project-config/+/939282 | 22:18 |
| clarkb | fungi: I'll try to forget you said that until later :) | 22:18 |
| clarkb | one problem at a time for now | 22:18 |
| fungi | indeed | 22:22 |
| clarkb | arg I think I just foudn another issue with the 1.1.x to 1.2 upgrade: the exclude paths have leading /s in them and apparently 1.2 doesn't want that? | 22:31 |
| clarkb | https://borgbackup.readthedocs.io/en/stable/changes.html#borg-1-1-x-to-1-2-x | 22:31 |
| clarkb | I think our CI job should cover that so I'll try and look into that next | 22:32 |
| clarkb | https://borgbackup.readthedocs.io/en/1.2-maint/usage/help.html#borg-patterns so I think it won't just staright up break but they do convert it to the format they want | 22:36 |
| clarkb | "A leading path separator is always removed." | 22:40 |
| clarkb | this means we don't need to rewrite all of our rules. I'm not sure how 1.1.18 will handle that so for now keeping them as is and letting it remove the leading separator is probably best | 22:40 |
| clarkb | then just check we don't have any rules that produced unexpected results with unanchored paths? Though I suspect they are functionally anchored since it is processing the listed paths that way. Its like an implicit anchor? | 22:41 |
| clarkb | " In case your OS does not provide Python >= 3.8, consider using our binary, which does not need an external Python interpreter." this may be another option we could consider to get everything running a consistent version | 22:42 |
| clarkb | https://borgbackup.readthedocs.io/en/stable/usage/general.html#return-codes this seems to confrirm the rc code behavior we saw | 22:48 |
| *** jhorstmann is now known as Guest6115 | 22:53 | |
| clarkb | fungi: if you get a chance if you can read over https://borgbackup.readthedocs.io/en/1.2-maint/changes.html#pre-1-2-5-archives-spoofing-vulnerability-cve-2023-36811 and https://borgbackup.readthedocs.io/en/stable/changes.html#borg-1-1-x-to-1-2-x before we proceed tomorrow just to sanity check that would be great | 22:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use newer borgbackup on Ubuntu Noble https://review.opendev.org/c/opendev/system-config/+/939491 | 23:01 |
| clarkb | it seems the rc 1 for warning behavior isn't consistent? | 23:01 |
| clarkb | at least the docs explicitly say 1 means warning so we can be a little ness concerned about it | 23:02 |
| clarkb | if the second pass fails then I'll be concerned. Might be worth some rechecks too maybe | 23:02 |
| clarkb | ok found it in the mess of log output /var/log/borg-backup-borg-backup01.region.provider.opendev.org.log: file changed while we backed it up | 23:04 |
| clarkb | this means that we can always return rc 1 and not even the second pass is safe /me will take another approach | 23:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use newer borgbackup on Ubuntu Noble https://review.opendev.org/c/opendev/system-config/+/939491 | 23:11 |
| clarkb | I've tried to braindump a bunch of this into our virtual meetup etherpad too | 23:16 |
| clarkb | I was hoping to spend time today preping for that while waiting for gitea to deploy etc, but that hasn't happened yet. If the latest ps above ^ is happy I'll start digging into that agenda more braodly and not just on that single topic | 23:17 |
| clarkb | progress but now our borg-mount script isn't working or at least not the way our test expects it to | 23:44 |
| fungi | clarkb: sorry, stepped away to get dinner, taking a look now | 23:48 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use newer borgbackup on Ubuntu Noble https://review.opendev.org/c/opendev/system-config/+/939491 | 23:53 |
| clarkb | I think I decided that I should go ahead and hold the nodes. But I noticed the borg mount script isn't set -e so the mount might be failing. That latest patchset makes it -e so that we can get that infor more clearly if that is the caase | 23:54 |
| clarkb | and now to add an autohold | 23:54 |
| fungi | i've got the changelogs up in my browser and will take a careful read through in a few minutes when i get back to the hotel | 23:54 |
| clarkb | thanks I don't think its urgent given how slowly I'm working through the CI job results | 23:55 |
| clarkb | I've got an autohold in place that we should be able to use now to debug this | 23:56 |
| clarkb | the odd thing is both borg create and borg verify worked so you'd expect mount to be happy but maybe therei s some new flag we need to provide or similar | 23:57 |
| clarkb | I've got a family dinner I'm going to need to prepare for and attend shortly too. I'm hoping I can get a new ps that pushing things forward before that but we shall see | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!