Saturday, 2023-12-09

clarkbother thoughts: what if borg is failing too00:02
clarkband that breaks the pipe and then mysql can no longer write00:02
clarkbthe way logging is setup in the borg stream setup is it logs the first error in the pipeline which implies the problem is on the first command to fail, but I suppose to could be the second command (borg-backup) that is erroring00:03
clarkbI'm guessing that would be a good thing to try and get better logs for first00:03
fungiclarkb: i suspect there's some delay between when the dump is initiated and when the data begins to flow. if there's some middlebox terminating "idle" ssh sessions then using ssh keepalives might help?13:10
opendevreviewBirger J. Nordølum proposed openstack/diskimage-builder master: feat: add almalinux-container element
opendevreviewMerged openstack/project-config master: Remove retired js-openstack-lib from infra made it into the charts on
Clark[m]fungi: the error occurs after a minute or two. It shouldn't be long enough for typical keepalives to help. It's a really weird one. After sleeping on it I think I want to try the dump to file then cat it out process manually to see if that has different behavior. And also update the script to check errors for both sides of the pipe15:27
fungii remember years ago the default "idle" session timeout on both juniper netscreen and f5 bigip was 120 seconds unless configured otherwise15:29
fungiso i usually set my ssh servers and clients to a 60-second keepalive ping15:30
Clark[m]Huh that seems really low. I guess we can try that15:30
fungiit does indeed seem low, when by comparison openbsd was something like 2 weeks15:30
Clark[m]That would go into .ssh/config?15:30
fungiServerAliveInterval is what you want in .ssh/config, according to the manpage15:33
fungi"Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server."15:34
fungiit defaults to 0, which disables that behavior15:35
fungithere's also TCPKeepAlive but i've seen middleboxes ignore those15:36
JayFfungi: is the `openstackci` pypi account human-accessible at all?21:16
JayFfungi: asking in context of -- I'm thinking given that's a github repo, and we don't know the shape of the CI21:16
JayFmight be wise to have openstackci + a human (me, for now?) get granted access21:16
JayFbut I figured I'd ask about how frequently humans have to use that account already before worrying about complicating the message21:17
fungiJayF: yes, opendev sysadmins have access to shared login credentials and otp for that account21:43
Clark[m]I think uploads use tokens now? So in theory we can make a token for that repo and give that to the maintainers but I'm not sure if they can be scoped like that22:27
fungithey can, there are account-scoped and project-scoped upload tokens23:41

Generated by 2.17.3 by Marius Gedminas - find it at!