| *** jroll07 is now known as jroll0 | 07:16 | |
| opendevreview | Merged openstack/project-config master: Migrate base propose-updates job to Jammy https://review.opendev.org/c/openstack/project-config/+/961475 | 09:59 |
|---|---|---|
| frickler | mordred: the secret for OSC you created in https://review.opendev.org/714502 needs rotation, seems you might be the only person having access? we could also drop the osc-image uploads instead if nodody is using them? | 10:08 |
| frickler | as a side note it seems weird to me that the job is running in gate, blocking merges if it fails. wouldn't it be better to do it in the post pipeline? e.g. https://review.opendev.org/c/openstack/python-openstackclient/+/962778 | 10:09 |
| fungi | still no newer hits to update_bug.py in the gerrit error_log since we rotated its application credentiall | 13:59 |
| fungi | https://launchpad.net/~hudson-openstack/+karma shows a bunch of bug comments though only some of them line up with the approximate time of release changes merging, suggesting that e.g. the one "4 hours ago" was from gerrit at least | 14:27 |
| fungi | (no openstack release requests merged between when we made the credential change and 4 hours ago) | 14:27 |
| frickler | yes, I don't think we have any releases pending that would trigger LP updates currently | 14:55 |
| fungi | bug comments are working, https://bugs.launchpad.net/keystonemiddleware/+bug/2129018 got them in the past few minutes | 15:36 |
| fungi | i'll go ahead and revoke authorization for the old credential now | 15:36 |
| clarkb | awesome thank you for handling that | 15:45 |
| frickler | ah, let's hope we get a couple of fresh releases from that soonish, too | 16:00 |
| fungi | right, i'll be monitoring that bug and we should see the zuul secret exercised that way | 16:02 |
| fungi | so far we only know that the token for gerrit updates is working, and that's configured via private hostvars rather than zuul secret (which now has a dedicated token to itself, for future flexibility and improved separation of concerns) | 16:03 |
| fungi | and i reduced the scope of privileges for these new tokens to just what i think they need, which makes all of this a bit more secure than what we had before too | 16:04 |
| *** gmaan is now known as gmaan_afk | 16:33 | |
| opendevreview | Mohammad Issa proposed openstack/project-config master: Add repo app-openvswitch for starlingx https://review.opendev.org/c/openstack/project-config/+/973521 | 17:31 |
| clarkb | fungi: I'm looking at the failed borg backup for lists and it looks like we've had a backup running to the ord server since the 12th which is why subsequent backups have failed (they cannot get the lock to backup to that server) | 17:33 |
| fungi | oh, i wonder what got it stuck | 17:33 |
| clarkb | fungi: strace on the python process has it stuck on a read | 17:34 |
| clarkb | a quick google serach indicates that borg should handle interrupted backups safely | 17:34 |
| fungi | i don't see anything remotely related to the lists server nor backups happening on monday, so it was probably a fluke | 17:35 |
| clarkb | I suspect that we can probably kill the process and then monitor tomorrow's backup pass to ensure it is happier? | 17:35 |
| clarkb | ya maybe something in the networking between them? | 17:35 |
| fungi | yeah, sgtm | 17:35 |
| clarkb | fungi: 2769786 is the python process and it has a child ssh (the transport mechanism) with pid 2769789 | 17:36 |
| clarkb | fungi: I'm thinking I'll just do a sudo kill 2769789 and see what happens? | 17:36 |
| fungi | right, interrupt the child and let the parent clean up | 17:36 |
| clarkb | ok I'll do that now | 17:37 |
| fungi | thanks! | 17:37 |
| clarkb | https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there here are the upstream docs. Killing ssh doesn't seem to have killed the python process | 17:39 |
| clarkb | so I think I'll kill 2769786 next | 17:39 |
| clarkb | fungi: looking at docker ps -a I think that docker restarted the mariadb server which interrupted our mysqldump | 17:40 |
| clarkb | and that is what we must be stuck on a read for | 17:40 |
| clarkb | 2769798 is the mysql dump so I'm going to try and kill it first | 17:41 |
| clarkb | the trace of the borg process is proceeding now with munmaps | 17:43 |
| clarkb | I suspect that if we wait a bit now it will figure out that things went side ways | 17:44 |
| clarkb | as for preventing this in the future I wonder if we should run borg under timeout? Say 48 hours? | 17:44 |
| clarkb | and it emailed us when the old running process exited with failure. | 17:45 |
| clarkb | tomorrow we should check that it succeeds and in the meantime think about how we might avoid this problem in the future. I'm surprised that mysqldump doesn't exit with an error if the mysql server goes away mid stream | 17:46 |
| *** gmaan_afk is now known as gmaan | 17:46 | |
| fungi | interesting, yeah i guess we pull from an outside mariadb container source and it updated? | 17:49 |
| fungi | timeouts make sense in that situation, i suppose any of our many database backup streams could get hit by the same | 17:49 |
| clarkb | yup I think we must've merged something that cause ansible to run on lists and it saw a mariadb image update and updated it | 17:53 |
| fungi | well, we were updating a lot of things earlier in the week, obviously | 17:54 |
| clarkb | exactly. I'm not surprised that mariadb updated. More surprised that mysqldump can't handle that | 18:04 |
| fungi | or the kernel... maybe the client side of the local network socket just hung when the server stopped listening? | 18:06 |
| clarkb | perhaps. the python side was reading fd 0 which I think is due to it being in a pipe from mysqldump? Maybe stringingthe process together like that is part of the problem | 18:07 |
| clarkb | we have decent alerting from when this goes wrong so I'm not too worried about it. We can probably just observe and see if any action needs to be taken | 18:07 |
| fungi | well, by "client" i meant myseldump itself. i assume it's connecting over loopback but maybe it's a unix socket/fifo instead? | 18:09 |
| fungi | looks like we did get a backup failure e-mail about the one you terminated too | 18:13 |
| clarkb | fungi: yup at ~17:44 UTC | 18:14 |
| clarkb | fungi: we don't specify a host value to mysql dump which causes the host to be localhost. And if you use localhost it will use the domain socket | 18:15 |
| clarkb | so that may be why we don't notice that things go aware | 18:15 |
| clarkb | s/aware/away/ | 18:15 |
| fungi | yeah, maybe that doesn't force close the fd | 18:15 |
| clarkb | since having the socket open will keep it around similar to having a normal file open and you delete the directory it is in recursively? | 18:15 |
| fungi | whereas network to loopback would probably have done tcp/rst once the service was down, if not a proper tcp/fin | 18:16 |
| fungi | during shutdown | 18:16 |
| clarkb | yup | 18:16 |
| clarkb | I wonder if we can update that mysqldump command to use -h 127.0.0.1 and have it just work | 18:17 |
| clarkb | mysql/mariadb differentiate auth based on auth method so that may not work as expected. But is somethign we could test via a held node in zuul or just manually | 18:17 |
| fungi | it might be marginally less efficient, but probably not in any way that matters for our case | 18:17 |
| clarkb | looking in system-config none of our current db backup scripts use -h at all so I think they are all using the default (I was hoping for one example to indicate this may just work) | 18:24 |
| fungi | it's super easy to test from the command line though | 18:24 |
| fungi | just direct stdout to /dev/null rather than piping through compression | 18:25 |
| fungi | unless you want to be able to check the contents of the dump i guess | 18:25 |
| fungi | launchpad is inconveniently down at the moment | 18:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DO NOT MERGE Test mysql connectivity via tcp instead of local socket https://review.opendev.org/c/opendev/system-config/+/973535 | 18:28 |
| fungi | looks like they moved from irc to matrix semi-recently. i'll have to update my client config | 18:29 |
| clarkb | fungi: oh ^ I did that because it is easy | 18:29 |
| clarkb | if that works I think it should work and we can test mysqldump itself to be sure. If it doesnt' work then we're into debugging | 18:29 |
| fungi | https://status.canonical.com/ says major outage for lp | 18:44 |
| clarkb | I think github had an issue today too. And yesterday verizon was down for your part of the world | 18:50 |
| clarkb | holidays are over and we're getting back to work and breaking things :) | 18:50 |
| fungi | but not moving fast, just breaking things | 18:51 |
| clarkb | fungi: my simple test ni 973535 did work. So the next step is probably updating one of the mysqldump scripts to do the same thing? | 19:02 |
| clarkb | either that or manually testing it first | 19:02 |
| clarkb | lets see if we have any low impact hosts to yolo with | 19:03 |
| clarkb | lodgeit? I'll propose a change to use -h 127.0.0.1 there | 19:03 |
| fungi | sgtm | 19:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Perform lodgeit mysqldump via tcp instead of local unix socket https://review.opendev.org/c/opendev/system-config/+/973540 | 19:12 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run all mysqldumps over tcp 127.0.0.1 instead of localhost socket https://review.opendev.org/c/opendev/system-config/+/973541 | 19:12 |
| clarkb | things look quiet I'm going to take advantage of that and the weather and go for a bike ride | 21:31 |
| clarkb | Looking ahead to tomorrow as long as things are still quiet I think I would like to try the gerrit java 21 cut over | 21:31 |
| clarkb | infra-root ^ just fyi and if you have reason to believe that is a bad idea please let me know | 21:31 |
| fungi | sounds good, i plan to be around | 21:37 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!