| @arnaudm:matrix.org | hey team, do you know who is managing https://www.stackalytics.io/ ? It seems the barbican metrics are not collected correcly | 10:37 |
|---|---|---|
| @mnasiadka:matrix.org | I think the tool that OpenInfra and Linux Foundation are exposing for this case is https://insights.linuxfoundation.org - stackalytics has been sort of on and off and incosistently managed outside of OpenDev/OpenInfra | 11:07 |
| @arnaudm:matrix.org | ack thanks mnasiadka | 12:06 |
| @fungicide:matrix.org | huh, gerrit seems to maybe be down, i'm having trouble reaching it | 12:24 |
| @fungicide:matrix.org | will check the nova status | 12:24 |
| @fungicide:matrix.org | no, looks like maybe i'm having ipv6 problems, i can reach it over ipv4 | 12:25 |
| @fungicide:matrix.org | yeah, i can't reach v6 addresses in other networks either | 12:27 |
| @fungicide:matrix.org | looks like my isp's v6 routing has broken somewhere past the first hop, since i can get responses from my firewall internally and it reports its v6 default gateway is reachable | 12:39 |
| @fungicide:matrix.org | corvus: please let me know when you're done editing the passwords list | 13:15 |
| @jim:acmegating.com | fungi: done | 13:23 |
| @fungicide:matrix.org | thanks corvus! | 13:29 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 1000395: Document domain creation for Mailman https://review.opendev.org/c/opendev/system-config/+/1000395 | 14:04 | |
| @mnasiadka:matrix.org | https://review.opendev.org/c/opendev/system-config/+/999795 needs a second +2 and we'll get node-exporter on the prometheus hsot | 16:07 |
| @mnasiadka:matrix.org | * https://review.opendev.org/c/opendev/system-config/+/999795 needs a second +2 and we'll get node-exporter on the prometheus host | 16:08 |
| @fungicide:matrix.org | #status log Manually cleaned up cronjobs for the retired backup02.ca-ymq-1.vexxhost Borg target from all production servers | 16:15 |
| @status:opendev.org | @fungicide:matrix.org: finished logging | 16:15 |
| @clarkb:matrix.org | fungi: thank you | 16:37 |
| @clarkb:matrix.org | mnasiadka: ya maybe fungi will be able to review that change? its on the meeting agenda to followup on that too later today | 16:38 |
| @clarkb:matrix.org | also apparently its the beginning of august when service coordinator election stuff should happen and not the end... I'll work on a last minute draft proposal that we can discuss in the meeting really quickly before making something official later today | 16:39 |
| @clarkb:matrix.org | Reminder that the weekly team meeting will be in the new #opendev-meeting:opendev.org room today and not the old #opendev-meeting channel on OFTC | 18:26 |
| @clarkb:matrix.org | fungi: backup03 is the failing backup target | 19:16 |
| @clarkb:matrix.org | fungi: you can see that in the backup03 log file on review03. Looks like a permissions error on the target side. Maybe due to the compaction? | 19:17 |
| @fungicide:matrix.org | ah, possible something with the compaction triggered those errors to start | 19:17 |
| @clarkb:matrix.org | its possible we need to prune and compact as the user owning the backups rather than as root | 19:17 |
| @fungicide:matrix.org | yeah, entirely possible. i think that was the one i did by hand before i edited the script to do the others | 19:17 |
| @fungicide:matrix.org | maybe umask differed | 19:17 |
| @clarkb:matrix.org | so probably just need to cross check the perms and ownership of the data files on some other backup target then correct review03's data files | 19:19 |
| @fungicide:matrix.org | yeah, i can look shortly. between meetings and discovering my isp has broken their ipv6 routing while i was sleeping, my day hasn't been all that productive | 19:20 |
| @fungicide:matrix.org | okay, should be fixed now. indeed some of the files were root-owned because i missed that the backup operations are run in the context of the per-server users | 19:32 |
| @fungicide:matrix.org | i did a `chown -R borg-review03:borg-review03 /opt/backups/borg-review03/backup` on backup03.ca-ymq-1.vexxhost | 19:32 |
| @fungicide:matrix.org | the others should be fine because i edited the command into the prune-borg-backups script to do them, which runs commands as the correct user for each repository | 19:34 |
| @fungicide:matrix.org | as for the `Found errors in Matrix room configuration!` failure on 1000376, i can reproduce it locally where it's also uselessly silent, but at least i can try to step through the checker and find where it goes slantways | 19:43 |
| @clarkb:matrix.org | fungi: corvus its the channel collision | 19:49 |
| @clarkb:matrix.org | since limnoria is still in #opendev-meeting in irc | 19:49 |
| @clarkb:matrix.org | I think thats fine since we know not to run a meeting there for now. But we can fix that up and get the change merged | 19:49 |
| @fungicide:matrix.org | yeah, i just spotted that as well | 19:49 |
| @fungicide:matrix.org | and had previously been considering checking whether we had a change to drop the bot from that channel | 19:49 |
| @fungicide:matrix.org | i'll propose a change that adds some actual output for that check ;) | 19:50 |
| @jim:acmegating.com | thanks, i'll fixup the original change and leave the linter output to fungi | 19:50 |
| @jim:acmegating.com | though... | 19:51 |
| @jim:acmegating.com | i mean, there isn't a conflict | 19:51 |
| @jim:acmegating.com | so maybe the change should be to not make that a conflict :) | 19:51 |
| @clarkb:matrix.org | do they write to the same log files? | 19:51 |
| @clarkb:matrix.org | I think that may be the concern | 19:51 |
| @clarkb:matrix.org | with the logging I mean | 19:51 |
| @jim:acmegating.com | yes, but that's set explicitly; so a check for colliding log files would make sense | 19:52 |
| @jim:acmegating.com | (it's not because they have the same name) | 19:52 |
| @clarkb:matrix.org | yes I think the check is specifically on the log paths | 19:52 |
| @jim:acmegating.com | ok if that's what it's checking, then just, like outputting the error would be good then :) | 19:53 |
| @clarkb:matrix.org | anyway the bot should leave matrix in ~ 15 minutes and then we can figure it out from there | 19:53 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [opendev/system-config] 1000376: Add matrix-eavesdrop to new #opendev-meeting room https://review.opendev.org/c/opendev/system-config/+/1000376 | 19:54 | |
| @fungicide:matrix.org | because of my broken ipv6 i have to wait for git to time out and fall back to v4, git-review doesn't seem to handle the `-4` option like some git subcommands | 19:55 |
| @fungicide:matrix.org | so eventually i'll have the change for that showing up, but i lost the race to have it fail stacked on top of the broken state of 1000376,1 | 19:55 |
| @clarkb:matrix.org | how concerned should we be about corrupting the log for the opendev-meeting channel/room? I think its not a big deal | 19:59 |
| @clarkb:matrix.org | worst case we have incomplete data from today and we can call those growing pains | 19:59 |
| @fungicide:matrix.org | i'm sure it's not a big deal | 19:59 |
| @clarkb:matrix.org | cool we will need to manually restart limnoria to pick up the channel drop there | 20:00 |
| @clarkb:matrix.org | I'm going to grab lunch now. Then will send an email about service coordinator elections. Then can help with the bot fixup/cleanup stuff | 20:00 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/system-config] 1000559: Meetbot linting: Add output for room collisions https://review.opendev.org/c/opendev/system-config/+/1000559 | 20:01 | |
| @fungicide:matrix.org | sgtm, thanks! | 20:11 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [opendev/system-config] 1000376: Add matrix-eavesdrop to new #opendev-meeting room https://review.opendev.org/c/opendev/system-config/+/1000376 | 20:51 | |
| @clarkb:matrix.org | I just sent the election email. Hopefully I got all the dates right | 21:00 |
| @clarkb:matrix.org | fungi: re ipv6 I have a small sysctl script to disable and enable ipv6 globally so that my wireguard traffic doesn't go over ipv6 and skip the tunnel. You could do something like that until your ISP corrects the upstream issue | 21:01 |
| @fungicide:matrix.org | yeah, i've been hoping it gets corrected soon, but with it having been this way for most of a day already my hopes are dwindling. i'll take more aggressive measures if it's ongoing tomorrow | 21:02 |
| @clarkb:matrix.org | 1000376's deploy failed in the base job on a lock contention issue with apt/dpkg on mirror02.regionone.osuosl.opendev.org | 21:05 |
| @clarkb:matrix.org | thats "ok" the hourly job will deploy it momentarily | 21:05 |
| @clarkb:matrix.org | I don't think we need to reenqueue anything | 21:05 |
| @clarkb:matrix.org | looks like apt-get has been running since 11:43 UTC | 21:06 |
| @clarkb:matrix.org | fungi: should I try to kill that process then run a manual apt-get update (the command that is stuck) to see if it succeeds? | 21:06 |
| @clarkb:matrix.org | also both bot configs have udpated at this point. Any objections to me restarting limnoria now? | 21:10 |
| @clarkb:matrix.org | the osuosl mirror's apt-get process was killed then I reran apt-get update by hand and that succeeded | 21:11 |
| @clarkb:matrix.org | ok no objections I am restarting the limnoria bot now | 21:16 |
| @fungicide:matrix.org | looking | 21:43 |
| @fungicide:matrix.org | Clark: i don't see apt-get running, maybe it finished? | 21:44 |
| @clarkb:matrix.org | fungi: I killed it | 22:08 |
| @clarkb:matrix.org | then reran apt-get update by hand to make sure it would exit in the current state of things | 22:08 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!