| @harbott.osism.tech:regio.chat | I'm still seeing regular issues with DNS resolution for opendev.org hosts. part of the equation seem to be connectivity issues towards vexxhost, but I'm also wondering whether we could increase the TTL in particular for the zone and maybe the ns hosts at least. currently everything seems to be at 3600 (1h), I would suggest 1d instead. if there isn't strong opposition to that, I'll prepare a patch. see https://paste.opendev.org/show/blqJsJOKsLy0htINQdtS/ for an example. not sure if someone could also ask the registrar about issues at their side potentially? I'm assuming the (now linux-) foundation is holder of the contract for the zone? | 08:30 |
|---|---|---|
| @mnasiadka:matrix.org | Well, we have two nameservers - one in DFW Rax and another in Vexxhost - is that a possibility that both have some networking issues? | 09:24 |
| @fungicide:matrix.org | looks like ipv6 is currently broken for ns04 (in vexxhost) | 12:59 |
| @fungicide:matrix.org | `host -t a review03.opendev.org 2604:e100:1:0:f816:3eff:feff:bd1c` | 12:59 |
| @fungicide:matrix.org | `;; communications error to 2604:e100:1:0:f816:3eff:feff:bd1c#53: host unreachable` | 13:00 |
| @fungicide:matrix.org | i can't `ssh -6` to it either | 13:00 |
| @fungicide:matrix.org | `ssh: connect to host ns04.opendev.org port 22: No route to host` | 13:00 |
| @fungicide:matrix.org | and now suddenly it's working again, but i briefly captured an `ip ad sh ens3` that only listed the linklocal address bound | 13:03 |
| @fungicide:matrix.org | https://paste.opendev.org/show/bw21YKiFxnqn24IapLmm/ | 13:05 |
| @fungicide:matrix.org | those were just a few seconds apart | 13:06 |
| @fungicide:matrix.org | `604786sec` for the expiry there is just 14 seconds shy of 604800, i.e. 7 days | 13:08 |
| -@gerrit:opendev.org- Benjamin Schanzel proposed: [zuul/zuul-jobs] 986645: FIX(roles/ensure-javascript-packages): Install version exact https://review.opendev.org/c/zuul/zuul-jobs/+/986645 | 13:09 | |
| @fungicide:matrix.org | suggesting that it had just refreshed its global address moments after i checked it the first time | 13:09 |
| @fungicide:matrix.org | not sure what would ave triggered that, since my only actions on the server were unprivileged (basically just examining the state of the interface) | 13:10 |
| @clarkb:matrix.org | fungi: that sounds like the issue we've mitigated with the yaml network config on some other hosts in the past | 13:26 |
| @clarkb:matrix.org | to statically configure the address to prevent it from flapping | 13:27 |
| @clarkb:matrix.org | if we do change the ttl to one day we will need to be better about reducing ttls before replacing servers. Not impossible but extra overhead. Also it won't fix the issue if the problem is with ipv6 connectivity it will just make it happen less frequently | 13:27 |
| @jim:acmegating.com | maybe we can try the non-ttl option first | 13:32 |
| @clarkb:matrix.org | https://opendev.org/opendev/system-config/src/commit/fa64689c32218ef78a4496ad30813b2029dc76c7/playbooks/service-review.yaml#L8-L44 | 13:34 |
| @clarkb:matrix.org | this is where we had it set up for the old gerrit server. I left it in that commented out state when we switched to the new server as it wasn't clear if the new one would have the same problem. And as far as Ican tell it hasn't. But porting that to ns04 should be straightforward if someone wants to gather the network info from the host and create the netplan config | 13:35 |
| @jim:acmegating.com | I'm pausing the zuul trigger event queues in order to perform some zuul maintenance | 14:01 |
| @fungicide:matrix.org | noted | 14:01 |
| @scott.little:matrix.org | zuul issues this morning ? | 15:31 |
| @clarkb:matrix.org | we're in the middle of updating zuul and it will queue up work in the meantime | 15:33 |
| @clarkb:matrix.org | the status page should give you a little banner notification of this | 15:34 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 986695: Upload-logs: skip symlinks https://review.opendev.org/c/zuul/zuul-jobs/+/986695 | 16:39 | |
| @fungicide:matrix.org | scott.little: and it's unpaused again | 17:23 |
| -@gerrit:opendev.org- Zuul merged on behalf of Monty Taylor https://matrix.to/#/@mordred:inaugust.com: [opendev/project-config] 985898: Add vouched pipeline to OpenDev tenant https://review.opendev.org/c/opendev/project-config/+/985898 | 17:24 | |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!