| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/zuul-providers] 1004382: Make image build jobs non-voting in gate https://review.opendev.org/c/opendev/zuul-providers/+/1004382 | 06:09 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed: [opendev/zuul-providers] 1004382: Make image build jobs non-voting in gate https://review.opendev.org/c/opendev/zuul-providers/+/1004382 | 06:11 | |
| @harbott.osism.tech:regio.chat | fyi I've self-approved ^^ now to unblock the other pending changes. for reference, the latest periodic builds all passed, from a rough look the periodic jobs tend to have less errors than the gate jobs, too. https://zuul.opendev.org/t/opendev/buildset/8ff2e6c2d8c24c41b08da1b84940bd59 | 06:21 |
|---|---|---|
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [opendev/zuul-providers] 1004382: Make image build jobs non-voting in gate https://review.opendev.org/c/opendev/zuul-providers/+/1004382 | 07:32 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/zuul-providers] 989136: Drop haveged and urandom fiddling from images https://review.opendev.org/c/opendev/zuul-providers/+/989136 | 08:38 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Dr. Jens Harbott: [opendev/zuul-providers] 1000965: Update cached images https://review.opendev.org/c/opendev/zuul-providers/+/1000965 | 09:54 | |
| @priteau:matrix.org | Gerrit feels quite slow today, and I am regularly seeing "Server unavailable" in the web UI | 11:50 |
| @priteau:matrix.org | Is it under crawler DoS again? 😢 | 11:51 |
| @fungicide:matrix.org | Pierre Riteau: server load is nominal and apache has plenty of open worker slots available to process requests, it's webui is responding quickly for me | 12:17 |
| @fungicide:matrix.org | i don't see any signs to indicate that it could be slow, but maybe there are backbone/routing problems somewhere affecting communication between other parts of the internet | 12:18 |
| @fungicide:matrix.org | i can't reach the gerrit server at the moment, either over ipv4 or ipv6. i get destination unreachable over v6 and ping timeouts over v4. not sure if anyone else is having trouble with it. i'll check nova | 17:02 |
| @fungicide:matrix.org | it's in shutoff state, probably another oom on the hypervisor. i'll work on getting it booted | 17:03 |
| @fungicide:matrix.org | openstack server show indicates it was last updated at 2026-09-07T16:59:41Z so that's probably when it happened | 17:03 |
| @fungicide:matrix.org | (5 minutes ago) | 17:04 |
| @fungicide:matrix.org | it came back to active and i can ping it now | 17:04 |
| @tkajinam:matrix.org | it seems service is not yet up ? I receive 503 from https://review.opendev.org | 17:06 |
| @fungicide:matrix.org | i can ssh into it, starting the containers | 17:06 |
| @tkajinam:matrix.org | ah, ok | 17:06 |
| @fungicide:matrix.org | yeah, we don't have the containers set to start automatically at boot | 17:06 |
| @fungicide:matrix.org | it's working via ipv4 but still not reachable over ipv6 yet | 17:07 |
| @fungicide:matrix.org | it's got a global address assigned and a default route over v6 | 17:08 |
| @fungicide:matrix.org | and it can ping its v6 gateway | 17:09 |
| @fungicide:matrix.org | `2604:e100:1::1` is reporting "Destination unreachable: Address unreachable" so that's inside vexxhost already | 17:11 |
| @fungicide:matrix.org | my guess is something's broken with internal ipv6 routing for at least the hypervisor host the gerrit server is scheduled to | 17:11 |
| @fungicide:matrix.org | oh, though i'm able to ssh to it over ipv6, so the icmp6 echo requests are being rejected in a confusing way by the router there i guess | 17:13 |
| @mnaser:matrix.org | [@fungicide:matrix.org](https://matrix.to/#/@fungicide:matrix.org) where is the :1 IP coming from? We rely on RA for the next hop | 17:14 |
| @fungicide:matrix.org | mnaser: pinging review.opendev.org from outside (multiple places on the internet) over ipv6 i get "From 2604:e100:1::1 icmp_seq=51 Destination unreachable: Address unreachable" | 17:15 |
| @fungicide:matrix.org | but routing is apparently working for tcp6 | 17:16 |
| @mnaser:matrix.org | Interesting. What's the next hop on ipv6 for that system ? | 17:16 |
| @fungicide:matrix.org | `default via fe80::21c:73ff:fe00:2000 dev ens3 proto ra metric 100 expires 1753sec mtu 1500 hoplimit 64 pref medium` | 17:16 |
| @mnaser:matrix.org | Interesting .. just one next hop? | 17:17 |
| @fungicide:matrix.org | yes, not a pair | 17:17 |
| @fungicide:matrix.org | so maybe icmp6 inbound is getting flowed to the other side and it doesn't see the nova host | 17:18 |
| @mnaser:matrix.org | Checking.. | 17:19 |
| @fungicide:matrix.org | i suppose this could also explain Pierre Riteau's "slowness" observation earlier today, if his connections were trying over v6 first and getting flowed to the "bad" side, then retrying over v4 and establishing normally | 17:22 |
| @mnaser:matrix.org | [@fungicide:matrix.org](https://matrix.to/#/@fungicide:matrix.org) are you still seeing the issue ? I seek to be able to run external probes using global ping and they're all working | 17:24 |
| @fungicide:matrix.org | mnaser: now it's started working | 17:27 |
| @fungicide:matrix.org | from all the different locations i was testing at | 17:27 |
| @fungicide:matrix.org | dtill only one default v6 gateway on the gerrit server though | 17:28 |
| @fungicide:matrix.org | s/dtill/still/ | 17:28 |
| @fungicide:matrix.org | thanks for the help! | 17:29 |
| @mnaser:matrix.org | Weird. All I did was ping it from our spine.. maybe that unkludged it | 17:44 |
| @fungicide:matrix.org | possible that dislodged some neighbor discovery issue | 17:46 |
| @jim:acmegating.com | ping of life | 17:48 |
| @fungicide:matrix.org | muahahahaha | 17:48 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!