| -@gerrit:opendev.org- Dmitriy Rabotyagov proposed: [opendev/system-config] 1005081: Update mirrored Ceph versions https://review.opendev.org/c/opendev/system-config/+/1005081 | 08:19 | |
| @priteau:matrix.org | Hello. What is the max-job-timeout in OpenDev Zuul? We have a kayobe job which is hitting the 3-hour timeout (10800 which is the default for max-job-timeout in upstream Zuul) but could probably complete most times if it had a few minutes more. Would it work if I bump the job timeout to 3.5 hours? | 11:25 |
|---|---|---|
| @fungicide:matrix.org | Pierre Riteau: we don't set one, so use the default value of 10800 seconds (3 hours) https://zuul-ci.org/docs/zuul/latest/tenants.html#attr-tenant.max-job-timeout | 12:59 |
| @priteau:matrix.org | Thanks. Unless you would be willing to bump this, I think we will need to figure out ways to make this job faster | 13:06 |
| @fungicide:matrix.org | one of the main reasons we wouldn't want to increase it is that in order to gracefully restart zuul executors during our weekly upgrades we pause acceptance of new builds but have to wait for any it's running to complete before it can be safely stopped | 13:12 |
| @fungicide:matrix.org | so longer-running builds directly impact the speed at which our zuul cluster can upgrade itself | 13:13 |
| @priteau:matrix.org | Understood | 13:19 |
| @priteau:matrix.org | These jobs used to take closer to two hours but have drastically slowed down recently | 13:26 |
| @priteau:matrix.org | Any thoughts about what could have changed around August 27 on vexxhost-ca-ymq-1-main (which appears to be the only cloud picked for this job) that would have slowed down the builds by ~50%? | 13:31 |
| @fungicide:matrix.org | we've also seen reports of internal and external communication problems in that provider region | 13:32 |
| @fungicide:matrix.org | test nodes struggling to get content from our mirror server there or from the internet at large, and it's also where we have our gerrit server which has been experiencing some ipv6 communication issues from outside | 13:33 |
| @fungicide:matrix.org | however it's the only provider with the highest ram flavors we have for our larger test node labels, so if we stopped scheduling jobs to it then any relying on those labels would instantly fail with a node_error result instead | 13:35 |
| @priteau:matrix.org | Yes of course | 13:36 |
| @priteau:matrix.org | Strangely it's the ubuntu-noble job that became much slower, the rocky-10 equivalent is not affected in the same way | 13:37 |
| @fungicide:matrix.org | i wonder if the rocky jobs have a lot of tests disabled that the ubuntu ones don't? | 13:40 |
| @priteau:matrix.org | They should be the same, really. This is Kayobe (part of Kolla project) so maybe the Ubuntu kolla images are bigger and take longer to fetch | 13:41 |
| @fungicide:matrix.org | ah okay so not a devstack-tempest job or anything | 13:41 |
| @priteau:matrix.org | https://zuul.opendev.org/t/openstack/builds?job_name=kayobe-overcloud-upgrade-ubuntu-noble&project=openstack%2Fkayobe&branch=master&result=SUCCESS&skip=100 | 13:42 |
| @priteau:matrix.org | That's the second page of results, so older builds | 13:43 |
| @priteau:matrix.org | They all take around 2 hours, give or take | 13:43 |
| @priteau:matrix.org | Recent jobs are around 3 hours, or longer (not displayed here because I filter on Result = SUCESS): https://zuul.opendev.org/t/openstack/builds?job_name=kayobe-overcloud-upgrade-ubuntu-noble&project=openstack%2Fkayobe&branch=master&result=SUCCESS&skip=0 | 13:43 |
| @fungicide:matrix.org | in addition to a release job failure for openstack earlier today where a `git review` invocation hit a protocol-level error communicating with our gerrit server to push a change, i just now had to remove and re-add approval on another change to get zuul to notice and trigger the gate pipeline so i suspect zuul is missing events from gerrit communication problems | 15:55 |
| @jim:acmegating.com | fungi: which change? | 15:56 |
| @fungicide:matrix.org | https://review.opendev.org/1004496 | 15:56 |
| @fungicide:matrix.org | i also can't ping the gerrit server's ipv6 address from anywhere again, getting "Destination unreachable: Address unreachable" `from 2604:e100:1::1` | 15:57 |
| @fungicide:matrix.org | mnaser: any chance you can tell if the network issues there have resurfaced? it seemed to clear up earlier in the week after you forced something | 15:58 |
| @fungicide:matrix.org | oh, actually i just got it to start working again for now | 15:58 |
| @fungicide:matrix.org | since 2604:e100:1::1 was claiming not to know the route to the gerrit server, i did `ping 2604:e100:1::1` from a shell on the gerrit server and now i can reach it from outside again | 15:59 |
| @jim:acmegating.com | fungi: 2026-09-11 15:19:21,932 ERROR zuul.GerritEventProcessor: urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) | 16:01 |
| @fungicide:matrix.org | looks about like what i expected | 16:01 |
| @jim:acmegating.com | yeah, that was an http request though, not a problem with the ssh stream | 16:01 |
| @jim:acmegating.com | but still certainly looks like network shenanigans | 16:02 |
| @fungicide:matrix.org | the earlier job failure i mentioned was https://zuul.opendev.org/t/openstack/build/0b5b62d6019f460ea9f1ad3edca7e3e7 which terminated the `git review` command with `Received disconnect from 2604:e100:1:0:f816:3eff:fe31:926a port 29418:2: Protocol error or corrupt packet` | 16:02 |
| @jim:acmegating.com | looks very similar (despite different protocols / ports) | 16:03 |
| @jim:acmegating.com | wonder if it's a rst or fin, not from the app. | 16:04 |
| @mordred:waterwanders.com | in addition to the gerrit issues I see above - I just saw a POST_FAILURE from the opendev-buildset-registry job at the end of a buildset, with no logs for thejob, so I'm guessing the POST_FAILURE was a log upload failure | 18:09 |
| @mordred:waterwanders.com | https://zuul.opendev.org/t/opendev/build/a71164b6aad2497eba44db84efb1097c | 18:09 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!