| fungi | 85795 failed the infra-prod-letsencrypt deploy job. i'm on airplane wifi for the moment so can't dig into the logs that easily | 00:03 |
|---|---|---|
| fungi | er, 885795 | 00:03 |
| fungi | https://zuul.opendev.org/t/openstack/build/47d4b3554e2f42768d661b5909c68106 | 00:04 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 00:10 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 00:10 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 00:12 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 00:12 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 00:19 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 00:19 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 00:21 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 00:21 |
| ianw | [Fri Jun 9 23:56:00 UTC 2023] The dns manual mode can not renew automatically, you must issue it again manually. You'd better use the other modes instead. | 00:35 |
| ianw | Unknown failure: 3 | 00:35 |
| ianw | i have dejavu about this | 00:35 |
| ianw | # less letsencrypt.yaml.log | grep failed=1 | wc -l | 00:36 |
| ianw | 25 | 00:36 |
| ianw | i feel like 3 might be the "everything is ok" status | 00:36 |
| fungi | that does indeed sound familiar | 00:41 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 00:42 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 00:42 |
| fungi | ianw: my irc history says see 2022-11-28 20:11:27 | 00:43 |
| ianw | heh yeah my diary for 2022-11-29 has "fix acme.sh breakage" :) | 00:44 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/865930 was the crux of it iirc | 00:44 |
| fungi | and https://github.com/acmesh-official/acme.sh/issues/4416 | 00:46 |
| ianw | given everyone's ability to really debug at this point in time, i'm thinking a revert might be the best option | 00:47 |
| fungi | wfm. i can push it if you want to review, or vice versa | 00:47 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Revert "Update acme.sh 3.0.6" https://review.opendev.org/c/opendev/system-config/+/885807 | 00:50 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Revert "Update acme.sh 3.0.6" https://review.opendev.org/c/opendev/system-config/+/885808 | 00:50 |
| fungi | d'oh | 00:50 |
| fungi | i'll abandon mine | 00:50 |
| fungi | thanks ianw! | 00:52 |
| fungi | clarkb: ^ not urgent, but heads up when you're around | 00:52 |
| Clark[m] | Ya that is unfortunate | 00:52 |
| fungi | i've already approved the revert | 00:52 |
| ianw | mea culpa i should have left better breadcrumbs on that pin since we had an idea there was an isue | 00:54 |
| fungi | no biggie | 00:56 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 01:06 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 01:06 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 01:36 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 01:36 |
| opendevreview | Merged opendev/system-config master: Revert "Update acme.sh 3.0.6" https://review.opendev.org/c/opendev/system-config/+/885807 | 03:32 |
| fungi | looks like the deploy on the revert ran fine | 05:45 |
| fungi | i don't seem to be able to reach ipv6 addresses in vexxhost's sjc1 region (for example our mirror.sjc1.vexxhost) | 07:37 |
| fungi | i can get to the v6 addy of mirror.ca-ymq-1.vexxhost just fine though | 07:37 |
| ianw | 64 bytes from 2604:e100:3:0:f816:3eff:fee3:150 (2604:e100:3:0:f816:3eff:fee3:150): icmp_seq=1 ttl=53 time=173 ms | 07:38 |
| ianw | ok from australia, if that helps :) | 07:38 |
| fungi | yeah, i think it's a routing issue in zayo again | 07:41 |
| fungi | traceroute seems to die in or at the border of zayo's atlanta pop | 07:41 |
| fungi | i can reach it from the rackspace mirrors, for example | 07:43 |
| fungi | so seems like a backbone/transit or selective bgp announcement filtering issue | 07:44 |
| opendevreview | Radosław Piliszek proposed zuul/zuul-jobs master: Add `collect-kubernetes-pods-logs` role https://review.opendev.org/c/zuul/zuul-jobs/+/885773 | 11:11 |
| opendevreview | Radosław Piliszek proposed zuul/zuul-jobs master: Add `collect-kubernetes-pods-logs` role https://review.opendev.org/c/zuul/zuul-jobs/+/885773 | 11:29 |
| opendevreview | Radosław Piliszek proposed zuul/zuul-jobs master: Add `collect-kubernetes-pods-logs` role https://review.opendev.org/c/zuul/zuul-jobs/+/885773 | 12:44 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!