| opendevreview | chandan kumar proposed openstack/project-config master: Add stable branch IRC notifications for Cyborg https://review.opendev.org/c/openstack/project-config/+/996426 | 08:16 |
|---|---|---|
| sean-k-mooney | are there currently knwo issue with the ovh mirrors | 12:02 |
| sean-k-mooney | WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='mirror.gra1.ovh.opendev.org', port=443): Read timed out. (read timeout=60.0)")': /pypi/simple/netaddr/ | 12:02 |
| sean-k-mooney | i have noticed a number of job failure on ovh due to pipe package downloads although not a specific package | 12:02 |
| frickler | that's not a mirror really, but a caching proxy towards pypi, so likely the issues are within the upstream CDN. we also saw this for other regions | 12:04 |
| sean-k-mooney | ya so there were 3000+ failure today | 12:06 |
| sean-k-mooney | fo ovh alone | 12:06 |
| sean-k-mooney | https://tinyurl.com/2chcxd6v | 12:07 |
| sean-k-mooney | frickler: i assome we cant really do much to work around this for now | 12:09 |
| sean-k-mooney | just recheck and hope it works its self out? | 12:09 |
| frickler | well if it really is only affecting gra1, we could consider shutting down that region temporarily. best discuss that in #opendev though | 12:12 |
| sean-k-mooney | ack i can swap client but maybe we coudl just disabel the pipi mirror in that region instead? | 12:13 |
| frickler | if it doesn't work via the mirror, I don't have much hope that a direct access will work better | 12:14 |
| sean-k-mooney | well if the issue is the proxy | 12:16 |
| sean-k-mooney | then direct access would work right | 12:16 |
| sean-k-mooney | if we are seing that in outher region then your right it wont help | 12:16 |
| sean-k-mooney | i have only noticed this in ovh on the hand full of jobs i noticed | 12:16 |
| frickler | hmm, looks like the mirror vm might be having more general issues | 12:18 |
| sean-k-mooney | im wondieringif its a storage issue | 12:19 |
| sean-k-mooney | these caches typicly rely on local storage and i have seen issues like that when said local storage is sad | 12:20 |
| sean-k-mooney | but it coudl be something else | 12:21 |
| sean-k-mooney | im not sure why they are usign as the pypi proxy/cache | 12:21 |
| fungi | i would try to help look into it, but my home broadband connection is down for the past several hours somewhere upstream of me so i'm limping along on a phone tether at the moment and struggling to connect to stuff as a result | 12:35 |
| fungi | check df on the mirror vm to see if the fs is full first, of course, then dmesg to see if there are write errors reported by the kernel or something | 12:36 |
| fungi | assuming you can ssh into it at all, otherwise check the nova console log and openstack server show | 12:37 |
| fungi | never mind, i had yet to notice that the discussion moved to matrix, continuing there | 12:42 |
| sean-k-mooney | :) | 12:42 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!