| *** mazzy50988129295808594943 is now known as mazzy5098812929580859494 | 07:25 | |
| mnaser | infra-root: +2 on https://review.opendev.org/c/opendev/base-jobs/+/843717/ to fix job logs for now | 07:36 |
|---|---|---|
| mnaser | ftr, it looks like CORS headers are missing | 07:36 |
| opendevreview | Merged opendev/base-jobs master: Temporarily stop uploading logs to Rackspace https://review.opendev.org/c/opendev/base-jobs/+/843717 | 08:32 |
| frickler | fyi epel8 mirroring is still broken https://paste.opendev.org/show/buFteHaLXtz7vLoKyEvj/ , breaking kolla jobs | 09:15 |
| jrosser_ | i think we see the same epel trouble on OSA as well | 11:44 |
| yoctozepto | frickler: hmm, but the used mirror seems to be fine now; the checksums agree; need to restart the mirroring mayhaps? | 15:36 |
| yoctozepto | or maybe the issue is with the rsync view of that mirror? | 15:36 |
| yoctozepto | I checked http | 15:36 |
| frickler | yoctozepto: indeed, the second run today succeeded. an attempt is made every 6 hours, no restart needed. let's see what the kolla builds think about it | 15:42 |
| fungi | yoctozepto: also you can see a timestamp at the root of each mirror indicating when we pulled it, so you can tell if it's updated since a failure you saw | 16:10 |
| fungi | a timestamp file, specifically, e.g.: https://mirror.ca-ymq-1.vexxhost.opendev.org/epel/timestamp.txt | 16:12 |
| fungi | and what i mean by "each mirror" is each subtree we're building via rsync, not each server. it should be consistent across all our mirror servers because it's coming from a shared network filesystem | 16:13 |
| yoctozepto | fungi: ack | 16:16 |
| Clark[m] | I'm not looking at it today (and unsure when I'll have time to) but one thing to check with the rax logs is if older uploads have the cors headers. If so that would likely point to some change on our side. If not then to their side | 16:50 |
| Clark[m] | And if our side maybe we just downgrade openstacksdk and see if it is happier | 16:50 |
| fungi | Clark[m]: they do, it was just uploads circa friday which seem to exhibit this | 16:58 |
| fungi | er, do have cors headers served i mean | 16:58 |
| fungi | and yeah, i figure next order of business after holidays is we can downgrade openstacksdk and then recheck some dnm change which reparents a job to base-test and see what happens when its logs end up in rax swift | 16:59 |
| fungi | though wondering how to experimentally downgrade openstacksdk on the executors. may be easier to come up with a manual reproducer | 17:00 |
| Clark[m] | You can run the upload playbook library stuff directly it has an if name is main block | 17:01 |
| Clark[m] | We can also downgrade the library directly in the running containers maybe | 17:02 |
| Clark[m] | But ya that would seem to point to the sdk update pulled in by the restart of the executors | 17:02 |
| Clark[m] | Other than that I couldn't find anything that had changed recently that should affect the logs | 17:03 |
| fungi | yeah, makes sense | 17:41 |
| fungi | anyway, try to holiday weekend ;) | 17:41 |
| fungi | this'll keep | 17:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!