Sunday, 2022-05-29

*** mazzy50988129295808594943 is now known as mazzy509881292958085949407:25
mnaserinfra-root: +2 on to fix job logs for now07:36
mnaserftr, it looks like CORS headers are missing07:36
opendevreviewMerged opendev/base-jobs master: Temporarily stop uploading logs to Rackspace
fricklerfyi epel8 mirroring is still broken , breaking kolla jobs09:15
jrosser_i think we see the same epel trouble on OSA as well11:44
yoctozeptofrickler: hmm, but the used mirror seems to be fine now; the checksums agree; need to restart the mirroring mayhaps?15:36
yoctozeptoor maybe the issue is with the rsync view of that mirror?15:36
yoctozeptoI checked http15:36
frickleryoctozepto: indeed, the second run today succeeded. an attempt is made every 6 hours, no restart needed. let's see what the kolla builds think about it15:42
fungiyoctozepto: also you can see a timestamp at the root of each mirror indicating when we pulled it, so you can tell if it's updated since a failure you saw16:10
fungia timestamp file, specifically, e.g.:
fungiand what i mean by "each mirror" is each subtree we're building via rsync, not each server. it should be consistent across all our mirror servers because it's coming from a shared network filesystem16:13
yoctozeptofungi: ack16:16
Clark[m]I'm not looking at it today (and unsure when I'll have time to) but one thing to check with the rax logs is if older uploads have the cors headers. If so that would likely point to some change on our side. If not then to their side16:50
Clark[m]And if our side maybe we just downgrade openstacksdk and see if it is happier16:50
fungiClark[m]: they do, it was just uploads circa friday which seem to exhibit this16:58
fungier, do have cors headers served i mean16:58
fungiand yeah, i figure next order of business after holidays is we can downgrade openstacksdk and then recheck some dnm change which reparents a job to base-test and see what happens when its logs end up in rax swift16:59
fungithough wondering how to experimentally downgrade openstacksdk on the executors. may be easier to come up with a manual reproducer17:00
Clark[m]You can run the upload playbook library stuff directly it has an if name is main block17:01
Clark[m]We can also downgrade the library directly in the running containers maybe17:02
Clark[m]But ya that would seem to point to the sdk update pulled in by the restart of the executors17:02
Clark[m]Other than that I couldn't find anything that had changed recently that should affect the logs17:03
fungiyeah, makes sense17:41
fungianyway, try to holiday weekend ;)17:41
fungithis'll keep17:42

Generated by 2.17.3 by Marius Gedminas - find it at!