| opendevreview | Amit Uniyal proposed openstack/whitebox-tempest-plugin master: Adds libvirt watchdog https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/921092 | 06:20 |
|---|---|---|
| *** elodilles_ooo is now known as elodilles | 06:32 | |
| opendevreview | Amit Uniyal proposed openstack/whitebox-tempest-plugin master: extend pre-commit with basic python linting https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/916533 | 12:12 |
| opendevreview | Amit Uniyal proposed openstack/whitebox-tempest-plugin master: add bashate to pre-commit https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/916534 | 12:12 |
| dansmith | hmm, is the python38-fips jobs known broken? seems like consistent failure due to missing mirror data: https://zuul.opendev.org/t/openstack/build/998d462e08c346f0b86a7dbfc004114b | 17:55 |
| clarkb | its likely that the centos mirrors were not updated in a safe order. This happens occasionally. I wuold expect that to affect non fips jobs too | 17:56 |
| dansmith | okay not sure there are any other non-fips centos jobs in this repo, everything else passed | 17:57 |
| dansmith | will it resolve itself or do we need to do something? | 17:58 |
| clarkb | typically it resolves itself once the upstream mirror is in a proper state that we can sync from | 17:59 |
| dansmith | to be clear, it's rax's centos mirror that is the problem yeah? | 17:59 |
| clarkb | this is/was an issue that I think people hoped would go away by having us pull directly from the main mirrors rather than the second level mirrors but it seems the problems originate there | 17:59 |
| clarkb | dansmith: no its centos' main mirror aiui | 17:59 |
| clarkb | what happens is they update the repo in an order that isn't consistent then we sync from that inconsistent state. | 18:00 |
| dansmith | oh I see, okay I see, I saw mirror-int.dft.rax and didn't read past that to opendev.org | 18:01 |
| clarkb | dansmith: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L54 this is what we're syncing from which is the main repo I think | 18:01 |
| clarkb | we got some exception made so that we could sync from them instead of second level repos (since we aren't a true public repo mirror) | 18:01 |
| clarkb | the idea was that maybe the second level mirrors were the source of the problem, but I think we now know the problem is happenign directly as the source unfortunately | 18:01 |
| clarkb | this is a solvable issue upstream, they just have to ensure they add new packages first, then new indexes, then remove old indexes, then remove old packages in that order aiui | 18:02 |
| clarkb | in this case we're looking for an index that wasn't added so they got things out of order | 18:02 |
| dansmith | ack, and what triggers our sync? the clock or something else? | 18:02 |
| clarkb | ya it is just a cron job running every 4 hours I think | 18:03 |
| dansmith | okay | 18:03 |
| dansmith | clarkb: should I be able to see the last time it ran in zuul jobs or something? | 18:14 |
| clarkb | dansmith: https://mirror.dfw.rax.opendev.org/centos/timestamp.txt if you navigate to the root of each of the mirror repos we put a timestamp there | 18:16 |
| clarkb | there are also logs somewhere I'll dig up | 18:16 |
| dansmith | aha, okay, so that looks like about 30 minutes until it runs again | 18:16 |
| dansmith | if it's every 4h | 18:16 |
| dansmith | er, no | 18:17 |
| dansmith | that timestamp is almost 6h old | 18:17 |
| clarkb | https://mirror.dfw.rax.opendev.org/logs/rsync-mirrors/centos.log | 18:18 |
| clarkb | ya its every six hours | 18:18 |
| clarkb | so it should be running nowish | 18:18 |
| dansmith | ah, cool then | 18:18 |
| clarkb | oh no in about 30 minutes but ya soon | 18:18 |
| dansmith | yeah cool | 18:18 |
| clarkb | dansmith: for the rax mirrors we use a -int dns record for the private rax network address as throughput is much better within the cloud that way. But you can drop the -int and hit the public IPs and should see all the same stuff | 18:19 |
| clarkb | and then under logs/ we have various syncing logs for each of the things in afs | 18:20 |
| dansmith | ack | 18:20 |
| clarkb | dansmith: on the server I think I'm seeing hte log file shows an 18:43 update but my browser doesn't see that. Maybe a cache problem | 18:58 |
| clarkb | hrm I'm not seeing any updated files though | 18:59 |
| clarkb | so maybe upstream is still out of sync | 18:59 |
| dansmith | yeah the log is not updated even with a force refresh | 18:59 |
| clarkb | even if it were sent 102 bytes received 63 bytes 110.00 bytes/sec is all that was transfered implying upstream is still a problem? | 19:00 |
| clarkb | we may need to compare manually if this persists | 19:00 |
| dansmith | okay I guess I don't understand why the timestamp/log isn't updated though | 19:07 |
| clarkb | dansmith: it may be due to how things get copied to afs the file I'm looking at is on nromal disk. Maybe we lag the copy to afs for some reason | 19:08 |
| clarkb | dansmith: oh ha! https://mirror.dfw.rax.opendev.org/centos/8-stream/ that is what | 19:11 |
| clarkb | centos 8-stream is dead | 19:11 |
| clarkb | they pulled the mirror entirely.... | 19:11 |
| clarkb | I think that means it is actually time to start removing centos 8 stream jobs and the nodes and mirrors and so on | 19:11 |
| dansmith | sheesh | 19:15 |
| dansmith | yeah, okay I'll slide the job removal in front of my backports | 19:15 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!