Thursday, 2021-09-30

stevebakerhey all, there seems to be a problem with this fedora mirror https://zuul.opendev.org/t/openstack/build/a49e22e39bd94e098836a6dd46ad386c/log/logs/fedora_build-succeeds.FAIL.log#102102:43
stevebakerhttp://mirror-int.iad.rax.opendev.org02:43
fungistevebaker: i've observed the same, we've pulled from the source mirror at least several times since i saw that error, so if it persists much longer we might have to switch to syncing from somewhere else02:46
stevebakerfungi: I get a "cannot resolve host" on that name02:48
stevebakerhmm, but a 404 in the logs02:48
fungithe mirror-int addresses are rfc 1918, drop the -int part to reach the public address on those02:49
fungistevebaker: according to  https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/fedora-mirror-update#L39 we're pulling that state from mirrors.mit.edu, if there's a different rsync mirror we should be pulling from instead of that one we can switch02:50
fungii was hoping they'd have fixed it by now02:50
stevebakerfungi: I see, thanks. I'll keep an eye on it02:51
fungikeep in mind that for our distro package mirrors they're all backed by a shared network filesystem, so if you see an error like that on one of our mirrors you'll generally see it on all of them02:52
stevebakerok02:52
fungiand we've been successfully pulling from the source mirror at mirrors.mit.edu every couple of hours, so the broken state seems to be getting copied from there02:53
fungino idea whether wherever mit pulls from is similarly broken02:53
fungiif so, we might get the same errors even if we switched to a different rsync source02:54
stevebakeryeah the 404s are coming from this mostly empty dir http://mirrors.mit.edu/fedora/linux/updates/34/Everything/x86_64/repodata/02:54
stevebakerfungi: curling this from your rsync host might give the best mirror potential hosts https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-34&arch=x86_6402:57
fungii'll take a look02:59
stevebakeralso fedora 34 is old, we should probably update that dib job02:59
fungimmm, the mirrorlist doesn't seem to indicate which ones have rsync support03:00
fungijust http and https03:00
stevebakerthis may be some fedora-34 eol thing. This looks fine03:01
stevebakerhttp://mirrors.mit.edu/fedora/linux/updates/36/Everything/x86_64/repodata/03:01
stevebakerianw: my fix for the dib gate is blocked by another dib CI issue ^^ https://review.opendev.org/c/openstack/diskimage-builder/+/81104803:02
fungihttp://download-ib01.fedoraproject.org/pub/fedora/linux//releases/34/Everything/x86_64/os/repodata/ seems to still have content, so i don't expect it's been removed from the primaries03:03
fungialso the mit mirror seems to have at least some indices for older releases like f33 and f3203:04
fungilooks like i switched to the mit mirror a year ago with https://review.opendev.org/746587 because pubmirror[12].math.uh.edu ceased updating their (at the time f31) copy03:06
fungiwe could switch back to it, http://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/34/Everything/x86_64/repodata/ has indices03:07
fungii've pushed up a revert as https://review.opendev.org/811832 but will try it manually first03:10
fungistevebaker: ianw is on vacation for a few more days, i think03:11
stevebakerok03:11
fungianyway, i've got a refresh running with the revert manually applied, no idea how long it may take to complete though, switching source mirrors often results in having to retransmit the entire set03:15
fungior at least large swaths of it03:16
*** ykarel_ is now known as ykarel06:07
*** ysandeep|out is now known as ysandeep06:36
*** jpodivin is now known as jpodivin|ruck06:36
*** jpena|off is now known as jpena07:00
*** elodilles_pto is now known as elodilles07:12
*** ykarel is now known as ykarel|lunch09:13
*** ykarel|lunch is now known as ykarel10:05
*** jcapitao is now known as jcapitao_lunch10:39
*** rlandy is now known as rlandy|ruck10:39
*** jpena is now known as jpena|lunch11:25
*** tobberydberg_ is now known as tobberydberg11:51
*** jcapitao_lunch is now known as jcapitao12:09
*** jpena|lunch is now known as jpena12:16
*** bhagyashris is now known as bhagyashris|rover12:43
mnaserim confused14:00
mnaseri dont think fedora 34 is eol? isn't it the latest release? lol14:01
fungimnaser: yeah, not sure what the cause was for mit's mirror being incomplete so we're in the process of syncing from uh14:16
fungiit's settling now and getting ready to update the public volume replicas starting in roughly 10 minutes14:17
fungibut given it ran for most of the night transferring data, i expect the volume release to take hours as well14:18
*** bhagyashris_ is now known as bhagyashris|rover14:19
clarkbmnaser: fedora 35 is just releasing now aiui. So ya 34 is the current release unless you do betas14:26
fungihttps://static.opendev.org/mirror/fedora/timestamp.txt14:28
fungii think maybe it's updated now, i'll recheck one of my failing changes14:29
fungii'm going to run a second manual sync once this one finishes though, just to make sure we're completely caught up14:29
clarkb++14:30
fungilooks like it's using slightly more of the volume after the source switch, but not significantly, and there's still plenty of headroom there: https://grafana.opendev.org/d/T5zTt6PGk/afs?viewPanel=29&orgId=1&from=now-24h&to=now14:34
fungialso the afs02.dfw replica is still releasing, and will be for a while, but looks like afs01.dfw is what the mirrors are serving and it released quickly since it's local14:35
fungistevebaker: okay, as of moments ago i've got passing f34 jobs again14:53
*** promethe- is now known as prometheanfire15:11
*** prometheanfire is now known as Guest141515:12
*** chandankumar is now known as raukadah15:21
*** Guest1415 is now known as prometheanfire15:35
*** jpena is now known as jpena|off15:52
gmannfungi: mnaser this is to add stable/xena periodic jobs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/81099916:09
*** ysandeep is now known as ysandeep|dinner16:10
fungigmann: thanks, i'll take a look here shortly after the zuul restart16:17
*** ysandeep|dinner is now known as ysandeep|out17:09
rlandy|ruckhello17:42
rlandy|ruckwe have failing tox jobs everywhere17:42
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/194568217:42
rlandy|ruckhttps://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml#L3817:42
rlandy|ruckfrom this code ^^17:43
rlandy|ruckfungi: hi - ^^ any thoughts here?17:44
clarkbrlandy|ruck: it always always always :) helps if you can link directly to a failure17:44
fungirlandy|ruck: checking, we did just merge some fixes for that17:44
clarkbis https://zuul.opendev.org/t/openstack/build/f8c23147316040beb63b29d8885fa7eb an example of it?17:44
rlandy|ruckyes - that is it 17:45
rlandy|ruckclarkb: noted (about the linking) :)17:45
clarkbI'm going to guess this is some corner case triggred by the tripleo project tox.ini fiels because other projects are running tox successfully17:45
fungispecifically we fixed an instance where the tox_extra_args role var wasn't being applied to the tox invocations in the siblings tasks17:45
rlandy|ruckpossibly - it is hitting tripleo-related repos17:46
clarkb[line  3]: 'using tox-3.24.4 from /home/zuul/src/opendev.org/openstack/tripleo-quickstart-extras/.tox/.tox/lib/python3.8/site-packages/tox/__init__.py (pid 2919)\n' is what it doesn't like17:46
fungioh, yep, looks like its failing in the tox showconfig output parser17:46
clarkbI wonder if https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/tox.ini#L5-L6 causes that17:47
fungii wonder what leads to the "using tox-..." line getting added to the output17:47
clarkbthe .tox/.tox venv is the boostrapping venv that tox creates when you use requires17:47
clarkbfungi: its that requires block I think. It creates a meta venv that it actually runs tox and virtualenv and so on out of17:48
clarkbThis allows you to bootstrap a newer tox than is present on the local system using tox (among other things)17:48
fungiahh, yeah i don't think we have any testing of tox.requires in the jobs which exercise the tox role17:49
fungiaccording to zbr there's a *very* recent fix to tox which tries to guarantee the stdout from tox --showconfig will be parseable ini data, but we need to be able to parse output from older tox versions as well17:51
fungiclarkb: rlandy|ruck: i've got a revert on the way, when gerrit finally accepts my push17:56
rlandy|ruckfungi: thanks for the quick action on that17:57
fungiit's https://review.opendev.org/81200117:58
rlandy|rucksaw https://review.opendev.org/c/zuul/zuul-jobs/+/812001/ merged - thank you!18:26
fungirlandy|ruck: yw, i'll hit you up once i have a fixed revert of the revert passing a new test (once i come up with a good way to exercise it) and maybe you can depends-on from something to verify it's clean for a tripleo change18:50
rlandy|rucksure18:51
fungiokay, i see the problem, and i agree it's the tox.requires triggering it19:03
fungithe way the siblings lib tries to parse tox --showconfig -vv is by discarding lines up until it sees one starting with [ since that would be a section heading in the ini output19:04
fungiexcept the verbose output in the nested pip install going on when there's tox.requires specified also prefixes some lines of output with what are (i think?) pids like "[176766] /some/path/to/a/command --with --options"19:05
fungii bet if it were to discard until it reaches a line which begins with a [ AND ends with a ] we'll be set19:06
opendevreviewMerged openstack/openstack-zuul-jobs master: Add stable/xena to periodic-stable templates  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/81099919:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!