Tuesday, 2022-04-12

ianwre 263490 -- 34094 * 5 / 60 /60 == 47.3 hours by my calculation.  that's essentially two days worth of cpu time we give to failing diff query from stackalytics a month00:01
ianwat least if we mined bitcoin with that we might have something to show for it :)00:02
clarkbooh that is interesting analysis00:05
*** rlandy_ is now known as rlandy|out00:05
clarkbI wonder why it cares about diffs anyway00:08
clarkbmaybe line count info.00:08
clarkbbut also why does it need to requery it if it is failing already00:08
clarkbianw: re https://review.opendev.org/c/opendev/system-config/+/445616/ the old system set it to 4096 and then multiplied that by two for ulimits. Then we ported that to the docker system with the chagne you linked I think. So this cahnge proposed way back when doubling it to 8192 * 2 and I updated it to do that under docker00:27
clarkbbasically I think we maintained status quo with the chagne you pushed. We didn't actually double as proposed00:27
ianwoh, hrm, ok very possible i misread the history :)00:28
ianwi don't see any of those errors in the logs00:29
clarkbnow whether or not we really need that I dunno its been years at the lower limit. But bumping it seems safe and we thought it might be a good idea at one time00:29
clarkbya I think we've been ok with the current limits. Possibly the thing that changed is we repack more often now00:29
clarkbwhich means far fewer object files for gerrit to open00:29
clarkbanyway if we want to say this isn't needed anymore we should abandon the change00:30
clarkband that is a perfectly valid answer00:30
ianw# ls -l /proc/1877/fd | wc -l00:31
ianwthat's how many files it current has open00:32
fungiso about an order of magnitude headroom00:32
ianwwe'd probably need to track it to be sure, especially around say periodic job kick-off, etc.00:34
clarkbya though an order of magnitude difference is a good indication we're good as is00:35
clarkbI've got to help with dinner now but if you want to -1 with "open file count is <500" that is good for me00:36
ianwyeah updated my comment00:36
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add per-build WinRM cert generation  https://review.opendev.org/c/zuul/zuul-jobs/+/83741600:43
opendevreviewMerged opendev/git-review master: Drop support for Python 3.5  https://review.opendev.org/c/opendev/git-review/+/83722203:47
opendevreviewIan Wienand proposed opendev/git-review master: Enforce minimum git version  https://review.opendev.org/c/opendev/git-review/+/83745104:26
chkumar|ruckianw: hello05:14
chkumar|ruckianw: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_103/836104/21/gate/tripleo-ci-centos-9-containers-multinode-wallaby/103c74c/job-output.txt is failing with following error05:14
chkumar|ruckmirror-workspace-git-repos : Synchronize src repos to workspace directory task05:15
chkumar|ruckECDSA host key for has changed and you have requested strict checking.05:15
chkumar|ruckHost key verification failed.05:15
chkumar|ruckand another example is this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_971/836104/21/gate/tripleo-ci-centos-9-containers-multinode-wallaby/97147e1/job-output.txt05:16
chkumar|ruckCan you check on infra side? when free, thanks!05:16
ianwhrm, has this happened multiple times?05:16
chkumar|ruckone with retry and one with retry_limit05:17
chkumar|ruckanother retry does not have logs05:17
chkumar|ruckianw: just 2 logs05:18
ianwhttps://zuul.opendev.org/t/openstack/build/97147e1f93714596a3db512f7512f35b/logs - iweb-mtl05:20
ianwhttps://zuul.opendev.org/t/openstack/build/103c74c36e3443be9a5ae0ae9d53f4d4 - iweb-mtl05:20
chkumar|ruckso it is linked with the provider iweb-mtl01.05:21
chkumar|ruckhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby jobs running on ovh-bhs1 is working fine, can we avoid this provider iweb-mtl01 ?05:25
ianwmaybe; seems like recycled nodes possibly ...05:28
ianw53f4d4 was
ianw512f35b was
chkumar|ruckianw: ok, thank you for looking into that.05:36
chkumar|ruckianw: is there anything we can do to avoid that? or can we remove recycled nodes?05:36
ianwit's clearly not supposed to do that :)  i'm just trying to see if there's any obvious errors 05:36
ianw198.72.124.40 has been up for 1:1705:52
ianwthat would be since ... just after 04:3605:53
ianw53f4d4 failed at 2022-04-12 03:27:0505:56
*** pojadhav- is now known as pojadhav05:57
ianwi dunno, i can't see clearly what caused this failure06:03
chkumar|ruckianw: I will keep an eye on this job, if more comes will let you know with a bug06:20
chkumar|ruckianw: ++ thank you for checking on it :-)06:20
ianwwell, an unsatisfying conclusion so far.  it could be something on the cloud side, with the floating-ip not updating, or being reused06:21
ianwI think at this point, yeah, we should monitor for more cases06:21
opendevreviewIan Wienand proposed opendev/git-review master: Enforce minimum git version  https://review.opendev.org/c/opendev/git-review/+/83745106:30
fricklerianw: those two IPs seem to be long running instances again. wasn't iweb supposed to get turned down anyway or am I mixing this up?06:37
frickleralso iirc mgagne wanted to look into cleaning things up some time ago but I don't remember seeing a response06:38
opendevreviewDr. Jens Harbott proposed openstack/diskimage-builder master: Update gentoo python version to 3.9  https://review.opendev.org/c/openstack/diskimage-builder/+/83623707:27
*** Tengu_ is now known as Tengu08:26
hrwcan zuul directory view can get sortable headers?09:07
hrwhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0e6/837293/4/check/kolla-ansible-ubuntu-source-masakari/0e653ec/primary/logs/ansible/ begs for 'sort by date' being available09:07
*** rlandy|out is now known as rlandy10:31
fungifrickler: inap has ended their openstack public cloud, but the iweb staff (different company) are in control of the servers there are trying to keep things running for us a while longer11:30
fungihrw: sortability might be hard since that index you linked is a static html file served from a swift container. if you can come up with a way to serve sortable indexes from the arbitrary swift services in public cloud providers, then perhaps11:32
fungiianw: `ssh root@ uptime` says "up 2 days"11:34
fungithe problem is that there's more than one virtual machine with that ip address, so just because the one you end up at when you check has a short uptime, that doesn't mean it's the problem vm11:34
fungianyway, i'm going to tell it to poweroff11:34
hrwfungi: ok.11:35
hrwor will look for some userscript to handle that11:36
fungihrw: it's far more likely the logs view of the build result page could grow some smarter indexes, since it's already a javascript app. i almost never use the raw indexes in swift anyway11:37
hrwtimes when I understood javascript passed long time ago11:40
fungii still don't understand javascript, if that helps11:41
hrwhttps://gist.github.com/ameboide/368797ecaa18bb9ac11f866e12156c95 adds some simple sorting. good enough11:43
hrwfungi: ;)11:46
fricklerrepo mirror issues seem to continue, now something fedora in case somebody wants to dig deeper https://78fd1c0a167dfb96b269-6e86dc8a2337ee139316a184fc899cae.ssl.cf5.rackcdn.com/836237/2/check/dib-functests/ae0b17d/logs/fedora_build-succeeds.FAIL.log https://zuul.opendev.org/t/openstack/build/ae0b17d50bcb488b8b5206fac036aa6611:52
fricklerand I also never use the build result page for logs because it tends to clog up my firefox11:53
hrwnot only yours11:54
fungiStatus code: 404 for http://mirror.iad3.inmotion.opendev.org/fedora/updates/35/Everything/x86_64/repodata/026490e6ea124e04df9d50fd577848426ac58a09b7401a8dca4a7cc43bde172d-filelists.xml.zck11:55
hrwrepodata out of sync11:55
fungifrickler: that looks like what we saw last week when there was some significant churn for f35 package updates and the second-tier mirror we were syncing from was apparently mid-update11:55
fungiit cleared on its own eventually when they finished updating their mirror11:56
fricklerwould be nice if we had a way to keep snapshots of working states until that happens11:58
fungifrickler: the trick would be to identify "working states"11:59
fungiif we could do that automatically, we could just have the rsync script check mirror consistency and exit nonzero for problems, at which point we'd avoid calling vos release and the read-only replicas would remain unchanged until the version in the read-write volume achieved a consistent state12:00
fricklerfungi: just select some set of test jobs that you can point at a proposed update12:00
fungiour mirror updates aren't triggered by zuul though12:00
fungi"some random jobs pass" is a rather poor (and likely inefficient) means of checking that the mirror content is internally consistent12:01
fungimaybe the fedora maintainers have a tool which can be used to check mirror consistency quickly?12:02
fricklermaybe inefficient but essentially the kind of check we care about in the end12:02
fungino, not really. "all jobs are fine with the mirror content" is what we care about12:02
fungi"random jobs are fine with the mirror content" isn't the same12:02
fungiif there's a corrupt package then you won't find out in your canary if it doesn't install that package12:03
fricklerI didn't suggest random, selecting a representative subset of all jobs might be possible12:03
frickleralso it doesn't need to have 100% coverage, the usual repodata issues should get caught by almost any job I guess12:04
fricklerthese also don't seem to be specific to fedora, more a general issue for rpm-based distros12:06
fungia tool we can run from the rsync script would be an efficient way to block releasing an inconsistent/incomplete content update. finding ways to have jobs run against candidate states of the mirror and then somehow provide a signal which blocks calling vos release until it's working seems very fragile, as well as unlikely to provide complete coverage12:08
fungiif we redid our mirror update mechanism to be entirely run from zuul, that might be slightly easier to engineer, but still at a minimum implies shadow paths or vhosts for the mirror sites so that nodes could be switched to fetch the candidate content12:10
fungisince we'd also need to serve the read-write volumes alongside the read-only replicas12:10
fungiand would still be an indirect/partial test of mirror consistency12:12
hrwhttps://fedoraproject.org/wiki/Infrastructure/Mirroring can be handy12:12
fungiyep, quite familiar with it, but we don't qualify to use that and sync from their first-tier mirrors based on prior discussions12:13
hrwimho CI should count as private mirror12:14
fungihttps://fedoraproject.org/wiki/Infrastructure/Mirroring#How_can_someone_make_a_private_mirror.3F "Private mirrors cannot pull from the master Fedora download servers. They must pull from another listed public mirror."12:16
hrwso Fedora master -> 3rd party mirror -> infra mirror, right?12:18
fungiyes, and the problem arises when "3rd party mirror" is in an inconsistent state, we sync from it, and then our mirrors reflect the same state12:20
fungii've been trying to figure out if https://pagure.io/quick-fedora-mirror/blob/master/f/quick-fedora-mirror has any consistency checking features, i know we've discussed using it in the past12:21
fungilooks like we've been talking about quick-fedora-mirror for 5 years now: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2017-04-04.log.html#t2017-04-04T17:40:2212:26
fungiunfortunately i'm still not seeing anything in quick-fedora-mirror which would notice when the mirror you're syncing from is in an inconsistent state12:29
funginot even a simple index reference checker12:29
fungiwe could probably write one, but i get the impression rpm/yum/dnf/whatever change their indexing models over time so it may need to do different things for centos 7 mirror indices than for fedora 3512:30
fungiand if it's not something written/used by the distros themselves, then it will probably bitrot and need regular maintenance12:31
hrwthis is where reprepro from Debian does the job better iirc as it fetches packages and rebuilds indexes locally12:33
hrwiirc because I use it for own repos only nowadays12:33
fungiwith the downside being that the package manager can no longer rely on the distro's index signatures12:35
fungi(we could sign them ourselves and bake our signing key into our images, but there's not much point)12:35
fungiand yeah, i'm familiar enough with apt repository indices that i could whip up a consistency checker fairly easily, but i know very little about what dnf expects12:36
hrwno idea too12:43
Clark[m]Not really here yet today, but I feel like official public mirrors are broken and we reflect that state isn't a bug on our end. If it is something we can address via verification (as with reprepro) then great let's do it. But it is also a very low priority thing for me and ideally upstream would fix their update order12:54
Clark[m]It's similar to CentOS breaking ping. We could fix that but it would be counterproductive to as it would require everyone to magically know for a month while it is broken how we fixed it. Instead pressure should be pushed upstream to correct those problems. We correctly caught and reported them and that is a success of the ci system12:56
*** rlandy is now known as rlandy|mtg13:03
*** pojadhav is now known as pojadhav|afk14:02
*** dviroel is now known as dviroel|mtg14:15
opendevreviewTristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix version to include wipness  https://review.opendev.org/c/opendev/system-config/+/83758014:42
* frickler likes "wipness" :)14:56
clarkbfungi: https://review.opendev.org/c/opendev/bindep/+/837394 and its child may be of interest to you15:49
clarkbbindep updates for dependency change and alma linux support15:49
fungithanks, and yeah i've been meaning to look at those15:50
*** dviroel|mtg is now known as dviroel|lunch16:02
*** rlandy|mtg is now known as rlandy16:03
clarkbthe gerrit mailing list has an interesting discussion on rehoming a project from one gerrit server to another16:08
clarkbthe tldr is that you cannot do this easily because you need to update the installation uuid in the repo and all of the change numbers in a repo need to be update so that they do not collide with existing changes in the existing server16:09
clarkbcalling this out beacuse this is theoretically something we need to be on the lookout for when importing projects into our gerrit. However, I think the way jeepyb does the imports it will fail to push any existing changes into our gerrit thus bypassing the issue16:09
*** pojadhav|afk is now known as pojadhav16:14
fungii thought the ideas with the new project-specific change urls was that gerrit would eventually be able to move away from a global index number16:24
fungibut i guess that will take a while and break lots of other assumptions16:24
clarkbya so far they haven't done that. YOu can still go to https://review.opendev.org/$number and it works16:26
opendevreviewsean mooney proposed openstack/project-config master: update Review-Priority lable for nova related projects  https://review.opendev.org/c/openstack/project-config/+/83759516:32
*** marios is now known as marios|out16:40
*** dviroel|lunch is now known as dviroel16:58
clarkbfungi: re iweb it seems they haev gone into a "all nodes are stuck in deleting" state17:00
*** artom_ is now known as artom17:01
clarkbon the nodepool side that seems to manifest as a timeout waiting for node cleanup. I suspect that means the apis are still up otherwise we'd get 500 errors or network connectivity problems17:01
opendevreviewJeremy Stanley proposed opendev/git-review master: Revert "Drop support for Python 3.5"  https://review.opendev.org/c/opendev/git-review/+/83760217:21
clarkbI guess due to centos 7 we'd need to detect git version anyway which means that old python can live another day too17:23
fungiyeah, i meant for those two changes to be approved together, they were just separated for ease of evaluation17:25
clarkbfwiw I think I'm ok with telling centos 7 users to install old git-review17:25
clarkbwhich is what I also suggested that potential old python3 users could do17:25
clarkbthey would do it for different reasons but that end result would be the same.17:26
fungiyep, i'm okay with it either way, we just technically don't need to drop py35 testing unless there's a specific change which causes us to be unable to continue17:26
clarkbI have removed my WIP from https://review.opendev.org/c/opendev/system-config/+/831441 with an indication of why I think it is safe to land that now. SHould be an easy review if you ahve a moment17:36
clarkbNot sure how much longer we want to keep the grafana graphs for that cloud: https://review.opendev.org/c/openstack/project-config/+/831442 but that chagne is still WIP'd17:37
mgagneclarkb fungi : I'm working on it right now. I managed to find and delete a bunch of rogue instances. There was also an incident that messed up our cells v1 databases.17:59
fungioh thanks mgagne!18:01
mgagneI'll take a look at the ones stuck in deleting.18:01
clarkbmgagne: not urgent I just wanted to make note of that as I took a look to ensure it wasnt' something on our end18:01
clarkband thank you!18:02
fungimgagne: also i ssh'd in and directly powered off a couple of rogue virtual machines we noticed at and which had been up for a day or two and were conflicting with new nova instances which got those addresses assigned18:04
mgagneyes, I found some rogue instances that were marked as deleted in the api cell database but not compute cell database. we are still running cells v1 because why not.18:05
fungiclarkb: this is an interesting behavior... look at the three most recent zuul comments on https://review.opendev.org/83711818:08
fungiside effect of the depends-on failure reporting when used for another change in the same project?18:08
mgagneinstances stuck in deleting are now deleted. I see a whole bunch of instances in spawning, I suppose a new focal image was uploaded recently?18:14
clarkbsorry was grabbing early lunch due to the noon infra meeting18:33
clarkbfungi: looks like the parent did fail twice?18:35
clarkbso I think those messages are correct18:36
clarkband I would expect them to be posted even without the depends on18:36
fungimgagne: looks like the most recent focal image uploaded to iweb was 1.5 hours ago18:36
clarkbmgagne: yes an hour an a half ago according to an image listing18:36
fungiclarkb: oh! it looks like maybe my comment on 837394 at 17:59 triggered a reentry into the gate pipeline18:39
fungii didn't start it with the word "recheck" so wasn't expecting that to happen18:39
fungii wonder why that triggered it18:39
clarkbthe failures are related to the fedora mirroring torubles that were discussed earlier today18:40
fungiright, hence my comment18:41
fungithough looking into the mirror-update logs, the last time rsync saw file changes was around 13 hours ago, so the mirror we're pulling from must be stuck in a bad state18:42
ianwthanks for looking into those nodes.  it seems obvious now that the I ended up at might not have been the same seen from another location.  i wonder if there's some sort of LB effect going on where the source location matters19:19
fungiit's almost certainly just your basic arp overwrites at work19:21
fungiso which one you end up connecting to depends on which mac the gateway has seen use that address and recorded to its arp table at the time19:21
ianwI wonder why https://mirror.aarnet.edu.au/pub/fedora/linux/updates/35/Everything/x86_64/repodata/ has more files than http://mirror.iad.rax.opendev.org/fedora/updates/35/Everything/x86_64/repodata/19:45
opendevreviewMerged opendev/system-config master: Remove byobu profile configuration  https://review.opendev.org/c/opendev/system-config/+/52942619:46
clarkbianw: we filter out things like source rpms and large image files19:47
ianwthis is the repodata though, this other mirror has different -primary.xml files etc.19:48
ianwi'm wondering if maybe we should be doing additive updates or something, then pruning old files after a given time (maybe, clutching at straws)19:49
clarkbianw: have you checked our upstrema though?19:49
clarkbI htink fungi was indicating it too lacks that content19:49
fungii don't think delayed deletes help in this case, without knowing that we need to delay until the parent mirror finishes updating and reaches coherency19:50
ianwyes, we look the same as upstream (http://pubmirror2.math.uh.edu/fedora-buffet/fedora/linux/updates/35/Everything/x86_64/repodata/)19:51
ianwdifferent to https://lon.mirror.rackspace.com/fedora/updates/35/Everything/x86_64/repodata/ though19:52
ianwso http://mirror.iad.rax.opendev.org/fedora/updates/35/Everything/x86_64/repodata/repomd.xml == https://lon.mirror.rackspace.com/fedora/updates/35/Everything/x86_64/repodata/repomd.xml19:55
ianwbut 8da0646b3c0dea2866247260d4b2d05073d92a7addf1c9612b09b2c67b1dadc2-primary.xml.gz just doesn't exist on the mirror19:57
fungiyeah, that's about where i got to with it20:01
ianwi feel like if i go back through history i'm going to see lon.mirror.rackspace.com being taken out for some reason, but it does seem like a good option...20:02
ianwalthough does that "lon." stand for "london"?20:02
ianwUnited Kingdom,20:03
opendevreviewClark Boylan proposed opendev/base-jobs master: Stop submitting logstash/subunit gearman jobs  https://review.opendev.org/c/opendev/base-jobs/+/83761920:03
clarkbfungi: ^ I went ahead and decied to remove the roles too. But let me know if you think we should do it more carefully only removing the use of the roles first. I figure a revert is mostly the same either way so why not rip it all out20:04
clarkbianw: we have asked a couple of times for thoes involved in fedora to try and help us find a better mirror and people tend to point at the most top level mirror which we cannot pull from as we are not publicly listed20:05
ianwweird, so that's the one listed in the fedora mirror page.  but mirror.rackspace.com responds and is us local20:05
fungii didn't check the git history for that line, but wouldn't be surprised to find it there20:06
ianwi'll try a manual pull from mirror.rackspace.com on mirror-update for f35 and see if it works ...20:06
ianwit looks in sync20:06
opendevreviewMerged opendev/statusbot master: twitter: send log level messages  https://review.opendev.org/c/opendev/statusbot/+/83708220:07
clarkbfungi: I've +A'd https://review.opendev.org/c/opendev/git-review/+/837451/ I think to remove frickler's wip vote on the parent change we need to escalte privs? Is that osmething you want to do?20:07
corvusianw: clarkb fungi : ^ i left a comment on the twitter change20:08
clarkbhrm I'm not familiar enough with twitter to know if there are expectations one way or the other about verbosity20:09
clarkbto be fair we tend to not #status log super often either and that info is still probably within realms of reason?20:10
corvusi think it's less a platform expectation, and what is it we were actually trying to offer to folks20:10
corvusi log boring stuff with commit shas all the time20:10
corvusbut if "log of zuul commit shas deployed" is what our users want, who am i to argue :)20:11
fungii'm okay with reverting it too, i similarly don't use twitter so i'm not sure what the balance should be between "look opendev is transparent and you can follow what's going on in excruciating detail" vs "please not another log entry about package upgrades"20:11
clarkbI wonder if we hashtagged them with say #notice or #log if users could filter them more readibly20:12
clarkbthat might be a good compromise?20:12
corvusyeah.  it's a tradeoff and i don't know what balance we're aiming for.  mostly i wanted to highlight that if the goal is "make sure people know about service interruptions and changes" this may be changing the balance in the other direction.20:13
corvusnot a twitter expert, but i think the options are pretty coarse.  another option may be to use multiple accounts (one for high-value/low-volume, another for firehose).20:14
corvuspretty sure twitter themselves have tons of official accounts :)20:14
fungithat's certainly an idea20:14
corvusanyway, that's totally a +0 comment from me.  i am not in the constituency :)20:15
ianwwell, it hasn't worked for, a long time -- since twitter dropped part of their API -- so wasn't achieving anything.  maybe many years ago, when we had 2-to-3 times as many people actively involved in infra things, it would be "too much".  clearly, since i proposed it, i think we're better to try and drive some engagement with the day-to-day things at this point though20:16
clarkbya I think I'm willing to try something different and if people complain we can revert or split to two accounts etc20:17
corvusokay, i just didn't see any discussion of the flip side.  should i continue to log "boring" things, or only "interesting" things?  is there any change we should make to the structure of log entries now that they have an audience?20:17
corvusi certainly have phrased notice and alerts different from log because of the differences in publishing targets20:18
corvuslike, i use capital letters and complete sentences20:18
clarkbI don't think we need to change how we use status log. Its always professional and has useful info (if potentially boring)20:19
opendevreviewMerged opendev/system-config master: Use deb-docker mirror for upload image too  https://review.opendev.org/c/opendev/system-config/+/68985820:20
ianwi guess these rackspace mirrors don't actually do rsync, despite what the mirror page says :/20:23
opendevreviewClark Boylan proposed openstack/project-config master: Remove geard graphing from zuul-status dashboard  https://review.opendev.org/c/openstack/project-config/+/83762120:26
clarkbcorvus: ^ thtas the graph cleanup which includes clenaup of an old zuul geard graph too20:26
corvushah, that really messes up the layout doesn't it?  we need to add 2 graphs or drop 1 :)   or maybe change that to a 2x2?20:28
corvusi like the 2x2 idea20:29
corvusrow1: zuul jobs/hour | node requests20:29
corvusrow2: gerrit events/hour | test nodes20:30
clarkblet me see how to do that20:30
corvusclarkb: i think change it to `span: 6` for the remaining 4 graphs20:30
corvusthey're already in the right order so that should be it20:30
corvus(a row is 12 units total)20:31
opendevreviewClark Boylan proposed openstack/project-config master: Remove geard graphing from zuul-status dashboard  https://review.opendev.org/c/openstack/project-config/+/83762120:31
clarkbgot it20:31
ianwwell, trying a dry-run with mirrors.us.kernel.org brings in : Everything/x86_64/repodata/8da0646b3c0dea2866247260d4b2d05073d92a7addf1c9612b09b2c67b1dadc2-primary.xml.gz20:38
ianwi.e. should be in sync20:38
ianwagain, i feel like we've switched away from this mirror before20:38
clarkbyes pretty sure we've used that one before20:39
ianwthough that was "mirrors.kernel.org", rather than "mirrors.us.kernel.org" ... that may make a difference20:40
clarkbthey resolve to the same location for me20:41
clarkbbut that may not be true from our mirror updater20:41
ianwyeah it does seem to change20:41
ianwi got a sfo once, then mirror-update just got pdx20:41
ianwi think that it's also been probably years since we tried it, so it's not like a constant flip-flop20:42
ianwtibbs is in #fedora-admin as is noted as the contact, i'll try reaching out quickly20:46
opendevreviewIan Wienand proposed opendev/system-config master: mirror-update: switch fedora to kernel.org mirror  https://review.opendev.org/c/opendev/system-config/+/83762520:50
*** dviroel is now known as dviroel|out20:55
opendevreviewMerged opendev/system-config master: mirror-update: switch fedora to kernel.org mirror  https://review.opendev.org/c/opendev/system-config/+/83762522:17
*** rlandy is now known as rlandy|bbl22:35
ianwi'm running a fedora sync with the lock22:37
clarkbwe're looking at a thing in #openstack-qa that seems to be causing all devstack jobs on ubuntu focal to fail due to pip/pbr/setuptools not being able to install keystone22:44
clarkbI'm working on holding a node to look at and there is some suspicion that a git package update from today could be to blame22:44
clarkbjust fyi in case we start to see problems like that elsewhere22:44

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!