Friday, 2023-10-06

opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
opendevreviewMerged openstack/project-config master: Normalize projects.yaml
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube
priteauIs there an issue with opendev rockylinux mirrors? Kayobe CI keeps failing with: Depsolve Error occurred: \n Problem: cannot install the best candidate for the job\n  - nothing provides python3 = 3.9.16-1.el9_2.2 needed by python3-devel-3.9.16-1.el9_2.2.x86_64\n  - nothing provides python3-libs(x86-64) = 3.9.16-1.el9_2.2 needed by python3-devel-3.9.16-1.el9_2.2.x86_6407:59
fricklerpriteau: I don't thing we mirror rocky, that should be just a caching proxy. can you check the status of upstream mirrors? I seem to remember that there were some more issues in the recent past already. NeilHanlon might know more08:06
priteauMaybe we're just unlucky and are hitting an unsynced mirror. python3-devel is in appstream but python3 and python3-libs are in baseos.08:10
fricklerinfra-root: wheel builds for debian seem to be failing for 1 month according to the release timers on , maybe we should just stop doing wheel builds in general with nobody really around to maintain them any longer 08:13
fungifrickler: interesting, looks like the wheel architecture changed from debian-11-x86_64 to debian-11.7-x86_6411:49
fungiwheel builds seem to be succeeding, it's writing them into afs that breaks because the directory name is now different11:50
fungii think this is an ansible change11:53
fungiwe're populating that from ansible_distribution_version, which for bullseye changed from 11 to 11.7 around the time we made the default ansible version update11:55
fungii guess we need to adjust playbooks/publish/wheel-mirror.yaml to treat debian the same way as centos now and just use ansible_distribution_major_version11:55
opendevreviewJeremy Stanley proposed openstack/project-config master: Use Debian major version in Debian wheel filenames
fungifrickler: that ^ ought to address it12:04
fricklerfungi: oh, interesting, iirc osa had a similar issue with the name changing recentish, one month ago might fit12:06
fungiwell, i narrowed down the start of the post_failure results to be the next run after the ansible default change merged for that tenant12:07
fungiso hard to imagine anything else might have changed on that same day to break it12:07
frickleroh, so it was the zuul ansible version change. noonedeadpunk jrosser ^^ not sure if your issue has gotten resolved, but it may still be interesting for you12:10
noonedeadpunkI do recall that, but can't recall detail already :(12:12
noonedeadpunkthough... we could forget to address that...12:12
noonedeadpunkso it's good you reminded about that:)12:13
jrosserI think we saw different versions reported from the host and from an lxc built with debootstrap12:17
jrosserfor reasons which were completely unclear12:17
NeilHanlonfrickler, priteau: we had (another) outage yesterday evening--but I've put in place the hopefully final mitigation against the issues we're having that will, at the very least, bring back stability to and our mirrorlist server12:32
NeilHanlon(cc noonedeadpunk, jrosser -- and thanks for putting up with the noise these last few weeks)12:33
fricklerNeilHanlon: thx for the update12:41
*** blarnath is now known as d34dh0r5313:11
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: DNM Add workaround for unqualified-search in Minikube
opendevreviewJeremy Stanley proposed openstack/project-config master: Use Debian major version in Debian wheel filenames
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: DNM Add workaround for unqualified-search in Minikube
clarkbfungi: any opinion on whether we should land the ua update first or the gitea upgrade?15:16
clarkbI'm good to approve either or both right now15:16
fungii guess as long as we keep in mind that a problem with requests getting through could be due to either, i'm fine merging both at once15:17
fungiwhat do you think about merging 897086 ? the spf one is seeming less urgent based on the current finger-pointing going on between zdi/exim/libspf215:19
fungii approved both 897244 and 89750015:20
Clark[m]897086 seems fine. We already do that for other names.15:22
Clark[m]And ya test coverage for UA filters is actually decent15:23
fungicool, approved 897086 too15:24
fungiClark[m]: do you have an opinion on 897545? seems like a behavior change in ansible 8 we hadn't previously spotted15:26
clarkbfungi: do you have a link to example build logs?15:44
clarkbat least for current debian builds would be nice15:45
fungiclarkb: in scrollback, but i added the link in a review comment just now15:46
clarkbthanks does appear to show the difference15:48
opendevreviewMerged openstack/project-config master: Use Debian major version in Debian wheel filenames
mnaserfungi: can i know what is the ips that you see as unreachable via ipv6? i see a /47 from that range being announced but nothing else which is 2600:6c00:2::/4715:58
fungimnaser: 2600:6c60:5300:375:96de:80ff:feec:f9e7 and 240d:1a:6af:1b00:6ee4:daff:fe51:3c95 are two in different parts of the world that don't seem to be routable in ca-ymq-1 but are routable in sjc115:59
mnaserinteresting i dont even see an announcement for that15:59
fungii see them in looking glasses for various backbones15:59
mnaserah ok i see it now 1 sec16:00
fungimnaser: i'm getting responses--thanks!!!16:02
mnaserone sec, i just wanna make sure it doesnt break again16:02
fungisure, just confirming that there do seem to be routes now, so you found it16:02
mnaserare they both good?16:03
mnaseri hate to tell you what i did16:03
fungii don't control the other one, i'll have to give that user a heads up to test16:03
mnaser'clear bgp ipv6 *' on a friday16:03
mnaserbut i dont see a routing loop in traceroute liek i did before16:03
fungiany idea why they were being filtered? ran out of table space?16:04
mnaserno there's plenty, it's actually frr/quagga that does the bgp bits but who knows if something got stuck so kernel/frr was not in sync16:04
fungioh fun16:04
fungii'll let you know if it crops up again, but i'm guessing we just got "lucky" that it was an announcement for my isp that started to get dropped on the floor16:05
mnasersounds like it might be one of those16:05
fungiso if it does happen again, it may be different users impacted next time16:06
fungithanks again mnaser! very happy to be able to drop my forced-v4 workarounds for talking to gerrit16:07
mnaserno problem, sorry it took so long16:07
fungino apologies needed16:08
fungiwe're all quite busy, i understand completely16:08
guilhermespThanks mnaser and appreciate the patience fungi !16:17
fungino problem, i'm just happy i could help16:18
fungii know how hard it can be to deal with vague reports from users16:19
fungimy new netbook picked last night to go toes-up, so i've scrambled to switch my workstation over from pulseaudio to pipewire/wireplumber to get it working reliably with my bt headset and wireless mute button in time for today's openmetal call. seems i've got everything working, but apologies in advance if my audio is terrible, and if i run into technical difficulties please proceed without me16:23
fungitravelling for ato a week from tomorrow will be even more fun. at least my new phone is just a compact debian-based computer, and i had the foresight to buy a laptop-like "dock" for it, so i can use it as a normal laptop too16:33
fungibut i haven't copied my important stuff onto it yet, so i guess that's yet one more thing i need to do over the coming week16:34
fungithe entire drive is luks-encrypted (have to supply a keyfile on removeable media or enter a passphrase when powering it on), so i feel pretty safe using it for opendev stuff16:36
clarkbmy laptop is also sad and I really need to call lenovo but its low on my priority list due to lack of planned travel16:43
opendevreviewRadosÅ‚aw Piliszek proposed openstack/project-config master: Add nebulous/resource-manager
opendevreviewMerged opendev/system-config master: Upgrade to gitea 1.20.5
opendevreviewMerged opendev/system-config master: Update the apache ua filter set
opendevreviewMerged opendev/system-config master: Blackhole deliveries for Mailman v3 local user
clarkbfungi: thank you for taking notes18:04
clarkbIt looks like gitea is generally happy but I still need to check all 6 backends updated18:04
fungifor anyone who is curious about the openmetal redeployemnt discussion, raw notes can be found at
fungigitea is still working for me after both those changes deployed18:08
clarkbI liek stepping through all 6 urls like to confirm the page loads and the version reported is as expected. I've done so and this lgtm18:11
fungiexim deferral queue on lists01 is down to 43 messages after the blackhole alias for the mailman@ address deployed18:11
fungii need to run to the hardware store and pick up some supplies for more weekend renovation projects, but should be back in about an hour18:12
opendevreviewMerged openstack/project-config master: Add nebulous/resource-manager
fungiexim mainlog is no longer filling up with errors for that address either18:16
fungispeaking of logs, i guess we should see if there are still nefarious crawler agent strings in the gitea server apache logs worth blocking18:17
fungiokay, back for now19:25
clarkbI'm going to stop banging my head against the wall called java and find lunch then go do some school stuff19:40
fungiwall o' java19:40
fungii may be grabbing early dinner shortly myself. nice, quiet friday in opendev19:47
fungiyeah, heading out again for a bit19:51
fungiand still nothing's broken22:08
clarkbI ended up pushing what I've got in response to their comments. I believe I have it filtering refs properly or at least under my test case. But Now I've made the laeked file problem workse :)22:08
fungigreatest friday ever22:08
clarkbI've asked for feedback on the approach though. I don't really want to keep banging my head agains the wall if this isn't what they had in mind22:08
fungiyeah, it doesn't sound great22:09
clarkbthe upside is I think it is progress? For a while there it was just replicating everything including what it shouldn't22:09
clarkbso getting back to proper filtration is at least something I guess22:09
clarkbside note: I can't figure out how to see the replication log when running tests which makes it really difficult/annoying to see what is going on22:10
clarkbI've asked for infos on how to do that as well22:10
clarkbI think if I can get that info I can trace things and make sense of it all. I could also theoretically attach jdb but this is all multithreaded and I worry that would be even more confusing22:10
clarkbfungi: for next week you still think Monday is a good day to take the Gerrit bookworm plunge?22:11
clarkbnot sure if you ever thought that was a good idea. Asking now :)22:12
fungiyep, i have nothing major planned for that day other than openstack artifact signing key rotation22:12
fungisounds like a good plan22:12
clarkbcool. I think I verified what I could previously but I'll try to do a quick recheck Monday of things and then we can go for it I guess22:12
clarkbI was hoping we could bundle in this replication file leak fix but that seems very unlikely22:13
fungianother time22:13

Generated by 2.17.3 by Marius Gedminas - find it at!