Thursday, 2021-08-26

xinliangianw: The openEuler mirror repo we sync from is fixed.
fungithanks xinliang!01:34
fungiianw: did we disable anything for that we need to put back now?01:36
ianwxinliang/fungi: thanks, i don't think so01:36
fungiokay, great01:36
xinliangThe openEuler mirror repo we sync from is fixed.
xinliangfungi: that repo is maintained by openEuler, should be good to sync with01:49
xinliangThey just ran out of space yesterday and fix01:49
fungiyep, thanks for the added detail. i was just making sure we hadn't disabled any tests or something in the meantime while it was down02:59
*** ysandeep|away is now known as ysandeep05:39
*** iurygregory_ is now known as iurygregory06:28
opendevreviewMerged opendev/system-config master: Retire openstack-i18n-de mailing list
*** rpittau|afk is now known as rpittau07:24
*** jpena|off is now known as jpena07:37
*** ykarel is now known as ykarel|lunch08:28
*** sshnaidm|afk is now known as sshnaidm09:38
*** ykarel|lunch is now known as ykarel10:18
dtantsurfungi: yep, the job seems to pass now, thank you!10:23
ykarelfrickler, the infra mirror pypi issue happening now, recall the issue with boto3 some days back?10:40
ykarelfailing: pip install --index-url= horizon==20.0.010:41
ykarelexample log
ykarel#infra-root if someone can check ^ what's wrong with mirrors10:41
ykarelalso it's specific to mirrors, as i see some working like one10:48
ykarel and failing, not checked others10:49
*** ksambor__ is now known as ksambor10:55
*** ysandeep is now known as ysandeep|mtg10:57
*** dviroel|out is now known as dviroel|ruck11:14
opendevreviewSlawek Kaplonski proposed openstack/project-config master: Update neutron-lib grafana dasboard
fungiykarel: we don't mirror pypi. the problem is almost always an issue with some fastly cdn endpoints in specific parts of the world returning old index copies11:25
fungiyou'd see the same thing from the same clouds whether you went through the proxy there or not11:25
fungiit's not specific to our mirrors, it's specific to where on the internet you're trying to reach pypi from11:25
fungialso it seems to happen with fastly's endpoints near montreal canada more than anywhere else, from what we've seen. it's possible they have one or more bad systems there11:26
funginote that ovh's bhs1 region and inap's mtl01 region are both in the montreal area, which is a strong indication it's this same issue again11:28
*** jpena is now known as jpena|lunch11:42
dtantsurhi folks! I have a zuul job that for one task has "Waiting on logger" instead of the task output. Still, the task and the job end successfully. What could cause that?11:52
*** dviroel|ruck is now known as dviroel|ruck|brb11:58
*** dviroel|ruck|brb is now known as dviroel|ruck12:20
*** jpena|lunch is now known as jpena12:38
fungidtantsur: zuul polls trying to connect to the log streamer to build that log, and logs that it's doing so. if the log streamer process doesn't start or is unreachable, you'll see that printed12:49
fungithat's more for the benefit of people watching the live log stream, but generally it's so that you don't simply see silence or a gap in timestamps and think zuul sat and did nothing for that span of time12:50
fungimordred: neat! so it really was an emulation bug?12:51
mordredfungi: it seems so? I'm working on reproducing it locally so I can try the fix12:54
mordred(I have a local focal box)12:54
mordredit does not fail locally \o/12:54
fungisame qemu package/version as we have on the job node?12:59
mordredfungi: it's been a while - what is the current correct way to create an autohold?12:59
fungisudo docker-compose -f /etc/zuul-scheduler/docker-compose.yaml exec scheduler zuul autohold --tenant openstack --project opendev/system-config --job system-config-run-gitea --change 805458 --reason "fungi testing matrix cors" --count 113:00
fungithat's one i set recently13:00
fungi(from the scheduler's command line)13:00
fungimight also be possible to do it with zuul-client now13:00
mordredand not sure about the version ... 13:00
lucasagomesmorning/afternoon folks, if you have some time mind taking a quick look at this new project proposal ? Thanks in advance13:15
*** ysandeep|mtg is now known as ysandeep13:29
ykarelfungi, Thanks for checking, how much it takes to clear it, i see it's still failing13:48
ykarelfrickler last time told that once it reproduces can have them try to flush those from their cdn network13:49
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Promote debian bullseye job to non-voting
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Add non-voting debian stable job
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Add non-voting debian stable job
clarkbykarel: I think you run a curl -XPURGE against the resource url on pypi14:06
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Add non-voting debian stable job
clarkbeg curl -XPURGE but I'm not sure if that works on indexes like it does package data14:07
clarkbthe problem has to do with cdn fallback iirc so it may just fallback to a the broken nodes and reset the state to broken14:07
fungiit varied. in recent events it wasn't serving copies of indices without relevant metadata, it was just serving old copies of indices which didn't include the versions of things we were pinning to14:10
fungithough i suppose that still might be a fallback, if they fixed the fallback site to include the same metadata warehouse serves but just happens to have some old indices14:11
fungilike maybe it stopped getting synced or something14:11
ykarelyes ^ seems correct14:18
opendevreviewMonty Taylor proposed zuul/zuul-jobs master: Update binfmt support image used
ykarelfungi, last time you said you can ask them to flush those cached data from their cdn network14:26
ykarelcan you do that14:26
fungiykarel: i'll need to find what specific page/url needs flushing, but i can try to do it myself as clarkb mentioned above. if they've got a fallback backend with outdated content however that's not going to fix it14:27
ykarelfungi, okk Thanks14:28
fungiykarel: do you happen to know what it downloaded (or wanted to download) which didn't exist or was too old? the pip error is bubbling up from the dep solver unfortunately so it will take some tracking down, and if you've already done that it will save me the time14:30
ykarelfungi, horizon==20.0.014:30
fungioh, it's coming from the horizon===20.0.0 constraint, okay14:31
fungiso it'll be the horizon index page i guess14:32
fungi$ curl -XPURGE
fungi{ "status": "ok", "id": "12826-1629899565-9246" }14:34
fungiso the api replied to that purge request with an ok status at least14:35
fungii did it from the inap-mtl01 mirror, no idea if that matters14:35
fungii also did one with a trailing / because without the / is a redirect if you do a get14:36
Clark[m]I think it is supposed to do a global purge when you do that so location shouldn't matter14:38
fungialso the inap-mtl01 and ovh-bhs1 mirrors currently include horizon 20.0.0 links in the simple index they're proxying14:39
fungiso we may need to wait a while to see it happen again14:39
ykarelfungi, now install is working14:39
fungiykarel: thanks, hopefully it stays that way14:40
ykarelThanks fungi Clark[m] 14:47
fungimordred: you've got a node held now:
mordredfungi: yes! I have used it to verify a fix for  thanks!15:33
fungioh awesome15:35
fungiand i just approved it. thanks!15:36
funginow i need to get ready for the lists.k.i server upgrade starting in 20 ~minutes15:37
*** jpena is now known as jpena|off15:39
fungiclark is otherwise engaged, so i'll be doing the upgrading on the server, but we tested it very thoroughly and he made extensive notes in
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Add non-voting debian stable job
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Fix debian-minimal security repos
rpittauhello everyone! Can anyone please check this ^ ? It's quite urgent as at the moment debian-minimal is broken and as a consequence also the ironic-python-agent-builder CI, thanks!15:59
fungistarting on the server upgrad enow16:00
fungitaking the server offline for a clean image16:02
*** rpittau is now known as rpittau|afk16:02
opendevreviewMerged zuul/zuul-jobs master: Update binfmt support image used
fungi#status log The service is offline for scheduled upgrades over the next two hours16:05
opendevstatusfungi: finished logging16:05
mordredmnaser: ^^ the zuul-jobs change there should fix the issues with arm docker builds16:07
opendevreviewMerged openstack/project-config master: Retire puppet-monasca - Step 3: Remove Project
fungiimage creation complete, bringing the server back up with mailman/exim4 disabled16:19
fungiserver is running again, and i've started a root screen session in which i'll perform the ubuntu upgrade steps16:21
fungithere's an openssl security update today, and for some reason the server is getting a 401 unauthorized trying to upgrade libssl/openssl packages even though `ua status` claims esm-infra channel is entitled and enabled16:28
fungii don't think this is a critical blocker for the xenial to bionic upgrade, so i'm going to ignore it, but would be good to circle back around after and see if any of our other xenial servers with esm set up are having similar problems16:29
fungioh, except do-release-upgrade  refuses to proceed16:30
Clark[m]Not really here but maybe the disable on the snapshot server did disable thing but it didn't know it on the other server.16:31
Clark[m]You could try disabling then reenabling or just disable entirely since you are upgrading16:32
fungiyes, i'm thinking the same, i'm detaching it from esm now16:32
fungijust need to clean up the sources.list entries16:32
Clark[m]Be careful enabling again because by default it enrolls in fips type stuff we don't want16:32
Clark[m]The Ansible we do to enable has the steps to avoid that16:33
fungiyeah, ua detach, removing the corresponding sources.list.d file and then updating the package list again was enough to get do-release-upgrade to proceed16:34
fungii also have the server in the emergency disable list so that ansible won't try to undo anything while i'm working16:35
*** ysandeep is now known as ysandeep|away16:39
fungiinfra-root: expedited review of would be much appreciated, i've got the mirror-update lock held for opensuse already synced from that source it appears to have solved the package update problem for ironic's jobs16:46
fungithe server is at bionic now and sanity checks are good, so i've initiated do-release-upgrade to take it to focal16:59
opendevreviewMonty Taylor proposed opendev/system-config master: Add backports repos to base and builder images
fungiwe're at the one hour mark for the lists.k.i upgrade, so doing well17:00
mordredwoot. the zuul-jobs fix fixed the python-builder issue. now we just have a uwsgi library version issue from the bullseye to deal with. I betcha this is going to turn green!17:01
fungiuntil we found the sea of green17:03
fungithe mailman package upgrade from bionic to focal complains about /var/lib/mailman/qfiles not being empty, investigating17:05
fungistarted mailman services but it's continually filling /var/lib/mailman/qfiles/out/ with new files and removing them17:10
fungilooks like it's because i didn't stop apache and someone tried to use the webui to subscribe to a list while exim was down but the upgrade had restarted the mailman qrunners17:14
fungiokay, cleaned out that looping qfile with mailman stopped again, and continued the upgrade process17:16
fungii also stopped apache once i realized that was problematic17:17
fungifinal reboot onto focal is underway now17:25
fungii lied, one more reboot after i enable exim4 and mailman again17:27
fungi#status log Upgrades completed for the server, which is now fully back in service17:30
opendevstatusfungi: finished logging17:30
fungii sent a message to the kata-dev ml letting them know everything's done, and received a copy of it as a list subscriber. also the archives still look intact17:31
fungii've taken the server back out of the emergency disable list now as well17:33
mordredcorvus, fungi, Clark: is green!19:06
Clark[m]Ok things are slowing down around here. I'm running errands and then lunch then another appointment then hopefully I can look at some changes in like 3 hours19:11
Clark[m]mordred the issue was the python base images bumped to bullseye?19:15
fungiwell, the trigger was that the images updated to bullseye19:15
fungithe problem was a couple of new and wondrus bugs that exposed19:15
mordredyeah. you'll love the solution ;)19:16
mordredit's nice and generally opaque19:16
fungisegfaults during package installation are always a sign your day is looking up19:23
*** sshnaidm is now known as sshnaidm|afk19:32
opendevreviewShnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Add building for CentOS 9
mordredfungi: oh - should I rebase that backports change on master - and then also update the matrix-eavesdrop dockerfile to appropriately drop the buster-backports line?20:06
corvusmordred: are the updated opendev images published -- or will the first bullseye publication be after 800318 merges?20:07
mordredyeah - I think so - we reference a few different buster thigns in there20:07
mordredcorvus: after20:07
corvusmordred: ack.  also, i had just approved 800318, but based on your rebase comment i have downgraded that to a +220:08
corvusbut let me know if you want me to put the +w back20:08
mordredcool. I'll rebase, then push up a modified patch (for ease of review)20:08
fungimordred: you could do it as a separate change, but i expect any builds of the matrix-eavesdrop image in the interim may break20:08
mordredor - I suppose I could just do a followup change to update the matrix-eavesdrop stuff20:09
mordredjust approving what's there now and doing a followup will waste fewer nodes20:09
corvus+3 restored20:09
opendevreviewMonty Taylor proposed opendev/system-config master: Update matrix-eavesdrop for bullseye
mordredcorvus, fungi : ^^ used depends-on to stack it :)20:13
opendevreviewMerged opendev/system-config master: Add backports repos to base and builder images
fungi#status log Used rmlist to remove the openstack-i18n-de mailing list from now that has merged20:44
opendevstatusfungi: finished logging20:44
mordredcorvus: :sadpanda: system-config-promote-image-python-base-3.8 : FAILURE in 58s21:41
mordredcorvus: got a 500 from dockerhub on the promote:
corvusmordred: ugh, we've been getting more of those lately.  good news is that it was the cleanup phase, so it should actually be published21:43
corvusmordred: i don't think you need to do anything and you can just ignore it21:44
mordredoh cool21:53
mordredI think this: is good to go now then21:53
corvusoff it goes :)21:54
ianwif there's a tl;dr and i can help with the images/arm64 issues etc. LMN22:12
mordredianw: I think we got it sorted:
mordredianw: tl;dr is - upstream python image is based on debian - and updated from buster to bullseye - which broke something in qemu/binfmt support22:16
mordredwaves hands furioulsy22:16
mordredso - updating the binfmt config support image to use qemu-*-static instead fixed that - the rest was just followups to bindep versions22:17
*** dviroel|ruck is now known as dviroel|out22:18
ianwexcellent.  well hopefully another 2 years of stable operation :)22:18
fungiianw: is mirror related and could use a look22:26
clarkbI just +2'd it22:27
ianwsure, i feel like we got leaseweb fixed before via a contact, can't remember22:28
clarkbya it is logan- related iirc22:29
ianwthat sounds right :)22:30
fungioh, is leaseweb a.k.a. limestone?22:31
clarkbI don't know if it is aka/== but there is a relationship there iirc22:31
fungianyway, the "official" opensuse mirrors list is a minefield of incomplete and outdated mirrors, so the one i picked was a compromise of having what we needed, being up to date, and not being too far from afs01.dfw22:32
fungiwhere "not too far" is defined as "on the same continent"22:33
opendevreviewMerged opendev/system-config master: Update matrix-eavesdrop for bullseye
ianwhah in .au we consider "not too far" as "only crosses 1 or 2 oceans" :)22:34
fungi"same hemisphere" (latitudinally or longitudinally? you may only pick one)22:35
fungifrom australia you get a choice of mirrors in siberia or chile22:36
opendevreviewMerged opendev/system-config master: Update OpenSUSE mirror source
fungionce deploy is done i'll release the flock for that23:09
mordredI choose to believe that is somehow related to clarkb 23:16

Generated by 2.17.2 by Marius Gedminas - find it at!