Wednesday, 2021-08-25

Clark[m] I think that is what you did?00:01
ianwit really seems to me isn't being speculatively tested, i'm not sure if i know why00:20
opendevreviewIan Wienand proposed zuul/zuul-jobs master: build-docker-image: Add flag to use BuildKit
ianwoh, i guess it's trusted00:25
opendevreviewMerged zuul/zuul-jobs master: build-docker-image: Add flag to use BuildKit
xinliangianw: just start to work for today, will look into the repo sync issue of openEuler, thank you for letting  me know this01:37
opendevreviewIan Wienand proposed zuul/zuul-jobs master: build-docker-image: fix indentation of documentation
opendevreviewIan Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle
opendevreviewIan Wienand proposed opendev/system-config master: gitea: use assets bundle
opendevreviewIan Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle
opendevreviewIan Wienand proposed opendev/system-config master: gitea: use assets bundle
ianwclarkb: ^ i've tested locally and that does work ... but not sure how to make it work in gate.  because it doesn't want to run the job to make opendevorg/assets as nothing that depends on changed.  but it also doesn't exist yet04:48
ianwanyway, maybe we approve 805932 first if we like the idea04:49
*** ysandeep|away is now known as ysandeep05:11
*** jpena|off is now known as jpena06:58
*** rpittau|afk is now known as rpittau07:06
*** mgoddard- is now known as mgoddard07:47
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Use Centos 8 Stream jobs for ironic jobs
PaulFertserHey :) I can confirm current master works against 3.4.0 out of the box. The special filename /PATCHSET_LEVEL is still visible in one place (diffstat):
opendevreviewMerged zuul/zuul-jobs master: build-docker-image: fix indentation of documentation
*** ykarel is now known as ykarel|lunch08:52
opendevreviewMerged opendev/elastic-recheck rdo: Ensure git review uses correct branch
*** ykarel|lunch is now known as ykarel10:17
*** dviroel|ruck|afk is now known as dviroel|ruck10:59
*** ysandeep is now known as ysandeep|away11:07
*** jpena is now known as jpena|lunch11:29
*** jpena|lunch is now known as jpena12:30
*** ysandeep|away is now known as ysandeep13:58
*** lbragstad_ is now known as lbragstad14:39
*** jpena is now known as jpena|off15:06
opendevreviewSorin Sb├órnea proposed openstack/project-config master: Ensure promote-tox-docs-infra runs only on master
*** ysandeep is now known as ysandeep|away15:24
dtantsurhi folks! we're having an opensuse deja vu:
*** rpittau is now known as rpittau|afk16:04
dtantsurseems to have started today, may be transient16:04
fungidtantsur: thanks, that usually indicates that wherever we're rsync'ing our mirror data from is broken16:09
fungii'll check a few things16:09
dtantsurthank you!16:09
fungi indicates the last successful rsync completed at 16:07 today, just a few minutes ago16:10
fungithis is where we set the rsync source:
fungilooks like they have an https version of the mirror at the same hostname, so probably the same fs16:14
*** sshnaidm is now known as sshnaidm|afk16:14
fungidtantsur: i also don't see any 6bb9fc59e4b8c010181e0ba4414df87a516782ba743df2c3317170d412756f4a-deltainfo.xml.gz file at but given we just finished re-syncing our mirror in the past few minutes, it's possible things are consistent there and that's no longer the deltainfo file it will be looking for16:17
dtantsurokay, I'll recheck something to verify16:17
fungithanks, if it's continuing to break we can look at switching to a different mirror or reach out to the opensuse folks if all their mirrors have gone sideways (maybe leaseweb mirrored the same bad state from the primary?)16:18
* dtantsur is curious how many projects use opensuse nodes16:19
fungidiskimage-builder at a minimum16:20
dtantsurright, makes sense16:21
fungii think devstack did too, but those jobs have been failing since ages so are either still non-voting or have maybe been removed16:21
dtantsurWhile I have your attention, when should we brace for opensuse 15.3 becoming the default?16:22
fungi will give you some idea of where the images might be used for jobs16:22
dtantsurmm, I keep forgetting about codesearch. good point.16:23
fungidtantsur: i don't think we have a set schedule, looks like we may rely on someone in the know pushing a patch to playbooks/roles/mirror-update/files/opensuse-mirror-update and maybe other places to update the version16:23
dtantsurI see. I suspect Bifrost will break when it happens, hence my question. But the job is non-voting, so it's not the end of the world :)16:24
fungiyeah, it's a tradeoff. the problem is that the new minor versions come along often enough that updating distinct nodeset and label names everywhere is an even bigger pain16:25
fungiwe go through something similar with fedora, trying to work out who's still using eol fedora images so we can clean them up is an almost sysiphean task16:26
dtantsurI can imagine. Yeah, Fedora is another thing we fix from time to time in Bifrost16:51
fungifor lts distros like opensuse we just treat the main version number (major for opensuse, centos, debian, minor for ubuntu) as the thing we track with a persistent name. for short-support distros like fedora it's more challenging, we'd likely be better off just offering a fedora-latest image and trying to keep that up to date16:55
fungior maybe fedora-latest and fedora-previous so projects can test functionality on whatever two fedora versions are not eol at any given point in time16:56
fungibut for projects like openstack, it doesn't make a lot of sense to do long-term testing on distros which aren't supported for as long as our software is16:59
dtantsurfungi: hmm, failed again on the same file: worth rechecking again?17:38
zbri personally like the idea of movable nodeset `{os}-(stable|current|latest)`.17:59
fungidtantsur: no, at this point we probably need to inspect a "more official" opensuse mirror and see if it's also in the same state, and then either look for a different rsync mirror or reach out to people we know who work on opensuse (if we still know anyone)18:06
dtantsurokay, I see18:07
fungii'll see if i can identify something more likely to be in a sane state than that leaseweb mirror18:07
dtantsurI've tried `zypper update` in a fresh 15.2 container, it worked18:08
fungi claims the mirror we're using is current for leap 15.2 updates, but i don't know how current that list really is (says it's updated every 30 minutes, though no clue how thoroughly the entries there get checked at each update)18:19
fungi (updated an hour or two ago) has way more recent content than (last updated in march), but both are in the mirrors list as usable for 15.2 updates, so i'm not going to put a lot of trust in the list :/18:22
mordredfungi: "#21 170.1 qemu: uncaught target signal 11 (Segmentation fault) - core dumped" ... *wow18:35
mordredfungi: :) 18:35
mordredClark, corvus: you may or may not find that interesting too ... that's attempting to install curl in a docker build of python-builder - fails consistently on all three versions of python18:37
fungi also has timestamps within the past few hours, but completely different hashes18:38
mordredlooks like it's on the arm builder backends18:38
fungii guess we're doing arm64 under emulation there? the job doesn't have any arm64 nodes in its inventory anyway18:40
fungidtantsur: bad news is every opensuse mirror i check seems to have a different set of metadata, even the ones updated today, so without some guidance from suse folks i'm likely to be picking alternatives at random18:46
fungieven the one we're mirroring from appears to have updated files more recently than we pulled from it18:47
fungiit's clearly missing a deltainfo file still though, but it's far from the only one. is in a similar situation18:48
fungii wonder if opensuse is in the middle of pushing out a large update to 15.2 today and updates across their mirror system are simply not safely implemented18:49
fungiwho do we know at or with contacts in suse these days? we used to ping dirk with such questions but i haven't seen him around for the past couple of months18:51
fungilooks like pulling updates from the current rsync mirror are also failing for us at times with "@ERROR: max connections (20) reached -- try again later" so we're not getting updates every two hours like we normally would19:12
fungii'm trying to manually run an interim update from another mirror to see if we get a more sane state19:15 looks like it has a fairly current copy, updated a few hours ago, while not missing the diffinfo or similar metadata files the mirrors look like they should have19:21
fungialso seems to be a reasonably complete mirror of various opensuse package suites and versions, unlike some of the mirrors on the list19:22
fungithough that may also mean we need to tweak our mirror script to exclude more directories, hard to say just yet19:23
fungilooks like the last time the opensuse mirror source host was updated was two years ago with (we were previously using until it became stale)19:33
fungidid not complete within the 30-minute timeout encoded in the mirror script so i've resumed it now, but may need to temporarily customize it to remove the timeout instead19:57
opendevreviewJeremy Stanley proposed opendev/system-config master: Update OpenSUSE mirror source
fungiokay, finished. if anything, it seems to have freed up a little space on the opensuse mirror volume. i've temporarily held the update lock for it to avoid undoing this while it's re-tested, and then we can approve ^ if it works out20:16
fungidtantsur: when you get time, see if a recheck is happy20:16
mordredmnaser: do I remember correctly that you're actually using python-base/python-builder on arm somewhere?20:22
*** timburke_ is now known as timburke20:27
*** timburke_ is now known as timburke20:55
mordredfungi, Clark, corvus : <-- I rechecked and it seems to be consistently failing21:16
fungithat's super fun21:21
fungiianw: does that ring a bell? i have a vague recollection of hitting a segfault while installing some package on arm64 a while back21:23
corvusmordred: do you think a noop change would fail too?  that looks pretty innocuous21:30
fungiyeah, it certainly doesn't look related to the change. i wonder if we're somehow trying to install x86 packages on aarch or something21:33
mordredcorvus: yeah - the dockerfile hasn't gotten to the point where the change is yet21:35
fungiright, the apt-get update doesn't mention buster-backports21:36
fungioh, but... it's bullseye now21:36
fungimordred: so at a minimum, i think that change may no longer be needed? the job seems to be building bullseye images rather than buster there21:38
fungii guess the base images switched over shortly after the bullseye release a couple weeks ago21:39
fungialternatively it could be updated to add bullseye-backports instead of buster-backports now21:40
fungitoo many consecutive release codenames starting with b. and not getting any better, as the next one will be bookworm21:41
fungiupdates aside, something may have broken with how we're emulating arm64 there for some new packages in bullseye21:43
PaulFertserI wonder what's the story behind not being able to add reviewers by gertty.21:45
fungiPaulFertser: gertty doesn't have complete feature parity with the webui, there's lots you can do in the webui or through the api(s) which isn't implemented in gertty21:46
PaulFertserfungi: yes, but how do you deal with it in your workflow?21:46
fungifor the most part, adding reviewers feels a lot like "assigning work" to people, which isn't how our projects generally operate21:47
fungiinstead people create dashboards to surface the things they specifically want to prioritize reviewing21:47
PaulFertserWhat about "hey buddy please take a look, that's your code according to git blame" kind of adding reviewers?21:47
fungiwe mostly use irc/matrix for informal chat about reviews21:48
fungiand project mailing lists if we need to reach a broader audience21:48
PaulFertserInteresting, thanks for the clarification.21:48
corvusPaulFertser: i'd welcome a patch to add it, but it's not an itch i personally have21:48
PaulFertsercorvus: understood, thank you. BTW, the fixes work nicely, and I appreciate your fast response. Noticed another place where "special" name is still shown (diffstat)21:49
mordredcorvus: it looks like it's at least pulling different shas for the layers on the two different arches:21:53
fungimordred: could it be mixing buster and bullseye packages somehow?21:54
fungi(or rather, installed binaries from them)21:54
mordreddoesn't look like it reading through the logs22:00
fungiso the next most likely probability seems to be a conflict between some arm64 binaries in bullseye and however we're implementing that cross-arch install/build mechanism22:04
fungicould be we haven't exercised it since the base images updated from buster to bullseye22:05
corvusmordred, fungi: i'm confused -- isn't it wrong to say 'buster-packports' when it should say bullseye?22:07
corvusoh wait, sorry, that would, again, only happen after we get to the line in the change22:08
fungiyeah, but you're right that it does mean the change needs updating (as i suggested earlier)22:10
corvusyep, but we don't need to wait for that to know something's broke22:12
fungii concur22:14
fungiseparate issues22:14
ianwhrm, that segfault doesn't ring an immediate bell22:32
ianwi feel like i saw cases where we appeared to pull x86 layers instead of arm64 layers from the buildset registry.  but that has got a lot further than that22:32
opendevreviewMonty Taylor proposed zuul/zuul-jobs master: Update binfmt support image used
opendevreviewMonty Taylor proposed opendev/system-config master: Add backports repos to base and builder images
mordredfungi, ianw, corvus: new hypothesis: ^^ image updated from buster to bullseye and maybe we're using old binfmt support. I updated to the binfmt support instructions in the official buildx docs (which seems to now be a supported non-experimental thing)22:42
ianwsounds promising22:44
corvusmordred: ++22:50
*** dviroel|ruck is now known as dviroel|out22:59
fungiyeah, that seems an angle worth investigating further23:35

Generated by 2.17.2 by Marius Gedminas - find it at!