Wednesday, 2021-08-25

Clark[m]https://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html I think that is what you did?00:01
ianwyep00:03
ianwit really seems to me https://review.opendev.org/c/zuul/zuul-jobs/+/805911/2/roles/build-docker-image/tasks/build.yaml isn't being speculatively tested, i'm not sure if i know why00:20
opendevreviewIan Wienand proposed zuul/zuul-jobs master: build-docker-image: Add flag to use BuildKit  https://review.opendev.org/c/zuul/zuul-jobs/+/80591100:25
ianwoh, i guess it's trusted00:25
opendevreviewMerged zuul/zuul-jobs master: build-docker-image: Add flag to use BuildKit  https://review.opendev.org/c/zuul/zuul-jobs/+/80591101:28
xinliangianw: just start to work for today, will look into the repo sync issue of openEuler, thank you for letting  me know this01:37
opendevreviewIan Wienand proposed zuul/zuul-jobs master: build-docker-image: fix indentation of documentation  https://review.opendev.org/c/zuul/zuul-jobs/+/80592802:40
opendevreviewIan Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle  https://review.opendev.org/c/opendev/system-config/+/80593203:23
opendevreviewIan Wienand proposed opendev/system-config master: gitea: use assets bundle  https://review.opendev.org/c/opendev/system-config/+/80593303:23
opendevreviewIan Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle  https://review.opendev.org/c/opendev/system-config/+/80593203:54
opendevreviewIan Wienand proposed opendev/system-config master: gitea: use assets bundle  https://review.opendev.org/c/opendev/system-config/+/80593303:54
ianwclarkb: ^ i've tested locally and that does work ... but not sure how to make it work in gate.  because it doesn't want to run the job to make opendevorg/assets as nothing that depends on changed.  but it also doesn't exist yet04:48
ianwanyway, maybe we approve 805932 first if we like the idea04:49
*** ysandeep|away is now known as ysandeep05:11
*** jpena|off is now known as jpena06:58
*** rpittau|afk is now known as rpittau07:06
*** mgoddard- is now known as mgoddard07:47
opendevreviewRiccardo Pittau proposed openstack/diskimage-builder master: Use Centos 8 Stream jobs for ironic jobs  https://review.opendev.org/c/openstack/diskimage-builder/+/80596007:54
PaulFertserHey :) I can confirm current master works against 3.4.0 out of the box. The special filename /PATCHSET_LEVEL is still visible in one place (diffstat): https://paste.debian.net/1208962/08:10
opendevreviewMerged zuul/zuul-jobs master: build-docker-image: fix indentation of documentation  https://review.opendev.org/c/zuul/zuul-jobs/+/80592808:30
*** ykarel is now known as ykarel|lunch08:52
opendevreviewMerged opendev/elastic-recheck rdo: Ensure git review uses correct branch  https://review.opendev.org/c/opendev/elastic-recheck/+/80585809:29
*** ykarel|lunch is now known as ykarel10:17
*** dviroel|ruck|afk is now known as dviroel|ruck10:59
*** ysandeep is now known as ysandeep|away11:07
*** jpena is now known as jpena|lunch11:29
*** jpena|lunch is now known as jpena12:30
*** ysandeep|away is now known as ysandeep13:58
*** lbragstad_ is now known as lbragstad14:39
*** jpena is now known as jpena|off15:06
opendevreviewSorin Sb├órnea proposed openstack/project-config master: Ensure promote-tox-docs-infra runs only on master  https://review.opendev.org/c/openstack/project-config/+/80601215:22
*** ysandeep is now known as ysandeep|away15:24
dtantsurhi folks! we're having an opensuse deja vu: https://zuul.opendev.org/t/openstack/build/be23920121a745d9a97ca1425661b32b16:04
*** rpittau is now known as rpittau|afk16:04
dtantsurseems to have started today, may be transient16:04
fungidtantsur: thanks, that usually indicates that wherever we're rsync'ing our mirror data from is broken16:09
fungii'll check a few things16:09
dtantsurthank you!16:09
fungihttps://mirror.dfw.rax.opendev.org/opensuse/timestamp.txt indicates the last successful rsync completed at 16:07 today, just a few minutes ago16:10
fungithis is where we set the rsync source: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/opensuse-mirror-update#L2816:13
fungilooks like they have an https version of the mirror at the same hostname, so probably the same fs16:14
*** sshnaidm is now known as sshnaidm|afk16:14
fungidtantsur: i also don't see any 6bb9fc59e4b8c010181e0ba4414df87a516782ba743df2c3317170d412756f4a-deltainfo.xml.gz file at https://mirror.us.leaseweb.net/opensuse/update/leap/15.2/oss/repodata/ but given we just finished re-syncing our mirror in the past few minutes, it's possible things are consistent there and that's no longer the deltainfo file it will be looking for16:17
dtantsurokay, I'll recheck something to verify16:17
fungithanks, if it's continuing to break we can look at switching to a different mirror or reach out to the opensuse folks if all their mirrors have gone sideways (maybe leaseweb mirrored the same bad state from the primary?)16:18
* dtantsur is curious how many projects use opensuse nodes16:19
fungidiskimage-builder at a minimum16:20
dtantsurright, makes sense16:21
fungii think devstack did too, but those jobs have been failing since ages so are either still non-voting or have maybe been removed16:21
dtantsurWhile I have your attention, when should we brace for opensuse 15.3 becoming the default?16:22
fungihttps://codesearch.opendev.org/?q=opensuse-15 will give you some idea of where the images might be used for jobs16:22
dtantsurmm, I keep forgetting about codesearch. good point.16:23
fungidtantsur: i don't think we have a set schedule, looks like we may rely on someone in the know pushing a patch to playbooks/roles/mirror-update/files/opensuse-mirror-update and maybe other places to update the version16:23
dtantsurI see. I suspect Bifrost will break when it happens, hence my question. But the job is non-voting, so it's not the end of the world :)16:24
fungiyeah, it's a tradeoff. the problem is that the new minor versions come along often enough that updating distinct nodeset and label names everywhere is an even bigger pain16:25
fungiwe go through something similar with fedora, trying to work out who's still using eol fedora images so we can clean them up is an almost sysiphean task16:26
dtantsurI can imagine. Yeah, Fedora is another thing we fix from time to time in Bifrost16:51
fungifor lts distros like opensuse we just treat the main version number (major for opensuse, centos, debian, minor for ubuntu) as the thing we track with a persistent name. for short-support distros like fedora it's more challenging, we'd likely be better off just offering a fedora-latest image and trying to keep that up to date16:55
fungior maybe fedora-latest and fedora-previous so projects can test functionality on whatever two fedora versions are not eol at any given point in time16:56
fungibut for projects like openstack, it doesn't make a lot of sense to do long-term testing on distros which aren't supported for as long as our software is16:59
dtantsurright17:18
dtantsurfungi: hmm, failed again on the same file: https://zuul.opendev.org/t/openstack/build/4ab4e2fe65dd42628deef3f1893869e9. worth rechecking again?17:38
zbri personally like the idea of movable nodeset `{os}-(stable|current|latest)`.17:59
fungidtantsur: no, at this point we probably need to inspect a "more official" opensuse mirror and see if it's also in the same state, and then either look for a different rsync mirror or reach out to people we know who work on opensuse (if we still know anyone)18:06
dtantsurokay, I see18:07
fungii'll see if i can identify something more likely to be in a sane state than that leaseweb mirror18:07
dtantsurI've tried `zypper update` in a fresh 15.2 container, it worked18:08
fungihttps://mirrors.opensuse.org/ claims the mirror we're using is current for leap 15.2 updates, but i don't know how current that list really is (says it's updated every 30 minutes, though no clue how thoroughly the entries there get checked at each update)18:19
fungihttps://mirror.wdc1.us.leaseweb.net/opensuse/update/leap/15.2/oss/repodata/ (updated an hour or two ago) has way more recent content than https://mirror.dal10.us.leaseweb.net/opensuse/update/leap/15.2/oss/repodata/ (last updated in march), but both are in the mirrors list as usable for 15.2 updates, so i'm not going to put a lot of trust in the list :/18:22
mordredfungi: "#21 170.1 qemu: uncaught target signal 11 (Segmentation fault) - core dumped" ... *wow18:35
mordredfungi: https://zuul.opendev.org/t/openstack/build/7faae3085d364b43a852a8b7363a8fcf :) 18:35
mordredClark, corvus: you may or may not find that interesting too ... that's attempting to install curl in a docker build of python-builder - fails consistently on all three versions of python18:37
fungihttp://mirror.datto.com/opensuse/update/leap/15.2/oss/repodata/ also has timestamps within the past few hours, but completely different hashes18:38
mordredlooks like it's on the arm builder backends18:38
fungii guess we're doing arm64 under emulation there? the job doesn't have any arm64 nodes in its inventory anyway18:40
fungidtantsur: bad news is every opensuse mirror i check seems to have a different set of metadata, even the ones updated today, so without some guidance from suse folks i'm likely to be picking alternatives at random18:46
fungieven the one we're mirroring from appears to have updated files more recently than we pulled from it18:47
fungiit's clearly missing a deltainfo file still though, but it's far from the only one. https://ewr.edge.kernel.org/opensuse/update/leap/15.2/oss/repodata/ is in a similar situation18:48
fungii wonder if opensuse is in the middle of pushing out a large update to 15.2 today and updates across their mirror system are simply not safely implemented18:49
fungiwho do we know at or with contacts in suse these days? we used to ping dirk with such questions but i haven't seen him around for the past couple of months18:51
fungilooks like pulling updates from the current rsync mirror are also failing for us at times with "@ERROR: max connections (20) reached -- try again later" so we're not getting updates every two hours like we normally would19:12
fungii'm trying to manually run an interim update from another mirror to see if we get a more sane state19:15
fungimirror.clarkson.edu looks like it has a fairly current copy, updated a few hours ago, while not missing the diffinfo or similar metadata files the mirrors look like they should have19:21
fungialso seems to be a reasonably complete mirror of various opensuse package suites and versions, unlike some of the mirrors on the list19:22
fungithough that may also mean we need to tweak our mirror script to exclude more directories, hard to say just yet19:23
fungilooks like the last time the opensuse mirror source host was updated was two years ago with https://review.opendev.org/695846 (we were previously using mirrors.rit.edu until it became stale)19:33
fungidid not complete within the 30-minute timeout encoded in the mirror script so i've resumed it now, but may need to temporarily customize it to remove the timeout instead19:57
opendevreviewJeremy Stanley proposed opendev/system-config master: Update OpenSUSE mirror source  https://review.opendev.org/c/opendev/system-config/+/80605020:16
fungiokay, finished. if anything, it seems to have freed up a little space on the opensuse mirror volume. i've temporarily held the update lock for it to avoid undoing this while it's re-tested, and then we can approve ^ if it works out20:16
fungidtantsur: when you get time, see if a recheck is happy20:16
mordredmnaser: do I remember correctly that you're actually using python-base/python-builder on arm somewhere?20:22
*** timburke_ is now known as timburke20:27
*** timburke_ is now known as timburke20:55
mordredfungi, Clark, corvus : https://zuul.opendev.org/t/openstack/build/1f14f740b8ac4fe1948344e6ad2a6e0b <-- I rechecked and it seems to be consistently failing21:16
fungithat's super fun21:21
fungiianw: does that ring a bell? i have a vague recollection of hitting a segfault while installing some package on arm64 a while back21:23
corvusmordred: do you think a noop change would fail too?  that looks pretty innocuous21:30
fungiyeah, it certainly doesn't look related to the change. i wonder if we're somehow trying to install x86 packages on aarch or something21:33
mordredcorvus: yeah - the dockerfile hasn't gotten to the point where the change is yet21:35
fungiright, the apt-get update doesn't mention buster-backports21:36
fungioh, but... it's bullseye now21:36
fungimordred: so at a minimum, i think that change may no longer be needed? the job seems to be building bullseye images rather than buster there21:38
fungii guess the base images switched over shortly after the bullseye release a couple weeks ago21:39
fungialternatively it could be updated to add bullseye-backports instead of buster-backports now21:40
fungitoo many consecutive release codenames starting with b. and not getting any better, as the next one will be bookworm21:41
fungiupdates aside, something may have broken with how we're emulating arm64 there for some new packages in bullseye21:43
PaulFertserI wonder what's the story behind not being able to add reviewers by gertty.21:45
fungiPaulFertser: gertty doesn't have complete feature parity with the webui, there's lots you can do in the webui or through the api(s) which isn't implemented in gertty21:46
PaulFertserfungi: yes, but how do you deal with it in your workflow?21:46
fungifor the most part, adding reviewers feels a lot like "assigning work" to people, which isn't how our projects generally operate21:47
fungiinstead people create dashboards to surface the things they specifically want to prioritize reviewing21:47
PaulFertserWhat about "hey buddy please take a look, that's your code according to git blame" kind of adding reviewers?21:47
fungiwe mostly use irc/matrix for informal chat about reviews21:48
fungiand project mailing lists if we need to reach a broader audience21:48
PaulFertserInteresting, thanks for the clarification.21:48
corvusPaulFertser: i'd welcome a patch to add it, but it's not an itch i personally have21:48
PaulFertsercorvus: understood, thank you. BTW, the fixes work nicely, and I appreciate your fast response. Noticed another place where "special" name is still shown https://paste.debian.net/1208962/ (diffstat)21:49
corvusagree21:50
mordredcorvus: it looks like it's at least pulling different shas for the layers on the two different arches:21:53
mordredhttps://zuul.opendev.org/t/openstack/build/1f14f740b8ac4fe1948344e6ad2a6e0b/log/job-output.txt#1078-110621:53
fungimordred: could it be mixing buster and bullseye packages somehow?21:54
fungi(or rather, installed binaries from them)21:54
mordreddoesn't look like it reading through the logs22:00
mordredhttps://bugs.launchpad.net/ubuntu/+source/qemu/+bug/192807522:01
fungiso the next most likely probability seems to be a conflict between some arm64 binaries in bullseye and however we're implementing that cross-arch install/build mechanism22:04
fungicould be we haven't exercised it since the base images updated from buster to bullseye22:05
corvusmordred, fungi: i'm confused -- isn't it wrong to say 'buster-packports' when it should say bullseye?22:07
corvusoh wait, sorry, that would, again, only happen after we get to the line in the change22:08
corvuspaper-bag-head-emoji22:09
fungiyeah, but you're right that it does mean the change needs updating (as i suggested earlier)22:10
corvusyep, but we don't need to wait for that to know something's broke22:12
fungii concur22:14
fungiseparate issues22:14
ianwhrm, that segfault doesn't ring an immediate bell22:32
ianwi feel like i saw cases where we appeared to pull x86 layers instead of arm64 layers from the buildset registry.  but that has got a lot further than that22:32
opendevreviewMonty Taylor proposed zuul/zuul-jobs master: Update binfmt support image used  https://review.opendev.org/c/zuul/zuul-jobs/+/80605722:41
opendevreviewMonty Taylor proposed opendev/system-config master: Add backports repos to base and builder images  https://review.opendev.org/c/opendev/system-config/+/80031822:41
mordredfungi, ianw, corvus: new hypothesis: ^^ image updated from buster to bullseye and maybe we're using old binfmt support. I updated to the binfmt support instructions in the official buildx docs (which seems to now be a supported non-experimental thing)22:42
ianwsounds promising22:44
corvusmordred: ++22:50
*** dviroel|ruck is now known as dviroel|out22:59
fungiyeah, that seems an angle worth investigating further23:35

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!