| Clark[m] | https://docs.ansible.com/ansible/latest/user_guide/playbooks_environment.html I think that is what you did? | 00:01 |
|---|---|---|
| ianw | yep | 00:03 |
| ianw | it really seems to me https://review.opendev.org/c/zuul/zuul-jobs/+/805911/2/roles/build-docker-image/tasks/build.yaml isn't being speculatively tested, i'm not sure if i know why | 00:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: Add flag to use BuildKit https://review.opendev.org/c/zuul/zuul-jobs/+/805911 | 00:25 |
| ianw | oh, i guess it's trusted | 00:25 |
| opendevreview | Merged zuul/zuul-jobs master: build-docker-image: Add flag to use BuildKit https://review.opendev.org/c/zuul/zuul-jobs/+/805911 | 01:28 |
| xinliang | ianw: just start to work for today, will look into the repo sync issue of openEuler, thank you for letting me know this | 01:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix indentation of documentation https://review.opendev.org/c/zuul/zuul-jobs/+/805928 | 02:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle https://review.opendev.org/c/opendev/system-config/+/805932 | 03:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 03:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle https://review.opendev.org/c/opendev/system-config/+/805932 | 03:54 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 03:54 |
| ianw | clarkb: ^ i've tested locally and that does work ... but not sure how to make it work in gate. because it doesn't want to run the job to make opendevorg/assets as nothing that depends on changed. but it also doesn't exist yet | 04:48 |
| ianw | anyway, maybe we approve 805932 first if we like the idea | 04:49 |
| *** ysandeep|away is now known as ysandeep | 05:11 | |
| *** jpena|off is now known as jpena | 06:58 | |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| *** mgoddard- is now known as mgoddard | 07:47 | |
| opendevreview | Riccardo Pittau proposed openstack/diskimage-builder master: Use Centos 8 Stream jobs for ironic jobs https://review.opendev.org/c/openstack/diskimage-builder/+/805960 | 07:54 |
| PaulFertser | Hey :) I can confirm current master works against 3.4.0 out of the box. The special filename /PATCHSET_LEVEL is still visible in one place (diffstat): https://paste.debian.net/1208962/ | 08:10 |
| opendevreview | Merged zuul/zuul-jobs master: build-docker-image: fix indentation of documentation https://review.opendev.org/c/zuul/zuul-jobs/+/805928 | 08:30 |
| *** ykarel is now known as ykarel|lunch | 08:52 | |
| opendevreview | Merged opendev/elastic-recheck rdo: Ensure git review uses correct branch https://review.opendev.org/c/opendev/elastic-recheck/+/805858 | 09:29 |
| *** ykarel|lunch is now known as ykarel | 10:17 | |
| *** dviroel|ruck|afk is now known as dviroel|ruck | 10:59 | |
| *** ysandeep is now known as ysandeep|away | 11:07 | |
| *** jpena is now known as jpena|lunch | 11:29 | |
| *** jpena|lunch is now known as jpena | 12:30 | |
| *** ysandeep|away is now known as ysandeep | 13:58 | |
| *** lbragstad_ is now known as lbragstad | 14:39 | |
| *** jpena is now known as jpena|off | 15:06 | |
| opendevreview | Sorin Sbârnea proposed openstack/project-config master: Ensure promote-tox-docs-infra runs only on master https://review.opendev.org/c/openstack/project-config/+/806012 | 15:22 |
| *** ysandeep is now known as ysandeep|away | 15:24 | |
| dtantsur | hi folks! we're having an opensuse deja vu: https://zuul.opendev.org/t/openstack/build/be23920121a745d9a97ca1425661b32b | 16:04 |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| dtantsur | seems to have started today, may be transient | 16:04 |
| fungi | dtantsur: thanks, that usually indicates that wherever we're rsync'ing our mirror data from is broken | 16:09 |
| fungi | i'll check a few things | 16:09 |
| dtantsur | thank you! | 16:09 |
| fungi | https://mirror.dfw.rax.opendev.org/opensuse/timestamp.txt indicates the last successful rsync completed at 16:07 today, just a few minutes ago | 16:10 |
| fungi | this is where we set the rsync source: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/opensuse-mirror-update#L28 | 16:13 |
| fungi | looks like they have an https version of the mirror at the same hostname, so probably the same fs | 16:14 |
| *** sshnaidm is now known as sshnaidm|afk | 16:14 | |
| fungi | dtantsur: i also don't see any 6bb9fc59e4b8c010181e0ba4414df87a516782ba743df2c3317170d412756f4a-deltainfo.xml.gz file at https://mirror.us.leaseweb.net/opensuse/update/leap/15.2/oss/repodata/ but given we just finished re-syncing our mirror in the past few minutes, it's possible things are consistent there and that's no longer the deltainfo file it will be looking for | 16:17 |
| dtantsur | okay, I'll recheck something to verify | 16:17 |
| fungi | thanks, if it's continuing to break we can look at switching to a different mirror or reach out to the opensuse folks if all their mirrors have gone sideways (maybe leaseweb mirrored the same bad state from the primary?) | 16:18 |
| * dtantsur is curious how many projects use opensuse nodes | 16:19 | |
| fungi | diskimage-builder at a minimum | 16:20 |
| dtantsur | right, makes sense | 16:21 |
| fungi | i think devstack did too, but those jobs have been failing since ages so are either still non-voting or have maybe been removed | 16:21 |
| dtantsur | While I have your attention, when should we brace for opensuse 15.3 becoming the default? | 16:22 |
| fungi | https://codesearch.opendev.org/?q=opensuse-15 will give you some idea of where the images might be used for jobs | 16:22 |
| dtantsur | mm, I keep forgetting about codesearch. good point. | 16:23 |
| fungi | dtantsur: i don't think we have a set schedule, looks like we may rely on someone in the know pushing a patch to playbooks/roles/mirror-update/files/opensuse-mirror-update and maybe other places to update the version | 16:23 |
| dtantsur | I see. I suspect Bifrost will break when it happens, hence my question. But the job is non-voting, so it's not the end of the world :) | 16:24 |
| fungi | yeah, it's a tradeoff. the problem is that the new minor versions come along often enough that updating distinct nodeset and label names everywhere is an even bigger pain | 16:25 |
| fungi | we go through something similar with fedora, trying to work out who's still using eol fedora images so we can clean them up is an almost sysiphean task | 16:26 |
| dtantsur | I can imagine. Yeah, Fedora is another thing we fix from time to time in Bifrost | 16:51 |
| fungi | for lts distros like opensuse we just treat the main version number (major for opensuse, centos, debian, minor for ubuntu) as the thing we track with a persistent name. for short-support distros like fedora it's more challenging, we'd likely be better off just offering a fedora-latest image and trying to keep that up to date | 16:55 |
| fungi | or maybe fedora-latest and fedora-previous so projects can test functionality on whatever two fedora versions are not eol at any given point in time | 16:56 |
| fungi | but for projects like openstack, it doesn't make a lot of sense to do long-term testing on distros which aren't supported for as long as our software is | 16:59 |
| dtantsur | right | 17:18 |
| dtantsur | fungi: hmm, failed again on the same file: https://zuul.opendev.org/t/openstack/build/4ab4e2fe65dd42628deef3f1893869e9. worth rechecking again? | 17:38 |
| zbr | i personally like the idea of movable nodeset `{os}-(stable|current|latest)`. | 17:59 |
| fungi | dtantsur: no, at this point we probably need to inspect a "more official" opensuse mirror and see if it's also in the same state, and then either look for a different rsync mirror or reach out to people we know who work on opensuse (if we still know anyone) | 18:06 |
| dtantsur | okay, I see | 18:07 |
| fungi | i'll see if i can identify something more likely to be in a sane state than that leaseweb mirror | 18:07 |
| dtantsur | I've tried `zypper update` in a fresh 15.2 container, it worked | 18:08 |
| fungi | https://mirrors.opensuse.org/ claims the mirror we're using is current for leap 15.2 updates, but i don't know how current that list really is (says it's updated every 30 minutes, though no clue how thoroughly the entries there get checked at each update) | 18:19 |
| fungi | https://mirror.wdc1.us.leaseweb.net/opensuse/update/leap/15.2/oss/repodata/ (updated an hour or two ago) has way more recent content than https://mirror.dal10.us.leaseweb.net/opensuse/update/leap/15.2/oss/repodata/ (last updated in march), but both are in the mirrors list as usable for 15.2 updates, so i'm not going to put a lot of trust in the list :/ | 18:22 |
| mordred | fungi: "#21 170.1 qemu: uncaught target signal 11 (Segmentation fault) - core dumped" ... *wow | 18:35 |
| mordred | fungi: https://zuul.opendev.org/t/openstack/build/7faae3085d364b43a852a8b7363a8fcf :) | 18:35 |
| mordred | Clark, corvus: you may or may not find that interesting too ... that's attempting to install curl in a docker build of python-builder - fails consistently on all three versions of python | 18:37 |
| fungi | http://mirror.datto.com/opensuse/update/leap/15.2/oss/repodata/ also has timestamps within the past few hours, but completely different hashes | 18:38 |
| mordred | looks like it's on the arm builder backends | 18:38 |
| fungi | i guess we're doing arm64 under emulation there? the job doesn't have any arm64 nodes in its inventory anyway | 18:40 |
| fungi | dtantsur: bad news is every opensuse mirror i check seems to have a different set of metadata, even the ones updated today, so without some guidance from suse folks i'm likely to be picking alternatives at random | 18:46 |
| fungi | even the one we're mirroring from appears to have updated files more recently than we pulled from it | 18:47 |
| fungi | it's clearly missing a deltainfo file still though, but it's far from the only one. https://ewr.edge.kernel.org/opensuse/update/leap/15.2/oss/repodata/ is in a similar situation | 18:48 |
| fungi | i wonder if opensuse is in the middle of pushing out a large update to 15.2 today and updates across their mirror system are simply not safely implemented | 18:49 |
| fungi | who do we know at or with contacts in suse these days? we used to ping dirk with such questions but i haven't seen him around for the past couple of months | 18:51 |
| fungi | looks like pulling updates from the current rsync mirror are also failing for us at times with "@ERROR: max connections (20) reached -- try again later" so we're not getting updates every two hours like we normally would | 19:12 |
| fungi | i'm trying to manually run an interim update from another mirror to see if we get a more sane state | 19:15 |
| fungi | mirror.clarkson.edu looks like it has a fairly current copy, updated a few hours ago, while not missing the diffinfo or similar metadata files the mirrors look like they should have | 19:21 |
| fungi | also seems to be a reasonably complete mirror of various opensuse package suites and versions, unlike some of the mirrors on the list | 19:22 |
| fungi | though that may also mean we need to tweak our mirror script to exclude more directories, hard to say just yet | 19:23 |
| fungi | looks like the last time the opensuse mirror source host was updated was two years ago with https://review.opendev.org/695846 (we were previously using mirrors.rit.edu until it became stale) | 19:33 |
| fungi | did not complete within the 30-minute timeout encoded in the mirror script so i've resumed it now, but may need to temporarily customize it to remove the timeout instead | 19:57 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Update OpenSUSE mirror source https://review.opendev.org/c/opendev/system-config/+/806050 | 20:16 |
| fungi | okay, finished. if anything, it seems to have freed up a little space on the opensuse mirror volume. i've temporarily held the update lock for it to avoid undoing this while it's re-tested, and then we can approve ^ if it works out | 20:16 |
| fungi | dtantsur: when you get time, see if a recheck is happy | 20:16 |
| mordred | mnaser: do I remember correctly that you're actually using python-base/python-builder on arm somewhere? | 20:22 |
| *** timburke_ is now known as timburke | 20:27 | |
| *** timburke_ is now known as timburke | 20:55 | |
| mordred | fungi, Clark, corvus : https://zuul.opendev.org/t/openstack/build/1f14f740b8ac4fe1948344e6ad2a6e0b <-- I rechecked and it seems to be consistently failing | 21:16 |
| fungi | that's super fun | 21:21 |
| fungi | ianw: does that ring a bell? i have a vague recollection of hitting a segfault while installing some package on arm64 a while back | 21:23 |
| corvus | mordred: do you think a noop change would fail too? that looks pretty innocuous | 21:30 |
| fungi | yeah, it certainly doesn't look related to the change. i wonder if we're somehow trying to install x86 packages on aarch or something | 21:33 |
| mordred | corvus: yeah - the dockerfile hasn't gotten to the point where the change is yet | 21:35 |
| fungi | right, the apt-get update doesn't mention buster-backports | 21:36 |
| fungi | oh, but... it's bullseye now | 21:36 |
| fungi | mordred: so at a minimum, i think that change may no longer be needed? the job seems to be building bullseye images rather than buster there | 21:38 |
| fungi | i guess the base images switched over shortly after the bullseye release a couple weeks ago | 21:39 |
| fungi | alternatively it could be updated to add bullseye-backports instead of buster-backports now | 21:40 |
| fungi | too many consecutive release codenames starting with b. and not getting any better, as the next one will be bookworm | 21:41 |
| fungi | updates aside, something may have broken with how we're emulating arm64 there for some new packages in bullseye | 21:43 |
| PaulFertser | I wonder what's the story behind not being able to add reviewers by gertty. | 21:45 |
| fungi | PaulFertser: gertty doesn't have complete feature parity with the webui, there's lots you can do in the webui or through the api(s) which isn't implemented in gertty | 21:46 |
| PaulFertser | fungi: yes, but how do you deal with it in your workflow? | 21:46 |
| fungi | for the most part, adding reviewers feels a lot like "assigning work" to people, which isn't how our projects generally operate | 21:47 |
| fungi | instead people create dashboards to surface the things they specifically want to prioritize reviewing | 21:47 |
| PaulFertser | What about "hey buddy please take a look, that's your code according to git blame" kind of adding reviewers? | 21:47 |
| fungi | we mostly use irc/matrix for informal chat about reviews | 21:48 |
| fungi | and project mailing lists if we need to reach a broader audience | 21:48 |
| PaulFertser | Interesting, thanks for the clarification. | 21:48 |
| corvus | PaulFertser: i'd welcome a patch to add it, but it's not an itch i personally have | 21:48 |
| PaulFertser | corvus: understood, thank you. BTW, the fixes work nicely, and I appreciate your fast response. Noticed another place where "special" name is still shown https://paste.debian.net/1208962/ (diffstat) | 21:49 |
| corvus | agree | 21:50 |
| mordred | corvus: it looks like it's at least pulling different shas for the layers on the two different arches: | 21:53 |
| mordred | https://zuul.opendev.org/t/openstack/build/1f14f740b8ac4fe1948344e6ad2a6e0b/log/job-output.txt#1078-1106 | 21:53 |
| fungi | mordred: could it be mixing buster and bullseye packages somehow? | 21:54 |
| fungi | (or rather, installed binaries from them) | 21:54 |
| mordred | doesn't look like it reading through the logs | 22:00 |
| mordred | https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1928075 | 22:01 |
| fungi | so the next most likely probability seems to be a conflict between some arm64 binaries in bullseye and however we're implementing that cross-arch install/build mechanism | 22:04 |
| fungi | could be we haven't exercised it since the base images updated from buster to bullseye | 22:05 |
| corvus | mordred, fungi: i'm confused -- isn't it wrong to say 'buster-packports' when it should say bullseye? | 22:07 |
| corvus | oh wait, sorry, that would, again, only happen after we get to the line in the change | 22:08 |
| corvus | paper-bag-head-emoji | 22:09 |
| fungi | yeah, but you're right that it does mean the change needs updating (as i suggested earlier) | 22:10 |
| corvus | yep, but we don't need to wait for that to know something's broke | 22:12 |
| fungi | i concur | 22:14 |
| fungi | separate issues | 22:14 |
| ianw | hrm, that segfault doesn't ring an immediate bell | 22:32 |
| ianw | i feel like i saw cases where we appeared to pull x86 layers instead of arm64 layers from the buildset registry. but that has got a lot further than that | 22:32 |
| opendevreview | Monty Taylor proposed zuul/zuul-jobs master: Update binfmt support image used https://review.opendev.org/c/zuul/zuul-jobs/+/806057 | 22:41 |
| opendevreview | Monty Taylor proposed opendev/system-config master: Add backports repos to base and builder images https://review.opendev.org/c/opendev/system-config/+/800318 | 22:41 |
| mordred | fungi, ianw, corvus: new hypothesis: ^^ image updated from buster to bullseye and maybe we're using old binfmt support. I updated to the binfmt support instructions in the official buildx docs (which seems to now be a supported non-experimental thing) | 22:42 |
| ianw | sounds promising | 22:44 |
| corvus | mordred: ++ | 22:50 |
| *** dviroel|ruck is now known as dviroel|out | 22:59 | |
| fungi | yeah, that seems an angle worth investigating further | 23:35 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!