Monday, 2023-03-20

*** dhill is now known as Guest828300:15
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788400:21
ianwhttps://imgur.com/a/2Sx426k or https://104.130.253.50/c/x/test-project/+/3 are some markdown comments00:41
ianwi think it's a release note bug that it's listed as an experiment, it's enabled by default00:49
ianwthe @user tagging in comments is not enabled afaict, but is an experiment.  i don't think we need to turn that on explicitly, let's just let upstream work out the kinks in the main instance00:50
opendevreviewMerged openstack/diskimage-builder master: Removal focal pins for testing  https://review.opendev.org/c/openstack/diskimage-builder/+/87744300:52
opendevreviewMerged openstack/diskimage-builder master: Update Fedora to 37  https://review.opendev.org/c/openstack/diskimage-builder/+/87648200:52
ianwclarkb: https://review.opendev.org/c/opendev/system-config/+/868054 is one that adds some s-r and copyconditions labels to the gerrit testing, just to finalise the testing there01:00
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788402:09
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788403:51
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788404:31
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788406:25
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788407:18
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788407:19
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788407:53
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788408:31
opendevreviewIan Wienand proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788409:19
opendevreviewdaniel.pawlik proposed zuul/zuul-jobs master: DNM - Provide ensure-microshift role  https://review.opendev.org/c/zuul/zuul-jobs/+/87608110:39
*** arxcruz is now known as arxcruz|rover10:57
clarkbGitea 1.19 released over the weekend. I'll look into the release notes and update my rc change (I don't think we should land that this week with the openstack release, but good to get it out of the way)15:13
clarkbAlso I need to followup on updating jitsi meet's docker iamge locations15:14
clarkbbut first kernel updates!15:15
clarkbjitsi meet have not updated their docker image location yet15:28
fungimmm15:28
*** dhill is now known as Guest833815:33
opendevreviewJames E. Blair proposed opendev/system-config master: wip: test idea for registry pull  https://review.opendev.org/c/opendev/system-config/+/87788415:37
clarkb"Return 404 instead of 403 if user can not access the repo (#23155) (#23158)" <- from gitea 1.19 release notes. Similar to what I was grumping about the quay API on friday15:43
clarkb"Repositories: by default disable all units except code and pulls on forks (#22541)" is another fun one that won't affect us15:44
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.19.x  https://review.opendev.org/c/opendev/system-config/+/87754116:04
clarkbthere was a template update between rc and final release16:04
clarkbgood think I double checked16:04
clarkbfungi: ianw: more people are complaining about Gerrit 3.7.1's related changes behavior16:26
clarkbI'm hopeful that will get a fix landed before we upgrade16:26
fungiseems likely16:29
fungieven if there's no 3.7.2 yet, we're deploying images autobuilt from the tip of their 3.7 stable branch anyway right?16:29
clarkbcorrect16:29
clarkbcorvus: https://review.opendev.org/c/zuul/zuul-jobs/+/877834 is where I ended up with the precreation of quay repos. I need to catch up on your aliasing of registries work. Is that basically just look at https://review.opendev.org/c/opendev/system-config/+/877884 ?16:32
fungiianw spotted a misbehavior in testing it, per his last comment16:37
clarkbianw: the markdown thing?16:38
clarkber sorry fungi  ^?16:39
fungiclarkb: incorrect pattern match with the redirect in 87785916:40
clarkbah thanks16:41
fungisuggested going from * to +16:41
corvusyeah and i agree with that (i even meant to type that, but pasted the wrong thing)16:47
corvusi think ianw's change supercedes the testinfra change, since it does the full docker/podman pull, so i rebased it away from the testinfra change.  it's possible that ianw also wanted a testinfra that does a curl...  if so, we can add that.  it basically would have nothing in common with the testinfra change anyway, so i'd probably just stick the new one on the end...16:52
corvusi think 877884 looks good now after that rebase away from testinfra -- https://zuul.opendev.org/t/openstack/build/5eedf23dc6274ba893f6342aba3e2545/console is showing the actual pull16:54
corvusianw did make one change i wasn't planning on doing -- he made /v1/ work.  i was only planning on proxying v2 since i'm not sure how to test that v1 is actually working, but i guess we can try proxying v1 and see how it goes :)16:55
corvuswe will need to check the logs though to make sure that we're not falling back to v116:55
clarkbI think just about everything is v2 only at this point?16:55
corvusyeah, that's why that was my plan16:55
corvusif we don't proxy v1, then we know if we've broken v2.16:56
corvuswith the v1 proxy in there, i'm not actually sure i would have noticed that we needed the empty index file for v2 to work.16:56
corvusmy preference still would probably be to only support v216:56
corvusso if we want to support v1, thin i think my change + ianw's are ready to merge (except ianw's needs a new commit msg).  if we want to disable v1 we need to tweak it.16:58
clarkbI can go either way on it. I'm not actually sure modern clients fall back to v1 it would just be for old cliens?16:59
corvuspodman tries to16:59
clarkbhuh til16:59
opendevreviewJames E. Blair proposed opendev/system-config master: Add a functional test for registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87788417:00
corvusokay that's ianw's change with the commit msg updated17:00
clarkboh I guess that is what ianw's -1 comment was getting at. Podman is falling back17:00
clarkbcorvus: one reason to proxy v1 would be ifwe end up sitting in front of a registry that is v1 only. But we can add that support when it becomes necessary and it should be less and less necessary over time17:02
corvushttps://8464fd24502a6c238377-7f41bd38e61f614f484bdc6903cb8f38.ssl.cf2.rackcdn.com/877884/11/check/system-config-run-static/5eedf23/static99.opendev.org/apache2/registry.zuul-ci.org_access.log confirms that we are supporting v2 correctly17:02
corvus(and that the v1 proxy isn't masking a v2 failure)17:02
corvusthat's actually a pretty good argument for the standalone testinfra check -- if we proxy v1, then we definitely should have a testinfra that does a "GET /v2/" to make sure that the v2 empty index page works and is not redirected17:03
opendevreviewJames E. Blair proposed opendev/system-config master: Remove v1 proxy from registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87798917:05
corvusokay that's a followup to remove v1.  i don't feel strongly about that, but that puts the system in the state that i originally imagined.  let's see if ianw wants to keep v1.  if so, i have no objection.17:05
clarkbhttps://gerrit-review.googlesource.com/c/gerrit/+/364214 is the fix for the gerrit thing whcih has landed. This means we should be able to hold a 3.7.1-foo build and double check it on our end17:11
opendevreviewJames E. Blair proposed opendev/system-config master: Add testinfra for registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87786017:15
opendevreviewJames E. Blair proposed opendev/system-config master: Add testinfra for registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87786017:17
corvusclarkb fungi ianw : okay i think 877859 877884 877860 are ready for immediate merging; i think we agree on everything in them. then 877989 is the open question about whether or not we support v1.17:18
corvustopic:reg-proxy17:18
clarkbok I'll review now17:18
fungioh, i see, ianw's -1 on 877859 is actually about the testing change not the change he technically voted against17:20
fungino, nevermind it's about the RewriteRule in 87785917:20
corvusyep17:21
clarkbcorvus: two questions. The old registry.zuul-ci.org wasn't used for anything? Or do we need to ensure what we are proxying to is valid first? Second do you intend on proxying to corvus/ long term or should we update that now?17:21
corvusfungi: and i'm proposing we land my "broken" change and let ianw's testing change fix it.  that's okay because it's not in production.17:21
fungiso is that rewriterule being fixed in a followup change?17:21
fungiaha, yes it is. thanks! wfm17:21
fungii should have just reviewed the others first17:22
corvusclarkb: registry.zuul-ci.org didn't exist before a few days ago, so no worries there.  not long term, will proxy to some as-yet-undetermined org.17:22
corvusi'd like to merge this and double check everything is working (since this is a functional change from my ".htaccess" approach i started last week)17:23
clarkbcorvus: the index file we create isn't accessible due to the redirect is it?17:23
corvusthen: pick a quay org; copy the repos, then update this to the real production values17:23
fungiclarkb: the followup change fixes that17:23
corvusclarkb: it is not accessible in my first change, it is with ianw's fix17:23
clarkbfungi: right, but the followup is the one we aren't sure we will merge?17:23
clarkbwondering if we should create the index file in ianw's fixup too?17:24
corvuser no17:24
corvusi'm so sorry17:24
corvusi feel like i've burned 200 hours of developer time because i typed a "*" instead of a "+" on a saturday...17:24
fungii'm not following. the rewriterule in 877859 shadows the index.html file, but then in 877884 it changes to no longer shadow it17:25
corvusgive me a minute to prepare an explanation17:25
corvusokay, first; i updated the hashtags, so there are 4 changes here under hashtag:reg-proxy17:26
fungiclarkb: i didn't realize we were still unsure about merging 877884, i'd approved it17:26
corvuswait why isn't that hashtagging working17:26
clarkbcorvus: one of them has hashtag:hashtag17:27
clarkbthe other three seem to have set properly17:27
corvusyeah ianw set that... that's the one i'm trying to change...17:27
corvusi guess i can't because i don't own it or something?17:28
corvusoy17:28
corvusi thought we made hashtags editable?17:28
fungiwe still haven't officially decided to allow non-owners to edit hashtags17:28
fungiwhich is gerrit's default17:28
fungii'm in favor of doing so17:28
corvusit makes zero sense i can't edit the hashtag but i can edit the topic17:28
fungiyep17:28
JayFThat can be changed per-project; Ironic already has fwiw17:29
corvuszuul has too i think17:29
fungisome openstack teams played with granting core reviewers exclusive rights to set hashtags and were worried about random users adding/removing them, but i don't know that the concern was warranted nor even still relevant17:29
clarkbI think the main concern is that its an unbounded value unlike topic?17:30
corvusokay, let me explain, i mean sum up17:30
corvushttps://review.opendev.org/q/topic:reg-proxy17:30
clarkbbut ya we can probably just go ahead and update All-Projects17:30
corvusall of those changes expect "remove v1 proxy" are ready to merge now, and when all 3 are merged, they will produce a working system and there are no unresolved concerns with those 3.17:31
corvusthe only remaining unresolved concern is with whether or not we proxy v1 requests, so let's at least wait for ianw on that one.17:31
clarkbgotcha I misinterpreted the earlier statement about maybe merging things to mean the one that adds v1 not removes it.17:31
clarkbI do think the most correct thing is to add the index file when it becomes accessible, but it isn't worth restacking again to make that change17:32
clarkband we are using 302s since the resources may move again17:34
corvusi don't disagree17:34
corvusyep17:34
clarkboh and it will change very soon to address corvus/ but even then we still want to use 302s17:36
corvusyep17:36
clarkbfungi: do you want to review https://review.opendev.org/c/opendev/system-config/+/877859 ? I +2'd it but you approved the child so wanted to make sure you had a chance to look at it first17:37
fungii thought i already had?17:37
corvusoh, re-reading ianw's comment on podman and v1 -- i think he's saying that even podman won't talk to v1 even though it hits the v1 ping url.  that seems to suggest it's really six-of-one whether we proxy v1 or not.17:38
fungigertty still shows me +2 on it17:38
clarkbfungi: oh it shows now. I don't know how I missed it17:38
clarkbthats probabl pebkac on my end17:38
fungiyeah, i +2'd that one yesterday, it hasn't changed since17:38
corvusi'm going to go ahead and +w that because i believe all of ianw's concerns have been addressed17:39
clarkbwfm17:39
corvusclarkb: and you have a +2 from me on 87783417:39
clarkbcorvus: cool. I think we can adjust that one too if we find the api isn't quite in line with what we expect out of the updated jobs17:40
clarkbcorvus: you were still going to work on the push side of the jobs right?17:40
corvusclarkb: yes -- the registry proxy did bump that a bit though; hopefully within a day or two?17:46
Clark[m]corvus: that last message didn't make it to IRC :( but sounds good to me17:46
Clark[m]and it just went through. A lag of a minute or two17:46
opendevreviewMerged opendev/system-config master: Use a static Apache config for registries  https://review.opendev.org/c/opendev/system-config/+/87785918:16
opendevreviewMerged opendev/system-config master: Add a functional test for registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87788418:16
opendevreviewMerged opendev/system-config master: Add testinfra for registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87786018:28
noonedeadpunkHey folks. I'm having issue with rocky-container element and wanted to raise a concern on approach of taking upstream containerfiles. As it's suuuper anoying to do that in environments with no internet connectivity. Now just to build couple of images we kind of need to mirror dockerhub which is really meh.19:21
noonedeadpunkBut you're still maintianing mirrors for DNF as they're required regardless19:21
noonedeadpunkBut dockerhub or smth simmilar is quite a pite to have as a requirement for such simple thing. And building rocky container image kind of jeopardize whole idea of having dib19:22
Clark[m]I'm not sure I fully agree. You can cache a single rocky image and that's all you need19:23
noonedeadpunkIt's still yet another system to maintain19:23
noonedeadpunkAnd ensure it's state and monitor, etc19:23
Clark[m]But also you can't build rocky/fedora/centos-stream using dnf on a number of platforms because it needs to be new enough. We decided that trying to sort that out for all platforms is basically impossible hence relying on the upstream minimal container images to bootstrap the container19:24
Clark[m]The alternative is you can't build the image at all which really does negate the purpose of div19:24
Clark[m]noonedeadpunk: it doesn't need to be a complicated system. You can just import the image into your builder and be done with it19:24
Clark[m]A full mirror isn't required19:25
noonedeadpunkOr maintain https://review.opendev.org/c/openstack/diskimage-builder/+/805800 as internal element...19:25
Clark[m]Sure can always have other elements. The main issue is you can't get a new enough dnf on debuntu to build newer rpm based distros iirc19:26
noonedeadpunkWell, I think c9s is now built using dnf?19:27
noonedeadpunkSo r9 kind of should not have any issue to be built as well19:27
Clark[m]Using container images as a base addresses that as well as in the opposite direction (debootstrap on rpm distros)19:27
Clark[m]The problem is you have to install dnf on Ubuntu/debian19:27
Clark[m]And that isn't new enough so it just doesn't work19:27
noonedeadpunkYeah, or build on Centos 19:28
noonedeadpunkBut to use container you need to install podman19:28
Clark[m]Rather than try and solve this dependency problem (which exists elsewhere too) the decision was to rely on container images to bootstrap the base chroot19:28
noonedeadpunkAnd then to ensure that image I'm using is in sync and always latest19:28
Clark[m]And that solves the issue basically universally19:28
funginoonedeadpunk: not sure if it was already mentioned, but NeilHanlon is also talking about having rocky container images somewhere other than dockerhub in the future19:28
Clark[m]You don't need to ensure the image is in sync and the latest19:28
Clark[m]The dib build will update the image.19:29
Clark[m]This is why I'm saying it really shouldn't be a big deal you import the image locally once and you are done19:29
noonedeadpunkThant kind of reminds me approaches when instead of regulary update images some cloud providers just inject `apt update` to cloud-init...19:29
NeilHanlonfwiw, and this doesn't exactly solve the problem but I guess it could... Rocky also publishes our container rootfs as .tar.xz19:30
funginoonedeadpunk: building on centos gets to be problematic when you want to generate a very new debian or ubuntu and centos only has old versions of apt available19:30
NeilHanlone.g. https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-Container-Base.latest.x86_64.tar.xz19:30
Clark[m]Yes the problem exists in both directions hence the containerfile approach to avoid it in the first place19:30
NeilHanlontime to unite and make sure we all have up to date versions of one another's tools 👀19:31
noonedeadpunk^++19:31
Clark[m]noonedeadpunk this is different than having cloud init update at boot because it happens pre boot19:31
fungithe idea was to find a universal bootstrapping solution, so if centos, fedora and other distros also provide a rootfs tarball then that could be an alternative to containers i guess19:32
Clark[m]It's just a bootstrap build env. You build should then update itself19:32
fungisince all the distros seem to provide container images capable of running their bootstrapping tools, containers seemed like the solution19:32
fungirather than maintaining a different bootstrapping solution for each distro19:33
fungior, rather, different solution to setting up the pre-bootstrapping environment for each distro19:33
noonedeadpunkProbably I'm jsut not convinced about idea that 2yo container image can be jsut updated with dnf and work nicely...19:33
noonedeadpunkAnd won't be messed up with any new element19:34
noonedeadpunkespecially when package names are changed as well as repo names19:34
fungidib isn't building a container image though, it's using an existing container image to get the necessary tools to bootstrap a working rootfs chroot19:34
noonedeadpunkYeah, exactly my point, that I just was percieving dib as autonomous image building tool19:35
Clark[m]noonedeadpunk it has to be or the distro is broken as I may turn on a two year old install and update it. The only distro this has ever broken for me is Arch19:35
noonedeadpunkAnd it appears to be not anymore19:35
Clark[m]I think you are mischaracterizing things19:35
fungiseverely19:35
noonedeadpunkFor me CentOS used to be pita19:35
Clark[m]Dib has chosen to shift the tools it uses for bootstrapping19:35
noonedeadpunkIt added an external requirement for some images19:36
fungiusing the containerfile method, we don't have to worry about whether ubuntu lts-1 has new enough dnf or apt or whatever to create a chroot of another distro19:36
noonedeadpunkYeah, I got what problem you was solving19:36
noonedeadpunkI'm jsut saying that it requires now external connection to other sources, it uses more traffic and I would be surprised if bootstrap time went down19:38
fungifor some images maybe, but overall it replaced the need to have an cross-distro package of an external package manager compatible with the distro you wanted to bootstrap with needing a container image of the distro you want to bootstrap19:38
Clark[m]So I think there are two important things to keep in mind. Dib has made a decision to solve a specific problem that has long plagued it. Yes this added a new dependency, but not with any malicious intent. And you can address your issue with the new tool in a straightforward manner. Second dib is flexible and allows you to build images however you like using it's run phases design. We aren't stopping you from doing something different 19:39
fungiso for cross-distro building (what dib is primarily designed for) it replaced one dependency with another19:39
fungiinstead of needing to find an installable and working version of the package manager for the distro you're building, you need a container image for the distro you're building. it's not really an additional dependency, if anything it can reduce dependencies when you're building lots of different distro images19:40
NeilHanlonquestion: i see some elements seem to have logic which downloads qcow2 files and extracts a rootfs from them (centos, e.g.) - is that still in use?19:41
noonedeadpunkWell, in rocky-container element I saw great comment, like `Distributions vary in their podman version and various bugs related to this.`19:41
fungiNeilHanlon: likely still in use somewhere, yes. opendev doesn't and hasn't used those but they were added by people who did19:42
Clark[m]NeilHanlon: yes but they aren't minimal builds. Which is what the container file element provides19:42
noonedeadpunkAlso my point was mainly in external dependency. so instead of need for mirrors you also now need dockerfiles19:42
noonedeadpunk*docker images19:42
Clark[m]noonedeadpunk yes, but that isn't a fatal flaw and I don't think you should characterize it that way. It is a compromise 19:43
noonedeadpunkIs anything except rocky use container approach as of today?19:44
clarkbnoonedeadpunk: fedora19:44
noonedeadpunkah,ok19:44
fungialso i don't think anyone's said dib is going to start deleting other ways of building images, but this was a new solution which specifically solves a problem opendev has building lots of different distros, including very new releases of distros, from a single stable server distro which can be years old19:45
noonedeadpunkbtw, question then - do you happen to know if fcos is supported by dib?19:45
fungifedora was the one which prompted adding the containerfile method, because we couldn't install new enough dnf on ubuntu lts19:45
clarkbI have no idea if coreos can be built by the fedora elemnts19:46
noonedeadpunkhm... Ok, so basically if we work on https://review.opendev.org/c/openstack/diskimage-builder/+/805800 to pass it CI it has chances to be merged?19:46
noonedeadpunkSorry, I'm jsut trying to understand all options19:46
clarkbNeilHanlon: historically dib has had two major styles of image build. The first is one that starts with the upstream published images and modifies them. This has a couple downsides mostly in that its hard t oremove things from the base image so you end up stuck with stuff ou ma not want (for example cloud-init). The other style is to start with yum/dnf/debootstrap/apt and create a19:47
clarkbnew image from scratch in the chroot. The containerfile stuff is addressing this second style of image19:47
clarkbnoonedeadpunk: I'm not sure that ianw wants to commit to maintaining that in dib directly. You can definitely have out of tree elements though19:47
noonedeadpunkAnd I think for rocky/fedora there's also currently no option for the first style (of upstream qcow being modified)?19:48
fungiyeah, there are multiple ways of building the same distros already included in dib though, so it might be accepted. would be up to the dib maintainers to decide19:48
clarkbnoonedeadpunk: There might be for fedora. But rocky is new enough that I don't think anyone added it19:48
clarkbwe (opendev) are not fans of the first style of image and if we had our way we'd probably delete all those elements but enough people use them that that would be problematic19:49
clarkbits funny because for a long time red hatters kept saying we didn't support that method...19:50
clarkbbut we definitely did despite our interest in not doing so19:50
clarkbthat always came up with that alternative tool was discussed. libguestfs based thing19:50
clarkbeventually I stopped trying to argue and piont out the exact use case they had is already supported19:50
clarkbwe just don't actively use it in opendev because we don't want cloud init in our images19:51
fungialso the opendev sysadmins as a group aren't the ones deciding what goes into dib, it's maintained by its own core reviewers within an openstack sig19:51
clarkbyes its like 50% opendev and 50% ironic though19:51
clarkbso lots of overlap19:51
noonedeadpunkwell, I'm considering dib for building public images, so it's they'd be kind of close to the upstreamed qcow 19:52
NeilHanlonI also have personal interest in building the Rocky images with DIB, so.. there are some "synergies" here19:52
noonedeadpunkBut I was also trying to build minimal from scratch rather then full qcow19:52
NeilHanlonRight now our images are built using anaconda and it's a ... mess :) 19:53
noonedeadpunkBut do need cloud-init for sure, though there's an element for it)19:53
noonedeadpunkoh, ok, good point not to use them then :D19:53
NeilHanlon:P 19:53
clarkbNeilHanlon: we also had centos-stream images available before centos did aiui19:53
clarkbbecause with dib adding support was pretty quick but whatever upstream uses was not19:54
fungiyeah, our avoidance of cloud-init is somewhat situational. we test a lot of python software, including projects whose jobs like to install python libs into the system python context, so the slew of python libraries cloud-init depends on can create conflicts with testing19:54
clarkbfungi: that and cloud init didn't support clouds like rackspace for a long time (this was fixed though)19:54
noonedeadpunkand cloud-init is useless with fedora now... which is super annoying to be frank...19:55
fungiyeah, i mean, we could have worked with the cloud-init maintainers to fix that problem, "fixing" their reliance on third-party libraries far less likely to be accepted though19:55
clarkbbut ya you should be able to write out of tree elements at the very least. And can have conversations in #openstack-dib with the dib maintainers about whether or not any of the proposed elements make sense in tree19:56
NeilHanlonmore than happy to collaborate on that19:56
fungieven we have out-of-tree dib elements which contain specific things for our environment19:56
clarkbmostly I think its a problem of having 3 different ways to express the same thing creating confusion and maintenance overhead. But if there is user interest and people willing to help then thats probably addressable19:56
noonedeadpunkanyway thanks, I will try to think about options... Maybe will attempt to revive patch with building rocky from dnf... as it should work jsut nice as well as c9s does atm... But given fedora... doh... Probably we will end up having local copy of dockerhub or smth, which is annoying, but seems not much options19:57
clarkbnoonedeadpunk: again I really don't think you need a proper mirror19:57
clarkbyou just need to seed your local image repo with the base images19:57
clarkbskopeo can do this iirc19:57
fungii have a feeling a lot of distros are going to start putting container images in other/additional registries soon anyway19:58
clarkbfungi: ya I don't think the issue is docker hub as much as the assumption that they will need torun a docker registry cache/mirror19:58
noonedeadpunkI can already feel how messy that could be19:58
clarkbinstead you just do skopeo something something this image upstream to my local disk please19:58
NeilHanlonnoonedeadpunk, btw since you mentioned FCOS... http://dl.rockylinux.org/pub/sig/8/ostree/x86_64/isos/19:58
noonedeadpunkclarkb: I will read about skopeo for sure19:59
fungiwe do maintain a caching dockerhub proxy, though it's mildly convoluted mainly because of how braindead their api is19:59
noonedeadpunkNeilHanlon: I was mentioning it mainly because of magnum being dependant on it20:00
fungiand also because we're trying to get by with apache mod_proxy instead of something more full-featured like squid20:00
noonedeadpunkI'm not sure rocky would be compatible with it?20:00
clarkbnoonedeadpunk: oh magnum should stop relying on fedora but no one listens to me on that one :)20:00
noonedeadpunkI can't agree more on that....20:00
clarkbnoonedeadpunk: the problem there is a cloud gets deployed with magnum and at that point the fedora release magnum relies on has at best 13 months of support. But then you still have users running k8s clusters more than 13 months later20:01
* NeilHanlon adds it to the list of things to put rocky in20:01
fungimagnum's going to need to find a way to make old magnum versions support new fedora versions if their plan is to stick with fedora20:01
fungii tried to point that out to them on the ml recently20:01
clarkbthis is one of the major reasons we (opendev) do not use magnum20:01
NeilHanloni know centos stream is doing releases of ostree/coreos stuff now20:01
NeilHanlonthat might be their end game20:01
ianwcorvus: yeah, sorry, i left that stack in a bit of a state as it was getting late here too.  sorry i mentioned in https://review.opendev.org/c/opendev/system-config/+/877859 the /v1 thing was a red-herring20:03
ianwwhen i was looking at the apache logs, i at first didn't realise the first "ping" podman does is a GET to just /v220:03
ianwwhen that fails, it seems to fall back to /v1/_ping 20:04
ianwso, before i realised that, i thought maybe we had to proxy /v1 to get the first ping to work or something.  but agree we should drop it20:04
ianwoh, and i set the hashtag just as a test of what zuul does to hashtag events actually, which is a separate thing ...20:11
clarkbI think it just ignores them currently?20:12
ianwnoonedeadpunk: i do get that building in an isolated environment the containerfile's dependency on pulling from the registry makes things more difficult.  tbh i don't think we really considered that case20:22
ianwi think cross-distro minimal builds are probably the biggest non-starter20:26
ianwi will say that it has not been easy running podman inside docker in the build environment, it fairly constantly breaks20:28
ianwultimately we just make a tarball of that.  so i wonder if perhaps something like having the ability to take a pre-cached tarball might work better20:29
ianwi've vaguely thought about this as an escape hatch for opendev infra too, should we get to the point we can't get podman running inside the nodepool-builder containers20:30
ianwessentially pre-cache the root image and then use that20:30
ianw... which in the end is, i guess, basically what a .qcow2 is ...20:31
noonedeadpunkWell, we're running dib with zuul, so we technically can select image where dib will run20:34
noonedeadpunkIt creates some chicken-egg though...20:34
clarkbianw: fwiw when containerfile stuff was first envisioned I thought that is what it was going to be. fetch the image and extract it then execute dib steps. But we modified that to fetch the image using Dockfile which can/does run a docker image build before the extraction step20:35
noonedeadpunkAnd root images usually are way more minimal then any qcow is20:35
clarkbianw: if we stopped doing Dockerfile evaluation we could still use the container image as the "tarball" and extract it to the chroot without needing a container runtime20:35
clarkbI think skopeo can do this out of the box?20:36
opendevreviewMerged opendev/system-config master: Remove v1 proxy from registry.zuul-ci.org  https://review.opendev.org/c/opendev/system-config/+/87798920:41
ianwright, i don't think we *need* to do anything in the dockerfile, it's just convenience 20:42
ianwdoes skopeo want to setup any sort of cgroups or namespaces etc. to do that?  it's the nesting of all that which causes problems.  if it's just pulling a manifest via http and dumping it to a .tar.gz that seems much more nested-container friendly20:43
ianwand yeah, apart from just being huge, .qcow2's have traditional been a source of all sorts of problems too.  files moving around, packages changing, bootloader layouts changing, etc.20:46
clarkbianw: I'm not 100% sure, but it definitely can write an image out to a filesystem prefix iirc20:46
ianwso the smallest target with the least changing parts is the goal (as has been discussed above, i know that's not new info :)20:46
* NeilHanlon has a master plan which involves a digraph of image dependencies and a single command that builds every variant of the images Rocky makes20:47
clarkbhttps://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md#examples20:47
corvusianw: no worries about the state of the stack -- thank you for the test!  and sorry for the regex error20:48
clarkblooks like you end up with each layer in a different tarball then presumably you can extract them in order to get a flattened filesystem result?20:48
clarkbinfra-root I've just updated the meeting agenda with the updates I had. If you've got something to add now is a good time :) I'll send it out later today20:50
corvusianw: clarkb fwiw, "use dockerfile syntax as an alternative to writing elements" was definitely a goal for the containerfile stuff.  i think that's worth keeping, so if you want to mod it to do something else, maybe a new element for that would be in order?20:52
clarkboh yup I think the Dockerfile interpretation has been useful. I can just see how the runtime requirements might make an alternative a good choice for the main distro elements20:53
corvus++20:53
clarkbit is a much more succint syntax for editing an image and if you as a user want/need that it is a useful tool20:53
ianw$ skopeo copy docker://busybox:latest docker-archive:archive-file.tar:busybox:latest20:54
ianwFATA[0000] initializing destination docker-archive:archive-file.tar:docker.io/library/busybox:latest: docker-archive doesn't support modifying existing images20:54
corvusyes, i have used it to build like 5 line dockerfile images20:54
corvusianw clarkb : the nodepool-in-zuul work will also eventually address the cross-platform build case by running dib in standard zuul/nodepool vms20:54
corvusso that's another escape valve we should have20:54
ianwthat's true, but also i don't know if we have the resources to keep dib working as a build *host* on many different platforms20:55
clarkbcorvus: that assumes that everyone is running dib in zuul though20:56
corvus(i'm expecting to start in earnest on the zuul work after the openstack state machine driver finishes stabilizing)20:56
clarkbI think for zuul users its an outlet, but dib has users that run it outside of zuul and that is a use case we shouldn't drop20:56
corvusoh sure, i was responding to ianw's opendev escape valve20:56
clarkbah20:56
corvus(so, in the future, opendev won't need to run nb03 or whatever because it can just run dib on a real arm vm)20:57
corvusianw: point taken; i think all options are open to us this way.  we can build fedora-x86-via-containerfile on a debian-x86 zuul vm;  and we can build ubuntu-arm-via-debootstrap on a debian-arm zuul vm.21:01
corvus(iow we can cross-build with dib where it makes sense -- keep the supported host systems small; and native-build where it's near-impossible to avoid, like arm)21:02
fungithough jobs which need dib's target platform to run dib to build new images does pose a bit of a catch-2221:04
fungialbeit we solve similar problems manually now21:04
corvusfungi: yeah, i anticipate we would use cloud images for the image build jobs -- either in all cases, or perhaps only when bootstrapping.  depending on whichever we think is easier.21:08
clarkblast call for meeting agenda updates23:39
ianwfungi: if you have a sec, would you mind checking https://review.opendev.org/c/openstack/project-config/+/875997/3 which is a linter update for submit-requirements.  fairly straight forward i think, but a double check would be good incase i missed a corner case or something23:41
fungiianw: the function keyword got moved out of alpha order in the list, was that intentional?23:45
clarkbbecause I'm looking at the agenda. fungi do you know if it is appropriate for openstack to adopt xstatic libs given their apparent deal with the moinmoin folks?23:47
clarkbit seems like maybe xstatic libs should not be in openstack if they are comanaged by people who hvae zero interest in gerrit?23:47
ianwfungi: there is no reason for that.  i think it just got mixed up when i shuffled the changes around23:47
fungii think it depends on the lib23:47
fungisome are maintained exclusively by the horizon team23:48
clarkboh I see it wasn't a universal shared setup?23:49
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : check for function/s-r in normalize  https://review.opendev.org/c/openstack/project-config/+/87599723:49
opendevreviewIan Wienand proposed openstack/project-config master: gerrit/acl : check for capital booleans in normalize  https://review.opendev.org/c/openstack/project-config/+/87757123:49
opendevreviewIan Wienand proposed openstack/project-config master: openstack/release : return to non-blocking submit rule  https://review.opendev.org/c/openstack/project-config/+/87772123:50
clarkbany idea if this one that is being renamed is shared? I guess that is what we need to sort out23:50
clarkbok the agenda is official and will be in your inbox momentarily23:53

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!