19:02:34 <clarkb> #startmeeting infra
19:02:34 <opendevmeet> Meeting started Thu Mar 16 19:02:34 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:34 <opendevmeet> The meeting name has been set to 'infra'
19:02:59 <clarkb> #topic What do we do about docker deleting free orgs
19:03:05 <clarkb> #link https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo
19:03:55 <clarkb> Long story short Docker will remove access to free orgs on April 14. I believe this means we won't be able to manage our images including push new version of them to opendevorg/* and zuul/*
19:04:22 <clarkb> If I've read their comms properly the public images already there will be accessible for 30 days after April 14
19:04:41 <clarkb> This puts us in a position where we need to find a new home for the images we host at opendevorg/ and zuul/
19:05:01 <clarkb> The etherpad linked above attempts to capture what our options are
19:06:05 <clarkb> We can A) Move to a different free public registry like quay or github etc B) Host our own registry C) Apply for docker's DSOS open source program D) pay Docker to keep our existing orgs as is. THere is also a E) which is do more than one of options A-D
19:06:29 <clarkb> informal discussion between different infra-root leave me to believe that none of us are too thrilled about C or D
19:06:41 <fungi> and for completeness, f) stop hosting/publishing images
19:06:59 <NeilHanlon> ... and move to somewhere and raise goats
19:07:15 <clarkb> yup that is technically an option though likely to be the most effort of all of them due to our reliance on containers for our service deployments
19:07:21 <fungi> goat farming has started to become tempting
19:07:31 <clarkb> considering all that I believe A and B are the serious contenders here.
19:07:56 <corvus> for clarity: if we did want to do (F), i think we could still run our own tiny registry for our own operational needs
19:08:00 <corvus> i'm not pushing for (F)
19:08:20 <clarkb> My personal inclination is to avoid running more services if we can. To this end I created a quay account and tested things there. I believe that quay, despite some oddities, would be a viable option for us
19:08:45 <fungi> your progress reports on quay experience in #opendev seemed promising
19:09:19 <corvus> yes, based on that progress, i like: A (quay); B, F in that order.
19:09:45 <fungi> and while it doesn't get us away from "some vendor-run registry that could be pulled out from under us at any moment" our communities at least have fairly strong relationships with red hat, unlike docker inc
19:09:48 <corvus> [but if someone has a herd of goats ready to go, that really does make F more tempting]
19:09:56 <clarkb> I think the rough setup I would advocate there is that infra-root create personal accounts via https://quay.io (you click sign in then register an account). We would then create an org for opendevorg/ and another for zuul/ and add our personal accounts to those orgs
19:10:21 <clarkb> One thing to note is that orgs appear to require an email address and that email address must be distinct
19:10:39 <fungi> do + extensions count as distinct?
19:10:43 <NeilHanlon> they do
19:10:44 <clarkb> which means we'll probably want the opendevorg/ email addr to be something like infra-root+quayopendevorg@...
19:10:54 <fungi> good enough for me
19:11:04 <clarkb> and then I'll defer to corvus on what he feels is most appropriate for zuul/ but could do similar there
19:11:11 <NeilHanlon> that is the setup, too, that we use for Rocky Linux, which seems to work well
19:11:23 <corvus> that's a little awkward for other orgs, in that it requires some interaction with infra-root
19:11:31 <clarkb> NeilHanlon: thats good input that others are able to do similar
19:11:34 <clarkb> corvus: agreed
19:11:49 <corvus> but that's probably scalable to the number of orgs we're contemplating for the next little while
19:12:10 <clarkb> corvus: yup and it sounds like a lot of other ajdacent orgs/communities are already on quay and have solved this themsevles
19:12:21 <corvus> (or we could ask projects like zuul to get their own secret shared mailbox, but that's a lot of work that's probably not necessary)
19:12:23 <fungi> it can certainly be handled the same way as it is for dockerhub: let community leadership create accounts, set up orgs, encode secrets for zuul, done
19:12:44 <corvus> fungi: right, just the email address thing is new
19:12:51 <clarkb> Then once we have an org we could create an appliction/robot within that org for api access and push credentials. This is what our CI jobs would be configured with
19:13:06 <fungi> yeah, they'd need e-mail addresses. most of them will probably create a gmail inbox (eww) and call it done, but whatever
19:13:32 <corvus> clarkb: did the email get used for anything?  confirmation dance?
19:13:34 <clarkb> Another oddity is that if you docker image push to create a new image in quay it is by default private even if your account is private only. You can manually set this as public via the web UI or emilienM wrote a script and blog post on how to do it via the api
19:13:44 <clarkb> #link https://my1.fr/blog/moving-container-images-from-docker-io-to-quay-io/
19:14:15 <clarkb> corvus: they didn't ask for confirmation now that I've double checked. When I set a docker cli password on the account to docker image push they sent me an email to alert me that this event occurred
19:15:09 <clarkb> oh account sign up does require a phone number, email address, and physical address. I don't know how much validation of those fields they do. They have not called or texted my phone number
19:15:48 <fungi> my assumption is that those and handed directly to your friendly red hat sales team for followup
19:16:00 <clarkb> but if you sign up via https://quay.io there are fewer questions to answer than when you sign up via sso.redhat.com so I suggest signing up via quay
19:16:18 <corvus> okay, so the org email address is basically something that's unused for now, but could be important later.  i think letting our orgs use infra-root+ORG@.. is reasonable in that case, and better than asking people to make gmail accounts and losing the passwords
19:16:30 <clarkb> corvus: wfm
19:16:47 <fungi> yeah, i'm cool with it
19:16:59 <clarkb> it is possible that the org creation process is different than the individual user process and they will validate it. I can't be sure of that yet since I did a slightly different process
19:17:43 <clarkb> Another thing I noticed is that quay's security scanning is very aggressive to the point of not being useful imo. It calls out a bunch of issues with up to date debian packages
19:18:22 <clarkb> But ya I would say that if others are comfortable creating personal accounts for quay/red hat we can start there then create orgs and then figure out what zuul job updates are required to migrate our images over
19:18:49 <clarkb> everything I've seen indicates it should be possible to automate what we need in order to create new images on quay and push to them via zuul
19:19:34 <corvus> sounds like we have 2 tasks there: 1) do the create public repo api call at the start of the promote job; and 2) update the docker-specific tag promote to something that works with quay
19:19:40 <corvus> i can volunteer for #2
19:19:50 <fungi> i'm assuming their security scanning is really only accurate if the images are rhel/centos/fedora based
19:20:10 <clarkb> corvus: and 3) create org in quay for opendevorg and 4) create org in quay for zuul
19:20:27 <clarkb> I'm happy to do 3 once we have general agreement and more than one user account that has managed to sign up successfully
19:20:31 <corvus> oh yeah that too, sorry i was just thinking of job related work
19:20:32 <NeilHanlon> in my experience, it's on par with most other scanners... which is to say: mediocre on a good day
19:20:40 <clarkb> corvus: I can do 4 too fwiw but thought you might prefer to do it
19:21:05 <corvus> yeah i can do #4
19:21:05 <fungi> do we want to push in parallel to both dockerhub and quay for some period of time, or just make a hard cut-over? i guess for opendev's server images it can be a hard cut. other projects will likely migrate at their own pace
19:21:10 <clarkb> I think 1) becomes easier to test once 3 and/or 4 are done since we'll be able to create an application that can talk to the API at that point
19:21:25 <clarkb> I'm willing to drive 1 and can holler for help if I need it
19:21:38 <corvus> my read on the zuul community is that we probably don't really need to get specific feedback and whatever we decide here will work for zuul.
19:22:12 <corvus> (specifically, everyone i've talked to in the past about this either (a) would prefer quay to dockerhub or (b) doesn't care)
19:22:12 <clarkb> fungi: I think I'd just hard cutover the publication location, then we update opendev's pull locations
19:22:27 <fungi> presumably zuul will do an announcement at least, opendev's server deployments likely don't even need that
19:22:37 <clarkb> its not like the docker hub images are going away immediately and if we need to update for security updates anyway we're doing a build and publish and pull anyway
19:22:50 <corvus> fungi: re announcement, yep, at least one
19:23:13 <clarkb> we also need to decide if we care about rebuilding imgaes multiple times or not. If we do care we should start with our base images and then their consumers
19:23:28 <clarkb> but I think we can rebuild if those move later and it will be fine too
19:23:33 <corvus> i'd like to float the idea of deleting our repos from dockerhub shortly before we lose access.  that way there isn't an "official looking" repo serving content that we don't control.
19:24:18 <clarkb> thats a reaosnable thing to do. I wonder if that would impact their promise to not allow name reuse. eg if you intentionally delete it vs them doing it
19:24:34 <clarkb> but I do agree not having stale images that we know are bad is preferable to potentially having bad images later
19:25:11 <fungi> we could always delete the orgs and then try to squat the same names with personal accounts
19:25:27 <fungi> but yeah there could be some delay in them releasing the names, if they do
19:25:31 <clarkb> it is probably sufficient to just delete our existing images within the orgs and let them do what they are going to do to the org
19:25:36 <corvus> i think they're talking about the org.... i assume that would stay.  we could always create a new repo so there's at least one repo in the org.
19:26:03 <corvus> "dear-docker-please-do-not-release-this-namespace"
19:26:06 <fungi> oh, got it, delete the repos but not necessarily the orgs
19:26:08 <corvus> fungi: i'm not sure that's possible (deleting orgs)
19:26:10 <NeilHanlon> someone in the github issue commented they removed an older org the had and the namespace wasn't cleaned up within 30 mins
19:26:22 <NeilHanlon> corvus: throw in some zero width spaces or something to make it hard
19:26:30 <corvus> lol
19:26:52 <clarkb> I'm going to try and summarize where I think the plan is headed on the etherpad
19:27:02 <NeilHanlon> just use chatgpt /s
19:27:04 <clarkb> oh looks like corvus already is thanks!
19:27:06 <corvus> oh i started a section at the bottom
19:27:20 <NeilHanlon> (just wanted to make fungi's blood pressure rise ;) )
19:27:36 <fungi> they make medication for that these days
19:27:54 <corvus> new task on that list: copy existing repos and all tags to quay.io -- i think that would be a good idea, yeah?
19:28:21 <clarkb> corvus: It was something I wondered about. I think for opendev its far less important but mostly because we just have :latest on most images except gerrit
19:28:29 <clarkb> But for zuul I think that makes a lot of sense
19:28:39 <clarkb> and if we are doing it for zuul we may as well do it for opendev
19:28:57 <corvus> true; that might be something i manually do for zuul then
19:29:32 <clarkb> emilienm's blog/script will be useful for that
19:29:38 <clarkb> since that is essentially what it was for
19:29:41 <corvus> ah cool
19:30:59 <fungi> is that linked in the pad?
19:31:08 <clarkb> fungi: not yet I'll add it
19:31:13 <fungi> thanks
19:31:22 <clarkb> heh corvus wins again
19:31:37 <fungi> i saw it in #opendev, just want to make sure we don't have to go digging it back out of our irc logs
19:33:45 <clarkb> ok I think we've got a good foundation for zuul/ and opendevorg/ things. This doesn't address what we'll do with other things like zookeeper mariadb etc
19:33:55 <clarkb> or graphite/grafana
19:34:17 <ianw> i don't want to be annoying, but i just wonder if we are setting ourselves up here to once again be disappointed by basically doing the same thing over again.  since we are have a rather large consortium behind us, perhaps we should consider using some of the resources available to the project to start this on a paid plan, and then we seem less likely to be back here having another meeting
19:34:19 <fungi> which will depend on what decisions their communities make, i suppose?
19:34:42 <clarkb> there are two classes of image there. THe first are the "docker official" images like python, zookeeper, mariadb, debian, ubuntu and it sounds like most people expect docker to continue to maintain those
19:34:54 <clarkb> but graphite and jitsi meet we may need to watch for new image locations for those as well
19:35:25 <clarkb> ianw: I think we can make a decision like that further down the road (there is a "upgrade" button to do that in account settings and I assume org settings)
19:35:40 <fungi> ianw: assuming you mean pay for quay, we can talk to the foundation executive team and board of directors, but it may get weird with red hat also being a member of the organization
19:35:45 <clarkb> ianw: my concern with trying to start that way is it will slow down the move while we sort out if that is ok or even appropriate and who to pay and so on.
19:36:03 <clarkb> also once you sign up its really clear they don't expect open source orgs to pay
19:36:15 <corvus> also before you sign up that's pretty clear
19:36:19 <clarkb> in particular when you create a new org the free tier is public images only and labeled "open source"
19:36:58 <clarkb> I don't think any part of this plan prevents us from doing that later
19:37:33 <fungi> as far as paying for things, i wouldn't be surprised if red hat paying for membership in a non-profit which in turn buys services from red hat might have tax law implications, so will want to make sure we talk to the right people to nail down all the gotchas if we want to go that direction
19:39:08 <ianw> nothing really stops us paying docker either, but i feel like the major concern with that is not really technical but more "they bait and switched us".  which, ok, but if RH starts imposing rate limits etc. we do have to sort of say "well we walked into that"
19:39:19 <corvus> putting on my free software hippie hat for a moment -- if the norm in open source communities becomes "pay to host container images" then i do think a serious consideration of (F) -- don't publish images just like we don't publish .rpm or .deb files is worth having.  "docker build" is not actually hard for end users to run.
19:40:37 <clarkb> ianw: yes we could pay docker hub as well and there is potential for this or something like it to occur with quay as well. I've heard an explicit interest in not paying docker from other infra rooters. And I think those opinions are valid
19:40:49 <fungi> and we do already operate on a ton of donated services which could be pulled out from under us at any time, so free dockerhub or quay aren't all that different from those
19:41:02 <clarkb> Where it gets weird paying quay is they explicitly sya they don't want money for what we are planning to do
19:41:44 <ianw> i don't really want to argue this, i know everybody understands what we're getting into.  quay seems fine and i'm happy to be on board with helping switching.  just if it turns out that unlimited free repos isn't sustainable, i don't think we should be surprised, again
19:41:55 <clarkb> ianw: agreed
19:42:03 <clarkb> fool me once etc
19:42:24 <corvus> ianw: i agree too.  and i don't think we have any implied or explicit SLA to our users here either.  moving stuff like this around quickly, with little notice, and even having interruptions in service should be expected, i think.
19:42:30 <fungi> it's definitely worth calling out, and i wouldn't be surprised if quay is getting inundated with a flood of dockerhub evacuees too which might change their viewpoint on financial matters
19:42:59 <clarkb> the short timeline involved makes it difficult for me to say we should do something like f which is almost certainly going to be more effort
19:43:31 <corvus> yeah, f is a bit "change how people think" which, you know, we don't shy away from, but also tends to have an extended time line
19:43:42 <ianw> ok, the fact that we even discussed this, which i don't feel like was ever discussed with dockerhub as it was pretty much assumed it would remain around forever, is probably enough
19:44:21 <clarkb> ok cool. And please do bring up concerns if we run into anything unexpected and new
19:44:25 <corvus> i may have joked about dockerhub pulling the plug while drunk... but yes, i don't recall a serious discussion :)
19:44:33 <fungi> fwiw, i don't assume any gratis service will be around/available forever. we've seen enough examples ourselves that's not reasonable
19:44:59 <fungi> transifex did the same to us years ago
19:45:06 <corvus> yep, sounds like we're eyes wide open on this
19:45:23 <NeilHanlon> fwiw, i did start conversations with some of Rocky's sponsors about the possibility of a community-owned/operated registry. of course: with all the problems which come along with that... but, I think it's not a bad idea in general to have something without a commercial interest...
19:45:24 <clarkb> re other images. I'm thinking for now we should maybe keep our ears open for news on any changes to those images. And update quickly assuming they move.
19:45:55 <clarkb> But I think it may be too early to say "we need to build our own ubuntu images from scratch" and I think it highly unlikely that docker just deletes those
19:46:10 <clarkb> er I guess it would be debian since that is what we predominantly use for our container base
19:46:59 <clarkb> #link https://github.com/jitsi/docker-jitsi-meet/issues/1502 jitsi meet move to github container registry will happen soon
19:47:50 <clarkb> it wouldn't surprise me if grafana stays on docker hub
19:48:44 <clarkb> seems like discussion is slowing down. Is there anything else we want to discuss around the docker hub org deletion?
19:48:58 <clarkb> Maybe we've got enough sorted out to beging to make progress then resync on tuesday
19:49:07 <clarkb> *to begin
19:49:11 <corvus> sounds like we have consensus (maybe even complete agreement) and a task list?
19:49:20 <fungi> i agree on the task list
19:49:58 <corvus> i concur
19:50:03 <fungi> #link https://wiki.debian.org/Cloud/CreateDockerImage if we do decide we need to build our own
19:50:18 <clarkb> fungi: we can also fork what docker is/was doing too
19:50:49 <clarkb> (I don't have an opinion on the best method but they are all open source and forkable aiui)
19:51:18 <NeilHanlon> #link https://github.com/docker-library/official-images/ "Library"/"Official" image sources for many images that can be rebuilt
19:51:45 <NeilHanlon> this is "just" references to other repos, so in theory they should be reproducible
19:52:39 <clarkb> sounds like that is probably it for now. We can sync up Tuesday at our normal meeting time
19:52:47 <fungi> i guess an #agreed is in order?
19:54:00 <clarkb> sure. How about #agreed Move opendevorg and zuul container images from docker hub to quay.io. This will require modifications to zuul jobs and management of accounts/orgs in quay.io. We recognize there is a risk that we may end up in the same spot later down the road but this seems the least painful option for today.
19:54:12 <clarkb> ianw: ^ want to make sure I'm representing what you've said accurately
19:54:22 <fungi> wfm
19:54:35 <NeilHanlon> sgtm
19:55:35 <corvus> ++
19:55:56 <clarkb> #link https://www.docker.com/blog/we-apologize-we-did-a-terrible-job-announcing-the-end-of-docker-free-teams/
19:56:30 <clarkb> "We’d also like to clarify that public images will only be removed from Docker Hub if their maintainer decides to delete them."
19:56:54 <clarkb> I thought we had waited long enough for clarification from them. However reading this I don't really think it changes my inclination for the plan
19:57:08 <clarkb> maybe makes it slightly less urgent? but its still not super clear so I'm more happy to move to something that is clearer
19:57:08 <NeilHanlon> obviously i'm external to opendev.. but.. IMO: too little, too late
19:57:43 <corvus> i think the only thing that warrants a reconsideration is: Can I migrate to a Personal account?
19:57:44 <corvus> You can migrate to a Free Team organization to a Personal account by opening a support ticket. No action will be taken against your account while your ticket is being processed.
19:57:59 <corvus> that was a question we had that was unanswered until now.
19:58:18 <corvus> but i'm still ready to proceed as agreed.
19:58:23 <fungi> NeilHanlon: you don't seem all that external to me. you're participating in our discussions and helping support our testing efforts
19:58:31 <clarkb> me too after skimming this.  Idon't think it functionally changes anything for us
19:58:53 <corvus> aside from that clarification, it's basically "here's what we said but with more words"
19:59:43 <NeilHanlon> fungi: well, thank you for that :) I'm glad to be involved with an org like opendev, and try to model what we do at Rocky after y'alls model
19:59:49 <fungi> "We have assigned more staff to promptly review all applications." i guess that's their "we're sorry we ignored all your applications previously"
20:01:10 <clarkb> ianw: fwiw I'm happy if you nack that draft as well :)
20:01:45 <clarkb> Maybe I'll leave it out of the official meeting notes just to avoid implication everyone ack'd it. But the log captures that we have general consensus if not complete consensus
20:02:40 <clarkb> ya I'll leave it there. Thank you everyone for the input.
20:02:45 <clarkb> #endmeeting