19:01:13 <clarkb> #startmeeting infra
19:01:13 <opendevmeet> Meeting started Tue Mar 21 19:01:13 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:13 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:13 <opendevmeet> The meeting name has been set to 'infra'
19:01:20 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/QH5SYKQ7JY6DF65R2NQQ7RTLONFQQIXR/ Our Agenda
19:01:28 <clarkb> #topic Announcements
19:01:48 <clarkb> OpenStack is doing a release this week (should happen early tomorrow relative to me)
19:02:00 <clarkb> Please refrain from updating base jobs or restarting zuul if we can avoid it
19:02:32 <fungi> though we do have a release job configuration change to merge later today in support of that
19:02:39 <clarkb> is that the gpg key update?
19:02:57 <fungi> #link https://review.opendev.org/877552 Temporarily remove release docs semaphores
19:03:03 <clarkb> oh that one
19:03:12 <fungi> i'm un-wipping it now
19:03:16 <clarkb> we don't need the gpg key update to be clear? that happens after the release?
19:03:37 <fungi> after, yes. i've set that up for monday
19:03:43 <clarkb> cool
19:04:35 <clarkb> And next week is the ptg. Similar story around avoiding making changes but to a different set of tools. Etherpad and meetpad in particular for the PTG
19:05:18 <clarkb> I suspect at least some of us will be participating in he PTG as well so we may want to skip next week's meeting. But I'll wait for more of the schedule to congeal as I'm not sure yet if it poses a problem
19:06:04 <fungi> our usual meeting time is during the break
19:06:11 <fungi> just to be clear
19:06:16 <clarkb> yes, but if you've spent 4 hours in meetings ou may need a break
19:06:23 <clarkb> I know I would :)
19:06:24 <fungi> yes, i likely will do ;)
19:06:53 <clarkb> #topic Topics
19:07:05 <clarkb> #topic Docker Free Team Organization Shutdown
19:07:24 <clarkb> last week docker announced that the free team organization setup we rely on for our docker images is going away on April 14
19:07:50 <clarkb> We had a meeting on thursday to discuss options and create a plan to move forward. Where we ended up was to move our container images to quay.io
19:07:58 <clarkb> #link https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo Notes on the situation and planning
19:08:11 <clarkb> We've taken some decent notes through the whole thing which can be found there
19:08:18 <clarkb> #link https://quay.io/organization/opendevorg Will become the new home for opendev images
19:08:40 <clarkb> I've created a new org on quay and I think all infra-root but frickler have been added? Happy to add frickler when notified of an account name
19:08:55 <clarkb> There are also some job/role updates that we need to make this move happen
19:09:00 <clarkb> #link https://review.opendev.org/c/zuul/zuul-jobs/+/877834 Role to precreate public images on quay.io
19:09:31 <clarkb> If you simply push to quay.io you end up with a private repo by default. The way around this is to precreate the repo which this role does (or you can switch it to public after pushing with the private default)
19:09:40 <clarkb> #link https://review.opendev.org/c/zuul/zuul-jobs/+/838919/ Updates to container roles to make them replace docker roles for quay.io
19:09:53 <clarkb> and this stack of changes updates the generic container roles to add functionality that was in he docker roles that we need
19:10:09 <clarkb> corvus: and to be clear the reason we can't use the docker roles is they have hardcoded docker hub as the only destination?
19:10:25 <clarkb> corvus: after reviewing this stack I wondered why we couldn't just tell the docker roles to talk somewhere else
19:10:29 <corvus> yes, they are designed specifically for docker hub
19:10:37 <clarkb> thanks, that helps
19:10:47 <corvus> mostly the promote role uses docker specific apis
19:11:07 <clarkb> on the whole I think we've got a reasonable plan and there is progress. Reviews and help are welcome
19:11:27 <clarkb> On top of those changes we'll need to update our jobs themselves to make use of the updated roles
19:11:34 <corvus> the other roles are supposed to be pretty similar, just with a few minor assumptions removed and defaults changed
19:11:35 <corvus> however
19:11:37 <corvus> the build-container-image role does not appear to have any support for multi-arch builds
19:12:10 <corvus> is there anyone interested in multi-arch support who is willing to look into bring build-container-image up to par?
19:12:36 <clarkb> corvus: the main use of that today in our world is nodepool-builder right?
19:12:46 <clarkb> I think opendev and zuul don't multiarch anywhere else.
19:13:05 <corvus> i think so; but is there base image support to facilitate that?
19:13:07 <fungi> which role are we building our multi-arch containers with currently?
19:13:18 <corvus> fungi: build-docker-image
19:13:32 <clarkb> corvus: oh yes, the base images are also built for arm64 and x86_64
19:13:59 <clarkb> I should be able to take a look at that if no one else is interested. I've poked at our base images and the multi arch stuff in the past
19:14:02 <ianw> i'll have to context switch it all back in, reviewing those changes should help with that, but i can probably take a look at it
19:14:18 <clarkb> ianw: oh cool. and ya I would have to context swithc it in. buildx and manifests and all that
19:14:21 <fungi> okay, so we'd need to port the multi-arch bits from build-docker-image to build-container-image sounds like
19:14:23 <ianw> ++ i'm sure we can figure something out
19:14:38 <corvus> thanks; i'm willing to pitch in to help, but it's too much for me alone right now.
19:15:02 <fungi> thanks for calling it out, i wouldn't have thought to check
19:15:03 <clarkb> fwiw I think some of the reason this was missing is buildah didn't support multiarch buidls until more recently
19:15:23 <clarkb> however, you can run docker with the container roles too (and in fact I think that is what opendev at least should do for max compatibility for now)
19:15:34 <clarkb> I think changing to buildah is a possibility but something we shouldn't worry about now
19:15:35 <fungi> buildah is a non-docker-specific buildx replacement?
19:15:38 <corvus> agreed re using "docker" with the "container" roles
19:15:57 <clarkb> fungi: buildah is the podman image builder. docker does both the container runtime and image building via one interface but podman and buildah split it out
19:16:32 <corvus> they both use buildkit as the underlying implementation for the new stuff though, so convergence is hopefully possible
19:17:19 <clarkb> ok cool ianw I'll probably try to look at this late today/early tomorrow for me
19:17:36 <clarkb> anything else related to the container registry move?
19:17:56 <corvus> i've completed work on a redirection system for zuul
19:18:12 <fungi> that was an awesome idea, btw
19:18:17 <corvus> so i will propose that zuul use "registry.zuul-ci.org/..." as the canonical locations for images
19:18:59 <corvus> it's a pretty easy option for any other domains that want to do something similar
19:19:08 <clarkb> ya we should probably do similar for opendev.org.
19:19:51 <ianw> it's probably worth advertising it somehow?
19:20:03 <clarkb> ianw: advertising that redirects work?
19:20:09 <corvus> ianw: can you elaborate?
19:20:17 <corvus> oh that it's a thing that other projects can do?
19:20:31 <corvus> like registry.kolla could be a thing if they wanted?
19:20:32 <fungi> like a blog post about the design or something?
19:20:34 <ianw> yeah, particularly like openstack projects might want to consolidate, etc.
19:21:01 <corvus> okay i can write something up
19:21:13 <corvus> maybe send it to the service-discuss list...
19:21:15 <ianw> yes, i already saw one similar blog post rise to the top of HN and i think they got it wrong too (might not have worked with podman)
19:21:21 <corvus> and then copy or link other lists to that?
19:21:34 <corvus> i suppose i could write an actual blog post
19:21:34 <fungi> for kolla specifically, it sounded like they were already deploying to multiple registries and were just planning to drop dockerhub, but there are potentally projects even outside opendev that could benefit from seeing that the idea is simple and works
19:22:25 <fungi> s/deploying/publishing/
19:22:40 <clarkb> ya its not something people may realize is possible so getting the word out could be useful
19:22:49 <ianw> yep, just think it would to get the word out that it's a thing you can do, and here's an example
19:23:27 <corvus> i think there's 2 audiences: how to do this in general; and how to do this in opendev
19:23:33 <corvus> for the latter, a pointer to code/changes might be in order
19:23:58 <corvus> #action corvus write up blog post/email/something to share registry redirect idea/configuration
19:24:18 <clarkb> ++
19:24:25 <fungi> probably solveable by just linking the opendev examples in the general writeup, but yeah i agree it may be two audiences for one post even
19:24:35 <corvus> if we're happy with the idea for opendev, i can try to make a clean series of changes for that
19:24:50 <corvus> (since the git history for zuul is not clear due to experimentation)
19:24:51 <clarkb> corvus: I think I like the idea since it should reduce some over the necessary changes if we do this again
19:25:13 <corvus> k.  expect that later this week :)
19:25:18 <fungi> yes, also if we're going that route we should do it before we switch registries, in order to save ourselves additional patches to review
19:25:47 <clarkb> exactly this is the best opportunity for doing it
19:26:36 * fungi prepares for docker inc to break the official docker client's ability to follow redirects
19:26:55 <clarkb> ok lets move on as we're almost half out of time and only one topic in. THank you everyone who has helped push this along. Its going to be a bit of work and won't happen overnight but we're making progress
19:27:05 <fungi> yes, thanks!
19:27:05 <clarkb> #topic Bastion Host Updates
19:27:25 <clarkb> I mostly wanted to call out that fungi had to boot the old bastion briefly to grab some gpg data off of it. Now is a great time to brainstorm if anything else needs moving
19:27:45 <clarkb> I suspect that given how long its been and we've only run into this one issue that there isn't. But the sooner we address anything else missing the better
19:28:08 <fungi> yeah, it's not still booted, was only up for a few minutes, but we should be very sure before we delete the server so that we don't need to restore from backups later
19:28:30 <ianw> ++ the other thing is, if we do want to go down the path of distributed backups of this (reviews welcome :) then we need a list of things to backup as well.  so thinking about what to grab can be codified into that list
19:28:41 <clarkb> ianw: that is a great idea
19:28:45 <fungi> /root ;)
19:28:55 <clarkb> and /etc/ansible
19:29:15 <ianw> fungi: did you say it was ~gb though?
19:29:33 <fungi> i don't think it needs to be. there was a metric ton of ansible tempfiles in there
19:29:50 <ianw> as long as nobody puts anything in /root that isn't critical to backup, that can work, but i also thing huge backups will be hard to deal with
19:29:57 <fungi> i just copied it all because i didn't want to have to wade through it while i was in a hurry to do key generation stuff
19:30:39 <clarkb> hrm I bet the new server has tempfiles for ansible too though
19:30:44 <clarkb> because ansible runnign as root
19:30:49 <clarkb> something to look at I guess
19:30:49 <fungi> yeah. i was being glib. specifically /root/signing.gnupg
19:30:56 <fungi> and /root/certs
19:31:08 <fungi> and /root/passwords
19:31:10 <ianw> fair enough, i think i didn't copy the whole thing as sort of a counter-point to that, trying to just keep the important bits
19:31:57 <ianw> the idea with the distributed backups is all that will go in a ansible variable; it could be either checked in, or in /etc/ansible
19:32:11 <fungi> a copy of the /root from old bridge can be found in /root/old-bridge-root-homedir.tar.xz for now, until we decide to toss it
19:32:36 <ianw> (i was thinking /etc/ansible, to make it less inspectable, but open to ideas)
19:32:46 <clarkb> ianw: as far as landing the backups goes we're mostly waiting on reviews?
19:33:16 <fungi> i remember saying i was going to review those changes and then not doing so
19:34:22 <clarkb> anything else bridge related?
19:34:36 <ianw> not this week
19:34:50 <clarkb> #topic Mailman 3
19:35:01 <clarkb> fungi: anything new with mailman 3 domain management stuff?
19:35:33 <fungi> negatory, sorry
19:35:39 <clarkb> #topic Gerrit Updates
19:36:03 <clarkb> ianw: looks like the last submit-requirements and copyConditions change (the one for testing it in system-config-run-review-*) landed
19:36:20 <clarkb> ianw: the conversion process should be largely done at this point?
19:36:33 <ianw> yes; i've spent some time updating
19:36:36 <ianw> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.7
19:36:44 <ianw> for that, and other issues
19:37:13 <ianw> we just merged a change to rebuild images from 3.7-stable, so i plan to bring up a test host with that
19:37:13 <clarkb> ya the other thing was the 3.7.1 bug
19:37:44 <ianw> that could likely be our production image.  if there's another release before our upgrade, we can rebuild at that
19:37:49 <clarkb> sounds good
19:38:05 <clarkb> in particular the bug we rebuilt to grab a fix for has to do with navigating related chnges in the ui
19:38:15 <clarkb> something I know I do so good to have fixed
19:38:27 <ianw> it would be worth going through the "DONE" tags on that page just to double check people agree with my conclusions about various issues
19:38:49 <clarkb> I'll try to review that today
19:39:14 <clarkb> this upgrade is very similar to prior upgrades except that we must take a logner downtime for offline reindexing
19:39:24 <clarkb> for whatever reason they apparently didn't support online reindexing for this one
19:40:41 <clarkb> #topic Project renames
19:40:49 <clarkb> related to the gerrit upgrade is the plan for project renames
19:41:00 <clarkb> we've decided to do both one after the other on April 6 starting at 22:00 UTC
19:41:43 <clarkb> fungi: other than better understanding our infra-project-manage-projects job sequencing when we've got old state in the earlier chagnes the other open question I've got is if we need to have any more care around openstack gettingthe xstatic library
19:41:44 <fungi> fwiw, i did an online reindex recently trying to run down a problem that turned out to be unrelated to indexing, but it only took a few hours to complete
19:42:25 <clarkb> fungi: should we maybe bring that up in the tc meeting tomorrow?
19:42:36 <fungi> xstatic-angular-fileupload?
19:42:42 <clarkb> "are you ok with this and does it create any conflict with moin moin?"
19:42:48 <clarkb> fungi: yes
19:44:16 <fungi> looks like it's got some shared authorship, yeah
19:44:23 <fungi> worth bringing up
19:44:39 <fungi> homepage link for it on pypi goes to https://github.com/danialfarid/ng-file-upload
19:44:57 <clarkb> ok I'll try to be there tomorrow morning (shouldn't be an issue other than being a few minutes late for a school run)
19:45:05 <fungi> last touched a little over 4 years ago there
19:46:01 <clarkb> #topic Updating old servers
19:46:38 <clarkb> The docker situation has distracted me from this. Which is mostly fine since I was going to hold off on gitea and etherpad work until later (can finalize gitea situation end of this week early next week and do etherpad after the ptg)
19:46:52 <clarkb> ianw: was there anything new with the nameserver planning?
19:47:34 <ianw> no, i've also been distracted.  there are some preparatory changes to merge, then i need to finish the checklist for switching and we can review.  parallel i can start new servers
19:48:01 <clarkb> ok ping me if I can help. I think I've reviewed changes though
19:48:17 <clarkb> #topic AFS disk utilization
19:48:40 <clarkb> Nothing really new here. I've left it on the agenda as a reminder it is something we need to keep an eye on. We do seem to have stablized near 10% free on the main server
19:48:59 <clarkb> and eventually fedora 36 should get cleaned up relieving some pressure just in time for new debian :)
19:49:23 <ianw> yep i'll try to get 37 builds going soon
19:50:09 <clarkb> #topic Gitea 1.19
19:50:33 <clarkb> Over the weekend gitea released 1.19.0. I had already pushed a chagne to sanity check 1.19.0-rc1 and there were minimal updates necessary to get 1.19.0 going
19:50:43 <clarkb> that said I think we should hold off until we've stabilized the gitea replacements
19:50:55 <clarkb> which is also planned for after the openstack release
19:51:01 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0
19:51:28 <clarkb> this release adds an actions runner system which is disabled by default. I have explicitly disabled it in this change to avoid that default changing on us later since we've got zuul
19:52:06 <clarkb> Otherwise the update seems pretty straightforward. THe commit message in that change tries to highlight all of the breaking changes from the release notes with my own notes on how/why things do or don't affect us
19:52:34 <clarkb> interestingly there were html template updates between 1.19.0-rc1 and 1.19.0. I'm not sure that has happened before
19:52:43 <clarkb> a good reminder for us to double check that between releases though
19:53:24 <clarkb> #topic Quo vadis Storyboard
19:53:57 <clarkb> last week frickler pointed out that there may be storyboard discussions at the ptg next week. In #opendev I mentioned that I think we should continue to encourage any projects migrating off of storyboard to collaborate to avoid duplicate effort on tooling/process
19:54:12 <fungi> so i guess keep an ear to the ground for projects bringing this up at the ptg and encourage them to weigh in on the opendev discussion?
19:54:32 <clarkb> fungi: yup and/or maybe push them towards each other and have shared discussion at the ptg too?
19:54:41 <fungi> sounds good
19:54:42 <clarkb> I'm also happy to jump into those sessions if people notify me
19:55:17 <fungi> yeah, one up-side to vptg, easy to drop from one call and jump into another (faster than running across the hall, much less to an adjacent building)
19:55:33 <clarkb> I also mentioend that I think longer term we (opendev) should think about sunsetting storyboard due to our lack of maintenance for the server and service
19:55:56 <clarkb> one good idea that came out of me mentioning that is we could try to make some sort of read only copy of the content and host that hopefully with much simpler hosting demands
19:56:22 <clarkb> but I also poitned out that I'm not the lone decision maker and welcomed feedback on that idea. Consider this an official reuqest for feedback from this group
19:57:04 <clarkb> #topic Open Discussion
19:57:10 <clarkb> We've got ~3 minutes for anything else
19:57:26 <fungi> having a search engine friendly view of the content instead of just browser-side rendering was, ironically, one of the wishlist items for further sb development
19:57:43 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/877735
19:57:50 <ianw> is one to split out the ssh server setup
19:58:10 <ianw> from base.  i'd like to try running that against the linaro cloud to make sure the root login auth, etc. are setup right
19:58:37 <clarkb> we can also potentially extend that to inmotion
19:58:54 <clarkb> but I Thin kstarting with linaro is great
19:58:56 <ianw> the prod job we added is working; but i did have to manually allow the zuul login
19:59:12 <ianw> (which is ok, that's setup by the launch node usually, which obviously didn't run)
19:59:26 <clarkb> ianw: its not zuul but bridge right?
19:59:31 <clarkb> I mean zuul via bridge
19:59:33 <clarkb> not direct
20:00:17 <ianw> yeah, basically allowing root logins from bridge.  but that config will be deployed with the ssh role, so if we change bridge again it will be ok
20:00:56 <fungi> speaking of linaro, i think we had a good justification to add them to the donor logos on https://opendev.org/
20:01:05 <fungi> anybody know who we should reach out to about that?
20:01:09 <fungi> kevinz?
20:01:16 <clarkb> fungi: yes kevinz is who I would start with
20:01:19 <fungi> k
20:01:35 <clarkb> and we are over time
20:01:40 <fungi> thanks clarkb
20:01:42 <fungi> !
20:01:44 <ianw> or perhaps ... sorry i'm blanking on the name ... but the person we sent the usage blurb to
20:01:51 <fungi> ianw: good call
20:01:56 <clarkb> thank you everyone! feel free to continue discussion in #opendev or on the mailing list. But I don't need to keep you in here any longer
20:01:56 <fungi> works on arm person
20:02:11 <clarkb> #endmeeting