19:01:33 <clarkb> #startmeeting infra
19:01:34 <openstack> Meeting started Tue Jul 21 19:01:33 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:38 <openstack> The meeting name has been set to 'infra'
19:01:48 <zbr|rover> o/
19:02:16 <corvus> o/
19:02:26 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000058.html Our Agenda
19:02:34 <clarkb> #topic Announcements
19:03:32 <clarkb> OpenDev virtual event #2 happening July 20-22
19:04:00 <clarkb> This event is happening. The upgraded version of etherpad seems to be holding up just fine
19:04:13 <clarkb> #topic Actions from last meeting
19:04:41 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-14-19.01.txt minutes from last meeting
19:05:16 <clarkb> No actions recorded
19:05:21 <clarkb> #topic Specs approval
19:05:38 <clarkb> #link https://review.opendev.org/#/c/731838/ Authentication broker service
19:05:48 <clarkb> again not quite ready for approval, but definitely available for feedback
19:06:46 <clarkb> fungi: ^ anything else to add on this? You saw my feedback?
19:09:24 <fungi> yep, thanks for reviewing that, will incorporate into the next revision
19:10:28 <clarkb> #topic Priority Efforts
19:10:37 <clarkb> #topic Update Config Management
19:11:47 <clarkb> Zuul and Nodepool are running on containers now with the exception of nb03
19:12:21 <clarkb> nb03 runs on an arm64 host so needs an arm64 image
19:12:56 <clarkb> I've collected a number of arm64 container image build fixes for nodepool in this change https://review.opendev.org/#/c/741942/
19:13:27 <clarkb> latest problem is that cryptography did a release between the time we had working wheel cache setup on cryptography 2.9.2 and vhd-util ppa being fixed so now there is an uncached cryptography 3.0.0 and we try to build that on buildx and timeout
19:13:32 <corvus> ping me if you need me here; i'm working on zuul emergency restart
19:13:57 <clarkb> ianw: ^ we expect that mirror to update periodically right? any idea how soon we should have a new wheel for cryptography?
19:14:48 <clarkb> but I think ocne that happens we can land that change then we will have an image and can start looking at updating nb03
19:15:28 <ianw> yes, wheels *should* update daily, however, i had a look and there were network issues the other day
19:16:24 <ianw> kevinz fiddled something with routers in the cloud, and i haven't looked since.  so i'll get back to it and see -- but the jobs couldn't get a reliable ipv4 connection and so afs was just not happy
19:16:50 <clarkb> got it, in any case that seems to be the current issue and so getting the wheel cache updated would be helpful
19:17:11 <clarkb> Related to services running in containers we've just discovered that docker-compose up -d doesn't pull new images if it already has images but they are out of date
19:17:25 <clarkb> something to be aware of and maybe something we'll address in our playbooks
19:17:29 <ianw> one thing is that the release job only releases if *all* the wheel builds pass ... i'll look at that.  it might have just been done like that before we could do something like a semaphore
19:17:59 <clarkb> ianw: if we're able to safely copy out successful builds doing partial updates seems fine?
19:18:25 <clarkb> I mean if we don't have new A or B package we'll try to  build both in the downstream jobs. If we have only A or B, we'll try to build the other and that is still a net win time wise
19:18:42 <clarkb> if we want different versions that needs to be controlled by requirements or constraints
19:18:52 <ianw> yeah, there's no comments that suggest it was done for particular consistency reasons
19:20:08 <clarkb> http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016022.html is also related to updating config management
19:20:45 <clarkb> I'ev sent a response and approved on of the changes. But more input on the general goal there would be good. The mailing list would probably be best for that so that Javier sees it too though
19:21:43 <clarkb> #topic OpenDev
19:21:50 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000057.html Advisory Board welcomed.
19:22:00 <clarkb> I think that mostly makes the advisory board and its membership official
19:22:12 <clarkb> I've encouraged our volunteers to get involved and we'll see where it takes us
19:22:14 <fungi> ianw: possibly related to network issues, but sslcheck is also saying for the past couple of days that the mirror cert there is is expiring unexpectedly soon
19:22:41 <ianw> fungi: ok, will look
19:22:47 <clarkb> #link http://lists.opendev.org/pipermail/service-announce/2020-July/000007.html Gerrit /p/ mirror deprecation.
19:23:08 <clarkb> This is another email I wrote. With plan to get ahead of gerrit repurposing /p/ for project dashboards instead of git mirrors
19:23:19 <clarkb> this simplifies the number of locations we need to replicate and makes branch management simpler
19:23:33 <clarkb> and since it will happen when we update gerrit anyway we may as well do it now and get those other wins :)
19:24:48 <clarkb> I'll propose a change to make those urls 403 when I get a momement, but I have about a week and a half
19:25:05 <clarkb> and I'll go ping the cinder channel too as they seem to be the bulk of the users of that old path
19:25:20 <clarkb> Anything else OpenDev related to bring up?
19:26:21 <clarkb> #topic General Topics
19:26:25 <fungi> the openstack-infra ml is now closed
19:26:32 * fungi wasn't quick enough
19:26:36 <clarkb> fungi: oh ya, thank you for doing that
19:26:42 <fungi> np
19:26:52 <clarkb> #topic Bup and Borg Backups
19:27:14 <clarkb> ianw: for Bup I was curious if you had success doing a restore from the hosts that had their local indexes cleaned up.
19:27:28 <clarkb> and for Borg I think it would probably be good to have a short intro to your thoughts/plan around it
19:28:02 <ianw> ahhh, yes, i didn't get to that restore.  i will loop back on that
19:28:35 <ianw> mostly because i started looking at a new backup server, and it seems, as we've discussed before, bup is effectively dead
19:29:03 <ianw> there's no python3 support, and it doesn't seem to be coming unfortunately, and it's dropped from focal
19:29:47 <ianw> so i had a serious look at rdiff-backup and borg; both are very close to bup in terms of the hosts backing themsevles up to a central location and deduping on the server side
19:30:18 <ianw> borg seemed to be more active, had clarkb's approval, and had a lot of other features
19:30:32 <fungi> the clarkb seal of approval
19:30:43 <ianw> #link https://review.opendev.org/741366
19:30:52 <clarkb> (ya I use borg locally, I've not had to use it in an emergency yet but seems to have a number of good features like append only support, encryption at rest if we want it, fuse mounts for restoration)
19:30:52 <ianw> so that is the roles to do borg backups
19:31:49 <ianw> i'd suggest, if we're happy to move on with it, we bring up a borg backup server and switch in a ... lesser value host (maybe not start with review i mean) ... and start from there to get a bit of experience
19:32:21 <clarkb> I like that idea
19:32:25 <fungi> seems we're backing up review-dev
19:32:31 <clarkb> also we potentially get the ability to prune append only backups?
19:32:32 <fungi> or we were anyway
19:32:42 <clarkb> not sure how effective that will be but that could be a great feature
19:33:08 <ianw> append-only is implemented in above -- it does seem that we could run jobs on the backup server to then prune repositories
19:33:38 <ianw> so clients can't delete their history (restricted ssh command), but we can on the server side
19:35:20 <clarkb> and in reviewing the change this is implemented completely to the side of bup so we could switch a host over with little pain I expect?
19:35:34 <clarkb> basically add it to borg backups, let it backup with borg, check things, then disable bup ?
19:35:57 <ianw> yes, i deliberately kept them totally separate, with separate group names etc. so they can run completely in parallel
19:36:17 <fungi> thanks, wise plan
19:37:42 <ianw> that's more or less it; if we like it, i'm happy to move on with starting the server and initial backup runs (as mentioend eview-dev seems a good place to start)
19:38:53 <corvus> ++
19:39:04 <clarkb> I think I'm ok with it, but corvus would be a good source of feedback too having built the bup stuff
19:39:14 <clarkb> I'll review the latest ps looks like some of my feedback was addressed
19:39:49 <clarkb> #topic Project Renames
19:40:13 <clarkb> Last week we pencilled in Friday for project renames. I think I'm going to be in a reasonable spot to do that still. fungi are you still able to help?
19:40:22 <fungi> i am indeed
19:40:35 <fungi> still wide open for friday
19:40:50 <clarkb> great, should we say ~15:00 UTC on friday for that? I've been having ealry days with opendev event happening so keeping to that schedule seems fine
19:41:06 <clarkb> and thursday ish we can curate an etherpad and the chagnes and make sure we are ready
19:41:34 <fungi> i'm good with 1500z
19:42:17 <clarkb> great. I can sent an email about that to service-announce today after lunch
19:43:38 <clarkb> #topic Server Upgrades
19:43:53 <clarkb> fungi: anything new on wiki side of things? I don't expect so but didn't want to leave it out if there is
19:44:03 <fungi> zilch
19:44:21 <clarkb> #topic Open Discussion
19:44:31 <clarkb> That was it for our scheduled agenda. Anything else to bring up?
19:45:23 <clarkb> I guess I should mention that https://review.opendev.org/#/c/741277/ is the next thing in my series of branch management changes. Land that should be safe and it won't be used until we tag the library. https://review.opendev.org/741279 tests it
19:49:17 <clarkb> Sounds like that may be it? Thank you everyone. I know it was a somewhat distracted meeting. See you next week.
19:49:24 <clarkb> #endmeeting