19:01:17 <clarkb> #startmeeting infra
19:01:18 <openstack> Meeting started Tue Nov 17 19:01:17 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:21 <openstack> The meeting name has been set to 'infra'
19:01:25 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2020-November/000135.html Our Agenda
19:02:01 <clarkb> #topic Announcements
19:02:09 <clarkb> Wallaby cycle signing key has been activated https://review.opendev.org/760364
19:02:13 <clarkb> Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html
19:02:21 * clarkb makes a note to sit down after lunch and get this done
19:02:33 <ianw> i think i did
19:04:55 <clarkb> #topic Actions from last meeting
19:05:01 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-10-19.01.txt minutes from last meeting
19:05:28 <clarkb> No actions recorded. Moving on
19:05:34 <clarkb> #topic Priority Efforts
19:05:41 <clarkb> #topic Update Config Management
19:05:52 <clarkb> ianw: looks like you've started investigating a docker deployed hound?
19:06:22 <ianw> yes the project has had a bit of a second life and moved in github, breaking the puppet
19:06:43 <ianw> it's going to be a pretty straight-forward container + apache, so i think it's best to just modernise it
19:06:53 <fungi> what's still lacking for elasticsearch backend in gitea?
19:07:22 <clarkb> fungi: a release for 1.13.0
19:07:31 <clarkb> then testing to see how to deploy it and all that
19:07:42 <fungi> oh, got it. for some reason i thought that already got released
19:08:03 <clarkb> they were down to like two issues remaining for the release on the issue tracker but it got a new rc and not a release
19:08:08 <clarkb> maybe that means it is close?
19:08:31 <ianw> yeah, i think it's probably best to have this in our pocket so that can be done ad-hoc, given it will be a fair bit more deployment work
19:09:12 <clarkb> ya lot of unknowns (like sizing of elasticsearch deplioyments)
19:09:59 <clarkb> any other config management items to bring up?
19:11:17 <clarkb> #topic OpenDev
19:11:33 <clarkb> We are just a few days from the planned Gerrit upgrade to 3.2
19:11:45 <clarkb> #link http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html
19:12:02 <clarkb> Need to land the non WIP changes at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep
19:12:07 * yoctozepto is here for the meeting too
19:12:31 <clarkb> Looks like those have received recent reviews so after the meeting I think we can start landing them. fungi did you want to review those and maybe approve them if they look good? I will be around to help if things get sad
19:13:02 <clarkb> https://review.opendev.org/#/c/762712/ in particular is important because it conflicts with the WIP portion of my stack so I need to rebase once this one lands
19:13:21 <clarkb> #link https://etherpad.opendev.org/p/opendev-gerrit-3.2-upgrade-plan the plan is here. A few items still need a little work. Please take a look.
19:13:52 <clarkb> For the items that need work the first is deciding how we want to backup the 2.13 state as part of the upgrade. After some thought I think I prefer mounting a new volume on the side, then rsyncing into it
19:14:00 <clarkb> if you have better ideas please leave them on the etherpad
19:14:37 <fungi> the alternative would probably be to umount the volume and then snapshot it with openstack api calls
19:14:59 <clarkb> fungi: yup, my concern with that is that we're relying on cinder to do all that successfully for us, which it has done previously if we prefer that appraoch
19:15:00 <fungi> down-side to that is there are extra steps involved in turning that into a mountable replacement
19:15:54 <clarkb> The other big todo item was thinking about how to get config management into sync with the post upgrade state. I decided that squashing the 4 changes that reflect new state into one change and landing it before zuul starts is the best option. I have done that squashing and kept the old changes with distinct changeids to ease reviewing
19:16:04 <fungi> i'm in favor of the rsync plan. i wonder if pre-syncing anything into it would speed up the final sync during the maintenance, or if all the repack/gc would still require replacing most of it
19:16:23 <clarkb> fungi: I think we backup prior to gc'ing so maybe we should run that early
19:16:59 <clarkb> I like that idea
19:17:14 <clarkb> re config management syncup I need to update the etherpad to reflect that I did the squash
19:17:28 <clarkb> If you can think of anything else to consider please reach out and update the etherpad too
19:18:52 <clarkb> One thing to note is I tried to put time stamps/estimates in the etherpad to give people a sense for how this process should progress over time
19:19:08 <clarkb> this info is based on what we learned doing the review-test upgrades and should be fairly accurate particularly for the gerrit upgrade steps
19:19:38 <ianw> i agree an rsync copy seems the simplest way to keep things separate
19:19:59 <ianw> can always flip it r/o mount to avoid any mistakes too
19:20:35 <ianw> borg is running too :)
19:20:48 <clarkb> yup  Ifigure we've always got borg/bup to fall back on if we need it
19:20:57 <clarkb> this is more of a convenience for quicker recovery if necessary
19:23:03 <clarkb> cool. I'll work on updating the etherpad to better reflect those choices today
19:23:25 <clarkb> then hopeflly the next cuple of days we can do some last minute checks as well as prep the "backup" volume
19:25:18 <clarkb> Anything else on this topic?
19:26:14 <ianw> just a big thanks to you and fungi for working through it!
19:26:39 <clarkb> #topic General topics
19:26:45 <clarkb> #topic Bup and Borg Backups
19:27:07 <clarkb> This was just mentioned and it seems like keeping up to date on progress here is a good thing too
19:27:28 <clarkb> ianw: I know you had talked about starting the backups over now that you've trimmed them a bit. Has that happened or have we decided that isn't necessary?
19:27:53 <ianw> yes, i did that, particularly for review and zuul
19:28:04 <ianw> so everything has been running in production mode now for ~ a week
19:28:20 <ianw> i think let's get through the gerrit upgrade, then we can audit the borg logs look good and retire bup
19:28:28 <ianw> s/logs/backups
19:28:45 <clarkb> great, figuring out when to stop using bup was my next question. Doing audits after the upgrade sounds good to me
19:29:05 <clarkb> also we're backup up to two separte providers as a result of this as well?
19:29:11 <clarkb> that is a good improvement regardless of the tool in use
19:29:25 <ianw> yep, vexxhost & rax.ord with borg
19:29:52 <ianw> the vexxhost only has a 1tb volume, as we ran out of quota.  we can re-assign some of the bup quota to that i imagine
19:30:16 <ianw> but we can also prune now, which we couldn't before with bup
19:31:46 <clarkb> neat
19:31:54 <clarkb> Any thing else on bup/borg/backups?
19:32:07 <ianw> nope
19:32:23 <clarkb> #topic General Topics
19:32:47 <clarkb> There were a number of other topics including, PTG followups, meetpad access in china, openstackid.org, and splitting up the puppet else job
19:33:02 <clarkb> I've been super head down on the gerrit stuff and not had time for any of ^ myself
19:33:15 <clarkb> is there anything new on any of these topics others would like to bring up?
19:35:14 <clarkb> Sounds like no
19:35:16 <ianw> none for me -- still trying to reduce puppet :)
19:35:22 <clarkb> ++ thank you for working on that
19:35:27 <clarkb> #topic Open Discussion
19:35:42 <clarkb> Anything else?
19:37:41 <clarkb> Next week is a US holiday Thursday and Friday. I plan to be around for the first half of the week and will host a meeting at our regular time
19:38:03 <clarkb> I'll likely be looking for followups to the gerrit upgrade more than anything else and try and enjoy some quiet time after a busy few months
19:38:50 <ianw> i have visited the USA outlet store and acquired frozen cranberries for sauce, as that's not really a thing here
19:39:28 <fungi> all you need is a bog and some cool weather... oh
19:41:28 <clarkb> ianw: the best cranberries come in a can and look like jello
19:41:47 <clarkb> we can probably ship some to you though I have no idea what that would cost or how long it would take
19:42:17 <ianw> clarkb: i think that's one of those things you had to grow up with to like :)  i couldn't get into the beans with mushroom soup either.  but yams and marshmallows, that was a revelation
19:42:28 <clarkb> ha
19:43:34 <fungi> i have to disagree on the congealed canned berries. but then again i grew up with stewed fresh ones
19:43:56 <clarkb> we only got the canned stuff because we lived on the island. Fresh and dried are relatively new to me.
19:44:04 <fungi> anything which comes in a can should not be served still in can shape
19:44:09 <ianw> fungi you might like to look in on https://review.opendev.org/#/c/761621/ and the follow-on https://review.opendev.org/762588 as that updates the build table to not be unselectable in zuul ui.  i think i remember you having some troubles there
19:44:57 <fungi> ianw: ooh, thanks! yes that limitation has continued to plague me
19:45:11 <clarkb> alright sounds like that may be it. Feel free to follow up with us in #opendev or at service-discuss@lists.opendev.org
19:45:24 <clarkb> I'll call the meeting now and we can all have 15 minutes of breakfast/lunch/dinner back :)
19:45:29 <clarkb> thank you everyone!
19:45:32 <clarkb> #endmeeting