19:01:38 <clarkb> #startmeeting infra
19:01:39 <openstack> Meeting started Tue Mar 24 19:01:38 2020 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:42 <openstack> The meeting name has been set to 'infra'
19:01:46 <clarkb> Anyone else here for the meeting?
19:01:52 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006612.html Our Agenda
19:01:57 <frickler> o/
19:02:10 <diablo_rojo> o/
19:02:11 <clarkb> #topic Announcements
19:02:14 <mordred> o/
19:02:21 <clarkb> #link http://lists.openstack.org/pipermail/foundation/2020-March/002854.html PTG going virtual
19:02:38 <clarkb> I'm sure most or all of us have seen this news already, but wanted to call it out if you hadn't
19:02:49 <fungi> thanks for the heads up!
19:03:09 <clarkb> The big impact for us is that we'll likely try to support the virtual event though tools
19:03:33 <ianw> o/
19:03:39 <clarkb> corvus has begun work on a jitsi deployment incorporating our etherpad which looks promising but I'm sure tehre will be other things as well
19:03:58 <corvus> clarkb: any idea what other things?
19:04:11 <clarkb> corvus: right now? no :)
19:04:48 <clarkb> though collecting artifacts might be something that becomes more important
19:05:12 <clarkb> especially if conversations span timezones and we need to forward more concrete thoughts/items
19:05:22 <mordred> like scrolls and treasure chests and magical amulets?
19:05:44 <fungi> dibs on the +1 sword
19:05:54 * mordred weilds the +2 mace of bludgeoning
19:06:05 <corvus> i thought that was what the etherpad was for...
19:06:30 <fungi> that probably is what the etherpad is for
19:06:48 <corvus> just trying to figure out if i'm barking up the wrong tree
19:06:55 <clarkb> corvus: I don't think you are
19:06:58 <mordred> I think your tree is great!
19:07:00 <fungi> i'm guessing "other things" might be improvements to ptgbot
19:07:11 <fungi> scheduling tools? dunno
19:07:15 <clarkb> but there are a lot of moving parts and we've been involved in the technology side a bit in the past
19:07:17 <mordred> I'm guessing clarkb is just trying to pessimistically anticipate a general need for "other things"
19:07:20 <clarkb> (so I expect more moving parts)
19:07:31 <corvus> yeah, lots of other opportunities
19:07:38 * mordred thinks meatpad is amazing
19:07:49 <fungi> heh
19:07:54 * mordred would like to propose that we cname it so that both spellings are acceptable
19:08:09 <fungi> veggiepad is leaner and better for you
19:08:17 <mordred> ok - maybe 2 cnames :)
19:08:47 <clarkb> we should continue and then we can talk about meetpad in more detail
19:08:50 <clarkb> #topic Actions from last meeting
19:09:03 <clarkb> ianw had an action to cleanup files02 and old static resources
19:09:09 <clarkb> I think I saw a statusbot log action for that?
19:09:15 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-17-19.01.txt minutes from last meeting
19:09:26 <ianw> yes, the servers are gone.  there was some fun when i tried to look at the volumes :)
19:09:58 <ianw> i think that is sorted now, need to catch up, but i'll leave the old static volume and have a note to clear it out in a few months, just in case
19:10:12 <clarkb> sounds good, thanks
19:10:23 <ianw> ... so ... i think the transition is done.  i can move it in specs to the done pile later
19:10:28 <clarkb> ++
19:10:45 <clarkb> #topic Specs approval
19:10:51 * mordred hands a +2 chicken of awesome to ianw
19:10:56 <clarkb> really quickly before we talk meetpad is the xwiki spec
19:11:00 <clarkb> #link https://review.opendev.org/#/c/710057/ xwiki for wikis
19:11:28 <clarkb> fungi: ttx ^ I think we probably can put this up for approval this week given the discussion we had on the topic in last weeks meeting
19:11:44 <clarkb> I think we are largely on the same page that experimentation around the wiki problem is worthwhile
19:11:56 <clarkb> is there any disagreement with that?
19:12:16 <fungi> i'm fine with putting it up for a vote but also don't expect it's too urgent
19:12:39 <fungi> good either way
19:13:08 <clarkb> ya I'm not sure we'll get too much additional feedback if we drag it out, and the topic is fresh in minds now given the discussion last week
19:14:02 <clarkb> ok I'll look to landing that EOD thursday local time.
19:14:06 <clarkb> #link https://review.opendev.org/714189 meetpad
19:14:44 <clarkb> For anyone not familiar the meetpad idea is to combine jitsi meet and etherpad into a remote collaboration tool for events like a virtual PTG (though sure to have uses elsewhere too)
19:14:52 <clarkb> corvus: ^ probably ahs a lot more to say about it
19:15:50 <corvus> i think that about covers it
19:16:12 <corvus> i'm optimistic that having a tool focused on the design session workflow will be a passable substitute
19:16:22 <corvus> i've tested it small-scale, but not large scale
19:16:46 <corvus> since this is a low-commitment standalone service, we might want to think about going aheand and spinning it up to see how it goes
19:16:52 <mordred> ++
19:16:53 <clarkb> ya I'm hoping we can use it at medium scale a bit before the main event
19:16:58 <corvus> we can always shut it down later; it has no persistent storage or anything
19:17:00 <fungi> i can imagine ways in which it could even make some remote participation in otherwise in-person events more viable
19:17:02 <mordred> especially since you've got the ansbiels already
19:17:13 <corvus> and yes, the complete implementation is ready to go
19:17:26 <clarkb> On the deployment side my one concern is that the shape of the deployment is very different than our existing docker stuff
19:17:37 <corvus> well, not *very* different
19:17:44 <corvus> i managed to get it using host networking
19:17:58 <clarkb> ya, but instead of process per container its lots of processes per container (including cron)
19:18:11 <clarkb> it will just look different from the management side which is probably ok if people are aware of it
19:18:36 <corvus> fwiw, i'm not really aware of it :)
19:18:45 <corvus> i think the cron stuff is only for certbot?  which we also won't use
19:18:52 <corvus> because i hooked it into our existing acme stuff
19:18:54 <clarkb> ah ok so we can trim out the bits we don't need
19:19:04 <mordred> yeah - I thnk the images have more stuff in them - but we're not necessarily using those images in that way
19:19:05 <corvus> well, i wouldn't trim it out
19:19:15 <mordred> which is fine by me
19:19:45 <corvus> i'd like to use upstream images if possible
19:20:01 <corvus> (right now, we need a customized local image because i have a patch outstanding)
19:20:03 <mordred> yeah. we're really only building these atm because we have to carry a patch right?
19:20:05 <mordred> yeah
19:20:06 <clarkb> ya I'm not opposed, more I wanted to call out how they seemed to be different than how we've tried to construct things
19:20:19 <clarkb> if we run the upstream images it will be even more clear
19:20:30 <corvus> well, i'm not sure they're that different
19:20:58 <corvus> they're using our acme letsencrypt stuff, and there are 4 containers for the 4 processes
19:21:23 <clarkb> ok
19:21:27 <clarkb> I've misread the cron stuff then
19:21:38 <corvus> it's there, we just won't use it so we can ignore it
19:22:08 <corvus> i think it fits in pretty well with our model; our docker-compose file is really close to the upstream one
19:22:08 <fungi> so the images have more in them that we'll be leveraging is the main concern?
19:22:18 <corvus> so i think it's worked out pretty nice actually
19:22:27 <fungi> er, more in then THAN we'll be leveraging i meant
19:22:33 * fungi can't type
19:22:34 <corvus> it's close enough to our model that i was able to complete the entire implementation stack in less than a day
19:22:53 <clarkb> fungi: on my initial read it appeared that we would be using cron to run certbot
19:23:13 <clarkb> but I guess thats all there to avoid divergence from the upstream images and not actually to be consumed in our setup
19:23:13 <fungi> and we're not, got it
19:23:35 <corvus> we may yet be
19:23:37 <corvus> but it won't do anything
19:24:11 <corvus> if we really want to pull it out, we can, but i don't think it's a big deal
19:24:37 <clarkb> if the plan is to use upstream images anyway may as well keep it (since we'll get it back again once we move to their images
19:24:49 <corvus> i'd like to do that, yes
19:24:52 <mordred> ++
19:25:19 <fungi> seems fine to me
19:25:27 <corvus> so what's the timing here?
19:25:35 <clarkb> please review the spec if you haven't already. Then plan to land it next week? Though at the rate implementation is going maybe we should try to get people to review it this week and land the spec sooner than later
19:26:07 <corvus> to be clear, implementation is done :)
19:26:15 <corvus> the next thing is to create the server after the implementation patches land
19:26:36 <fungi> i'd be cool landing it this week
19:26:39 * mordred too
19:26:58 <fungi> some of us have already reviewed and voted in favor of the spec
19:27:06 <clarkb> of active roots its missing ianw and myself (and I've actually read it just not finalized my vote (which is positive I'm excited)
19:27:07 <frickler> +1, looking forward to tsting this
19:27:09 <fungi> doesn't seem like it needs adjustments
19:27:18 <clarkb> ianw: ^ you ok with landing that this week?
19:27:39 <ianw> yes
19:27:41 <fungi> also, it's a timely effort, so the less we unnecessarily delay progress on it, the better
19:27:52 <clarkb> cool I'll give people until thursday to review it then plan to approve it
19:28:04 <corvus> should we ping anyone from other communities about this?
19:28:16 <corvus> or stand up the server and then do that?
19:28:20 <clarkb> corvus: I half expect it would be easier to explain once the service is up
19:28:24 <mordred> me too
19:28:27 <fungi> i think asking them to help test it out makes more sense
19:28:28 <mordred> "go look at this"
19:28:43 <corvus> yeah, i tend to think so too.  there's so little cost in standing it up
19:29:10 <fungi> i expect we'll want to conduct at least one "load test" where we try to get lots of folks using it at the same time
19:29:32 <fungi> so we can work out sizing before it gets slammed and falls over at ptg time
19:29:45 <clarkb> as a time check we are about halfway through our allotted time and have more to discuss. Do you think we've gotten the important bits for this spec sorted (until thursday at least)?
19:29:47 <corvus> ya
19:29:57 <corvus> clarkb: yep, thanks!
19:30:05 <clarkb> #topic Priority Efforts
19:30:12 <clarkb> #topic Update Config Management
19:30:30 <clarkb> mordred: (or fungi or corvus ?) can you fill us in on the gerrit on containers status?
19:30:50 <mordred> well ...
19:30:57 <mordred> we ALMOST updated on friday during renames
19:31:07 <mordred> but then there was some container content issues - mostly the fact that jeepyb was missing
19:31:08 <fungi> the maintenance went really well, right up until.. yeah. so close!
19:31:25 <mordred> we've been unwinding an fixing some issues in the job
19:31:32 <fungi> hook script entrypoints were missing
19:31:42 <fungi> was the main setback
19:31:49 <mordred> as soon as I get another set of images built with the latest jeepyb changes (so close last time) - we'll be in a good position to do another quick restart
19:32:00 <mordred> that should be no longer than a stop/start
19:32:12 <mordred> in the process - we learned a gerrit thing that's worth sharing ...
19:32:19 <fungi> this has also spurred greater testing for jeepyb, so it's not all bad
19:32:37 <mordred> gerrit init does three things: it downloads the database driver, it expands the plugins from the war, and it runs db migrations
19:32:55 <mordred> we updated the image build script to do the first two - as they make no sense to be runtime activities in a container image
19:33:11 <mordred> so now seriously we only have to run init for db migrations when we upgrade
19:33:18 <mordred> so - just a note for everyone to be aware of
19:34:17 <clarkb> and until we get restarted on that new image project creation is not expected to work
19:34:21 <clarkb> (so we are holding new projects for now)
19:34:39 <clarkb> mordred: and the testing will incorporate testing of manage-projects?
19:34:59 <clarkb> it just occured to me that we may want to bind mount in the jeepyb cache for that? somethign to sort out side of the meeting
19:35:09 <mordred> we already do!
19:35:16 <clarkb> excellent
19:35:18 <mordred> although that's worth a mention
19:35:31 <fungi> new project creation and anything else on the gerrit server is blocked, right (this includes gerritbot config changes)
19:35:34 <mordred> I added helper scripts for manage-projects and track-upstream that run docker run with the appropriate bind-mounts
19:35:38 <mordred> fungi: yah
19:35:51 <corvus> do we need to hold new projects right now?
19:35:57 <mordred> so if you go to review-dev and just run manage-projects - it should work
19:36:07 <corvus> it's still in emergency, right?
19:36:09 <mordred> yah - I think AJaeger has been holding them
19:36:10 <mordred> that's right
19:36:12 <corvus> so i guess nothing will run automatically
19:36:19 <corvus> ok.  just wanted to double check my understanding on that
19:36:20 <fungi> corvus: seems like it's safe to keep approving them, they just won't take effect
19:36:36 <corvus> yeah, though that's potentiall confusing
19:37:15 <fungi> right
19:37:41 * fungi was confused that the infra-manual rename hadn't taken effect for gerritbot, for example)
19:38:18 <clarkb> ok sounds like that may be it. Lets move on as we have even more to talk about :)
19:38:21 <clarkb> #topic OpenDev
19:38:49 <clarkb> We've updated the way Gitea restarts happen. This should allow gerrit's replication plugin to properly detect the remote service is offline which will result in retries
19:39:17 <clarkb> https://review.opendev.org/714691 is small change that will run this in production (we do test it in checka nd gate too) just to be sure it doesn't cause problems unexpectedly later
19:39:43 <clarkb> Also the OpenStack TC governance change to excise OpenDev has landed (mentioend this last week) and we've landed our documentation updates
19:40:10 <mordred> woot
19:40:10 <clarkb> Two big todos that come out of that is I think we need to formally elect a project lead as well as start forming the Advisory Board
19:40:28 <mordred> clarkb: my understanding is that you are now lifetime project lead, right?
19:40:36 <clarkb> mordred: I was going to nominate you
19:40:38 <clarkb> :P
19:40:45 <fungi> hot potato
19:41:04 <corvus> not it
19:41:12 <clarkb> more seriously though I was going to start noodling on how the process for both of those. We've got lots of hsitory to work with can liekly come up with something sane
19:41:26 <clarkb> if you've got ideas or thoughts let me know or feel free to start a mailing list thead
19:41:40 <clarkb> and that takes us to the last set of opendev related topics I had on the agenda (which I'm going to reorder sligthly)
19:41:56 <clarkb> location of future meetings as well as need for opendev-discuss mailing list
19:42:24 <clarkb> initially I had thought we could do our weekly meetings in #opendev but frickler has indicated that the overlap in communications would cause problems
19:43:01 <clarkb> I think that concern is valid, so maybe we should create an #opendev-auxillary (I can't spell) where we can have meetings as well as use it for quiet space like the old infra-incident?
19:43:01 <frickler> yes, I've experienced that e.g. in the qa team and I would not like it
19:43:31 <clarkb> then anyone looking to catch up can check that channel for any super important updates and if empty skim #opendev at their leisure
19:43:47 <ianw> ++ to separate
19:43:55 <fungi> a related question is whether we need something similar to or replacing #openstack-infra-incident
19:43:56 <clarkb> corvus: any concerns with that from a freenode perspective? I suppose its another channel for bots but we are trimming channels elsewhere for the bots
19:44:11 <clarkb> fungi: ya I kinda expect we can simply use a single side channel for both
19:44:23 <clarkb> because if there is an important incident we won't be meeting anyway
19:44:44 <fungi> wfm
19:44:48 <clarkb> (I'm trying to be friendly to bots and not add too many extra channels)
19:44:48 <frickler> or we make an incident-themed meeting
19:44:52 <clarkb> frickler: ya or that
19:45:15 <fungi> so bikeshed some name which covers meetings and incidents
19:45:16 <corvus> i think we own the #opendev namespace
19:45:20 <frickler> then the channel could in fact be named #opendev-meeting, I'd think that would make it's purpose more easily visible
19:45:27 <fungi> #opendev-bikeshed isn't taken ;)
19:45:41 <clarkb> frickler: thats a good point. I'm happy to go with that
19:46:31 <clarkb> anyone want to take an action to push up changes to make that official?
19:46:38 <clarkb> we can bikeshed further in review if necessary
19:46:45 <fungi> i can do that unless someone else really wants it
19:47:09 <clarkb> #action fungi Add new opendev meeting (and incident) channel
19:47:26 <clarkb> the other related piece of this is mailing lists
19:47:47 <clarkb> I thinkwe can just go ahead aand create an opendev-discuss list on lists.opendev.org and start transitioning more of our email communication there
19:47:58 <clarkb> Then we can possibly fold openstack-infra into openstack-discuss as well
19:48:00 <fungi> service-discuss@ would make sense given we already have service-announce@
19:48:10 <clarkb> fungi: ++
19:48:27 <fungi> i can do the change for that as well
19:48:44 <clarkb> #action fungi Add service-discuss@lists.opendev.org mailing list
19:48:47 <clarkb> thank you!
19:48:54 <mordred> ++
19:49:19 <fungi> and again, names can be debated in the change
19:49:49 <clarkb> yup
19:50:05 <clarkb> #topic General Topics
19:50:15 <clarkb> The last agenda item was a catchup on mediwiki updates/cahnges
19:50:26 <clarkb> I don't think there has been any movement on the puppet retrofit
19:50:51 <clarkb> but did want to have this as an opportunity to talk about the possibility of dockerizing the deployment
19:51:07 <clarkb> mnaser: corvus ^ I think you've both been busy so don't expect there is anything new to discuss, but if there is feel free to share
19:51:26 <corvus> i don't have anything to report there
19:52:07 <fungi> if i find time i'll still try to make progress on the puppet module because it's the closest thing we have to documentation for the target configuration
19:52:46 <fungi> but mostly i suspect it needs troubleshooting done with extensions and possibly extension conflicts
19:54:04 <clarkb> ++
19:54:17 <clarkb> and that takes us to the free for all portion of the meeting
19:54:21 <clarkb> #topic Open Discussion
19:56:04 <clarkb> we lucked out and our monthly delivery of TP arrived (a few days late) from amazon. But now I'm distributing TP to family
19:56:27 <mordred> we have plenty for ourselves
19:58:27 <clarkb> Also oreilly has apparently cancelled all in person events forever. If you were set up to speak at one of their events that info may be useful to you
19:58:31 <fungi> people here tend to stockpile it to fling during the st patrick's day parade (which wound up cancelled). this produced an odd dynamic with people unexpectedly sitting on stockpiles of the stuff and stores already running low before the panic hit
19:59:11 <clarkb> (or attend I suppose)
19:59:43 <clarkb> And that is time. Thank you everyone!
19:59:46 <clarkb> #endmeeting