19:01:09 <clarkb> #startmeeting infra
19:01:10 <openstack> Meeting started Tue Jan 19 19:01:09 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:14 <openstack> The meeting name has been set to 'infra'
19:01:17 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-January/000166.html Our Agenda
19:01:36 <clarkb> #topic Announcements
19:01:50 <zbr> o/
19:01:59 <clarkb> I didn't have any announcements. Continuing on
19:02:05 <clarkb> #topic Actions from last meeting
19:02:11 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-12-19.01.txt minutes from last meeting
19:02:21 <clarkb> iane had an action to check on wiki backups after the borg migration
19:02:25 <clarkb> ianw ^
19:02:28 <clarkb> typing is hard sometimes
19:03:00 <clarkb> ianw: has that happened or should we tack it back on (I know last week was crazy)
19:03:19 <ianw> i don't believe it is backing up to borg, but i haven't fixed it, so yeah, leave it on
19:03:36 <clarkb> #action ianw confirm wiki is still backed up after bup to borg migration
19:04:01 <clarkb> and now we'll jump right into mnaser's topic to avoid a conflict
19:04:03 <clarkb> #topic General topics
19:04:09 <clarkb> #topic Discuss infra-core (on behalf of OpenStack)
19:04:21 <clarkb> we'll resume regularly scheduled agenda planning after this item
19:04:31 <mnaser> thanks.  so it was recently brought up that andreas who's done a ton of infra reviews will not be able to help out as much
19:05:11 <mnaser> and the biggest load of project-config changes right now happen to openstack/project-config (i assume) and the opendev team is already a small one so .. wondering if it's an issue right now?  the reviews that come in seems to be very little right now
19:05:27 <mnaser> mostly: how do we make sure we stay on top of this, i guess
19:05:35 <clarkb> active infra-core is roughly myself, fungi, ianw, corvus, and mnaser I think
19:05:52 <clarkb> and by active I mean active in the community but not necessarily with reviews on that particular project
19:05:53 <fungi> er, project-config-core?
19:06:01 <clarkb> fungi: ya sorry
19:06:23 <clarkb> config-core is the general term we use for highlights iirc
19:06:34 <corvus> andreas did a lot of things, but in addition to reviewing project-config changes that added or updated projects, he also reviewed zuul-jobs changes, especially ones that affected openstack's use of zuul.
19:06:59 <fungi> and translations and docs tooling
19:07:14 <corvus> so there's likely to be some "opendev maintenance" work shortfall as well as openstack tact work shortfall
19:07:30 <corvus> yep those too
19:07:30 <zbr> my concern on reviews was around small projects which were nurtured by infra historically, getting a review on these requires a lot of effort, as infra-cores do always have more urgent priorities. what can we do to improve?
19:07:31 <fungi> but yeah, one thing which came up during the openstack tc meeting was that if we have some volunteers from the openstack community with bandwidth to review project-config changes and who can show some understanding of the material in there, we could likely expand the core review team for that fairly quickly
19:07:52 <clarkb> zbr: I think that is a separate but related concern (it has its own agenda item)
19:08:08 <clarkb> not sure if we want to combine them due to the bit of overlap (basically lack of resources overall being the overlap)_
19:08:08 <corvus> fungi: ++
19:08:24 <corvus> clarkb: separate i think
19:08:43 <fungi> mnaser: also it's not just openstack/project-config, as corvus points out there's zuul/zuul-jobs and also openstack/openstack-zuul-jobs (and to a lesser extent, opendev/base-jobs)
19:09:22 <mnaser> yes, i believe the concern here isn't around tooling but more around the project config reviews (tooling is another valid, but seperate issue)
19:10:02 <fungi> well, the bulk of the reviewing is job content, job definitions, job roles... and some project creation changes
19:10:04 <clarkb> I agree we could probably bootstrap a few interested individuals quickly. Maybe start by identifying who they are and ask them to do reviews with +1/-1 and they can reach out to us with questions or concerns with changes?
19:10:23 <clarkb> basically get them involved then once we've established comms and some trust we can bump up to +2?
19:10:26 <corvus> mnaser: i also think the revies he did on project-config are a little unique -- a *lot* of the technical stuff of that is actually automated or rigidly documented.  most of what he did there was, i feel, a form of hand-holding folks through the process.
19:10:48 <clarkb> corvus: yes that was definitely a big part of what ajaeger was doing
19:11:02 <corvus> that is in contrast to all the other repos which involve things like "knowing what tox is"
19:11:16 <fungi> helping folks who have submitted config changes understand the job failures which arise from those
19:11:19 <mnaser> corvus: i agree, it was a lot of "enforcing" the process
19:11:51 <fungi> and being aware of openstack policy (pti, guidelines in the ptoject teams guide, release policies, et cetera)
19:12:11 <fungi> and processes outlined in the infra manual
19:12:19 <corvus> clarkb: i also agree that it's the easiest team to expand
19:13:27 <corvus> and that the knowledge needed is a mix of community and tech.
19:13:49 <fungi> i've tried to step up my reviewing of project-config and related repos in recent weeks, but that's also eating into my available time for other opendev work
19:14:08 <clarkb> maybe the thing to do then is write down these things that we've identified ajaeger was doing in an email and request volunteers from the openstack community. Then once we've identified people interested do our best to help them do reviews and gain confidence in the processes and tech?
19:14:51 <mnaser> sounds good to me clarkb.
19:15:07 <ianw> that's also not a bad idea to try and find things we can reconsider/rework in 2020 that we've haven't looked at in a long time because ajaeger was doing such a good job keeping things humming along
19:15:15 <ianw> 2021 even
19:15:38 <fungi> i'm not so keen to go back and try 2020 again
19:15:49 <clarkb> ianw: ++ perspective from some new volunteers might help us identify those areas too
19:16:16 <clarkb> anyone want to volunteer to write that email? I can probably do it though unlikely today
19:16:24 <clarkb> but probably this week
19:16:55 <fungi> if one of us drafts it in an etherpad i suppose others can help flesh it out too
19:17:03 <clarkb> ++ to an etherpad
19:17:15 <clarkb> I'll go ahead and put myself down for it and if someone beats me to it they can just let the group know
19:17:25 <fungi> as tact sig chair, i'll consider it my duty
19:17:36 <fungi> you can look it over once i have a go at it
19:17:40 <clarkb> k
19:17:41 <mnaser> i'll bring up this discussion in the tc meeting on thursday too
19:18:14 <corvus> mnaser: thanks for being on top of this :)
19:18:16 <clarkb> #action fungi Draft email describing ajaeger's tasks and asking for new volunteers to help with those tasks. Also open possibility for improving processes and tools
19:18:31 <clarkb> Anything else on this subject or should we move on?
19:18:32 <mnaser> no problem :) i'll try to look at project-config changes too on my side as well
19:18:38 <mnaser> that's it from me for this
19:18:42 <fungi> #link https://etherpad.opendev.org/p/tact-sig-2021-rfh OpenStack TaCT SIG 2021 Request for Help
19:18:46 <gmann> o/
19:18:48 <fungi> i'll write something there
19:18:51 <clarkb> fungi: thank you!
19:18:56 <clarkb> #topic Priority Efforts
19:19:03 <clarkb> #topic OpenDev
19:19:23 <clarkb> A reminder that nominations for the Service Coordinator position are open.
19:19:28 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html
19:19:33 <clarkb> Please let us know if you are interested
19:19:53 <clarkb> I was going to bring up the zuul status change list, but it looks like ianw got that landed and applied
19:20:11 <clarkb> ianw: anything else to bring up on the subject of the zuul status work in gerrit?
19:20:27 <ianw> not, good team effort with clarkb getting it across the line, thanks :)
19:20:39 <clarkb> it looks great in gerrit too, thank you for writing thatp lugin
19:20:49 <corvus> it looks great, thanks!
19:21:07 <clarkb> Zuul's gerrit WIP change support should be deployed as of this last weekend
19:21:12 <fungi> discussion on the openstack-discuss ml thread about it suggested reviewing the zuul status plugin as a potential next candidate
19:21:28 <clarkb> fungi: another candidate to deploy in our installation?
19:21:50 <fungi> right, the live test progress display in the gerrit webui change view
19:21:57 <clarkb> gotcha
19:22:01 <ianw> fungi: that one might be a bit harder to get screen shots of in testing, but i think the framework for including it is fairly straight forward now
19:22:34 <clarkb> for WIP support has anyone tested that yet?
19:22:47 <clarkb> I believe we want to confirm that zuul will not enqueue a wip change to the gate
19:23:00 <corvus> ianw: is non-voting handled?
19:23:08 <corvus> (by handled i mean displayed)
19:23:54 <fungi> i did just approve zbr's gerritbot change today to start ignoring wip changes, and start announcing them if they transition out of wip (there's a separate event for that now)
19:23:58 <ianw> corvus: in the in-progress plugin, or our summary-status?
19:24:10 <corvus> ianw: summary -- and yes, i see it now
19:24:21 <corvus> shows up on the right with the rest of the time comment
19:24:32 <corvus> fungi: oh um
19:24:35 <corvus> fungi: can we not do that?
19:24:40 <corvus> i really want to see wip changes announced
19:24:42 <fungi> we can revert it
19:24:58 <fungi> it was up for review for a while
19:25:00 <ianw> corvus: yeah, i did think looking at that yesterday on a massive cinder change it could a) sort the NV together and b) show them somehow differently
19:25:20 <corvus> fungi: well, i may be getting ahead of things here, but i don't want anything about gerritbot to change :)
19:25:41 <fungi> sure, i'm pushing up a revert now and then we can discuss further
19:26:09 <clarkb> fungi: I think you tested if zuul ignored WIP chagnes in the gate previously. do you know if there is a good change to test that on?
19:26:16 <corvus> and i realize we may be getting into personal preferences, but i have found it really useful when, say, we're having some convo in irc, and someone pushes up a wip patch, to go look at it right then.
19:26:53 <clarkb> I guess we can maybe WIP that revert and approve it and see what zuul does then unwip it and merge it?
19:27:15 <clarkb> that should sufficiently exercise the new zuul behavior
19:27:40 <fungi> i've pushed up a revert and a revert of the revert, approving the former now and we can discuss it in the latter
19:27:51 <clarkb> ok
19:28:30 <clarkb> The last major opendev item I wanted to bring up is gitea 1.13.1 upgrade. I've put this off as fires have crept up and needed attention. I think we're largely stabilized and I hope to approve the upgrade tomorrow (when I'll be able to monitor it)
19:28:37 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/769226
19:28:41 <zbr> wip change on gerritbot was open for 2 months
19:28:59 <clarkb> the upgrade is reasonable well tested so I don't expect issues but the upgrade is fairly large in terms of new features and stuff in gitea
19:30:06 <clarkb> Any other opendev items or should we continue on?
19:30:44 <fungi> clarkb: i think zbr may have had a wip change which was approved, and we saw zuul attempt to merge it and fail leaving a verified +2 behind... i don't recall which change it was off the top of my head though
19:30:49 <zbr> corvus: in fact regarding WIP on gerritbot, i think that revert is not the best idea as it would be easy to enable announcement with two lines of code.
19:31:29 <zbr> yep, we experimented it and found that zuul was trying to merge as wip change and gerrit was not letting it do it.
19:32:03 <corvus> fungi: to be clear, are you saying that happened *after* we restarted zuul with the wip change support, or before?
19:32:22 <fungi> corvus: before
19:32:29 <fungi> before the wip support change was written
19:32:35 <corvus> okay, so that's a description of the behavior to be on the lookout for
19:32:41 <fungi> right
19:33:11 <fungi> zbr: on the gerritbot change, we can edit the revert of the revert to perhaps make it configurable by channel
19:33:23 <corvus> zbr: i certainly don't oppose the option for gerritbot users to disable wip notices
19:34:00 <zbr> do we really need it configurable?
19:34:16 <fungi> if some channels want to see wip changes announced and others don't
19:34:26 <corvus> i'm just volunteering that i'm happy with the current workflow, don't think it's spammy, and would miss functionality if it changed.  i would probably run my own gerritbot or something like it to make up for the loss of functionality.
19:35:21 <clarkb> lets move on to ensure we get to the other items, but can get back to this discussion if we have time at the end of the meeting
19:35:25 <clarkb> #topic Update Config Management
19:35:30 <zbr> current behavior is that is even missing to anounce a change getting out of WIP, which is quite important IMHO
19:35:50 <clarkb> ianw has started work to ansible our afs servers
19:35:51 <zbr> because that is the moment that marks "that is ready for review"
19:35:57 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/771268
19:36:09 <clarkb> this is related to the next topic whcih is openafs cluster updates
19:36:21 <clarkb> why don't we just jump ahead there so that meeting notes are a bit cleaner
19:36:26 <fungi> there's been a flurry of openafs configuration management improvement yeah
19:36:30 <clarkb> #topic General Topics
19:36:40 <clarkb> #topic OpenAFS Cluster Status
19:36:53 <ianw> i can give a quick summary
19:36:56 <clarkb> ianw: please do
19:37:06 <fungi> probably better not to burn meeting time getting into the history of what's transpired, but current status would be great i think
19:37:14 <ianw> yes, current summary sorry :)
19:37:30 <ianw> so afs01/02.dfw and afs01.ord are now all running afs 1.8 from our ppa
19:37:46 <ianw> we should have the changes in system-config to manage them via ansible, but when i left yesterday the CD jobs were blocked
19:37:52 <ianw> (think that's fixed now)
19:37:55 <ianw> so i will chase up on that
19:37:57 <fungi> as are all clients (at least installed, many will run it on next reboot)
19:38:13 <ianw> today, i'd like to upgrade afsdb01/02 in turn to 1.8, then we'll be 100% 1.8
19:38:24 <ianw> i have the system-config change to manage them with ansible too, and remove the puppet bits
19:38:31 <clarkb> ianw: that would be great
19:38:39 <clarkb> this way we'll be able to respond to future bugs more directly
19:38:44 <fungi> and jobs are using either patched 1.8.6 or 1.8.7 which has the patches officially applied
19:38:49 <clarkb> and as you mentioend it likely simplifies upgrading the servers
19:38:54 <ianw> then i'd like to upgrade the hosts one-by-one to focal, in place.  i don't think it's worth making new servers, especially for the db which are annoying to change ip addresses
19:39:16 <clarkb> ianw: yup we upgraded them in place previously so I expect that is still our best option
19:39:37 <ianw> i had planned to start that with afs01.ord and see how it goes.  this should be zero downtime as we can now do it one-by-one
19:39:57 <ianw> that's about it
19:40:10 <fungi> we're holding off the focal upgrades though until we have fully recovered from the storage outage on afs02.dfw
19:40:20 <fungi> which is still in progress
19:40:21 <clarkb> there were two things I wanted to call out the first is debian buster test nodes will use our bionic ppa package until debain fixes their distro packages
19:40:24 <ianw> oh yes, sorry, after we're back in a regular release schedle
19:40:49 <clarkb> second is the openafs-client role seems to not try to install our ppa packages on xenial
19:40:53 <fungi> by my estimate we've got full releases of roughly half the mirror volumes left to complete, everything else is back on track
19:41:12 <clarkb> evidence from the zuul executors which are our only xenial openafs-clients shows taht we seem to be using the ppa there anyway
19:41:17 <fungi> so it's progressing faster than i'd expected anyway
19:41:34 <clarkb> maybe someone else can look at that and we can decide if we should remove the xenial exclusion condition from the ansible role?
19:41:43 <ianw> clarkb: huh, wasn't aware of that.  we can chase up after meeting, definitely sounds like a bug
19:41:48 <fungi> one theory on the executors is that puppet installed the ppa config, and so it's lingered there after we switched them to ansible
19:41:54 <clarkb> ianw: cool I'll get links together for that after the meeting
19:42:04 <corvus> fungi: is that true for ze12?
19:42:32 <fungi> corvus: was it built after the ansibilification? if so then i gues sthat theory can be discounted
19:42:51 <fungi> but even ze12 has a lot of ppa config crufy in place
19:42:52 <corvus> i don't know, but worth looking into if we need to verify/falsify that
19:42:54 <clarkb> it does have the ppa installed and did update the package at least
19:42:58 <fungi> er, cruft
19:43:13 <corvus> but also maybe we don't care about the past and only the future in this case :)
19:43:17 <fungi> on ze12 there are three different files all adding that same ppa
19:44:06 <ianw> all of the afs servers also had some old cruft from when we tried something that added all the ubuntu repos twice
19:44:25 <clarkb> so maybe thats a larger general cleanup (the extra ppa configs)
19:44:55 <clarkb> we can follow up on that after the meeting just wanted to make people aware
19:45:00 <clarkb> #topic Picking up steam on Puppet -> Ansible rewrites
19:45:06 <fungi> the ppa addition files on ze12 all date from may 2020
19:45:16 <fungi> oops, sorry, will discuss after the meeting
19:45:28 <clarkb> no worries just want to get through the agenda :)
19:45:53 <clarkb> one thing I noticed when we looekd at the openafs stuff last week is we have a number of servers still running xenial and for many of them not called zuul or review they are still running puppet
19:46:20 <clarkb> we did not solve puppet on bionic or newer which means that one of the things we need to keep in mind is xenial -> newer upgrades will want ansible (and maybe docker) on the config managment side
19:46:48 <clarkb> This is an early call out that we need to start looking at this (work like ianw's openafs role is great!) so that we can also do server upgrades
19:47:06 <fungi> there was renewed interest expressed in last week's storyboard meeting about finishing the docker deployment automation for sb at least
19:47:14 <clarkb> I'm hoping that if things remain calm I can start putting together and audit so that we have a todo list we can pick things off of and annotated with interest like ^
19:47:23 <clarkb> there is also interest from refstack folks
19:47:23 <fungi> as that's blocking us from deploying a number of other fixes to production at this point
19:47:37 <ianw> ++; things like docker for storyboard seem like something that is good for us, but good for everyone else too
19:47:51 <clarkb> #action clarkb put together audit/todo list for ansiblification and server upgrades
19:48:04 <fungi> in the case of storyboard, we already have docker images publishing on each new change merged (thanks mordred!)
19:48:12 <clarkb> I've got a few things I need to get done in my backlog but thats near the top of the next items to pull off
19:48:47 <clarkb> #topic two-review rule impact on low-activity projects
19:49:10 <clarkb> zbr added this to the agenda and this is the topic I was referring to during mnaser's topic as being related but separate
19:49:34 <clarkb> we've got a number of low activity projects like git review and bindep and gerritbot etc that don't get regular reviews
19:49:45 <fungi> fwiw, we've already said it's okay to single-core approve stuff in these as long as we observe a quick-revert policy should concerns be expressed by another core reviewer later (as we've just done with gerritbot, for example)
19:49:50 <clarkb> a consequence of that is that it is hard to get two +2s to land changes
19:50:10 <clarkb> fungi: yes, I think the bigger issue is more a lack of time for any reviews on those change due to higher priority issues/fires
19:50:33 <clarkb> at least for me it is rare that I'm able to make a chunk of time for reviews in those projects
19:50:55 <fungi> yep, i've been trying to strafe some of them as i get time. i should probably convert a number of my solo +2's to approvals
19:51:10 <corvus> i think we've always been flexible with this.  i think we can continue to do so.  i think that procedural/minor technical changes are especially good candidates for sigle-core reviews.  i think substantial changes benefit from more reviews.  again, using the gerritbot change as an example: more reviewers might have surfaced an alternate view.
19:51:54 <fungi> for example, someone e-mailed an archlinux fix for bindep to openstack-discuss, i proposed it to gerrit for them in https://review.opendev.org/771108 and waffled on whether i should just approve it too
19:52:09 <clarkb> some things are pretty tightly coupled to our service (git review and gerritbot for example) so I'd be wary of putting them up to adoption.
19:52:16 <corvus> but i'd also say that some of the low activity projects are low activity for a reason: in my view some of them are pretty close to "done" and i view them as being in maintenance mode.
19:52:24 <clarkb> projects like bindep are much less so (and we really only got bindep beacuse no one elsewanted ti it seems like)
19:52:32 <corvus> i don't want git-review, gerritbot, and gear to change.
19:53:02 <fungi> gerritlib is also in that space a bit
19:53:12 <fungi> since jeepyb depends on it fairly heavily
19:53:25 <fungi> (as does gerritbot)
19:53:49 <clarkb> I know some projects have relied heavily on priority lists (either generated by gerrit and a review priority category) or manually curated in things like an etherpad
19:54:07 <fungi> we do get soe good bug fixes for those from time to time, but also they tend to attract feature additions which are harder for me to justify as they almost always imply some scope creep
19:54:21 <clarkb> we could try something like that to not just surface changes but more publicly communicate what we think is important right now and maybe people will unerstand why other changes are being passed over
19:54:35 <zbr> someone said a sw is done only when is dead (nobody is using it) :p
19:54:49 <clarkb> I think the big win with something like ^ would be more to expose "this is important and keeping us from other work" more so than helping us get the other work done quicker
19:55:00 <corvus> zbr: i recognize that's a common view of folks who get paid to write software, but i don't entirely agree with it
19:55:32 <fungi> i use a ton of 'nix utilities day in and day out which haven't added a new feature in countless years
19:55:36 <fungi> doesn't make them less useful
19:55:47 <zbr> be aware that not reviewing patches from others does not help us get more people to join our effort
19:56:20 <corvus> i think if people want to join our effort, then focusing work on where our priorities are would be best
19:56:30 <corvus> and we have quite a few ways of conveying that
19:56:30 <zbr> in fact is quite an effective deterrent, and creates a vicious cycle
19:56:39 <clarkb> corvus: ya thats why I'm wondering if pointing at the priorities more clearly will help
19:56:48 <clarkb> "this is where you can help us" also "this is why we are busy"
19:57:04 <fungi> zbr: yep, in my case i think i struggle to break the bad news to people that i'm not interested in approving their pet feature addition. if i harden my heart a bit maybe i can find more time to approve bug fixes in those projects and not worry about spending as much time explaining to contributors why their proposal doesn't fit with the project
19:57:54 <corvus> clarkb: yeah, it looks like we need a link to infra-specs in https://docs.opendev.org/opendev/system-config/latest/project.html#priority-efforts
19:57:58 <fungi> sometimes it's easier to just put off leaving a review on something i know i'm not going to approve, because i don't have time to write an essay explaining why i'm rejecting it
19:58:21 <clarkb> corvus: ya I think specs as well as current fires
19:58:29 <zbr> fungi: yep, that is tricky, but is still better in long term to cut it short (but politely)
19:58:36 <clarkb> at least the last few months have felt more like fire fighting than proper maintenance work. I'll think on how to expose that
19:58:46 <clarkb> maybe just start with a simple etherpad then look into automating it osmehow
19:58:49 <corvus> https://docs.opendev.org/opendev/infra-specs/latest/#priority-efforts
19:58:50 <fungi> zbr: i totally agree
19:58:58 <fungi> something i need to improve at personally
19:59:08 <corvus> clarkb: i'd rather just see that kept up to date
19:59:21 <corvus> i don't think we need more places to share our priorities
19:59:34 <clarkb> corvus: I agree keeping it up to date is a good thing, but I'm not sure it captures "here are the five things we're dealing with to unbreak afs"
19:59:45 <clarkb> which lately seems to be the type of thing consuming my time
19:59:45 <zbr> i would be very upset if spend 10 days on making something work, and failing to deliver it than just a couple of hours and being notified politely that that feature is outside the scope of the project.
20:00:05 <ianw> i also do not mind if people have a change that seems stuck calling it out as a meeting topic.  i have found that very useful over the years, and i think it's reasonable to expect some outcome from calling a specific change out in a topic
20:00:23 <clarkb> (as a time check we are at the end of our scheduled hour)
20:00:59 <corvus> clarkb: ianw has tried to keep storyboard up to date for some efforts like that
20:01:17 <clarkb> corvus: good point, so maybe its tagging in storyboard and makign a dashboard out of that
20:01:28 <clarkb> and then linking to that in the docs next to the long term maintenance priority efforts
20:01:38 <clarkb> and being better about filing bugs and tagging them appropriately
20:01:44 <clarkb> (I know I'm really bad at that bit myself)
20:01:47 <corvus> yeah, if you want to go down that road, i think that may fit the bill
20:02:14 <clarkb> I like that idea, I'll see if I can make that work reasonably well
20:02:48 <clarkb> another thing to keep in mind on this topic is that we are all human and trying our best to keep up with what at times is a flood of inputs. I doubt anything we do will ever make this problem go away
20:03:20 <clarkb> which means we probably need to both set expectations appropriately but also figure out methods for doing better
20:03:29 <corvus> the only thing that has even remotely made a dent is having the ability to hire more people to do more work.
20:03:50 <clarkb> yup
20:04:15 <corvus> and that was just less of a backlog than now.
20:05:27 <clarkb> I'll leave this open for another minute or two if there are any last thoughts
20:05:39 <clarkb> But feel free to continue any conversations had in this meeting in #opdnev
20:05:43 <clarkb> * #opdnev
20:05:50 <clarkb> ugh I cannot type today. #opendev
20:05:51 <fungi> something like that
20:05:58 <fungi> thanks clarkb!
20:06:32 <zbr> thank!
20:07:45 <clarkb> thanks everyone!
20:07:47 <clarkb> #endmeeting