19:01:19 <clarkb> #startmeeting infra
19:01:20 <opendevmeet> Meeting started Tue Jun  1 19:01:19 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:21 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:23 <opendevmeet> The meeting name has been set to 'infra'
19:01:29 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-June/000252.html Our Agenda
19:01:37 <clarkb> #topic Announcements
19:02:42 <clarkb> I didn't have any announcements.
19:02:52 <fungi> welcome to oftc?
19:02:58 <clarkb> It is a later topic in the meeting but ya we are here on oftc now
19:03:13 <clarkb> I know a few of us are still hanging around in a few freenode channels but I'll probably clean those up next week
19:03:15 <fungi> hope everyone found us okay
19:03:39 <fungi> and yeah, we can discuss later in the meeting
19:03:39 <clarkb> *clean those up in my client.
19:04:23 <clarkb> But ya welcome and let us know if you have qusetions/concerns/problems with the switch
19:04:27 <clarkb> #topic Actions from last meeting
19:04:32 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-05-25-19.01.txt minutes from last meeting
19:04:47 <clarkb> There were no actions and even if there were I think we ended up fairly swamped by irc and related tasks
19:04:58 <fungi> a swamp it was
19:05:09 <clarkb> #topic Topics
19:05:24 <fungi> how topical
19:05:25 <clarkb> The most generic of topics. I reorged the meeting agenda as discussed last week
19:05:32 <clarkb> #topic Switch to OFTC now largely complete
19:05:57 <fungi> there's some patches up to convert ptgbot
19:05:57 <clarkb> fungi: want to give us an update on where we are with the switch and any outstanding issues we need to clean up?
19:06:25 <fungi> and someone still needs to follow up on the #openstack-sahara registration
19:06:44 <fungi> i also have a patch out there to drop #edeploy from the statusbot config
19:07:03 <clarkb> #link https://review.opendev.org/c/openstack/ptgbot/+/793792 ptgbot oftc conversion
19:07:36 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/793813 drop edeploy channel from statusbot config
19:07:49 <fungi> on the topic of statusbot, we've seen an unexplained crash but the way it logs fails to capture exceptions/tracebacks, so it's running foreground currently in a root screen session on eavesdrop with an su to the ptgbot account
19:08:08 <clarkb> fungi: and I guess we'll keep running it that way in the hopes of catching the crash?
19:08:27 <fungi> yeah
19:08:29 <fungi> likely some unforeseen problem in the hurried python3 conversion we did for it over the weekend
19:09:01 <fungi> and ianw has been tackling the converged bot/limnoria spec, has a poc up i believe though i haven't had time to look
19:09:05 <clarkb> fungi: for sahara we need them to give accessbot master perms via chanserv?
19:09:13 <clarkb> #link https://review.opendev.org/q/topic:%22limnoria%22+status:open Limnoria bot rewrite
19:09:16 <fungi> correct on #openstack-sahara
19:09:45 <clarkb> ianw: based on the commits there you're porting the existing supybot config to limnoria?
19:10:07 <fungi> the pending patch to add #openstack-sahara back to acessbot and gerritbot is...
19:10:18 <fungi> #link https://review.opendev.org/792857 Revert "Accessbot OFTC channel list stopgap"
19:10:20 <ianw> that stack (completely uncommented) is intended to port meetbot to python3/limnoria
19:10:35 <clarkb> ianw: nice, I'll try to take a look today
19:10:39 <ianw> however limnoria is so far exactly the same as supybot
19:10:54 <ianw> i will clean up that stack iwth actual change comments
19:11:01 <clarkb> ianw: sounds good
19:11:15 <ianw> it's probably more useful to have a poke at it in #opendev-sandbox atm and see what's right/wrong
19:11:29 <clarkb> ah ok interactive debugging more so than code review
19:11:56 <ianw> yeah, the other part is
19:11:56 <fungi> oh, that reminds me, the supybot package on eavesdrop has one of its files edited to basically roll in a newer (but many years old) commit which never made it into ubuntu
19:12:01 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/793704
19:12:13 <ianw> which builds a limnoria/meetbot container
19:12:16 <fungi> which makes the limnoria work slightly more important
19:12:21 <clarkb> fungi: ++
19:12:31 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/793719/8
19:12:36 <ianw> is then installing that
19:12:57 <ianw> i've tried my best to port over the existing config, but there's *a lot* of options in there
19:13:29 <ianw> small details all still WIP, but big picture comments definitely useful at this point.
19:13:51 <clarkb> cool, I will attempt to interact with the bot and see if that produces any feedback
19:13:58 <fungi> also more generally, we introduced some known security risks during the bot conversion by switching some of them to identify where previously they authed at connection time, and by no longer having an atomic means of verifying the nicks they get messages from are authentic
19:14:19 <clarkb> fungi: we should set enforce on our nicks with nickserv ya?
19:14:33 <ianw> i don't know if supybot had certificate support, but there are options in it for limnoria
19:14:58 <clarkb> ianw: its verification of the nicks interacting with the bots that is the struggle
19:14:59 <fungi> clarkb: yes, it's already set for our the bot accounts but we should encourage people to set it on their own nicks
19:15:06 <fungi> if we can set the new converged bot up to do certfp auth, that eliminates one of those risks
19:15:15 <clarkb> other networks have extensions that include with every message the auth/ident status when each message is sent but oftc does not
19:15:40 <fungi> or if it does, i wasn't readily able to find it
19:16:08 <clarkb> Overall I think the transition went well. We hit a few speedbumps but nothing major and we managed to work through almost all of them. Thank you to all who helped with this process.
19:16:13 <fungi> granted, the cap freenode was relying on for that also didn't seem to be part of the irc3 standard caps
19:16:29 <clarkb> fungi: it was but then got superceded by another cap that does similar
19:17:07 <fungi> ahh, yeah i'll admit i only skimmed them to see which ones were implemented in freenode and oftc
19:17:20 <clarkb> alright anything else on this topic or should we move on?
19:17:50 <fungi> i think we can move on, but please anyone with questions or problems do reach out to us
19:18:02 <clarkb> #topic Gerrit Account cleanup
19:18:06 <fungi> and i have an oftc-related policy question to bring up during open discussion if there's time
19:18:12 <clarkb> noted
19:18:28 <clarkb> For gerrit account cleanups this is a reminder that I've got a list up of slightly more dangerous but probably safe account cleanups on review
19:18:48 <clarkb> We've all been busy and not able to review that list (and at this point I feel like I should rereview it myself), but if you get time to look over it that would be great
19:19:28 <clarkb> I think doing that cleanup in a multistep process where we disable teh accounts first then sit and wait a while then cleanup the conflicting external ids has worked well and should be relatively safe here. When done I think we end up with about 80 accounts that we should reach out to users about
19:20:01 <clarkb> #topic Refreshing SSL certs
19:20:24 <clarkb> We have made some good progress with these. A number of services are now using LE certs. We also cleaned up certcheck a bit.
19:20:48 <clarkb> The last remaining cert we need to renew is for wiki and I'll be doing that manually after this meeting because we don't have that service in config mgmt
19:21:10 <clarkb> Then we should be good for another year.
19:21:29 <fungi> thanks!
19:21:30 <clarkb> If you notice old certs hanging around that may be due to apache not cleaning up all child processes when we tell it to reload. I think the risk is low but it does happen
19:21:41 <clarkb> Definitely say something if you see a cert expiring soon
19:21:57 <fungi> and we do have workarounds we can apply for cases like that if they keep cropping up
19:22:31 <clarkb> #topic Switch Vexxhost to provide only specialized labels in Nodepool
19:22:38 <clarkb> #link https://review.opendev.org/c/openstack/project-config/+/785769
19:22:44 <clarkb> This has been rebased and looks mergeable
19:23:09 <clarkb> fungi: did you want to discuss it further or just ask for reviews and possible +A at this point?
19:23:37 <fungi> i guess i was hoping for some consensus on either the ml or review that this was something we wanted before approving it
19:23:58 <clarkb> ok so more than just one or two +2s before approving it
19:24:02 <fungi> but maybe the lack of general interest in the topic is a sign that there's no objection
19:24:25 <fungi> i can go ahead and approve it after the meeting, it's always easy enough to revert
19:24:58 <clarkb> I added my thoughts to the change in review and ya basically said it should be safe and it is easy to revert if we discover it causes problems somehow
19:25:37 <clarkb> #topic Server Upgrades
19:26:09 <clarkb> Once I've sorted out the wiki ssl cert I plan to set up a hold for change that runs system-config-run-lists and then do in plcae upgrades of the resulting test nodse as a first pass check of that upgrade.
19:26:37 <clarkb> Then based on how the test nodes handle it we can decide if we need to do a snapshot of the prod list server(s) and test that upgrade with real prod like images
19:26:51 <fungi> that sounds like a reasonable check
19:27:04 <clarkb> I don't expect any trouble. iirc last time we did this we had to modify some mailman pipelines but otherwise it went well
19:28:03 <clarkb> #topic Scheduling project renames
19:28:27 <fungi> we've still just got the one requested for now, i guess
19:28:34 <clarkb> I still feel like it is a bit early to schedule these as we're still coming up for air from the last fire... Maybe next week we'll have some thoughts based on how playbook updates go?
19:29:03 <clarkb> ya I'd like to think we can do it this month sometime. I'ev got family visiting through the month but should have time for this at some point. But I need to catch up with everything else going on before I can commit to that
19:29:17 <fungi> i think the osf/* repos are probably wanting to move to something like openinfra/ but i don't know that i have time to push on the patches and coordination for that as it's not at all urgent
19:29:32 <clarkb> fungi: do you think someone else at the foundation might be interested?
19:29:44 <clarkb> maybe jimmy?
19:29:45 <fungi> not really sure, tbh
19:29:48 <clarkb> I guess we can ask around and see
19:29:53 <fungi> can ask around, yeah
19:30:00 <clarkb> it would be good to do more than one if we can
19:30:15 <fungi> should probably be all of those in one go
19:31:02 <clarkb> ok, I think that is all for the agenda. I ran through it fairly quickly.
19:31:05 <clarkb> #topic Open Discussion
19:31:12 <clarkb> fungi: plenty of time for your extra item
19:31:38 <fungi> yeah, so there's this change...
19:31:46 <fungi> #link https://review.opendev.org/793999 Rename IRC channel for Octavia
19:32:09 <fungi> it raises the question of whether we should continue to squat abandoned channels when we move or otherwise clean our bots out of them
19:32:30 <clarkb> hrm my initial though is that keeping things cleaner is a good thing particularly after all the audit work you had to do
19:32:41 <fungi> given there's no join forwarding on oftc, there's less of an incentive to keep those old channels registered
19:33:12 <fungi> in a similar vein, i'd like to try to root out other abandoned channels (the ones we log are easy to audit, those we don't log are not so easy)
19:33:14 <clarkb> I'm ++ on renaming and dropping the old for simplicity
19:33:24 <corvus> yeah; and we're not actually revoking our access; we could conceivably still step in and set a topic if we needed to
19:33:39 <fungi> also to be polite to the fine folks who run the servers, i feel like we should unregister channels when we stop using them
19:33:51 <corvus> (no guarantee of course, but mostly just saying we're not likely to box ourselves into a corner and be sad)
19:34:23 <fungi> if we unregister them, we would theoretically lose control of them
19:34:26 <corvus> or, should we consider doing all of that 6 months after we stop using them?
19:34:46 <fungi> yeah, i think a delay before cleaning up is warranted, for sure
19:34:51 <clarkb> doing it with a delay seems reasonable. We still eventually reduce network overhead, but give us an out if things change in the short term
19:34:55 <corvus> because if we are going to unregister them, then losing privs immediately after the switch might make us sad
19:35:14 <fungi> and we can always appeal to the network operators in the (very rare, i'm sure) case where we'd need to do something to regain control of a channel we unregistered
19:35:26 <corvus> i wouldn't want someone to accidentally (or purposefully) hinder a move by squatting the channel right as the move happens
19:36:07 <fungi> so maybe we should comment out those entries with a date or something, so we remember when to unregister them?
19:36:35 <fungi> i don't think a heavyweight process is warranted
19:37:21 <clarkb> fungi: ++
19:37:31 <fungi> also worth noting we haven't rewritten the channel renaming section of our system-config irc doc yet, so i can take a shot at drafting the new policy there
19:37:37 <fungi> policy/process
19:37:43 <clarkb> thanks
19:38:01 <fungi> and big thanks to frickler for rewriting a good chunk of that document
19:38:18 <fungi> i've already pointed folks to the updated channel registration instructions
19:38:31 <fungi> so it's definitely come in handy
19:39:50 <corvus> comment process sgtm :)
19:40:17 <corvus> if we have time for another unscheduled topic...
19:40:25 <clarkb> we do, go for it
19:41:03 <corvus> i pushed up a zuul spec to see if the zuul project can achieve consensus to move to matrix; i consider that step 0 before talking to opendev about that
19:41:29 <corvus> like, my understanding of how we want these relationships to work is for projects to express desires/needs/etc and work with opendev to achieve them
19:41:53 <fungi> sounds right to me
19:42:00 <clarkb> yup (also Ishould add repsonding to that spec to my todo list)
19:42:00 <corvus> cool
19:42:23 <corvus> #link zuul spec about using matrix https://review.opendev.org/793669
19:42:28 <corvus> there's some positive response already
19:42:52 <corvus> i think it would be useful for people to weigh in both with their zuul hat and their opendev hat
19:43:16 <corvus> since most of the folks who haven't weighed in yet are in the overlapping area on the venn diagram :)
19:43:26 <clarkb> ++
19:44:07 <corvus> and if that proceeds, then i figure i'll be working on an infra-spec as a followup and talking about it more here
19:44:16 <fungi> which is probably at least subconsciously intentional since i personally want to know what other folks think before i potentially go clouding the input with my overlapping opinion space
19:45:28 <fungi> but more generally, i'm comfortable connecting to both irc and matrix from the same unified client for different communities, some of which aren't on irc networks which have matrix bridges
19:46:23 <clarkb> Anything else? if not we can get to breakfast/lunch/dinner 15 minutes early :)
19:46:25 <fungi> so weechat having a matrix plugin for matrix-only channels and native irc protocol support for non-matrix-accessible channels is going to be necessary for me anyway
19:46:35 <clarkb> fungi: ++
19:47:31 <clarkb> I'll go ahead and call it there. Thank you everyone. We'll see you here next week
19:47:51 <clarkb> #endmeeting