19:01:07 <clarkb> #startmeeting Infra
19:01:08 <openstack> Meeting started Tue Jan 12 19:01:07 2021 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:11 <openstack> The meeting name has been set to 'infra'
19:01:27 <clarkb> #topic Announcements
19:01:53 <clarkb> The foundation's board of director elections are happening this week. If you are a foundation member you shoulve have received an email with your voting instructions in it
19:02:07 <clarkb> Please take a few minutes to go and vote if you are able to
19:02:17 <clarkb> #topic Actions from last meeting
19:02:23 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-01-05-19.01.txt minutes from last meeting
19:02:51 <clarkb> corvus had an action to implement Gerrit WIP support in Zuul. I believe the change was written and I reviewed it. corvus if you are around has that merged yet?
19:03:12 <fungi> if not, i'd appreciate a link so i can review
19:03:18 <clarkb> ya trying to find that now
19:03:38 <clarkb> #link https://review.opendev.org/c/zuul/zuul/+/769436 looks like it merged according to gerritbot which is where I Found the link
19:03:43 <fungi> awesome
19:03:51 <fungi> so we need a restart to pick that up
19:03:53 <clarkb> that means the next step for us in supporting WIP is to restart the zuul scheduler and then test again
19:03:55 <clarkb> yup
19:04:02 <fungi> at least the scheduler, presumably
19:04:14 <fungi> ahh, you said those things
19:04:26 * fungi is a bit scattered today
19:04:31 <clarkb> me too
19:04:37 <clarkb> #topic Priority Efforts
19:04:42 <clarkb> #topic OpenDev
19:05:12 <clarkb> First up it was pointed out that we were overdue for service coordinator nominations (and an election if necesasry)
19:05:20 <clarkb> #link http://lists.opendev.org/pipermail/service-discuss/2021-January/000161.html
19:05:34 <clarkb> I sent email dscribing what I think is a reasonable plan for addressing this miss
19:05:55 <clarkb> if you think that plan is flawed in some way please respond to the mailing list and help us set up a new less flawed plan :)
19:06:09 <clarkb> any other election feedback is also appreciated there.
19:06:39 <clarkb> If you are interested in taking on the service coordinator role I'm happy to talk about it if you need more info before committing. Otherwise please send email to the service-discuss list nominating yourself
19:07:01 <clarkb> as mentioned in the email I linked I've done it for a number of cycles now and think that new perspectives would be a good thing to have.
19:07:19 <clarkb> I'm not saying I won't run again if necessary, but do strongly feel that having a bit more rotation would be a good thing
19:07:32 <fungi> prepare for zbr to volunteer
19:08:04 <fungi> zbr: you know you want to!
19:08:33 <clarkb> the email I sent also sets up a week for electiosn if they become necessary. And to avoid this problem of missing them in the future I set out dates for the next set of nominations and elections
19:09:05 <clarkb> I'll add them into my calendar reminders if later if no one ends up objecting to that proposed plan
19:09:38 <clarkb> The opendev project update for the foundation's annual report is basically finalized now
19:09:40 <clarkb> #link https://etherpad.opendev.org/p/opendev-2020-annual-report
19:09:52 <clarkb> I think those are due tomorrow so if there are important edits get them in now (or let me know and I can make them)
19:10:40 <clarkb> The last opendev topic I wanted to bring up was the gitea 1.13.1 upgrade.
19:10:42 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/769226
19:10:43 <mordred> lgtm
19:11:19 <clarkb> I think this is ready for serious consideration. There is a held gitea test node (somewhere I need to find it again via nodepool) running 1.13.1 to help confirm we're happy with it
19:11:46 <clarkb> my biggest concern is that gitea 1.13 added some big new features like kanban boards to projects so want to make sure we're presenting a gitea that is consistent with our current setup
19:12:07 <mordred> patch itself looks good - assuming the test node is solid
19:12:25 <clarkb> ya I looked it over (and I think fungi did too?) and it seemed fine
19:12:35 <fungi> i did, think i already +2'd
19:12:56 <clarkb> if it does look good I should have plenty of time to watch it land and monitor it thursday
19:13:02 <fungi> trying to get through some project-config backlog today, i noticed this one has a potential for global disruption but would like to merge and watch it closely after the meeting:
19:13:06 <fungi> #link https://review.opendev.org/760495 Use internal mirror for RAX IAD/DFW
19:13:33 <fungi> that switches the interface for mirror connections in two regions to use the second nic where bandwidth may be less constrained
19:13:39 <clarkb> ++
19:13:44 <fungi> (we're already doing it that way in one)
19:14:16 <mordred> ++
19:14:39 <mordred> might be worth simplifying that to "if nodepool.cloud == 'rax'" at this point
19:14:57 <fungi> true
19:15:01 <fungi> ianw: ^ wdyt?
19:15:23 <mordred> not that I expect us to grow new rax regions or anything
19:15:46 <fungi> i'm happy to tweak it before approving if folks prefer
19:15:47 <ianw> yeah, i don't mind; we have used this before for switching in other clouds so it might be useful to keep the cloud/region append just as an example of what to do
19:16:13 <fungi> i'm good with it as-is too
19:16:58 <clarkb> why don't we land it as is then just to avoid unnecessary churn
19:17:07 <fungi> wfm, will do
19:17:36 <clarkb> #topic General Topics
19:17:43 <clarkb> #topic Bup and Borg Backups
19:18:01 <clarkb> ianw: this was on my list to check up on after the holidays. Are we completely off of bup at this point for new backups?
19:18:12 <clarkb> and if so should I drop this item from our meeting agendas?
19:18:28 <ianw> yep, since https://review.opendev.org/c/opendev/system-config/+/766300
19:18:44 <ianw> i still have to finish the cleanup with https://review.opendev.org/c/opendev/system-config/+/766630/
19:19:23 <ianw> i will work on that
19:19:26 <clarkb> thanks
19:19:31 <clarkb> and thank you for working on that
19:19:41 <clarkb> we should be able to start looking at focal nodes now I think
19:19:47 <clarkb> (since this was a big hold up for that iirc)
19:20:03 <ianw> maybe keep it for one more week as i cleanup the old servers
19:20:12 <clarkb> can do
19:20:23 <fungi> i can't remember, was borg manually added to the wiki server? if not, i'll try to prioritize that
19:20:33 <clarkb> fungi: I'm not sure
19:20:39 <clarkb> probably not?
19:20:50 <ianw> fungi: not sure either.  i don't remember doing it.  i can look into that
19:21:02 <fungi> it was being backed up with bup
19:21:08 <fungi> (still is afaik)
19:21:11 <clarkb> it likely still is ya
19:21:16 <clarkb> since the bup bits are still there iirc
19:21:24 <clarkb> (we have to keep them around for backup retention anyway)
19:22:26 <ianw> maybe give me an action item to confirm wiki being backed up so we don't forget
19:22:41 <clarkb> #action ianw confirm wiki is still backed up after bup to borg migration
19:23:13 <clarkb> #topic openstackid.org scale down
19:24:02 <clarkb> fungi and I conferred with smarcet to confirm that the desired running state for openstackid.org is to scale it down after we scaled it up for the summit
19:24:41 <clarkb> fungi returned openstackid.org to its ansible + puppet managed state and I have shut down the two new servers in vexxhost and removed their A and AAAA dns records
19:24:55 <clarkb> in a day or two when we're happy that we've cleaned things up without disruption I will delete them
19:25:29 <clarkb> we also discovered that smarcet uses docker for openstackid development so it sounds like we can collaborate to convert that over to a ansible + docker-compose + docker deployment
19:25:41 <clarkb> however, that is more of a "yes we can do that" idea at this point
19:26:04 <clarkb> mostly an fyi on that since that service had some changes and now we've undone them. And now everyone else is caught up
19:26:12 <clarkb> #topic Open Discussion
19:26:40 <clarkb> That was what I had written down in notes really quickly after my previous meetings ended. We've got plenty of time to tlk about other topics if we need to
19:27:03 <clarkb> oh!
19:27:13 <clarkb> https://bugs.chromium.org/p/gerrit/issues/detail?id=13930 is worth pointing out
19:27:26 <clarkb> I think I managed to figure out why fedora 33 users are still having trouble with new gerrit and rsa
19:27:57 <clarkb> long story short is upstream openssh has only deprecated ssh-rsa for hostkey signature exchanges. Fedora has disabled it for hostkey stuff and for public key auth
19:28:38 <clarkb> Gerrit does rsa-sha2-* just fine for hostkey stuff but it does not work for pubkey auth because that requires supporting the server-sig-algs kex extension in the server and gerrit's server doesn't seem to support that
19:28:41 <corvus> oh sorry i got sucked into an issue
19:29:04 <clarkb> fedora 33 users can work around this by enabling ssh-rsa or switching to an ed25119 or ecdsa key for auth
19:29:21 <ianw> i feel like i'm using fedora 33 and it is working
19:29:31 <ianw> ohhh, i'm using and ed25119 key
19:29:41 <clarkb> ya this is specific to using rsa keys to auth
19:29:48 <fungi> ianw: see, you anticipated this
19:30:32 <clarkb> if fedora users have qusetions we can point them to that bug. I'm hoping upstream will say "oh thats an easy fix" and it will magically happen but I think it may be more involved
19:30:45 <ianw> heh, yep, istr having to merge some changes to our puppet to handle ed keys years ago :)
19:30:46 <clarkb> in particular I think the proper way to fix this is to update mina upstream since other mina sshds will want the same fix
19:31:31 <clarkb> eventually ssh clients should switch their fallback rsa pubkey auth type to rsa-sha2-something. But until that happens I expect this will be a problem for people
19:31:44 <clarkb> related: if anyone knows fedora devs ^ it might be worth suggesting they make that switch
19:31:52 <clarkb> since they are disabling the alternative
19:31:59 <fungi> well, and also fedora could improve the situation by not still looking for sha-1 with ssh-rsa and instead trying sha-2 first
19:32:09 <fungi> er, what you also just typed
19:32:25 <clarkb> ya it feels like fedora's disabling of ssh-rsa missed an important step
19:32:46 <clarkb> which was to not fallback to ssh-rsa when doing rsa pubkey auth talking to a server that doesn't do server-sig-algs
19:33:21 <clarkb> https://tools.ietf.org/html/rfc8332#section-3.3 notes that this is the expected end state once rsa-sha2 is sufficiently ubiquitous (and it seems fedora is saying that it is)
19:34:28 <ianw> is it https://bugzilla.redhat.com/show_bug.cgi?id=1881301 ?
19:34:31 <openstack> bugzilla.redhat.com bug 1881301 in openssh "openssh-clients do not accept PubkeyAcceptedKeyTypes rsa-sha2-512/256" [Unspecified,Closed: errata] - Assigned to jjelen
19:35:03 <clarkb> ianw: ya I think that is the bug on the fedora side
19:36:28 <ianw> i'd like to get the zuul summary plugin going
19:36:32 <ianw> the review stack is @ https://review.opendev.org/q/topic:%22gerrit-admin-user%22
19:37:12 <clarkb> ianw: and that plugin is hosted upstream now too right?
19:37:28 <ianw> yes, that's right
19:37:57 <clarkb> excellent I'll add that very high on the review todo list once I've got time to do that (probably tomorrow?)
19:38:45 <ianw> ok, yeah the stuff underneath is to cleanup review-dev, then initalize and populate gerrit during testing, then add selenium testing and take screenshots, and then finally add the plugin
19:39:23 <ianw> with a little bazelisk stuff for good measure :)
19:41:11 <clarkb> alright anything else?
19:41:14 <zbr> i am back
19:41:24 <zbr> (reading backlog)
19:42:22 <fungi> zbr: just me encouraging folks to make our service coordinator election an election this time
19:42:30 <clarkb> I'll give zbr a coupel of minutes to catch up but then if that is it we can call it a meeting
19:42:45 <fungi> (was the nick highlight i mean)
19:45:03 <zbr> sure. ok to call the meeting off.
19:45:28 <clarkb> thanks everyone. Sorry I missed the agenda. I'll do my best to not dismiss the alert until actually done in the future :)
19:45:46 <clarkb> I think what happened was I habitually swiped it away when my phone made noise and had a thing pop up
19:45:52 <clarkb> #endmeeting