19:00:07 <clarkb> #startmeeting infra
19:00:07 <opendevmeet> Meeting started Tue Feb  6 19:00:07 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:07 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:07 <opendevmeet> The meeting name has been set to 'infra'
19:00:21 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/B7IJ56QOFLPJMGXRLJE4I44KG2RWPH4C/ Our Agenda
19:00:24 <clarkb> #topic Announcements
19:00:39 <clarkb> Service coordinator nominations are now open. You have until end of day February 20 to nominate yourself
19:00:53 <clarkb> We said we'd use UTC times for figuring out what end of day means
19:01:09 <clarkb> and then next week we'll have our preptg for opendev
19:01:19 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ Service coordinator election details
19:01:30 <clarkb> #link #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ preptg info
19:01:50 <clarkb> I'ev been trying to add background/depth/info to topics on the preptg etherpad
19:02:17 <clarkb> I expect that the later half of this week I'll dive in and do some of the last minute planning and organization around what to cover when. if you can add yuor interest to topics that would be helpful
19:02:25 <clarkb> feel free to add new topics or info to existing ones
19:03:50 <clarkb> #topic Server Upgrades
19:04:02 <clarkb> tonyb's meetpad stack got reviews (thank you for that)
19:04:23 <clarkb> frickler had some questions but I think the changes are mergeable. tonyb  if you happe nto be around any thing holding that up?
19:05:36 <fungi> i guess technically the keycloak change is also a server upgrade since it's being rebuilt on jammy instead of focal, but we also have a dedicated topic for that
19:06:06 <clarkb> ya
19:06:34 <clarkb> we can continue on and if tonyb joins us we can revisit this topic at the end of the meeting. I expect we'll finish early today and will have time for that
19:06:34 <fungi> i can be around to help tonyb test jitsi-meet servers and keep an eye on deploy jobs too
19:06:44 <clarkb> #topic Python container updates
19:06:59 <clarkb> I'm tempted to drop this agenda item for now and revisit it if necessary in the future.
19:07:25 <clarkb> We did rebuild our base images yesterday to pick up python and os updates though so also worth rebuilding and redeploying things
19:08:20 <clarkb> but nothing else to say about these at this time
19:08:36 <clarkb> Oh on the topic of containers skopeo can't talk to latest dockerd
19:09:05 <clarkb> there is an api protocol mismatch. This is less of a problem for us in opendev as we primarily use docker currently but some jobs do rely on skopeo and we may see this being problematic at some point
19:09:07 <clarkb> mostly a heads up
19:09:28 <clarkb> #topic AFS Quota Issues
19:09:38 <clarkb> I saw there was a chagne to stop mirroring some isos for centos stream
19:10:14 <fungi> yep, that merged
19:10:23 <clarkb> however the latest sync fungi did against better upstreams seems to have put us right up near the quota limit for centos stream again
19:10:52 <frickler> I also only did a very small quota bump
19:11:02 <frickler> planning for tonyb to do another one as training
19:11:43 <clarkb> cool I guess something to continue to improve and keep an eye on
19:11:53 <clarkb> and I've still got the old ubuntu ports cleanup on my todo list somewhere
19:12:40 <clarkb> #topic Keycloak Upgrade Struggles
19:12:56 <fungi> this is ready for reviews now
19:13:07 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/907141 Keycloak upgrade change
19:13:30 <fungi> there's a proposed plan etherpad, which includes the ip address of the latest held sample
19:13:54 <clarkb> #link https://etherpad.opendev.org/p/keycloak-refresh-2024 keycloak upgrade plan
19:14:27 <clarkb> fungi: all of the outstanding issues have been addressed then? The good news is we have much better testing now too which will hopefully make this better in the future
19:14:38 <fungi> last week i noticed that it wasn't actually using the database, so fixed the compose file and added an explicit testinfra test for it
19:14:40 <clarkb> as well as using a proper database (at least what keycloak considers to be proper)
19:15:21 <fungi> and yeah, no known issues at this point, other than just the mechanical process of needing to redo the zuul realm and add accounts to it from scratch
19:15:45 <fungi> there is an export/import feature, but starting over might be cleaner regardless
19:16:01 <fungi> and would allow us to more explicitly document the manual setup steps
19:16:07 <clarkb> sounds good. Thank you for working through this. And ya its a small enough setup that reproducing it shouldn't be ap roblem
19:16:21 <clarkb> and once this is all done we'll have to add db backups too
19:16:43 <fungi> correct. that's noted in the pad
19:16:54 <clarkb> perfect. Anything else?
19:17:03 <fungi> if anyone spots anything else we should do afterward, please add a note in the pad
19:17:10 <fungi> nothing else from me on this
19:17:39 <clarkb> #topic Gitea Upgrade and DB Config Changes
19:18:00 <clarkb> frickler noticed that some CI jobs that fetch constraints failed on http 500 errors from gitea
19:18:13 <clarkb> I was able to trace this back to mariadb errors for connection limits being reached
19:18:21 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/907500 Increase DB connection limits after HTTP 500 errors due to hitting the limit
19:18:31 <clarkb> I wrote this chagne to icnrease the limit on our dbs and deployed it yesterday
19:18:53 <clarkb> by default mariadb has a limit of 150 connections. Our container images reduce that to 100 (where we hit the limit and had the errors) so I doubled it to 200
19:19:08 <clarkb> separately gitea has released a new bugfix release
19:19:14 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/907472 Upgrade to the latest bugfix release
19:20:03 <clarkb> there is a fix for access to containers in the container registry that users shouldn't have. but we make everything public anyway and don't use the registry so that isn't a major issue for us
19:20:07 <clarkb> but good to catch up on the bugfixes
19:21:01 <clarkb> There are also more bugs around rendering things that have been reported upstream. but they are in code review comments and similar so again features we don't use.
19:21:07 <clarkb> apparently the next bugfix release will fix those though
19:22:03 <clarkb> Reviews welcome and let me know if you think we need to hold a node. Historically we've only done that for the major upgrades
19:22:11 <clarkb> but happy to set that up if we find something we are concerned about
19:22:18 <clarkb> #topic Etherpad 1.9.7 Upgrade
19:22:37 <clarkb> This is mostly a heads up that there is a new etherpad version available. The changelog for the new version makes note of changes to plugin installations
19:22:58 <clarkb> I think we'll want to ensure that doesn't affect ouf docker image (cross check with chagnes to upstream's docker image) and hold a node to see if our plugins work
19:23:10 <clarkb> I haven't written a change for this yet. Happy for someone else to if they are interested
19:24:03 <clarkb> #topic Open Discussion
19:24:08 <clarkb> That was all I had on the agenda
19:24:35 <clarkb> I did want to note that we had a user request to redact/delete a gerrit comment. corvus took care of that for us and wrote a tool to make it easy which is now in system-config/tools
19:24:39 <fungi> our matrix homeserver hosting plan with ems will be upgraded tomorrow
19:24:51 <fungi> shouldn't be any user-facing impact, but keep an eye out anyway
19:26:07 <clarkb> good reminder.
19:27:23 <clarkb> I'll probably have weird availability tomorrow as well due to family stuff
19:27:28 <corvus> i can make a system-config docs patch with the procedure if folks want
19:27:31 <clarkb> but not sure yet
19:27:43 <clarkb> corvus: a high level doc would probably be good as a pointer
19:27:54 <corvus> not sure the appropiate level of visibility for that
19:28:06 <fungi> or even just a few sentences in the gerrit.rst doc in system-config
19:28:35 <corvus> ok i'll add it
19:28:57 <fungi> thanks!
19:30:03 <clarkb> I'll give it a few more minutes for any other topics. Please go add your interest and topics to the preptg etherpad
19:30:26 <fungi> corvus: is https://zuul-ci.org/docs/zuul/latest/howtos/openid-with-keycloak.html the best place to start with redoing our keycloak config?
19:31:02 <fungi> looks like it's already pretty step-by-step but not sure what else you might have set up on the existing server beyond what's mentioned there
19:31:23 <corvus> fungi: i think so
19:31:41 <fungi> cool, i'll work from that
19:31:43 <fungi> thanks!
19:32:16 <corvus> ping me if you have q's
19:32:22 <fungi> gladly!
19:32:23 <clarkb> thank you everyone. We'll be back here next week and then we'll have the preptg stuff
19:32:33 <clarkb> #endmeeting