19:00:07 #startmeeting infra 19:00:07 Meeting started Tue Feb 6 19:00:07 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:07 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:07 The meeting name has been set to 'infra' 19:00:21 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/B7IJ56QOFLPJMGXRLJE4I44KG2RWPH4C/ Our Agenda 19:00:24 #topic Announcements 19:00:39 Service coordinator nominations are now open. You have until end of day February 20 to nominate yourself 19:00:53 We said we'd use UTC times for figuring out what end of day means 19:01:09 and then next week we'll have our preptg for opendev 19:01:19 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ Service coordinator election details 19:01:30 #link #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ preptg info 19:01:50 I'ev been trying to add background/depth/info to topics on the preptg etherpad 19:02:17 I expect that the later half of this week I'll dive in and do some of the last minute planning and organization around what to cover when. if you can add yuor interest to topics that would be helpful 19:02:25 feel free to add new topics or info to existing ones 19:03:50 #topic Server Upgrades 19:04:02 tonyb's meetpad stack got reviews (thank you for that) 19:04:23 frickler had some questions but I think the changes are mergeable. tonyb if you happe nto be around any thing holding that up? 19:05:36 i guess technically the keycloak change is also a server upgrade since it's being rebuilt on jammy instead of focal, but we also have a dedicated topic for that 19:06:06 ya 19:06:34 we can continue on and if tonyb joins us we can revisit this topic at the end of the meeting. I expect we'll finish early today and will have time for that 19:06:34 i can be around to help tonyb test jitsi-meet servers and keep an eye on deploy jobs too 19:06:44 #topic Python container updates 19:06:59 I'm tempted to drop this agenda item for now and revisit it if necessary in the future. 19:07:25 We did rebuild our base images yesterday to pick up python and os updates though so also worth rebuilding and redeploying things 19:08:20 but nothing else to say about these at this time 19:08:36 Oh on the topic of containers skopeo can't talk to latest dockerd 19:09:05 there is an api protocol mismatch. This is less of a problem for us in opendev as we primarily use docker currently but some jobs do rely on skopeo and we may see this being problematic at some point 19:09:07 mostly a heads up 19:09:28 #topic AFS Quota Issues 19:09:38 I saw there was a chagne to stop mirroring some isos for centos stream 19:10:14 yep, that merged 19:10:23 however the latest sync fungi did against better upstreams seems to have put us right up near the quota limit for centos stream again 19:10:52 I also only did a very small quota bump 19:11:02 planning for tonyb to do another one as training 19:11:43 cool I guess something to continue to improve and keep an eye on 19:11:53 and I've still got the old ubuntu ports cleanup on my todo list somewhere 19:12:40 #topic Keycloak Upgrade Struggles 19:12:56 this is ready for reviews now 19:13:07 #link https://review.opendev.org/c/opendev/system-config/+/907141 Keycloak upgrade change 19:13:30 there's a proposed plan etherpad, which includes the ip address of the latest held sample 19:13:54 #link https://etherpad.opendev.org/p/keycloak-refresh-2024 keycloak upgrade plan 19:14:27 fungi: all of the outstanding issues have been addressed then? The good news is we have much better testing now too which will hopefully make this better in the future 19:14:38 last week i noticed that it wasn't actually using the database, so fixed the compose file and added an explicit testinfra test for it 19:14:40 as well as using a proper database (at least what keycloak considers to be proper) 19:15:21 and yeah, no known issues at this point, other than just the mechanical process of needing to redo the zuul realm and add accounts to it from scratch 19:15:45 there is an export/import feature, but starting over might be cleaner regardless 19:16:01 and would allow us to more explicitly document the manual setup steps 19:16:07 sounds good. Thank you for working through this. And ya its a small enough setup that reproducing it shouldn't be ap roblem 19:16:21 and once this is all done we'll have to add db backups too 19:16:43 correct. that's noted in the pad 19:16:54 perfect. Anything else? 19:17:03 if anyone spots anything else we should do afterward, please add a note in the pad 19:17:10 nothing else from me on this 19:17:39 #topic Gitea Upgrade and DB Config Changes 19:18:00 frickler noticed that some CI jobs that fetch constraints failed on http 500 errors from gitea 19:18:13 I was able to trace this back to mariadb errors for connection limits being reached 19:18:21 #link https://review.opendev.org/c/opendev/system-config/+/907500 Increase DB connection limits after HTTP 500 errors due to hitting the limit 19:18:31 I wrote this chagne to icnrease the limit on our dbs and deployed it yesterday 19:18:53 by default mariadb has a limit of 150 connections. Our container images reduce that to 100 (where we hit the limit and had the errors) so I doubled it to 200 19:19:08 separately gitea has released a new bugfix release 19:19:14 #link https://review.opendev.org/c/opendev/system-config/+/907472 Upgrade to the latest bugfix release 19:20:03 there is a fix for access to containers in the container registry that users shouldn't have. but we make everything public anyway and don't use the registry so that isn't a major issue for us 19:20:07 but good to catch up on the bugfixes 19:21:01 There are also more bugs around rendering things that have been reported upstream. but they are in code review comments and similar so again features we don't use. 19:21:07 apparently the next bugfix release will fix those though 19:22:03 Reviews welcome and let me know if you think we need to hold a node. Historically we've only done that for the major upgrades 19:22:11 but happy to set that up if we find something we are concerned about 19:22:18 #topic Etherpad 1.9.7 Upgrade 19:22:37 This is mostly a heads up that there is a new etherpad version available. The changelog for the new version makes note of changes to plugin installations 19:22:58 I think we'll want to ensure that doesn't affect ouf docker image (cross check with chagnes to upstream's docker image) and hold a node to see if our plugins work 19:23:10 I haven't written a change for this yet. Happy for someone else to if they are interested 19:24:03 #topic Open Discussion 19:24:08 That was all I had on the agenda 19:24:35 I did want to note that we had a user request to redact/delete a gerrit comment. corvus took care of that for us and wrote a tool to make it easy which is now in system-config/tools 19:24:39 our matrix homeserver hosting plan with ems will be upgraded tomorrow 19:24:51 shouldn't be any user-facing impact, but keep an eye out anyway 19:26:07 good reminder. 19:27:23 I'll probably have weird availability tomorrow as well due to family stuff 19:27:28 i can make a system-config docs patch with the procedure if folks want 19:27:31 but not sure yet 19:27:43 corvus: a high level doc would probably be good as a pointer 19:27:54 not sure the appropiate level of visibility for that 19:28:06 or even just a few sentences in the gerrit.rst doc in system-config 19:28:35 ok i'll add it 19:28:57 thanks! 19:30:03 I'll give it a few more minutes for any other topics. Please go add your interest and topics to the preptg etherpad 19:30:26 corvus: is https://zuul-ci.org/docs/zuul/latest/howtos/openid-with-keycloak.html the best place to start with redoing our keycloak config? 19:31:02 looks like it's already pretty step-by-step but not sure what else you might have set up on the existing server beyond what's mentioned there 19:31:23 fungi: i think so 19:31:41 cool, i'll work from that 19:31:43 thanks! 19:32:16 ping me if you have q's 19:32:22 gladly! 19:32:23 thank you everyone. We'll be back here next week and then we'll have the preptg stuff 19:32:33 #endmeeting