| clarkb | Just about meeting time | 18:59 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Feb 6 19:00:07 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/B7IJ56QOFLPJMGXRLJE4I44KG2RWPH4C/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | Service coordinator nominations are now open. You have until end of day February 20 to nominate yourself | 19:00 |
| clarkb | We said we'd use UTC times for figuring out what end of day means | 19:00 |
| clarkb | and then next week we'll have our preptg for opendev | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ Service coordinator election details | 19:01 |
| clarkb | #link #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ preptg info | 19:01 |
| clarkb | I'ev been trying to add background/depth/info to topics on the preptg etherpad | 19:01 |
| clarkb | I expect that the later half of this week I'll dive in and do some of the last minute planning and organization around what to cover when. if you can add yuor interest to topics that would be helpful | 19:02 |
| clarkb | feel free to add new topics or info to existing ones | 19:02 |
| clarkb | #topic Server Upgrades | 19:03 |
| clarkb | tonyb's meetpad stack got reviews (thank you for that) | 19:04 |
| clarkb | frickler had some questions but I think the changes are mergeable. tonyb if you happe nto be around any thing holding that up? | 19:04 |
| fungi | i guess technically the keycloak change is also a server upgrade since it's being rebuilt on jammy instead of focal, but we also have a dedicated topic for that | 19:05 |
| clarkb | ya | 19:06 |
| clarkb | we can continue on and if tonyb joins us we can revisit this topic at the end of the meeting. I expect we'll finish early today and will have time for that | 19:06 |
| fungi | i can be around to help tonyb test jitsi-meet servers and keep an eye on deploy jobs too | 19:06 |
| clarkb | #topic Python container updates | 19:06 |
| clarkb | I'm tempted to drop this agenda item for now and revisit it if necessary in the future. | 19:06 |
| clarkb | We did rebuild our base images yesterday to pick up python and os updates though so also worth rebuilding and redeploying things | 19:07 |
| clarkb | but nothing else to say about these at this time | 19:08 |
| clarkb | Oh on the topic of containers skopeo can't talk to latest dockerd | 19:08 |
| clarkb | there is an api protocol mismatch. This is less of a problem for us in opendev as we primarily use docker currently but some jobs do rely on skopeo and we may see this being problematic at some point | 19:09 |
| clarkb | mostly a heads up | 19:09 |
| clarkb | #topic AFS Quota Issues | 19:09 |
| clarkb | I saw there was a chagne to stop mirroring some isos for centos stream | 19:09 |
| fungi | yep, that merged | 19:10 |
| clarkb | however the latest sync fungi did against better upstreams seems to have put us right up near the quota limit for centos stream again | 19:10 |
| frickler | I also only did a very small quota bump | 19:10 |
| frickler | planning for tonyb to do another one as training | 19:11 |
| clarkb | cool I guess something to continue to improve and keep an eye on | 19:11 |
| clarkb | and I've still got the old ubuntu ports cleanup on my todo list somewhere | 19:11 |
| clarkb | #topic Keycloak Upgrade Struggles | 19:12 |
| fungi | this is ready for reviews now | 19:12 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/907141 Keycloak upgrade change | 19:13 |
| fungi | there's a proposed plan etherpad, which includes the ip address of the latest held sample | 19:13 |
| clarkb | #link https://etherpad.opendev.org/p/keycloak-refresh-2024 keycloak upgrade plan | 19:13 |
| clarkb | fungi: all of the outstanding issues have been addressed then? The good news is we have much better testing now too which will hopefully make this better in the future | 19:14 |
| fungi | last week i noticed that it wasn't actually using the database, so fixed the compose file and added an explicit testinfra test for it | 19:14 |
| clarkb | as well as using a proper database (at least what keycloak considers to be proper) | 19:14 |
| fungi | and yeah, no known issues at this point, other than just the mechanical process of needing to redo the zuul realm and add accounts to it from scratch | 19:15 |
| fungi | there is an export/import feature, but starting over might be cleaner regardless | 19:15 |
| fungi | and would allow us to more explicitly document the manual setup steps | 19:16 |
| clarkb | sounds good. Thank you for working through this. And ya its a small enough setup that reproducing it shouldn't be ap roblem | 19:16 |
| clarkb | and once this is all done we'll have to add db backups too | 19:16 |
| fungi | correct. that's noted in the pad | 19:16 |
| clarkb | perfect. Anything else? | 19:16 |
| fungi | if anyone spots anything else we should do afterward, please add a note in the pad | 19:17 |
| fungi | nothing else from me on this | 19:17 |
| clarkb | #topic Gitea Upgrade and DB Config Changes | 19:17 |
| clarkb | frickler noticed that some CI jobs that fetch constraints failed on http 500 errors from gitea | 19:18 |
| clarkb | I was able to trace this back to mariadb errors for connection limits being reached | 19:18 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/907500 Increase DB connection limits after HTTP 500 errors due to hitting the limit | 19:18 |
| clarkb | I wrote this chagne to icnrease the limit on our dbs and deployed it yesterday | 19:18 |
| clarkb | by default mariadb has a limit of 150 connections. Our container images reduce that to 100 (where we hit the limit and had the errors) so I doubled it to 200 | 19:18 |
| clarkb | separately gitea has released a new bugfix release | 19:19 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/907472 Upgrade to the latest bugfix release | 19:19 |
| clarkb | there is a fix for access to containers in the container registry that users shouldn't have. but we make everything public anyway and don't use the registry so that isn't a major issue for us | 19:20 |
| clarkb | but good to catch up on the bugfixes | 19:20 |
| clarkb | There are also more bugs around rendering things that have been reported upstream. but they are in code review comments and similar so again features we don't use. | 19:21 |
| clarkb | apparently the next bugfix release will fix those though | 19:21 |
| clarkb | Reviews welcome and let me know if you think we need to hold a node. Historically we've only done that for the major upgrades | 19:22 |
| clarkb | but happy to set that up if we find something we are concerned about | 19:22 |
| clarkb | #topic Etherpad 1.9.7 Upgrade | 19:22 |
| clarkb | This is mostly a heads up that there is a new etherpad version available. The changelog for the new version makes note of changes to plugin installations | 19:22 |
| clarkb | I think we'll want to ensure that doesn't affect ouf docker image (cross check with chagnes to upstream's docker image) and hold a node to see if our plugins work | 19:22 |
| clarkb | I haven't written a change for this yet. Happy for someone else to if they are interested | 19:23 |
| clarkb | #topic Open Discussion | 19:24 |
| clarkb | That was all I had on the agenda | 19:24 |
| clarkb | I did want to note that we had a user request to redact/delete a gerrit comment. corvus took care of that for us and wrote a tool to make it easy which is now in system-config/tools | 19:24 |
| fungi | our matrix homeserver hosting plan with ems will be upgraded tomorrow | 19:24 |
| fungi | shouldn't be any user-facing impact, but keep an eye out anyway | 19:24 |
| clarkb | good reminder. | 19:26 |
| clarkb | I'll probably have weird availability tomorrow as well due to family stuff | 19:27 |
| corvus | i can make a system-config docs patch with the procedure if folks want | 19:27 |
| clarkb | but not sure yet | 19:27 |
| clarkb | corvus: a high level doc would probably be good as a pointer | 19:27 |
| corvus | not sure the appropiate level of visibility for that | 19:27 |
| fungi | or even just a few sentences in the gerrit.rst doc in system-config | 19:28 |
| corvus | ok i'll add it | 19:28 |
| fungi | thanks! | 19:28 |
| clarkb | I'll give it a few more minutes for any other topics. Please go add your interest and topics to the preptg etherpad | 19:30 |
| fungi | corvus: is https://zuul-ci.org/docs/zuul/latest/howtos/openid-with-keycloak.html the best place to start with redoing our keycloak config? | 19:30 |
| fungi | looks like it's already pretty step-by-step but not sure what else you might have set up on the existing server beyond what's mentioned there | 19:31 |
| corvus | fungi: i think so | 19:31 |
| fungi | cool, i'll work from that | 19:31 |
| fungi | thanks! | 19:31 |
| corvus | ping me if you have q's | 19:32 |
| fungi | gladly! | 19:32 |
| clarkb | thank you everyone. We'll be back here next week and then we'll have the preptg stuff | 19:32 |
| clarkb | #endmeeting | 19:32 |
| opendevmeet | Meeting ended Tue Feb 6 19:32:33 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:32 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-02-06-19.00.html | 19:32 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-02-06-19.00.txt | 19:32 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-02-06-19.00.log.html | 19:32 |
| fungi | thanks clarkb! | 19:32 |
| clarkb | I've got 20 something minutes before my next meeting. Long enough to eat some lunch | 19:33 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!