19:00:26 <clarkb> #startmeeting infra
19:00:26 <opendevmeet> Meeting started Tue Feb 13 19:00:26 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:26 <opendevmeet> The meeting name has been set to 'infra'
19:00:33 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/RNNSC7LBS44L3RKUSPLERGLNFQSMY4LT/ Our Agenda
19:00:37 <clarkb> #topic Announcements
19:00:54 <clarkb> First up a reminder that nominations for the OpenDev Service Coordinator position are now open (close eod utc time on the 20th)
19:01:08 <clarkb> This email has more details
19:01:10 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/
19:01:26 <clarkb> Second a reminder that we are having our preptg starting tomorrow
19:01:48 <clarkb> This event will occur February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service
19:01:53 <clarkb> #link https://etherpad.opendev.org/p/opendev-preptg-202402 PrePTG Agenda
19:02:03 <clarkb> The meetpad for this agenda is where we'll meet during those times
19:02:13 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/
19:02:23 <clarkb> that email has all the details I sent out previously
19:02:46 <clarkb> Finally please indicate your interest on topics so that we can try and organize. THough I'm happy to try and bridge discussions between the timezones as well if we have overlap
19:04:09 <clarkb> #topic Server Upgrades
19:04:27 <clarkb> I think tonyb was out last week so no new progress on the meetpad stuff. tonyb anything to add to that?
19:04:42 <clarkb> #link https://review.opendev.org/q/topic:jitsi_meet-jammy-update
19:05:04 <tonyb> Not really.
19:05:24 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels.
19:05:31 <tonyb> Perhaps if we get to the point that meetpad02 is a thing we could try it during the preptg
19:05:59 <clarkb> This is something that came up last week when fungi was booting keycloak servers. Basically some servers in rax don't have sse features that ubi9 needs
19:06:14 <clarkb> so we should filter out those servers as early as possible to avoid problems in the future as glibc base versions update
19:06:23 <clarkb> tonyb: ack and if not I'm always happy to do a test run
19:07:34 <clarkb> would be good to have 908512 in place before booting any new servers for meetpad or otherwise
19:07:49 <clarkb> though I just realized we may need to have it check for equiavlent amd flags?
19:07:56 <clarkb> not sure if amd flags for sse are the same as for intel
19:08:32 <fungi> oh, good point, i tested that patch on rackspace but not other providers
19:08:33 <clarkb> spot checking gitea lb it is the same flag so thats fine
19:09:02 <clarkb> Anything else server related?
19:09:39 <clarkb> #topic AFS Quota Issues
19:10:20 <clarkb> I don't think there is anything new to report here. The ubuntu ports and centos stream volumes continue to be near their limits. I think frickler was going to work with tonyb on increasing quotas for something as a learning example
19:10:36 <clarkb> tonyb: did that happen if so/if not any questions about it?
19:11:08 <tonyb> It didn't happen as I was AWOL last week.
19:11:36 <tonyb> no questions at this point
19:11:41 <fungi> i'm just glad you were found and brought back to us safely
19:11:53 <clarkb> #topic Keycloak upgrade struggles
19:12:04 <fungi> struggles are over, i think?
19:12:22 <fungi> at this point, i've created an account for myself in the zuul realm on the new keycloak03
19:12:25 <clarkb> at this point we have a new server (keycloak03) running the latest version of keycloak and a proper database backend. fungi has configured it using the doc from zuul (since that was the only use case so far)
19:12:49 <clarkb> ya so the next steps are for people to recreate their accounts and swap the server into production
19:12:59 <clarkb> corvus indicated a preference to do the recreation of accounts after the server was in prod
19:13:05 <clarkb> should we go ahead and update dns for that?
19:13:12 <corvus> well my own at least :)
19:13:25 <clarkb> it makes things simpler and I'm a fan of simpler :)
19:13:32 <fungi> is there anyone who wants to create an account before we switch over? if not, i'll un-wip and approve the changes nowish
19:13:54 <fungi> i'll also add a blurb to our docs with instructions for account creation in the zuul realm
19:14:09 <clarkb> ++ to that
19:14:11 <tonyb> Sounds like a solid plan.
19:14:19 <fungi> okay, i'll get on that after the meeting
19:14:23 <clarkb> thanks!
19:15:10 <clarkb> #topic Etherpad 1.9.7 upgrade
19:15:16 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/908328 Update Etherpad to 1.9.7
19:15:35 <clarkb> My concerns with plugin installations don't seem to have been a problem
19:15:51 <clarkb> the upstream dockerfile did not change as part of the update and the held test node was rendering things with our plugins just fine
19:15:56 * clarkb finds the test node ip
19:16:26 <clarkb> 104.130.4.87 is the held node if anyone else wants to test it
19:16:38 <clarkb> I'm happy to hold off approving this until after the preptg since we're relying on etherpad for that
19:16:51 <clarkb> but would be good to get this off the todo list soon
19:17:29 <clarkb> but also I suspect its fine to upgrade whenever we want. It seem to be working on the held node
19:17:44 <clarkb> #topic Zuul Circular Dependency Refactor
19:18:12 <clarkb> if you aren't aware Zuul did a fairly large refactor around how it schedules changes to better handle circular dependencies
19:18:35 <clarkb> OpenDev's Zuul doesn't do circular deps so the main impact would be side effects ofthe refactor. A few web ui issues were discovered and have been addressed
19:18:44 <clarkb> please let the zuul team know if you find any unexpected behavior
19:18:51 <fungi> technically inaccurate
19:18:58 <corvus> the status page is a little different; some api endpoints provide different data
19:19:15 <fungi> the zuul tenant on opendev's zuul is configured to support circular deps, so we could dogfood it, right?
19:19:17 <clarkb> fungi: yes zuul's tenant did add circular dep support for dogfooding/testing
19:19:27 <clarkb> but it is temporary aiui and not intended to be a long term thing
19:19:37 <fungi> makes sense
19:19:53 <corvus> i put a cycle through its paces
19:19:54 <corvus> i can get a link
19:19:59 <clarkb> and zuul maintainers were told not to merge changes that way
19:20:20 <corvus> https://zuul.opendev.org/t/zuul/buildset/a7e74d0cd40048c9aa85be8245ca8bd0
19:20:24 <fungi> right. we did exercise it in production basically, though
19:20:43 <corvus> that's a cycle of 2 changes if you want to see the buildset/build page changes
19:20:49 <clarkb> Once zuul is happy with the results of the refactor the next step in zuul is the 10.0.0 release whcih will also drop ansible 6 support and add ansible 9
19:21:12 <clarkb> I expect that the rate of change in zuul will have us deploying that when we do our weekly upgrades on satruday
19:21:24 <clarkb> possibly sooner if corvus or others want to restart thigns quicker to verify
19:21:40 <clarkb> should we send email to service-announce letting people know this is coming soon?
19:21:55 <clarkb> we did move all our tenants ot ansible 8 last year and I'm not aware of anyone relying on ansible 6
19:22:27 <fungi> probably with regard to dropping ansible 6, couldn't hurt to give a heads up but i wouldn't worry too much about precise or lengthy advance notice
19:22:46 <clarkb> ok I'll send a note about it after lunch today
19:22:46 <fungi> more like "if you see problems this might be why"
19:22:53 <corvus> sounds good to me
19:24:02 <clarkb> And ya if you see anything weird let zuul know
19:24:02 <corvus> oh fun side note:
19:24:19 <corvus> apparently the command/shell warn setting is removed and it's just not going to warn anymore
19:24:26 <corvus> in ansible 9
19:24:37 <clarkb> but if you keep setting the flag to ignore the warning on 8 it breaks 9
19:24:48 <clarkb> so you have to accept the warning in 8 if you want to work with 9
19:24:52 <clarkb> progress
19:24:55 <corvus> so in all the places where we turn warnings off because we're running "git" or "apt" or whatever, we can remove those
19:25:27 <corvus> zuul-jobs should already be cleaned up -- but i mention it since we may be able to clean up some stuff in system-config i think?
19:25:55 <corvus> well https://review.opendev.org/908671 still needs to merge in zuul-jobs
19:25:57 <clarkb> yes possibly. Though much of system-config runs in nested ansible so may also require upgrading ansible on bridge to get to that point
19:26:12 <corvus> clarkb: yep; so its a ways out
19:26:20 <corvus> but just mentioning it as a point of shared interest
19:27:20 <clarkb> I guess the email should also indicate peopel can opt into ansible 9 after the switch and some point later we'll default to 9
19:27:35 <corvus> ++
19:28:23 <clarkb> #topic Open Discussion
19:28:56 <clarkb> I debugged a stuck job in zuul today that ended up being due to leaked nodes in rax dfw preventing any new boots there but the nodepool provider was still grabbing a single request that it paused after checking quota
19:29:22 <clarkb> I manually deleted a subset of nodes that nodepool wasn't deleting. I have taken a record of those I didn't delete in my homedir on bridge and will try to get to those later today
19:29:51 <clarkb> Second reminder that we're meeting tomorrow and the day after for our pre ptg
19:30:08 <clarkb> our agenda etherpad is here: https://etherpad.opendev.org/p/opendev-preptg-202402 and we'll use the corresponding meetpad
19:30:29 <clarkb> I've tried to organize topics and clump semi related items together
19:30:38 <clarkb> as well as add background where I can
19:31:10 <clarkb> Anythine else?
19:32:03 <fungi> i think i forgot that was happening on top of valentine's day, but i'll make sure i'm around
19:32:25 <clarkb> fungi: ya it sounded like that wasn't a huge imposition and for some it was preferable to scheduling during later weeks
19:32:34 <fungi> not a problem
19:32:44 <fungi> i haven't made any reservations or anything
19:33:15 <clarkb> my kids have to make cards for everyone in their class this afternoon so today is worse than tomorrow for me :)
19:33:23 <fungi> hah
19:33:32 <fungi> good luck!
19:34:23 <clarkb> Last call for anything else. I'm not surprised the meeting is short when we plan a bunch of discussion for the next coupel of days
19:36:14 <clarkb> thank you all for your time today and later this week and otherwise!
19:36:19 <clarkb> #endmeeting