19:00:26 #startmeeting infra 19:00:26 Meeting started Tue Feb 13 19:00:26 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:26 The meeting name has been set to 'infra' 19:00:33 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/RNNSC7LBS44L3RKUSPLERGLNFQSMY4LT/ Our Agenda 19:00:37 #topic Announcements 19:00:54 First up a reminder that nominations for the OpenDev Service Coordinator position are now open (close eod utc time on the 20th) 19:01:08 This email has more details 19:01:10 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ 19:01:26 Second a reminder that we are having our preptg starting tomorrow 19:01:48 This event will occur February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service 19:01:53 #link https://etherpad.opendev.org/p/opendev-preptg-202402 PrePTG Agenda 19:02:03 The meetpad for this agenda is where we'll meet during those times 19:02:13 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ 19:02:23 that email has all the details I sent out previously 19:02:46 Finally please indicate your interest on topics so that we can try and organize. THough I'm happy to try and bridge discussions between the timezones as well if we have overlap 19:04:09 #topic Server Upgrades 19:04:27 I think tonyb was out last week so no new progress on the meetpad stuff. tonyb anything to add to that? 19:04:42 #link https://review.opendev.org/q/topic:jitsi_meet-jammy-update 19:05:04 Not really. 19:05:24 #link https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. 19:05:31 Perhaps if we get to the point that meetpad02 is a thing we could try it during the preptg 19:05:59 This is something that came up last week when fungi was booting keycloak servers. Basically some servers in rax don't have sse features that ubi9 needs 19:06:14 so we should filter out those servers as early as possible to avoid problems in the future as glibc base versions update 19:06:23 tonyb: ack and if not I'm always happy to do a test run 19:07:34 would be good to have 908512 in place before booting any new servers for meetpad or otherwise 19:07:49 though I just realized we may need to have it check for equiavlent amd flags? 19:07:56 not sure if amd flags for sse are the same as for intel 19:08:32 oh, good point, i tested that patch on rackspace but not other providers 19:08:33 spot checking gitea lb it is the same flag so thats fine 19:09:02 Anything else server related? 19:09:39 #topic AFS Quota Issues 19:10:20 I don't think there is anything new to report here. The ubuntu ports and centos stream volumes continue to be near their limits. I think frickler was going to work with tonyb on increasing quotas for something as a learning example 19:10:36 tonyb: did that happen if so/if not any questions about it? 19:11:08 It didn't happen as I was AWOL last week. 19:11:36 no questions at this point 19:11:41 i'm just glad you were found and brought back to us safely 19:11:53 #topic Keycloak upgrade struggles 19:12:04 struggles are over, i think? 19:12:22 at this point, i've created an account for myself in the zuul realm on the new keycloak03 19:12:25 at this point we have a new server (keycloak03) running the latest version of keycloak and a proper database backend. fungi has configured it using the doc from zuul (since that was the only use case so far) 19:12:49 ya so the next steps are for people to recreate their accounts and swap the server into production 19:12:59 corvus indicated a preference to do the recreation of accounts after the server was in prod 19:13:05 should we go ahead and update dns for that? 19:13:12 well my own at least :) 19:13:25 it makes things simpler and I'm a fan of simpler :) 19:13:32 is there anyone who wants to create an account before we switch over? if not, i'll un-wip and approve the changes nowish 19:13:54 i'll also add a blurb to our docs with instructions for account creation in the zuul realm 19:14:09 ++ to that 19:14:11 Sounds like a solid plan. 19:14:19 okay, i'll get on that after the meeting 19:14:23 thanks! 19:15:10 #topic Etherpad 1.9.7 upgrade 19:15:16 #link https://review.opendev.org/c/opendev/system-config/+/908328 Update Etherpad to 1.9.7 19:15:35 My concerns with plugin installations don't seem to have been a problem 19:15:51 the upstream dockerfile did not change as part of the update and the held test node was rendering things with our plugins just fine 19:15:56 * clarkb finds the test node ip 19:16:26 104.130.4.87 is the held node if anyone else wants to test it 19:16:38 I'm happy to hold off approving this until after the preptg since we're relying on etherpad for that 19:16:51 but would be good to get this off the todo list soon 19:17:29 but also I suspect its fine to upgrade whenever we want. It seem to be working on the held node 19:17:44 #topic Zuul Circular Dependency Refactor 19:18:12 if you aren't aware Zuul did a fairly large refactor around how it schedules changes to better handle circular dependencies 19:18:35 OpenDev's Zuul doesn't do circular deps so the main impact would be side effects ofthe refactor. A few web ui issues were discovered and have been addressed 19:18:44 please let the zuul team know if you find any unexpected behavior 19:18:51 technically inaccurate 19:18:58 the status page is a little different; some api endpoints provide different data 19:19:15 the zuul tenant on opendev's zuul is configured to support circular deps, so we could dogfood it, right? 19:19:17 fungi: yes zuul's tenant did add circular dep support for dogfooding/testing 19:19:27 but it is temporary aiui and not intended to be a long term thing 19:19:37 makes sense 19:19:53 i put a cycle through its paces 19:19:54 i can get a link 19:19:59 and zuul maintainers were told not to merge changes that way 19:20:20 https://zuul.opendev.org/t/zuul/buildset/a7e74d0cd40048c9aa85be8245ca8bd0 19:20:24 right. we did exercise it in production basically, though 19:20:43 that's a cycle of 2 changes if you want to see the buildset/build page changes 19:20:49 Once zuul is happy with the results of the refactor the next step in zuul is the 10.0.0 release whcih will also drop ansible 6 support and add ansible 9 19:21:12 I expect that the rate of change in zuul will have us deploying that when we do our weekly upgrades on satruday 19:21:24 possibly sooner if corvus or others want to restart thigns quicker to verify 19:21:40 should we send email to service-announce letting people know this is coming soon? 19:21:55 we did move all our tenants ot ansible 8 last year and I'm not aware of anyone relying on ansible 6 19:22:27 probably with regard to dropping ansible 6, couldn't hurt to give a heads up but i wouldn't worry too much about precise or lengthy advance notice 19:22:46 ok I'll send a note about it after lunch today 19:22:46 more like "if you see problems this might be why" 19:22:53 sounds good to me 19:24:02 And ya if you see anything weird let zuul know 19:24:02 oh fun side note: 19:24:19 apparently the command/shell warn setting is removed and it's just not going to warn anymore 19:24:26 in ansible 9 19:24:37 but if you keep setting the flag to ignore the warning on 8 it breaks 9 19:24:48 so you have to accept the warning in 8 if you want to work with 9 19:24:52 progress 19:24:55 so in all the places where we turn warnings off because we're running "git" or "apt" or whatever, we can remove those 19:25:27 zuul-jobs should already be cleaned up -- but i mention it since we may be able to clean up some stuff in system-config i think? 19:25:55 well https://review.opendev.org/908671 still needs to merge in zuul-jobs 19:25:57 yes possibly. Though much of system-config runs in nested ansible so may also require upgrading ansible on bridge to get to that point 19:26:12 clarkb: yep; so its a ways out 19:26:20 but just mentioning it as a point of shared interest 19:27:20 I guess the email should also indicate peopel can opt into ansible 9 after the switch and some point later we'll default to 9 19:27:35 ++ 19:28:23 #topic Open Discussion 19:28:56 I debugged a stuck job in zuul today that ended up being due to leaked nodes in rax dfw preventing any new boots there but the nodepool provider was still grabbing a single request that it paused after checking quota 19:29:22 I manually deleted a subset of nodes that nodepool wasn't deleting. I have taken a record of those I didn't delete in my homedir on bridge and will try to get to those later today 19:29:51 Second reminder that we're meeting tomorrow and the day after for our pre ptg 19:30:08 our agenda etherpad is here: https://etherpad.opendev.org/p/opendev-preptg-202402 and we'll use the corresponding meetpad 19:30:29 I've tried to organize topics and clump semi related items together 19:30:38 as well as add background where I can 19:31:10 Anythine else? 19:32:03 i think i forgot that was happening on top of valentine's day, but i'll make sure i'm around 19:32:25 fungi: ya it sounded like that wasn't a huge imposition and for some it was preferable to scheduling during later weeks 19:32:34 not a problem 19:32:44 i haven't made any reservations or anything 19:33:15 my kids have to make cards for everyone in their class this afternoon so today is worse than tomorrow for me :) 19:33:23 hah 19:33:32 good luck! 19:34:23 Last call for anything else. I'm not surprised the meeting is short when we plan a bunch of discussion for the next coupel of days 19:36:14 thank you all for your time today and later this week and otherwise! 19:36:19 #endmeeting