| clarkb | almost time for our weekly meeting. We'll get started in just a few minutes | 18:56 |
|---|---|---|
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Feb 13 19:00:26 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/RNNSC7LBS44L3RKUSPLERGLNFQSMY4LT/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | First up a reminder that nominations for the OpenDev Service Coordinator position are now open (close eod utc time on the 20th) | 19:00 |
| clarkb | This email has more details | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/TB2OFBIGWZEYC7L4MCYA46EXIX5T47TY/ | 19:01 |
| clarkb | Second a reminder that we are having our preptg starting tomorrow | 19:01 |
| clarkb | This event will occur February 14+15 at 14:00-16:00UTC and 22:00-00:00UTC both days on our meetpad service | 19:01 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-preptg-202402 PrePTG Agenda | 19:01 |
| clarkb | The meetpad for this agenda is where we'll meet during those times | 19:02 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/KUZG7DPU77GC3M3MKKTSYY655JLIYV4Z/ | 19:02 |
| clarkb | that email has all the details I sent out previously | 19:02 |
| clarkb | Finally please indicate your interest on topics so that we can try and organize. THough I'm happy to try and bridge discussions between the timezones as well if we have overlap | 19:02 |
| clarkb | #topic Server Upgrades | 19:04 |
| clarkb | I think tonyb was out last week so no new progress on the meetpad stuff. tonyb anything to add to that? | 19:04 |
| clarkb | #link https://review.opendev.org/q/topic:jitsi_meet-jammy-update | 19:04 |
| tonyb | Not really. | 19:05 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/908512 Ensure new servers have sufficient CPU version levels. | 19:05 |
| tonyb | Perhaps if we get to the point that meetpad02 is a thing we could try it during the preptg | 19:05 |
| clarkb | This is something that came up last week when fungi was booting keycloak servers. Basically some servers in rax don't have sse features that ubi9 needs | 19:05 |
| clarkb | so we should filter out those servers as early as possible to avoid problems in the future as glibc base versions update | 19:06 |
| clarkb | tonyb: ack and if not I'm always happy to do a test run | 19:06 |
| clarkb | would be good to have 908512 in place before booting any new servers for meetpad or otherwise | 19:07 |
| clarkb | though I just realized we may need to have it check for equiavlent amd flags? | 19:07 |
| clarkb | not sure if amd flags for sse are the same as for intel | 19:07 |
| fungi | oh, good point, i tested that patch on rackspace but not other providers | 19:08 |
| clarkb | spot checking gitea lb it is the same flag so thats fine | 19:08 |
| clarkb | Anything else server related? | 19:09 |
| clarkb | #topic AFS Quota Issues | 19:09 |
| clarkb | I don't think there is anything new to report here. The ubuntu ports and centos stream volumes continue to be near their limits. I think frickler was going to work with tonyb on increasing quotas for something as a learning example | 19:10 |
| clarkb | tonyb: did that happen if so/if not any questions about it? | 19:10 |
| tonyb | It didn't happen as I was AWOL last week. | 19:11 |
| tonyb | no questions at this point | 19:11 |
| fungi | i'm just glad you were found and brought back to us safely | 19:11 |
| clarkb | #topic Keycloak upgrade struggles | 19:11 |
| fungi | struggles are over, i think? | 19:12 |
| fungi | at this point, i've created an account for myself in the zuul realm on the new keycloak03 | 19:12 |
| clarkb | at this point we have a new server (keycloak03) running the latest version of keycloak and a proper database backend. fungi has configured it using the doc from zuul (since that was the only use case so far) | 19:12 |
| clarkb | ya so the next steps are for people to recreate their accounts and swap the server into production | 19:12 |
| clarkb | corvus indicated a preference to do the recreation of accounts after the server was in prod | 19:12 |
| clarkb | should we go ahead and update dns for that? | 19:13 |
| corvus | well my own at least :) | 19:13 |
| clarkb | it makes things simpler and I'm a fan of simpler :) | 19:13 |
| fungi | is there anyone who wants to create an account before we switch over? if not, i'll un-wip and approve the changes nowish | 19:13 |
| fungi | i'll also add a blurb to our docs with instructions for account creation in the zuul realm | 19:13 |
| clarkb | ++ to that | 19:14 |
| tonyb | Sounds like a solid plan. | 19:14 |
| fungi | okay, i'll get on that after the meeting | 19:14 |
| clarkb | thanks! | 19:14 |
| clarkb | #topic Etherpad 1.9.7 upgrade | 19:15 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/908328 Update Etherpad to 1.9.7 | 19:15 |
| clarkb | My concerns with plugin installations don't seem to have been a problem | 19:15 |
| clarkb | the upstream dockerfile did not change as part of the update and the held test node was rendering things with our plugins just fine | 19:15 |
| * clarkb finds the test node ip | 19:15 | |
| clarkb | 104.130.4.87 is the held node if anyone else wants to test it | 19:16 |
| clarkb | I'm happy to hold off approving this until after the preptg since we're relying on etherpad for that | 19:16 |
| clarkb | but would be good to get this off the todo list soon | 19:16 |
| clarkb | but also I suspect its fine to upgrade whenever we want. It seem to be working on the held node | 19:17 |
| clarkb | #topic Zuul Circular Dependency Refactor | 19:17 |
| clarkb | if you aren't aware Zuul did a fairly large refactor around how it schedules changes to better handle circular dependencies | 19:18 |
| clarkb | OpenDev's Zuul doesn't do circular deps so the main impact would be side effects ofthe refactor. A few web ui issues were discovered and have been addressed | 19:18 |
| clarkb | please let the zuul team know if you find any unexpected behavior | 19:18 |
| fungi | technically inaccurate | 19:18 |
| corvus | the status page is a little different; some api endpoints provide different data | 19:18 |
| fungi | the zuul tenant on opendev's zuul is configured to support circular deps, so we could dogfood it, right? | 19:19 |
| clarkb | fungi: yes zuul's tenant did add circular dep support for dogfooding/testing | 19:19 |
| clarkb | but it is temporary aiui and not intended to be a long term thing | 19:19 |
| fungi | makes sense | 19:19 |
| corvus | i put a cycle through its paces | 19:19 |
| corvus | i can get a link | 19:19 |
| clarkb | and zuul maintainers were told not to merge changes that way | 19:19 |
| corvus | https://zuul.opendev.org/t/zuul/buildset/a7e74d0cd40048c9aa85be8245ca8bd0 | 19:20 |
| fungi | right. we did exercise it in production basically, though | 19:20 |
| corvus | that's a cycle of 2 changes if you want to see the buildset/build page changes | 19:20 |
| clarkb | Once zuul is happy with the results of the refactor the next step in zuul is the 10.0.0 release whcih will also drop ansible 6 support and add ansible 9 | 19:20 |
| clarkb | I expect that the rate of change in zuul will have us deploying that when we do our weekly upgrades on satruday | 19:21 |
| clarkb | possibly sooner if corvus or others want to restart thigns quicker to verify | 19:21 |
| clarkb | should we send email to service-announce letting people know this is coming soon? | 19:21 |
| clarkb | we did move all our tenants ot ansible 8 last year and I'm not aware of anyone relying on ansible 6 | 19:21 |
| fungi | probably with regard to dropping ansible 6, couldn't hurt to give a heads up but i wouldn't worry too much about precise or lengthy advance notice | 19:22 |
| clarkb | ok I'll send a note about it after lunch today | 19:22 |
| fungi | more like "if you see problems this might be why" | 19:22 |
| corvus | sounds good to me | 19:22 |
| clarkb | And ya if you see anything weird let zuul know | 19:24 |
| corvus | oh fun side note: | 19:24 |
| corvus | apparently the command/shell warn setting is removed and it's just not going to warn anymore | 19:24 |
| corvus | in ansible 9 | 19:24 |
| clarkb | but if you keep setting the flag to ignore the warning on 8 it breaks 9 | 19:24 |
| clarkb | so you have to accept the warning in 8 if you want to work with 9 | 19:24 |
| clarkb | progress | 19:24 |
| corvus | so in all the places where we turn warnings off because we're running "git" or "apt" or whatever, we can remove those | 19:24 |
| corvus | zuul-jobs should already be cleaned up -- but i mention it since we may be able to clean up some stuff in system-config i think? | 19:25 |
| corvus | well https://review.opendev.org/908671 still needs to merge in zuul-jobs | 19:25 |
| clarkb | yes possibly. Though much of system-config runs in nested ansible so may also require upgrading ansible on bridge to get to that point | 19:25 |
| corvus | clarkb: yep; so its a ways out | 19:26 |
| corvus | but just mentioning it as a point of shared interest | 19:26 |
| clarkb | I guess the email should also indicate peopel can opt into ansible 9 after the switch and some point later we'll default to 9 | 19:27 |
| corvus | ++ | 19:27 |
| clarkb | #topic Open Discussion | 19:28 |
| clarkb | I debugged a stuck job in zuul today that ended up being due to leaked nodes in rax dfw preventing any new boots there but the nodepool provider was still grabbing a single request that it paused after checking quota | 19:28 |
| clarkb | I manually deleted a subset of nodes that nodepool wasn't deleting. I have taken a record of those I didn't delete in my homedir on bridge and will try to get to those later today | 19:29 |
| clarkb | Second reminder that we're meeting tomorrow and the day after for our pre ptg | 19:29 |
| clarkb | our agenda etherpad is here: https://etherpad.opendev.org/p/opendev-preptg-202402 and we'll use the corresponding meetpad | 19:30 |
| clarkb | I've tried to organize topics and clump semi related items together | 19:30 |
| clarkb | as well as add background where I can | 19:30 |
| clarkb | Anythine else? | 19:31 |
| fungi | i think i forgot that was happening on top of valentine's day, but i'll make sure i'm around | 19:32 |
| clarkb | fungi: ya it sounded like that wasn't a huge imposition and for some it was preferable to scheduling during later weeks | 19:32 |
| fungi | not a problem | 19:32 |
| fungi | i haven't made any reservations or anything | 19:32 |
| clarkb | my kids have to make cards for everyone in their class this afternoon so today is worse than tomorrow for me :) | 19:33 |
| fungi | hah | 19:33 |
| fungi | good luck! | 19:33 |
| clarkb | Last call for anything else. I'm not surprised the meeting is short when we plan a bunch of discussion for the next coupel of days | 19:34 |
| clarkb | thank you all for your time today and later this week and otherwise! | 19:36 |
| clarkb | #endmeeting | 19:36 |
| opendevmeet | Meeting ended Tue Feb 13 19:36:19 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:36 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2024/infra.2024-02-13-19.00.html | 19:36 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2024/infra.2024-02-13-19.00.txt | 19:36 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2024/infra.2024-02-13-19.00.log.html | 19:36 |
| fungi | thanks clarkb | 19:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!