| corvus | matrix comm check | 18:58 |
|---|---|---|
| clarkb | its working | 18:59 |
| clarkb | and we'll get started momentarily | 18:59 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Aug 12 19:00:18 2025 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/B2OLLYGACZDURRJWKZSHE34JVQ5HT5RY/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | I didn't have anything to announce. Did anyone else? | 19:00 |
| fungi | not i | 19:01 |
| clarkb | ok I'll probably keep things rolling today as we have a lot of agenda items. I believe we should be able to get through them all in our hour but want to do my best to ensure we do | 19:02 |
| clarkb | #topic Gerrit 3.11 Upgrade Planning | 19:02 |
| clarkb | I don't have anything new on this item. Its perpetually on my list of todos but requires I be able to sit down for a good chunk of time and not get distracted and that hasn't happened :/ | 19:02 |
| clarkb | Were there any new questions, comments, or concerns on this subject? | 19:02 |
| fungi | i didn't have any | 19:03 |
| clarkb | #topic Upgrading old servers | 19:04 |
| clarkb | This topic does have updates largely due to the efforts of fungi | 19:04 |
| clarkb | fungi: want to fill us in on the refstack and eavesdrop01 changes? | 19:04 |
| * fungi deletes things | 19:04 | |
| fungi | eavesdrop was a pretty straightforward server replacement with a cinder volume move | 19:05 |
| fungi | old server has now been cleaned up along with its dns records | 19:05 |
| fungi | refstack server has been imaged for archival and deleted, dns records for that cleaned up as well as our deployment automation and documentation purged | 19:05 |
| fungi | the refstack service sunsetting was announced to the openstack-discuss ml in case anyone asks | 19:06 |
| fungi | since it was an openstack-specific service | 19:06 |
| fungi | any questions? presumably no | 19:07 |
| clarkb | thank you for taking care of that. These updates are important for two reasons. The first is it concludes the upgrades for the "easy" list of servers that need upgrading. And second this means all of our python containers are running on Ubuntu Noble now which means we can potentially make changes to our base container image locations (whcih is a topic for later) | 19:07 |
| clarkb | The leftovers are AFS servers, kerberos servers, graphite, and backup servers | 19:07 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-server-replacement-sprint | 19:07 |
| clarkb | I updated the lists on ^ | 19:07 |
| clarkb | Of those servers I think afs and kerberos servers may want to be done in place rather than with replacements | 19:08 |
| clarkb | due to dns record management for those services as well as the size of the data its just easier this way I suspect. And we have redundancy that allows us to step through it one at a time | 19:08 |
| fungi | i can probably arrange to drive that if it's the plan, i'm rather well-versed at in-place debuntu upgrades | 19:08 |
| clarkb | then graphite and backup servers would follow our normal new server replacement process | 19:08 |
| clarkb | fungi: that would be great | 19:08 |
| clarkb | I suspect of those in place upgrades the kerberos servers are easiest | 19:09 |
| corvus | i agree that in-place replacement sounds best | 19:09 |
| clarkb | then the afs db servers then the afs fileservers | 19:09 |
| fungi | afs servers probably won't be that hard either | 19:09 |
| fungi | especially since we build our own packages | 19:09 |
| fungi | so the openafs versions aren't really changing much, if at all | 19:10 |
| clarkb | I'm happy to help as well | 19:10 |
| clarkb | we can sync up outside of the meeting and start sketching out some plans (and not even necessarily today just some point soon hopefully) | 19:10 |
| fungi | yep, sgtm | 19:10 |
| clarkb | any other inputs on this subject? | 19:10 |
| clarkb | #topic Matrix for OpenDev comms | 19:11 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing | 19:11 |
| clarkb | I pushed a new patchset yseterday to address the reviews I got. Thank you for the feedback. Looks like ianw has some followup feedback I should probably go ahead and address today as well | 19:12 |
| clarkb | then once that new patchset is up followup reviews would be appreciated | 19:12 |
| clarkb | I think we can keep conversation in the review I just wanted to call out there was an update and there should be another one soon | 19:12 |
| clarkb | #topic Working through our TODO list | 19:13 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-running-todo-list | 19:13 |
| corvus | did you find out whether ems will run molnir? | 19:13 |
| clarkb | #udno | 19:13 |
| clarkb | #undo | 19:13 |
| opendevmeet | Removing item from minutes: #link https://etherpad.opendev.org/p/opendev-running-todo-list | 19:13 |
| clarkb | #undo | 19:13 |
| opendevmeet | Removing item from minutes: #topic Working through our TODO list | 19:13 |
| clarkb | corvus: I couldn't find anything in their docs indicating they do but didn't login to check the toggles in the dashboard | 19:13 |
| clarkb | corvus: but reading up on mjolnir it seems very straightforward to run like one of our other bots so that was what I proposed in the spec | 19:14 |
| corvus | ok. we should probably check the dashboard, but ack, agree we can just run it if we need to | 19:14 |
| clarkb | its basically a container with a filesystem bind mount for data storage and a config file configuring its management room (where anyone in that room gets to control the bot) | 19:14 |
| corvus | ++ | 19:15 |
| corvus | (that's all i had on this topic; thx) | 19:15 |
| clarkb | #topic Working through our TODO list | 19:15 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-running-todo-list | 19:15 |
| clarkb | This is a reminder that if you get bored and want to pick up something new starting with this list is a good start | 19:15 |
| clarkb | I think I'll drop this off of the meeting agenda after this meeting though as the list has a proper location that should be discoverable at this point so I don't think manual reminders in the meeting are as valuable | 19:16 |
| clarkb | feel free to update that list. The idea is that its informal and somewhere we can capture the very beginnings of efforts before they become specs or changes etc | 19:16 |
| clarkb | #topic Pre PTG Planning | 19:17 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document | 19:17 |
| clarkb | Proposed times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700 | 19:17 |
| clarkb | I think fungi and I actually have a conflict on the 7th from 1800-1900 UTC but we should be able to reschedule that conflict. And this way I think we can avoid having multiple meetings that day and just replace our meeting with the pre ptg | 19:18 |
| fungi | right, i can be flexible | 19:18 |
| clarkb | Please leave feedback on the proposed times as well as the topics (and/or add your own topic ideas) | 19:18 |
| clarkb | On my own I was able to come up with a good set of ideas which helps validate that this is a useful thing to do | 19:19 |
| tonyb | I'm still planning to be in MN for the pre-PTG | 19:19 |
| clarkb | tonyb: those time blocks should be CDT friendly. But let me know if they aren't for some reason | 19:19 |
| fungi | clarkb: the meeting you're thinking of isn't on the 7th from what y calendary claims | 19:19 |
| clarkb | fungi: oh good | 19:19 |
| tonyb | They seem good to me :) | 19:20 |
| fungi | that's the off-week for the meeting | 19:20 |
| clarkb | and if we end up not needing all three blocks of time we can cancel and/or stop early | 19:20 |
| clarkb | I just want to have a reasonable amount of time available to us upfront so that we have the opportunity to use it if necessary | 19:21 |
| clarkb | and ya please update the topics list with your own ideas | 19:21 |
| clarkb | #topic Service Coordinator Election Planning | 19:22 |
| clarkb | Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. | 19:22 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/YXRD23ZWJGDPZ3WESBNZNEYO7NBCXFT4/ | 19:22 |
| clarkb | This is your "one week remaining" warning notice | 19:22 |
| clarkb | I'm more than happy to support someone in this role if there is interest. Let me know if there are any questions about what is involved | 19:22 |
| fungi | also similarly happy to help support someone in that role who isn't me ;) | 19:23 |
| clarkb | #topic Loss of upstream Debian bullseye-backports mirror | 19:24 |
| clarkb | the workaround to have reprepro ignore errors landed so that we'll continue to update debian packages that do exist | 19:24 |
| clarkb | but now we need to make a plan for the longer term. I think what I'd like to see if have opendev base jobs stop configuring backports by default on debian to match the upstream behavior. Then we can delete the bullseye backports content and the xenial and bionic-arm64 conent | 19:25 |
| clarkb | I think the process for all three of those is basically identical so bundling them all up and oing it at once helps avoid unncessary context switching | 19:25 |
| fungi | the only question i have is whether it makes sense to announce and adjust in zuul/zuul-jobs instead | 19:25 |
| fungi | because if opendev has this problem, other users of that role also may | 19:26 |
| clarkb | that is a good point. I think if we do this for bullseye alone we can avoid big announcements as upstream has forced our hand. But if we do it for all of debian then we should announce | 19:26 |
| fungi | but as corvus pointed out last week, that role is also on the road to deprecation if someone gets time to replace it with the new design | 19:26 |
| clarkb | (and we should announce whether we do it for opendev or zuul or both) | 19:26 |
| clarkb | my hunch here is that relatively little stuff is relying on backports since it is a non default opt in choice | 19:27 |
| fungi | yeah, i have no problem announcing whatever we do wherever we do it, just wondering if diverging from the zuul default makes sense in this case | 19:27 |
| clarkb | so blast radius should be small and we should rip the bandaid off | 19:27 |
| fungi | or if freezing the zuul stdlib implementation in order to reduce churn for anyone working on a replacement is better | 19:28 |
| clarkb | ya probably comes down to corvus' preference for the zuul community | 19:28 |
| clarkb | if corvus thinks we should fix this at the root in zuul-jobs we'll do that otherwise we'll modify things in opendev only | 19:28 |
| corvus | i don't think anyone's working on a replacement... and i don't have strong feelings about it... | 19:29 |
| corvus | i'm not 100% caught up on the issue | 19:29 |
| fungi | there's also not a ton of urgency from our perspective, the workaround is functional but we'd like to be able to free up the space | 19:29 |
| corvus | so i'd say: if it makes sense for everyone to change, then go ahead and do it in zuul jobs... otherwise meh | 19:29 |
| clarkb | I think the main reason it applies broadly is it helps mimic upstream debian behavior by default better | 19:30 |
| clarkb | which leads to fewer surprises as people are testing software | 19:30 |
| corvus | sounds reasonable :) | 19:30 |
| fungi | corvus: essentially upstream debian deleted the backports suite for bullseye a few weeks ago. the configure-mirrors role defaults to enabling the backports suite on debian nodes, so if we delete it from our mirrors then any bullseye jobs are going to fail unless they're overridden to disable backports in that role | 19:30 |
| clarkb | sounds like we should make the change in zuul-jobs and announce it then go from there? | 19:31 |
| fungi | and yeah, disabling backports by default instead of enabling would be an easy switch in configure-mirrors, and would be closer to normal debian installs these days, but is a clear behavior change | 19:31 |
| fungi | clarkb: probably the reverse. announce and *then* switch | 19:32 |
| clarkb | er yes | 19:32 |
| clarkb | we want to announce it before we break things | 19:32 |
| fungi | agreed | 19:32 |
| fungi | i ca ntake care of that | 19:32 |
| clarkb | thanks! | 19:32 |
| clarkb | #topic Adding Ansible 11 to Zuul | 19:32 |
| clarkb | Next up in potentially breaking changes for jobs run by zuul :) | 19:33 |
| corvus | ha, but hopefully not! | 19:33 |
| clarkb | we switched the zuul and opendev tenants to ansible 11 last week and as far as I can tell things seem to be working | 19:33 |
| clarkb | zuul in particular has been running a good number of jobs | 19:33 |
| corvus | i haven't heard of any issues. should we switch the rest of the tenants? if so when? | 19:33 |
| clarkb | I think we should switch the other tenants. I think we can send an announcement ~today for a switch early next week? That way we can communicate to people the change is happening and they can override the version back to 9 if things break | 19:34 |
| clarkb | its also probably just early enough in the openstack release cycle that we can work through problems | 19:34 |
| fungi | yeah, much longer and we're getting close to the rush/freeze | 19:34 |
| clarkb | if we didn't have the fallback to 9 in job overrides I would worry more. but we do have that fallback so I think we should proceed | 19:35 |
| corvus | sounds good | 19:35 |
| clarkb | corvus: do you want to send that announcement or should I? | 19:35 |
| corvus | clarkb: if you don't mind, that'd be great | 19:36 |
| corvus | https://paste.opendev.org/show/bu8VuxDAKYEzai5Caik7/ | 19:37 |
| clarkb | ack /me scribbles a note | 19:37 |
| corvus | i did just find an old copy of a message from you on the subject ^ | 19:37 |
| clarkb | I should be able to do that after lunhc today | 19:37 |
| corvus | #link old message https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/message/A7SOKNMQLZXZSTXUI4UOIIGOQHIQSMZ6/ | 19:37 |
| corvus | (yay archived-at header) | 19:37 |
| clarkb | #topic Etherpad 2.4.2 Upgrade | 19:37 |
| clarkb | #link https://github.com/ether/etherpad-lite/blob/v2.4.2/CHANGELOG.md | 19:38 |
| clarkb | There are some etherpad updates. Nothing critical according to the release notes but I've been doing my best to keep us up to date on services like this | 19:38 |
| clarkb | Problem is the new versions break every skin except for colibris | 19:38 |
| clarkb | #link https://github.com/ether/etherpad-lite/issues/7065 Theming updates have broken the no-skin and custom skin themes. We use no-skin. | 19:38 |
| clarkb | someone else followed up on the issue I filed to point out it breaks custom skins too | 19:39 |
| clarkb | Unfortunately, no feedback from the maintainers yet | 19:39 |
| clarkb | I think our options are either wait more and see if they fix it. Or consider using colibris. As mentioned this doesn't appear to be urgent but if you have 10 minutes it might be a good idea to check out colibris on the held node | 19:39 |
| clarkb | 158.69.64.117 is the ip address to update in /etc/hosts for etherpad.opendev.org to do that | 19:40 |
| clarkb | the biggest difference I find is that colibris mimics the sheet of paper appraoch used by google docs | 19:40 |
| clarkb | this narrows the useable space of the pads. Not necessarily always a good or bad thing. Depends on context | 19:41 |
| fungi | i personally find it unnecessarily wastes screen space | 19:41 |
| clarkb | if you do check it out and have thoughts on whether or not it would be problematic for us let me know. If we think it is a valid switch then we can proceed that way | 19:41 |
| clarkb | fungi: ya I feel like it may also imply that these are less ether pads and more perma pads | 19:42 |
| corvus | yeah that's kind of silly, but, probably not a showstopper if we have to switch | 19:42 |
| fungi | which isn't a show-stopper but feels like a regression | 19:42 |
| fungi | yes, agreed | 19:42 |
| clarkb | probably the thing to do is wait for now and see if they respond to my issue. Then if there are any important reasons to update sooner (security updates major bug fixes) we can switch to colibris and do our best | 19:43 |
| corvus | who needs more than 87 columns anyway? /s | 19:44 |
| clarkb | #topic Cleaning Up the Old voip.ms Account | 19:44 |
| clarkb | yesterday fungi and I were made aware that the voip.ms account is making small infrequent charges | 19:45 |
| clarkb | since we dropped asterisk and switched to meetpad we haven't been using this service | 19:45 |
| clarkb | before we formally shutdown the account/cancel the service I wanted to double check there wasn't a reason to avoid doing so. I think if we decide to tie POTS into meetpad via SIP we can always resurrect the account or create a new one at that point | 19:46 |
| clarkb | oh hey etehrpad literally just responded wtih a question. I'll followup after the meeting | 19:46 |
| corvus | i imagine the chargers are for maintaining the DID (phone number); part of shutting it down will be releasing that | 19:46 |
| clarkb | corvus: yes that was what I thought. fungi thought it could also be spam calls | 19:47 |
| corvus | so as long as we don't feel super attached to that number, then that should be fine. | 19:47 |
| clarkb | I think giving up the number is also fine. Using a new number shouldn't be a big impact | 19:47 |
| corvus | not sure what the plan was, but i wouldn't expect non-terminated incoming calls to be charged | 19:47 |
| clarkb | ack | 19:48 |
| fungi | yeah, so probably not minutes getting eroded by random call-ups then | 19:48 |
| corvus | last i looked it was something like $2/mo to maintain a did | 19:48 |
| corvus | (and last i looked was 4 years ago) | 19:48 |
| clarkb | to summarize we would give up the existing phone number and do whatever the equivalent of shutting down the voip.ms account is (not sure what options they give us there) | 19:49 |
| clarkb | and I'm not hearing any objections to that plan. Wont' get to that until at least tomorrow so if there are objections feel free to raise them after the meeting too | 19:49 |
| corvus | sgctm | 19:49 |
| corvus | sgtm | 19:50 |
| fungi | wfm too | 19:50 |
| clarkb | #topic Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay | 19:50 |
| clarkb | As mentioned earlier in the meeting all of our python based container images (that we build ourselves) are running on Ubuntu Noble now | 19:50 |
| clarkb | that means we should be good to move the python base images to quay.io then update all of the consumers of the images to pull them from there | 19:51 |
| clarkb | and we shouldn't lose speculative image testing. However, there are a couple of caveats/concerns with that | 19:51 |
| clarkb | The first is that zuul and vexxhost also use these images so not sure how much communication we feel is necessary to ensure they don't accidentally use orphaned images. Then these images are primarily used in the image build process not the container deployment and testing process. That means that the existing changes to noble to use podman for deployments may not be | 19:52 |
| clarkb | sufficient to ensure specualtive testing continues to work | 19:52 |
| clarkb | Docker build via buildx/buildkit does support alternative mirrors and is enabled by default in modern docker. But we may need extra configuration to ensure the mirrors are set? | 19:52 |
| clarkb | alternatively we could switch to building with podman/buildah and I think that should just work now after we update all the jobs | 19:53 |
| clarkb | so I guess I'm asking if we think we need a deprecation announcement period before switching and if we should continue building with docker or switch to buildah | 19:53 |
| corvus | i think we should look into continuing to build with docker buildx/buildkit -- so looking into what we need to do to make the mirrors used there, if anything. | 19:54 |
| fungi | i suppose uploading to both registries for a transitional period is a no-go (there was almost certainly a reason we didn't do it before now if not) | 19:54 |
| clarkb | fungi: it is possible. I just want to avoid doing that long term if possible since that would continue to make us vulnerable to docker rate limits | 19:55 |
| clarkb | corvus: ok I can dig into that a bit more to understand that better | 19:55 |
| corvus | (i'm less confident in podman build and buildah; but not opposed if it works) | 19:55 |
| corvus | i think a note to service-discuss should be fine. consider the zuul community notified already. :) | 19:56 |
| clarkb | ack thanks | 19:56 |
| clarkb | so step 0 is become better informed on the docker build behavior and then make an announcement and plan the swithc | 19:56 |
| clarkb | I'll work on that | 19:56 |
| clarkb | #topic Open Discussion | 19:56 |
| clarkb | We have a few minutes left. Anything we missed that is worth covering? | 19:56 |
| corvus | and test between steps 0 and 1 to make sure the mirror stuff is right | 19:57 |
| clarkb | ++ | 19:57 |
| corvus | it's not terrible if we have to roll back, but would be nice if we could avoid that and break our streak :) | 19:57 |
| corvus | 1 quick thing | 19:58 |
| corvus | there's an upcoming syntax change for zuul-launcher; i have the change to zuul-providers prepped: https://review.opendev.org/956946 | 19:58 |
| corvus | that's reviewed, and so is the zuul change; when the zuul change merges, i'll handle restarting zuul with it, then merging the zuul-providers change | 19:59 |
| corvus | just wanted to let folks know that's upcoming | 19:59 |
| clarkb | will test node boots break during the time between restarting launcher and updating zuul-providers? | 20:00 |
| clarkb | and if so does that imply we'll need to force merge that update? | 20:00 |
| corvus | (during this unstable development period for niz, we might have the occasional backwards-incompat change like that -- basically, it's two commits in zuul so it's just long enough for opendev to make the change without breaking). | 20:00 |
| corvus | clarkb: nope ^ 3 changes total: add new syntax, update zuul-providers, remove old. so i'll sequence everything so it's never broken for opendev | 20:01 |
| clarkb | got it | 20:01 |
| clarkb | we're over tiem. Thank you everyone! We'll be back next week at the same time and location | 20:01 |
| corvus | [eot] | 20:01 |
| corvus | thanks! | 20:01 |
| clarkb | #endmeeting | 20:02 |
| opendevmeet | Meeting ended Tue Aug 12 20:02:00 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-12-19.00.html | 20:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-12-19.00.txt | 20:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-12-19.00.log.html | 20:02 |
| fungi | thanks clarkb! | 20:02 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!