| clarkb | ~5 minutes to our meeting | 18:55 |
|---|---|---|
| corvus | read you loud and clear | 18:56 |
| clarkb | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Aug 5 19:00:06 2025 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/GPTGNBBSJONHY3AZX64L5HECQ2YZYXB7/ Our Agenda | 19:00 |
| clarkb | #topic Announcements | 19:00 |
| clarkb | I didn't have anything to announce. Did anyone else? | 19:00 |
| clarkb | I guess I can note the service coordinator nomination period is open now | 19:00 |
| clarkb | we'll discuss that in more depth later in the meeting | 19:00 |
| fungi | i didn't, no | 19:02 |
| clarkb | #topic Gerrit 3.11 Upgrade Planning | 19:03 |
| clarkb | Nothing really new here. I had hoped to start testing upgrade things with our latest image builds last week but then got distracted by other things | 19:03 |
| clarkb | it is still relatively high on my todo list though so I'm hoping to have updates soon | 19:03 |
| clarkb | #link https://www.gerritcodereview.com/3.11.html Gerrit 3.11 Changelog | 19:03 |
| clarkb | #link https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3 Held Nodes A | 19:03 |
| clarkb | #link https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818 Held Nodes B | 19:04 |
| clarkb | and these are useful links if anyone else wants to look into it a bit more too | 19:04 |
| clarkb | any questions/concerns/comments around Gerrit 3.11 upgrades before we move on? | 19:04 |
| clarkb | #topic Upgrading old servers | 19:06 |
| clarkb | There are updates here | 19:06 |
| clarkb | fungi replaced eavesdrop01 with eavesdrop02 updating it to Noble | 19:06 |
| clarkb | there were some minor hiccups during the replacement but overall things went smoothly | 19:06 |
| clarkb | fungi: is the old server still running? And if so when do you think we should consider cleaning it up? | 19:07 |
| fungi | still up at the moment | 19:07 |
| fungi | but yes needs cleaning up along with dns | 19:07 |
| fungi | i was going to do it this week | 19:08 |
| clarkb | sounds good. Any thing else to note on the eavesdrop server replacement? | 19:08 |
| fungi | other than me forgetting to add the acme cname initially, it went fine | 19:08 |
| clarkb | yup the actual data recording outage ended up being pretty minimal as planned | 19:09 |
| fungi | the cinder volume name still references eavesdrop01 of course | 19:09 |
| clarkb | the ssl cert problem only affected reading the logs back out again | 19:09 |
| clarkb | I wonder if you can rename a volume | 19:09 |
| fungi | we can do a seamless replacement later if we want, using pvmove | 19:09 |
| clarkb | good point | 19:10 |
| fungi | but it's mainly cosmetic | 19:10 |
| clarkb | fungi also announced the plan to shutdown refstack and retire its git repos | 19:10 |
| clarkb | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/WNI4PE2TZ3G52C3U5FT2YNVRUAJB3CMO/ | 19:10 |
| corvus | google suggests "openstack volume set --name" can rename but that should be verified | 19:10 |
| clarkb | I don't think any of those changes have actually happend yet but announcing that they will happen is a good start | 19:10 |
| fungi | yeah, side note, turns out refstack crashed weeks ago and nobody noticed | 19:11 |
| corvus | (maybe worth a try before doing pvmove) | 19:11 |
| fungi | so it's already down | 19:11 |
| clarkb | ha confirming that shutting it down is the right course of action | 19:11 |
| clarkb | fungi: is there anything you need to continue to make progress on the refstack cleanup? | 19:12 |
| fungi | nah | 19:12 |
| clarkb | thank you for pushing these along. It helps a lot when we're all chipping away at the big list of server upgrades | 19:12 |
| fungi | i just wanted to give the announcement thread a few days before i stop/snapshot/delete the server | 19:12 |
| clarkb | anything else on the larger server upgrades topic? | 19:13 |
| clarkb | #topic Matrix for OpenDev comms | 19:14 |
| clarkb | #link https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing | 19:14 |
| clarkb | I've gotten a couple more reviews now. I'll plan to make some updates and push a new patchset this week in response to those | 19:14 |
| clarkb | thank you for that. I haven't really seen any feedback that diverges from what we've already talked about. But some good ideas on tightening up the plan | 19:15 |
| clarkb | we can keep the discussion largely in Gerrit so that we don't have diverging threads. I just want to keep calling attention to the spec here while it is getting iterated on | 19:15 |
| clarkb | #topic Working through our TODO list | 19:16 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-running-todo-list | 19:16 |
| clarkb | we have a new dedicated etherpad that is also linked to from the front page of our specs site | 19:16 |
| clarkb | https://docs.opendev.org/opendev/infra-specs/latest/#todo-list | 19:16 |
| corvus | oh matrix | 19:16 |
| corvus | ianw suggested looking into mjonlr | 19:17 |
| clarkb | Idea is to capture things on our radar that we either haven't been able to get to yet or are slowly making progress on and don't want to forget | 19:17 |
| clarkb | corvus: ya I was going to look at the options there then update the spec | 19:17 |
| clarkb | I thought we were already subscribed to some sort of global moderation system but maybe this is an extra layer on top of that? | 19:17 |
| corvus | ok... cool... yeah, that might change the work item stream. | 19:17 |
| corvus | i think maybe they just provide access to it and we have to start using it | 19:17 |
| clarkb | got it. I'll dig in and see what I find | 19:18 |
| corvus | sounds good | 19:18 |
| corvus | thanks! | 19:18 |
| clarkb | on the todo list front feel free to update that document with other things you're aware of that aren't already captured | 19:18 |
| clarkb | or mark things done if I've missed things getting completed | 19:18 |
| clarkb | the idea is to have an easy system to acpture stuff at a very high level so that we don't completely forget but also give people a thread to pull on if/when they are ready to start the next thing (which may lead to writing a spec etc) | 19:19 |
| clarkb | and related to that is pre PTG planning | 19:19 |
| clarkb | #topic Pre PTG Planning | 19:19 |
| clarkb | #link https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document | 19:19 |
| clarkb | I've added some real content ideas there and everyone should feel free to add more | 19:19 |
| clarkb | One thing I wanted feedback on as we get closer is how much time do we think we can dedicate to this in october and what time of day are we thinking? | 19:20 |
| clarkb | I was thinking may be 2 hours a day tuesday through thursday and if we run out of discussion topics we can cancel the later blocks of time | 19:20 |
| clarkb | tonyb: will you be in an au timezone or somewhere else? Wondering if we should try to have an early and late session or maybe something around 1900 UTC which may "work" (for some value of work) for everyone? | 19:21 |
| clarkb | we are about 2 months away now so plenty of time to figure that out, but I don't want to forget that we need a more concrete set of times so am starting that discussion now | 19:22 |
| fungi | should be doable for me | 19:22 |
| clarkb | if I don't hear and specific feedback I'll probably just pick some times in a couple of weeks then see if anyone complains :) | 19:23 |
| clarkb | so ya get back to me on what times you expect to work for you and I'll see what I can do schedule wise | 19:23 |
| clarkb | #topic Service Coordinator Election Planning | 19:23 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/YXRD23ZWJGDPZ3WESBNZNEYO7NBCXFT4/ | 19:23 |
| clarkb | Nomination Period open from August 5, 2025 to August 19, 2025. If necessary we will hold an election from August 20, 2025 to August 27, 2025. All date ranges and times will be in UTC. Is the summary | 19:23 |
| clarkb | this has all been made official on the mailing list now | 19:24 |
| clarkb | I am more than happy for someone else to step into the role and can act as backup and contineu to support whoever it may be | 19:25 |
| clarkb | let me know if you have any questions about what is involved | 19:25 |
| clarkb | #topic Loss of upstream Debian bullseye-backports mirror | 19:26 |
| clarkb | #link https://review.opendev.org/956497 temporarily ignore undefinedtarget | 19:26 |
| clarkb | fungi: want to catch us up on this one? | 19:26 |
| fungi | yeah... | 19:27 |
| fungi | so, the backports suite in debian for bullseye was eol for a while, finally removed from the upstream mirrors a couple weeks ago | 19:27 |
| fungi | this broke our reprepro mirroring of course, so there was a change to remove it from the config | 19:28 |
| fungi | unfortunately, because it still persists in the reprepro database, it's treated as an error causing the script to abort | 19:28 |
| fungi | in the short term we can work around it by telling reprepro to ignore that error and get the mirror back up to date | 19:29 |
| clarkb | fungi: so we've already removed it from the config? | 19:29 |
| fungi | yes, in git | 19:29 |
| fungi | the script change you linked above refers to the earlier config change | 19:29 |
| clarkb | so I guess we need to run the steps we have documented for cleaning up targets like that? Similar to what we would do after the changes to cleanup xenial land? | 19:29 |
| clarkb | in the case of xenial we probably won't get an error because ubuntu isn't removing that content from the mirrors I guess. But I suspect the cleanup fixup steps are similar/the same? | 19:30 |
| fungi | we could instead just purge the bullseye-backports suite, but that's going to cause jobs which are using bullseye with the configure-mirror role to start breaking unless they override its default | 19:30 |
| clarkb | got it. Do we know if any actually depend on that? | 19:31 |
| clarkb | just thinking that bullseye is old enough and backports are uncommon enough that maybe it is a non issue? | 19:31 |
| fungi | i highly doubt they're installing packages from it | 19:31 |
| clarkb | and if it is an issue maybe that is an indication the jobs should be updated | 19:31 |
| fungi | or the default | 19:32 |
| fungi | #link https://opendev.org/zuul/zuul-jobs/src/commit/06098ab/roles/configure-mirrors/defaults/main.yaml#L14 | 19:32 |
| clarkb | I'm ok with that workaround change and I've +2'd it. But I suspect we can probably just proceed with clearing out backports as the longer term solution | 19:32 |
| fungi | as we've discussed before, that role enables the backports repo in all jobs that use it, unless they explicitly set `configure_mirrors_extra_repos: False` as an override | 19:33 |
| clarkb | fungi: I think we can set a site vars override for that | 19:33 |
| clarkb | then you don't have to do it on a per job basis | 19:33 |
| fungi | one possibility that could be less intrusive is to make the default version-dependent | 19:33 |
| clarkb | or we could set it in the base jobs that configure mirrors I guess which is only a small number of locations | 19:33 |
| clarkb | fungi: well I'm thinking that we should set it to false because debian doesn't keep the packages up for long enough | 19:34 |
| clarkb | rather than make it version dependent just force things to opt in and then deal with the fallout when the packlages go away in those locations | 19:34 |
| fungi | yeah, i just wonder if (and this is more a topic for zuul discussion venues) that role's default should be rethought | 19:34 |
| corvus | there's a completely different way of configuring mirrors already specced out | 19:35 |
| corvus | but no one is working on it | 19:35 |
| corvus | https://zuul-ci.org/docs/zuul-jobs/latest/mirror.html | 19:35 |
| clarkb | ya that appraoch is far more specific and less convention based. Definitely a good thing to pick up if someone has time | 19:36 |
| clarkb | so for next steps I think we can land the workaround to start syncing packages again, then concurrently flip the var value in opendev one way or another, then finally cleanup the old packages db and revert the workaround | 19:37 |
| clarkb | if that seems reaosnable i think fungi can go ahead and +A the workaround | 19:37 |
| fungi | yeah, can do | 19:38 |
| fungi | this had enough moving parts i didn't want to go making too many changes before we had a chance to talk through the options | 19:39 |
| clarkb | #topic Adding Ansible 11 to Zuul | 19:39 |
| clarkb | Zuul is growing support for Ansible 11 (this may already be deployed in opendev as of our friday restarts?) and losing support for ansible 8 | 19:40 |
| corvus | yep is deployed | 19:40 |
| corvus | #link ansible 11 test https://review.opendev.org/956490 | 19:40 |
| corvus | that initial sanity check came back green | 19:40 |
| corvus | so do we want to go ahead and set the tenant defaults for opendev and zuul to 11? | 19:41 |
| clarkb | I'm open to try that and see what if anything breaks | 19:41 |
| clarkb | have we run into any problems at all yet? | 19:41 |
| clarkb | (just wondering if there are known gotchas) | 19:41 |
| corvus | nothing yet; i don't expect the changes to impact us (or probably almost anyone) | 19:42 |
| corvus | but nobody expects, etc... | 19:42 |
| clarkb | then ya I think the next step is to toggle over a tenant or two and exercise it | 19:42 |
| corvus | cool, i'll propose a change | 19:43 |
| clarkb | #topic Gitea 1.24.4 Upgrade | 19:43 |
| clarkb | #link https://github.com/go-gitea/gitea/blob/v1.24.4/CHANGELOG.md | 19:43 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/956591 | 19:43 |
| clarkb | this appears to be a fairly straight update for us. I suggested earlier today that we land fungi's opendev.org main page update first then land this to upgrade the service itself | 19:44 |
| fungi | yeah, i'll take a look after the meeting | 19:44 |
| clarkb | the main page update has enough reviews now but my upgrade change could use some reviews | 19:44 |
| clarkb | thanks! | 19:44 |
| clarkb | so ya not a major item but wanted to bring it up as I think we can get it done and rolled out fairly quickly | 19:45 |
| clarkb | #topic Etherpad 2.4.2 Upgrade | 19:45 |
| clarkb | This change however is probably a bit more involved | 19:45 |
| clarkb | #link https://github.com/ether/etherpad-lite/blob/v2.4.2/CHANGELOG.md | 19:45 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/956593 | 19:45 |
| fungi | i love the bug fix "fix various bugs" | 19:45 |
| clarkb | in particular they've added dark mode support that I guess toggles based on your browser settings | 19:45 |
| fungi | (that was about the gitea upgrade) | 19:46 |
| clarkb | and they added a home button to the toolbar list which I'm not entirely sure what it does | 19:46 |
| clarkb | so anyway I've held a node looks like it is 158.69.64.117 | 19:46 |
| clarkb | we'll want to edit local /etc/hosts to point etherpad.opendev.org to that IP and test out these new UI updates | 19:46 |
| clarkb | reviews welcome as well as any feedback on the ui updates | 19:47 |
| clarkb | #topic Open Discussion | 19:48 |
| clarkb | I did want to call out I've got a couple changes up to clear out xenial mirror content and bionic arm64 content | 19:48 |
| clarkb | but I'm thinking now the priority is probably in fixing the bullseye situation so that we've got a stable and happy reprepro before we start trying to modify the ubuntu content? | 19:48 |
| clarkb | Anything else? | 19:50 |
| clarkb | sounds like no. Thank you for your time and help everyone! Let me know about timing for pre ptg schedule blocks and consider running for service coordinator | 19:52 |
| clarkb | we'll be back here next week at the asme time and location | 19:52 |
| clarkb | #endmeeting | 19:52 |
| opendevmeet | Meeting ended Tue Aug 5 19:52:23 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:52 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-05-19.00.html | 19:52 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-05-19.00.txt | 19:52 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2025/infra.2025-08-05-19.00.log.html | 19:52 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!