| ianychoi | Hi team, I cannot join in upcoming infra meeting in 3 hours.. but would be great if your team can give me / seongsoocho some hints how to move forward https://review.opendev.org/c/openstack/project-config/+/961499 so that he can say during OpenInfra Asia Summit 2026 that I18n SIG "migrated" to Weblate like https://kubecon-cloudnativecon-openinfra-pytorch-2026.sessionize.com/session/1225189 . | 15:48 |
|---|---|---|
| fungi | ianychoi: in the future we can add things to the meeting specifically if it's more than 24 hours ahead of the meeting time, but i'll bring it up during open discussion | 15:50 |
| fungi | though that's more of an openstack topic than an opendev topic. clarkb has been travelling for the past month and i've been buried under other tasks so haven't had a chance to look at it. i think mnasiadka was also talking about reviewing some of that | 15:51 |
| clarkb | yes, there is overlap in people but its likely we need to motivate people from the openstack side to get it over the finish line | 15:52 |
| ianychoi | Thank you fungi for the comments and I agree with you to have raised in early manner is encouraged. Note that I pinged 1-2 weeks ago on #openstack-infra channel so I ping here again.. but I suggest seongsoocho to more actively discuss with OpenStack Infra team if there are any issues. Appreciate all the help, honestly. | 15:53 |
| mnasiadka | fungi: thanks for reminding me :) | 16:21 |
| fungi | meeting in ~2 minutes! | 18:57 |
| fungi | #startmeeting infra | 19:00 |
| opendevmeet | Meeting started Tue Aug 4 19:00:00 2026 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:00 |
| opendevmeet | The meeting name has been set to 'infra' | 19:00 |
| fungi | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/2NBGQXHA3KHZALTRU3K5LAEVOEYZC3EF/ Our Agenda | 19:00 |
| fungi | #topic Announcements | 19:00 |
| fungi | i didn't have any announcements recorded, did anyone else? | 19:00 |
| clarkb | I'm back to a normal timezone | 19:01 |
| fungi | welcome back! | 19:01 |
| fungi | seems like nothing else for announcements this week, so moving on.. | 19:02 |
| fungi | #topic Upgrading Old Servers | 19:02 |
| fungi | #link https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents | 19:02 |
| fungi | the change to remove backup02.ca-ymq-1.vexxhost has deployed, but that didn't clean up the old cronjobs on our servers | 19:03 |
| clarkb | ya I looked at the ansible code (role install-borg) and it seems to only add cron rules | 19:03 |
| fungi | so it's still getting a redundant copy of backups for now, and having to be pruned periodically | 19:04 |
| clarkb | so I think it should be safe to either manually clear those out or figure out a way to have ansible drop the cron rules (they are atuomatically generated based on group membership in the borg server group | 19:04 |
| fungi | yeah, i can just manually `crontab -e` on them | 19:04 |
| clarkb | we may need a list that we manually edit in an ansible var that we loop over and ensure absent with | 19:04 |
| fungi | it wouldn't take too long to go through the servers that have been backing up to it | 19:04 |
| clarkb | but for now manually dropping it is probably fine | 19:05 |
| fungi | also we noticed that the new borg version on backup03.ca-ymq-1.vexxhost no longer compacts backups automatically, causing it to fill up fairly quickly | 19:05 |
| fungi | i manually ran a compact cycle across all the backups on it last week just to keep things running for now | 19:05 |
| corvus | interesting enhancement choice | 19:05 |
| fungi | #link https://review.opendev.org/999465 Compact borg repos after pruning | 19:06 |
| fungi | that's wip for the moment, because the old version of borg on our other server doesn't have that subcommand | 19:06 |
| fungi | but i'll adjust it to be version dependent, unless we're close to being able to replace the other older server in rax ord | 19:06 |
| fungi | that's probably all the updates about server upgrades for the moment, though related to the next topic | 19:07 |
| fungi | #topic Deploying a Prometheus for Server Metrics | 19:07 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/999383 deployment automation continues from here | 19:08 |
| clarkb | corvus: they made that change so you can better schedule high iop activity I guess | 19:08 |
| fungi | the server exists (thanks mnasiadka for creating it) | 19:08 |
| corvus | clarkb: ah interesting. makes sense i guess | 19:08 |
| fungi | the next step is adding the deployment job | 19:09 |
| fungi | which that change does, but needs more eyes | 19:09 |
| clarkb | fungi: mnasiadka are there hosts in the production inventory yet? | 19:09 |
| clarkb | mostly wondering if that change is expected to noop or not | 19:09 |
| fungi | yes | 19:09 |
| clarkb | ok so it will not noop | 19:10 |
| fungi | prometheus01 exists and is in inventory and dns | 19:10 |
| fungi | but right now it's not got prometheus installed on it | 19:10 |
| clarkb | ack I will review it with that in mind thanks. Maybe that means I don't approve it this afternoon before doing router upgrades | 19:10 |
| fungi | the playbooks have previously been exercised by test jobs, just not used to deploy on the prod server yet | 19:11 |
| fungi | which is why that change just adds a job to run the existing playbooks | 19:11 |
| fungi | one thing i have not double-checked yet is whether any private hostvars are already set up on bridge | 19:12 |
| fungi | just dawned on me | 19:12 |
| clarkb | ++ that is a good thing to check before that change gets approved | 19:12 |
| fungi | it's an easy step to forget | 19:12 |
| fungi | #topic Dealing with alien zuul config errors in the openstack tenant | 19:13 |
| fungi | clarkb reached out to starlingx about this | 19:13 |
| clarkb | the change does lgtm but I didn't approve it due to ^ and needing to upgrade my router after this meeting | 19:13 |
| clarkb | I asked ildiko to ping them about it and scott little responded a few days back asking how urgent this is | 19:13 |
| fungi | #link https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/ Moving StarlingX into a Dedicated Zuul Tenant | 19:13 |
| clarkb | sounds like they are getting into the busy part of their release process? I was planning to respond that this isn't super urgent but that they should work on correcting their current zuul configuration errors in the interim | 19:14 |
| clarkb | (as those should be more straightforward and are less likely to be negatively impactful to their release) | 19:14 |
| fungi | that makes sense | 19:14 |
| fungi | #topic Gitea 1.27.1 Upgrade | 19:15 |
| clarkb | I've got writing that response on my todo list as of this morning. Should get to it eventually | 19:15 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/998742 Upgrade Gitea to 1.27.1 | 19:15 |
| fungi | we're running that now | 19:16 |
| fungi | when i checked yesterday there wasn't a newer version yet | 19:16 |
| clarkb | yup this is the latest version at the moment | 19:16 |
| clarkb | we can probably drop this item from the agenda. Thank you for getting that landed | 19:16 |
| fungi | no problem | 19:16 |
| fungi | #topic Matrix Meetbot Porting | 19:17 |
| fungi | here's an exciting one! | 19:17 |
| fungi | corvus just tested this again a few minutes ago | 19:17 |
| fungi | when should we switch to meeting in matrix, and what room (our normal room or a separate meeting room)? | 19:17 |
| fungi | are folks fairly confident this is working and feature-complete now? | 19:17 |
| mnasiadka | clarkb: there was no deploy job, I think I raised a patch for one, so once it merges we should get prometheus up and running | 19:18 |
| fungi | mnasiadka: yes, that was the change we discussed which he's planning to review | 19:18 |
| clarkb | fungi: I think I'm happy to move the meeting to matrix as soon as the test meetings look good | 19:18 |
| corvus | yeah, i think the meetbot is ready for us | 19:18 |
| clarkb | as for whether or not we need a new channel I'm indifferent. Seems like the existing channel should be fine. But if we want to avoid the clutter we can create another room too | 19:19 |
| corvus | i lean slightly toward "new room" | 19:19 |
| corvus | just because we do have people dropping in in the main room during meetings sometimes | 19:19 |
| corvus | so it's nice not to have to say "pls wait 1 hour" :) | 19:20 |
| clarkb | in that case a new #opendev-meeting:opendev.org room seems like it would work well | 19:20 |
| fungi | and even when we had a non-retired #opendev irc channel we used this #opendev-meeting channel to hold our meetings | 19:20 |
| fungi | so yes, that sounds fine to me, i don't think our reasons for separating them have changed, just the protocol/venue | 19:21 |
| clarkb | so we need to create the room then configure the bot to live there. Then can announce the change of meeting venue? | 19:21 |
| fungi | yes i think so | 19:21 |
| corvus | ++ | 19:21 |
| fungi | who wants to create the room and push the one-liner change to add it to the config? | 19:22 |
| clarkb | I'm still building and catching up on my backlog so not sure if I would be able to get to that before next meeting. | 19:22 |
| clarkb | but I can be the backup/fallback if others don't beat me to it | 19:22 |
| fungi | i'm hesitant to pretend it won't fall too far down my list of priorities for this week | 19:23 |
| corvus | i can try, no promises | 19:23 |
| fungi | okay, let's all try to find a few minutes for it and maybe one of us will get to it | 19:24 |
| clarkb | sounds good | 19:24 |
| fungi | and if not, then we'll reconvene here next week until we get to it | 19:24 |
| fungi | #topic Gerrit 3.13.8 upgrade | 19:24 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/994938 Upgrade Gerrit to 3.13.8 | 19:24 |
| fungi | we talked about maybe doing it tomorrow, is that still feasible? | 19:25 |
| fungi | i expect to be around most of the day, though do need to run some errands around lunch | 19:25 |
| clarkb | we are running 3.13.6. There have been a couple of bugfix updates since. One nice thing is they include a fix for the replication plugin on a tag so we can stop building the branch version there. But otherwise there don't seem to be super critical updates just good to stay up to date generally | 19:25 |
| clarkb | fungi: yes I expect to be around tomorrow as well | 19:25 |
| clarkb | if tomorrow is still good for you then I think it should work for me | 19:25 |
| fungi | what time do you like? 20:00 utc? | 19:25 |
| clarkb | yes I think that should work. I can eat lunch while the changes land | 19:26 |
| fungi | perfect, i've put it on my reminders as well | 19:26 |
| fungi | it should be a quick restart so i don't think we need more than a sttatus notice when we're starting | 19:27 |
| clarkb | works for me | 19:27 |
| fungi | and maybe a warning shot an hour before | 19:27 |
| clarkb | ++ | 19:27 |
| fungi | #topic Update Anubis to 1.26.2 | 19:27 |
| fungi | this has deployed, and also the wiki server's package was manually upgraded to 1.26.2 | 19:27 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/998741 says the change is still open | 19:28 |
| clarkb | was there a differetn change that landed or is this not deployed yet? | 19:28 |
| fungi | oh, wait, i thought i had approved it | 19:28 |
| fungi | ah, no you're right | 19:28 |
| fungi | #link https://review.opendev.org/c/opendev/system-config/+/998741 Upgrade Anubis to 1.26.2 | 19:29 |
| fungi | i did upgrade it on wiki, but only revised the change to take it from 1.26.0 to 1.26.2 | 19:29 |
| fungi | so we can merge this whenever, i think | 19:29 |
| clarkb | yes I would expect that this is something we can land whenever | 19:29 |
| fungi | and i saw they pushed a v1.27.0-pre2 tag too, if we want to start a wip change to test that in advance | 19:30 |
| fungi | anyway, 1.26.2 seems to be working fine on wiki so far | 19:31 |
| fungi | it's been runnig since last week with no complaints | 19:31 |
| clarkb | I need to upgrade my home network gear after the meeting but am happy if thatgets approved I think it should be safe enough with a straightforward rollback path | 19:31 |
| fungi | yeah, i need to disappear briefly right after the meeting, but can approve it later when i get back or tomorrow morning | 19:32 |
| fungi | #topic Open discussion | 19:32 |
| fungi | ianychoi and seongsoocho[m] were looking for feedback on openstack's translation jobs migration to weblate | 19:33 |
| fungi | #link https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/991432 Add Weblate client Ansible role | 19:33 |
| fungi | that looks like the next step there | 19:33 |
| seongsoocho[m] | Hi fungi. yes. 991432 is the first patch to be reviewed. | 19:34 |
| fungi | i think clarkb and frickler have looked at some of the other changes in the set, and mnasiadka was talking about taking a look over them too | 19:35 |
| clarkb | one thing I've brought up before but can't recall if it was ever clarified is do we know who can/how to reset the credentials for weblate if we expose them? | 19:35 |
| clarkb | I think that is the biggest risk here. And maybe we should go ahead and accept something we think is close and start working on improvements rather than the big bulk update change | 19:35 |
| fungi | like if a bug in the job causes them to be unexpectedly leaked/disclosed in public logs | 19:36 |
| clarkb | correct | 19:36 |
| seongsoocho[m] | Our current approach is to never expose any credentials. They should only be read from the configuration file. | 19:37 |
| clarkb | I'll try to rereview the change. I think if we can clarify that credential reset path then my suggestion would be to proceed with what we've got unless there is something obviously wrong with it and then iterate from there | 19:37 |
| clarkb | seongsoocho[m]: yes, but sometimes ansible doesn't do what you expect etc | 19:38 |
| corvus | does weblate support oidc? if so, using zuul as a federated idp could be an option | 19:38 |
| seongsoocho[m] | Okay. Also, the current patch does not use any credentials. | 19:39 |
| clarkb | ah right this change is only adding the role | 19:39 |
| clarkb | so we have to get to the point where we run the role to risk credentials | 19:39 |
| fungi | right it's not a concern for 991432 but in later parts of the implementation the question will still be relevant | 19:40 |
| clarkb | so ya I think my suggestion would be to proceed here if there isn't anything obviously wrong | 19:40 |
| clarkb | then get to the point we can run something and iterate from there (with a plan for what to do if we expose credentials) | 19:40 |
| clarkb | corvus: I have no idea if they support that. | 19:40 |
| clarkb | google seems to think it may be possible. Might be a good improvement in the iteration process | 19:41 |
| fungi | yes, i think we discussed it in the past | 19:42 |
| fungi | well, oidc support i mean, not zuul's ability to do federated authorization | 19:43 |
| fungi | anyway, i guess that's all for open discussion unless anybody else had anything? | 19:43 |
| clarkb | I didn't have anything | 19:44 |
| seongsoocho[m] | We use the Weblate API, which requires an API key for authentication. Even if Weblate supports OIDC, I think we still need to keep using an API key for this use case. | 19:44 |
| clarkb | seongsoocho[m]: the idea with OIDC is that zuul can act as a broker for the authentication details and provide short term tokens to the jobs | 19:45 |
| fungi | yes, the weblate api would need oidc support for that to be a possibility | 19:45 |
| clarkb | and yes depends on how they have implemented oidc | 19:45 |
| clarkb | and whether or not you can configure weblate to trust zuul | 19:45 |
| fungi | right, i.e. not hard-coded to a list of popular providers | 19:46 |
| seongsoocho[m] | ah okay... | 19:46 |
| fungi | but yes down the road we can evaluate that as a potential improvement, then you wouldn't need to maintain any credentials for the jobs that interact with the weblate api | 19:47 |
| fungi | and if a job leaked an oidc token it probably wouldn't be usable by the time someone saw it | 19:48 |
| fungi | okay, seems like discussion is winding down so i'll wrap the meeting up | 19:49 |
| fungi | find us in #opendev:opendev.org matrix, or on the service-discuss@lists.opendev.org mailing list for anything else opendev between now and the next meeting | 19:50 |
| fungi | thanks everyone! | 19:50 |
| fungi | #endmeeting | 19:50 |
| opendevmeet | Meeting ended Tue Aug 4 19:50:40 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:50 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2026/infra.2026-08-04-19.00.html | 19:50 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2026/infra.2026-08-04-19.00.txt | 19:50 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2026/infra.2026-08-04-19.00.log.html | 19:50 |
| clarkb | thank you for running the meeting again fungi | 19:50 |
| fungi | any time | 19:50 |
| seongsoocho[m] | thank you for your help! | 19:50 |
| fungi | you're welcome! | 19:52 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!