19:01:08 <clarkb> #startmeeting infra
19:01:08 <opendevmeet> Meeting started Tue Jan 17 19:01:08 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:08 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:08 <opendevmeet> The meeting name has been set to 'infra'
19:01:15 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/MB3XWSVFXEROBXFXU5V3DX3D3QVMYLQR/ Our Agenda
19:01:58 <clarkb> #topic Announcements
19:02:14 <clarkb> I didn't have anything to announce. Did anyone else?
19:02:46 <fungi> nothing particularly exciting, no
19:03:16 <clarkb> #topic Topics
19:03:22 <clarkb> #topic Bastion Host Updates
19:03:47 <clarkb> I have a small update for ansible installation https://review.opendev.org/c/opendev/system-config/+/843330
19:04:23 <clarkb> Old bridge is still up. But I expect we'll shut it down real soon now
19:05:26 <clarkb> Other than that I think new bridge has been happy. Was there anything else to mention on this topic? Probably drop it from the weekly agenda once old bridge is shutdown
19:05:59 <ianw> (sorry bouncer issues)
19:06:22 <clarkb> ianw: anything to add before we move onto the next item?
19:06:41 <ianw> not really, there's still some things with some backup changes, and possibly some work on the parallel running
19:06:56 <ianw> i haven't looked at either lately though
19:06:58 <clarkb> oh right. I've been bad and mean to do reviews of those changes but keep getting distracted
19:07:06 * clarkb makes a note on the todo list and maybe that will help
19:07:53 <fungi> yeah, i had some down time over the holiday weekend and tried to dig into older system-config changes which have been growing stale
19:08:27 <clarkb> #topic Mailman 3
19:08:28 <fungi> it would be good to get the parallel stuff finished
19:08:33 <clarkb> ++
19:08:57 <fungi> no progress on the mm3 front since last week's meeting
19:08:59 <clarkb> fungi: lsat week I said feel free to reach out if we can help with the mailman 3 stuff. You've got changes but they are WIP'd. Anything new?
19:09:03 <clarkb> ah that answers that.
19:09:11 <clarkb> Anything we can do to help move it along?
19:09:27 <fungi> but year end/beginning paperwork is mostly wrapped up now, so i should have more time to get back to it
19:10:08 <clarkb> sound good.
19:10:26 <ianw> not mailman3, but on mailman2 ...
19:10:37 <ianw> it's currently in emergency with some custom apache rules
19:10:57 <fungi> oh, right, that's one thing that sucked up a bit of our collective time last week
19:11:13 <ianw> #link https://review.opendev.org/c/opendev/system-config/+/869779
19:11:28 <clarkb> I'll review that change after the meeting
19:11:34 <fungi> thanks!
19:11:40 <clarkb> but I guess we can remove the host from emergency once that lands?
19:11:47 <fungi> should be able to, yes
19:11:48 <ianw> would allow us to codify those rules
19:12:01 <ianw> we'd have to make sure we define the current rules in the secrets for the host
19:12:04 <clarkb> excellent, ya I'll get on that as soon as the meeting is done
19:12:29 <ianw> although, i'm not 100% sure if that issue is ongoing
19:13:25 <ianw> anyway, can double check and we can discuss outside meeting
19:14:22 <clarkb> #topic Quo vadis Storyboard
19:14:40 <clarkb> This is something i Just haven't had time to get to with all of the other paperwork and distractions. I'm hoping to be able to get to it at some point though
19:15:22 <clarkb> #topic Gerrit 3.6
19:15:29 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/870118 Fix Gerrit 3.6 image build
19:15:58 <clarkb> Gerrit updated the bazel version they use which is stricter about checking some flogger logging code. This has resulted in one of the plugins we build erroring on the bazel build
19:16:19 <clarkb> The change above has us consume the latest version of that plugin which fixes the error. Looking at the difference between 3.6.3 and latest of that plugin I expect this to be safe
19:16:42 <clarkb> Then separately I've got a stack of changes to drop gerrit 3.5 image builds and add 3.7 image builds and test all that
19:16:49 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/869763 beginning of stack to remove 3.5 images and add 3.7 and associated testing.
19:17:19 <ianw> ++ will review
19:17:50 <clarkb> probably the most interesting bit of all that is the modifications to the upgrade testing to make the upgrade functional
19:18:26 <clarkb> converts Verified to a submit requirement dropping the function specification (the docs actually say we should set function to noop but they also say function is deprecated so I decided to try just not setting it and that appeas to work)
19:18:48 <clarkb> but ya its straightforward to start, the real fun will be as we test 3.7 more and figure out what all in our acls needs updating
19:19:02 <ianw> ++
19:19:04 <ianw> ... on that
19:19:32 <ianw> as we found out late last week, we may have committed some bad ACL's and not noticed
19:19:35 <ianw> full details are at
19:19:37 <ianw> #link https://etherpad.opendev.org/p/760YNeM5OEFS1hlr7bE5
19:20:11 <ianw> i think maybe if fungi could review that, i can take on doing an audit of where we're at, following the checklist in there, at some quiet time
19:21:10 <clarkb> ianw: my only other piece of feedback is that afte rremoving the cache file maybe we want to manually run the tool against a specific project or three just to be sure we don't have unexpected behavior with the cache file out of the way before we do it against everything
19:21:12 <clarkb> otherwise lgtm
19:21:59 <fungi> i'll take a look, but may not have time until tomorrow
19:22:52 <ianw> good idea, actually it's probably better to do something like loop the project list and keep each in a separate log file.  i can make an update to the procedure
19:23:31 <ianw> i think manage-projects works by getting the list of all projects, but then filtering it by any command-line parameters, iirc from code inspection last week
19:23:40 <clarkb> yes that sounds right
19:25:14 <clarkb> Sounds like that is all for Gerrit
19:25:18 <clarkb> #topic Linaro Cloud Move
19:25:39 <clarkb> The new nb04 arm64 builder is up and running in osuosl and I think ianw mentioned the new linaro cloud is running jobs?
19:25:55 <clarkb> Ed emailed earlier today asking for an update. YOu can update us then copy paste into an email :)
19:26:17 <ianw> i'm not sure on that -- a node did get allocated, but the wrong flavor :)  so there is some communication happening
19:26:50 <ianw> but yeah, nb04 has been my main focus and that seems to be working.  so i think we can get rid of nb03 to have one less thing in linaro
19:27:05 <ianw> but now i need to debug the node launches in the new cloud
19:27:26 <clarkb> ianw: one easy thing might be to rename the flavor if we can
19:27:48 <clarkb> the other idea I had was maybe we can set max-servers in the old linaro cloud to zero at this point?
19:28:09 <ianw> oh that should be fixed -- by dropping the min-ram in the config, it makes it an exact match on the flavor name
19:28:17 <clarkb> ah
19:28:31 <ianw> but something is up with launching the nodes i'm not sure about yet, it seems, there's nothing running
19:28:35 <clarkb> I see
19:29:10 <clarkb> probably a good idea to give ed that update. Basically we're close and shutting down resources in the old cloud just have to figure out new boots in the new cloud?
19:29:15 <ianw> i'll have to dig through the logs.  i got a bit sidetracked because there's a ton of extra messages in there from a leaking node issue, that i had a fix for but now is in merge conflict with the new statemachine stuff :)
19:29:32 <ianw> yep, i will respond, sorry haven't even opened emails yet :)
19:29:42 <clarkb> thanks! and thank you for pushing this along
19:30:04 <ianw> no worries, getting closer!  always interesting finding the things we didn't expect
19:30:12 <clarkb> #topic Upgrading Old Servers
19:30:44 <clarkb> As mentioned last week I didn't have any news and didn't expect to have news today. I don't. That said I'm hopeful if I can burn through some of my backlog this week I might actually get to booting a new server before our next meeting
19:30:57 <clarkb> #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades Notes
19:31:08 <clarkb> Please add your name next to any hosts you plan on dealing with
19:31:16 <clarkb> to avoid stepping on toes as we work through this
19:31:35 <clarkb> #topic Fips job updates
19:31:48 <clarkb> ade_lee added this to the agenda after I sent it it out but we've got plenty of time so I think we can dig in
19:32:13 <clarkb> there are two outstanding changes to make this work. The one ot zuul-jobs is the one I'm concerned about since it makes a backward incompatible change. Does anyone know if that has been communicated like fungi requested?
19:32:26 <clarkb> https://review.opendev.org/c/zuul/zuul-jobs/+/866881 that change
19:32:51 <clarkb> then the other question I had is if the second change can land at any time or if it needs to wait for ^
19:32:55 <clarkb> https://review.opendev.org/c/openstack/project-config/+/867112
19:33:17 <ade_lee> clarkb, it hasn't yet - but I can send it out -- I wanted to get some +2's first
19:33:27 <ade_lee> in case there were any other changes needed
19:33:49 <clarkb> ade_lee: gotcha, in that case I can try to review the zuul-jobs change today and then you can send email based on that I Guess
19:34:01 <ade_lee> clarkb, as far as I know only one team is using these jobs (and that is tripleo which is setting the boolean true in any case)
19:34:04 <clarkb> ade_lee: is the project-config chang eeffecitvely a noop until the zuul-jobs change lands?
19:34:06 <fungi> i think the project-config change can probably land at any time since it's just supplying additional context to the zuul-jobs parent?
19:34:26 <fungi> but i didn't review it with that possibility in mind
19:34:35 <ade_lee> yeah I think so
19:34:45 <clarkb> ade_lee: zuul-jobs is a bit of a global zuul library. What this means is within opendev there may be only one team but we have to be mindful of other potential users
19:34:52 <ade_lee> I already did talk with the tripleo folks
19:35:01 <ade_lee> ack understood
19:35:37 <ade_lee> clarkb, if you're good with the change, I'll send it out once I get a +2
19:35:55 <clarkb> ok I'll try to take a look today (it won't happen immediately after the meeting as I need to review themailman change then find lunch)
19:36:04 <clarkb> #topic Gitea Upgrades
19:36:07 <ade_lee> cool thanks
19:36:12 <clarkb> Another new topic since we have time.
19:36:14 <ianw> fungi: i we don't really care if the extant parent job is passed the secret
19:36:25 <ianw> i.e. if we merge that project config change now?
19:36:28 <fungi> ianw: right, it doesn't currently do anything with it
19:36:45 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/870850 Upgrade Gitea to 1.17.4 and its child upgrades us to 1.18.1
19:37:23 <clarkb> Over the holidays there were a couple of gitea updates that I meant to upgrade us to. I think the 1.17.4 upgrade is very straightforward and quite safe. The 1.18.1 upgrade form 1.17.4 is mroe involved. THere are template updates and golang is bumped to 1.19 and there are some config changes
19:37:49 <clarkb> anyway reviews welcome and I think we can hold a 1.18.1 host for manual interaction too just to be extra sure (I haven't gotten that far yet)
19:38:36 <clarkb> I wanted zuul to tell me if we got anything catastrophically wrong before I worried about holding stuff
19:39:26 <ianw> ++ will review
19:39:26 <clarkb> #topic Open Discussion
19:39:31 <clarkb> That was all I had on the agenda
19:39:34 <clarkb> Anything else?
19:39:54 <fungi> i got nuthin'
19:40:07 <ianw> apropos nothing, but i was wondering if maybe we should do something like apache redirect opendev.org / to some sort of fancier static page
19:40:25 <clarkb> we can also have gitea serve it via the template file pretty easily
19:40:35 <clarkb> the upside to that is it helps people find repos and orgs in gitea more easily
19:40:44 <clarkb> I want ot say fungi has been noolding on that sort of change
19:40:58 <ianw> i still feel like that kind of goes from 0 - 100mph on changes via gerrit and gating really quickly
19:41:25 <ianw> "that" being the current page we have there
19:41:43 <clarkb> ya I think the content can certainly use editing. But I think having gitea be navigable from the content is a good thing (and using the gitea templates gets us that integration for free ish)
19:41:59 <fungi> yeah, for now the change i have up to add donor logos takes the current assets directory approach
19:42:19 <frickler> what became of the afs disk usage topic? do we want to add it to the agenda to at least watch it regularly?
19:42:39 <ianw> fungi: oh, i feel like i've missed that, is it just one change?
19:42:43 <fungi> the last time i talked about shadowing the main page in apache, one of the objections was that we need to be careful any sub-paths it incorporates don't shadow git namespaces
19:43:06 <ianw> frickler: we did purge some openeuler stuff, but it didn't make a big difference
19:43:09 <clarkb> frickler: sure, can you add it?
19:43:19 <clarkb> the openeuler cleanup would've been about 100GB iirc
19:43:24 <fungi> #link https://review.opendev.org/869091 Feature our cloud donors on opendev.org
19:43:26 <fungi> ianw: ^
19:43:31 <ianw> will look, thanks
19:44:17 <fungi> ianw: and i too am interested in some reworking of the page content at the very least
19:44:32 <frickler> oh, I missed the openeuler cleanup. with that we're at least well below 90% again
19:44:38 <fungi> ianw: like some of the faq we have up there is probably no longer necessary now that we've been around for a while
19:45:02 <clarkb> frickler: I do think more cleanup would be good especially to understand the growth in some of the volumes
19:45:16 <clarkb> but openeuler cleanup was a quick bandaid since the change was already written
19:45:35 <ianw> i think i mentioned it last week, but i feel like purging xenial nodes, and everything related to it in various mirror volumes would be a help
19:45:43 <clarkb> ya that would make a good dent
19:45:48 <fungi> absolutely
19:46:02 <clarkb> but also doesn't explain why ubuntu has grown so much over the last 6 monmths
19:46:16 <clarkb> (which may also expose somethign inefficient we are doing)
19:47:27 <ianw> we could make the mirror scripts dump some sort of du log
19:47:46 <ianw> although i'm not sure how long that would take as it stats everything
19:48:03 <frickler> ok, I'll add the topic to the agenda with those ideas listed
19:50:23 <clarkb> Sounds like that may be it for today? Thank you for your time and your help running OpenDev
19:50:38 <clarkb> Oh! I've just remembered one other item
19:51:03 <clarkb> the service coordinator election. I plan to send email to service-discuss making january 31 - february 14 the official time frame for nominations since no one objected
19:51:19 <clarkb> I'll try yo get that out after lunch today
19:51:27 <fungi> thanks!
19:51:38 <clarkb> #endmeeting