19:02:26 <clarkb> #startmeeting infra
19:02:26 <opendevmeet> Meeting started Tue Apr 11 19:02:26 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:26 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:02:26 <opendevmeet> The meeting name has been set to 'infra'
19:02:44 <frickler> \o
19:02:48 <clarkb> As mentioned yesterday I didn't expect this to be well attended but did want to write down some of what has happened recently
19:02:58 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/LC5RWSADKSROICHN7YZKPOZX4AV55N34/ Our Agenda
19:04:18 <clarkb> #topic Announcements
19:04:22 <clarkb> #link https://meetpad.opendev.org/opendev-contributor-bootstrap-202304 New Contributor Onboarding Wednesday April 12, 2023 at 14:00 UTC
19:04:47 <clarkb> We've had at least two people interested in contributing and I'll have a bootstrapping session on meetpad tomorrow morning (for me)
19:05:03 <clarkb> Others are welcome to join if interested. Definitely not limited to those who have already chimed in
19:05:21 * frickler plans to join
19:05:29 <ianw> ++ i've mentioned to a few others too
19:05:44 <clarkb> #link https://etherpad.opendev.org/p/opendev-contributor-bootstrap-202304 Add your thoughts and ideas here
19:05:54 <clarkb> thats a link to the etherpad we'll use for ease of use if you have anything to add to the agenda
19:06:12 <frickler> do we want to order that a bit? looks pretty overwhelming to me
19:07:08 <clarkb> yes, my idea was to go over basics like gerrit and zuul and ensuring everyone knows how to communicate, push changes, join the meeting etc. Then dig into the ideas for contributions. Grouping the contribution ideas by interest is probably a good idea
19:07:17 <frickler> like select some (<5 ?) most important topics?
19:07:51 <frickler> the getting started section is fine, but some more structure to the other ideas
19:07:59 <ianw> yes, sorry that's probably largely my fault dumping a lot of todo things
19:08:24 <clarkb> I hesitate to prioritize on our side as I think that might discourage people who are interested in other things. More like here is some CI related stuff, here is some container related stuff, here is some something else stuff?
19:08:58 <clarkb> or maybe just reorder so that the first things you see are priorities and then the list can go on if you don't find something that interests you to start
19:09:09 <clarkb> I'll have to look at it more closely since it got updated
19:09:34 <ianw> yes i agree, i think the point is that doing any one thing is a way into contributing, where you'll start to see the bigger picture
19:09:50 <clarkb> ++
19:10:08 <clarkb> I'll take a look at organizing it a bit more after lunch today
19:10:12 <ianw> even zuul/zuul-jobs things expose you to a lot of the mechanics of ci, etc.
19:10:53 <clarkb> I do think we've got a good set of tasks that don't require root involvement. And some that do as things progress
19:11:16 <ianw> i think the question of where we keep these backlog items is a good one
19:11:32 <ianw> i don't think we have a good answer atm
19:11:36 <clarkb> ianw: ya I was thinking a bit about that after the storyboard question yesterday
19:12:03 <clarkb> I think there are options but nothing is used consistently today. (options include wiki, etherpad, less formal section of the specs repo, etc)
19:12:43 <clarkb> ok lets continue with the other items on the agenda. Looking forward to seeing people tomorrow
19:12:49 <clarkb> #topic Migrating to Quay
19:13:24 <clarkb> At last check all of the outstanding changes to update jobs merged. However, I think corvus may still have one to implement the intermediate registry promotion path that isn't quite ready?
19:13:40 <corvus> yep that's on my list
19:13:48 <clarkb> I think ianw and corvus were also working on testing the updates that had landed using zuul/zuul-client.
19:14:01 <clarkb> corvus: ianw: is there anything the rest of us can do at this point to help?
19:14:49 <ianw> that does work now with zuul-client, so that path of pushing and re-tagging via the api is ok
19:15:03 <corvus> also, just a friendly reminder from the last topic... there's some zuul stuff on there which is great, but keep in mind that zuul is an independent project and opendev's task list is kind of a separate thing :)
19:15:54 <clarkb> yes, I see those as tasks that would benefit opendev and we would work on anyway
19:16:05 <clarkb> but that would be contribution to zuul that aids opendev not direct contribution to opendev
19:16:17 <corvus> and yep, zuul-client is looking good with that container build path
19:16:22 <corvus> so i think the intermediate registry thing is the next step; i should have time for that this week.
19:17:04 <corvus> clarkb: yep.  95% of what i see there matches that.  a few of those are... kind of zuul-project level work.
19:17:36 <clarkb> excellent re intermediate registry stuff.
19:17:41 <clarkb> Let me know if I can help with reviews etc
19:17:48 <corvus> i don't think we need to get into specifics, but if we're indoctrinating new contributors, it might be good to highlight the separation.
19:17:55 <clarkb> corvus: can do
19:18:30 <clarkb> Also worth noting that April 14 (Friday) was the original you can't push to these orgs day on docker hub. They have said they will not do that anymore but we should keep our eyes open for anything unexpected
19:19:11 <clarkb> #topic Bastion Host Updates
19:19:18 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/879388 Fix rax rdns
19:19:37 <clarkb> This change is related to launching new servers recently and could use a second review if anyone has time
19:19:55 <clarkb> Also ianw you reported the fixes to reinstalling the launch venv have rax volume listing working now?
19:20:29 <ianw> yeah, it has now updated itself to the latest everything
19:21:04 <ianw> i think that's probably appropriate for this venv; as it is a bit of a dogfood for communicating with all our various clouds
19:21:38 <ianw> sorry, i mean to say, it updating itself periodically is probably appropriate ....
19:22:03 <clarkb> yup and we've identified specific versions of things where necessary too
19:22:05 <clarkb> which pip should respect
19:23:24 <clarkb> Anything else bridge related?
19:24:13 <ianw> not really; the backup stuff still hasn't had a lot of review, but i don't expect movement on that for a bit
19:24:46 <clarkb> everyone has been taking vacation :)
19:24:52 <clarkb> #topic Mailman 3
19:25:09 <clarkb> related to my last comment I don't think there is any movement on this either. I expect fungi should be able to get to it once he returns
19:25:22 <clarkb> #topic Gerrit Upgrade and Project Renames
19:25:52 <clarkb> We upgraded Gerrit and renamed three projects last Thursday/Friday (depending on timezone). As far as I can tell this went well though we did need almost all of our alotted time
19:26:21 <clarkb> One thing Gerrit did with its notedb migration was update all labels to have a copyCondition including changekind:NO_CHANGE
19:26:37 <clarkb> I feel like this was a bit over the top and filed a bug about it. But haven't seen any resposne from upstream on whether or not they think it is a bug
19:27:00 <clarkb> fungi reverted this update in All-Projects but we had updates to all of the labels in project specific configs too that we need to evaluate
19:27:42 <clarkb> Additionally gerrit updated the project specific config files to use hard tabs in the config file that our normalize config script removes. I think we should actually match what gerrit wants as much as possible to keep diffs as small as possible in situations like this
19:27:44 <ianw> yeah i need to finish that audit, sorry, got distracted
19:27:46 <corvus> what's wrong with it? (why not leave it?)
19:27:49 <clarkb> #link https://review.opendev.org/c/openstack/project-config/+/879906 Gerrit also normalized indentation of config files we should consider this to be in sync
19:28:20 <ianw> i still don't understand fully how a series of client events leads to a NO_CHANGE event being matched
19:28:23 <clarkb> corvus: two things. The first is that many of our copyConditions already include changekind:TRIVIAL_REBASE which is a superset of NO_CHANGE so it is confusingly redundant
19:28:56 <corvus> it=copycondition
19:28:57 <clarkb> corvus: the second is that for Verified and Workflow we explicitly had no copyConditions and I don't think it is appropriate for Gerrit to add any copy conditions in those cases.
19:29:19 <corvus> got it, thx.
19:29:21 <clarkb> We may decide to add copyConditions to Verified and/or Workflow but that should be done by users not automation
19:29:58 <clarkb> ianw: ya I don't think we need to rush on this as I suspect in projects specific configs it is rare to lack a copycondition
19:30:09 <ianw> i think it may relate to merge changes
19:30:11 <corvus> and yeah, that sounds potentially dangerous for project-specific "core approval" types of labels too.  :(
19:31:03 <corvus> is there some kind of "copycondition:never" that we should look for that might prevent this?
19:31:23 <ianw> there isn't afaik
19:31:48 <ianw> ... but yeah, i think fully understanding the conditions that lead to NO_CHANGE might help evaluate the impact
19:31:59 <clarkb> ++
19:32:15 <clarkb> and if we do bulk updates we should consider bundling in a hard tab change along with that
19:32:23 <ianw> "Matches when a new patch set is uploaded that has the same parent tree, code delta, and commit message as the previous patch set."
19:32:48 <ianw> i'm not sure how you do that, because i feel like gerrit usually rejects that as "nothing changed"?
19:33:07 <clarkb> ianw: it has and hasn't depending on the version. Maybe it won't reject it today anymore? Might be worth testing that
19:33:40 <clarkb> In any case doing an audit of what changed and understanding the situation when NO_CHANGE matches is the first step then we can plan any updates
19:33:45 <ianw> ++
19:33:51 <corvus> copycondition: "before:1970-01-01"
19:34:16 <clarkb> #link https://groups.google.com/g/repo-discuss/c/kkbyIKkzlAU/m/4t789yerBAAJ new usp query parameters
19:34:20 <corvus> (^ amusing way to get a copycondition that never matches?)
19:34:46 <clarkb> This is something frickler called out today. Gerrit is adding query strings to change urls to annotate where the link originated from. I've sent email asking them to clarify how this is actually used
19:35:47 <clarkb> As far as positives go the web ui now supports bulk actions (something gertty did first). Comments support markdown syntax.
19:35:57 <corvus> perhaps used by google's gerrit and they expect everyone else to ignore it / don't really care
19:36:22 <clarkb> Some of the diff annotations are nice too giving you a better indication of what is changing if not just file content
19:36:28 <corvus> they should do threaded change relationships next :)
19:36:46 <clarkb> ++
19:37:32 <clarkb> The only other next step I'ev currently got in mind is cleaning up the 3.6 image and adding 3.8 images and upgrade testing when we are ready
19:38:00 <clarkb> Thank you to everyone who helped, test, review, upgrade, fix etc during this process
19:39:04 <clarkb> #topic Upgrading Old Servers
19:39:14 <clarkb> #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades Notes
19:39:47 <clarkb> static02 should be serving all of the static.o.o and related content now. I used the LE cert config to determine what all needed updating and think I've got that all changed at this point
19:40:35 <clarkb> yesterday I shutdown apache2 on static01 as a sanity check. corvus found that zuul-ci.org was struggling this morning possibly due to stale dns pointing at static01? So I'll leave static01 alone for another day or two before cleaning it up to be sure there aren't any gremlins in the works
19:40:59 <clarkb> One thing to note is that Jammy's openafs-client package is newer than our PPAs which means static02 is using the upstream distro package version
19:41:32 <clarkb> I've also got an etherpad02 in the works which should be deployed now with an empty db. I'll need to sort out an etherpad downtime and db migration and dns update soon (hopefully tomorrow)
19:42:03 <clarkb> please don't put anything you want ot persist in etherpad02's db as the db migration from 01 will dump what is in 02
19:42:23 <clarkb> ianw: nameservers are the other push on this item. Anything new to report for them?
19:42:45 <ianw> no but soon i hope!
19:43:11 <clarkb> thanks, let me know if I can help. This generally has become relatively high priority for me now that docker doom is cancellled/delayed
19:43:22 <clarkb> #topic AFS volume quotas and utilization
19:43:47 <clarkb> I've been keeping an eye on this and there hasn't been any dramatic change but he trend is up slowly
19:44:03 <clarkb> I know that with osme of the wheel mirror cleanup work ideas we will delete content from AFS
19:44:12 <clarkb> And also that fedora 36 mirroring will go away soon
19:44:33 <clarkb> I think that those are good next steps. But please if you see something going sideways more quickly say something as we might need to intervene in different ways
19:45:36 <ianw> i also have about 20gb of wheels to remove when i get back to it
19:46:18 <clarkb> yup I think the current status/plan is good I jsut want to keep this on the agenda in case anyone has specific concerns or wants to tackle this in a different way. I'm not hearing anything like that so we can probably move on
19:46:25 <clarkb> #topic Gitea 1.19
19:46:31 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0
19:47:06 <clarkb> This change is ready I think. There is a held node for it too if anyone else wants to look. But it looks like 1.19.1 is going to happen very soon judging by their issue tracker's milestone for 1.19.1
19:47:23 <clarkb> I feel like we have enough going on and there isn't a presssing reason to upgrade quickly so waiting for the bugfix version is a good idea
19:48:06 <clarkb> That said I don't expect the change we land to upgrade to differ much between 1.19.0 and 1.19.1 if you have time to review the existing change that may be a good way to get ahead of things
19:48:54 <clarkb> #topic Quo vadis Storyboard
19:49:15 <clarkb> As expected a number of projects decided to move to launchpad at the PTG. You may see openstack/project-config changes to update their projects.yaml metadata to reflect this
19:49:34 <clarkb> I think we can also continue to encourage projects to work together if they end up building tools to migrate content or similar.
19:50:03 <clarkb> Oh also we can set projects read only in storyboard via admin config stuff in the storyboard UI (and likely API) should we get requests for that
19:50:48 <clarkb> #topic Open Discussion
19:50:51 <clarkb> anything else?
19:51:31 <clarkb> As noted previously my priorities this week are largely static02, etherpad02, and then supporting the work everyone else has going on.
19:51:45 <clarkb> I'm going to attempt to take Friday off as my kids are out of school Thursday and Friday. We'll see how that goes
19:54:44 <clarkb> Sounds like that may be everything. Thank you everyone! We'll be back here same time and location next week.
19:54:47 <clarkb> #endmeeting