19:00:35 <clarkb> #startmeeting infra
19:00:35 <opendevmeet> Meeting started Tue May 28 19:00:35 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:35 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:35 <opendevmeet> The meeting name has been set to 'infra'
19:00:44 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/HUXFNIX5ZHYFHSF4SRXSKN25P3CXES4E/ Our Agenda
19:00:49 <clarkb> fungi: there should be plenty of time I suspect
19:00:52 <clarkb> #topic Announcements
19:01:18 <clarkb> The OpenInfra Summit CFP closes tomorrow. I'm not sure what timezone that is relative to but get your proposals in if you plan to submit one
19:02:12 <clarkb> #topic Upgrading Old Servers
19:02:15 <clarkb> Let's dive right in
19:02:31 <clarkb> tonyb it sounds like you have learned things about the wiki migration/upgrade plan. Anything new you'd like to share?
19:02:57 <tonyb> I have the basics fleshed out
19:03:46 <tonyb> I should have the container stuff published this week and hopefully enough of the service-mediawiki stuff up that we can hold a node and start looking more carefully
19:04:11 <clarkb> that sounds like good progress. Anything concerning other than the theme may have to go away?
19:04:23 <tonyb> .... Unless I should also target Noble and just make mediawiki a real challenge
19:05:14 <tonyb> I haven't gotten far enough to verify if all the extensions work but they load so that's something
19:05:15 <clarkb> hopefully the container stuff isolates us quite a bit from the underlying platform
19:05:33 <clarkb> so even if we deploy on jammy moving to noble later should be more straightforward
19:05:51 <tonyb> Yup for sure.
19:06:12 <tonyb> That's the main aim
19:06:39 <tonyb> getting to a newer mediawiki is kinda secondary
19:06:43 <fungi> we haven't tested deploying any servers on noble yet, so i wouldn't recommend gating this work behind figuring that out
19:07:00 <clarkb> thats a good point too.
19:07:12 <fungi> one problem (or set of problems) at a time
19:07:18 <tonyb> Okay
19:07:47 <tonyb> I'm hoping that cacti will be easy after this ;P
19:07:53 <clarkb> anything we can do to help at this point or are we waiting for the held node to poke around on?
19:08:09 <tonyb> I think its that one
19:08:39 <clarkb> great I guess let us know when you get to that point. Anything else related to upgrading servers?
19:09:06 <tonyb> I have some "local hacks" that let me spin up a VM and then run roles etc from system-config so it's going okay and slightly faster than doing everythign in the gate
19:09:42 <tonyb> nothing else from me
19:09:42 <fungi> as far as deploying things on noble, the first one should probably be an upgrade of something we've already got in containers
19:09:53 <clarkb> fungi: or a mirror node
19:09:59 <fungi> yep. just to limit the amount of effort involved
19:10:23 <fungi> i wouldn't mix "make this work on noble" with "make this work in containers, and with ansible" was my point
19:10:43 <clarkb> ++
19:10:51 <tonyb> Sounds fair
19:11:16 <fungi> jammy has plenty of support lifetime still
19:11:49 <clarkb> #topic AFS Mirror Cleanups
19:12:10 <clarkb> Speaking of things with less lifetime I've sort of stalled on the xenial cleanup being distracted by things like gerrit and gitea upgrades
19:12:30 <clarkb> however there is one more change in topic:drop-ubuntu-xenial that I think is landable now. Just need to recheck the change now that its depends-on has merged
19:13:08 <clarkb> And then its back to pruning usage of xenial out where appropriate. I'll see if I find time for that soon (maybe tomorrow)
19:13:27 <clarkb> #topic Gerrit 3.9 Upgrade
19:13:36 <clarkb> #link https://etherpad.opendev.org/p/gerrit-upgrade-3.9 Upgrade prep and process notes
19:13:57 <clarkb> If you haven't yet it would be great if you can take a look at this document to ensure my plan looks sane and doesn't have any obvious issues
19:14:28 <clarkb> we have up to date image builds for 3.8 and 3.9 so we're upgrading from latest to latest of each release branch. And my concerns seem to be largely non issues after testing
19:14:46 <clarkb> assuming nothing comes up before May 31 (Friday) at 1600 UTC I'll proceed with doing the upgrade then as previously announced
19:15:54 <clarkb> The upgrade doesn't require any schema changes to the db and all reindexing can be done online. Should make the actual process relatively quick but I still allocated an hour in case we decide to rollback (which requires offline reindexing)
19:16:11 <fungi> it looks good to me, but as previously mentioned i'll be in a car all day so can't help. take my evaluation with a grain of salt ;)
19:16:45 <tonyb> I'll be around and and moved meetings out of the conflict zone
19:16:54 <fungi> i do still think this is good timing relative to release cycles of the larger communities we're hosting
19:17:15 <clarkb> ya I did cross check against the openstack release cycle and it looks clear. Starlingx just did a big release so expect them to be relatively quiet
19:17:31 <fungi> if there are any short-term hitches, it shouldn't be too hard for them to absorb gracefully
19:18:01 <fungi> but also the preliminary upgrade (and downgrade) testing has eliminated most concerns
19:19:04 <clarkb> cool I peeked at 3.10 upgrade notes and I think the lucene stuff that 3.9 initially tripped over may now be somewhat intentional in the 3.10 release so that upgrade is likely to be more interesting than this one
19:19:52 <clarkb> they say the indexes are automatically rebuilt when upgrading to 3.10 from 3.9 but I get the sense that isn't via normal online reindexing and its automatic offline reindexing instead
19:20:02 <clarkb> but we'll figure that out in future testing. No need to worry about it now
19:20:13 <clarkb> #topic Linaro Cloud Cert Renewal
19:20:38 <clarkb> Just a reminder that this is still nearing its expiry. I think fungi mentioned possibly looking at it later as it became more urgent
19:21:35 <fungi> yeah, i'm happy to serve as a last-minute backup if nobody else takes it
19:21:52 <clarkb> thanks
19:21:54 <tonyb> I can do it later today or tomorrow
19:21:58 <clarkb> double thanks!
19:22:05 <clarkb> #topic Gitea 1.22 Upgrade
19:22:18 <clarkb> The long awaited Gitea 1.22 release has finally arrived (yesterday)
19:22:35 <fungi> i'll stop gripping my seat now
19:22:41 <clarkb> This change does include the updates to the doctor tool to fixup mysql dbs that we will want to apply to our databases
19:22:45 <clarkb> s/change/release/
19:22:53 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/920580/ Change implementing the upgrade
19:23:21 <clarkb> I've gone ahead and pushed a change up that tests the upgrade (and will perform the upgrade when merged) and most things seem to be working
19:23:42 <clarkb> the appearance is changed slightly as part of some web css refactoring that I had to accomodate in our template overrides
19:24:07 <clarkb> Anyway I plan to test the doctor tool on a held node for that upgrade before we upgrade anything. That said I suspect the plan will be to do the upgrade and then do the doctoring as separate steps
19:24:26 <clarkb> since the doctor fixups are not strictly required for this upgrade we'll just continue to have the same case sensitivity problems until we fix things
19:24:35 <clarkb> #link https://104.130.74.99:3081/opendev/system-config Held Node
19:24:45 <tonyb> I did notice that 'code search' is back on the test server.  I don't know if we want to hide that in css again
19:24:47 <clarkb> this is the held node if anyone wants to take a look at the new gitea appearance or check things otherwise
19:25:02 <clarkb> tonyb: were we intentioanlly hiding it before?
19:25:31 <clarkb> https://opendev.org/explore/code I think its there today as well
19:25:31 <tonyb> I thought so, as it was generally not "right" in our usecase
19:25:53 <tonyb> Hmm okay I'll look again
19:25:58 <fungi> i wasn't aware there was a code search link previously, maybe that's new?
19:26:08 <clarkb> fungi: its in the current release too
19:26:09 <fungi> it's always been reachable via the explore tab
19:26:26 <clarkb> oh the difference is whether it is available in the in repo view
19:26:32 <fungi> ahh
19:26:50 <clarkb> I don't think it is a problem to have, we haven't removed codesearch because the gitea version is deficient but it isn't actively harmful as far as I can tell
19:27:01 <tonyb> Okay.
19:27:17 <clarkb> oh no its in the code repo view too just in a different spot (they really refactored the UI)
19:28:01 <tonyb> Ooooo so it is
19:28:07 <clarkb> But ya I'm not sure its a problem and I wasn't aware that we had taken any active steps to disable it. It was more of a "we can't remove codesearch until gitea works better with search"
19:28:40 <fungi> right, in my opinion we've continued to operate hound as a fallback option
19:28:58 <fungi> people can (and do) use either
19:29:10 <clarkb> keep calling out unexpected changes though. This release doesn't have a ton of "breaking" changes listed but it does seem to have a lot of user facing changes
19:29:16 <fungi> they both return slightly different results
19:29:50 <fungi> i think the main deficiency in gitea's search is a lack of regex support, yeah?
19:30:15 <clarkb> that and it doesn't reindex things aggressively so some stuff just doesn't show up in results
19:30:22 <fungi> also that it doesn't return all matches within each repo for a cross-repo search, but that's probably more of a pet peeve of mine
19:30:33 <clarkb> however, I wonder/suspect if the db problems are related to that and fixing up the db might fix a number of these weird slightly broken behaviors we've seen
19:30:55 <fungi> definitely worth investigating further
19:31:36 <clarkb> I'll dig into this upgrade more after gerrit is done
19:31:45 <clarkb> until then let me know if you notice anything weird on the held node
19:31:50 <clarkb> #topic Open Discussion
19:31:52 <clarkb> Anything else?
19:32:29 <fungi> the openstack tc is working on removing individual collaborators on pypi projects
19:32:32 * gouthamr is lurking
19:33:25 <fungi> the pypi interface for doing that is clunky and there is no programmatic api available for it, so removing hundreds of collaborators on openstack deliverables is a lot of work clicking things
19:33:58 <fungi> one option would be to temporarily share access to the openstackci pypi account with openstack tc members so they can share the workload
19:34:36 <fungi> though thinking through it, that will also be somewhat hindered by the new mfa requirement for pypi
19:35:09 <fungi> since they would each need some way to access multi-factor auth credentials to that account
19:35:22 <gouthamr> ah; bummer
19:35:27 <clarkb> two concerns: the account is used for more than just openstack (due to zuul multitenancy migrations being half done) and there is no way to promote an existing user like we can with gerrit group membership (for example we in the past promoted tonyb temporarily for brnach management before we grew the features and acls to just delegate that properly)
19:35:28 <fungi> the fallback option is that tact sig members who have access to our openstackci account do all the work
19:35:39 <clarkb> this means that once we share the credenitals the only way to unshare them is to rotate credentials
19:36:13 <fungi> yes, we'd presumably need to change the password and re-enroll the mfa with a new secret
19:36:23 <clarkb> I think I'm generally ok with the idea given previous escalations of privileges to people like tonyb for other tasks as long as we keep the group small like that. But am concerned that the amount of work rotating and testing and fixing things after the fact is worse than just clicking buttons
19:36:41 <clarkb> fungi: yes and then update zuul configs and test that it all still works with new new stray newlines involved
19:36:56 <fungi> well, the zuul configs should be using api keys now
19:37:02 <clarkb> if we could do this programmatically then we could create a token that we revoke I think, but I guess there is no way to do that via a token and requires an actual user session?
19:37:26 <clarkb> fungi: I'm operating under the assumtion we would have to roll those too
19:37:32 <clarkb> but maybe that is a bad assumption
19:37:56 <fungi> yeah, from what i gather there's an open warehouse feature request to add api methods for collaborator maintenance
19:38:26 <fungi> i don't think password and mfa changes invalidate api keys that have been issued, but i suppose it's possible
19:39:54 <clarkb> maybe we can test that with a different account and if it doesn't invalidate/force rolling them due to exposure then the effort involved is likely much less than I feared
19:40:59 <clarkb> then we can get a small set of volunteers for a small time window before we roll the main credentials. The other concern I have is that it is difficult to audit this stuff (again due to the lack of groups and separate accounts) so keepign the window involved small helps
19:41:01 <corvus> so... the end state would be that only the opendev credential has owner or maintainer status?
19:41:23 <fungi> yes, for official openstack deliverables
19:41:32 <corvus> i feel like that is slightly misaligned
19:42:13 <fungi> it would probably be an improvement for openstack to also have more direct control over these packages
19:42:19 <corvus> (in that it means that opendev is on the hook for any maintenance of the ownership list (including this situation))
19:42:52 <corvus> would it be appropriate for openstack to have its own (essentially never-used) credential for this?
19:43:04 <fungi> the way i see it for now is that the openstack tact sig and opendev sysadmins are sharing that responsibility (and account), but i can see room for improvement
19:43:23 <clarkb> corvus: yes that could possibly work, but we'd still need to help in the transition work I think
19:43:35 <corvus> hrm, so other tenants should get their own accounts?
19:43:35 <clarkb> but in theory we'd do it once then be done
19:43:48 <fungi> making a new account for openstack raises the question of who stores and safeguards the password and mfa secret
19:43:57 <tonyb> Can we simulate the clicky clicky with something ike selenium?
19:44:06 <clarkb> corvus: I think long term that would be ideal. THe current account is called "openstackci" iirc
19:44:25 <clarkb> yes confirmed
19:44:29 <fungi> tonyb: possibly, but https://xkcd.com/1205
19:44:42 <corvus> yeah... i'd be more comfortable with opendev being the only holder of these credentials if there were some low-impact way of us doing that maintenance (ie, script)
19:45:22 <fungi> our irc accessbot is probably the closest example we have currently
19:45:38 <frickler> I can check until next week how much work this actually would be and whether I could volunteer to execute it in a reasonable time frame
19:45:40 <corvus> agree, and that's pretty self-service
19:45:57 <tonyb> fungi: fair point
19:46:17 <fungi> frickler: it's about 3 clicks per package plus re-entering the package name, and potentially several seconds wait between each operation
19:47:00 <fungi> er, 3 clicks per collaborator on a package, plus re-entering the collaborator's username i mean
19:47:05 <fungi> on each package
19:47:30 <frickler> but usually there is only one other co-owner or multiple ones? do we know?
19:47:42 <fungi> pypi has a "are you really sure, type the name into this text input box to confirm you know what you're doing" step
19:47:56 <clarkb> I think we can get that programmitcally as you can list all of the maintainers + owners together (it doesn't distinguish them though)
19:47:59 <fungi> frickler: on some packages there are more than one
19:48:14 <fungi> gouthamr has a list
19:48:58 <gouthamr> yes; i have the extra-maintainers in a yaml dump here:
19:49:00 <gouthamr> #link https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L52
19:49:29 <clarkb> side note looking at that list you should probably sort it by priority
19:49:38 <clarkb> a number of those are pacakges we don't care about much anymore so are less important
19:49:46 <frickler> so I can just start to work and see how it goes. and then judge how many weeks I think it might take me to go through gathering enough concentration in between
19:51:16 <frickler> and that would avoid all issues with creds rotation being needed
19:51:20 <fungi> also, to be clear, the triggering event for this effort was that a historical collaborator on a package which was considered to be under the governance of openstack decided to push their own release for that project outside of what code was being maintained in gerrit, and the interest in preventing that in the future was amplified somewhat by the xz-utils compromise more recently
19:51:22 <corvus> ballpark that's maybe ~175 people to remove?
19:51:55 <clarkb> and more recently people have asked people who have moved on from poenstack to take over openstack packages because they are still listed on the pypi package
19:53:42 <clarkb> anyway seems like frickler putting some effort into it to get a sense of the scale is a good first step
19:53:45 <corvus> i agree with the effort; just not sure i love the idea that if anything comes up, opendev admins may be on the hook for updating pypi as packages are removed from openstack official governance.
19:53:45 <frickler> gouthamr: do we actually care about (possibly soon to be) retired projects?
19:54:36 <frickler> corvus: well that's the case already for the majority of the projects anyway?
19:54:36 <corvus> i have no better ideas given the current constraints; only a desire to change the constraints.  :)
19:54:38 <fungi> corvus: i'm happy to reframe that as openstack tact sig representatives within the opendev sysadmins will take on those tasks as they arise, at least until we come up with a more self-service option like accessbot
19:54:52 <clarkb> corvus: what sorts of updates are you concerned about? I think much of the actual package properties and attributes can be driven through the release process. The main bit that can't is the maintainers/owners aiui
19:56:01 <corvus> the only concrete example i can think of is "hey opendev, can you add <username> to <project> because it's being released from openstack governance".
19:56:03 <clarkb> and yes I agree getting us out of this business is ideal, but currently we're stuck in it one way or another (we could add a second account for openstack to manage things with but that would flow through the openstackci account anyway)
19:56:42 <clarkb> corvus: ack. One option there may be to say you can't publish to the old name on pypi. But that may be problematic for other reasons (needing to update deps everywhere for example)
19:56:42 <fungi> yeah, i mean, the majority of official openstack packages were *already* in this situation
19:57:20 <clarkb> ya I think the current proposal is still an improvement if not the ideal state. And we can revisit if we need to take additional steps to get closer to that ideal
19:57:22 <fungi> because an initial upload of a release on pypi creates the project there and adds the uploader as the only owner
19:59:10 <corvus> to be clear, i'm not objecting to this work, or to frickler, or anyone else doing it.  just noting that we may be signing up for more work in the future depending on openstack project lifecycle, and that this is not as self-service as we would like.
19:59:32 <clarkb> we are just about at time now. I think it is reasonable for frickler to investigate the effort required to do this directly. Then next week at the TC meeting we can discuss the idea of package ownership having a secondary account to move us out of the dependency tree
19:59:41 <clarkb> corvus: ++
19:59:49 <fungi> so anyway, the current state seems to be that frickler and i can look into sharing the work the tc is requesting, and separately we can brainstorm something like accessbot to create and add accounts on pypi (though the lack of an api for that in warehouse makes that a bit of a challenge at present)
20:00:12 <clarkb> fungi: or just do another pass of adding openstackowner to all the packages
20:00:17 <fungi> yeah
20:00:27 <fungi> which i can also volunteer for if needed
20:00:40 <clarkb> then we can state "openstackci" is only used for publication. Other actions like changing owners when packages retire within openstack should be handled by the openstackowner account
20:00:51 <clarkb> and we are at time. Thank you everyone!
20:00:58 <tonyb> Thanks all
20:01:17 <corvus> and if there ever is an api, we can change the name of the publishing account :)
20:01:17 <clarkb> I'm going to end it here since I need to eat lunch, but feel free to continue discussion in our other venues if you'd like to finish up on any of these thoughts
20:01:23 <clarkb> corvus: ++
20:01:25 <clarkb> #endmeeting