19:01:26 <clarkb> #startmeeting infra
19:01:27 <openstack> Meeting started Tue Mar 12 19:01:26 2019 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:30 <openstack> The meeting name has been set to 'infra'
19:01:35 <clarkb> #link http://lists.openstack.org/pipermail/openstack-infra/2019-March/006293.html
19:01:41 <clarkb> #topic Announcements
19:01:50 <clarkb> PTL Candidate nominations close today (March 12) at 23:45UTC.
19:02:11 <clarkb> I have submitted a self nomination to be infra PTL. Its not too late for someone else to run if they would like to :)
19:02:30 <clarkb> These nominations are for the Train cycle
19:02:30 <anteaya> is infra still governed by the tc?
19:02:36 <clarkb> anteaya: it is
19:02:42 <anteaya> and will it continue to be in future?
19:02:47 <anteaya> and thanks
19:03:07 <fungi> not all of what the openstack infrastructure team does is moving into opendev
19:03:33 <clarkb> that and I don't expect us to start really thinking about breaking pieces out until we've got a bit more opendev work done (probably during the next cycle)
19:03:51 <anteaya> okay thanks
19:03:52 <fungi> at some point it might be worthwhile to talk about converting the team to a sig, but i wouldn't begin to consider that until after opendev is nearing some steady state
19:04:21 <clarkb> On the agenda email I had noted you could submit forum sessions but I'm told that closed late yesterday?
19:04:26 <anteaya> and thanks for standing as ptl again clarkb
19:04:33 <clarkb> so that announcement item is no longer valid re forum ssessions
19:04:41 <mordred> yes - forum session selection time has started
19:05:00 <clarkb> #topic Actions from last meeting
19:05:04 <fungi> though if there's something we really need to squeeze in, we might be able to talk the selection committee into it
19:05:12 <clarkb> fungi: good to know thanks
19:05:15 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-03-05-19.01.txt minutes from last meeting
19:05:24 <clarkb> ianw had an action to rotate our backup server volumes
19:05:29 <clarkb> ianw: has that happened?
19:05:40 <ianw> no, on my todo list, sorry was out a couple of days last week
19:05:52 <clarkb> #action ianw rotate backup server disks
19:06:53 <clarkb> #topic Specs approval
19:07:15 <clarkb> There are no specs up for approval. However I wanted to quickly follow up on the letsencrypt spec
19:07:33 <clarkb> ianw: re ^ are you happy with the dns subdomain for acme validation? and do we think that needs a spec update?
19:07:54 <ianw> clarkb: i think we can handle it in reviews at this point
19:07:58 <clarkb> cool thanks
19:08:10 <clarkb> #topic Priority Efforts
19:08:19 <clarkb> #topic Update Config Management
19:09:05 <clarkb> On the puppet 4 side of things I've done some cleanup of the upgrade process so that we don't install puppet4 then downgrade to puppet 3 then back to puppet 4 again. This cleaned up puppet logs on review-dev
19:09:42 <clarkb> Otherwise I've mostly used the dev servers as a set to clean up some of the things that puppet 4 points out. thank you for the reviews on that
19:10:05 <clarkb> On the docker side of things corvus is ready to document the speculative image building system and mark it as in production
19:10:21 <clarkb> This allows us to test changes to docker deployed systems before we merge their image build updates
19:10:47 <clarkb> should end up being super useful particularly as we end up deploying more complex systems with docker
19:11:17 <clarkb> corvus: ^ other than the password change happenign today and the docs updates that need to happen is there anything to keep in mind or look at for that now?
19:11:45 <corvus> clarkb: nope -- just note that the jobs in system-config are the vanguard
19:11:52 <corvus> so look to them for reference for how to do things
19:11:58 <corvus> also zuul-preview
19:12:13 <corvus> i'll bring zuul and nodepool forward to use those jobs soon
19:13:31 <clarkb> I'll be looking to merge the run puppet 4 in more places change(s) later this week. I'll probably ping people for reviews once I am happy that all the minor issues we've foudn on the dev servers are ignorable or not worth fixing
19:13:50 <clarkb> #topic OpenDev
19:14:06 <corvus> i sent out the email announcement
19:14:10 <corvus> #link opendev gerrit repo migration announcement http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
19:14:24 <corvus> and got some replies
19:14:28 <corvus> #info jroll and dtroyer are POCs for openstack and starlingx respectively
19:14:41 <corvus> i didn't see a reply from airship
19:14:51 <roman_g> Airship is here as requested in an e-mail
19:14:57 <corvus> roman_g: yay!
19:15:03 <roman_g> Hello all. =)
19:15:16 <corvus> #undo
19:15:17 <corvus> #info jroll, dtroyer, and roman_g are POCs for openstack, starlingx, and airship respectively
19:15:30 <fungi> um, i think you're forgetting one
19:15:33 <fungi> starts with a z?
19:15:53 <corvus> yeah... no one replied to that either.  i'm going to assume everyone assumes i'll handle it...
19:15:59 <corvus> unless anyone wants to volunteer real quick? :)
19:16:08 <roman_g> zuul?
19:16:25 <corvus> #undo
19:16:26 <corvus> #info jroll, dtroyer, roman_g, and corvus are POCs for openstack, starlingx, airship, and zuul respectively
19:16:38 <corvus> okay, we have a full complement :)
19:16:42 <clarkb> thank you to all our volunteers :)
19:16:44 <roman_g> :)
19:16:53 <anteaya> are edge or kata affected/
19:16:56 <anteaya> ?
19:17:04 <fungi> i can volunteer to represent zuul in opendev discussions if you really want, but i have a feeling your bandwidth with yourself might be higher
19:17:13 <clarkb> anteaya: kata doesn't really host with us currently :/ so not really
19:17:21 <anteaya> ah okay thanks
19:17:31 <clarkb> but I expect ttx would be happy to help there should we need a volunteer
19:17:35 <corvus> there's two main things we need to coordinate on -- any repository renames, and scheduling
19:17:49 <fungi> the one repo kata does have in our gerrit is unlikely to be heavily impacted by this
19:18:05 <fungi> and teh edge effort doesn't have any software
19:18:19 <anteaya> oh
19:18:33 <fungi> they are/were primarily a whitepaper writing effort
19:18:39 <clarkb> corvus: for scheduling I expect on our side of things we'll want to do things no sooner than the summit
19:18:43 <corvus> regarding renames, i think it would be best if each of the POCs come back with a list of repos that should be renamed when we perform the migration
19:18:52 <clarkb> corvus: ++
19:19:36 <dtroyer> roger
19:19:42 <clarkb> corvus: along with the new names?
19:19:43 <corvus> roman_g, dtroyer, corvus: for your groups, i'm kind of imagining you'd take the opportunity to just forklift all your repos to your own namespace.  eg openstack-infra/zuul will move to zuul/zuul.
19:20:00 <roman_g> #action each of the POCs come back with a list of repos that should be renamed when we (#openstack-infra) perform the migration
19:20:19 <corvus> i mean, other options exist, but that's sort of what i'm assuming things would look like
19:20:25 <dtroyer> corvus: that is our plan, yes
19:20:37 <fungi> openstack will, predictably, probably me much more messy
19:20:39 <clarkb> dtroyer: will you drop the stx- prefix?
19:20:45 <fungi> s/me/be/
19:20:49 <mordred> the larger/complicated list is potentially openstack - due to maybe moving stackforge thigns out
19:20:52 <mordred> fungi: jinx
19:20:53 <dtroyer> clarkb: still an open question, I imagine we will.
19:20:54 <clarkb> dtroyer: those details can be sorted out with the listings, but that is stuff to think about probably
19:20:56 <corvus> jroll: your job is harder :)  since openstack/ is hosting a bunch of unofficial projects, you (as in the TC) get to decide what you want to do about that.
19:21:11 * mordred believes in jroll
19:21:20 <fungi> there was also interest in openstack of not using just one namespace
19:21:43 <corvus> jroll: you could leave them in openstack/ for now, we could move them all... or.. whatever :)
19:21:51 <fungi> e.g., neutron/neutron-lib and nova/python-novaclient and so on
19:22:04 <mordred> yup. it's all good by us!
19:22:23 <clarkb> There were rumblings of bringing stackforge back
19:22:51 <corvus> i don't think we need a dedicated org for unofficial projects
19:22:51 <fungi> heh, yeah, as the catch-all namespace for opendev hosting of projects who don't request any particular namespace
19:23:22 <fungi> would we just duplicate their repository name as their namespace prefix too?
19:23:34 <mordred> we certainly _could_
19:23:39 <fungi> i gather gitea requires some namespace prefix
19:23:50 <corvus> i did take the liberty of starting one action item for this, which is to set up a place where folks with unofficial projects can self-select a new name.  so regardless of whether the tc decides to let folks stay in the openstack namespace, projects can self-select to move along with the migration.
19:24:05 <mordred> ++
19:24:05 <corvus> #link unofficial project move self-selection: https://ethercalc.openstack.org/opendev-transition
19:24:13 <corvus> we don't have many takers there yet.  :)
19:24:35 <corvus> but you can see my suggested answer to the namespace question :)
19:24:50 <fungi> hard to pass up that clickbait
19:26:00 <corvus> anyway, we'll just need that list from each of the 4 osf projects before we execute the migration.  any questions on this point?
19:26:00 <clarkb> On the opendev/infra side of things do we think that automated rename tooling that doesn't require a gerrit outage is required before we make the transition (that will likely inform the timing of this whole process)
19:26:12 <corvus> clarkb: i don't think so
19:26:39 <fungi> even if there were automated rename tooling we wouldn't want to use it for this batch
19:26:43 <corvus> i think our rename policy should stay the same for now -- we do it periodically as necessary and get really annoyed about it.  :(
19:27:04 <clarkb> ya I think I'm mostly fine with that as it is no longer cool to rename a project every other week
19:27:08 <mordred> yeah. although I do think it would be great if the gerrit rename plugin became magically usable for online renames
19:27:15 <fungi> for reasons such as task 29708 (which i haven't done more than ponder so far, sorry)
19:27:50 <fungi> well, i take that back, i did a little bit of prototyping
19:27:50 <anteaya> when was the last time a rename was done?
19:28:02 <corvus> that is something for jroll/openstack to consider in developing a policy for the openstack/ namespace.  if the idea is to revert to "official only" then keep in mind we're still not any better at renames as the last time.  however, we do have fewer projects incubating.
19:28:06 <clarkb> anteaya: we did some renames a few months ago.
19:28:15 <clarkb> I wnt to say november ish
19:28:19 <anteaya> how long did the reindex take?
19:28:28 <clarkb> anteaya: we are able to do it online now
19:28:33 <anteaya> oh lovely
19:28:34 <roman_g> Шэь еруку фдкуфвн
19:28:35 <fungi> reindexes have been less of a concern now that they're no longer offline
19:28:38 <roman_g> sorry
19:28:57 <fungi> that came with gerrit 2.11 i think
19:29:03 <mordred> roman_g: I'm going to assume that translates to "mordred is super awesome" :)
19:29:13 <fungi> (for us anyway)
19:29:13 <roman_g> mordred: yes, sure =)
19:29:21 <anteaya> cool, thanks
19:29:30 <corvus> the other point we need to coordinate on is scheduling
19:30:03 <corvus> broadly speaking, we have a couple weeks, then things get hairy for the openstack release
19:30:39 <corvus> let's assume we still need those couple weeks to finish up our pre-tasks, which means we're probably looking at after the stein release, week of apr 12
19:30:43 <fungi> it sounded like openstack was wanting to see it happen after release activity for stein concludes
19:30:50 <anteaya> is there pressure to complete this prior to a certain time?
19:31:09 <clarkb> corvus: ya then we run into summit/ptg prep
19:31:11 <fungi> anteaya: the sooner we get it over with the sooner we can move on to other parts of the opendev transition
19:31:20 <anteaya> ah okay
19:31:26 <clarkb> corvus: I was thinking the ptg might be a good time for a sanity check of planning then we can do the transition after that?
19:31:31 <corvus> as the zuul representative, i would like it to happen asap :)
19:31:35 <fungi> this bit is a blocker for many other activities
19:31:40 <anteaya> :)
19:31:42 <clarkb> corvus: also things tend to be quiet in the week or two after a big event like that which likely makes it a good time for our users
19:31:44 <dtroyer> StarlingX is on a release schedule of OpenStack+N weeks, where N is currently between 4 and 6
19:31:55 <corvus> clarkb: hrm.  i wouldn't mind it happening before the summit
19:32:46 <dtroyer> I think we would prefer either before summit or a month or so later
19:32:53 <corvus> dtroyer: summit is ~3 weeks after release, so ... yeah that
19:32:56 <clarkb> corvus: my worry is that between summit prep, trusty upgrades, and gitea prep we don't have a lot of time between now and then
19:33:31 <anteaya> will you rename all projects at once?
19:33:48 <corvus> i am performing zero summit prep, so april 15-26 are excellent for me
19:33:51 <fungi> anteaya: that's the current plan, yes
19:34:07 <anteaya> if yes, then I think jroll's thoughts on how long it will take to sort out openstack names will factor
19:34:14 <corvus> we're changing a hostname, so there isn't really much of a choice
19:34:31 <mordred> I think before the summit is potentially nice for being able to talk about things. maybe provisionally plan for april 15 week - but also plan for a "crap we're not ready" contingency?
19:34:38 <clarkb> fwiw I have no objection to doing it pre summit if we get things lined up by then
19:35:00 <corvus> anteaya: if it takes longer than we have before the migration to sort that out, then i think it makes sense to perform further renames after the migration to opendev.
19:35:15 <anteaya> that would make sense
19:35:26 <corvus> i look at it this way -- we *must* rename all projects at least once.  we can *optionally* take the opportunity to avoid a second rename for anyone who is ready.
19:35:36 <clarkb> mordred: ya I can see that. basically tell everyon we are targetting that week and then have an out if we need to
19:35:42 <fungi> and basically results in at least two renames causing significant impact to each project involved
19:35:53 <clarkb> That gives us ~ 5 weeks
19:36:03 <corvus> yes, from a technical point of view, i think we should be able to make that
19:36:15 <corvus> given the remaining tasks on:
19:36:16 <corvus> #link opendev gerrit story https://storyboard.openstack.org/#!/story/2004627
19:36:36 <clarkb> is default branch setting still broken for unknown reasons?
19:36:42 <anteaya> well two scheduled renames might be better than one big rename that can't be scheduled
19:36:48 <mordred> clarkb: we have workaround in place
19:36:49 <corvus> clarkb: i think it's fixed
19:37:11 <mordred> we don't know the root cause in upstream gitea and I donm't believe that's fixed - but I believe it's fixed in our automation
19:37:11 <fungi> anteaya: it's one big rename followed by maybe an additional rename for the same projects (first rename is the change of domani)
19:37:17 <fungi> s/domani/domain/
19:37:24 <fungi> i was typing latin there for a moment
19:37:25 <clarkb> mordred: thanks
19:37:26 <anteaya> fungi /nod
19:37:34 <corvus> do we want to do it during the week of 15-19, or do we want to, erm, perform the move on easter weekend? :)
19:37:37 <anteaya> don't bring god into it
19:37:50 <clarkb> then ya I agree I think the major known problems are addressed and the other technical work is coordination and planning
19:38:20 <mordred> corvus: I do not personally have any conflicts on the weekend known by some as easter
19:38:26 <corvus> i'm happy to volunteer to work on the weekend if it helps reduce disruption.
19:38:43 <clarkb> I've probably got egg hunting acitvities with the kids on sunday but otherwise am free
19:38:57 <corvus> if it would be better to do it during the week because more folks would be available (either infra folks to move, or project folks to deal with unanticipated fallout), that wfm too
19:39:34 <mordred> same. I'm fine either way - happy to work on that weekend, or to not work on that weekend, whichever provides the most happiness for the largest number of humans
19:39:54 <clarkb> Friday is a holiday for many, maybe do friday?
19:40:03 <clarkb> that then gives us the weekend to sort stuff out if necessary
19:40:11 <mordred> 4/19?
19:40:17 <corvus> yeah, 4/19 wfm
19:40:20 <clarkb> yup
19:40:30 <corvus> dtroyer: that still close enough to the release to make it okay-ish for starlingx?
19:40:35 <mordred> yeah. also - it'll probably be slower then because of pre-summit anyway
19:41:11 <corvus> and post release tension release :)
19:41:12 <dtroyer> corvus: I think so, I'll ask tomorrow in our community meeting.
19:41:26 <dtroyer> Our RC1 is scheduled for 4/29 right now
19:41:32 <corvus> roman_g: any concerns?
19:41:48 <roman_g> corvus: no concerns. thanks for your work.
19:42:19 <corvus> #info tentative date for opendev migration: april 19
19:42:47 <roman_g> corvus - are you POC for this change?
19:42:54 <corvus> let's wait to hear back from dtroyer and jroll, and confirm that next week, but start planning for that.
19:42:58 <corvus> roman_g: yep
19:43:11 <roman_g> great, thanks
19:43:52 <corvus> roman_g, jroll, dtroyer: i don't think we need to ask you to attend every infra meeting, but you're welcome to.  this is a standing item on the agenda until its done, and we'll talk about it a little every week.
19:44:16 <corvus> if we do need to get everyone back for another real-time discussion, i'll let you know
19:44:25 <corvus> and that's EOT from me
19:44:52 <dtroyer> corvus: ack
19:45:13 <clarkb> fwiw I've grabbed the set launchpad when not using storyboard task. I don't know what fixup current errors means but we can followup on that outside of the meeting
19:45:27 <fungi> sorry, getting pulled in several conversational directions and then my workstation decided to spontaneously reboot, but ~easter week should work for me. there may be a day when i have company and possibly a wedding anniversary, but i can work around those
19:45:40 <corvus> clarkb: means "everyone is currently set to use storyboard, so update the existing other projects to use lp"
19:45:45 <mordred> clarkb: I believe that means "do a $something to change the links for existing projects in the gitea db"
19:45:59 <clarkb> got it
19:46:02 <corvus> clarkb: and thanks!
19:46:25 <anteaya> fungi happy anniversary
19:46:31 <clarkb> thank you to our volunteers
19:46:41 <clarkb> and now lets continue on as we only have ~14 minutes left
19:46:55 <clarkb> #topic General Topics
19:47:16 <clarkb> I wanted to do a quick update on the trusty server upgrade progress.
19:47:29 <clarkb> I managed to get all three of the afs fileservers upgraded yesterday doing an inplace upgrade
19:47:57 <clarkb> There was a small hiccup where afs clients didn't seem to notice that they could use a different server to pull data from when the one they were talking to went down
19:48:01 <clarkb> otherwise I think it went really well
19:48:18 <clarkb> I've since starting sketching a plan for the afsdb server upgrades. https://etherpad.openstack.org/p/afs-dbserver-trusty-to-xenial
19:48:35 <fungi> and the replacement wiki-dev is in the inventory now. i was going to check on it after the meeting, but instead i'll be figuring out why my workstation rebooted
19:49:16 <clarkb> One question I had for the group re ^ is do we feel comfortable using the upgrade of the fileservers as the template for upgrading the db servers? or do you think I should make a db server snapshot, boot that, then run through a do-release-upgrade?
19:49:34 <clarkb> from what I can tell there is only a one package difference ebtween the two servers. openafs-dbserver is installed on the dbservers and not on the fileservers
19:49:42 <clarkb> then the differences are in openafs configs
19:50:17 <clarkb> given that I don't really expect any trouble from the ubuntu upgrade process, but maybe those that know more about openafs think we should test the upgrade first?
19:50:27 <corvus> i think the utility of a snapshot would be limited.  if things go very very badly, i'm not at all sure restoring from snapshot would be easier than starting from scratch.
19:50:49 <corvus> and i don't think this is possible to test except by creating a new cell
19:51:32 <zbr> re bindep question from yesterday, maybe you can cast some votes on https://ask.openstack.org/en/question/120501/what-atom-should-bindep-expose-about-official-distro-python-version/ ;)
19:51:36 <clarkb> given ^ and the reliability of do-release-upgrade on 3 very similar hosts so far I think I'm comfortable just doing do-release-upgrade and answering the questions as I did for the fileservers
19:51:50 <corvus> clarkb: ++
19:51:55 <clarkb> I'll copy the process over from the fileserver plan and we can look at it in the new context
19:52:22 <clarkb> otherwise thank you all for the help on getting that done as well as for the other server upgrades
19:52:38 <clarkb> one thing to note is that when we do an in place upgrade from trusty to xenial we have to clean up apt things for our puppet install
19:52:54 <clarkb> this process is documentedin my etherpads above, but whoever does lists.o.o will likely need that too
19:53:33 <clarkb> #topic Open Discussion
19:53:41 <ianw> i guess i must have missed the 24 hour cut off for the agenda, but i did want to discuss git: -> https: as part of opendev
19:53:58 <clarkb> ianw: go for it
19:54:11 <ianw> i have a bunch of links on the agenda wiki
19:54:28 <ianw> in short, before i send out 400+ changes to move git:// -> https:// i'd like to discuss it :)
19:54:32 <corvus> can you link that?
19:54:36 <clarkb> https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:54:49 <ianw> do we have a service user to submit these from?
19:55:03 <ianw> or, should we do that
19:55:12 <clarkb> ianw: we have the openstack gerrit bot it does the requirements update pushes iirc
19:55:57 <fungi> i think the bigger problem is not who submits the changes but getting them merged
19:55:58 <clarkb> ianw: dhellmann has recruited our contributors that like to push a lot of changes like this to do pushes like this in the past
19:56:17 <clarkb> ianw: dhellmann et al may have tooling that is useful
19:56:22 <corvus> i think that's the "openstack release bot" ?
19:56:27 <ianw> clarkb: well the scripts are ready and aimed ... it's just a matter of pushing enter :)
19:56:32 <clarkb> ianw: got it
19:56:34 <mordred> yeah. I'd say dhellmann is the most recent "submit changes to every repo" champion
19:56:53 <fungi> for the git-review and zuul configuration in repositories we're planning to just bypass ci entirely and (at least it's my current plan) merge them directly on disk while gerrit is offline
19:56:56 <clarkb> ianw: I think we should communicate it to the discuss list first and explain why we want this so we don't get blanket -1s
19:57:07 <corvus> fungi: ++
19:57:11 <dhellmann> oh, god, please do not submit changes to every repo
19:57:21 <mordred> but I don't think there's any specific reason to do it as not-ianw
19:57:43 <ianw> dhellmann: well, i'm not sure what else to do?  some are +1/-1 diffs, others are not
19:57:48 <clarkb> ya I agree with mordred I think if ianw did the work to write the commits (even if automated) it is perfectly accurate for ianw to be author and committer
19:57:50 <roman_g> ianw: for how many says would git:// still work after a set of changes is pushed?
19:57:54 <roman_g> *days
19:57:59 <fungi> we've also talked to the openstack tc about the possibility to say an infra representative is clear to just use gerrit admin privs to approve changes like that which block significant infrastructure activities
19:58:11 <clarkb> roman_g: it will work until our transition date (that we just pencilled in for april 19)
19:58:24 <roman_g> good to know
19:58:29 <dhellmann> fungi : yes, do that
19:58:46 <dhellmann> submitting and managing hundreds of patches is just going to burn you out
19:58:48 <corvus> i don't think that most *infrastructure* things will be broken if git/https changes don't merge, as devstack in the gate bypasses this.  but end-user use of things may break
19:59:04 <ianw> #link https://review.openstack.org/#/q/topic:opendev-gerrit+owner:iwienand
19:59:05 <clarkb> dhellmann: fungi is the concern use of gerrit and reviews or the waiting on people to review the cahnges?
19:59:07 <ianw> is a sample
19:59:19 <fungi> so at least we're covered where it comes to openstack official repositories if we want to give the tc a heads up that we're going to "just merge it"
19:59:21 <clarkb> My preferences is that we still go through gerrit reviews but then we can certainly approve via our infra backdoor
19:59:45 <fungi> clarkb: my main concern is waiting for the cat herding which comes with expecting different review teams to approve the various changes
19:59:49 <corvus> i believe that is why we are reluctant to invoke our "let us just merge this" authority on this (unlike .gitreview and .zuul.yaml, where we clearly know it will break infrastructure and we know exactly what we need to change)
19:59:53 <dhellmann> clarkb : you will have to explain individually to hundreds of reviewers who don't read the mailing list why the patch is needed. You will have a bunch of folks show up out of nowhere and start submitting similar patches, frequently incorrectly. It's just a huge pain.
20:00:43 <anteaya> I wouldn't have expected copycat patches
20:00:44 <clarkb> (as a note we are offocially at time, but don't want to end this discussion until ianw has what he needs)
20:00:45 <corvus> i'm open to the suggestion that this, while perhaps not technically an infrastructure blocker, is just too durned annoying to handle another way.
20:00:50 <anteaya> guess its a thing
20:00:51 <mordred> I'm in favor of infra-approving the patches in this case -- also, we should maybe do this over a weekend bease it's going to slam nodepool hard
20:00:55 <fungi> ianw: any feel for how many of those patches are actual functional changes vs how many are just readmes and other documentation?
20:00:58 <dhellmann> anteaya : yep
20:01:03 <anteaya> dhellmann: wow
20:01:03 <corvus> we could consider rolling it up into the rename changes
20:01:23 <mordred> corvus: we could do that for sure
20:01:27 <dhellmann> corvus : I like that. Make it a "fix git things after migration" patch
20:01:28 <corvus> merge .gitreview, zuul.yaml, and git/https all at once during the outage.
20:01:31 <fungi> oh, also we're over time (though i don't know if anyone else is waiting to start a meeting in here)
20:01:40 <ianw> fungi: this initial list is for anything with a devstack plugin.  i feel like they're all very minimally "functional" as in part of code that will execute, but very low risk
20:01:41 <clarkb> fungi: there is no meeting that follows us
20:02:05 <mordred> corvus: although if we do the git:// removal first, we coudl actually roll out the git.openstack.org redirect before doing the larger cutover, since the redirects will keep stuff working
20:02:15 <dhellmann> anteaya : look through some of the abandoned python3-first patches from this past cycle if you want examples
20:02:32 <mordred> corvus: I'm not sure we want to do that - but it would be open to us once git:// is gone
20:02:42 <cmurphy> clarkb: do we care enough about the cgit servers to bother upgrading their puppet?
20:02:43 <anteaya> dhellmann: happy to take your word for it, and I thank you for the gerrit topic pointer
20:02:45 <ianw> personally i feel like doing this first, then taking on a smaller mop-up of any remaining issues would be easier
20:02:53 <mordred> cmurphy: I do not think so, no
20:02:57 <clarkb> cmurphy: at this point given the 4/19 date probably not
20:03:03 <cmurphy> okie
20:03:13 <fungi> yeah, the fact that the git://->https:// changes *should* be able to happen at any time between now and the maintenance makes it less critical that we roll it into the git-review and zuul configuration patch scripting, though it's certainly worth debating
20:03:20 <cmurphy> those were the only centos ones then i think so that makes it easy
20:03:26 <corvus> mordred: that's true... though it may make the migration more difficult... worth thinking about.
20:03:30 <mordred> corvus: ++
20:03:36 <clarkb> corvus: mordred ianw if ianw pushes teh change then self approves them that reduces the things we have to do
20:03:39 <clarkb> ya what fungi said
20:04:28 <mordred> biggest issue with self-approve will be tracking down/rechking phantom failures
20:04:40 <clarkb> mordred: unless we also submit
20:05:15 <corvus> if we did it along with the renames, we could do "git://git.openstack.org" -> "https://opendev.org" in one pass.
20:05:19 <mordred> clarkb: if we also submit, will anything cancel the now-not-needed zuul jobs in flight?
20:05:27 <corvus> let me clarify that
20:05:51 <dhellmann> as far as tooling goes, be careful linking that many patches to 1 story in storyboard (the UI can't cope and starts timing out), and the stuff I build for python3-first was all pretty custom for that work but it's in the goal-tools git repo if you want to look at it
20:06:01 <corvus> because we could technically do that now, but we could do "git://git.openstack.org/openstack/unofficial-project" -> "https://opendev.org/foobar/unofficial-project" during the migration
20:06:25 <mordred> corvus: that is true and a good point
20:06:58 <clarkb> ianw: is it relatively easy to generate the todo list?
20:06:59 <corvus> dhellmann: oh good point, thanks.  if we do push patches, we could send a ML post and link the commit message to that instead <-- ianw
20:07:09 <clarkb> we'll need to generate that list just before the migration if we do it all in one go then
20:07:44 <corvus> clarkb: i bet if we re-do the list a few days before the migration, that'll be enough
20:07:47 <ianw> clarkb: it's all scripts, i can certainly beat them into a more generic shape, currently i hand run it
20:08:00 <clarkb> ianw: as long as it is repeatable in advance should be fine
20:08:05 <corvus> clarkb: we can freeze new repo creation a couple days before migration too, so we don't have any surprise new devstack plugin repos show up
20:08:18 <clarkb> given corvus' point I think there is value in doing it in the migration
20:08:29 <clarkb> so if want to do it then wfm
20:08:48 <fungi> concern with doing more than protocol changes is we have some fairly complex rules for mapping urls from cgit and git http backend, which is happening on the old vhosts under the present plan
20:09:26 <fungi> so if we do that in the patch we need to take care to perform the complete set of transformations because we can't rely on gitea for all of them once the requests no longer hit the redirect site for git.openstack.org
20:09:32 <clarkb> fungi: I think most of that complexity is to capture the difference in cgit and gitea parameters
20:09:35 <mordred> fungi: yeah. I don't thin we should try to auto-change any cgit links
20:09:36 <dhellmann> corvus : +1 for a mailing list post instead
20:09:37 <ianw> i think the protocol changes are pretty straight forward.  i think if we URL renames, that should probably at least be in a separate step
20:09:45 <corvus> fungi: correct -- but we can and should go directly to the end state.
20:09:50 <fungi> clarkb: and also the cgit path prefix
20:10:04 <clarkb> fungi: oh I thought we were handling that with project renames in gitea
20:10:17 <clarkb> (corvus' got that patched merged upstream to make that work)
20:10:27 <fungi> oh, opendev.org/cgit/... works?
20:10:32 <fungi> i didn't even think to try that
20:10:35 <clarkb> fungi: oh that prefix
20:10:36 <mordred> yeah - cgit rewrite rules will make the old cgit urls work - and renames in gitea will make renames work
20:10:45 <clarkb> sorry I thought you mean openstack/zuul -> zuul/zuul
20:10:56 <corvus> were there ever any "git://git.openstack.org/cgit" links?
20:11:00 <clarkb> corvus: no
20:11:02 <mordred> no
20:11:05 <fungi> great point
20:11:48 <fungi> which brings up the other question... is there a huge benefit to rewriting the domain part of git:// urls in repository content when we're not doing that for other protocols?
20:12:32 <fungi> feels like it just increases the set of things we might possibly break in a scripted transition
20:12:46 <corvus> well, anything that uses git:// is going to break with the transition
20:13:08 <fungi> not what i was asking, but i agree that's true
20:13:08 <mordred> not a technical one - the system should handle all the https:// urls already. BUT - if we're going to make a mass change like that - it might be nicer to do the mass change to a correct final state
20:13:24 <corvus> that's probably the best argument for doing something ahead of time.  it lets the change propagate to downstream users before we turn it off.
20:13:40 <mordred> so the smallest possible working change is git:// to https:// and changing nothing else
20:13:41 <corvus> fungi: oh, sorry, i grok your question
20:14:09 <clarkb> As a time check we are almost 15 minutes over now and I can smell my lunch :) this may warrant a mailing list thread?
20:14:29 <clarkb> sounds like there are enough moving parts with enough options that writing them down so that we can talk about them distinctly may help?
20:14:39 <corvus> wfm
20:14:39 <ianw> clarkb: sure, i think i've got enough to get that together, thanks
20:14:56 <mordred> https://review.openstack.org/#/c/642652/ is an example change from ianw's script btw
20:15:00 <ianw> tldr; i will *not* press enter and submit the changes right now :)
20:15:03 <clarkb> great, I think it will help me to see them listed as individual items
20:15:09 <clarkb> thank you everyone
20:15:17 <clarkb> #endmeeting