16:00:11 <smcginnis> #startmeeting releaseteam
16:00:12 <openstack> Meeting started Thu Mar 21 16:00:11 2019 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:15 <openstack> The meeting name has been set to 'releaseteam'
16:00:18 <smcginnis> Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong
16:00:24 <hberaud> o.
16:00:27 <hberaud> o/
16:00:27 <smcginnis> #link https://etherpad.openstack.org/p/stein-relmgt-tracking Agenda
16:00:28 <evrardjp> o/
16:00:32 <dhellmann> o/
16:00:40 <fungi> aloha
16:00:43 <ricolin> o/
16:00:46 <smcginnis> Way down on R-3.
16:00:49 <smcginnis> Man time flies.
16:00:58 <evrardjp> yeah
16:01:00 <evrardjp> very bottom
16:01:07 <ttx> o/
16:01:23 <ttx> woohoo
16:01:24 <smcginnis> #topic Review missing RC1 PTL feedback
16:01:42 <smcginnis> #link  http://tiny.cc/a1mkwy
16:02:02 <ttx> We've had decent participation, I feel like posting "by default" helped
16:02:03 <smcginnis> I think we've gotten quite a few PTL acks, but a lot without yet.
16:02:23 * ttx counts
16:02:28 <diablo_rojo> O/
16:03:13 <ttx> about 20 without -1 or +1
16:03:30 <ttx> out of...
16:03:32 <smcginnis> No so bad really.
16:04:04 <ttx> 66
16:04:23 <ttx> For Thursday it's not bad at all
16:04:43 <smcginnis> I think we can process the ones that have had an ack so far.
16:04:48 <ttx> I guess the uestion is when would we force-approve
16:04:51 <ttx> tomorrow?
16:05:05 <smcginnis> And either wait until way late tonight, or maybe even tomorrow morning to do the ones that haven't had any response.
16:05:08 <ttx> The ones I +2ed should be ok to merge
16:05:08 <fungi> we usually release stuff on thursdays, right?
16:05:17 <ttx> fungi: RCs are ok any time
16:05:23 <smcginnis> Yeah, and Thursday is the stated deadline.
16:05:38 <ttx> Friday is a way to do EOD Thursday
16:05:41 <fungi> ahh, yeah and i guess if today is the deadline then waiting until tomorrow is fine to consider those have missed it
16:05:42 <smcginnis> But we've been a little lenient the last couple cycles I think.
16:06:13 <dhellmann> I think our past goal has been no later than monday?
16:06:21 <smcginnis> We always have had a couple Friday morning from folks not paying enough attention to realize that Thursday is the deadline and not the end of RC1 week.
16:06:27 <smcginnis> dhellmann: ++
16:06:32 <dhellmann> we used to do them "after the release meeting on friday" but that's on thursday now
16:06:41 <ttx> smcginnis: how about sending an email today to the list highlighting the ones still needing feedback
16:06:45 <smcginnis> What do folks think, tomorrow or wait until Monday?
16:06:48 <dhellmann> so maybe 24 hrs from now?
16:06:49 <ttx> last chance before forcemerge
16:06:53 <ttx> dhellmann: ++
16:07:18 <smcginnis> ttx: That's a good idea. I'll mention in the countdown email, then send a separate full list of projects.
16:07:37 <smcginnis> #action smcginnis to send ML post for projects with no ack
16:07:42 <ttx> Since it's the first time we do it, i wonder if a specific email warning that we'll do it would be preferable
16:08:05 <efried> o/ question on that
16:08:06 <ttx> I mean, both is good
16:08:15 <ttx> efried: hello!
16:08:15 <efried> When stuff is ready, do I update the hash in the patch or do you?
16:08:26 <dhellmann> efried : you should do it
16:08:29 <smcginnis> efried: That's up to the projects to do.
16:08:34 <efried> ack, thx
16:08:43 <ttx> we just propose a default in case nobody talks to us
16:08:52 <dhellmann> the proposed patches are the "it's better than nothing" version
16:09:11 * ttx fetches an example
16:09:18 <efried> It's cool, I understand.
16:09:19 <efried> thanks
16:09:30 <smcginnis> We can also do some spot checks before approving the un-ack'd releases to see if anything has merged since we generated them. Though that could take some time to do.
16:09:36 <ttx> https://review.openstack.org/#/c/644317/
16:09:54 <ttx> That's exactly how it should work (thanks cmurphy) ^
16:10:08 <efried> ack
16:10:34 <dhellmann> smcginnis : that feels scriptable
16:10:47 <smcginnis> Yeah, I suppose it could be.
16:11:02 <smcginnis> Or at least run our unreleased commit script to have an easy reference.
16:11:03 <dhellmann> git review -d $patch && list_unreleased_changes.sh $deliverable
16:11:24 <smcginnis> Oh yeah, that should do it.
16:11:30 <evrardjp> oh yeah
16:12:03 <ttx> next topic?
16:12:14 <smcginnis> Yep, I think we have a plan on this one.
16:12:25 <smcginnis> #topic Post-deadline tasks
16:12:29 <dhellmann> hmm, that might not work, because the unreleased script looks at actual tags
16:12:47 <ttx> yeah so I wanted to discuss when to start post-deadline tasks
16:12:52 <smcginnis> Maybe we could tweak the script.
16:12:56 <ttx> Friday is not an awesome day to land those
16:13:11 <smcginnis> Maybe Monday after we approve the remaining RC's?
16:13:13 <ttx> Maybe prepare them on Friday, land them on Monday?
16:13:17 <smcginnis> ++
16:13:31 <ttx> smcginnis: I thought we would approve remaining RCs on Friday
16:13:48 <smcginnis> We state "minimum set of projects" for the devstack branching, but do we have that set defined?
16:14:02 <dhellmann> I think it's whatever is in the integrated-gate
16:14:03 <ttx> It's the one that are devstack-integrated
16:14:09 <ttx> ones*
16:14:26 <smcginnis> ttx: OK, I asked what folks thought but don't think we really said. Friday is good. I like sticking closer to the stated deadline.
16:14:41 <ttx> I mean, it's not the end of the world really
16:14:42 <dhellmann> +1 for friday
16:14:50 <ttx> They can ask for an RC2
16:14:55 <smcginnis> Yeah
16:15:03 <ttx> worst case scenario the RC1 is unusable
16:15:17 <ttx> But that would be indicative of a deeper problem
16:15:18 <smcginnis> Probably wouldn't be the first time.
16:15:42 <smcginnis> We do kind of miss the visibility that missing milestones gave us.
16:16:21 <smcginnis> I think gmann has taken care of the devstack branching in the past.
16:16:23 <ttx> There are a bunch of comments on those post-deadline tasks
16:16:27 <smcginnis> Grenade too?
16:16:53 <ttx> "IS THIS STILL RIGHT?: "
16:17:11 <smcginnis> I think we had the same question in Rocky, so we should get some clarification on that.
16:17:18 <smcginnis> Really a QA task I think.
16:18:01 <gmann> smcginnis: you mean after branch cut work right ?
16:18:22 <smcginnis> gmann: Yeah, we are looking at the Post-deadline Tasks on line 464 here: https://etherpad.openstack.org/p/stein-relmgt-tracking
16:18:25 <ttx> https://opendev.org/openstack-infra/devstack-gate/src/branch/master/devstack-vm-gate-wrap.sh#L299
16:18:31 <ttx> Looks like that task is still relevant
16:18:50 <smcginnis> gmann: First 4 (?) tasks I think have usually been done by the QA team, IIRC.
16:19:54 <gmann> ttx: smcginnis yes
16:19:55 <dhellmann> ttx: yeah, I just kept hearing that devstack-gate was deprecated so I wasn't sure. it looks like we need that change.
16:20:19 <gmann> dhellmann: ttx yeah we still need that change in d-g as grenade still depend on that.
16:20:42 <gmann> those bits has not separated out yet
16:20:46 <smcginnis> gmann: OK, do you have those tasks covered? I think we have it listed in the release process guide just to make sure all the necessary things get done.
16:21:57 <gmann> smcginnis: not yet. we are at L464 ? i mean all projects branch already cut ?
16:22:19 <ttx> Looks like all the "IS THIS STILL RIGHT" is still right
16:22:38 <dhellmann> gmann : we have not cut all of the branches, but we want to make sure we have these other patches lined up and ready for when we do cut them
16:22:39 <gmann> yeah
16:22:40 <ttx> I ceommented on teh etherpad pointing to segments that seem to be branch-sensitive and need updating
16:22:42 <smcginnis> gmann: Right, not yet. But asking if you have those tasks or if anything is needed from the release team to follow through.
16:22:46 <dhellmann> ttx: ++
16:23:36 <gmann> smcginnis: L465 will be release team right? i have not done that in past
16:24:21 * ttx tries to find what L465 aligns with with myopic etherpad alignment
16:24:33 <smcginnis> gmann: That's this, right? https://review.openstack.org/#/c/591563/
16:25:00 <smcginnis> Or am I looking at the wrong thing?
16:25:15 <gmann> smcginnis: ah sorry i read it wrongly
16:25:28 <gmann> smcginnis: yeah i am on those.
16:26:25 <smcginnis> gmann: OK, thanks!
16:26:25 <gmann> i see March 22 as deadline for Post-deadline Tasks ?
16:26:55 <ttx> gmann: to prepare patches yes
16:27:10 <gmann> ttx: cool, I will do it tonight.
16:27:13 <ttx> We'd rather not break everything on Friday, so will likely approve them on Monday
16:27:29 <gmann> ok
16:27:43 <fungi> that does seem safer
16:27:52 <smcginnis> ++
16:28:00 <openstackgerrit> Matt Riedemann proposed openstack/releases master: nova: release rocky 18.2.0  https://review.openstack.org/645227
16:28:01 <mriedem> efried: ^
16:28:08 <smcginnis> Do we need to reach out to Frank for the translation branching?
16:28:40 <smcginnis> That's another one that seemed like the team just knew to do in the past.
16:29:09 <fungi> or ianychoi now
16:29:27 <smcginnis> Depends when you consider the hand over.
16:29:32 <fungi> yep
16:29:35 <smcginnis> But yeah, either. :)
16:29:51 * ttx dusts off his "release steward" proposal from 2016
16:30:11 <ttx> the infamous overlapping PTLs proposal
16:30:15 <smcginnis> We can ping prometheanfire on the release branching too (see what I did there0
16:30:16 * evrardjp has no idea what ttx means
16:30:40 <evrardjp> ttx: you can fill me in after the meeting :D
16:30:52 <smcginnis> evrardjp: There's been a lack of clarity or differing assumptions on whether the newly elected PTL takes over as soon as the election is done or on the cycle transition.
16:31:06 <evrardjp> I mean would you mind explaining this proposal to me later?
16:31:15 <evrardjp> oh
16:31:16 <evrardjp> that
16:31:24 <efried> IMO previous PTL should continue to do things related to the old release, new PTL should do things related to the new release.
16:31:33 <ttx> evrardjp: just an idea I floated some time ago that we should ahve people caring for a release before, during and after development cycle
16:31:38 <efried> In cases where the distinction is not clear, new PTL takes over as soon as election ends
16:31:40 <smcginnis> efried: That's always been my take.
16:31:54 <efried> in cases where race is uncontested, arguably when noms close
16:32:04 <efried> or by agreement between outgoing and incoming
16:32:16 <evrardjp> that's what I've been doing in the past -- worked fine for me
16:32:19 <smcginnis> Except when the previous PTL just disappears to a hippy fest and doesn't tell the new PTL that he has to do everything to wrap up the release for the first time ever.
16:32:22 <smcginnis> :)
16:32:25 <ttx> That was back when Forums happen in the middle of the cycle -- the same person would handle that, then the PTG and development cycle, then stable branch releases
16:32:31 <evrardjp> smcginnis: haha that happened to me too. :D
16:32:39 <smcginnis> Anyway, back on topic. ;)
16:32:42 <efried> yeah, smcginnis that's people being responsible citizens
16:32:44 <ttx> people were not that happy with the years-long commitment
16:32:57 <prometheanfire> ohai
16:33:24 <smcginnis> prometheanfire: We're going through the post-RC1 tasks. One of them is the requirements branching and lifting of the freeze.
16:33:32 <smcginnis> prometheanfire: Assuming you've got that all covered.
16:33:46 <smcginnis> prometheanfire: Even though your freeze meme game was sadly lacking this time around. :P
16:33:51 <prometheanfire> ya, if we are branched, have a script to run for that (from releases repo)
16:34:04 <evrardjp> if?
16:34:11 <dhellmann> a script?
16:34:21 <smcginnis> prometheanfire: We are going to process the last of the RC1 and branching tomorrow, so by Monday we should be good to go.
16:34:26 <evrardjp> a wild confusion appears!
16:34:44 <prometheanfire> smcginnis: kk
16:34:53 <smcginnis> prometheanfire: Thanks
16:35:20 <smcginnis> Last thing there is the addition of the new branch to openstack-zuul-jobs.
16:35:37 <smcginnis> I can take that one unless someone else wants it. (?)
16:36:04 <fungi> happy to review it, just ping me
16:36:47 <smcginnis> fungi: Will do, thanks
16:36:59 <smcginnis> That's it for the list of remaining tasks.
16:37:08 <smcginnis> Anything else there?
16:37:20 <ttx> We also have next week tasks!
16:37:34 <ttx> ah that was what you just discussed
16:37:34 <dhellmann> it's a shame some of these tools don't look at the series list to determine which branches are valid. maybe we could work on that during train, to cut the number of steps in this process
16:37:42 <smcginnis> #link https://releases.openstack.org/reference/process.html#between-rc1-and-final
16:37:54 <smcginnis> dhellmann: That's a good idea.
16:38:13 <dhellmann> the devstack-gate scripts in particular seem like they could be candidates, if we produced a CLI they could call
16:38:29 * ttx copy pastes tasks
16:40:16 <smcginnis> Added a few of those to the countdown email since they were for that.
16:43:26 <smcginnis> Two big tasks there are probably making sure all stable/stein branches are created and adding release note links.
16:44:05 <ttx> It's really almost all finalrc week tasks
16:44:13 <smcginnis> Yeah
16:44:29 <smcginnis> ttx: I think it looks good how you've moved things around.
16:44:44 <smcginnis> We could maybe update process.rst, but probably not a bad thing to review them ahead of time like this anyway.
16:44:45 <ttx> Did we discuss "Force missing cycle-with-intermediary releases" ?
16:45:20 <smcginnis> No, not really
16:46:02 <smcginnis> evrardjp or diablo_rojo: Want to take that for next week?
16:46:41 * diablo_rojo catches up on scrollback having just been finishing another simultanous meeting
16:46:50 * diablo_rojo not sure what she's signing up for but..
16:46:53 <diablo_rojo> Sure :)
16:46:57 <smcginnis> Hah
16:47:28 <smcginnis> diablo_rojo: Just more generating release patches. Any cycle-with-intermediary deliverables that have had commits merged but no release done yet.
16:47:47 <smcginnis> Well, I guess that would be only "other" and "service" types, right?
16:48:01 <smcginnis> We don't want to do more lib or client lib releases past the deadline.
16:48:03 <diablo_rojo> smcginnis, so basically the same as I did Monday but with cycle with intermediary
16:48:05 <evrardjp> we'll take it and come beg for comments as usual :)
16:48:22 <evrardjp> :D
16:48:57 <smcginnis> diablo_rojo: Basically, but they are only needed if they've committed anything since the last release.
16:49:05 <dhellmann> it looks like most of the unreleased things are tempest plugins
16:49:37 <smcginnis> Oh right, we do need to make sure all the tempest plugins get released and branched.
16:49:44 <dhellmann> evrardjp , diablo_rojo : I hope you're making notes about how to improve the details
16:49:51 <smcginnis> :)
16:50:09 * diablo_rojo knows where to find the meeting logs links :)
16:50:11 <evrardjp> dhellmann: We are opening this meeting log every time we do it.
16:50:14 <dhellmann> smcginnis : do we branch the tempest plugins?
16:50:15 <evrardjp> diablo_rojo: :)
16:50:19 <diablo_rojo> lol
16:50:25 <dhellmann> evrardjp : we should get those improvements into the process doc
16:50:32 <evrardjp> indeed
16:50:53 <smcginnis> diablo_rojo: Yeah, we started last cycle just mainly for the purpose of knowing which plugins are expected to work with which version of tempest.
16:50:56 <dhellmann> note that is the democratic "we" not the royal "we" (so "you")
16:51:08 <diablo_rojo> smcginnis, that makes sense
16:51:29 <evrardjp> dhellmann: :)
16:51:32 <smcginnis> OK, I'll move on then.
16:51:35 <smcginnis> #topic Ensuring stable-maint-core is in all official projects
16:51:52 <smcginnis> We have official projects that do not have stable-main-core in the ACL.
16:52:02 <ttx> ah?
16:52:02 <smcginnis> ttx: Is that something that could feasibly be added to the acl checker?
16:52:21 <ttx> Maybe? I'll have to look
16:52:37 <ttx> do you have one example of missing?
16:52:43 <smcginnis> OK, I just wanted to raise it. Not super important. Most have been like this for a long time.
16:52:50 <smcginnis> neutron-lib was one I noticed.
16:53:08 <ttx> How about adding that to the list of things to improve, if it's not urgent
16:53:19 <dhellmann> if we work out how to do that, we could add it to the branch validation rules in the releases repo
16:53:23 <smcginnis> Oh, down at the bottom?
16:53:28 <ttx> yes
16:53:55 <ttx> ++
16:53:57 <smcginnis> Done
16:54:09 <ttx> that will basically be our PTG agenda
16:54:28 <smcginnis> Last thing quick, though not necessarily have to cover it in the meeting.
16:54:31 <smcginnis> #topic Placement idea to use cycle-with-intermediary
16:54:44 <ttx> can you summarize the idea?
16:54:47 <evrardjp> oh
16:54:49 <smcginnis> Chris had some questions about releasing per microversion bump in placement.
16:54:58 <smcginnis> Since it's basically all an API.
16:55:17 <smcginnis> So if they bump microversion 1.35, release version 1.35.
16:55:29 <ttx> IIRC there was some difference between API microversioning and semver
16:55:32 <cdent> oh hi. it's a very speculative idea at this point, but wondered about the feasability
16:55:47 <dhellmann> they can release as often as they want, but to start every cycle they have to tag a new minor version at least in order to provide space for bug fixes, so things may come out of alignment quickly
16:55:48 <ttx> like x.y vs. x.y.z
16:56:19 <ttx> I guess it could be 1.35.0 for microversion 1.35
16:56:23 <smcginnis> Only drawbacks I could see are 1) if anything backward incompatible they would need to do a major bump, and 2, we currently require c-w-i at least every milestone and it's possible development stabilizes enough not to need a microversion bump each milestone period.
16:56:23 <cdent> ttx, if (and it is a big if) we did this: microversion 1.35 would be 1.35.0
16:56:29 <cdent> jinx
16:56:33 <ttx> if really all features are reflected in the API
16:56:37 <dhellmann> yeah, the problem isn't within the cycle, it's at the cycle boundaries
16:56:40 <smcginnis> And then still room for bugfixes.
16:57:33 <ttx> One issue that was raised in the past is that in some cases we twist the rule for the release number, but keep the microoversion honest. Imagine this scenario
16:57:40 <cdent> it sounds like more trouble than it is worht
16:57:44 <ttx> We are in the middle of U
16:57:52 <ttx> A security issue is found in T
16:58:14 <smcginnis> I should have stated, it was just an idea cdent was pondering, so I wanted to bring it up for issues I wasn't thinking of with doing it.
16:58:35 <ttx> hmm beh
16:58:39 <ttx> lost my thought
16:58:46 <smcginnis> Stable backport?
16:59:14 <ttx> ISTR there was this case where you would bump major API version but only bump .z in version number
16:59:30 <ttx> like the rules were just different
16:59:37 <cdent> I think the only way it would work, in any real way, would be if we stopped doing stable branches, which is possible in a microversioned world since they are opt in. If you want a security fix, you must upgrade, but you don't have to use the new features. In the context of the way openstack projects normally work that would be a non-starter...
17:00:06 <smcginnis> Oh well, it was an idea worth exploring.
17:00:32 <cdent> I reckon there are multiple ideas to take from it
17:00:42 <cdent> but not something that is immediately actionable
17:00:56 <dhellmann> dropping stable branches would break the ability for downstream consumers to deal with fixes without also breaking co-installability as dependency versions increase
17:01:18 <smcginnis> I'm going to end the meeting since we're over the time, but we can continue discussing.
17:01:21 <smcginnis> Thanks all!
17:01:23 <smcginnis> #endmeeting