16:18:36 <smcginnis> #startmeeting releaseteam
16:18:37 <openstack> Meeting started Thu Jun 11 16:18:36 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:18:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:18:40 <openstack> The meeting name has been set to 'releaseteam'
16:19:03 <smcginnis> Reminder to folks to add you nicks to the ping list in the new etherpad if you want that.
16:19:08 <smcginnis> #link https://etherpad.opendev.org/p/victoria-relmgt-tracking Agenda
16:21:04 <smcginnis> Well, I guess it's just us today. Probably not surprising since we haven't had meetings for a few weeks.
16:21:14 <smcginnis> #topic Review task completion
16:21:29 <smcginnis> Taks for this week was to get the list of cycle-trailing deliverables for ussuri.
16:21:52 <smcginnis> That probably needs to be updated in our process doc. I think it was this week before we pushed out the deadline for trailing projects.
16:22:08 <smcginnis> But I did run that and sent a post to the ML reminding those teams that it will need to be done.
16:22:24 <ttx> I think a reminder is always good
16:22:34 <ttx> well ahead of the deadline
16:22:44 <hberaud> o/
16:22:46 <smcginnis> Doesn't hurt. I think it did trigger at least one team to do a release.
16:23:02 <ttx> I was wondering about all those stable/train autoreleases -- why did we do them? is that something we should make part of the process?
16:23:15 <smcginnis> I was planning on bringing that up.
16:23:22 <smcginnis> I did that partly with my stable hat on.
16:23:31 <smcginnis> Partly as a way to test out some automation I was working on.
16:23:47 <ttx> (personally I think it's a slippery slope to start doing stable autoreleases... will add a lot of work for us?)
16:24:02 <smcginnis> I do think if the automation does help (it's pretty close) then we probably should add that to our process to do those at at least a couple times during the cycle.
16:24:22 <smcginnis> When we get around to the EM transition date, it's always this mad rush by some teams to quick get things backported while they can.
16:24:24 <ttx> ok, if we can get the cost near zero
16:24:37 <smcginnis> Rather than doing that regularly over the course of the stable branch lifecycle.
16:24:54 <ttx> What's costly generally is to handle the ones who do not get +1
16:24:58 <smcginnis> And there are a few teams that are pretty good about backporting bugfixes, but no so good about remembering to actually release those fixes.
16:25:10 <fungi> released stable branches aren't really cycle-oriented, though i guess the choice to tie timing of autoreleases to the cycle would be because the release team's activity schedule *is* cycle-oriented?
16:25:14 <ttx> so maybe we should decide what to do with those ahead of time
16:25:35 <smcginnis> ttx: Yeah. Since this isn't an official thing, I was thinking we could jsut abandon those unacknowledged ones if we don't get a response.
16:25:36 <ttx> fungi: it would not be on the release schedule, just in the team process timeline
16:25:45 <fungi> right, i figured
16:25:53 <smcginnis> I'm not going to go too far out of my way to help these teams that aren't paing any attention to stable maintenance.
16:26:07 <smcginnis> fungi: Yep.
16:26:11 <fungi> but since the team's timeline is arranged around the cycle, that's why you'd do stable autoreleases "a couple times a cycle"
16:26:12 <ttx> ok, as long as there is no chasing or endless meeting discussions to discuss what to do with the -1s or leftovers, I'm good :)
16:26:17 <smcginnis> I thought it would be a good quieter week to get something like that out.
16:26:43 <smcginnis> I'll go through and abandon any that get no response next week.
16:26:45 <ttx> I just don;t want to get the impression that they need to wait for us to propose it to get a stable release out
16:26:51 <smcginnis> True
16:27:03 <ttx> but I guess that can be worded in the commit message
16:27:04 <smcginnis> Maybe next time I should add more clarification to the commit message.
16:27:09 <ttx> +1
16:27:28 <smcginnis> Anyway, based on the responses so far, I think it was a good reminder for some teams.
16:27:49 <fungi> "since the team hasn't proposed a stable release in $x days and there are $y changes merged since then, the release team is proposing this point release on your behalf..."
16:28:41 <fungi> something which reminds them they really should have been proposing releases sooner on their own
16:28:46 <smcginnis> Probably even more clear that it's not something that's part of the release process, just something we're doing (or even just I'm doing) to help make sure our stable branches are in good shape and ready to be retired when the time comes.
16:29:26 <ttx> yeah, maybe not say "release team"
16:29:26 <smcginnis> Part of this is that when we've done the last couple of EM transitions, there were a few deliverables that had a TON of merged commits that never were released. Going back 1+ years.
16:29:32 <smcginnis> ++
16:29:44 <ttx> ok next
16:29:49 <smcginnis> I'll try to be more clear next time.
16:29:56 <smcginnis> topic Review next week's tasks
16:30:11 <smcginnis> We have the c-w-i lib releases.
16:30:24 <ttx> woohoo
16:30:25 <smcginnis> That ties in well to testing out this scripting I've been doing, so I'd like to take that one.
16:30:37 <ttx> permission granted
16:30:40 <hberaud> :)
16:30:42 <smcginnis> :)
16:30:49 <smcginnis> Next, ACL issues.
16:30:56 <ttx> I can take the acl unless someone wants it
16:31:11 <hberaud> not sure what's behind
16:31:19 <smcginnis> ttx: I think you've become the de facto ACL man. :)
16:31:30 <smcginnis> Maybe a chance to walk through it with hberaud?
16:31:34 <ttx> I have a few other consistency work I've been pushing
16:31:36 <hberaud> +1
16:31:40 <ttx> sure
16:32:02 <smcginnis> And I'll send the countdown email.
16:32:18 <smcginnis> #topic One release model to bind them all - open questions
16:32:29 <ttx> right, so the discussion opened a couple of questions
16:32:35 <ttx> first one is...
16:32:42 <ttx> Can we support stable branches from arbitrary SHAs (to support the Kolla use case, but generally do give more flexibility as to when the stabilization work starts)
16:33:02 <ttx> Currently we enforce that the branch point is at a tag, for most deliverables
16:33:14 <ttx> I'm.. not sure where that came from
16:33:17 <smcginnis> Our validation prevents that right now, but we can change it. We do that for non-cycle based things.
16:33:28 <ttx> historically we did manually create branches from arbitrary commits, on demand
16:33:52 <smcginnis> We allow it for things like feature branches, but enforce that tagging with stable branches.
16:33:54 <ttx> that allowed stable work to start before the RC release itself
16:34:02 <smcginnis> Not really sure the background on that either.
16:34:24 <ttx> well feature branches should obviously not require to start from a tag
16:34:38 <smcginnis> We could certainly change that to create an RC branch if we want to have the stability work separated from mainline work as we lead up to final releases.
16:34:47 <fungi> not having a tag there does make it harder for downstream consumers to try out the candidate though
16:34:50 <ttx> OK, if that does not ring a bell, I'll try to find where that requirement came from
16:35:09 <ttx> it might just be taht it started that way and was cargoculted
16:35:13 <smcginnis> Pretty sure it was from before my time, so that may take some digging.
16:35:18 <smcginnis> Could be.
16:35:43 <ttx> At first glance i can;t see why that would be necessary
16:36:01 <smcginnis> Release note generation?
16:36:25 <ttx> hmm
16:36:33 <fungi> for testing purposes, it may still be that you'd want to encourage most projects to branch from an initial tag anyway
16:36:50 <ttx> I'll ask dhellmann if that rings a bell for him
16:37:02 <fungi> and branching from an arbitrary commit in master would be an exception rather than commonplace
16:37:42 <ttx> anyway, allowing it now would give a bit more flexibility
16:38:10 <ttx> so you can start the stabilization work slightly ahead of when you think you have a great release candidate
16:39:07 <ttx> Most of the reaction on that thread seems to be around confusing marketing release numbers and semver numbers
16:39:18 <ttx> Like a release should be x.0.0!
16:39:21 <smcginnis> It may be reno. Thinking about that, the common tag between the two branches would be a point before the last cycle branched, so release notes may include duplicate changes in both N and N-1 notes.
16:40:08 <ttx> I'd say that finally letting go of marketing numbers is a good thing
16:40:34 <ttx> especially since half the openstack world has already moved on
16:41:03 <ttx> ok, second question...
16:41:05 <smcginnis> I'm a little concerned about the corner case I pointed out earlier today. Occasionally breaking changes have been merged between RCs.
16:41:25 <smcginnis> And two major version bumps within a short timeframe could be a little confusing.
16:41:29 <ttx> oh you mean respecting semver in prereleases
16:42:14 <ttx> release number inflation around pre-releases
16:42:24 <smcginnis> Yeah.
16:42:29 <ttx> It's another case of conflating marketing and semver numbers
16:42:37 <smcginnis> That's just semver.
16:42:44 <smcginnis> No marketing involved.
16:42:47 <fungi> the horizon ussuri example i gave on the ml thread was a fairly heavy exercise of all levels of semver over the course of a cycle, i don't think it created confusion for anyone
16:43:01 <ttx> right, but you;re afraid of the major number inflation because it "looks bad"
16:43:43 <ttx> Having 16.2.0 bumped to 17.0.0 days before final... looks bad?
16:44:07 <fungi> i suppose the concern is that people might be surprised if horizon had branched stable/ussuri at 18.3.0 but then tagged 19.0.0 on stable/ussuri before the ussuri release?
16:44:24 <smcginnis> Not just looks bad. That we are releasing a major release that could be bad. Someone could pick that up assuming it's something we intend for them to use. Then they see a new major release out, try to upgrade, and we break something that normally we would handle upgrading to better if we considered it a true release before.
16:44:43 <smcginnis> No more "quick, we have to fix that before the final release goes out" because every release is the final release.
16:44:46 <fungi> or might not realize 19.0.0 was the ussuri release and thought it was a master branch tag for victoria?
16:45:14 <ttx> I feel like it's already the case with all our libs though
16:45:24 <smcginnis> Regardless of branch/series/cycle/whatever.
16:45:33 <smcginnis> Yes, and we make sure we handle things right in libs.
16:45:34 <tosky> I believe (and then I will shut up about the topic) that having a 18.2.0->19.0.0 bump, leaving out of any release a 18.2.0 version which looks like a stable release but it's not, it is confusing
16:45:36 <tosky> to me
16:45:39 <tosky> and it's not about marketing
16:45:53 <tosky> that's it
16:46:12 <smcginnis> Less so in services because we've had this gray area window before final RC where we can make changes that may break things we determine we shouldn't have done in the first place.
16:47:16 <ttx> smcginnis: right... so RCs allows to have things that are "will be a release unless we find better"
16:47:27 <smcginnis> We can move on. It's just something I've seen happen that if we change how we do things, teams will need to make sure they are doing things appropriately when running into those kinds of situation.
16:47:28 <fungi> tosky: there was no stable release for horizon 17.x.x at all, did that cause confusion?
16:47:29 <ttx> without doing one just yet?
16:47:58 <fungi> (horizon train is 16.x.x, horizon ussuri is 18.x.x)
16:48:04 <ttx> how about... not doing a release until the very end.
16:48:18 <smcginnis> ttx: Will be a release but "oh crap, we messed up and didn't realize a kitten dies ever time this happens so we need to quick throw out what we had in there and do something different".
16:48:34 <ttx> And from a process persepctive, we would only require that a change proposing the final release is out there
16:48:41 <smcginnis> Having multiple major bumps in a release cycle is not the issue I'm pointing out.
16:48:51 <ttx> that allows to do ""release candidates"
16:49:05 <ttx> you just don;t tag one, you update the release patch
16:49:39 <ttx> so you have that patch up that tags nova 17.0.0 at sha foobar
16:49:41 <tosky> fungi: yes, it is for me; it seems I'm the only one
16:49:43 <fungi> smcginnis: the issue you're pointing out is having major bumps during the stabilization period, right?
16:49:45 <ttx> you keep it warm
16:49:58 <ttx> update it to another SHA if you have critical fixes
16:50:04 <smcginnis> fungi: No. Let's move on though, since I'm the only one that sees this issue.
16:50:22 <ttx> and we just push at at the very end... same result as RCs, withouthaving to maintain a separate model
16:50:24 <fungi> yeah, i guess i'm not grasping what you're trying to point out either
16:50:52 <ttx> I'll explain what I mean in a response to the thread
16:51:01 <ttx> no need to discuss it now
16:51:18 <ttx> so question 2 was
16:51:24 <ttx> Should we really support Rally being a part of OpenStack while being independent from the OpenStack release
16:51:30 <ttx> Rally is the exception
16:51:45 <ttx> the one service that is part of openstack but does not follow the cycle
16:51:56 <ttx> there used to be more of those, but it's the only one left
16:52:10 <ttx> I'm not sure we need to have that exception really
16:52:22 <ttx> You're either part of openstack or not
16:52:24 <smcginnis> Solum, storlets, cyborg, karbor, masakaru, monasca-events-api
16:52:30 <ttx> smcginnis: false
16:52:30 <smcginnis> kuyryr-libnetworl
16:52:37 <smcginnis> We have a few services under independent.
16:52:38 <ttx> those used to be independent, when they started
16:52:48 <ttx> before inclusion in first release
16:52:49 <fungi> is rally really still active? i don't ever see any discussions of it on the ml these days, or questions about it for that matter
16:52:55 <smcginnis> Ah, OK. Those are just the historical releases then.
16:52:57 <ttx> they all moved to cycle-based
16:53:09 <ttx> I checked, only Rally is left
16:53:21 <ttx> Kuryr too, but it's a library
16:53:39 <ttx> libraries still make sense as cycle-independent, if generally useful
16:53:48 <smcginnis> What is the concern having an independent service?
16:54:22 <ttx> 1/ The messaging is blurry -- it's part of openstack but not part of "openstack"
16:54:43 <ttx> 2/ we have to keep the possibility for services to pick that alternate model, just because one service uses it
16:54:54 <ttx> so, unnecessary complexity
16:55:48 <ttx> when there were 3 projects in that case I was like... maybe. But now there is only one left I'm like, you should either be a part of the openstack release or not
16:56:33 <smcginnis> Thinking about it - I don't think I've thought enough though to have a strong opinion either way.
16:56:50 <ttx> That would be a TC thing anyway... but was wondering if we wanted to raise it or not
16:57:22 <ttx> I guess we could work around it by calling it a testing tool rather than a service.
16:57:23 <smcginnis> Probably worth pointing it out to the TC to see what they think.
16:57:25 <dhellmann> o/
16:57:31 <smcginnis> dhellmann!
16:57:43 <dhellmann> hi, smcginnis :-)
16:57:54 <ttx> dhellmann: hi! TL;DR: do you remember why we enforce branches to be created at tag points?
16:58:06 <ttx> That limits flexibility on when stable branches start
16:58:42 <dhellmann> I think to start it was just because that's the way we'd always done it? The git history of that part of the release validation might help.
16:59:02 <dhellmann> I'm sure reno expects that, but we probably already had to work around it so it might be less of a hard requirement now
16:59:02 <ttx> dhellmann: historically the branches were manually created, so we gave Gerrit a SHA
16:59:11 <dhellmann> oh, hmm
16:59:44 <dhellmann> well then I'm not sure. maybe it had to do with making it easier for folks to enter data in the release repo?
16:59:52 <ttx> I remember the time when branching and RC1s were different deadlines
17:00:14 <ttx> dhellmann: could totally be an implementation detail yes
17:00:17 <smcginnis> Would we need to give reno a starting sha instead of tag to be able to generate release notes without including commits in both branches?
17:00:28 <ttx> Like we'd ask them to submit branching and RC1s at the same time
17:01:47 <dhellmann> smcginnis : yeah, that's what I would be worried about. the logic to find the base of the branch looks for a tag that's on both branches so if there isn't one it pulls in extra history
17:02:30 <ttx> also at one point we supported creating RCs for c-w-i deliverables,,, Like swift would do releases but do RCs for the final one. I wonder why we stopped allowing that
17:03:09 <dhellmann> I don't remember the swift team ever wanting to do that, so I'm not sure about that change either
17:03:13 <ttx> probably because that was confusing, but maybe as a single unified model that could work
17:03:26 <ttx> dhellmann: they did not want it, we forced them to do it
17:03:39 <ttx> so that they would fit in the coordinated release process
17:03:48 <ttx> and then we stopped
17:03:51 <dhellmann> maybe dropping it was all part of adding the release cycle-with-intermediary stuff then?
17:04:02 <ttx> yes that's what I think
17:04:20 <ttx> but maybe allowing RC tags for those who want it could be a possibility
17:04:29 <ttx> solving everyone's fears about it
17:04:42 <ttx> like a single model that allows both approaches
17:05:14 <ttx> ok, I'll continue brainstorming on the thread
17:05:19 <dhellmann> yeah, that might be a reasonable compromise. I wonder how many teams would use them optionally.
17:05:24 <fungi> i feel like most of the extreme objections were based on a misreading of the proposal anyway
17:05:51 <dhellmann> I got the sense that some people are thinking about the versions of individual projects vs. the final release version set of all of openstack
17:06:12 <ttx> smcginnis: we can move on :)
17:06:16 <smcginnis> It's a change in something we've been doing for years, so some people are going to be resistant to that anyway.
17:06:31 <smcginnis> #topic Countdown email
17:06:35 <ttx> dhellmann: yes... like zigo complaining about not being able to tell in the Debian page which one is the final at first glance
17:06:37 <smcginnis> #link https://etherpad.opendev.org/p/relmgmt-weekly-emails
17:06:46 <ttx> I think that's fair
17:06:47 <smcginnis> Please take a look and make any edits. I will send it tomorrow.
17:07:46 <hberaud> LGTM
17:07:48 <ttx> it /is/ difficult to see which component numbers are part of the coordinated release without looking at releases.o.o
17:07:59 <dhellmann> ttx: I wonder if branching at a tag may have been the way to avoid merging tags from one branch to another?
17:08:56 <fungi> but we used to merge the release tags back into master, not the rc tags
17:09:09 <ttx> maybe... In doubt it's probably simpler to ask that if you want to branch early, you need to tag a beta or RC
17:09:36 <dhellmann> right, but now we don't do either and at least we have that RC tag to be able to tell where the previous release forked off
17:09:48 <ttx> smcginnis: that looks like next week's message
17:10:12 <ttx> i.e. next week is victoria-1, not the week after
17:10:12 * dhellmann wanders off, but is only a ping away
17:10:18 <ttx> dhellmann: thanks!
17:10:24 <smcginnis> Thanks dhellmann
17:10:30 <smcginnis> ttx: You're right!
17:11:18 <ttx> I was like... wow, this week was victoria-1? Time flies! Hmm... wait a minute
17:11:22 <smcginnis> Hmm, I got the subject line right, just not the content.
17:12:37 <smcginnis> OK, fixed.
17:12:45 <smcginnis> I will review again before sending just to be safe. ;)
17:13:01 <smcginnis> I need to get going soon. Anything else for today?
17:14:02 <ttx> nope
17:14:06 <ttx> endmeeting!
17:14:07 <hberaud> nope
17:14:12 <smcginnis> Thanks everyone.
17:14:14 <smcginnis> #endmeeting