16:00:10 <smcginnis> #startmeeting releaseteam
16:00:11 <openstack> Meeting started Thu Jan  9 16:00:10 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:14 <openstack> The meeting name has been set to 'releaseteam'
16:00:16 <smcginnis> Ping list: ttx armstrong diablo_rojo, diablo_rojo_phon
16:00:24 <evrardjp> o/
16:00:24 <hberaud> o/
16:00:35 <diablo_rojo_phon> O/
16:00:49 <evrardjp> look who is back from holidays! me \o/
16:01:00 <armstrong> o/
16:01:15 <smcginnis> #link https://etherpad.openstack.org/p/ussuri-relmgt-tracking Agenda
16:02:04 <evrardjp> L225
16:02:09 <ttx> o/
16:02:14 * evrardjp guesses
16:02:25 <smcginnis> #topic Remaining ussuri-1 refresh
16:02:33 <smcginnis> #link https://review.opendev.org/#/c/698508/
16:03:35 <smcginnis> hberaud: Do you know if your concern has been addressed there? ^
16:04:02 * hberaud take a look
16:04:32 <smcginnis> Oh, sorry. Actual comment was from dtantsur
16:04:53 <dtantsur> what have I done? :)
16:04:53 <hberaud> yep the reno is still missing
16:05:16 <hberaud> dtantsur: ^^
16:05:25 <dtantsur> ah
16:05:27 <smcginnis> This is just milestone 1. What does everyone think? Should we just abandon this one and pick it up at milestone 2?
16:05:44 <dtantsur> mordred: ^^^
16:06:02 <hberaud> I think we can go ahead to catch error ASAP
16:06:26 <smcginnis> hberaud: You mean go ahead with getting this released now so we can catch errors sooner?
16:06:36 <hberaud> yes
16:06:48 <evrardjp> and have the next release have the reno?
16:06:53 <smcginnis> That makes sense to me. Finding out if there are issues with it sooner rather than later is better.
16:06:55 <evrardjp> that sounds okay to me
16:07:11 <hberaud> IMHO it's more safer
16:07:12 <smcginnis> It would be great if we can quick get a reno added, but not the end of the world.
16:07:23 <hberaud> smcginnis: sure
16:07:40 <smcginnis> I've removed my -1 on there. Would be great if the team can ack if they think it's good to go ahead with it now.
16:08:14 <dtantsur> I'm -0 on releasing anything so major without a reno..
16:08:32 <smcginnis> Is there one proposed yet?
16:08:37 <smcginnis> I haven't had a chance to look.
16:08:47 <evrardjp> dtantsur: that's why we hope to have a reno patch soon merged so we can bump the sha and release ? :D
16:08:56 <dtantsur> I haven't looked, I'm not following o-c-c closely
16:09:07 <dtantsur> I can prepare one if nobody else wants to
16:09:14 <smcginnis> That would be great!
16:09:15 <evrardjp> that would be awesome!
16:09:18 <evrardjp> smcginnis: haha
16:09:32 <smcginnis> Hopefully that can get through ASAP and we can get this released after the sha gets updated.
16:10:25 <smcginnis> Let's plan on that then. dtantsur, if you happen to remember, please let me know the patch adding the reno so I can review/watch it.
16:10:47 <smcginnis> #topic Validate countdown email content
16:10:55 <smcginnis> #link https://etherpad.openstack.org/p/relmgmt-weekly-emails
16:10:57 <evrardjp> or ping in this channel so other people can also vote
16:11:13 <smcginnis> ++
16:11:23 <smcginnis> Line 87 in the countdown etherpad.
16:11:28 * smcginnis reads
16:11:29 <evrardjp> I read mid-milestone2 can't find anything to say more. It looks good
16:12:24 <ttx> OK, I'll turn it into process tomorrow
16:12:40 <smcginnis> Yep, content looks good to me.
16:13:11 <smcginnis> #topic False positive tests
16:13:17 <dtantsur> smcginnis: https://review.opendev.org/#/c/701761/
16:13:20 <smcginnis> #link https://review.opendev.org/#/c/698845/1
16:13:22 <smcginnis> dtantsur: Thanks!
16:14:15 <smcginnis> I think we only have a check for all repos in later releases.
16:14:37 <smcginnis> So if one release contains, for example, three repos, then all subsequent releases need to contain all three.
16:15:00 <ttx> hmm
16:15:22 <ttx> does it sound like something we should add?
16:15:28 <ttx> It's a bit hard to spot
16:15:30 <smcginnis> Since some teams need to release some of their repos separately, I don't think we have any kind of reconciliation with the governance data.
16:15:58 <ttx> The gov/rel consistency check I'm workign on will catch it
16:16:10 <ttx> but it's a periodic check, not immediate
16:16:21 <smcginnis> How will that work if a team has some c-w-i deliverables and some c-w-rc?
16:17:54 <evrardjp> wasn't that a problem for OSA in some stable branches?
16:18:01 <ttx> those would be separate files right
16:18:11 <smcginnis> evrardjp: Maybe
16:18:31 <evrardjp> because some things were split out into different deliverables in governance
16:18:36 <smcginnis> ttx: Yes. So just looking at the governance data, how do we tell if something is missing or just not released separately?
16:18:42 <evrardjp> so inside a cycle, some have disappeared to move somewhere else
16:18:54 <ttx> governance defines deliverables
16:19:12 <evrardjp> some repo could have moved to a different deliverable :p
16:19:31 <smcginnis> Or deliverables added or removed mid-cycle.
16:19:41 <evrardjp> before m2 I guess but after m1 indeed
16:19:50 <ttx> Like governance says A contains a1 and a2... we could check that A.yaml contanis a1 and a2
16:19:52 <smcginnis> We can try it out. If it works in this consistency check, maybe we can extend that into our validate job.
16:20:01 <ttx> but yeah, stable branches would not work
16:20:05 <evrardjp> ttx: well governance is not stable branched
16:20:10 <evrardjp> :)
16:20:16 <evrardjp> you said it ! :p
16:20:24 <ttx> BUT stable used to be master once
16:20:40 <evrardjp> oh I see, you think it's fine forward
16:20:41 <ttx> and we have a check to check that later releases are ok
16:20:42 <smcginnis> I suppose we could start tagging governance. :)
16:20:58 <evrardjp> smcginnis: I thought of that multiple times tbh.
16:21:04 <evrardjp> would simplify releases...
16:21:05 <ttx> So if we just used governance to check that initial master releases are OK, it would propagate
16:21:32 <smcginnis> It's something at least.
16:21:34 <ttx> I can have a look at what it would take
16:21:39 <ttx> probably not that difficult
16:22:39 <ttx> ok next
16:23:03 <ttx> just not today (or this month really)
16:23:11 <smcginnis> Yeah...
16:23:15 <smcginnis> #topic cliff issue postmortem
16:23:23 <smcginnis> Worthy of its own topic. :)
16:23:35 <smcginnis> I haven't had a chance to try to triage what happened here.
16:23:36 <ttx> same thing... is that something we could/should have caught
16:23:48 <ttx> maybe we can postpone discussion then
16:23:49 <smcginnis> Somehow the automation picked up a very old commit for cliff.
16:23:58 <ttx> until somone has a look and can explain it
16:24:22 <ttx> it's not urgent... just whenever a probem slips through we should have a look and see if we could avoid it in the future
16:24:23 <smcginnis> For now at least, let's make sure we are extra careful reviewing those automated milestone patches.
16:24:50 <smcginnis> Definitely. I don't like not knowing how something like that happened when the script should be pulling the latest from HEAD.
16:24:51 <ttx> do you have a link?
16:24:59 <smcginnis> To the cliff release patch?
16:25:04 <ttx> yeah
16:25:14 <smcginnis> https://review.opendev.org/#/c/698485/
16:25:56 <evrardjp> is that the right link?
16:25:57 <smcginnis> Oddly, list-changes shows what changed between 2.11.0 and that.
16:26:04 <ttx> hmm we should definitely check the WHY before doing milestone2 autoreleases
16:26:07 <evrardjp> oh my bad
16:26:08 <smcginnis> When it should have been since 2.16.0
16:26:31 <ttx> who can have a look in the next 4 weeks?
16:26:41 <ttx> (hint: not me)
16:26:47 <smcginnis> It would have been a good clue looking at the "Release will NOT include" section.
16:26:51 <smcginnis> I will try to.
16:27:11 <ttx> no point in using meeting time to further investigate
16:27:33 <smcginnis> #topic "Decentralizing release approvals" thread and resulting TODOs
16:27:45 <ttx> yeah so there was this thread
16:27:59 <ttx> where some speed-up solutions were proposed
16:28:08 <ttx> first one is easy
16:28:20 <ttx> going back to single-approval of releases changes
16:28:42 <ttx> Waiting for two +2s can take some time
16:28:52 <ttx> especially when the proposer is Sean or me
16:28:55 <evrardjp> previous topic makes me think two approvals could (but no certainty) have helped
16:29:12 <ttx> It's obviously a trade-off
16:29:22 <evrardjp> yup but it's one in a x
16:29:23 <ttx> more reviewers means we catch more things, but delays releases
16:29:27 <smcginnis> Maybe for those auto-patches we keep the two +2's.
16:29:37 <smcginnis> As much as we can.
16:29:45 <evrardjp> I would be inclined to trust the auto-patches more, so I am sad here :p
16:29:49 <evrardjp> haha
16:29:57 <smcginnis> Yeah, very true. Very sadly true.
16:30:16 <ttx> I think single-approval is the right tradeoff
16:30:25 <ttx> between two reviews and no review at all
16:30:26 <smcginnis> Once we get root cause, maybe that will explain things sufficiently to not have to worry about more than 1 +2 on those.
16:30:39 <evrardjp> As long as it doesnt' prevent people to onboard releases.
16:30:45 <smcginnis> I agree. For the most part, our automation is good at catching most issues.
16:30:48 <ttx> there will be more errors yes, but also less frustration due to delays
16:30:57 <evrardjp> I think that's what matters
16:31:03 <smcginnis> So a single human review checking things like semver rules is probably enough in most cases.
16:31:18 <evrardjp> when projects do follow them... ahem.. :p
16:31:22 <ttx> The second proposal is around stable releases
16:31:30 <ttx> not waiting for a Monday to pass
16:31:37 <ttx> At this point the Monday does not help
16:31:52 <evrardjp> so wait, what's the difference? a day?
16:31:52 <ttx> since we don;t really get stable reviews by waiting
16:32:09 <smcginnis> For single +2, should we try +2 first, wait a bit to give someone else a chance, then +A after some set amount of delay? Or just go for it?
16:32:24 <smcginnis> Yeah, the Monday stable review day hasn't been working for quite awhile now I think.
16:32:33 <evrardjp> smcginnis: +1
16:32:39 <ttx> smcginnis: I'd just go for it if you're comfortable with it
16:32:54 <evrardjp> ttx: doesn't that apply for all votes? :p
16:32:57 <ttx> if you're not sure, just do not w+1
16:32:59 <smcginnis> Stable review in general. I've been thinking maybe those are just the ones where we require two +2's so we get more eyes on watching for stable policy issues.
16:33:07 <smcginnis> ttx: ++
16:33:28 <ttx> that would be fair
16:33:28 <evrardjp> yeah I think stable should keep the 2x+2 rule
16:33:44 <smcginnis> There's some "specialized" knowledge with stable, but nothing that much more that normal release team folks shouldn't be able to handle.
16:33:51 <ttx> So... master = single-approval OK, can wait for more in case of doubt
16:34:01 <ttx> stable = 2x +2s needed, no wait
16:34:13 <ttx> let's see how that goes
16:34:16 <smcginnis> That sounds like a good plan to me.
16:34:23 <evrardjp> yeah, and in doubt don't +2 just +1 ?
16:34:32 <smcginnis> We can always course correct down the road if we find it's causing issues.
16:34:35 <evrardjp> that's what I have been doing for now
16:34:46 <ttx> +2 in case of "yes but could use a second opinion"
16:34:54 <evrardjp> yup
16:35:01 <ttx> +2 w+1 in case of "I'm pretty sure this is good"
16:35:14 <ttx> Like when an experienced liaison submits it
16:35:17 <evrardjp> that's my rule :)
16:35:26 <smcginnis> These all seem reasonable to me.
16:35:28 <evrardjp> seems to work so far, I didn't break the world
16:35:32 <evrardjp> (yet)
16:35:33 <smcginnis> ;)
16:35:55 <ttx> smcginnis: maybe close the thread by saying we've relaxed the rules to speed up release processing, let's see how that goes?
16:36:21 <smcginnis> #agreed Current releases only require 1 +2 and can be approved if comfortable. Stable reviews should have two from any release manager.
16:36:30 <smcginnis> Will do.
16:36:33 <ttx> Note that it's also valid for all those patches from me that you have +2ed and nobody else reviewed :P
16:36:46 <smcginnis> Subtle hint? :)
16:36:53 <ttx> subtle is my middle name
16:37:00 <evrardjp> haha
16:37:05 <diablo_rojo_phon> Haha
16:37:21 <smcginnis> #topic Dropping cycle-automatic for tempest plugins
16:37:22 <evrardjp> ptl is different, you know that smcginnis has all the rights in the world for releases.
16:37:30 <ttx> "you are so subtle" said noone to me ever
16:37:36 <evrardjp> ttx: oh really?
16:37:41 <evrardjp> :p
16:37:54 <evrardjp> anyway
16:37:58 <smcginnis> I think I added this topic, though it was weeks (years) ago.
16:38:01 <ttx> a lot of people say "you are so devilishly crafty"
16:38:06 <smcginnis> hehe
16:38:26 <ttx> yeah, what's that about
16:38:38 <evrardjp> smcginnis: ofc. You want also to say it was last decade?
16:38:39 <smcginnis> With c-a release model, we originally conceived of that for things like tempest plugins that had not previously been released.
16:38:40 <ttx> like cycle-automatic was designed for tempest plugins
16:39:14 <smcginnis> With the idea that we just needed to get them tagged at the end of a cycle to have a way to track the version that goes with a given version of tempest.
16:39:16 <ttx> well c-a was mostly to make sure there would be a end release for those oft-overlooked things
16:39:24 <smcginnis> To know what is expected to work as tempest evolves.
16:39:35 <smcginnis> Yeah, that too.
16:39:38 <ttx> so why on Earth would you stop doing that?
16:39:59 <smcginnis> We originally had validation in place to make sure there weren't multiple releases, IIRC.
16:40:11 <ttx> ah, that sounds silly
16:40:24 <smcginnis> But now it turns out teams, and some downstream consumers like Red Hat, want multiple releases throughout the cycle.
16:40:35 <smcginnis> Making them look a lot more like cycle-with-intermediary.
16:41:18 <johnsom> Octavia is in that camp. We have had requests for "more often than once a release" tags.
16:41:20 <smcginnis> Since we now have been doing more final c-w-i automatically proposed releases, do we need cycle-automatic, or do we just include those things in our final checks for c-w-i that have outstanding commits to release?
16:41:38 <smcginnis> johnsom: Thanks, good to know that.
16:41:40 <ttx> hmm
16:42:01 <ttx> c-w-i would not do a final release at the very end
16:42:12 <smcginnis> Or, we keep cycle-automatic, we treat it just like we treat c-w-i, but no matter what we always do a final release at the end of the cycle.
16:43:01 <smcginnis> I guess that last one is really what we've been doing now.
16:43:06 <ttx> yeah c-a = c-wi + final release
16:43:11 <evrardjp> cycle-automatic doesn't prevent more releases , does it ?
16:43:15 <ttx> no
16:43:37 <smcginnis> Originally it did, but then folks had $reasons for wanting intermediate releases.
16:43:42 <ttx> + No stable branch will be automatically created.
16:43:52 <ttx> https://releases.openstack.org/reference/release_models.html#cycle-automatic
16:44:08 <ttx> The “cycle-automatic” model is used by specific technical deliverables that need to be automatically released once at the end of a cycle. Those may, optionally, also be released in the middle of the cycle. Those do not need a stable branch created. This may be applied only to “tempest-plugin” or “other” deliverables.
16:44:22 <ttx> I feel like that's a clear definition
16:44:41 <smcginnis> OK, that is clear enough. So I guess we are good then.
16:44:47 <tosky> thanks :)
16:44:48 <armstrong> @ttx best math equation: c-a = c-wi + final release
16:44:57 <smcginnis> :)
16:45:10 <smcginnis> #topic AOB
16:45:17 <smcginnis> Anything else to discuss today?
16:45:19 <ttx> final release = wi-a
16:45:54 <evrardjp> smcginnis: nope
16:46:22 <smcginnis> I do have a patch I think we should get through:
16:46:25 <smcginnis> https://review.opendev.org/#/c/700221/
16:46:31 <smcginnis> Our release announcements have bad links right now because of that.
16:46:42 * ttx will apply new rules
16:46:51 <ttx> :boom:
16:46:57 <smcginnis> Thank
16:47:00 <smcginnis> s
16:47:01 <smcginnis> :0
16:47:03 <evrardjp> wow
16:47:13 <evrardjp> wouldn't that be a rpoblem for a bunch of automation?
16:47:27 <smcginnis> I think it depends on the context that the job is run.
16:47:35 <evrardjp> oh I see
16:47:46 <smcginnis> Like our list-changes output includes an example of the release announcement, and I think it was fine there.
16:48:22 <smcginnis> Because in that case we call it differently and pass in the name or something. It's a little fuzzy now since I've swapped out that part.
16:48:40 <evrardjp> tarball/null does look good though :p
16:48:43 <smcginnis> The issue started because of an infra change I think that would clear the remote information.
16:48:57 <smcginnis> So suddenly we would extract "null" instead of the repo name.
16:49:19 <smcginnis> OK, if we don't have anything else, let's move along.
16:49:21 <smcginnis> Thanks everyone!
16:49:30 <evrardjp> thanks smcginnis and others!
16:49:38 <smcginnis> #endmeeting