16:00:01 <smcginnis> #startmeeting releaseteam
16:00:01 <openstack> Meeting started Thu Mar 19 16:00:01 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:04 <openstack> The meeting name has been set to 'releaseteam'
16:00:07 <ttx> o/
16:00:08 <smcginnis> Ping list: ttx armstrong diablo_rojo, diablo_rojo_phon
16:00:12 <hberaud> o/
16:00:14 <evrardjp> o/
16:00:14 <smcginnis> #link https://etherpad.openstack.org/p/ussuri-relmgt-tracking Agenda
16:00:34 <elod> o/
16:00:47 <smcginnis> Good turnout.
16:00:50 <smcginnis> #topic Outstanding rocky-em patches
16:01:01 <smcginnis> #link https://review.opendev.org/#/q/status:open+project:openstack/releases+branch:master+topic:rocky-em EM Patches
16:01:14 <smcginnis> Glance should be good to go now - https://review.opendev.org/709888
16:01:28 <smcginnis> Great if someone else can take a look and approve that.
16:01:30 <diablo_rojo> o/
16:01:43 <smcginnis> Cloudkitty - https://review.opendev.org/#/c/709920/
16:01:49 <evrardjp> I will have a look after the meeting.
16:01:56 <smcginnis> No response from PTL and it looks like there are unreleased commits.
16:02:17 <smcginnis> So... would be nice to get a release out for those, but also an indication that this may actually be unmaintained.
16:02:22 <smcginnis> Thoughts on that?
16:02:31 <evrardjp> Maybe the new PTL didn't get the memo. Will ping in their channel
16:02:48 <evrardjp> they changed PTL during the cycle
16:03:01 <smcginnis> Ah, OK. We can wait on that one a bit then.
16:03:09 <evrardjp> I expect some quirks on comms
16:03:15 <smcginnis> Telemetry - https://review.opendev.org/#/c/709880/
16:03:26 <smcginnis> PTL approved, but unreleased commits.
16:03:34 <evrardjp> :/
16:03:35 <smcginnis> No response to question regarding doing another release first.
16:03:47 <evrardjp> let's go ahead then ?
16:04:01 <evrardjp> I mean if it was approved by PTL....
16:04:12 <smcginnis> Yeah, kind of what I was thinking.
16:04:16 <smcginnis> Other thoughts?
16:04:27 <evrardjp> let's see if there are complaints later ;)
16:04:33 <evrardjp> haha
16:04:54 <elod> I think the same, if there are no answers then I guess those can be EM'd :/
16:04:55 <ttx> nope go for it
16:05:52 <smcginnis> At least looking at ceilometer, a few fixes, but nothing critical.
16:06:03 <diablo_rojo> I would hope that the PTL actually looked at the patches..
16:06:08 <smcginnis> And of course, distros and end users can always apply unreleased commits if actually needed.
16:06:27 <elod> exactly
16:06:34 <smcginnis> If someone could approve that telemetry one then...
16:06:46 <smcginnis> OSA - https://review.opendev.org/#/c/709875/
16:07:03 <smcginnis> These cycle trailing ones wanted to wait until other projects had finished going to EM.
16:07:28 <smcginnis> So I think this one can wait a little more.
16:07:38 <evrardjp> we should define a limit
16:07:53 <evrardjp> how long to wait ?
16:07:55 <ttx> telemetry one approved
16:08:09 <smcginnis> Next week maybe? I think glance and telemetry are the only big ones that OSA would need to wait for?
16:08:26 <smcginnis> Heat - https://review.opendev.org/#/c/709889/
16:08:45 <smcginnis> ricolin: There are some questions on the heat rocky em patch.
16:09:34 <smcginnis> That one at least needs the sha's updated.
16:09:46 <smcginnis> Validation has since been added to make sure that doesn't pass anymore.
16:10:39 <smcginnis> I guess go ahead and update that and transition if no response by next week? Any other ideas?
16:11:41 <smcginnis> We can go back to that one later.
16:11:47 <smcginnis> TripleO - https://review.opendev.org/#/c/709912/
16:11:48 <diablo_rojo> I think that is plenty of time that we've waited..
16:12:01 <smcginnis> That one actually looks it's OK, just never got a PTL response.
16:12:13 <smcginnis> So I think we can approve that.
16:12:27 <ttx> ok on it
16:12:50 <smcginnis> Last one, Kolla - https://review.opendev.org/#/c/709893/
16:13:03 <smcginnis> Commented on their that they wanted to wait for others to transition.
16:13:21 <smcginnis> So that one we can wait until glance and telemetry (and maybe Heat) make it through.
16:13:33 <smcginnis> I can comment on their now to ask Mark to reassess their readiness.
16:14:01 <evrardjp> sounds good
16:14:26 <smcginnis> Well, some progress on those at least.
16:14:44 <smcginnis> I will also put up a patch to reflect the stable phase on releases.o.o after the meeting.
16:14:58 <smcginnis> #topic Outstanding ussuri-cwi patches
16:15:07 <smcginnis> #link https://review.opendev.org/#/q/status:open+project:openstack/releases+branch:master+topic:ussuri-cwi
16:15:12 <smcginnis> Two outstanding ones.
16:15:19 <smcginnis> Freezer - https://review.opendev.org/#/c/709844/
16:15:50 <smcginnis> They have not done a release. A little confusing comment on there, but I think we are OK to go ahead and switch them to -rc.
16:15:56 <evrardjp> Isn't freezer already in the freezer?
16:16:01 <smcginnis> Hah
16:16:21 <smcginnis> Not sure on that projects state, but it has been iffy for awhile now I think.
16:16:33 <evrardjp> I thought only SUSE was working on it recently, but I might be wrong, I have to double check. It might be worth checking if we shouldn't retire it instead.
16:16:40 <smcginnis> So probably safe enough to do. Unless it needs to be completely dropped for Ussuri.
16:16:46 <smcginnis> Anyone know anything about its state?
16:16:51 <ttx> on a call, brb
16:16:56 <hberaud> nope...
16:17:12 <evrardjp> the safe bet would be to move it to cycle-with-rc and deprecate it next cycle
16:17:15 <smcginnis> Is the TC still doing health checks?
16:17:24 <evrardjp> but that increase the lifespan
16:17:31 <evrardjp> not actively as we used to have
16:17:39 <diablo_rojo> smcginnis, not really
16:17:59 <evrardjp> we are trying a different approach to see the healthiness of the project, using "golden signals"
16:18:15 <evrardjp> basically this is a signal :D
16:18:20 <diablo_rojo> PTL is from ZTE
16:18:25 <evrardjp> so I am thinking -- how should we interact on this
16:18:43 <diablo_rojo> I think we move it to cycle with rc
16:18:50 <diablo_rojo> and then we will see if they elect a PTL..
16:19:08 <smcginnis> I think it's the release teams place to decide if it's part of the coordinated release or not. But TCs role to see if it should even exist anymore.
16:19:25 <diablo_rojo> I would agree
16:19:28 <smcginnis> I guess since we got at least some activity from the PTL (even if it was confusing) we can do the switch to rc, then see how things go.
16:19:40 <evrardjp> agreed on both diablo_rojo and smcginnis comment
16:19:43 <smcginnis> So if someone wants to merge that patch, I think that's the right path at this point.
16:19:49 <evrardjp> I just want to make sure the TC is aware of this :)
16:20:10 <smcginnis> I suppose PTL election will be another "signal" on that.
16:20:20 <evrardjp> correct.
16:20:40 <evrardjp> at least that's my understanding. ttx can tell us more about the signals.
16:20:48 <smcginnis> The other one is similar - https://review.opendev.org/#/c/709845/
16:21:07 <smcginnis> Karbor. Some comment, but kept thiss time, on that one.
16:21:12 <smcginnis> But no follow up release.
16:21:31 <smcginnis> So I think we go ahead with the switch. Otherwise drop since technically they have not met the inclusion criteria.
16:21:44 <evrardjp> the problem I have is that the more we wait, the more it will take to deprecate :) but I think it's too late to decide for this cycle, and let's not take harsh decisions lightly
16:21:58 <evrardjp> (that's the tc hat speaking, sorry for that)
16:22:00 <diablo_rojo> +2 Sounds good to me. We can only give so much time.
16:22:11 <evrardjp> smcginnis: agreed
16:22:17 <smcginnis> "Projects must participate in at least two milestones in order to be considered part of the release."
16:22:22 <smcginnis> https://releases.openstack.org/ussuri/schedule.html#u-mf
16:22:35 <smcginnis> We do clearly state what is expected, and they have not met that expectation.
16:22:42 <evrardjp> agreed
16:22:59 <smcginnis> Would like ttx's input on this too. I don't think we've ever had to fully drop a whole project before.
16:23:30 <evrardjp> it's never too late to start those kind of things.
16:23:34 <diablo_rojo> Really? Never?
16:23:44 <diablo_rojo> I'm surprised.
16:23:56 <evrardjp> smcginnis: just a question: are you afraid of the impact on the tooling for the release?
16:24:01 <smcginnis> At least since I've been involved. The threat of the stick has usually worked to get them to do a release.
16:24:11 <smcginnis> evrardjp: No, I think we're fine.
16:24:21 <smcginnis> I will propose dropping the files.
16:24:27 <smcginnis> We can get comments on the review for that.
16:24:30 <evrardjp> yeah or release-management none
16:24:38 <evrardjp> or whatever
16:24:40 <evrardjp> :)
16:24:42 <smcginnis> But I wonder if maybe we should do the same for freezer.
16:24:59 <smcginnis> Something cycle based like this, I don't think we would want to say release-management none.
16:25:01 <evrardjp> we should probably drop an email on the ML
16:25:28 <evrardjp> smcginnis: I agree it was more: if there is a real urgency, maybe we can do something
16:25:40 <smcginnis> What do folks think about including freezer along with karbor?
16:25:45 <evrardjp> not that governance is the fastest for merging things :p haha
16:25:52 <smcginnis> ;)
16:26:10 <evrardjp> how long can this wait?
16:26:16 <diablo_rojo> smcginnis, freezer had one release but karbor has had none right?
16:26:34 <smcginnis> We're already way past the deadline. Not sure we want to wait much longer in the cycle to communicate something like this.
16:26:43 <ttx> back
16:26:46 <evrardjp> ok I guess we should go ahead then
16:26:51 <ttx> catching up
16:26:53 <smcginnis> evrardjp: No, neither.
16:27:01 <smcginnis> evrardjp: https://review.opendev.org/#/c/709844/1/deliverables/ussuri/freezer.yaml
16:27:40 <smcginnis> ttx: tldr; should we drop freezer and karbor from being included in the ussuri release since they have not done releases by the deadline.
16:28:09 <ttx> technically we are past membership freeze, so what's in we are committed to release
16:28:09 <evrardjp> I have the impression freezer is a repeating offender, but maybe it's my wrong impression
16:28:17 <evrardjp> oh that's true too
16:28:31 <evrardjp> so that's the solution
16:28:37 <smcginnis> But with no releases at all, just a placeholder file, are those actually "in"?
16:29:15 <ttx> smcginnis: the idea behind membership freeze was to communicate early what will be in release
16:29:18 <ttx> for packagers and all
16:29:33 <ttx> not sure who actually pays attention, but I like the concept
16:29:38 <smcginnis> True
16:29:38 <evrardjp> smcginnis: technically -- why not? assuming the project is very stable it might not require new patches and new releases, but be part of the coordinated release
16:29:53 <openstackgerrit> Merged openstack/releases master: [Telemetry] Transition Rocky to EM  https://review.opendev.org/709880
16:30:06 <smcginnis> OK, then just force both of these to -rc, even though at least one the PTL has said no.
16:30:07 <openstackgerrit> Elod Illes proposed openstack/releases master: Set Rocky status to Extended Maintenance  https://review.opendev.org/713926
16:30:07 <evrardjp> ttx: I have doubts about the usefulness, but that's a conversation for another time :p
16:30:29 <ttx> smcginnis: yes... that or proposing an intermediary release immediately
16:30:39 <smcginnis> I'm concerned about the lack of visibility into zombie projects that we just end up taking care of.
16:30:49 <ttx> I would switch karbor, and propose a release for freezer on HEAD
16:30:50 <smcginnis> We did add the "forced" flag awhile back.
16:31:14 <smcginnis> Maybe we should think about doing that as a way to track the ones that do not get any response from the teams.
16:31:27 <ttx> Feels like a ML thread about it could trigger some attention
16:31:34 <ttx> one for each
16:31:44 <smcginnis> ttx: I'm concerned that we asked them to get an intermediary release out to stay cwi, then no response at all.
16:31:46 <ttx> that way we (1) track it and (2) use the shame
16:31:47 <evrardjp> would you mind tackling this? :)
16:32:02 <evrardjp> I am afraid of my wording.
16:32:05 <ttx> smcginnis: then we force-approve it
16:32:58 <evrardjp> I agree with ttx plan. currently I have voted on those, and if they propose a patch, I can vote again.
16:33:01 <smcginnis> OK, switch karbor and force freezer. That works for me.
16:33:09 <ttx> then I'll create a thread for each
16:33:18 <smcginnis> ttx: Thanks for taking that.
16:33:25 <smcginnis> I can get the release proposed.
16:33:42 <ttx> mind you, the thread might conclude membership freeze should not prevent us from removing stuff at the last minute
16:33:58 <smcginnis> I like that too.
16:34:31 <ttx> hmm
16:34:32 <smcginnis> Oh good, the additional validations caught something with the tripleo patch. I'll get that fixed up too.
16:34:39 <ttx> I'm confused
16:35:15 <evrardjp> ttx: ?
16:35:34 <ttx> so freezer is the one where jiaopengju said "please ignore this comment."
16:35:47 <ttx> basically not -1 it
16:35:59 <ttx> so that would be the one we'd swtch to -rc
16:36:20 * hberaud need to eject, sorry, see you tomorrow
16:36:22 <ttx> karbor is the one with a standing -1
16:36:33 <ttx> so the one for which we should propose a release
16:36:43 <smcginnis> Yep. I think that's waht we said, release karbor, transition freezer.
16:37:01 <ttx> ok, you had said OK, switch karbor and force freezer
16:37:01 <smcginnis> Oh, nope, I stated that backwards earlier.
16:37:02 <smcginnis> Sorry.
16:37:18 <ttx> as  long as we are on the same boat
16:37:19 <smcginnis> Do what I meant, not what I said. :)
16:37:20 <evrardjp> oh interesting the three of us understood it correctly, but the words were wrong :)
16:37:47 <smcginnis> ttx: As long as it's not a cruise ship.
16:37:52 <evrardjp> smcginnis: hahaha
16:37:57 <evrardjp> #nevertoosoon
16:38:09 <smcginnis> #alwaystoosoon
16:38:26 <smcginnis> OK, I think that horse is well beaten.
16:38:28 <diablo_rojo> #neveracruiseship
16:38:36 <smcginnis> Hah~
16:38:38 <smcginnis> #topic Validate countdown email
16:38:45 <smcginnis> #link https://etherpad.openstack.org/p/relmgmt-weekly-emails
16:39:24 <smcginnis> Line 210
16:41:15 <smcginnis> diablo_rojo: Should we expand on the cycle highlights in there to point out appropriate content for that?
16:41:29 <smcginnis> We at least have the links.
16:41:42 <diablo_rojo> smcginnis, I was just debating that.
16:42:04 <smcginnis> I added one line to address a common confusion at least.
16:42:18 <diablo_rojo> I think there is enough detail there since you are linking to the email etc.
16:42:22 <diablo_rojo> Yeah that is a good addition
16:42:38 <diablo_rojo> Maybe add it to the dates list at the bottom
16:42:57 <smcginnis> What is the deadline for these?
16:43:31 <smcginnis> OK, perfect.
16:43:41 <smcginnis> Thanks ttx as always for getting these prepped.
16:43:50 <smcginnis> I will send out tomorrow, my morning.
16:44:02 <diablo_rojo> Oh interesting
16:44:11 <diablo_rojo> We dont have the deadline on the release schedule for ussuri
16:44:15 <diablo_rojo> I will get that patch out today
16:44:23 <smcginnis> Can't hurt.
16:44:44 <smcginnis> #topic Unmaintained for older branches
16:44:51 <smcginnis> Just wanted to check with the team.
16:44:59 <smcginnis> We have some really older stable branches.
16:45:10 <smcginnis> With some things being still "maintained" and some not.
16:45:36 <smcginnis> It was never really clear to me what the signal would be to know if and when something would transition to unmaintained.
16:45:43 <smcginnis> Other than the team expressing declaring it.
16:46:00 <smcginnis> But I'm fairly sure not a lot of teams are still paying much attention to Ocata.
16:46:19 <elod> if the branch of a project does not get patches about ~6 months. maybe?
16:46:38 <smcginnis> I guess that's one sign.
16:46:49 <smcginnis> Or proposed patches that are not getting merged.
16:46:51 <ttx> yeah I like objective signs
16:46:53 <smcginnis> Or broken CI.
16:47:06 <ttx> since if we ask around there will always be someone saving it
16:47:08 <elod> yes, those, too
16:47:45 <smcginnis> Secondary question, do we need to wait for all teams before we show these as unmaintained on releases.o.o?
16:48:08 <fungi> i suppose "broken ci" is only a signal if it's broken for a while and nobody is fixing it
16:48:26 <fungi> that may be hard to gauge in a simple spot-check
16:48:29 <smcginnis> Do we run stable periodic jobs on ocata yet?
16:48:39 <evrardjp> fungi: I am afraid of a move to a pseudo noop job to make things green
16:48:51 <smcginnis> Yeah, I've seen some of that.
16:48:54 <elod> yes, there are periodic jobs still against ocata
16:48:59 <smcginnis> elod: OK, great.
16:49:04 <fungi> plenty of stable branch jobs are perpetually broken because nobody looks at them, but they're also usually not the same jobs being run against proposed changes
16:49:08 <elod> and there are some constantly failing :)
16:49:43 <smcginnis> It does seem like there's a usual set of failures every day. I haven't been paying attention to which branches those were coming from.
16:49:44 <fungi> also one of the options spelled out in the extended maintenance section of the stable branch chapter in the project teams guide is to drop broken ci jobs
16:50:18 <smcginnis> So I guess as far as this team is concerned, we just leave these as Unmaintained until we are told otherwise?
16:50:51 <fungi> because the extended maintenance proponents felt having an untested (or nominally tested) branch still receiving patches was better than straight eol
16:51:15 <elod> heat, nova-powervm and nowadays octavia are failing in ocata
16:51:56 <johnsom> Ocata is long EOL for Octavia.....
16:52:04 <smcginnis> OK, not something this team needs to solve today. Just wanted to raise it to get folks thinking.
16:52:14 <smcginnis> johnsom: Was that announced on the mailing list?
16:52:25 <smcginnis> But makes me think, we need some central place to track that too.
16:52:39 <johnsom> Yeah, I would have to dig way back in the way-back machine
16:52:49 <elod> hmmm. then i guess the jobs should be dropped in such cases
16:53:00 <smcginnis> johnsom: Doesn't look like it every got an ocata-eol tag proposed.
16:53:01 <johnsom> Pike was the first 1.0 release, so we announced that it was as far back as we go
16:53:09 <smcginnis> That would reflect that state on the site.
16:53:14 <fungi> maybe we're missing a clear process for removing periodic jobs when branches transition to unmaintained
16:53:38 <smcginnis> We should document what needs to be done. Do the -eol tagging, remove periodic stable jobs, etc.
16:54:15 <johnsom> We were just having a conversation about this on our channel. It's really not clear how individual teams can EOL or "declare" Unmaintained. beyond the e-mail list which gets lost to history.
16:54:18 <fungi> the "new" extended maintenance process got rid of the eol tags, i thought
16:54:31 <smcginnis> IIRC, part of the process was a team had to announce it and wait 6 months in case someone else came along and volunteered to maintain the older branch than the official team would.
16:54:42 <fungi> or maybe it only got rid of deleting eol branches
16:54:54 <smcginnis> fungi: It didn't get rid of branches being EOL though. Just how it got to that phase.
16:55:04 <johnsom> Right, there was talk that the new process would no longer remove the branches like the old EOL did
16:55:05 <smcginnis> Yeah, maybe the deletion. That sounds right.
16:55:09 <elod> I can prepare something to the stable-branch page and we can refine there the general TODOs, is that OK?
16:55:19 <smcginnis> elod: That would be great!
16:55:22 <fungi> ahh, rereading there is an eol phase, which comes 6 months after unmaintained
16:55:30 <fungi> https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-life
16:55:47 <smcginnis> It's that process of getting between maintained and eol that isn't as clear as I would like.
16:55:53 <fungi> current process does still tag and delete branches at eol time
16:56:09 <smcginnis> Anyway, not something we need to solve in this meeting. I'm going to move on to our last agenda topic.
16:56:13 <smcginnis> #topic
16:56:19 <johnsom> Frankly, we are talking about EOL up to Rocky.
16:56:19 <smcginnis> #topic Check-release-approval false negatives
16:56:20 <evrardjp> that's quite the topic
16:56:35 <smcginnis> johnsom: I think that makes sense.
16:56:35 <ttx> yeah so the latest change did not really fix anything
16:56:44 <smcginnis> #link http://zuul.openstack.org/build/d385f0d89418420baaf4b8a1d05d345f
16:56:44 <fungi> looks like the transition from unmaintained to eol is pretty clear, it's the transition from extended maintenance to unmaintained which is muddy
16:56:45 <ttx> we already have a false negative
16:56:51 <smcginnis> Looks like the same symptoms.
16:56:59 <smcginnis> So something we need help on the gerrit side to figure out.
16:57:03 <ttx> Currently picking it apart to check if anything else is missing
16:57:23 <fungi> ooh, i like this game. i'll play too
16:57:26 <ttx> The code does not seem to have crazy code paths that would explain this behavior
16:57:46 <ttx> It's not limited to a specific executor
16:58:14 <smcginnis> Really seems like it would have to be a bug in gerrit.
16:58:31 <ttx> smcginnis: yeah, one which I can't reproduce locally
16:58:45 <ttx> I ran a lot of tests and never hit it
16:58:49 <smcginnis> fungi: Do you know if there are any debug logs on the gerrit side that could help?
16:58:57 <ttx> While it seems to be between 5% and 20% hit rate in zuul
16:59:11 <fungi> gerrit's only useful logging is for the ssh api, really
16:59:51 <ttx> interesting...
16:59:52 <smcginnis> ttx: And that's not an option from the executor?
17:00:08 <ttx> "permitted_labels": {}
17:00:20 <ttx> The failed JSON has ^
17:00:39 <ttx> Successful JSON has the data in that array
17:01:18 <smcginnis> We're over time, so I'm going to end the meeting. But we should keep on this.
17:01:46 <smcginnis> #endmeeting