15:01:56 <smcginnis> #startmeeting releaseteam
15:01:57 <openstack> Meeting started Fri Aug 10 15:01:56 2018 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:57 <ttx> o/
15:01:58 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:00 <openstack> The meeting name has been set to 'releaseteam'
15:02:01 <evrardjp> o/
15:02:07 <smcginnis> #link https://etherpad.openstack.org/p/rocky-relmgt-tracking Agenda
15:02:20 <smcginnis> Currently around line 503
15:02:42 <smcginnis> I won't do the ping list, but hopefully get by just mentioning ttx and dhellmann
15:03:03 <dhellmann> o/
15:03:03 <annabelleB> o/ (late to the party)
15:03:18 <armstrong> O/
15:03:25 <ttx> I was there already, see above :)
15:04:20 <prometheanfire> o/
15:04:31 <smcginnis> Good, the gangs all here.
15:04:44 <smcginnis> #topic RC status
15:05:02 <smcginnis> We have some details in the etherpad for various deliverables.
15:05:13 <smcginnis> prometheanfire: That near the bottom of https://etherpad.openstack.org/p/rocky-relmgt-tracking
15:05:45 <smcginnis> I think we have an update on RC1 deliverables except for cyborg.
15:05:53 * fungi is around, just having too many discussions at once
15:06:08 <prometheanfire> same, two meetings at once :P
15:06:16 * dhellmann is having 2 conversations with fungi
15:06:20 <smcginnis> :)
15:06:43 <smcginnis> Barbican - they thought deadline was today.
15:06:49 <prometheanfire> smcginnis: so the question I have is the same as all the FFE things, UC only? probably fine
15:07:32 <smcginnis> prometheanfire: We have a few client libraries in the list on the etherpad.
15:07:42 <smcginnis> Looks like all should just be u-c only.
15:07:49 <prometheanfire> ya, I see them, looks like 3
15:08:19 <smcginnis> prometheanfire: Do you want us to direct them to send a FFE?
15:08:25 <prometheanfire> iirc I think karborclient had a release already, but ya, UC only is fine
15:08:40 <ttx> smcginnis: Looks like only cyborg might need to be cat-herded?
15:08:45 <prometheanfire> if only to name and shame
15:08:47 <smcginnis> prometheanfire: We have a python-designateclient patch up.
15:09:03 <ttx> (in the cycle-with-milestones ones)
15:09:20 <smcginnis> mugsie: Can you send an FFE to openstack-dev for the python-designateclient? Then we can link to that for the release patch.
15:09:33 <smcginnis> ttx: Yeah, that is the only one I haven't seen activity from.
15:09:37 <prometheanfire> they could be batched into one request to cut the spam too
15:10:17 <ttx> smcginnis: + karbor and tricircle for their client ?
15:10:41 <smcginnis> So sticking with the RC list so we don't scatter all over - we are waiting for barbican, we may need to force cyborg, the rest have patches up.
15:10:50 <smcginnis> ttx: Karbor is separate.
15:11:02 <smcginnis> ttx: Oh, sorry, I see what you're saying now.
15:11:27 <ttx> I'd just cut a release for both
15:11:47 <ttx> if that's just CI/reqs changes
15:11:51 <smcginnis> I can send the FFE for those two at least as a way to announce the release team is following through with what we had said we would do and are forcing a release of them.
15:11:58 <ttx> ++
15:12:26 <prometheanfire> wfm
15:12:31 <ttx> and then if karbor fails to release an intermediary release we'd remove both
15:12:45 <ttx> (karbor and karborclient)
15:12:52 <smcginnis> Good point. I will note that as well.
15:13:36 <smcginnis> #action smcginnis to send FFE for karbor and tricircle clients noting release is being forced. Also noting karbor will be removed if service release is not requested.
15:13:40 <ttx> (tricircle ahs a relatively recent release and is branched)
15:14:01 <smcginnis> So maybe just some late patches for the client.
15:14:08 <ttx> ok next cycle-with-intermediary services
15:14:32 <ttx> we heard for cloudkitty earlier this week, they should send in something
15:14:44 <ttx> not sure we heard anything from karbor or magnum
15:14:50 <smcginnis> I have not.
15:14:55 * ttx checks his ping list
15:14:55 <dhellmann> nor i
15:15:07 <ttx> strigazi: ?
15:15:09 <smcginnis> And not sure how cloudkitty missed RC when they had been in touch prior to it.
15:15:38 <ttx> I think he said he might be one or two days late ?
15:15:53 <ttx> which for intermediary-released stuff is not so big
15:16:27 <prometheanfire> ya, triple-o is in that camp
15:16:31 <ttx> zhipeng is not on this channe;
15:16:33 <ttx> l
15:17:02 <dhellmann> the intermediary server projects aren't technically late, right?
15:17:15 <ttx> dhellmann: no they aren't
15:17:19 <dhellmann> we just set the deadline for them to the final deadline, according to the schedule page
15:17:23 <smcginnis> According to the clarification last week, they have until RC final.
15:17:32 <dhellmann> right
15:17:37 <dhellmann> that's what I meant
15:17:40 <ttx> we'd still prefer it earlier so we can cut stabel/rocky
15:17:47 <evrardjp> agreed
15:17:51 <smcginnis> Cuts it close for some of these that have not done any releases yet, but that is what we published.
15:18:04 <dhellmann> right, but it's their problem if they run into issues when we go ahead with the other branches
15:18:11 <smcginnis> True
15:18:12 <dhellmann> that's what we've communicated in the past
15:19:04 <smcginnis> So cloudkitty - we know they are working on it, karbor - will warn in client FFE that they need to do a release or risk being removed, magnum - no word yet.
15:19:10 <dhellmann> and since we're not syncing requirements any more, we shouldn't have any issues with versions going up on master then back down after the branch
15:20:16 <smcginnis> Clients: designate - patch is up, karbor and tricircle - will send FFE and force releases.
15:20:22 <dhellmann> should we warn them about the impending deadline and missing being included in rocky?
15:20:35 <dhellmann> (them == magnum)
15:20:46 <fungi> the castellan ffe e-mail just hot the -dev list too
15:21:05 <fungi> er, just hit
15:21:36 <smcginnis> We probably should warn magnum, though we have stated in multiple posts and it is documented.
15:23:19 <smcginnis> Horizon plugins?
15:23:33 <smcginnis> Cloudkitty is known.
15:23:43 <dhellmann> I'm trying to put the notes into the etherpad, and I missed the client ones. are those all warnings?
15:24:09 <smcginnis> Hopefully mugsie can send FFE for designate. I will send one for karbor and tricircle ones.
15:24:25 <smcginnis> With the warning that they will be removed if no service release by RC final.
15:24:31 <mugsie> smcginnis: I will do that now, before I forget
15:24:38 <smcginnis> mugsie: Thanks!
15:25:12 <dhellmann> ok, I think I'm caught up
15:25:43 <smcginnis> Not sure about tacker-horizon and zun-ui.
15:25:55 <smcginnis> zun-ui looks to be the bigger concern.
15:26:03 <dhellmann> those looked like there were enough changes to need a warning
15:26:03 <smcginnis> But they do still have until RC final.
15:27:26 <smcginnis> OK, "others"?
15:27:46 <ttx> TheJulia might know the bifrost situation
15:28:11 <dhellmann> the monasca-kibana-plugin had no real changes except the readme file format
15:28:30 <smcginnis> kuryr-* ones have some, but again have until RC final.
15:28:40 <dhellmann> I guess that means it's stable? it would be good to have confirmation that it's maintained.
15:28:51 <dhellmann> the tempest tag is called out in our process somewhere I think
15:29:27 <dhellmann> oh, just for the branching
15:30:18 <dhellmann> I do seem to remember that we wait to tag tempest as long as possible though
15:30:30 <smcginnis> I have not seen anything for networking-hyperv, and given the recent election results for the Win team, they might need a reminder.
15:30:42 <dhellmann> seems likely
15:30:49 <ttx> I feel like we should (in the future) mandate that intermediary-released stuff has one release and branch done by R-2
15:31:04 <dhellmann> that's early for a branch, but a release makes sense
15:31:15 <evrardjp> dhellmann: it would be nice to document the why
15:31:18 <dhellmann> we used to have that rule, iirc
15:31:21 <ttx> since the turnaround for removing that deliverable (and update release messaging) is REALLY short
15:31:38 <dhellmann> evrardjp : yes, definitely
15:31:41 <smcginnis> ttx: RC2 or RC3? It would be nice to make sure something is being done before the end.
15:32:13 <dhellmann> smcginnis : rc2 is the inclusion deadline for milestone projects, so that sort of makes sense. although rc3 would be ok, too.
15:32:15 <ttx> smcginnis: I'd either add a deadline at R-2, or require that anything that is past R-3 requires an exception
15:32:41 <ttx> Basically I'm fine cutting slack to people that are keeping up and reaching out
15:32:57 <ttx> I'm worried about peolpe we haven't heard from
15:33:05 <smcginnis> Me too.
15:33:35 <dhellmann> we don't really do negative messaging, right? so even if we remove the project at r2 folks who aren't checking until the end of the cycle won't notice until then.
15:33:50 <TheJulia> re bifrost, I know it needs to be released, but it is mainly geared on tooling to deploy ironic, so I just haven't gotten to that yet since there has a minor breakage
15:34:02 <smcginnis> TheJulia: OK, thanks.
15:34:16 <smcginnis> TheJulia: It is cycle-with-intermediary, so there's a little time yet.
15:34:54 <TheJulia> yeah, my perception was next week
15:35:05 <ttx> The original idea behind MembershipFreeze and requiring somethign is in by milestone-2 was to have those discussions about what is in the release a lot earlier in the cycle
15:35:23 <dhellmann> yeah, that's why I would support an rc2 deadline for intermediary projects
15:35:29 <ttx> removing stuff the week before release is not making us look good
15:35:30 <dhellmann> just make them all the same deadline
15:35:31 <smcginnis> That makes sense.
15:35:41 <ttx> and makes it very difficult to work on release messaging
15:35:44 <annabelleB> +1
15:35:47 <smcginnis> We really should not have questions about what is in or out this late.
15:36:16 <ttx> Also... intermediary-released was for things that do multiple releases
15:36:39 <ttx> things that can't make a release need the cycle-with-milestone baby-wheels
15:36:50 <evrardjp> I agree on the idea of having the "same deadline" as message, but it's far from possible due to a chain of dependencies
15:37:01 <ttx> I'm just ranting out loud
15:37:06 <dhellmann> evrardjp : ?
15:37:14 <smcginnis> ttx: No, it's all very valid.
15:37:51 <evrardjp> maybe it was out of context, continue :)
15:38:13 <evrardjp> "them all" was not clear to
15:38:14 <evrardjp> me
15:38:38 <annabelleB> (I see cyborg rc1 just appeared)
15:38:41 <dhellmann> evrardjp : ah, I meant "all cycle-with-intermediary and cycle-with-milestones projects"
15:38:49 <armstrong> thx: thx for the clarification on release models, it’s clear now to me.
15:38:52 <evrardjp> dhellmann: I realised that afterwards.
15:39:00 <smcginnis> OK, c-w-i outdated releases.
15:39:05 <smcginnis> Swift we know is coming.
15:39:11 <smcginnis> Not sure about heat.
15:39:20 <dhellmann> smcginnis , ttx: perhaps we should change the models for some of these projects that aren't really following the intermediary model expections
15:39:21 <smcginnis> We may just need to force a branch from the last release.
15:39:23 <ttx> The RC dance was so that we have a fallback release on R-3. intermediary-released did not have to go with the RC dance because we were supposed to have a fallback release (the last intermediary) already
15:39:54 <ttx> And now people use it to release a single thing at the end... and we lose our fallback
15:40:06 <smcginnis> dhellmann: I think I'd rather prod them to require an earlier intermediary release.
15:40:08 <dhellmann> right
15:40:18 <evrardjp> ttx: that's good to clarify this.
15:40:22 <dhellmann> smcginnis : well, like ttx said, if they're not actually producing intermediary releases...
15:40:38 <ttx> I think we should reach out to all those who did a single release and being intermediary
15:40:50 <ttx> and ask them about going with-milestones again
15:41:00 <evrardjp> or independant?
15:41:09 <ttx> The trick being, of course, that with-milestones is inferior as it's less semver
15:41:10 <dhellmann> perhaps we should suggest more strongly than we did in the past when we had those conversations
15:41:11 <smcginnis> I'm saying I'd rather force (stronly encourage) them to do intermediary releases like they should or miss being part of the coordinated release.
15:41:29 <ttx> evrardjp: we want "openstack" main components to be released on the cycle
15:41:35 <smcginnis> evrardjp: Yep, good point. If they can't do intermediary, then they probably should be independent.
15:41:36 <ttx> so that independent option is not open to all
15:41:46 <dhellmann> smcginnis : ok, I guess presenting the choice that way works for me. Either actually do intermediary releases, switch to milestones, or switch to independent.
15:41:49 <armstrong> ttx: how many RC should a release have?
15:41:50 <smcginnis> It would depend on what it is.
15:41:59 <evrardjp> ttx: that was a joke initially -- what I meant behind that is that maybe they are not using the right model
15:42:07 <smcginnis> armstrong: As close to 1 as possible.
15:42:13 <dhellmann> armstrong : for milestone-based projects we require 1 and often have a second for final translations.
15:42:15 <smcginnis> evrardjp: Could be.
15:42:26 * TheJulia finds this entire discussion really frustrating from a contribution standpoint.
15:42:28 <ttx> dhellmann: in theory, we /could/ fallback to the last release they did. The previous cycle.
15:42:33 <armstrong> ok thx
15:42:39 <dhellmann> but yeah, we don't want a lot of them
15:42:40 <dhellmann> TheJulia : what about it frustrates you?
15:42:46 <smcginnis> TheJulia: Can you share your frustrations?
15:42:58 <TheJulia> dhellmann: eating food real quick to make sure it is not low blood sugar
15:43:04 <smcginnis> :)
15:43:10 <evrardjp> TheJulia: that's why I am saying this: If your project doesn't meet certain procedure,maybe the procedure isn't the right one?
15:43:24 <evrardjp> TheJulia: I smiled on blood sugar.
15:43:25 <dhellmann> TheJulia : ack. I'm interested in your perspective. I'm probably too close to the other side of the question.
15:43:55 <smcginnis> All of this should be around helping teams make sure they are progressing along and able to be released together as a good, quality whole. If it's not encouraging that, then we should reevaluate.
15:44:01 <TheJulia> nomming
15:44:03 <dhellmann> evrardjp : right, a big purpose of the models is to describe what process the project uses for releases, so if the project isn't actually doing what the model says then the model is wrong
15:44:17 <ttx> basically I'd prefer not to remove anything from the release after milestone-2
15:44:35 <evrardjp> dhellmann: I don't want to look oversimplifying though :)
15:45:06 <dhellmann> evrardjp : I think the models are defined well, so what I mean is that they are not matched correctly to what the teams need.
15:45:13 <dhellmann> we're using the wrong models, not defining the models incorrectly
15:45:27 <evrardjp> that's what I meant.
15:45:35 <evrardjp> precisely!
15:45:42 <dhellmann> huzzah, we agree!
15:45:49 <smcginnis> We have a long list of ones missing stable branches. Some appear to have merged a lot since their last release, but I think we will just have to force the stable branch for those that do not have anything before final.
15:46:13 <dhellmann> what is the effect of not forcing stable branches now for projects that are not milestone-based?
15:46:44 <smcginnis> These are cycle-with-intermediary, so I don't think we should force until the RC final deadline.
15:46:58 <dhellmann> ah, right, I misread what you said
15:47:07 <smcginnis> Otherwise if they are planning to release more, they would end up needing to backport potentally a lot.
15:47:09 * dhellmann may need some of TheJulia's noms
15:47:15 <ttx> dumping a few more thoughts in #openstack-release to avoid disrupting meeting further
15:47:34 <smcginnis> Release team brunch.
15:47:35 <smcginnis> :)
15:47:41 <dhellmann> there's a plan
15:47:43 <annabelleB> here for this!
15:47:46 <dhellmann> does denver do brunch?
15:48:03 <evrardjp> will there be wine?
15:48:06 <smcginnis> I'm sure we can find somewhere.
15:48:08 <dhellmann> of course!
15:48:10 <smcginnis> Mimosas?
15:48:17 <dhellmann> bingo
15:48:22 <TheJulia> I see it as kind of two fold. We want fallbacks for messaging, but as pointed out we don't get them... because projects have the ability to break eachother kind of hard at times by minor actions or changes. I get the desire for messaging, but if you can't get to something to milestone-3 that is unrelated, then your essentially penalizing in my mind. We're essentially trying to drive things to move in lock-step, but
15:48:22 <TheJulia> it is really unreasonable imho. I think we can only really 'release' what we know works together too.
15:48:32 <smcginnis> Funny how several channels have all gravitated towards food lately.
15:48:58 <dhellmann> smcginnis : the memes own us
15:49:16 * fungi plans to go find food as soon as the meeting ends
15:49:26 <smcginnis> TheJulia: We want to make sure a release is done earlier in the cycle, but it would not be their final release.
15:49:29 <evrardjp> smcginnis: I am not at fault, but yeah that's my main hobby.
15:49:43 <smcginnis> TheJulia: Or maybe I'm misunderstanding what you're saying.
15:49:51 <dhellmann> TheJulia : I guess the point is to release early and often, or to use the milestone model which is strictly date-based and tests the release process without producing something that claims to be production ready
15:50:08 <TheJulia> smcginnis: so then we force number/release model, stable branch backporting for the final release?
15:50:21 <dhellmann> projects using the intermediary model claim to be stable, well tested, and well maintained. If that's not the case, then that's the wrong model for the project.
15:50:28 <TheJulia> but that feels like it fragments the deliverable product
15:50:49 <fungi> put another way, if the project is only creating one release per cycle, then it ought to be cycle-with-milestone?
15:50:58 <smcginnis> We don't force release model. That is something the team chooses. We just try to enforce that they follow their chosen model.
15:51:03 <ttx> fungi: well, yes.
15:51:25 <dhellmann> smcginnis : right. we want them to accurately describe how they manage releases.
15:51:38 <ttx> it's called cycle-with-intermediary, not cycle-with-one-single-release-that-if-you-miss-it-you-are-out
15:51:52 <smcginnis> We don't want to impose anything on the teams other than to follow through with what they've declared they would do.
15:51:53 <dhellmann> that's spelled "independent"
15:52:08 <fungi> heh
15:52:15 <evrardjp> smcginnis: should we?
15:52:23 <ttx> ok if we are having that discussion now... I'll paste from #opensatck-release
15:52:50 <ttx> 17:48 <ttx> In the name of giving teams control over their deliverables, we created a machine that removes components from OpenStack if someone misses a couple of checkboxes.
15:52:51 * dhellmann waits for the paste before responding
15:52:52 <ttx> 17:48 <ttx> I wonder if the process should not, by default, do releases for people
15:52:54 <ttx> 17:49 <ttx> and then those who want the extra control can assert it
15:53:07 <evrardjp> I agree.
15:53:24 <TheJulia> smcginnis: I think that is a good delineation, the conundrum is I think the milestone model is unsustainable without automated point in time "we're releasing this" which is out of control of contributiors aside from trying to get things in early, because we do get the gates bogged down at the end of the cycle, but I also think we have many smaller minor things that just don't have the resources to release any other
15:53:24 <TheJulia> way except as "hey, got this thing",  but then compatibility is completely unknown to everything else.
15:53:32 <evrardjp> it simplifies the model, and doesn't have a bad message.
15:53:44 <dhellmann> ttx: we have the split between servers and libraries to consider, too. I would support "milestone by default" for servers, but that doesn't work for libs. :-/
15:54:04 * TheJulia hits the same idea as ttx
15:54:07 <ttx> dhellmann: we could do regular library releases too ?
15:54:27 <TheJulia> if there have been changes landed, sure
15:54:29 <dhellmann> ttx: we could, but when something breaks as the result then that's on us instead of the team that owns the library
15:54:45 <dhellmann> we've had a few times where we knew and oslo release would break something, so we waited
15:54:57 <ttx> hmm
15:55:01 <smcginnis> "why did you release that - it's not ready and we've been trying to fix this bug!"
15:55:05 <dhellmann> right
15:55:18 <TheJulia> This also goes back to the earlier discussion where releasing libraries without a reason felt like the wrong thing to be doing
15:55:35 <dhellmann> that's less likely with servers, except the break would happen downstream of us where someone is automatically packaging based on tags
15:55:35 <TheJulia> smcginnis: "tough"?
15:55:39 <ttx> if you don't want your thing to be released, you can flag it
15:55:42 <smcginnis> :)
15:55:49 <fungi> releasing libraries when there are changes is the right thing to be doing. making unnecessary changes to libraries is the wrong thing to be doing
15:56:07 <dhellmann> when did we do unnecessary library releases?
15:56:20 <TheJulia> dhellmann: we were talking about forcing releases of libraries with no changes
15:56:22 <ttx> If you don;t flag it as "safe to release anytime", then it's on you to fix it if it breaks ?
15:56:28 <fungi> i assume it's conflated with releasing libs with just requirements bumps in them in the past
15:56:51 <fungi> and the idea that we should let our stable libs actually remain stable
15:57:06 <dhellmann> TheJulia : ah, maybe because there were CI changes and so if we create the branch without including them it would be broken immediately and require backports?
15:57:39 <evrardjp> ttx: isn't that what a PTL of a project can do? Like reviewing things is probably easier than remembering the deadlines in a busy world.
15:57:58 <dhellmann> yeah, we still need to talk about what to do with those. I think I'm leaning towards just making stable libraries independent.
15:58:11 <TheJulia> dhellmann: That is the risk with any branch or any release. Ironic has had to create its branch in a broken state because it was broken by things outside of our influence or control.  We had to backport everything to get things going again
15:58:24 <ttx> Anyway, that won't apply to rocky, but i think we should spend a few minutes on that in Denver
15:58:30 <fungi> what was the resolution on branching when there were no changes? did we decide to preemptively branch?
15:58:31 <dhellmann> TheJulia : sure. We're trying to reduce the number of times that happens, though.
15:58:53 <smcginnis> With 2 minutes left, this does seem like a good thing for the PTG.
15:59:04 <fungi> agreed
15:59:09 <TheJulia> dhellmann: agreed and I believe we should attempt that
15:59:11 <TheJulia> agreed
15:59:33 <evrardjp> +1
15:59:38 <smcginnis> I'll add a topic to our PTG etherpad.
16:00:00 <smcginnis> Anyting else in the last few seconds?
16:00:02 <dhellmann> fungi : I don't remember, but we should make sure it's written down somewhere
16:00:22 <ttx> cyborg request is now in
16:00:37 <ttx> and Barbican's
16:00:43 <smcginnis> Great!
16:00:45 <ttx> so we are good as far as Rocky process goes :)
16:00:51 <smcginnis> OK, we're at time. Thanks everyone.
16:00:56 <fungi> thanks smcginnis!
16:00:58 <smcginnis> #endmeeting