15:00:03 <smcginnis> #startmeeting releaseteam 15:00:04 <openstack> Meeting started Fri Jun 1 15:00:03 2018 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:07 <openstack> The meeting name has been set to 'releaseteam' 15:00:10 <smcginnis> Ping list: dhellmann, dims, fungi, tonyb, lbragstad, ttx, armstrong, annabelleB 15:00:16 <annabelleB> o/ 15:00:17 <ttx> Hi! 15:00:23 <smcginnis> #link https://etherpad.openstack.org/p/rocky-relmgt-tracking Agenda 15:00:23 <fungi> howdy 15:00:24 <dhellmann> o/ 15:01:12 <fungi> looks like we're around line 216 today? 15:01:31 <smcginnis> Yep, looks right. 15:01:37 <smcginnis> #topic Rocky-2 tasks 15:01:44 <smcginnis> #link https://github.com/openstack/releases/blob/master/PROCESS.rst 15:02:10 <smcginnis> So other than tagging the milestone releases, I guess I should have included a mention of the unrelease changes. 15:02:25 <smcginnis> I actually did run the report, but didn't include that. 15:02:28 <ttx> I'll be traveling Tue-Thu, but shoudl be around to take my share on teh Wednesday 15:02:33 <smcginnis> I'll get that next tune, 15:02:36 <smcginnis> *time 15:02:47 <smcginnis> ttx: Great, thanks. 15:03:01 <smcginnis> Any other issues/concerns for next week's milestone activities? 15:03:27 <smcginnis> OK, I don't think I have anything going on, so should be good on my end. 15:03:36 <smcginnis> #topic Tagging right fixes progress 15:03:37 <dhellmann> no concerns here 15:03:52 <fungi> no planned infrastructure disruption either 15:04:02 <smcginnis> fungi: OK, great! 15:04:10 <smcginnis> ttx: Was this your topic? 15:05:42 <ttx> yes 15:06:01 <ttx> so... ec2api was done already 15:06:23 <ttx> For dragonflow, I looked it up and they did not do a queens release 15:06:42 <dhellmann> that's unfortunate 15:06:44 <smcginnis> Surprised we missed that until now. 15:06:48 <ttx> which makes me question what to do there 15:06:51 <fungi> i thnik they may be one of those projects which trail substantially 15:06:54 <fungi> er, think 15:07:18 <fungi> as in they develop against the latest neutron release rather than against master? 15:07:27 <ttx> given where it sits on the map (neutron plugin) I would be inclined to consider it for unofficalnessity 15:07:41 <smcginnis> At least I don't see them listed as part of the coordinated release: https://releases.openstack.org/queens/index.html 15:07:49 <ttx> probably one to prioritize in the "health" TC activity 15:08:09 <ttx> last release was 9 months ago 15:08:10 <dhellmann> I'll make a note of that now 15:08:32 <ttx> They have a bunch of commits though, so it's not "dead" 15:08:35 <smcginnis> Ah, I seem to recall some internal mention of this. 15:09:11 <fungi> 2017-09-01 for their 4.0 release 15:09:12 <dhellmann> they have 393 commits since that last release 15:09:25 <dhellmann> that seems pretty active, are they just not tagging? 15:09:35 <ttx> It's hard to consider it a part of openstack if it's not released basically 15:09:40 <dhellmann> yeah 15:09:56 <fungi> so they basically released 4.0 immediately after the coordinated pile release 15:10:05 <ttx> I'm not saying it should die... I just question the value of it being seen as a part of the product rather than a product add-on 15:10:07 <fungi> er, pike release 15:10:16 <smcginnis> So do we propose to move them out of governance? 15:10:29 <ttx> smcginnis: I'd wait for someone from teh TC to dive into it 15:10:35 <dhellmann> fungi : quite the freudian slip that was :-) 15:10:39 <fungi> hah 15:10:42 <smcginnis> :D 15:10:43 <ttx> We just need to prioritize it up 15:10:46 <fungi> and get some feedback from the dragonflow devs too, of course 15:11:09 <dhellmann> I've added a note to the "actively monitoring teams" section of the tc tracker in the wiki 15:11:09 <ttx> next up: 15:11:16 <fungi> looks like oanson is probably the most active contributor there (and has been pushing their tags) 15:11:16 <smcginnis> I can start a ML thread after the meeting raising the question. 15:11:24 <dhellmann> I intended to raise that in office hours next week when most everyone is back online 15:11:30 <smcginnis> dhellmann: ++ 15:11:33 <ttx> Rally -- I overlooked their response to the last ML thread about it 15:11:41 <ttx> http://lists.openstack.org/pipermail/openstack-dev/2018-April/129405.html 15:11:41 <dhellmann> smcginnis : a ML thread from the release PTL would be a good start, too 15:11:57 <ttx> basically arguing that non-cycle-based things should retain tagging rights 15:11:58 <dhellmann> I don't find either of those arguments particularly convincing 15:12:04 <smcginnis> #action smcginnis to start ML thread asking for Dragonflow status update 15:12:21 <dhellmann> but yeah, if they don't want to be listed on the releases.o.o site I guess I don't want to argue over it 15:12:37 <smcginnis> We do state now in the documentation that coming under governance means the release team must be the ones handling tagging. 15:12:37 <ttx> dhellmann: yeah, so my initial "ACL compliance" target was the openstack / openstack-operations buckets from the map[tm] 15:12:58 <fungi> so rally wants to keep their release notes in the tag message rather than in their repository content? 15:13:15 <ttx> they want to keep not depending on us, I think 15:13:43 <dhellmann> that seems like the real reason, yes 15:14:11 <dhellmann> we have some old data about releases. we could just remove that so the site doesn't appear to be out of date. 15:14:18 <ttx> I'm fine with the "I should be able to do things how I damn well please" mentality, but I also think things can live outside of the "openstack" official space 15:14:25 <fungi> right, the message indicates a dislike for the rules about when releases are appropriate, and a concern that there might not be release team members around to approve (urgent?) release requests 15:14:30 <smcginnis> ttx: Agree 15:14:39 <ttx> wanting both is what I disagree with 15:14:52 <dhellmann> ++ 15:14:53 <smcginnis> Sounds like this one should be "OpenStack hosted" and not officially governed if that's what they want. 15:15:03 <ttx> we hold "openstack deliverables" to a given standard, and that standard now includes managed released 15:15:32 <ttx> if only so that you can not make non-peer-reviewed release blunders that would stain the openstyck name 15:15:52 <fungi> infra being the main outlier, and as noted in the most recent ml thread we want to move most of the infra repos out of openstack officially too 15:16:16 <fungi> so i think it's a fine precedent 15:16:38 <dhellmann> so, does someone need to reply to that ML thread proposing the 2 options? either letting us manage the releases or removing rally from governance? 15:16:59 <smcginnis> Sounds like something the TC chair should address. :) 15:17:04 <ttx> I'd say that's a TC-level discussion 15:17:08 <fungi> agreed 15:17:16 <ttx> enforcing that rule that is now in governance or not 15:17:17 <dhellmann> ok, maybe our release team PTL will raise that then? :-) 15:17:36 <smcginnis> Hehe, OK. 15:18:11 <ttx> I think the key is what are the minimal stquality andards we want to enforce for things to be associated with the "openstack" 15:18:25 <ttx> damn it weechat+mosh stop eating my characters 15:18:49 <dhellmann> smcginnis : if you lay out the reasons why we have the rule, I will offer the option of rally leaving governance 15:19:25 <ttx> I think those quality standards now include peer-reviewed releases basically 15:20:05 <ttx> and that is what that resolution said 15:20:18 <smcginnis> dhellmann: Are you thinking as a reply on that thread? Or should I bring this up in the next office hours? 15:21:06 <ttx> reply to thread would be good 15:21:10 <dhellmann> smcginnis : let's start by reviving the thread. I'll make sure the tc-members see it. 15:21:16 <fungi> i think if it's brought up on the thread as a discussion, that works 15:21:26 <dhellmann> maybe add [tc] to the subject if it isn't already there 15:21:27 <ttx> Andrey felt like his answer went into deaf ears 15:21:27 <smcginnis> OK, sounds like a plan. 15:21:55 <ttx> I for one missed it 15:22:05 <ttx> ok, next topic! 15:22:10 <smcginnis> I saw it, but didn't plan on responding right away. 15:22:10 <fungi> yeah, i honestly don't remember seeing his reply 15:22:23 <fungi> i probably did read it and immediately forgot 15:23:04 <dhellmann> yeah, I'm not sure why I glossed over that one 15:23:17 <smcginnis> #action smcginnis to respond to http://lists.openstack.org/pipermail/openstack-dev/2018-April/129405.html with reasons why governed projects need to use the release team 15:26:02 <smcginnis> ttx: Want to discuss the aclfixer patch? 15:26:15 <fungi> so what's the deal with the puppet-pacemaker addition then? just needs to have the acl updated to conform or is that an indefinite blacklist? 15:26:15 <ttx> should be straightforward 15:26:27 <ttx> just needs to be fixed, I think 15:26:30 <dhellmann> yeah, I wanted to ask about that, too 15:26:45 <dhellmann> should we just fix their acls? 15:26:47 <ttx> I could post the fix and update that aclfixer patch 15:27:10 <ttx> just wanted to have time to dive deeper into it see where that came from 15:27:18 <ttx> to prevent it from happening again 15:27:33 <dhellmann> ok 15:27:42 <ttx> I'll post the patch and update the aclfixer list 15:27:51 <ttx> so don't approve just now 15:28:03 <smcginnis> OK, sounds good. 15:28:23 <smcginnis> #topic Stable library maintenance 15:28:26 <ttx> OK, so I realized this morning that I signed up for that one, and wanted to discuss it for a bit 15:28:50 <ttx> what would be your definition of a stable library ? Wouldn't we expect req bumps on those ? 15:29:05 <dhellmann> since we no longer sync them, we might not have any 15:29:09 <dhellmann> think oslo.i18n 15:29:33 <dhellmann> that's basically never going to see new features, and will only get bug fixes if there are any, but it's a thin wrapper around gettext, so there aren't likely to be many 15:29:39 <smcginnis> I think we would need them. Aren't the older ones still just using g-r and u-c? 15:30:04 <dhellmann> we don't change g-r on older branches 15:30:10 <dhellmann> they do use u-c, but those aren't synced 15:30:26 <ttx> ok 15:30:33 <ttx> so in this case... 15:31:00 <ttx> the less-disruptive way is probably to allow multiple stable branches to be created from a given master release 15:31:14 <dhellmann> we'll have a lot of somewhat-but-less stable libs, but oslo.i18n seems likely to stay as it is for ages 15:31:31 <ttx> We would just branch it from the last master release if nothing was pushed there since last stable 15:31:59 <ttx> In case new things land in master, we just need to jump n stable branches in that versioning 15:32:05 <ttx> x.y+n.0 15:32:19 <ttx> Any obvious issue with that strawman ? 15:32:33 <ttx> I feel like not creating branches until they are needed is actually more painful 15:32:37 <dhellmann> it sounds like extra backporting work 15:32:48 <ttx> backporting? 15:33:03 <dhellmann> if we have a queens version that ends up being used for 3 cycles, then we have a fix, would we backport it to all of those other empty branches? 15:33:13 <dhellmann> why not just backport from master to queens and do 2 releases? 15:33:20 <fungi> seems like it might be challenging to predict version numbers for those branches? 15:34:09 <ttx> hmm let me think... we rely on constraints rather than branch names in tests, right 15:34:35 <dhellmann> in unit tests, yes 15:34:36 <dhellmann> the branch names are used when testing changes to the libraries in the integration tests 15:34:44 <dhellmann> ideally we'd be able to use version numbers instead of series names, but that's a lot of tooling updates 15:34:53 <dhellmann> stable/2.0 instead of stable/queens 15:34:56 <ttx> I fear that tracking which branch is used for where might be a bit challenging 15:35:03 <ttx> What we really need is branch aliases :) 15:35:05 <dhellmann> the telemetry team tried that and it turned into a lot of work 15:35:13 <fungi> say we branch stable/rocky and then later branch stable/stein from the same commit... later we discover a bug (security vulnerability fix?) which needs to be committed in master and backported... what do we tag for the stable branch release versions? 15:35:51 <ttx> so.. master has 1.18.0 15:35:57 <dhellmann> if the version before stable/rocky was 1.0.0 then the version on stable/rocky is 1.1.0 and the version on stable/stein is 1.2.0 15:35:59 <ttx> you create stable/rocky from that 15:36:04 <dhellmann> the release tools already require that 15:36:17 <ttx> then create stable/stein from that 15:36:27 <smcginnis> I have to admit, I'm not following all this. Why do we need to do something different than our current stable library releases? 15:36:32 <ttx> then you fix the thing in master and release 1.20.0 15:36:38 <fungi> so as long as we pick the stable/rocky version number before we pick the stable/steni version number, it could work out 15:36:41 <ttx> then 1.18.1 and 1.19.0 15:36:44 <dhellmann> fungi : yes 15:37:08 <fungi> but we need to make sure we don't tag something in stable/stein before stable/rocky 15:37:14 <ttx> what dhellmann says is that it's preferable to do 1.19.0 on master and 1.18.1 for rocky AND stein 15:37:19 <fungi> or we could paint ourselves into a corner with version numbering 15:37:28 <dhellmann> smcginnis : I guess the question is, do we need to create empty branches when there were no changes for several series? 15:37:35 <ttx> fungi: that's why I suggest 1.19.0 15:37:51 <dhellmann> ttx: right, we would just use the rocky version in the stein series, too 15:38:02 <smcginnis> dhellmann: Just speaking from experience with os-brick, we always create a stable branch. 15:38:04 <dhellmann> I don't know if that's going to be confusing for downstream folks, but that's how we treat 3rd party tools 15:38:24 <smcginnis> Of course, it always does have changes, so I guess that's the key here. 15:38:32 <fungi> ttx: right, my example could be extended to include stable/t and we need to make sure that we pick version numbers across r, s and t which are in order then 15:38:39 <dhellmann> right 15:38:45 <smcginnis> But I think we should still create branches each release for any "active" libs. 15:38:50 <ttx> I'd say the 3 options are 15:38:55 <fungi> so can't just rely on incrementing y in x.y.z 15:39:05 <dhellmann> fungi : yeah, we'll have to be careful with the reviews. or we could make the tool smart enough to detect those errors 15:39:12 <ttx> 1/ force "artificial" releases even if there are no changes and just reuse same process 15:39:30 <dhellmann> if we create a branch, there will always be at least 1 patch containing the .gitreview update 15:39:35 <ttx> 2/ do not force releases, but still create branches from latest releases before they are even needed 15:39:51 <ttx> 3/ do not force releases, and reuse stable branches from cycle to cycle 15:40:34 <ttx> 3 is conceptually the best, but I have a hard time to wrap my head around potential consequences 15:40:35 <smcginnis> 1 seems the safest to me. 15:40:48 <ttx> 1 is the safest but weirdest when there are no more changes 15:40:56 <ttx> 2 is a trade-off 15:41:29 <dhellmann> 2 is the case where we have to be careful with versions, as fungi pointed out 15:41:29 <ttx> do we agree on the 3 options ? 15:41:37 <fungi> yeah, #1 while safe deos also seem like a treadmill of process-imposed busywork 15:41:46 <dhellmann> I suppose option 4 is to not worry about stable branches at all for those things 15:41:47 <fungi> s/deos/does/ 15:42:01 <ttx> switch them to independent ? 15:42:10 <dhellmann> and option 5 is to not do anything and create the stable branches when we need them 15:42:39 <dhellmann> (option 5 is like 2, except for the timing) 15:42:46 <ttx> dhellmann: you mean create all the missing branches once you need them ? 15:42:50 <dhellmann> right 15:43:04 <dhellmann> I'm not sure it's a good idea, but for completeness sake I thought I'd mention it 15:43:04 <ttx> It's a 2bis 15:43:07 <fungi> i want to say there were some drawbacks to #4 and #5 (we did them in the past at various times) but trying to recall what the problems were now 15:43:37 <ttx> is #4 like switching things to be independent ? 15:43:45 * ttx documents in etherpad 15:43:50 <fungi> certainly not creating stable branches leads to problems when you have a new release you don't want used by stable branches of other projects 15:43:56 <dhellmann> well, we have stable branches for independent projects sometimes 15:44:19 <dhellmann> fungi : with the constraints system in place, new releases can be controlled 15:44:24 <fungi> though i suppose constraints on those stable branches help 15:44:29 <fungi> yeah, that 15:44:54 <smcginnis> Yeah, we're probably in better shape now to handle these. 15:44:54 <fungi> we didn't really have constraints widely used back when we were encountering the issues which caused us to force stable branches for all libs 15:44:54 <dhellmann> in fact 3 would require extra manual work to update u-c in stable branches 15:45:06 <dhellmann> right 15:45:24 <fungi> so... 4/5 actually seem sort of okay and would certainly be less work 15:45:46 <smcginnis> Not to be lazy, but less work is probably better at this point. 15:45:52 <dhellmann> option 5 feels appealing, because it lets us do the work we need to fix something at the time it's broken without having to do extra work up front 15:46:00 <fungi> i think it would end up being #5 because there will be critical fixes you need backported to an older version used by stable-branch projects 15:46:12 <dhellmann> otoh, doing it up front means it's done as part of a batch, so we're ready when things are broken 15:46:42 <dhellmann> so I'm sort of torn between 5 and 2 15:46:43 <ttx> yeah, it's self-imposed work, but part of a routine 15:46:46 <fungi> well, release requests can come with branching requests, right? so it's not really that much extra convenience to have the branches precreated 15:46:53 <ttx> I renamed 5 to 2bis in the etherpad 15:47:02 <dhellmann> fungi : they need the branch before they can backport a fix 15:47:14 <dhellmann> ok 15:47:17 <ttx> since it's close to 2 and helps estimating disruption potential 15:47:21 <fungi> oh, good point. so it's two release requests 15:47:35 <fungi> create the branch, backport the fix, tag the release 15:48:05 <dhellmann> they're likely going to want a release from master, and the branch request could come in that patch, but at that point it's not really much extra effort to make it separately 15:48:21 <dhellmann> maybe we try 2bis and if we run into problems we switch to 2? 15:48:21 <fungi> i still feel like #5 is preferable to #2 _if_ the frequency of backports is very low 15:48:26 <ttx> We don't have to decide now, just wanted to lay down the options to give it some extra thought 15:48:39 <smcginnis> Was just going to say, we don't need to decide today. 15:48:41 <dhellmann> I'm expecting a very small number of things in this category, and for those things to be very stable. 15:48:54 <smcginnis> Let's give it a bit and let things sink in and see if we think of any other concerns. 15:48:58 <ttx> One issue is how we "mark" those 15:49:05 <dhellmann> ++ to contemplation 15:49:16 <dhellmann> ttx: yeah, maybe a new release model? 15:49:31 <fungi> it's also possible to switch from #2 to #2bis in the future if we discover that #2 isn't buying us much for the additional effort 15:49:38 <dhellmann> fungi : good point 15:49:53 <dhellmann> it seems like we can switch between any of the options, really 15:50:06 <fungi> sure 15:50:08 <smcginnis> Yeah, 2bis is really just lazy loaded 2. 15:50:17 <ttx> ok... let's it simmer a bit. Next topic ? 15:50:19 <dhellmann> ++ for being lazy 15:50:36 <smcginnis> Any aspect of extended maintenance lib releases we should consider with this? 15:50:46 <smcginnis> I thought that was actually the topic at first glance. 15:51:01 <smcginnis> Probably the same applies? 15:51:20 <ttx> brain cells level: 0 15:51:22 <dhellmann> I think we decided to change the devstack job to install everything from source by default for EM branches, didn't we? 15:51:47 <smcginnis> Was that the decision in the session. I couldn't remember if we came to a decision. 15:52:01 <ttx> that was the consensus there yes 15:52:07 <dhellmann> I felt like it was the preferred option, although I'm not sure if someone stepped up to write that change. 15:52:20 <dhellmann> it may have been another case of dealing with it when it actually comes up 15:52:34 <smcginnis> OK, that works then. We can move on. 15:52:45 <smcginnis> #topic Planned time off 15:53:11 <ttx> I realized my summer vacation is on FF week, again 15:53:20 <smcginnis> I know I will be gone R-7, but that should be a quiet week. 15:53:21 <ttx> also in Europython week 15:53:30 <smcginnis> I shouldn't have any conflicts on FF week (R-5). 15:53:48 <dhellmann> I'll be traipsing all over scotland that week 15:53:51 <ttx> I refreshed my travel schedule for the rest of the cycle 15:53:56 <smcginnis> dhellmann: Nice! 15:54:00 <d0ugal> EuroPython \o/ 15:54:16 <apetrich> \o/ 15:54:18 <dhellmann> well,not so much "all over" as "Edinburgh and Glasgow" 15:54:49 <smcginnis> So I guess I'll be holding down the fort that week with whatever help I can get. 15:55:08 <smcginnis> And will be definitely leaning on fungi if any release/infra issues come up. :) 15:55:48 <fungi> i'm gone for the second half of july... trying to see what that works out to on the release schedule 15:55:50 <smcginnis> This cycle has been much more reliable with those things, so hopefully this is no issue. 15:55:53 * smcginnis knocks on wood 15:56:11 <ttx> it's a good test at least 15:56:22 <smcginnis> OK, then I can just cry for help in -infra if something happens. :D 15:56:32 <smcginnis> But not feeling too worried. 15:56:36 <fungi> yeah, i disappear in the middle of r5 and return in the middle of r3 15:56:40 <dhellmann> on conference days (the end of the week) I'll check in on irc periodically 15:56:49 <fungi> er, i mean r6 and r4 15:57:12 <smcginnis> dhellmann: That would be great. Don't want to distract you, but might be good if any questions come up. 15:57:33 <dhellmann> I'm on twitter and signal, too 15:57:51 <fungi> so yeah, my travel is july 18-31 according to my notes, though i'll have internet access during some of that 15:57:53 <smcginnis> I think I can hunt most of you down if I really need to. 15:58:13 * dhellmann turns on his cloaking device 15:58:19 <smcginnis> Hah ;) 15:58:29 <smcginnis> Anything else in the last minute+? 15:59:11 <dhellmann> nothing from me 15:59:13 <smcginnis> We need to talk succession at some point, but we have time yet. But time today. 15:59:23 <smcginnis> But not time today I mean. 15:59:34 <smcginnis> OK, thanks everyone! 15:59:45 <smcginnis> #endmeeting