13:01:33 <ttx> #startmeeting releaseteam
13:01:34 <openstack> Meeting started Mon Aug 24 13:01:33 2015 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:01:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:01:37 <openstack> The meeting name has been set to 'releaseteam'
13:01:59 <ttx> Had a number of topics ti discuss, with liberty-3 / feature freeze coming up next week
13:02:19 <ttx> The first one is the Oslo libraries stable branches
13:02:28 <ttx> following the recent thread
13:02:46 <ttx> I keep on getting more confused, maybe we changed too many things
13:03:05 <ttx> We had a plan out of that crazy session outside in YVR
13:03:11 <ttx> I posted it on the thread
13:03:18 <dhellmann> yeah, I'm reading that now
13:03:39 <ttx> So I'm surprised that we don't need stable branches to achieve that plan now
13:03:51 <dhellmann> I don't see dims on line right now
13:04:05 <dhellmann> I don't think anyone said we didn't need them?
13:04:16 <dhellmann> lifeless suggested we may not need requirements caps
13:05:12 <ttx> he said "stable branches are orthogonal to constraints IMO. If the only reason for the stable branch is the capped requirements, then I would not make the branch."
13:05:18 <dhellmann> because we'll have the constraints system to protect us at first, and we can add caps later for individual libs if we need to
13:05:29 <ttx> pretty much "we don't need them"
13:05:34 <dhellmann> yeah, but that's not the only reason
13:05:59 <ttx> right, which means we need to take them into account in the constraints checks
13:06:22 <dhellmann> as I pointed out later in the thread, landing changes in master becomes a mess if we don't have stable branches, because we have to test against the stable branch as well as master and then remove that testing when we have an incompatible change to make
13:06:56 <dhellmann> I think for the constraints what we do is leave requirements in libraries uncapped for good and leave requirements in servers uncapped *for now*
13:07:07 <dhellmann> create stable branches for everything (order to be determined)
13:07:16 <ttx> agreed. I just worry about that "Enable master->stable cross-check"
13:07:29 <ttx> which is very much to make contrsinats work in a stable-full world
13:07:37 <ttx> constraints
13:07:37 <sdague> dhellmann: I think it's just a trade off between managing backports vs. ensuring there are stable jobs on libs
13:07:37 <dhellmann> and then as we release new versions of the libraries, test them with constraint changes in the stable branches, and if they don't work introduce a cap instead
13:08:14 <dhellmann> sdague: the problem isn't making sure we have stable jobs, it's rushing to remove them when they start breaking our ability to land changes in master because we raise a minimum requirement somewhere
13:08:26 <sdague> dhellmann: right, gotcha
13:08:31 <dhellmann> sdague: which is sort of the reverse of what we have now, where master changes break stable
13:09:00 <dhellmann> I think with the constraints system fully in place, we're safe to not have caps until we actually discover an incompatibility somewhere, at which point it's relatively easy to do that
13:09:11 <dhellmann> it's not clear we have enough of the constraints work in place, yet, though
13:09:31 <ttx> dhellmann: regarding "as we release new versions of the libraries, test them with constraint changes in the stable branches, and if they don't work introduce a cap instead" ... does that mean we may end up with stable branches being tested with library versions that are on master ?
13:09:35 <dhellmann> the tox stuff, in particular
13:09:57 <dhellmann> ttx: we could have that if we explicitly update the constraints to allow it, yes
13:10:00 <sdague> dhellmann: right, I don't know exactly what lifeless has left
13:10:21 <ttx> dhellmann: under the current rules, we are supposed to accept any constraint change that passes tests
13:10:38 <ttx> stable branches would be frozen ?
13:10:40 <sdague> ttx: fwiw, that's the way this always used to work. The idea of stable branches in libraries is relatively new.
13:11:08 <dhellmann> sdague: yeah, he mentioned being blocked on some sort of governance change in his email to update the testing interface, but I think we can do that in parallel if not after the fact
13:11:41 <ttx> sdague: right, but that means we may release a .Z version on a stable branch and never test with it
13:12:17 <dhellmann> I would expect us to be making far fewer changes to the constraints on the stable branch than on master, but I wouldn't consider it frozen
13:12:31 <dhellmann> and I would definitely expect us to update it if we need a stable release of a library
13:12:44 <sdague> so, honestly, there are a lot of moving parts here. I'd suggest instead of everyone trying to keep it in their heads, get some bits written down with diagrams about the scenarios.
13:13:02 <dhellmann> given the rate at which we update some of these libs, I don't think we want to tell a deployer the only place they can get a fix for 1.2.x is in 1.15.x
13:13:15 <ttx> dhellmann: trying to see how that would work with the automated constraints bump suggestions.
13:13:27 <ttx> we would disable that on stable ?
13:13:45 <sdague> so, constraints shouldn't be landed in libraries
13:13:46 <ttx> or we would apply totally different rules on master vs. stable requirements ?
13:13:48 <dhellmann> ttx: maybe, that's a good question. We should get lifeless to chime in on this. I agree with sdague we need a good write-up
13:14:37 <ttx> sdague: we did diagrams and scenarios. It's just that we may not be were we were supposed to be
13:14:49 <sdague> ttx: url ?
13:15:26 <ttx> whiteboards in YVR that were posted on the ML at some point
13:15:26 <dhellmann> sdague: I think https://doughellmann.com/blog/2015/05/28/openstack-requirements-handling/
13:15:42 <dhellmann> those are the notes from the summit, at least
13:16:48 <dhellmann> re-reading them, they're pretty raw
13:16:53 <sdague> right, those are the notes. My suggestion is to turn them into a more detailed doc
13:17:01 <dhellmann> yeah
13:17:24 <sdague> because I think those notes can be interpretted differently, and some of the tricky specifics need to be fleshed out
13:17:45 <sdague> mostly for clarity, it's just a lot to keep in mental ram
13:18:24 <dhellmann> ttx: do you have time to start that, or should I put it on my list? Where do we want it to live?
13:19:29 <ttx> dhellmann: my gut feeling is that we built a system that is orthogonal to stable branches (like lifeless said), and putting stable branches back into it is likely to take more than the week we have to figure it out
13:20:00 <dhellmann> ttx: ok, I always understood the changes we were making to be on top of stable branches, so I'm surprised to hear that.
13:20:27 <ttx> dhellmann: in particular, my impression is that the system is designed to work with a single channel of releases
13:20:31 <dhellmann> if we don't create stable branches, we need to have test jobs for changes to master versions of all of the libraries running against stable versions of the server code
13:21:34 <dhellmann> and if we have that, we need some sort of quick response plan for the day a team is blocked making changes because we've introduced an incompatible dependency in master so that the master version of the lib no longer works with the stable version of the app -- that will happen, it happens all the time
13:21:37 <ttx> dhellmann: I think we need stable branches. Otherwise you need to branch whenever the library starts to implement changes that don't work with the previous release
13:21:44 <dhellmann> right
13:21:55 <ttx> either to mark that beforehand
13:21:56 <dhellmann> ok, I thought you were saying we didn't need them
13:22:05 <sdague> so, it's not a terrible thing to make libraries have that level of compatibility
13:22:07 <ttx> easier*
13:22:23 <sdague> that just means interfaces need a couple cycles to be dropped
13:22:34 <ttx> dhellmann: no, I'm surprised to discover lifeless thinks we don't need them.
13:23:03 <dhellmann> sdague: the issue is not usually with our code (though that does certainly come up), it's with wanting to add a dependency on a new thing (or a new version of a thing) we don't control, and having *that* break the stable branches
13:23:42 <ttx> stablea branches are just about freezing the state every 6 months so that maintaining it working is easier
13:23:47 <sdague> dhellmann: ok, so, perhaps documenting some cases of when that happened will make it more clear
13:25:04 <ttx> dhellmann: ok, I'll start a thread on the stable/release/requirements process, beyond only Oslo
13:25:20 <ttx> no point in discussing that without Robert
13:25:23 <sdague> anyway, probably worth figuring out lifeless's plan here
13:25:24 <dhellmann> yeah
13:25:25 <sdague> yeh
13:25:54 <ttx> Second topic I had is about intermediary-released thingies and RCs
13:26:16 <ttx> Historically Swift was the only thing we did intermediary-releases for (beyond libs)
13:26:45 <ttx> We would just push a tag there
13:26:56 <ttx> but the "coordinated" swift end-of-cycle release still used RCs
13:27:10 <ttx> we would create a release branch there and tag rc1
13:27:28 <ttx> To be consistent I think we need to just tag there
13:27:39 <ttx> the supposedly-final release
13:27:54 <ttx> and bump Z in case we need to have another "candidate"
13:28:17 <ttx> in summary...
13:28:43 <ttx> for projects with intermediary releases, the fnal release candidates are just intermediary releases
13:29:14 <ttx> and you just do another one on the stable/liberty release branch if it's too broken.
13:29:18 <dhellmann> yes, I think that's what I was expecting. release:cycle-with-intermediary projects don't tag betas or RCs, just actual releases.
13:29:33 <ttx> right. That means we need to communicate that to Swift
13:29:37 <ttx> since that's a change to them.
13:29:44 <ttx> I'll talk with notmyname about it
13:30:15 <ttx> I think they never wanted to have the final release being different than others anyway, but better doublecheck
13:30:16 <dhellmann> if he objects, I wouldn't have a problem with making that process change next cycle
13:30:36 <ttx> dhellmann: it came up when I started updating the rc*.sh scripts
13:31:03 <dhellmann> ok
13:31:20 <ttx> #action ttx to reach out to notmyname about the need for RCs for the "final" swift release of the cycle
13:31:45 <ttx> #action ttx to start a wider thread about the stable/release/requirements process, beyond only Oslo
13:32:29 <ttx> dhellmann: my next topics were about stable branch releases and release notes. But there may be other "urgent" topics to discuss before ?
13:33:06 <dhellmann> I don't have anything urgent, so let's keep going with your agenda
13:33:41 <ttx> #topic Stable branch releases from liberty on
13:34:04 <ttx> Current state of that is well summarized as options in :
13:34:37 <ttx> http://lists.openstack.org/pipermail/openstack-dev/2015-August/072603.html
13:34:50 <ttx> Tag-every-commit vs. Time to time tagging
13:35:15 <ttx> Have any preference ?
13:35:26 <ttx> At this stage I'm willing to opt for the less disruptive and less work
13:35:36 <ttx> i.e. "Time to time tagging"
13:35:54 <dhellmann> yeah, I think time-to-time makes sense
13:36:03 <dhellmann> that can always devolve into every commit, if we need it to
13:36:06 <ttx> I can see how it fails to reach the original objectives, so I sympathetize to Daviey
13:36:16 <ttx> but yeah, easier to evolve from there
13:36:18 <dhellmann> but I'm also inclined to see how far we can stretch that time between tags
13:36:24 <ttx> good incremental step
13:36:34 <dhellmann> maybe with projects or distros asking for tags when they feel they are needed
13:37:17 <ttx> Also a good thing to have release automation fully in place and see how much it changes things
13:37:47 <dhellmann> yeah
13:37:58 <ttx> #agreed Release team leaning toward "Time to time tagging" for stable/liberty as a more realistic objective and good incremental step
13:38:14 <dhellmann> oh, on stable branches we do post-versioning, right?
13:38:40 <ttx> I think we currently do pre-versioning
13:39:00 <ttx> but I don't think that's hard to change there
13:39:08 <dhellmann> yeah, we just need to remove the version string from setup.cfg
13:39:36 <ttx> anything else before we move to stable release notes ?
13:40:15 <ttx> I'll take that as no
13:40:16 <ttx> #topic Stable release notes
13:40:22 <dhellmann> no, sorry, phone rang
13:40:26 <ttx> So on this topic there might be more work involved
13:41:30 <dhellmann> I like the idea of having the release notes in tree. The scheme lifeless described seemed a bit more complex than necessary
13:41:39 <ttx> I think the two solutions are: lifeless's YAML directory for semi-continuous release note generation, and a single file in the root of the directory
13:41:41 <dhellmann> could we not just have a relnotes subdir under doc/source and let sphinx build the final document for us?
13:42:01 <ttx> oh, you're suggesting something in between now
13:42:18 <dhellmann> yeah, we get the merge protection of lifeless' yaml proposal without having to build any new software
13:42:54 <dhellmann> we could even have relnotes/$SERIES/*.rst and just keep the notes published forever
13:43:17 <ttx> dhellmann: yeah, I was thinking that part (determining which release notes to build) was a bit over the top
13:43:58 <ttx> dhellmann: are docs generated as part of the tarball creation ?
13:44:08 <dhellmann> I think so, but I'm not certain
13:44:39 <dhellmann> fungi: do you know if docs are built as part of tarball creation? ^^
13:44:49 <ttx> I can check really quick
13:45:25 <dhellmann> I don't get them in an sdist
13:45:42 <ttx> yeah, can't find them in built tarballs on tarballs.o.o either
13:47:04 <dhellmann> ok, so that may not be optimal
13:47:34 <ttx> So... we can still brainstorm solutions there. I guess plan B is to edit a single file (or let whoever triggers the time-to-time release update it before tagging)
13:48:08 <ttx> so we have a backup plan if we can't come up with something good before stable/liberty is cut
13:48:47 <dhellmann> right
13:49:12 <dhellmann> another benefit of keeping them in tree is we could encourage reviewers to require them as patches are made
13:49:37 <ttx> yeah, that's the main argument for it
13:49:40 <dhellmann> of course it opens us up to having notes changed for clarity after the fact, and then those changes not being backported, but I think we can live with that
13:50:20 <dhellmann> oh, I thought we were doing this because without official stable releases we didn't have anyone doing the gathering
13:50:25 <ttx> How would you reconcile building release notes continusouly and post-versioning ? We would generate a RELEASENOTES-NEXT.rst that would contain untagged notes ?
13:51:31 <dhellmann> if we go with the sphinx idea and group the notes by release series, we can have an N directory until N gets its name, and then that can be renamed. Or we could just use the letters to avoid having URLs change
13:52:06 <dhellmann> then release notes would be published live in the docs for the project as soon as the patch lands, and would be included in a discoverable directory afterwards
13:52:35 <dhellmann> if we want one big release notes file in the tarball, to complement the file published on docs.openstack.org, we could make pbr build that for us
13:53:17 <dhellmann> it could do something with docutils to convert the rst to plain text and concatenate the results together or something
13:53:18 <ttx> hmm, you need release notes per version, not per series
13:53:29 <ttx> although you can group them in a single file per series I guess
13:53:35 <dhellmann> oh, true
13:53:43 <ttx> you need to at least write them to a version
13:54:02 <ttx> hence my question on how to write to the "next" version in postver
13:54:14 <ttx> we'll likely need some placeholder
13:54:30 <dhellmann> I guess this is why lifeless' proposal was more complex :-)
13:54:35 <ttx> hehe
13:54:59 <ttx> He is often right, which makes it difficult to constantly disagree with him.
13:55:12 * dhellmann can't believe he has been building release notes management systems for nearly 20 years now
13:56:01 <ttx> ok, so I should have a side releaseteam-meeting with lifeless on APAC-friendly time (my tomorrow morning) to clarify both things
13:56:15 <dhellmann> and I should re-read his proposal
13:56:19 <ttx> #topic Open discussion
13:57:14 <dhellmann> you saw that I submitted all of the release tag changes?
13:57:16 <ttx> Anything else ? I'm a bit panicked because it feels like we changed the engine to the train, I don't know how it works anymore, the train is full speed ahead and the next train station is next week
13:57:34 <ttx> yes I saw that. I'll have a deep look soon
13:58:00 <dhellmann> I'm a little worried, but I think the only thing we actually have to solve by next week is understanding the order to do the tagging/branching
13:58:03 <ttx> also, the mechanic of the train leaves on the opposite side of the world
13:58:09 <ttx> lives*
13:58:13 <jokke_> ttx: http://www.imdb.com/title/tt0477080/ ? :)
13:58:14 <dhellmann> the release notes thing we can always do in the stable branches directly, we don't need that in master, yet
13:58:16 <ttx> and I make funny english mistakes, too
13:58:46 <dhellmann> ttx: I read it with a french accent, so I understood what you meant ;-)
13:59:11 <dhellmann> what else is inducing panic?
13:59:49 <ttx> stable branch / RC / cut order. I think we'll fumble it again this time
14:00:14 <ttx> I'm missing too many pages of that book to be sure.
14:00:42 <ttx> Can't wait until I reach firefighting stage and not panic anymore
14:00:52 <ttx> #endmeeting