14:00:20 <dhellmann> #startmeeting releaseteam
14:00:21 <openstack> Meeting started Fri Apr 15 14:00:20 2016 UTC and is due to finish in 60 minutes.  The chair is dhellmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:24 <openstack> The meeting name has been set to 'releaseteam'
14:00:31 <dhellmann> courtesy ping for ttx, dims, lifeless
14:00:46 <ttx> o/
14:00:51 <dhellmann> our agenda is at the bottom  of https://etherpad.openstack.org/p/mitaka-release-tracking
14:01:02 <dhellmann> #topic release postmortem review
14:02:17 <dhellmann> I've started copying some of the actionable items from the postmortem over to the newton planning document
14:02:22 <dhellmann> I've annotated those where appropriate
14:02:49 <dhellmann> the merge-release-tags job failures are pretty straightforward, so I don't know that we need to discuss those
14:03:17 <dhellmann> we should try to think about ways to mitigate the fact that we only run the release process for most projects twice a month
14:03:35 <dhellmann> maybe now that we have the release-test repo we can do the test work the week before each milestone/deadline?
14:04:56 <ttx> hm
14:05:46 <dhellmann> we can take that as it comes, if we change fewer things maybe it won't be as necessary
14:05:53 <dhellmann> though I expect we'll want to test the final release process
14:05:54 <ttx> It's tricky. We should I think determine what code paths in the automation only run on release day
14:06:05 <ttx> and apply extra testing / care to that
14:06:27 <ttx> But as we are still changing things it's difficult to tell now
14:06:48 <dhellmann> right. I'm not sure we actually have that many parts of the code that *only* run on the final release
14:07:15 <dhellmann> there's the announcement email thing, but if we're going to turn those off for milestone projects that will be a new change
14:07:27 <ttx> code paths is not the right term
14:07:27 <dhellmann> we'll want to test that when we make the change :-)
14:07:33 <dhellmann> process paths?
14:07:40 <ttx> It's just taht we want things to behave slightly differently on release day
14:07:47 <ttx> like announcements
14:08:04 <dhellmann> right
14:08:05 <ttx> or at least we need to think about it
14:08:19 <ttx> maybe it's ok to have separate emails announcing on release day
14:08:26 <dhellmann> I've been making notes in https://etherpad.openstack.org/p/newton-relmgt-plan for a while
14:08:30 <ttx> but I just didn't expect them so early
14:08:52 <ttx> Also difficultt o judge the result from a test repo
14:08:53 <dhellmann> yeah, it's tricky to coordinate the timing of those automatic emails with the press release
14:09:17 <ttx> FOr example we didn't spot the "only rc1..release" stuff
14:10:10 <dhellmann> yeah, that result makes sense but it's not what we want
14:11:18 <ttx> So short of running a dry-run with copies of repos and alternate email addresses for destination...
14:11:30 <ttx> it's actually tricky to spot all the potential issues
14:11:36 <dhellmann> yeah
14:11:41 <ttx> food for thoughts
14:11:46 <ttx> nothing urgent
14:12:02 <dhellmann> I'm not sure what the "Master Sample config" item means, can you elaborate on that one?
14:12:27 <dhellmann> is that just that because we're doing post-versioning the version of master hasn't actually changed yet?
14:12:28 <ttx> so yeah, it's a variant on the "colliding versions" between stable and master
14:12:53 <dhellmann> ok, that should be sorted out when we tag the first milestone
14:12:54 <ttx> Our response to that was that it shouldn't collide, since you pick a branch and stay there
14:13:05 <ttx> BUT the pbr version also appears elsewhere
14:13:19 <ttx> like in sample config and doc versioning
14:13:32 <dhellmann> I suppose we could have projects tag an a0 or b0 version if they want
14:13:39 <ttx> so for master it ends up showing 13.0.1.dev187 instead of 14.0.0.dev187
14:14:05 <ttx> You can read the logs of the discussion with sdague
14:14:17 <ttx> he was the one who brought the issue up
14:14:25 <dhellmann> ok, I'll do that later
14:14:38 <ttx> yes, it will fall into place when b1 is tagged
14:14:59 <ttx> but that may be another reason to do early a0 tagging
14:15:23 <sdague> right, this is about things like docs which we publish per commit, and start diverging within weeks of the release, but look like they are explaining the stable branch, which they aren't
14:15:55 <dhellmann> yeah, maybe we should add a step to tag something on master with an alpha tag to raise that version #
14:16:06 <sdague> yeh, that could work
14:16:17 <ttx> and would also solve the issue for packagers in the same go
14:16:18 <sdague> I'm open to any possible solutions, just raised the issue
14:16:21 <ttx> might be worth the hassle
14:16:51 <ttx> sdague: it's good, we oversaw that issue (focused on the colliding versioned tarballs)
14:16:52 <dhellmann> we would want a commit that's on master but not on the new stable branch, so we would need a way to determine that when the branch is created
14:17:08 <ttx> hmm
14:17:13 <dhellmann> sdague : thanks for raising it
14:17:21 <ttx> yeah, that was the issue
14:17:24 <ttx> I remember now
14:17:37 <ttx> we had to push setup.cfg bump as the very first commit in master
14:17:49 <ttx> and then create stable from the /previous/ commit
14:17:54 <ttx> craeted a bit of a painful window
14:17:56 <dhellmann> yeah, switching everything to post-versioning was to avoid having to do that dance
14:18:23 <ttx> the new model didn't require any commit
14:18:31 <dhellmann> right
14:18:48 <dhellmann> so we can stage the alpha commit at any point after the branch is created, when the project has a new commit of some sort
14:18:48 <ttx> now we would require a commit, which would generate a conflict version, and then tag
14:18:51 <ttx> a bit dirty
14:19:15 <dhellmann> it wouldn't be a problem if we just plan to fix it retroactively
14:19:27 <dhellmann> we could go back now and find the commit that landed in master right after the branch and tag that
14:19:49 <ttx> sure but we'd still generate at least one conflicting/wrong version
14:19:57 <dhellmann> for newton, we just encourage liaisons to do that right after we branch
14:20:04 <dhellmann> to shorten the window
14:20:21 <ttx> so we shorten the window but don't eliminate it
14:20:31 <dhellmann> we also move it to a time before the final release
14:20:43 <dhellmann> so that by the time the release comes, master and the branch have different versions already
14:20:57 <ttx> hmm
14:21:01 <ttx> makes me think
14:21:09 <dhellmann> yeah, we should discuss this more at the summit
14:21:12 <ttx> when we branch post-RC1... what does pbr generate for master versions ?
14:21:33 <ttx> 0rc1.dev1234
14:21:35 <ttx> ?
14:22:11 <dhellmann> something like that, I think
14:22:17 <ttx> or 13.0.1.dev123
14:22:29 <ttx> anyway, yes we need to revisit that one
14:22:31 <dhellmann> I'd have to go test that
14:22:37 <dhellmann> yep, noted for summit work session
14:22:39 <dhellmann> moving on?
14:22:41 <ttx> with more neurons
14:22:43 <ttx> yes
14:22:57 <dhellmann> "During the release process once master is unfrozen, master u-c and g-r may have either bot proposed or from others updates with library versions cut from stable"
14:23:13 <ttx> that was from dims
14:23:22 <ttx> not sure what he had in mind there
14:23:28 <dhellmann> this time around I resisted updating constraints to put mitaka updates into the master test environment
14:23:42 <dhellmann> late in the game he convinced me that was a bad idea
14:24:03 <dhellmann> so I think next time we just need to remember that it's ok to approve those constraint changes
14:24:23 <ttx> oh ok
14:24:25 <dhellmann> same for the next one, "Once the release announce email is sent, we can start processing pending release requests from master etc."
14:24:27 <ttx> yeah
14:24:44 <dhellmann> I thought it would be better to just focus on mitaka, and not do any newton releases during the mitaka release week
14:24:51 <dhellmann> maybe we can be less strict with that
14:24:58 <ttx> yeah, it's more "if we can"
14:25:01 <dhellmann> right
14:25:24 <dhellmann> "Project teams that were aggressively backporting fixes clogged up the review lists so we didn't know which things were critical to land and which were not."
14:25:31 <ttx> yeah, not sure how to fix that
14:25:34 <dhellmann> that was mostly an issue with the neutron team, but some others did, too
14:25:44 <ttx> It's true that it doesn't facilitate our work
14:25:49 <ttx> two options
14:26:00 <ttx> commuinicatinbg that backports should not be proposed unless critical
14:26:02 <ttx> or
14:26:12 <dhellmann> if we encourage them to WIP the changes they can still backport and we can filter those out of what we look at
14:26:15 <ttx> forget about that and let PTLs do stuff
14:26:24 * dims back
14:26:30 <dhellmann> letting the PTLs deal with it on their own is also a good option
14:26:33 <ttx> currently we use teh pre-release ACL to reduce approval to PTL+us
14:26:53 <dhellmann> trying to track all of the projects was a lot of effort, and I'd like to distribute that more
14:26:55 <ttx> I ended up reviewing most of the changes to check that the change was in master
14:27:15 <ttx> some PTLs also just waited for our approval and pinged us
14:27:42 <ttx> we couls remove ourselves from the preocess, give the pre-release acl top the release liaison and giev them the guidance
14:27:54 * ttx topes
14:27:57 <ttx> typos
14:28:06 <dhellmann> pushing that review work back on the PTL/core team would also solve the issue we identified with infra of not wanting to have to make acl changes as part of the release process
14:28:26 <ttx> well, you would still have an ACL issue
14:28:38 <ttx> PTL/core is different from stable
14:28:47 <dhellmann> ah, true
14:29:06 <ttx> and I kinda like to put the release liaison on the critical path
14:29:07 <dhellmann> clarkb suggested creating a new group and changing its membership. I sort of like that, if we can automate the setup.
14:29:16 <ttx> because they check FFEs and all
14:29:38 <dhellmann> if we have foo-newton groups for all projects, we can start them out with us, PTL, liaisons and then eventually add the foo-stable group
14:29:45 <ttx> so...
14:29:51 <dhellmann> that's a lot more gerrit groups, though, so I don't know if we want to do that
14:29:56 <ttx> wouldn't be an issue if we had kept proposed/*
14:30:01 <ttx> but we didn't
14:30:23 <ttx> ACLs cover stable/*, we can't have them target stable/latest
14:30:26 <dhellmann> I suppose resurrecting that is another option to consider
14:30:43 <ttx> but yeah, we can brainstorm
14:30:52 <dhellmann> sure we can, we just have to replace the * with the series name, right?
14:31:04 <ttx> how do you do that without updating the acl ?
14:31:38 <dhellmann> we could add those acls at any point, which would remove the project-config patch from the critical path for making the release happen
14:31:48 <dhellmann> which would at least be a timing improvement over what we have now
14:32:04 <ttx> oh sure, you can create the acl in adva,ce
14:32:17 <ttx> you still need to remove it postrelease though
14:32:22 <dhellmann> I guess we couldn't do it in one place, though, since it would be a team-specific group
14:32:41 <ttx> I was surprised how many people asked for it just after release
14:32:43 <dhellmann> hmm, I thought we would just change the members
14:33:08 <ttx> Ah. You would create different groups for stable/newton, stable/ocata
14:33:12 <dhellmann> right
14:33:21 <ttx> and leave them all in the acl
14:33:31 <dhellmann> yeah, or clean them up when we delete old branches
14:33:32 <ttx> and/or consolidate once per cycle
14:33:38 <ttx> when you add the new one
14:33:44 <dhellmann> sure
14:33:47 <ttx> yeah, that sounds doable
14:33:56 <dhellmann> ok, more for the summit discussion
14:33:57 <ttx> one level of indirection
14:34:13 <dhellmann> "Communication with PTLs and Liaisons was spotty for the entire cycle."
14:34:34 <dhellmann> I have no good ideas here, aside from stressing that everyone needs to actually figure out how to read a lot of email.
14:34:50 <dhellmann> this will be a topic of discussion for the fishbowl session, for sure
14:35:05 <ttx> frankly... given the number of PTLs who didn't read my design summit scheduling emails... I don't know what to do
14:35:41 <dhellmann> the other thing I'm going to do is discourage you from chasing folks down
14:35:45 <ttx> Over the last days I had about 6 of them asking me how they can edit the titles on their sessions
14:35:47 <EmilienM> ttx: that is something that could be addressed during a TC meeting
14:36:11 <ttx> EmilienM: how ? By throwing out of the tent projects that have PTLs that won't read their emails ?
14:36:13 <EmilienM> it's part of PTL's duty to take care of scheduling sessions
14:36:19 <EmilienM> I don't know :(
14:37:14 <ttx> The problem is that email is stateless notification. You don't have a list of standing items to process and check them one by one
14:37:16 <dhellmann> I want folks to come to rely on the weekly countdown emails for reminders and notices, so we can avoid doing lots of 1:1 discussions. We need to figure out how to train people to read those. :-/
14:37:25 <ttx> For that you need a task tracking system
14:37:34 <ttx> one that would be part of your daily workflow
14:37:56 <ttx> I think we are paying the fact that we don't have a good solution for task tracking today
14:38:01 <dhellmann> interesting insight, maybe that's what's missing -- copy the emails to your task system
14:38:17 <ttx> and we trained PTLS to ignore Launchpad/StoryBoard/.whatever
14:38:30 <ttx> at least as a work organization tool
14:38:31 <dhellmann> maybe we should experiment with something like trello
14:38:46 <EmilienM> mhh we tried in puppet group
14:38:53 <EmilienM> lot of people don't want to create an account
14:38:56 <dhellmann> that might help with another issue I noted, which was that keeping track of all of the state of all projects for RCs was a lot of manual work
14:39:09 <ttx> (FWIW it's the same with the periodic jobs failure, you want them to go in a queue and get processed by someone)
14:39:33 <ttx> email is good for discussion, not for deadlines/ work notification
14:39:36 <dhellmann> EmilienM : that's not going to be an acceptable excuse for whatever we decide to use
14:39:49 <ttx> unless you specifically set up specific folders and use them for task tracking
14:39:50 <EmilienM> dhellmann: right.
14:40:07 <dhellmann> ttx: yeah, I put task emails into evernote, myself
14:40:38 <ttx> I do use inbox 0 / action folder and process them all, but I realize I'm a bit special
14:41:11 <ttx> some people just go one week in vacation and just mark as read the 1500 emails when they come back
14:41:15 <dhellmann> would it make sense to do something like trello,  or try to figure out a way to use launchpad?
14:41:17 <ttx> some/most
14:41:24 <EmilienM> ttx: same ;-)
14:41:57 <dhellmann> maybe we can brainstorm more by the summit
14:41:57 <EmilienM> we have the trello/storyboard/launchpad discussion at every summit
14:42:03 <ttx> dhellmann: we need to think about something, because no amount of email/blog communication will cut it
14:42:19 <dhellmann> yeah, I like your insight that we need a task tracker
14:42:27 <ttx> you want to be able to put work on the PTLs/liaisons queue
14:42:32 <ttx> we have no way to do that
14:42:54 <ttx> the only way is to ping them and put the work in their face (several times)
14:43:07 <ttx> which does not scale
14:43:15 <dhellmann> I also need a good way to visually see status, and I don't think launchpad is a good answer for that, though I've seen some tools for copying LP to trello
14:43:32 <ttx> Storyboard has boards now
14:43:47 <dhellmann> I've never been able to figure out how to use storyboard, it surprises me too often
14:43:53 <ttx> anyway, also not a one-hour discussion ;)
14:43:55 <dhellmann> yeah
14:44:06 <dhellmann> #topic Release model tag changes
14:44:14 <EmilienM> o/
14:44:23 <ttx> In the meantime we can continue with the current set of tools, but we won't have 100% success until we switch to something else entirely
14:44:26 <dhellmann> I wrote up the release:cycle-trailing tag, and most of the deployment tool folks seemed to like it
14:44:36 <ttx> yeah, that sounds reasonable
14:44:43 <dhellmann> #link https://review.openstack.org/306037
14:44:47 <ttx> still a bit fuzzy on what is the release and what is trailing it
14:44:48 <EmilienM> that's a very good proposal
14:45:04 <dhellmann> I picked 2 weeks out of the air, so I'm willing to change it if we like a different number better
14:45:15 <ttx> I still would prefer to make it clear that the "release" is the thing that was out, and we have trailing projects
14:45:18 <dhellmann> I figured being generous with the upper bound was a good way to start
14:45:35 <ttx> we could still display them on the same page though
14:45:37 <dhellmann> yeah, we'll need to update the releases.o.o page to separate these projects out into their own section
14:45:42 <dhellmann> right
14:46:03 <ttx> I just don't want people to visit the page on mitaka release day and then one week later and see new projects added
14:46:17 <ttx> but if it's clear that those are trailing projects, it's fine
14:46:30 <dhellmann> it looks like there are a few more tag changes, so I'll go review those
14:46:46 <dhellmann> the one for adding the managed tag to congress may take some thought
14:46:50 <ttx> so, non-related to trailing are:
14:46:57 <ttx> Change magnum deliverables release tags https://review.openstack.org/300828
14:47:09 <dhellmann> I don't want to just add managed projects until we see that they're actually participating and their liaison is active
14:47:31 <ttx> I guess the magnum one is ok
14:47:52 <dhellmann> yeah, I'm happy with that one
14:48:01 <ttx> we still don't have a good story to display something that went from independent to intermediary though
14:48:48 <ttx> so we can either figure the display out, or retroactively add them to previous releases
14:49:02 <ttx> for the same reason I explained above I would rather not rewrite history
14:49:18 <ttx> although it's less of an issue for libs
14:49:31 <dhellmann> I thought we might try adding project pages to combine all of the deliverables for a given project, so maybe we can address the model change there
14:49:54 <dhellmann> so we would leave the deliverable files as they are, and add more rendering views
14:49:55 <ttx> I would add an entry to the independent file taht would explain and point to the new thing
14:50:08 <dhellmann> that's a good idea, too
14:50:15 <ttx> i.e. next versions are tied to the cycle, see there
14:50:31 <ttx> because that's the only place that would look funny
14:50:38 <ttx> anyway, approving this one
14:50:44 <EmilienM> can we talk about the puppet one please?
14:50:47 <EmilienM> #link https://review.openstack.org/297382
14:51:17 <dhellmann> EmilienM : you don't want to use the trailing model?
14:51:24 <EmilienM> I think we can follow release:cycle-with-intermediary
14:51:32 <EmilienM> we don't need 2 weeks more
14:51:51 <dhellmann> what benefit do you see of going with c-w-i instead of c-t?
14:52:01 <EmilienM> and if in 6 months we realize it was a mistake, we'll propose a change. But for Mitaka, we did it 1 week before.
14:52:08 <dhellmann> you don't have to wait the 2 weeks for a release, you know
14:52:37 <dhellmann> well, we would actually prefer that you not change models a lot, because as ttx was just pointing out it causes issues reporting what is going into a project
14:52:44 <dhellmann> s/project/release
14:52:53 <EmilienM> there is no benefit, it's just our group/project is able to make releases on time
14:53:07 <dhellmann> ttx: what if we put the release model in the deliverable file to make the rendering easier?
14:53:08 <EmilienM> but I don't mind using trailing model.
14:53:40 <ttx> EmilienM: let's wait for the end of the trailing discussion, maybe it will evolve one way or another
14:53:49 <EmilienM> okay
14:53:56 <dhellmann> EmilienM : ok, I think we'd prefer all deployment tools to use the trailing model for consistency, but this is new so I don't want to force anything
14:53:56 <ttx> If it ends up describing packaging things, you might want to be displayed in that category
14:54:07 <dhellmann> right
14:54:13 <ttx> dhellmann: we mighy want to bake that in the tag description
14:54:15 * EmilienM will wait
14:54:25 <ttx> dhellmann: we actually don't want random things to be trailing
14:54:34 <ttx> only things that package other stuff
14:54:35 <dhellmann> I thought I did cover that somewhat
14:54:49 <dhellmann> I didn't want to say "only packaging" but i did say most projects shouldn't use it
14:54:55 <ttx> It's a bit between a type: and a release: thing
14:55:15 <dhellmann> yeah, that's why I tried to express why it might be used, rather than the types of projects that should use it
14:55:18 <odyssey4me> that makes some sense, and yeah - perhaps it should describe specifically that the tag is primarily designed for packaging and deployment projects
14:55:33 <ttx> I like that those would all appear in their own section
14:55:53 <odyssey4me> ttx I was thinking about that earlier today, and I think I agree
14:56:12 <dhellmann> we group by type tags now, IIRC, so we can do that easily
14:56:13 <ttx> and sections are type-driven currently (pages are model-driven)
14:56:36 <ttx> the problem is that "trailing" can be milestones-driven or intermediary-driven
14:57:04 <dhellmann> yeah, I was trying to avoid having 2 separate tags
14:57:08 <ttx> and the model doesn't describe that
14:57:26 <ttx> We could propose it as a type tag that influences how the model operates
14:57:57 <dhellmann> that was the original proposal, but we agreed having a separate model entirely would be simpler for explaining what the difference was
14:58:11 <dhellmann> we put all of the cycle based things on the cycle pages, regardless of their actual model
14:58:15 <dhellmann> and we split that by type
14:58:19 <ttx> i.e. introduce the type and then explain in each model how the type changes the model
14:58:22 <odyssey4me> perhaps the 'type: deployment' and 'type:packaging' tags could inform the way the release cycle is applied?
14:58:34 <ttx> that would still be a single tag
14:58:52 <dhellmann> ttx: but it means we have a bunch of exceptions in all of the model tags
14:59:01 <ttx> yeah, just brainstorming out loud
14:59:12 <dhellmann> it also means we're not giving ourselves flexibility if we have another type that needs this model
14:59:42 <ttx> depends if we need to track the distinction between with-milestones and with-intermediary in trailing
14:59:48 <ttx> I fear we need to
14:59:49 <dhellmann> we could say that projects can combine cycle-trailing and cycle-with-* to indicate which of the latter they use
15:00:08 <dhellmann> I'm not sure we do, it's just a matter of how important they consider those milestone deadlines, right?
15:00:14 <dhellmann> and what version types they use?
15:00:25 <ttx> hmm
15:00:28 <ttx> yeah, guess os
15:00:30 <ttx> so
15:00:34 <ttx> oh well, out of time
15:00:38 <dhellmann> ah, ok
15:00:46 <dhellmann> we can move to #openstack-release
15:00:52 <dhellmann> #endmeeting