14:02:43 <dhellmann> #startmeeting releaseteam
14:02:44 <openstack> Meeting started Fri Jun 10 14:02:43 2016 UTC and is due to finish in 60 minutes.  The chair is dhellmann. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:02:45 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:02:47 <openstack> The meeting name has been set to 'releaseteam'
14:02:48 <ttx> o/
14:02:50 <dhellmann> courtesy ping: ttx, dims, lifeless, tonyb, stevemar, fungi
14:02:50 <dhellmann> If you would like for me to add you to the courtesy ping please say so
14:02:59 <dhellmann> our agenda is at https://etherpad.openstack.org/p/newton-relmgt-tracking
14:02:59 <dhellmann> this week is R-17
14:03:29 <dims> o/
14:03:37 <dhellmann> #topic tag merging issue
14:03:46 <dhellmann> #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/096895.html
14:03:58 <dhellmann> ttx, based on your email it seems like the consensus is that it's OK to stop merging tags, so unless anyone disagrees I'll work on that patch to project-config
14:04:32 <ttx> yeah... I tried to remember the initial reason
14:04:57 <dhellmann> I think what you described was similar to what fungi remembered when we talked about it the other day in -infra
14:05:15 <dhellmann> basically, it looks weird to humans who are expecting to see all versions when asking for tags along master
14:05:26 <ttx> I /think/ it's what I described (in which case it is not current anymore), so wanted to throw it out there and see if anybody screamed
14:05:51 <fungi> sorry, around now. had to step away for a bit
14:06:06 <dhellmann> fungi : np, thanks for making the time
14:06:20 <dhellmann> so should I wait a few days before making any changes?
14:06:35 <ttx> dhellmann: main issue with the solution is that we have to wait for the n+1 commit to tag
14:06:55 <ttx> we can trigger a n+1 commit but we still have delay while it gets accepted (much like setup.cfg)
14:07:13 <dhellmann> why do we need to wait?
14:07:18 <ttx> unless we can somehow automate it, but it sounds like a complex trigger
14:07:55 <ttx> you need a commit that is only on master, right, so a new one after branch point
14:08:27 <ttx> on which to apply the a0 tag on
14:08:39 <dhellmann> yeah, it means after creating the stable branch, some time in the next day or so we need to add a tag. I don't see that as too big of a deal if we miss a few patches, do you?
14:08:44 <fungi> yeah, otherwise git will show it on the old branch too, which you don't want
14:09:07 <fungi> so need at least one commit's divergence
14:09:14 <ttx> it's still painful to track
14:09:19 <dhellmann> yes, true
14:09:24 <ttx> some slow teams will take forever
14:09:31 <dhellmann> we already ask them to tag before branching. we could ask them to have that patch in place already.
14:09:36 <ttx> so ideally we would have some automation to do it for us
14:09:42 <dhellmann> even if it's just a whitespace change to the readme or something
14:10:12 <dhellmann> IOW, have both patches in place before creating the branch, and don't branch from HEAD
14:10:13 <ttx> the way I saw it, this is a technical tag that would not go through openstack/releases
14:10:24 <ttx> i.e. no need for a change by release liaison and all
14:10:25 <dhellmann> sure, that's fine
14:10:32 <dhellmann> I meant the patch in the project tree
14:11:01 <ttx> it still means tracking who did it and who did not
14:11:43 <dhellmann> true. we'll need to work that out. what we have now is producing erroneous release notes, so I really do think we need to change something.
14:12:06 <ttx> We alsmot need a zuul trigger that would detect first-patch-after-stable-branching and trigger a tag job
14:12:52 <ttx> dhellmann: sure, I'm not saying we shouldn't do it. Just saying it's inconvenient and we should not overlook that
14:12:58 <fungi> could just run a post job on every commit and have it no-op if it's not the first commit on master after the branch point
14:12:59 <dhellmann> that does sound challenging. it would be relatively straightforward to add a validation check for rc1 tags to say they can't be from HEAD.
14:13:21 <fungi> though that would no-op 10k times for every positive
14:13:25 <dhellmann> yeah
14:13:38 <ttx> dhellmann: ah, so force people to submit another patch before they can even have their rc1
14:13:51 <dhellmann> ttx: right
14:14:22 <dhellmann> we don't care what that patch is, we don't care if they eventually back-port it, etc. but it has to be present.
14:14:41 <ttx> dhellmann: that sounds a bit tricky to communicate but we can give it a try. No better answer anyway
14:14:42 <dhellmann> we can tag that first, then tag rc1, then branch
14:14:57 <dhellmann> yeah, it will take some documentation (more on that later)
14:15:02 <ttx> (for reference, that is what we did for setup.cfg)
14:15:18 <dhellmann> yeah, this will be a little looser, since the patch can be anything at all
14:15:24 <ttx> Historically we asked them to bump setup.cfg, and I would branch stable/* from previous commit
14:15:39 <ttx> so setup.cfg merging was the signoff for stable branching
14:16:07 <ttx> It was slightly more explicit and integrated than asking them to coordinate RC1 tag and extra commit
14:16:16 <ttx> but not impossible to explain
14:16:23 <fungi> though i thought that was recognized as a sort of fragile inconvenience in the old process
14:16:38 <dhellmann> having to synchronize this at all is annoying and fragile :-)
14:16:44 <fungi> so would be unfortunate if we're having to reintroduce a similar workflow
14:16:48 <ttx> well, the idea was to remove the core team from the equation
14:16:57 <dhellmann> another option is for us to stop merging tags and stop fretting about master appearing to be older than the stable branch.
14:16:57 <ttx> so that you could do everything with release team
14:17:18 <ttx> and asking for another patch to land on master before you tag RC1 means putting the core review team in the loop again
14:17:33 <fungi> i'm personally fine with just status quo and stop the merge job, but someone will have to field all the complaints
14:17:51 <dhellmann> we could also just wait until the week after the release and tag whatever head is at that point
14:18:04 <dhellmann> assuming it's not still their rc1
14:18:27 <ttx> dhellmann: we could have a script that we run weekly to catch the missing tags
14:18:38 <dhellmann> "list all cycle-with-milestone repos, get head of master, if there are no tags on that patch tag it a1"
14:18:57 <ttx> i.e. detect first-patch-after-stable-branching and see if it has tag
14:19:09 <dhellmann> ok, I sort of like that
14:19:11 <fungi> especially from continuous deployers who are now annoyed that their packages from master are "older" versions than the latest release
14:19:12 <ttx> right
14:19:14 <dhellmann> so a periodic job?
14:19:20 <fungi> sorry, lag
14:19:24 <ttx> we run it until it says everything is covered
14:19:32 <dhellmann> run by hand?
14:19:34 <ttx> doesn't have to be daily, can be weekly at release meeting
14:19:41 <ttx> yeah
14:19:51 <ttx> that way they are not in the critical path
14:19:56 <dhellmann> ok, that's better than coordinating with the core teams
14:20:14 <ttx> and at most one week worth of versioning confusion
14:20:36 <dhellmann> ttx: ok, do you want to think about how to write that script?
14:20:37 <ttx> and if for some reason it takes forever we can then push a change for a whitespace change
14:20:51 <ttx> dhellmann: that would be a good exercise for me yes
14:20:53 <fungi> i think we can accommodate just about any workflow you want via automation, as long as you can settle on which is the least annoying path for the release team from a human interaction and tracking perspective
14:21:24 <ttx> fungi: we could run that as a daily job, although that would mean unsigned tags
14:21:30 <ttx> or at least machine-signed tags
14:21:44 <dhellmann> if machine signed tags are good enough for the release, they should be good enough for this
14:21:50 <ttx> anyway, we have a path forward. Let's do it manually first
14:21:55 <dhellmann> sounds good
14:22:11 * ttx adds notes to meeting
14:22:13 <dhellmann> #agreed ttx to work on a script to identify "first commit after stable branch" and apply a tag
14:22:27 <dhellmann> #topic automation update (fungi)
14:22:41 <fungi> yeah, machine-signed
14:23:02 <dhellmann> so speaking of machine-signed tags... :-)
14:23:05 <fungi> no major update, pushing the change today that adds teh signing node
14:23:09 <ttx> dhellmann: oh, and would be great if somewhow that tag did not trigger post-tag jobs
14:23:21 <ttx> like announcements
14:23:29 <fungi> still have a little more work to do on the piece which handles key deployment to the machine
14:23:32 <dhellmann> ttx: I think we already don't announce pre-releases but we can make sure of that
14:23:37 <ttx> oh right
14:23:42 * ttx shuts up now
14:23:47 <fungi> could certainly adjust the announcements job to not announce alphas
14:24:00 <fungi> even if you wanted beta milestone tags announced
14:24:19 <dhellmann> fungi : so it sounds like we're getting close to ready to start moving some of the release tools scripts into project-config?
14:24:23 <fungi> anyway, that was more or less my update. should have more in place later today
14:24:32 <dhellmann> great, thank you
14:24:36 <fungi> and yes, we could in theory move those scripts at any point now
14:25:01 <dhellmann> ok, I have some other things that are more urgent, but that's getting close to the top of my priority list now
14:25:31 <dhellmann> moving on...
14:25:31 <dhellmann> #topic email notification for release pipeline job failures
14:25:32 <fungi> good. the release automation work has mostly bubbled to the surface for my priorities at this point in the cycle as well
14:25:42 <dhellmann> #undo
14:25:43 <openstack> Removing item from minutes: <ircmeeting.items.Topic object at 0x7fe23758dc90>
14:26:25 <dhellmann> ok, I've been holding off moving the scripts until we were ready to actually define a job and be able to run it, but I'll reconsider that (we can run the release scripts from project-config by hand just as easily as from release-tools)
14:26:34 <dhellmann> #topic email notification for release pipeline job failures
14:26:59 * fungi notes that #undo of a #topic doesn't roll back the irc channel topic
14:27:27 <dhellmann> we've had a little trouble keeping up with what release jobs have failed, relying on the project teams or consumers to notice that a package wasn't built and then debugging the failure
14:27:42 <dhellmann> yesterday jeblair pointed out that it's possible to have zuul send us email when a job fails
14:27:54 <dhellmann> or at least when something in a given pipeline fails, I'm not clear on all of the filtering abilities yet
14:28:12 <fungi> yeah, i think it's just all failures, all successes, or both
14:28:13 <dhellmann> my plan is to create a new mailing list, and set that list up to receive notifications when any tag, release, release-post, etc. jobs fail
14:28:15 <fungi> at the moment
14:28:39 <fungi> note there will be a ton of failures since a lot of non-reno'd projects seem to have added the release announcements jobs
14:28:44 <dhellmann> I think either "all" or "all failures" would be fine
14:29:02 <dhellmann> oh, probably because I added the announcement job to the template
14:29:09 <dhellmann> not having reno shouldn't be causing that to fail
14:29:15 <jeblair> 'all failures in pipeline' is what we have right now
14:29:26 <ttx> I think that's fine
14:29:37 <dhellmann> yeah, that's way better than "only when someone complains" :-)
14:29:39 <fungi> oh, i assumed it was lack of release notes, but regardless i notice lots of "failing" release announcement jobs and they're for stuff that isn't release-managed
14:29:56 <dhellmann> fungi : it's probably because they don't have URLs in their README for their bug tracker, etc.
14:30:07 <ttx> dhellmann: although we'd likely only cover release pipelines, not the post one
14:30:22 <dhellmann> yeah, I'll have to look at what's in each pipeline
14:30:28 <dhellmann> I'll include everyone on the patches to do that
14:30:35 <dhellmann> does anyone want to suggest a name for the mailing list?
14:30:56 <ttx> release-job-failures ?
14:30:56 <fungi> so, yeah, propose a change to openstack-infra/system-config for the modules/openstack_project/manifests/lists.pp file to add the ml you want to use for this once you bikeshed a name
14:31:20 <ttx> The most interesting mailing list you'll ever subscribe to !
14:31:21 <dhellmann> ttx: works for me
14:31:33 <dhellmann> fungi : yep, watch for that today or monday :-)
14:31:44 <ttx> 13-interesting-ways-release-jobs-can-fail@lists.openstack.org
14:32:00 <fungi> an alternative might be to get these reporting on the health dashboard (and so also available via rss). i don't know what's involved to make that happen though
14:33:15 <dhellmann> fungi : I'd like that, too. I think we need to set up the sql integration (I can't remember the name of that plugin thing mtreinish built). I like the push notification aspect of email, though
14:33:36 * dhellmann imagines a bat signal plugin for zuul
14:34:04 <mtreinish> dhellmann: subunit2sql. Fwiw, it's really easy to add subunit generation to jobs
14:34:17 <dhellmann> mtreinish : thanks, I was thinking testr2sql and knew that was wrong
14:34:33 <mtreinish> dhellmann: AJaeger and I did it pretty quickly for the translation jobs
14:34:34 <dhellmann> mtreinish : we don't have any "tests" in these jobs, and they don't use subunit.
14:34:42 <dhellmann> ah, ok, so maybe that's not a requirement
14:34:48 <fungi> dhellmann: all you'd need is a rpi2 with a cron job and a blinkenlicht
14:35:15 <dhellmann> fungi : so now I have a project for this spare rpi2 sitting on my shelf
14:35:24 <mtreinish> dhellmann: right, but there is a cli utility I wrote you give it a name and a couple timestamps and a status and it'll generate a subunit file for you
14:35:42 <dhellmann> mtreinish : oh, cool. are there docs for that somewhere?
14:35:58 <dhellmann> we talked about this a while back when I didn't have a lot of time to dig into it
14:36:54 <mtreinish> dhellmann: err, I don't think I actually wrote docs for generate subunit (I'll fix that today) but the script is here: http://git.openstack.org/cgit/openstack/os-testr/tree/os_testr/generate_subunit.py
14:37:09 <dhellmann> mtreinish : ok, I'll take a look at adding that, too
14:37:19 <dhellmann> thanks!
14:37:29 <mtreinish> dhellmann: and here is how the translation jobs call it: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n64
14:37:48 <dhellmann> oh, good, an example will help a lot
14:38:06 <dhellmann> cool, so I'll put that on our list, too
14:38:17 <dhellmann> we have 2 more topics to cover today, let's move on
14:38:22 <dhellmann> #topic reno changes for osc project
14:38:38 <dhellmann> dtroyer wants to have version-specific pages instead of release series pages, which is possible now, but it would be easier with a latest-release field similar to earliest-release
14:39:01 <dhellmann> I'd like to have someone else know the reno code a bit, so does anyone want to take a stab at adding that?
14:39:43 <ttx> depends how fast you need it
14:39:48 <dhellmann> there's no rush
14:40:21 <dhellmann> he has a list of versions now, and is ok with manually updating that, but over the long term this will make it easier
14:41:00 <ttx> I can add it to my list
14:41:16 <dhellmann> cool, thanks
14:41:35 <ttx> but that's the sort of things that tend to slip from my TODO forever, so at some point we should do a timecheck :)
14:41:53 <ttx> and someone with more bandwidth could pick it up
14:41:57 <dhellmann> will do. I'll add it to the tracking docs so we don't  lose it
14:42:14 <dhellmann> #topic release manager's manual
14:42:31 <ttx> dhellmann: ok, and if you could describe the required feature in a bit more detail (on the etherpad or on an email to me) that would be awesome
14:42:57 <dhellmann> ttx: of course, I'll write that up in the etherpad
14:43:14 <dhellmann> I'd like to get started building the release manual. Where should we put it and where should we publish it?
14:43:37 <ttx> define scope again. It's for release managers ?
14:43:48 <ttx> or liaisons ? Or both
14:44:28 <dhellmann> maybe a bit of both? basically, all of the procedural instructions we have
14:44:43 <dhellmann> some of what's in the releases/README.rst might move there
14:45:12 <ttx> My take is, instructions for project teams (release liaisons) should live in the pTG
14:45:16 <dhellmann> but we'd also have things like the processes for handling each milestone, the process making the stable branches, etc.
14:45:37 <ttx> while instructions for release managers should live somewhere specific
14:45:55 <fungi> releases repo readme?
14:45:56 <dhellmann> that makes sense
14:46:01 <dhellmann> the split does
14:46:24 <dhellmann> yeah, the readme may be a good place for release manager instructions. it's simpler than publishing a whole new manual if we're going to split the other bits out to the ptg
14:46:47 <ttx> For the team documentation, most other OpenStack project teams leverage an existing repo, or the wiki
14:47:09 * fungi shudders at mention of the wiki
14:47:12 <dhellmann> I don't want to use the wiki for this.
14:47:22 <dhellmann> we already have a todo to clean up our wiki presence
14:47:56 <dhellmann> I'll start working on patches for the existing documents, and if we decide something doesn't fit in either we can revisit
14:47:58 <ttx> so if we push it to openstack/releases, would we publish in some subdirectory there ?
14:48:17 <ttx> or somewhere else completely ? Or not publish ?
14:48:24 <dhellmann> if it's in the readme, I don't think we need to publish it
14:48:30 <ttx> ok
14:48:45 <fungi> "publish" it the same way http://git.openstack.org/cgit/openstack/requirements/tree/README.rst is "published"
14:48:51 <dhellmann> yeah
14:49:03 <fungi> i find that entirely readable
14:49:07 <ttx> I would probably separate README - for people submitting changes to that repo or trying to figure out what it is) from process
14:49:20 <dhellmann> ok, a separate rst file works, too
14:49:42 <dhellmann> I think a lot of what I have in mind can be written in a way that is usable both to the liaisons and release team, so that can go into the ptg
14:49:52 <ttx> So keep README.rst for "what is this about ?" and basic info on file structure
14:50:02 <dhellmann> even if it's just "this is what will happen, liaisons don't need to do it" it's still informative
14:50:22 <dhellmann> "please pay attention to the man behind the curtain"
14:50:28 <ttx> then some "Process.rst" for out team doc
14:51:07 <ttx> Yeah, I just don't want to overload the PTG too much
14:51:20 <dhellmann> ok, I'll start by assembling a list of topics to document and we can see what goes where
14:51:30 <ttx> Currently the PTG rewrite points to the README.rst
14:51:43 <ttx> https://review.openstack.org/#/c/324540/
14:52:13 <ttx> "See the `README file in that repository`_ for more details."
14:52:15 * dhellmann adds that to his review queue
14:53:14 <ttx> agreed / mix of README.rst, OtherFile.rst and PTG should do it
14:53:22 <dhellmann> speaking of reviews
14:53:25 <dhellmann> #topic priority reviews
14:53:28 <ttx> depending on level of external exposure wanted
14:53:30 <dhellmann> #link https://review.openstack.org/#/c/324540/ - updates the PTG
14:53:38 <dhellmann> #link https://review.openstack.org/#/c/322863/ - proposed release ACL dance model
14:54:08 <ttx> Anything in releases changes that I should pay immediate attention to ? Been in catchup mode today
14:54:12 <dhellmann> ttx: yeah, my original goal was to have enough instructions written down that someone could use them to train to do the release work. it changes enough that it might not be possible, though
14:54:43 <dhellmann> ttx: you may want to look at the patches we pushed through yesterday to add the "team" field to deliverable files instead of relying on the governance repo definition of teams
14:55:01 <dhellmann> those were turning into rebase issues, so we pushed them in with the agreement that we'd change things as needed
14:55:35 <ttx> ack, will post-review them :)
14:55:47 <dhellmann> otherwise the queue is empty so just the 2 items you listed
14:56:34 <dhellmann> I think we're going to need to do something similar to the team field for other changes. for instance, deliverable definitions change over time, and their release models and tags change
14:56:57 <dhellmann> we might need to turn off some of the validation we have for those things, or modify it to look at local tag definitions and only fall back to the governance repo
14:57:18 <dhellmann> otherwise we'll have trouble releasing stable versions of things if the governance repo has changed
14:57:51 <dhellmann> #topic open discussion
14:57:57 <dhellmann> is there anything else we need to mention?
14:58:08 <dhellmann> I have jury duty next week, so I don't know what my schedule is going to be like yet
14:59:31 <dhellmann> ok, that's time
14:59:33 <dhellmann> thanks everyone!
14:59:38 <ttx> ack
14:59:48 <dhellmann> #endmeeting