14:02:43 #startmeeting releaseteam 14:02:44 Meeting started Fri Jun 10 14:02:43 2016 UTC and is due to finish in 60 minutes. The chair is dhellmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:02:45 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:02:47 The meeting name has been set to 'releaseteam' 14:02:48 o/ 14:02:50 courtesy ping: ttx, dims, lifeless, tonyb, stevemar, fungi 14:02:50 If you would like for me to add you to the courtesy ping please say so 14:02:59 our agenda is at https://etherpad.openstack.org/p/newton-relmgt-tracking 14:02:59 this week is R-17 14:03:29 o/ 14:03:37 #topic tag merging issue 14:03:46 #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/096895.html 14:03:58 ttx, based on your email it seems like the consensus is that it's OK to stop merging tags, so unless anyone disagrees I'll work on that patch to project-config 14:04:32 yeah... I tried to remember the initial reason 14:04:57 I think what you described was similar to what fungi remembered when we talked about it the other day in -infra 14:05:15 basically, it looks weird to humans who are expecting to see all versions when asking for tags along master 14:05:26 I /think/ it's what I described (in which case it is not current anymore), so wanted to throw it out there and see if anybody screamed 14:05:51 sorry, around now. had to step away for a bit 14:06:06 fungi : np, thanks for making the time 14:06:20 so should I wait a few days before making any changes? 14:06:35 dhellmann: main issue with the solution is that we have to wait for the n+1 commit to tag 14:06:55 we can trigger a n+1 commit but we still have delay while it gets accepted (much like setup.cfg) 14:07:13 why do we need to wait? 14:07:18 unless we can somehow automate it, but it sounds like a complex trigger 14:07:55 you need a commit that is only on master, right, so a new one after branch point 14:08:27 on which to apply the a0 tag on 14:08:39 yeah, it means after creating the stable branch, some time in the next day or so we need to add a tag. I don't see that as too big of a deal if we miss a few patches, do you? 14:08:44 yeah, otherwise git will show it on the old branch too, which you don't want 14:09:07 so need at least one commit's divergence 14:09:14 it's still painful to track 14:09:19 yes, true 14:09:24 some slow teams will take forever 14:09:31 we already ask them to tag before branching. we could ask them to have that patch in place already. 14:09:36 so ideally we would have some automation to do it for us 14:09:42 even if it's just a whitespace change to the readme or something 14:10:12 IOW, have both patches in place before creating the branch, and don't branch from HEAD 14:10:13 the way I saw it, this is a technical tag that would not go through openstack/releases 14:10:24 i.e. no need for a change by release liaison and all 14:10:25 sure, that's fine 14:10:32 I meant the patch in the project tree 14:11:01 it still means tracking who did it and who did not 14:11:43 true. we'll need to work that out. what we have now is producing erroneous release notes, so I really do think we need to change something. 14:12:06 We alsmot need a zuul trigger that would detect first-patch-after-stable-branching and trigger a tag job 14:12:52 dhellmann: sure, I'm not saying we shouldn't do it. Just saying it's inconvenient and we should not overlook that 14:12:58 could just run a post job on every commit and have it no-op if it's not the first commit on master after the branch point 14:12:59 that does sound challenging. it would be relatively straightforward to add a validation check for rc1 tags to say they can't be from HEAD. 14:13:21 though that would no-op 10k times for every positive 14:13:25 yeah 14:13:38 dhellmann: ah, so force people to submit another patch before they can even have their rc1 14:13:51 ttx: right 14:14:22 we don't care what that patch is, we don't care if they eventually back-port it, etc. but it has to be present. 14:14:41 dhellmann: that sounds a bit tricky to communicate but we can give it a try. No better answer anyway 14:14:42 we can tag that first, then tag rc1, then branch 14:14:57 yeah, it will take some documentation (more on that later) 14:15:02 (for reference, that is what we did for setup.cfg) 14:15:18 yeah, this will be a little looser, since the patch can be anything at all 14:15:24 Historically we asked them to bump setup.cfg, and I would branch stable/* from previous commit 14:15:39 so setup.cfg merging was the signoff for stable branching 14:16:07 It was slightly more explicit and integrated than asking them to coordinate RC1 tag and extra commit 14:16:16 but not impossible to explain 14:16:23 though i thought that was recognized as a sort of fragile inconvenience in the old process 14:16:38 having to synchronize this at all is annoying and fragile :-) 14:16:44 so would be unfortunate if we're having to reintroduce a similar workflow 14:16:48 well, the idea was to remove the core team from the equation 14:16:57 another option is for us to stop merging tags and stop fretting about master appearing to be older than the stable branch. 14:16:57 so that you could do everything with release team 14:17:18 and asking for another patch to land on master before you tag RC1 means putting the core review team in the loop again 14:17:33 i'm personally fine with just status quo and stop the merge job, but someone will have to field all the complaints 14:17:51 we could also just wait until the week after the release and tag whatever head is at that point 14:18:04 assuming it's not still their rc1 14:18:27 dhellmann: we could have a script that we run weekly to catch the missing tags 14:18:38 "list all cycle-with-milestone repos, get head of master, if there are no tags on that patch tag it a1" 14:18:57 i.e. detect first-patch-after-stable-branching and see if it has tag 14:19:09 ok, I sort of like that 14:19:11 especially from continuous deployers who are now annoyed that their packages from master are "older" versions than the latest release 14:19:12 right 14:19:14 so a periodic job? 14:19:20 sorry, lag 14:19:24 we run it until it says everything is covered 14:19:32 run by hand? 14:19:34 doesn't have to be daily, can be weekly at release meeting 14:19:41 yeah 14:19:51 that way they are not in the critical path 14:19:56 ok, that's better than coordinating with the core teams 14:20:14 and at most one week worth of versioning confusion 14:20:36 ttx: ok, do you want to think about how to write that script? 14:20:37 and if for some reason it takes forever we can then push a change for a whitespace change 14:20:51 dhellmann: that would be a good exercise for me yes 14:20:53 i think we can accommodate just about any workflow you want via automation, as long as you can settle on which is the least annoying path for the release team from a human interaction and tracking perspective 14:21:24 fungi: we could run that as a daily job, although that would mean unsigned tags 14:21:30 or at least machine-signed tags 14:21:44 if machine signed tags are good enough for the release, they should be good enough for this 14:21:50 anyway, we have a path forward. Let's do it manually first 14:21:55 sounds good 14:22:11 * ttx adds notes to meeting 14:22:13 #agreed ttx to work on a script to identify "first commit after stable branch" and apply a tag 14:22:27 #topic automation update (fungi) 14:22:41 yeah, machine-signed 14:23:02 so speaking of machine-signed tags... :-) 14:23:05 no major update, pushing the change today that adds teh signing node 14:23:09 dhellmann: oh, and would be great if somewhow that tag did not trigger post-tag jobs 14:23:21 like announcements 14:23:29 still have a little more work to do on the piece which handles key deployment to the machine 14:23:32 ttx: I think we already don't announce pre-releases but we can make sure of that 14:23:37 oh right 14:23:42 * ttx shuts up now 14:23:47 could certainly adjust the announcements job to not announce alphas 14:24:00 even if you wanted beta milestone tags announced 14:24:19 fungi : so it sounds like we're getting close to ready to start moving some of the release tools scripts into project-config? 14:24:23 anyway, that was more or less my update. should have more in place later today 14:24:32 great, thank you 14:24:36 and yes, we could in theory move those scripts at any point now 14:25:01 ok, I have some other things that are more urgent, but that's getting close to the top of my priority list now 14:25:31 moving on... 14:25:31 #topic email notification for release pipeline job failures 14:25:32 good. the release automation work has mostly bubbled to the surface for my priorities at this point in the cycle as well 14:25:42 #undo 14:25:43 Removing item from minutes: 14:26:25 ok, I've been holding off moving the scripts until we were ready to actually define a job and be able to run it, but I'll reconsider that (we can run the release scripts from project-config by hand just as easily as from release-tools) 14:26:34 #topic email notification for release pipeline job failures 14:26:59 * fungi notes that #undo of a #topic doesn't roll back the irc channel topic 14:27:27 we've had a little trouble keeping up with what release jobs have failed, relying on the project teams or consumers to notice that a package wasn't built and then debugging the failure 14:27:42 yesterday jeblair pointed out that it's possible to have zuul send us email when a job fails 14:27:54 or at least when something in a given pipeline fails, I'm not clear on all of the filtering abilities yet 14:28:12 yeah, i think it's just all failures, all successes, or both 14:28:13 my plan is to create a new mailing list, and set that list up to receive notifications when any tag, release, release-post, etc. jobs fail 14:28:15 at the moment 14:28:39 note there will be a ton of failures since a lot of non-reno'd projects seem to have added the release announcements jobs 14:28:44 I think either "all" or "all failures" would be fine 14:29:02 oh, probably because I added the announcement job to the template 14:29:09 not having reno shouldn't be causing that to fail 14:29:15 'all failures in pipeline' is what we have right now 14:29:26 I think that's fine 14:29:37 yeah, that's way better than "only when someone complains" :-) 14:29:39 oh, i assumed it was lack of release notes, but regardless i notice lots of "failing" release announcement jobs and they're for stuff that isn't release-managed 14:29:56 fungi : it's probably because they don't have URLs in their README for their bug tracker, etc. 14:30:07 dhellmann: although we'd likely only cover release pipelines, not the post one 14:30:22 yeah, I'll have to look at what's in each pipeline 14:30:28 I'll include everyone on the patches to do that 14:30:35 does anyone want to suggest a name for the mailing list? 14:30:56 release-job-failures ? 14:30:56 so, yeah, propose a change to openstack-infra/system-config for the modules/openstack_project/manifests/lists.pp file to add the ml you want to use for this once you bikeshed a name 14:31:20 The most interesting mailing list you'll ever subscribe to ! 14:31:21 ttx: works for me 14:31:33 fungi : yep, watch for that today or monday :-) 14:31:44 13-interesting-ways-release-jobs-can-fail@lists.openstack.org 14:32:00 an alternative might be to get these reporting on the health dashboard (and so also available via rss). i don't know what's involved to make that happen though 14:33:15 fungi : I'd like that, too. I think we need to set up the sql integration (I can't remember the name of that plugin thing mtreinish built). I like the push notification aspect of email, though 14:33:36 * dhellmann imagines a bat signal plugin for zuul 14:34:04 dhellmann: subunit2sql. Fwiw, it's really easy to add subunit generation to jobs 14:34:17 mtreinish : thanks, I was thinking testr2sql and knew that was wrong 14:34:33 dhellmann: AJaeger and I did it pretty quickly for the translation jobs 14:34:34 mtreinish : we don't have any "tests" in these jobs, and they don't use subunit. 14:34:42 ah, ok, so maybe that's not a requirement 14:34:48 dhellmann: all you'd need is a rpi2 with a cron job and a blinkenlicht 14:35:15 fungi : so now I have a project for this spare rpi2 sitting on my shelf 14:35:24 dhellmann: right, but there is a cli utility I wrote you give it a name and a couple timestamps and a status and it'll generate a subunit file for you 14:35:42 mtreinish : oh, cool. are there docs for that somewhere? 14:35:58 we talked about this a while back when I didn't have a lot of time to dig into it 14:36:54 dhellmann: err, I don't think I actually wrote docs for generate subunit (I'll fix that today) but the script is here: http://git.openstack.org/cgit/openstack/os-testr/tree/os_testr/generate_subunit.py 14:37:09 mtreinish : ok, I'll take a look at adding that, too 14:37:19 thanks! 14:37:29 dhellmann: and here is how the translation jobs call it: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/common_translation_update.sh#n64 14:37:48 oh, good, an example will help a lot 14:38:06 cool, so I'll put that on our list, too 14:38:17 we have 2 more topics to cover today, let's move on 14:38:22 #topic reno changes for osc project 14:38:38 dtroyer wants to have version-specific pages instead of release series pages, which is possible now, but it would be easier with a latest-release field similar to earliest-release 14:39:01 I'd like to have someone else know the reno code a bit, so does anyone want to take a stab at adding that? 14:39:43 depends how fast you need it 14:39:48 there's no rush 14:40:21 he has a list of versions now, and is ok with manually updating that, but over the long term this will make it easier 14:41:00 I can add it to my list 14:41:16 cool, thanks 14:41:35 but that's the sort of things that tend to slip from my TODO forever, so at some point we should do a timecheck :) 14:41:53 and someone with more bandwidth could pick it up 14:41:57 will do. I'll add it to the tracking docs so we don't lose it 14:42:14 #topic release manager's manual 14:42:31 dhellmann: ok, and if you could describe the required feature in a bit more detail (on the etherpad or on an email to me) that would be awesome 14:42:57 ttx: of course, I'll write that up in the etherpad 14:43:14 I'd like to get started building the release manual. Where should we put it and where should we publish it? 14:43:37 define scope again. It's for release managers ? 14:43:48 or liaisons ? Or both 14:44:28 maybe a bit of both? basically, all of the procedural instructions we have 14:44:43 some of what's in the releases/README.rst might move there 14:45:12 My take is, instructions for project teams (release liaisons) should live in the pTG 14:45:16 but we'd also have things like the processes for handling each milestone, the process making the stable branches, etc. 14:45:37 while instructions for release managers should live somewhere specific 14:45:55 releases repo readme? 14:45:56 that makes sense 14:46:01 the split does 14:46:24 yeah, the readme may be a good place for release manager instructions. it's simpler than publishing a whole new manual if we're going to split the other bits out to the ptg 14:46:47 For the team documentation, most other OpenStack project teams leverage an existing repo, or the wiki 14:47:09 * fungi shudders at mention of the wiki 14:47:12 I don't want to use the wiki for this. 14:47:22 we already have a todo to clean up our wiki presence 14:47:56 I'll start working on patches for the existing documents, and if we decide something doesn't fit in either we can revisit 14:47:58 so if we push it to openstack/releases, would we publish in some subdirectory there ? 14:48:17 or somewhere else completely ? Or not publish ? 14:48:24 if it's in the readme, I don't think we need to publish it 14:48:30 ok 14:48:45 "publish" it the same way http://git.openstack.org/cgit/openstack/requirements/tree/README.rst is "published" 14:48:51 yeah 14:49:03 i find that entirely readable 14:49:07 I would probably separate README - for people submitting changes to that repo or trying to figure out what it is) from process 14:49:20 ok, a separate rst file works, too 14:49:42 I think a lot of what I have in mind can be written in a way that is usable both to the liaisons and release team, so that can go into the ptg 14:49:52 So keep README.rst for "what is this about ?" and basic info on file structure 14:50:02 even if it's just "this is what will happen, liaisons don't need to do it" it's still informative 14:50:22 "please pay attention to the man behind the curtain" 14:50:28 then some "Process.rst" for out team doc 14:51:07 Yeah, I just don't want to overload the PTG too much 14:51:20 ok, I'll start by assembling a list of topics to document and we can see what goes where 14:51:30 Currently the PTG rewrite points to the README.rst 14:51:43 https://review.openstack.org/#/c/324540/ 14:52:13 "See the `README file in that repository`_ for more details." 14:52:15 * dhellmann adds that to his review queue 14:53:14 agreed / mix of README.rst, OtherFile.rst and PTG should do it 14:53:22 speaking of reviews 14:53:25 #topic priority reviews 14:53:28 depending on level of external exposure wanted 14:53:30 #link https://review.openstack.org/#/c/324540/ - updates the PTG 14:53:38 #link https://review.openstack.org/#/c/322863/ - proposed release ACL dance model 14:54:08 Anything in releases changes that I should pay immediate attention to ? Been in catchup mode today 14:54:12 ttx: yeah, my original goal was to have enough instructions written down that someone could use them to train to do the release work. it changes enough that it might not be possible, though 14:54:43 ttx: you may want to look at the patches we pushed through yesterday to add the "team" field to deliverable files instead of relying on the governance repo definition of teams 14:55:01 those were turning into rebase issues, so we pushed them in with the agreement that we'd change things as needed 14:55:35 ack, will post-review them :) 14:55:47 otherwise the queue is empty so just the 2 items you listed 14:56:34 I think we're going to need to do something similar to the team field for other changes. for instance, deliverable definitions change over time, and their release models and tags change 14:56:57 we might need to turn off some of the validation we have for those things, or modify it to look at local tag definitions and only fall back to the governance repo 14:57:18 otherwise we'll have trouble releasing stable versions of things if the governance repo has changed 14:57:51 #topic open discussion 14:57:57 is there anything else we need to mention? 14:58:08 I have jury duty next week, so I don't know what my schedule is going to be like yet 14:59:31 ok, that's time 14:59:33 thanks everyone! 14:59:38 ack 14:59:48 #endmeeting