| *** bobh has joined #openstack-release | 01:44 | |
| *** oanson has quit IRC | 01:59 | |
| *** dave-mccowan has joined #openstack-release | 02:07 | |
| *** hongbin has joined #openstack-release | 02:11 | |
| *** hongbin has quit IRC | 02:28 | |
| *** bobh has quit IRC | 02:37 | |
| *** dave-mccowan has quit IRC | 02:47 | |
| *** bobh has joined #openstack-release | 02:49 | |
| *** hongbin has joined #openstack-release | 02:50 | |
| *** dangtrinhnt has quit IRC | 03:21 | |
| *** lbragstad has joined #openstack-release | 03:26 | |
| *** ykarel has joined #openstack-release | 03:43 | |
| *** udesale has joined #openstack-release | 03:49 | |
| *** armax has quit IRC | 03:57 | |
| *** bobh has quit IRC | 04:12 | |
| *** ricolin has joined #openstack-release | 04:31 | |
| *** hongbin has quit IRC | 04:37 | |
| *** ykarel has quit IRC | 04:40 | |
| openstackgerrit | John Dickinson proposed openstack/releases master: swift 2.20.0 release https://review.openstack.org/625483 | 04:58 |
|---|---|---|
| *** ykarel has joined #openstack-release | 05:01 | |
| *** ykarel has quit IRC | 05:10 | |
| *** ykarel has joined #openstack-release | 05:12 | |
| *** ekcs has quit IRC | 06:55 | |
| *** jtomasek has joined #openstack-release | 07:32 | |
| *** jtomasek has quit IRC | 07:33 | |
| *** jtomasek has joined #openstack-release | 07:33 | |
| *** oanson has joined #openstack-release | 07:34 | |
| *** e0ne has joined #openstack-release | 07:35 | |
| *** e0ne has quit IRC | 07:39 | |
| *** dangtrinhnt has joined #openstack-release | 07:48 | |
| *** ykarel is now known as ykarel|lunch | 07:58 | |
| *** pcaruana has joined #openstack-release | 08:18 | |
| *** witek_ is now known as witek | 08:27 | |
| *** witek has quit IRC | 08:35 | |
| *** ykarel|lunch is now known as ykarel | 08:39 | |
| *** hberaud|gone is now known as hberaud | 08:39 | |
| *** alexchadin has joined #openstack-release | 08:47 | |
| *** jpich has joined #openstack-release | 09:02 | |
| *** shardy has joined #openstack-release | 09:04 | |
| *** witek has joined #openstack-release | 09:08 | |
| *** ricolin has quit IRC | 09:10 | |
| *** ssbarnea|rover has joined #openstack-release | 09:47 | |
| *** e0ne has joined #openstack-release | 10:02 | |
| *** lbragstad has quit IRC | 10:10 | |
| *** slaweq has joined #openstack-release | 10:20 | |
| slaweq | hi release team, can You take a look at https://review.openstack.org/#/c/625363/ - I need this new release a bit. Thx in advance for Your help :) | 10:21 |
| *** aojea has joined #openstack-release | 10:37 | |
| *** luizbag has joined #openstack-release | 10:46 | |
| *** electrofelix has joined #openstack-release | 10:52 | |
| *** hberaud is now known as hberaud|lunch | 10:56 | |
| *** udesale has quit IRC | 11:10 | |
| *** e0ne has quit IRC | 11:31 | |
| *** hberaud|lunch is now known as hberaud | 11:50 | |
| *** luizbag has quit IRC | 11:56 | |
| *** luizbag has joined #openstack-release | 11:58 | |
| *** hberaud has quit IRC | 12:00 | |
| *** hberaud has joined #openstack-release | 12:01 | |
| *** bobh has joined #openstack-release | 12:34 | |
| *** bobh has quit IRC | 12:38 | |
| *** udesale has joined #openstack-release | 12:39 | |
| *** e0ne has joined #openstack-release | 12:47 | |
| *** bobh has joined #openstack-release | 12:53 | |
| *** e0ne has quit IRC | 13:11 | |
| *** e0ne has joined #openstack-release | 13:14 | |
| *** bobh has quit IRC | 13:16 | |
| *** luizbag has quit IRC | 13:18 | |
| *** dave-mccowan has joined #openstack-release | 13:24 | |
| *** alexchadin has quit IRC | 13:24 | |
| *** luizbag has joined #openstack-release | 13:30 | |
| *** e0ne has quit IRC | 13:42 | |
| *** dave-mccowan has quit IRC | 13:42 | |
| *** e0ne has joined #openstack-release | 13:45 | |
| *** pcaruana has quit IRC | 13:50 | |
| *** lbragstad has joined #openstack-release | 13:58 | |
| *** lbragstad has quit IRC | 13:59 | |
| *** lbragstad has joined #openstack-release | 14:04 | |
| *** pcaruana has joined #openstack-release | 14:05 | |
| *** dave-mccowan has joined #openstack-release | 14:06 | |
| *** mriedem has joined #openstack-release | 14:08 | |
| *** bobh has joined #openstack-release | 14:09 | |
| *** dave-mccowan has quit IRC | 14:11 | |
| *** bobh has quit IRC | 14:12 | |
| *** bobh has joined #openstack-release | 14:13 | |
| *** bobh has quit IRC | 14:31 | |
| dmellado | Same here for this ;) https://review.openstack.org/#/c/624341/ | 14:40 |
| dmellado | I've added a few folks but I'd totally use some reviews! thanks! | 14:41 |
| dmellado | evrardjp: o/ do you have super-powers here? ;) | 14:41 |
| *** bobh has joined #openstack-release | 14:48 | |
| *** mlavalle has joined #openstack-release | 14:51 | |
| *** bobh has quit IRC | 14:52 | |
| evrardjp | I do not :) Will still review | 14:56 |
| *** toabctl has quit IRC | 14:57 | |
| *** toabctl has joined #openstack-release | 14:57 | |
| *** beekneemech is now known as bnemec | 15:00 | |
| *** e0ne has quit IRC | 15:02 | |
| *** bobh has joined #openstack-release | 15:03 | |
| *** e0ne has joined #openstack-release | 15:06 | |
| dmellado | thanks! | 15:07 |
| *** ykarel is now known as ykarel|away | 15:26 | |
| openstackgerrit | Merged openstack/releases master: Release os-ken 0.3.1 https://review.openstack.org/625363 | 15:28 |
| *** pcaruana has quit IRC | 15:29 | |
| *** ykarel|away has quit IRC | 15:35 | |
| openstackgerrit | John Dickinson proposed openstack/releases master: swift 2.20.0 release https://review.openstack.org/625483 | 15:42 |
| *** hberaud is now known as hberaud|afk | 15:44 | |
| *** pcaruana has joined #openstack-release | 15:51 | |
| *** ykarel|away has joined #openstack-release | 15:52 | |
| *** slaweq has quit IRC | 15:54 | |
| *** armax has joined #openstack-release | 15:55 | |
| openstackgerrit | Merged openstack/releases master: Release Kuryr-Kubernetes 0.6.0 https://review.openstack.org/624341 | 15:56 |
| *** udesale has quit IRC | 15:57 | |
| dhellmann | dmellado , slaweq: today is my review day, so when I finish up a few tc-related tasks I'll be taking a look at releases | 15:59 |
| dmellado | dhellmann: smcginnis already got my patch in, thanks in any case! | 16:00 |
| dmellado | thanks smcginnis! ;) | 16:00 |
| dhellmann | smcginnis : please save some releases for me to use for onboarding diablo_rojo later today :-) | 16:00 |
| smcginnis | dhellmann: Will do! | 16:00 |
| *** aojea has quit IRC | 16:07 | |
| *** ykarel|away is now known as ykarel | 16:12 | |
| *** hberaud|afk is now known as hberaud | 16:18 | |
| *** N3l1x has joined #openstack-release | 16:23 | |
| *** N3l1x_ has joined #openstack-release | 16:23 | |
| *** pcaruana has quit IRC | 16:28 | |
| *** aojea has joined #openstack-release | 16:39 | |
| *** aojea has quit IRC | 16:51 | |
| *** jtomasek has quit IRC | 17:21 | |
| *** jpich has quit IRC | 17:23 | |
| *** ykarel is now known as ykarel|away | 17:37 | |
| *** e0ne has quit IRC | 17:46 | |
| *** diablo_rojo has joined #openstack-release | 17:54 | |
| *** ykarel|away has quit IRC | 18:25 | |
| *** ekcs has joined #openstack-release | 18:26 | |
| *** armstrong has joined #openstack-release | 18:30 | |
| *** electrofelix has quit IRC | 18:32 | |
| *** hberaud is now known as hberaud|gone | 18:34 | |
| armstrong | @dhell: Hello will the on-boarding training happen here? of which media? | 18:37 |
| *** e0ne has joined #openstack-release | 18:42 | |
| *** e0ne has quit IRC | 18:46 | |
| *** mriedem has quit IRC | 18:48 | |
| *** luizbag has quit IRC | 18:49 | |
| *** mriedem has joined #openstack-release | 18:51 | |
| dhellmann | armstrong : here in channel to start, and then we'll see whether we want to set up a hangout or other video chat | 18:51 |
| dhellmann | diablo_rojo , armstrong : I have an unexpected meeting right now, which should be wrapping up but I need to find a bit of food so I will be a few minutes late | 18:52 |
| armstrong | ok | 18:52 |
| dhellmann | and finding food likely means no video :-) | 18:52 |
| armstrong | hahaha i know right | 18:55 |
| diablo_rojo | dhellmann, no problem | 18:56 |
| * diablo_rojo thinks it would be entertaining if dhellmann tried to eat BBQ ribs while teaching her release things | 18:57 | |
| *** shardy has quit IRC | 18:58 | |
| fungi | that would be quite the feat | 19:01 |
| diablo_rojo | fungi, hence its entertainment value | 19:01 |
| *** e0ne has joined #openstack-release | 19:01 | |
| fungi | i just finished a larger pot of crab gumbo than i probably should have consumed | 19:02 |
| * diablo_rojo has consumed 21 grams of chicken jerkey so far today | 19:02 | |
| armstrong | @fungi: that is fun | 19:02 |
| diablo_rojo | fungi, homemade I would guess? | 19:02 |
| fungi | yeah, though i added some prickly pear and ghost pepper hot sauce i picked up from a local source over the weekend | 19:03 |
| fungi | that stuff is evil-good | 19:03 |
| diablo_rojo | Ha ha ha I wouldn't know. Too spicy for me. | 19:03 |
| *** luizbag has joined #openstack-release | 19:07 | |
| prometheanfire | a friend just made some mango/reaper jelly | 19:08 |
| prometheanfire | had some of his strawberry/reaper stuff, was good | 19:09 |
| * dhellmann returns | 19:11 | |
| dhellmann | diablo_rojo : today's lunch is a deconstructed burrito, because ribs are across town | 19:12 |
| diablo_rojo | dhellmann, 'deconstructed'..have you been watching a lot of chopped or cutthroat kitchen? | 19:12 |
| dhellmann | so, who's hanging around waiting for this onboarding chat? | 19:13 |
| armstrong | me | 19:13 |
| dhellmann | diablo_rojo : in a bowl instead of a tortilla because callories | 19:13 |
| dhellmann | I have some basic team stuff I can share, and then I thought we could go through a couple of reviews | 19:14 |
| dhellmann | before we do that, does anyone have any specific questions they want to have answered today? | 19:14 |
| diablo_rojo | not really? I've read though a ton of the docs and whatnot and nothing has raised questions so far | 19:15 |
| dhellmann | ok, great | 19:15 |
| diablo_rojo | it just seems... weirdly easy? | 19:15 |
| dhellmann | that's the idea :-) | 19:15 |
| armstrong | I will likely have questions as we go on | 19:16 |
| dhellmann | sure, I was more thinking if anyone had specific goals like wanting to know how the release jobs work or something | 19:16 |
| *** luizbag has quit IRC | 19:16 | |
| diablo_rojo | Nothing like that at this point. | 19:16 |
| dhellmann | I think it's best to start by focusing on the start of the workflow, with the releases repo first | 19:16 |
| dhellmann | but just in case.. | 19:16 |
| dhellmann | ok, let's go then | 19:16 |
| diablo_rojo | I think seeing someone review patches is the thing I want to see most. Thrilling I know. | 19:17 |
| dhellmann | at the start of each cycle the team meets up at the ptg/summit/whatever and plans out any major work we're going to do, just like all the other teams | 19:17 |
| dhellmann | we've been using storyboard for that lately, but did sometimes use etherpads | 19:17 |
| dhellmann | during the cycle we also use a "tracking" etherpad with sections for each week to help us keep up with what to do during the cycle | 19:17 |
| dhellmann | the stein pad is in https://etherpad.openstack.org/p/stein-relmgt-tracking | 19:18 |
| dhellmann | you've seen that already from the meetings | 19:18 |
| dhellmann | we populate that with steps based on the process outline, which is documented at https://releases.openstack.org/reference/process.html | 19:18 |
| dhellmann | none of that is likely new to either of you, so I won't go through it in depth unless you have questions about it? | 19:19 |
| armstrong | I am fine so far | 19:19 |
| dhellmann | a few other bits of documentation we have are the user-focused stuff in the project team guide: https://docs.openstack.org/project-team-guide/release-management.html | 19:19 |
| diablo_rojo | I'm keeping up so far too :) | 19:20 |
| dhellmann | and the reviewer guide in the releases repo https://releases.openstack.org/reference/reviewer_guide.html | 19:20 |
| diablo_rojo | Yeah thats one of the things I've been looking at | 19:20 |
| dhellmann | the reviewer guide is something we wrote based on the onboarding discussions I had with Anne, so we're probably going to cover a lot of the same stuff today | 19:20 |
| dhellmann | at the top of the tracking etherpad you'll find a link to a gerrit dashboard query that makes it easy to organize your release reviews | 19:21 |
| dhellmann | http://tiny.cc/a1mkwy | 19:21 |
| dhellmann | that has sections for each cycle, as well as our tools | 19:21 |
| diablo_rojo | A beautifully organized review inbox | 19:22 |
| dhellmann | there are basically 3 types of release reviews to consider | 19:22 |
| dhellmann | the current series, stable series, and independent | 19:22 |
| dhellmann | where independent means not tied to a series and using the "independent" release model | 19:22 |
| diablo_rojo | Makes sense. | 19:23 |
| dhellmann | let's take a look at an independent review first | 19:23 |
| dhellmann | karma-subunit-reporter -- https://review.openstack.org/#/c/625249/ | 19:23 |
| dhellmann | release requests have 3 jobs run to build the docs, apply hard validation rules, and produce the report for the human reviewer | 19:24 |
| dhellmann | the docs job almost never fails, but you never know so... | 19:24 |
| dhellmann | in this case the validate job has failed, and we'll look at that in a second | 19:24 |
| dhellmann | the list-changes job is the one with the report for the human reviewer | 19:25 |
| dhellmann | I'm going to keep going, so just jump in with questions if you have them | 19:25 |
| dhellmann | if you open up the validate job log link http://logs.openstack.org/49/625249/1/check/openstack-tox-validate/2596f89/ and then click on the "tox" directory you'll see a "validate-request-results.log" file | 19:26 |
| armstrong | ok, for the human review we see two reviewers right? | 19:26 |
| dhellmann | that file is the nice human readable report for the job | 19:26 |
| diablo_rojo | Will do. None so far | 19:26 |
| dhellmann | armstrong : well, you're the human reviewer when you're reading the log :-) | 19:26 |
| dhellmann | you can basically ignore the validate job if it passes, but if it fails you might have to help the author of the patch understand why it failed | 19:26 |
| smcginnis | Hah, I always just scan through job-output. Didn't even know it also went to its own log. :) | 19:26 |
| dhellmann | I got tired of parsing that long file or clicking 15 times in the ara report to get it | 19:27 |
| dhellmann | :-) | 19:27 |
| diablo_rojo | smcginnis, learn something new every day I guess :) | 19:27 |
| dhellmann | you can read through the validation report, but there's a summary at the bottom that just includes the warnings and errors | 19:27 |
| smcginnis | Indeed | 19:27 |
| dhellmann | so if you scroll all the way down you'll find what's wrong with this request | 19:27 |
| dhellmann | this is an example of language-specific validation | 19:28 |
| diablo_rojo | Not being able to find 0.0.5 | 19:28 |
| dhellmann | the package being released is javascript, I think | 19:28 |
| diablo_rojo | ? | 19:28 |
| dhellmann | and those packages have the version number inside the file | 19:28 |
| dhellmann | so the file and the tag have to match | 19:28 |
| dhellmann | in this case, the error is that ttx is trying to tag 0.0.5 but the file has 0.0.4 | 19:28 |
| armstrong | @diablo_rojo look at the bottom right | 19:29 |
| dhellmann | the reason behind that discrepancy is interesting for this patch, and that's why I picked it | 19:29 |
| dhellmann | here we're importing the history of an existing package | 19:29 |
| dhellmann | it already has 4 releases, and has been tagged 4 times | 19:29 |
| dhellmann | one of those tags is v0.0.4 and that doesn't match our required pattern because of the prefix "v" | 19:29 |
| smcginnis | I guess one benefit of job-output is you can link to timestamps... | 19:30 |
| smcginnis | http://logs.openstack.org/49/625249/1/check/openstack-tox-validate/2596f89/job-output.txt.gz#_2018-12-14_13_10_27_677787 | 19:30 |
| armstrong | @dhellmann is that the meaning of the 0.0.4 ? | 19:30 |
| dhellmann | so we were going to re-tag that same commit as 0.0.5 to differentiate it from the old v0.0.4 package | 19:30 |
| dhellmann | armstrong : yes, the file has 0.0.4 because that's the valid way to express the version and it was tagged v0.0.4 because the tagger did it by hand and did not realize that was a mistake | 19:30 |
| armstrong | ok | 19:31 |
| openstackgerrit | Matt Riedemann proposed openstack/releases master: nova: release rocky 18.1.0 https://review.openstack.org/625696 | 19:31 |
| dhellmann | now, because of this validation rule we realized we cannot retag it as 0.0.5 because they won't match | 19:31 |
| armstrong | ok | 19:31 |
| diablo_rojo | Okay I think I follow all that, not sure I would have figured it out completely on my own, but I am still with you dhellmann :) | 19:31 |
| dhellmann | yeah, this one is a bit advanced -- right into the deep end of the pool :-) | 19:32 |
| dhellmann | ok, let's look at a stable review next | 19:32 |
| dhellmann | stable pike ironic release: https://review.openstack.org/#/c/624472/ | 19:32 |
| diablo_rojo | Wait a sec. | 19:32 |
| dhellmann | ok | 19:33 |
| diablo_rojo | Trying to words.. | 19:33 |
| diablo_rojo | And read comments qucik | 19:33 |
| diablo_rojo | quick | 19:33 |
| dhellmann | smcginnis : yes, good point on the timestamps. I don't usually find that very useful for these logs, but it can be easier to link to specific lines | 19:33 |
| diablo_rojo | So it needs to make it 0.0.4? instead of 0.0.5? | 19:34 |
| dhellmann | ah, yes, right | 19:34 |
| diablo_rojo | Okay I think I got it. | 19:34 |
| diablo_rojo | Feel free to continue :) | 19:34 |
| dhellmann | we considered 2 options to fix this | 19:34 |
| dhellmann | either make the tag match, or skip the tag entirely | 19:34 |
| dhellmann | making the tag match will require making a new release from the same commit | 19:34 |
| dhellmann | skipping it will mean we leave some version history out of the releases.o.o web site | 19:35 |
| diablo_rojo | skipping entirely was what was done in the first patchset? 0.0.5? | 19:35 |
| diablo_rojo | But the consensus is now to make it match? | 19:35 |
| dhellmann | 0.0.5 was trying to tag the same commit with a new version number, so not quite the same as skipping | 19:35 |
| dhellmann | skipping would just leave that release out of the yaml file entirely | 19:35 |
| diablo_rojo | Ah | 19:35 |
| diablo_rojo | Okay cool. | 19:35 |
| diablo_rojo | I'm with you | 19:36 |
| dhellmann | ok, so back to https://review.openstack.org/#/c/624472/ | 19:36 |
| dhellmann | typically a stable release will only update the 3rd part of the version number, indicating that the release only contains fixes | 19:36 |
| dhellmann | that's what they've done in this case, going from 9.1.5 to 9.1.6 | 19:36 |
| dhellmann | now, as a reviewer, it's our job to ensure that the version information they're communicating is accurate | 19:37 |
| dhellmann | and we do that by applying the semver rules | 19:37 |
| dhellmann | https://docs.openstack.org/pbr/latest/user/semver.html | 19:37 |
| armstrong | @dhellmann we do that manually? | 19:37 |
| dhellmann | yes, that's right | 19:37 |
| dhellmann | if the version is X.Y.Z, backwards-incompatible changes mean incrementing X, features mean incrementing Y, and fixes mean incrementing Z | 19:38 |
| diablo_rojo | I think I've read through all that before.. but will brush up again. | 19:38 |
| diablo_rojo | Yeah that matches what I remember | 19:38 |
| dhellmann | we also require updates to Y for dependency updates | 19:38 |
| diablo_rojo | Example of a dependency update? | 19:39 |
| armstrong | Ok, now I understand this SimVer thing | 19:39 |
| dhellmann | if they update the version of a dependency or add a new one | 19:39 |
| diablo_rojo | Lol okay. | 19:39 |
| dhellmann | so if they go from oslo.config to 1.2 to 1.3 for example | 19:39 |
| diablo_rojo | Got it. | 19:39 |
| diablo_rojo | Just making sure it wasn't more complicated than that. | 19:39 |
| * dhellmann nods | 19:39 | |
| dhellmann | let's look at the list-changes output for this one | 19:40 |
| dhellmann | just like the validate job, there's a separate log file in the tox directory | 19:40 |
| dhellmann | although I'm going to look at the job_output.txt.gz to make it easier to share links here | 19:40 |
| dhellmann | and as a nod to smcginnis ;-) | 19:40 |
| smcginnis | :) | 19:41 |
| dhellmann | the interesting output starts around here: http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_40_59_236798 | 19:41 |
| dhellmann | list-changes runs in a tox job, which conveniently means you can run it yourself locally if you want to | 19:41 |
| diablo_rojo | I like the giant: "NEEDS STABLE POLICY REVIEW" | 19:42 |
| dhellmann | yes, that's intentionally hard to miss | 19:42 |
| dhellmann | since this is a stable release for a project with the stable:follows-policy tag, we give the stable maintenance team "at least one Monday" to review the release | 19:42 |
| dhellmann | in practice that's usually tonyb, but mriedem does help out quite a bit, too | 19:42 |
| diablo_rojo | 'stable team' being like.. tonyb | 19:43 |
| diablo_rojo | Oh cool | 19:43 |
| diablo_rojo | noted | 19:43 |
| dhellmann | 2 release managers can approve faster in cases where we have a serious issue and need a release sooner | 19:43 |
| dhellmann | and 2 release managers can approve if no stable team review happens on the monday | 19:43 |
| dhellmann | something new we've added recently that may not be in the reviewer docs yet is the section starting on http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_41_21_614717 | 19:44 |
| dhellmann | that's more relevant for the master branches, but this team has a nice list of deliverables so it's good to look at the list for context | 19:45 |
| dhellmann | the idea is that we want to ensure teams are not forgetting to release some of their deliverables | 19:45 |
| dhellmann | for example, the client | 19:45 |
| dhellmann | if something hasn't been released, it will show up here without a version number | 19:45 |
| dhellmann | and that's an opportunity for us to nudge the team to consider releasing it | 19:45 |
| diablo_rojo | But we are okay here because they all have been released? | 19:46 |
| dhellmann | yeah, they're fine here | 19:46 |
| diablo_rojo | Cool\ | 19:46 |
| dhellmann | we'll look at this again on the stein review I picked out when we get that far | 19:46 |
| openstackgerrit | Matt Riedemann proposed openstack/releases master: nova: release queens 17.0.8 https://review.openstack.org/625698 | 19:46 |
| *** bobh has quit IRC | 19:47 | |
| dhellmann | the program that produces this report is in the releases repo in openstack_releases/cmds/list_changes.py | 19:47 |
| dhellmann | by default it emits a bunch of information using the logging module at DEBUG level | 19:47 |
| dhellmann | you can almost always ignore that, but it helps when there's a bug in the program | 19:47 |
| dhellmann | so if we skip over all of this debug output and scroll down a bit we'll see where it starts showing the details about this new release | 19:48 |
| dhellmann | http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_41_25_106654 | 19:48 |
| armstrong | level 2? | 19:48 |
| dhellmann | I don't know the numbers for the different levels off the top of my head :-) | 19:48 |
| dhellmann | what I mean is that the output with "DEBUG:" at the start of the line can be ignored | 19:48 |
| dhellmann | for example http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_41_21_724964 | 19:48 |
| dhellmann | make sense? | 19:48 |
| *** bobh has joined #openstack-release | 19:49 | |
| armstrong | yes | 19:49 |
| armstrong | so far I am ok | 19:49 |
| dhellmann | ok | 19:49 |
| diablo_rojo | Yep. Still with you. | 19:49 |
| dhellmann | the reviewer guide explains what each of these sections contains, so I'm going to focus on the bits that are useful for this individual review | 19:49 |
| dhellmann | in this case for example, I want to see what git thinks the version is now for the commit being tagged | 19:50 |
| dhellmann | that's the first section | 19:50 |
| dhellmann | git describe shows 9.1.5-5-$sha which means 5 commits after 9.1.5 | 19:50 |
| dhellmann | I think the validation program would have complained if they had tried to tag this with a version less than 9.1.5, but it's nice to see the confirmation here | 19:51 |
| dhellmann | the next thing of interest is "Branches containing commit" | 19:51 |
| dhellmann | that shows all of the branches containing the commit being tagged | 19:51 |
| dhellmann | again, the validation logic would have complained if this pike release didn't match the pike branch | 19:52 |
| dhellmann | that validation rule isn't run for independent releases, though, so the output is still included here | 19:52 |
| dhellmann | the "Relationship to HEAD" section shows whether the commit being tagged is the tip of the branch | 19:52 |
| diablo_rojo | Makes sense | 19:52 |
| dhellmann | http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_41_25_345746 | 19:53 |
| *** N3l1x has quit IRC | 19:53 | |
| dhellmann | if it's not, I usually ask the author if they intended to leave some commits unreleased | 19:53 |
| dhellmann | especially on the stable branch, where releases are less common | 19:53 |
| dhellmann | or at least less frequent | 19:53 |
| smcginnis | It does happen from time to time for various reasons, but usually an oversight. | 19:53 |
| *** bobh has quit IRC | 19:54 | |
| dhellmann | down at "Requirements Changes" we can see the various places we look for dependency changes | 19:54 |
| dhellmann | http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_41_25_869572 | 19:54 |
| dhellmann | smcginnis : yes, good point | 19:54 |
| dhellmann | we diff the various requirements files and then in the next block we diff setup.cfg as well | 19:54 |
| dhellmann | here there are no dependency changes, so that part of the semver rules is satisfied with the version number they have specified | 19:55 |
| dhellmann | skipping down a couple of sections to "Release 9.1.6 will include" we can see the list of commits that will be in this release | 19:55 |
| dhellmann | http://logs.openstack.org/72/624472/1/check/releases-tox-list-changes/8fc4baa/job-output.txt.gz#_2018-12-11_19_41_26_169457 | 19:55 |
| dhellmann | this graph view summary, and the detailed output that follows, is the primary thing you're going to be looking at in this report | 19:55 |
| dhellmann | you want to look through the list for things that appear to be features or backwards-incompatible changes | 19:56 |
| diablo_rojo | To make sure they are all just fixes and not features? | 19:56 |
| dhellmann | most of the teams understand semver now, so we don't have a lot of issues | 19:56 |
| dhellmann | right | 19:56 |
| dhellmann | the thing that needs to change based on this content is almost always the version number | 19:57 |
| dhellmann | although we did have one case where someone backported a configuration option and then it was reverted | 19:57 |
| *** bobh has joined #openstack-release | 19:57 | |
| dhellmann | in this case, everything looks like a bug fix or CI change | 19:57 |
| dhellmann | and it's a nice short list | 19:58 |
| dhellmann | if you're comfortable that this release is OK, go ahead and +1 the request | 19:58 |
| diablo_rojo | Releasing often I imagine is helpful to keep them more reviewable. | 19:58 |
| dhellmann | otherwise, ask any questions you have about it | 19:59 |
| dhellmann | yes, we try to encourage teams to release as often as possible | 19:59 |
| dhellmann | the oslo team usually submits 1 request a week with several libraries in it, for example | 19:59 |
| dhellmann | although the sheer bulk there has slowed down somewhat lately | 19:59 |
| diablo_rojo | Seems logical to me | 19:59 |
| diablo_rojo | +1ed | 20:00 |
| dhellmann | every reviewer who votes on the patch will have their name attached to the message that goes along with the git tag | 20:00 |
| dhellmann | that's one reason we encourage PTLs to +1 the final release request at the end of the cycle, for example | 20:00 |
| diablo_rojo | Kind of like a signoff. 'I'm so and so and I approve this release' | 20:01 |
| dhellmann | now, this request was filed on the 11th, which was last tuesday | 20:01 |
| dhellmann | right | 20:01 |
| dhellmann | today is monday where I am, which means tomorrow it would be safe for me to approve it | 20:01 |
| dhellmann | I'll give it a +2 today though | 20:01 |
| diablo_rojo | You usually wait for like a week before giving the +W? | 20:02 |
| dhellmann | "at least one Monday" | 20:02 |
| dhellmann | tonyb does his reviews on monday | 20:02 |
| dhellmann | :-) | 20:02 |
| fungi | that's for stable-branch release requests specifically though, right? | 20:02 |
| *** bobh has quit IRC | 20:02 | |
| dhellmann | so if they had filed it Friday, I would still go ahead with tomorrow | 20:02 |
| dhellmann | yes, that's right, that only applies to stable releases | 20:02 |
| fungi | for releases on master you don't incur the additional delay? | 20:02 |
| dhellmann | releases on master go out as soon as the reviewer can get to them | 20:02 |
| fungi | thanks for the clarification | 20:03 |
| dhellmann | we'll go through and approve on on master next so we can talk about the jobs that run | 20:03 |
| * fungi is secretly following along | 20:03 | |
| dhellmann | yep, thanks for the questions | 20:03 |
| armstrong | @dhellmann: when do you give a +2 ? | 20:03 |
| dhellmann | armstrong : for this pike release I gave a +2 just a minute ago, and I will click the workflow button tomorrow if someone else doesn't do it before I get to it | 20:04 |
| dhellmann | this is a good time to talk about review scheduling | 20:04 |
| dhellmann | the team usually breaks down the week so we each cover 1-2 days of reviews | 20:04 |
| dhellmann | I have monday and tuesday, ttx has wednesday, and smcginnis has thursday | 20:04 |
| dhellmann | although smcginnis cheats and does reviews all the time | 20:05 |
| dhellmann | :-) | 20:05 |
| fungi | and friday is a free day! | 20:05 |
| smcginnis | I have Monday too, but I do try to keep on eye on things. | 20:05 |
| diablo_rojo | smcginnis, the line budger :) | 20:05 |
| dhellmann | we used to not do releases on fridays because it ruins the weekend when something goes wrong | 20:05 |
| dhellmann | and now I think we do allow releases, at the discretion of the reviewer | 20:05 |
| diablo_rojo | But no one wants to be THAT person I imagine :) | 20:06 |
| armstrong | @diablo_rojo hahaha ikr | 20:06 |
| smcginnis | We have a second level of protection that most things that will break others also need to go through a requirements update review. | 20:06 |
| smcginnis | But still, best to avoid that. | 20:06 |
| dhellmann | we break the world far less often now that we have the constraints system in place in the requirements repository | 20:06 |
| dhellmann | but yeah | 20:06 |
| dhellmann | ok, let's look at a review from master | 20:07 |
| dhellmann | swift 2.20.0 — https://review.openstack.org/#/c/625483/ | 20:07 |
| dhellmann | in particular on this one I want you to start by looking at the first revision of the patch https://review.openstack.org/#/c/625483/1 | 20:07 |
| dhellmann | and see if you can figure out why the validation job failed there | 20:07 |
| ttx | sorry could not join earlier | 20:07 |
| diablo_rojo | jumped two Y places | 20:08 |
| dhellmann | hi, ttx | 20:08 |
| diablo_rojo | Hello ttx :) I'm learning things! | 20:08 |
| notmyname | diablo_rojo: hint, I made a copy/paste error :-) | 20:08 |
| ttx | Great! | 20:08 |
| diablo_rojo | notmyname, lol | 20:08 |
| dhellmann | notmyname : sorry to use you as an example, but thanks for providing one :-) | 20:08 |
| notmyname | :-) | 20:09 |
| diablo_rojo | copy/paste is deceptively difficult | 20:09 |
| dhellmann | where would you look for the error message? | 20:09 |
| diablo_rojo | validate-request-results.log I would guess | 20:10 |
| dhellmann | yep, and what does the error there say? | 20:10 |
| fungi | 1 errors found | 20:10 |
| * dhellmann is secretly focus grouping his error message phrasing | 20:10 | |
| mriedem | dhellmann: do i get a pass on the tonyb stable release request thing because i'm a stable branch jockey? | 20:11 |
| dhellmann | mriedem : tonyb volunteered and is more active in release team planning, but we call on you when we need a hand | 20:12 |
| dhellmann | I'm sure smcginnis would love to have you doing reviews regularly, too | 20:12 |
| smcginnis | Absolutely | 20:12 |
| mriedem | i just mean for the stuff i'm pushing today, | 20:12 |
| ttx | historically mriedem got a free pass for being stable-maint-core yes | 20:12 |
| mriedem | do i need to wait a week for tony to review that? | 20:12 |
| dhellmann | but we're also aware that you're doing a ton of other things | 20:12 |
| dhellmann | mriedem : oh, sorry | 20:12 |
| fungi | dhellmann: so to focus group that error message a bit for you... ;) | 20:12 |
| dhellmann | I think we can probably let it slide | 20:12 |
| mriedem | b/c frickler was requesting these releases for a couple of weeks | 20:13 |
| mriedem | ok | 20:13 |
| fungi | dhellmann: i did find the phrasing obscure until i looked back at https://review.openstack.org/#/c/625483/1..2/deliverables/stein/swift.yaml | 20:13 |
| dhellmann | yeah, that one could be a little more verbose | 20:13 |
| fungi | dhellmann: and also https://git.openstack.org/cgit/openstack/swift to confirm my hunch | 20:13 |
| dhellmann | does the error make sense? esp. given notmyname's hint? | 20:13 |
| diablo_rojo | All I see is: Could not find 2.20.0? | 20:14 |
| dhellmann | diablo_rojo : I think you're looking at the logs for the most recent patch | 20:14 |
| dhellmann | the error was in the first version | 20:14 |
| dhellmann | http://logs.openstack.org/83/625483/1/check/openstack-tox-validate/123e866/tox/validate-request-results.log | 20:14 |
| fungi | very last line | 20:14 |
| dhellmann | I planned out this exercise expecting that to stay the current version of the patch, so it's a little confusing, sorry for that | 20:15 |
| armstrong | 1 errors found | 20:15 |
| armstrong | deliverables/stein/swift.yaml: validate_existing_tags: Version 2.18.0 in openstack/swift is on commit 'f270466de363499894317b7c671f65e8a912bd53' instead of '184fdf17ef7490038e89fe92a92de0fe4f2b36b7' | 20:15 |
| dhellmann | that's it | 20:15 |
| fungi | yep, that | 20:15 |
| diablo_rojo | So failed because 2.18.0 had already been tagged | 20:15 |
| dhellmann | right | 20:15 |
| ttx | ARA can be useful on those too | 20:15 |
| notmyname | dhellmann: FWIW, the error message was very obvious to me when I woke up this morning (but I know *how* I created the patch, so I have an advantage there) | 20:16 |
| dhellmann | that's a surprisingly common error, in part because of the copy-and-paste thing and also sometimes when people start out a new cycle and don't pay close attention to the previous cycle | 20:16 |
| fungi | it probably helps to know a little what this job is doing first, right. if you know that the job is testing out creating the requested tag and then checking that what got created is what was requested, then the possible reasons for that particular error are fairly few in number | 20:16 |
| dhellmann | notmyname : yeah, good point. it really should be obvious to the author of the patch, so that's good | 20:16 |
| dhellmann | fungi : that's a good mental model, although that's not actually how this test works | 20:17 |
| dhellmann | at least iirc, it just looks at the sha of the existing tag | 20:17 |
| fungi | oh, right, because it's checking potentially existing tags? | 20:17 |
| dhellmann | right | 20:17 |
| fungi | since this job could be run at any time for any tag, not just the one(s) being requested | 20:18 |
| fungi | so it's highlighting a mismatch between the yaml document which asserts a particular tag->commit mapping and what it's actually found in the repository? | 20:18 |
| dhellmann | right, it does ensure that any existing tags that are imported into the yaml files have the right settings | 20:18 |
| dhellmann | exactly | 20:18 |
| fungi | i guess it would help for me to look at the log from the working patchset | 20:19 |
| dhellmann | if you scroll up to the "Ensure tags that exist point to the SHAs listed" section you'll see it doing that work | 20:19 |
| fungi | yeah, so the various ERROR lines in the working run are how we know about new not-previously-existent tags | 20:20 |
| * ttx is called back to his regular evening activities | 20:20 | |
| dhellmann | right | 20:20 |
| ttx | looks like dhellmann has this under control | 20:20 |
| * dhellmann hands ttx a glass of wine | 20:20 | |
| fungi | i was about to ask ttx to bring us wine for this ;) | 20:20 |
| * ttx tastes and says " Hmm... not bad" | 20:20 | |
| dhellmann | so, let's talk about the special case this presents because it is the first release for this deliverable this cycle | 20:21 |
| dhellmann | the previous release in rocky was 2.19.0 | 20:21 |
| * ttx leaves the bottle on the table for the students to finish | 20:21 | |
| dhellmann | this release needs to be at least 2.20.0 | 20:21 |
| dhellmann | would anyone care to guess why? | 20:21 |
| armstrong | X=20.Y(changing).Z=0 | 20:22 |
| dhellmann | as a hint, I'll remind you that the releases are on different branches | 20:22 |
| fungi | i was in on this discussion so i'll refrain from giving it away | 20:23 |
| dhellmann | even if this release had only had bug fixes, we would have required a Y version increase as the first release of the new cycle | 20:23 |
| armstrong | ok | 20:23 |
| dhellmann | if we did not do that, and allowed 2.19.1 from master then there would have been no room for more patch releases on the rocky stable branch | 20:23 |
| dhellmann | we wouldn't want 2.19.2 to come from rocky if 2.19.1 came from stein, because not everything was backported | 20:24 |
| dhellmann | so the first release for each deliverable in each cycle must increment at least the minor version number | 20:24 |
| fungi | also, not a theoretical concern. it's happened before | 20:24 |
| dhellmann | many increment the major version number instead, and that's fine, too | 20:24 |
| dhellmann | yeah, I'm giving you all the tricky edge cases I see today because the easy stuff is in the docs you've read :-) | 20:25 |
| dhellmann | let's all vote on this patch and then we can approve the release and what what happens | 20:25 |
| dhellmann | unless anyone has questions? | 20:25 |
| openstackgerrit | Matt Riedemann proposed openstack/releases master: nova: release pike 16.1.7 https://review.openstack.org/625709 | 20:25 |
| * diablo_rojo is still processing for a sec | 20:26 | |
| dhellmann | k | 20:26 |
| diablo_rojo | Okay got it now | 20:27 |
| diablo_rojo | just had to reread a few times lol | 20:27 |
| dhellmann | no worries | 20:27 |
| dhellmann | go ahead and give this one a +1 and then I'll approve it | 20:28 |
| diablo_rojo | Done | 20:28 |
| dhellmann | ok, while I approve it, open up http://zuul.openstack.org/status | 20:28 |
| dhellmann | and then put "release,swift" in the search box on the left side | 20:28 |
| dhellmann | and click "expand by default" | 20:28 |
| armstrong | @dhellmann please for the link again | 20:28 |
| dhellmann | https://review.openstack.org/#/c/625483/ | 20:29 |
| armstrong | i got so many opened | 20:29 |
| dhellmann | haha, I know how you feel | 20:29 |
| dhellmann | armstrong : I'm going to step away for a second while you vote, please ping me when you're done | 20:30 |
| dhellmann | brb | 20:30 |
| armstrong | ok | 20:30 |
| armstrong | @diablo_rojo remind me on voting :) I want to give a +1 | 20:32 |
| diablo_rojo | armstrong, the reply button at the top | 20:33 |
| armstrong | Yes I click on it and added myself but didn't see the vote | 20:33 |
| armstrong | ok good now | 20:33 |
| armstrong | thx | 20:33 |
| diablo_rojo | Cool :) | 20:34 |
| dhellmann | ok, there we go | 20:34 |
| dhellmann | so, everyone open up your page to watch the zuul status | 20:34 |
| dhellmann | http://zuul.openstack.org/status | 20:34 |
| dhellmann | and in the search box on the left put "release,swift" | 20:34 |
| dhellmann | that will limit the jobs to the release jobs and anything running for swift | 20:34 |
| dhellmann | that way we can watch what happens when we approve this release | 20:35 |
| dhellmann | let me know when you're ready | 20:35 |
| * dhellmann doesn't want anyone to miss the excitement | 20:35 | |
| armstrong | ok | 20:35 |
| diablo_rojo | Ha ha I'm there | 20:35 |
| dhellmann | ok | 20:36 |
| dhellmann | the jobs for managing releases happen in 2 phases | 20:36 |
| dhellmann | we have a bunch of validation in the releases repo, and then when a patch merges there another job adds the tag to the thing being released | 20:36 |
| dhellmann | the tag event triggers other jobs | 20:36 |
| dhellmann | those depend on the repo and language of the deliverable | 20:36 |
| dhellmann | I suppose the gate is backed up enough right now I didn't need to worry about anyone missing anything :-) | 20:37 |
| diablo_rojo | Interesting | 20:37 |
| diablo_rojo | lol | 20:37 |
| dhellmann | that 2 part design comes from 2 things | 20:37 |
| dhellmann | first, it gives us a nice separation of concerns between managing the tagging and doing something when the tag happens | 20:37 |
| dhellmann | second, the tag jobs existed when we built the releases repo :-) | 20:37 |
| dhellmann | so it was sort of an obvious thing to keep them separate | 20:38 |
| dhellmann | there are several pipelines related to the tag event | 20:38 |
| dhellmann | tag, release, and pre-release | 20:38 |
| dhellmann | jobs that should run against any tag usually go in the tag pipeline | 20:38 |
| dhellmann | pre-release and release are different based on the type of version number | 20:39 |
| dhellmann | a pre-release version is an alpha, beta, or release candidate | 20:39 |
| dhellmann | a regular release version is anything that is not one of those 3 | 20:39 |
| dhellmann | at least in our terminology | 20:39 |
| diablo_rojo | Got it. | 20:39 |
| fungi | yeah, we perform a regular expression match on the tag name to determnie whether it's a pre-release or release | 20:39 |
| dhellmann | the release-post pipeline is a special version of the post pipeline that you're used to seeing in other contexts | 20:39 |
| dhellmann | it used to have a different priority, but now the main difference is that failures trigger an email to be sent to the release-failures mailing list | 20:40 |
| dhellmann | http://lists.openstack.org/cgi-bin/mailman/listinfo/release-job-failures | 20:40 |
| fungi | and if you don't feel like subscribing to that directly, you can always just refresh the web archive for it to see any new failures | 20:41 |
| dhellmann | you've probably noticed one of the release managers replying to one of those messages and directing the conversation to the -dev or -discuss list to ensure a team is aware of the issue with their release | 20:41 |
| fungi | but yeah, being subscribed does make it easier to follow up on the failure messages | 20:41 |
| diablo_rojo | Yep, definitely seen that. | 20:41 |
| dhellmann | so what we expect to have happen now is the gate jobs pass, the patch merge, and then some new jobs start in the release-post pipeline | 20:42 |
| armstrong | we are on gate 24 right? | 20:43 |
| dhellmann | yeah, if you put the search query in you should see 1 item for openstack/releases running 2 jobs, openstack-tox-docs and openstack-tox-validate | 20:44 |
| dhellmann | those will take another couple of minutes to finish | 20:44 |
| fungi | armstrong: an amusing interpretation. that's actually saying there are 24 changes enqueued in the gate pipeline right now, but you only see one because of the search filter | 20:44 |
| fungi | 23 now, you'll see it just updated | 20:44 |
| fungi | and it's back to 24 again | 20:45 |
| armstrong | ok I got it | 20:45 |
| armstrong | thanks @fungi: | 20:45 |
| dhellmann | that validation job is almost done | 20:45 |
| fungi | we have a zuul-status graph dashboard where you can see those counts being trended at the top: http://grafana.openstack.org/d/T6vSHcSik/zuul-status | 20:45 |
| openstackgerrit | Merged openstack/releases master: swift 2.20.0 release https://review.openstack.org/625483 | 20:46 |
| fungi | and it merged! | 20:46 |
| dhellmann | there we go | 20:46 |
| dhellmann | and now there is 1 item in the release-post pipeline with 2 jobs | 20:46 |
| dhellmann | the tag-releases job is the one we're interested in | 20:46 |
| diablo_rojo | Such a pretty CI system | 20:46 |
| dhellmann | that actually creates branches, too, when someone asks for them | 20:46 |
| dhellmann | when the job starts the name will become a link to the console output, and we can watch what it does | 20:47 |
| fungi | you'll see it's a 7-digit hexidecimal number, which is an abbreviation of the first 7 digits of the merge commit which resulted from the merger of the change we were watching in the gate pipeline earlier | 20:47 |
| dhellmann | the tools for tag-releases are all in openstack-infra/project-config/roles/copy-release-tools-scripts/files/release-tools | 20:48 |
| dhellmann | that location is an artifact of the migration from the original implementation to the zuul v3 version of the job | 20:48 |
| dhellmann | we'll save going through the scripts for another session, I think | 20:48 |
| dhellmann | ok, you should be able to open the tag-releases console up in another window now | 20:49 |
| dhellmann | it starts with some typical node configuration steps like ssh and bindep | 20:49 |
| dhellmann | and a bunch of other things, I guess -- I don't know what a lot of those are, tbh | 20:49 |
| diablo_rojo | Lol | 20:50 |
| dhellmann | mostly setting up the node to know about our infra | 20:50 |
| dhellmann | now it's copying those tools into place to run them | 20:51 |
| dhellmann | and setting up gpg so it can sign a tag | 20:51 |
| dhellmann | and now it runs the main release job content | 20:51 |
| dhellmann | which finds the new tag in the deliverable, and applies it to the repo | 20:51 |
| armstrong | @dhellmann "ok, you should be able to open the tag-releases console up in another window now" I am lost here | 20:52 |
| dhellmann | and then comments on all of the bugs closed by that release to indicate when the fix was released | 20:52 |
| dhellmann | http://zuul.openstack.org/stream/872854be92614c42805a9dead6576b97?logfile=console.log | 20:52 |
| armstrong | where is the console? | 20:52 |
| armstrong | thanks | 20:52 |
| dhellmann | the name of the job is a link to the console | 20:52 |
| armstrong | ok i see | 20:52 |
| dhellmann | now, if you go back to the zuul pipeline status page we should see the swift tagging jobs start up soon | 20:52 |
| dhellmann | and there they go | 20:53 |
| diablo_rojo | Beautiful :) | 20:53 |
| dhellmann | release notes in the tag pipeline, and the release itself in the release pipeline | 20:53 |
| dhellmann | the 3 jobs in the release pipeline depend on each other | 20:53 |
| dhellmann | release-openstack-python has to complete before the other 2 run | 20:54 |
| dhellmann | announce-release sends email with details of the new release to the release-announce list | 20:54 |
| dhellmann | http://lists.openstack.org/cgi-bin/mailman/listinfo/release-announce | 20:54 |
| dhellmann | propose-update-constraints is the interface with the openstack/requirements repo | 20:54 |
| dhellmann | when anything is released, we look at the list of constraints to see if it is present | 20:55 |
| dhellmann | if it is, we propose a new patch to that repo to update the constraint to the new version | 20:55 |
| dhellmann | that triggers a bunch of tests, which tells us if the new package is compatible with the other things in the list and the software that consumes it | 20:55 |
| dhellmann | prometheanfire can tell you lots more about how all of that works | 20:55 |
| dhellmann | oh, and of course release-openstack-python is the job that packages the deliverable and publishes it to pypi.python.org | 20:56 |
| dhellmann | as well as tarballs.openstack.org | 20:56 |
| dhellmann | that job would be different if this was a javascript repo or puppet repo | 20:56 |
| fungi | and signs it with our release automation openpgp key | 20:56 |
| dhellmann | yes, good point | 20:56 |
| dhellmann | looking at http://git.openstack.org/cgit/openstack/swift/refs/ we can see the new 2.20.0 tag at the top of the list | 20:57 |
| dhellmann | note that the author is the bot | 20:57 |
| dhellmann | but if you look at the tag metadata in http://git.openstack.org/cgit/openstack/swift/tag/?h=2.20.0 | 20:57 |
| dhellmann | you will see all of our names | 20:57 |
| * diablo_rojo feels important | 20:58 | |
| dhellmann | since we're not going to look at the details of the actual release job today, I think we can start wrapping up | 20:59 |
| fungi | that pgp signature at the bottom of the tag details is made using the same release automation openpgp key i mentioned earlier as well | 20:59 |
| dhellmann | are there any questions about what we did cover? | 20:59 |
| prometheanfire | sup? | 21:00 |
| fungi | prometheanfire: wave to the class | 21:00 |
| dhellmann | prometheanfire : hey, we're just doing some team onboarding with diablo_rojo and armstrong | 21:00 |
| fungi | the release-openstack-python job just finished running too | 21:00 |
| dhellmann | I referred them to you if they have questions about how constraints work | 21:00 |
| diablo_rojo | What is the current distribution of humans to days in terms of when people are reviewing? | 21:00 |
| diablo_rojo | My only question | 21:00 |
| armstrong | suppose a job fails at zuul will that affets the version number? X.Y.Z | 21:00 |
| dhellmann | and now you can see the new swift release on pypi at https://pypi.org/project/swift/#history | 21:00 |
| prometheanfire | o/ | 21:01 |
| notmyname | ...and I just told the superuser people that the article on 2.20.0 can be published :-) | 21:01 |
| dhellmann | diablo_rojo : we have 4 people with acls to approve releases | 21:01 |
| dhellmann | notmyname : woot! | 21:01 |
| fungi | notmyname: don't you feel like another cog in the release automation machine now? ;) | 21:01 |
| dhellmann | diablo_rojo : tonyb mostly focuses on the stable ones, but does get into the others sometimes | 21:01 |
| dhellmann | fungi , notmyname : we need to automate that! | 21:01 |
| dhellmann | and we tend to focus on the start of the week | 21:02 |
| dhellmann | if you're looking for a day to pick up, I think Tuesday is probably a good one, since Sean and I both have Monday covered and that means I could drop Tuesday | 21:02 |
| dhellmann | things tend to pile up over the weekend, so we want good monday coverage | 21:03 |
| dhellmann | armstrong : it depends | 21:03 |
| dhellmann | if the reason for the release is an infra failure of some sort or a bug in the release script, we can often re-enqueue the tag event and trigger the jobs again | 21:03 |
| dhellmann | whether we can do that does depend somewhat on where in the process the failure happens | 21:04 |
| dhellmann | if the package is uploaded to pypi, for example, we can't redo it because uploading the same version again is considered an error | 21:04 |
| dhellmann | if the reason for the failure is something in the repo itself, then we need the team to fix it | 21:04 |
| dhellmann | we had failures like that before we added the validation steps to check the javascript version file content with the tag, for example | 21:05 |
| *** dmellado has quit IRC | 21:05 | |
| dhellmann | we have less of those now, but they do still happen in some edge cases | 21:05 |
| dhellmann | as part of the work we did this cycle, we added a release test job to all of the python repos | 21:05 |
| dhellmann | that job tries to build a package if any of the packaging metadata has changed | 21:05 |
| *** stevebaker has quit IRC | 21:05 | |
| dhellmann | and it also verifies the content of the README.rst using twine | 21:06 |
| diablo_rojo | dhellmann, yeah I can definitely take on Tuesday | 21:06 |
| dhellmann | great! let's get you doing some reviews and then we can make it official | 21:06 |
| fungi | and while we're talking about failures, the zuul builds interface lets you query for failures (and other sorts of stuff too). so if you wanted to look at a variety of unsuccessful jobs in the pre-release, release, tag and release-post pipelines you could query it for | 21:06 |
| dhellmann | so every time we find an issue with the content of a package causing a problem, we try to add a new validation step up front to avoid having that trigger a release job failure | 21:06 |
| fungi | http://zuul.openstack.org/builds?pipeline=pre-release&pipeline=release&pipeline=tag&pipeline=release-post&result=FAILURE&result=POST_FAILURE&result=RETRY_LIMIT | 21:06 |
| dhellmann | but those classes of failures almost always require a new version number, because we cannot change a tag once we publish it | 21:07 |
| dhellmann | oh, nice, thanks fu | 21:07 |
| dhellmann | fungi | 21:07 |
| fungi | in theory that's mostly the same thing which gets reported to the release failures ml | 21:07 |
| armstrong | thanks I got it now | 21:07 |
| fungi | but accessible via a web dashboard and an api | 21:08 |
| dhellmann | yeah, most of the ones at the top of the list look familiar | 21:08 |
| clarkb | not sure if it was mentioned but the infra team tries to warn the release channel when we are doing work that may impact releases as well | 21:08 |
| clarkb | so checking there isn't any arnings about that before approving things is helpful | 21:08 |
| dhellmann | that's a good tidbit, thanks, clarkb | 21:08 |
| fungi | yep, or when there are other exceptional circumstances which may make release activity especially unsafe | 21:08 |
| dhellmann | those notices come here in channel, so always look at the scrollback for info before starting to approve releases | 21:08 |
| * diablo_rojo got sucked into another meeting, but it should be short | 21:08 | |
| diablo_rojo | Will catch up in a little bit | 21:08 |
| dhellmann | I think we're about done here | 21:09 |
| dhellmann | if you encounter questions as you do reviews, just ask either here in channel or on the mailing list | 21:09 |
| dhellmann | or on reviews, even | 21:09 |
| dhellmann | diablo_rojo, armstrong, and fungi : thanks for participating today, and I hope this was useful | 21:09 |
| fungi | i found it very helpful myself! | 21:10 |
| fungi | thanks for setting aside the time | 21:10 |
| diablo_rojo | dhellmann, very! Thanks for doing the walk through | 21:10 |
| armstrong | very helpful to me and wish I practice more to grasp it better | 21:10 |
| tonyb | ZOMG soo much scrollback | 21:11 |
| dhellmann | I encourage you to dig into the validation code a bit to see what the error messages mean, and help with wording if you think those can be clarified | 21:11 |
| dhellmann | tonyb : your name was invoked, but only in a good way :-) | 21:11 |
| tonyb | dhellmann: I did see that \o/ #phew | 21:11 |
| dhellmann | I'm going to drop offline for a bit to hang lights on the front of the house in preparation for our saturnalia celebration | 21:12 |
| dhellmann | I'll check in after it's too dark to be on a ladder | 21:12 |
| fungi | dhellmann: i don't even want to know how you decorate for lupercalia | 21:17 |
| tonyb | dhellmann: (when you're back) re: http://eavesdrop.openstack.org/meetings/releaseteam/2018/releaseteam.2018-12-14-15.00.log.html#l-107 | 21:21 |
| tonyb | I can't do 1600UTC as it's 0300am for me | 21:22 |
| tonyb | It's be great if the meeting were at a time I could attend but I think given it's only me down here keeping it at time that works for US+EU is fine | 21:23 |
| smcginnis | Yeah, wasn't sure that would be a great time for you. You're more of a morning person than a night owl from what I've seen. | 21:23 |
| * tonyb doesn't always read the logs but as always you can make me read at least parts of them my saying my name ;P | 21:23 | |
| smcginnis | :) | 21:24 |
| smcginnis | tonyb: Would still be nice to get your thoughts on https://review.openstack.org/#/c/625290/ | 21:24 |
| tonyb | Yeah. I wake just befoer 0600, I can be on a meeting by 0630, which given DST translates to about 2030->0930 UTC | 21:25 |
| smcginnis | tonyb: Looks like we would either have to lose you or ttx | 21:26 |
| tonyb | smcginnis: Well evrardjp and ttx should be about the same TZ wise so that's 2 in EU vs 1 in AU | 21:27 |
| *** tobias-urdin has joined #openstack-release | 21:29 | |
| tonyb | smcginnis: commented | 21:29 |
| tonyb | smcginnis: hopefully it makes sense and is slightly funny ;P | 21:30 |
| smcginnis | :) | 21:33 |
| smcginnis | tonyb: I didn't think the start dates really mattered too much since a bunch of different ones on different days all had the same start date. | 21:33 |
| *** bobh has joined #openstack-release | 21:33 | |
| *** mlavalle has quit IRC | 21:33 | |
| tonyb | smcginnis: Going back 1 step, if the meeting was at 2030 UTC that'd be late for ttx and evrardjp, early for me and reasonable for the US which might work I guess depending on family dynamics | 21:33 |
| tonyb | smcginnis: they were added as a 'default' when we added that feature | 21:34 |
| smcginnis | tonyb: How is that start date used? | 21:35 |
| tonyb | smcginnis: if you se thew startdate right you become immue to the bugs created by ISO week 53 being followed by ISO week 1 meaning you have 2 odd weeks in a row | 21:35 |
| tonyb | smcginnis: it's also used when we create the ICS file to all the 'old' release meetings will disappear and the first one will start in $start_date | 21:36 |
| * tonyb tries something to illustrate | 21:37 | |
| smcginnis | Cool, I can get that updated. Just was curious as I didn't see it reflected anywhere. | 21:37 |
| tonyb | [tony@thor irc-meetings]$ curl -s http://eavesdrop.openstack.org/calendars/release-team-meeting.ics | grep 11 | 21:38 |
| tonyb | DTSTART;VALUE=DATE-TIME:20161111T150000Z | 21:38 |
| tonyb | http://paste.openstack.org/show/737511/ | 21:40 |
| tonyb | sorry I probably shoudl have remove the tox output to make it standout | 21:40 |
| *** armstrong has quit IRC | 21:41 | |
| tonyb | smcginnis: you DO subsrcibe to release-team-meeting.ics right? | 21:41 |
| smcginnis | I probably should. ;) | 21:42 |
| *** bobh has quit IRC | 22:20 | |
| *** bobh has joined #openstack-release | 22:22 | |
| *** e0ne has quit IRC | 22:25 | |
| *** bobh has quit IRC | 22:56 | |
| *** mlavalle has joined #openstack-release | 23:06 | |
| dhellmann | fungi : no, you don't want to know. ;-) | 23:16 |
| *** mlavalle has quit IRC | 23:27 | |
| *** mriedem has quit IRC | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!