16:00:03 <smcginnis> #startmeeting releaseteam
16:00:03 <openstack> Meeting started Thu Feb 28 16:00:03 2019 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:07 <openstack> The meeting name has been set to 'releaseteam'
16:00:10 <smcginnis> Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong
16:00:19 <smcginnis> #link https://etherpad.openstack.org/p/stein-relmgt-tracking Agenda
16:00:19 <dhellmann> o/
16:00:22 <fungi> aloha
16:00:33 <smcginnis> Getting really far down in that tracking etherpad now.
16:00:51 <diablo_rojo> o/
16:01:40 <evrardjp> o/
16:01:52 <ttx> o/
16:02:10 <smcginnis> #topic TripleO release issues
16:02:22 <smcginnis> #link https://review.openstack.org/#/c/637857
16:02:30 <smcginnis> #link https://review.openstack.org/#/c/637858/
16:02:40 <smcginnis> #link https://review.openstack.org/#/c/638459/
16:02:40 <smcginnis> Tasks update
16:02:51 <smcginnis> Oops, a little too much grabbed on that last one. :)
16:03:00 <smcginnis> Not sure who added this topic.
16:03:05 <hberaud> o/
16:03:45 <smcginnis> http://logs.openstack.org/57/637857/4/check/openstack-tox-validate/502eeab/job-output.txt.gz#_2019-02-22_10_12_23_715253
16:04:17 <smcginnis> Looks like they need a release job defined and they have errors in their readme.
16:04:21 <evrardjp> it looks like ttx might have added this?
16:04:42 <smcginnis> ttx: This one yours?
16:04:44 <evrardjp> see also last comments on https://review.openstack.org/#/c/637857
16:04:50 <ttx> yes
16:05:10 <ttx> It looks like this pile of things are blocked
16:05:58 <ttx> First do fail due to a validation error
16:06:13 <ttx> (missing release job for instack-undercloud)
16:06:22 <ttx> but apparently fixing that is not the solution
16:06:30 <ttx> see https://review.openstack.org/#/c/638459/
16:07:08 <smcginnis> I thought that was actually what was needed.
16:07:14 <ttx> yeah, me too
16:07:24 <smcginnis> Maybe we just need to let Andreas know that?
16:07:40 <ttx> but sending Juan Antonio falling down the cracks between our teams is not the best we can do
16:07:56 <ttx> "You have everything set up to release - the release repo check just does not know ;)"
16:08:14 <smcginnis> https://github.com/openstack/tripleo-image-elements/blob/stable/queens/zuul.d/layout.yaml#L8
16:08:16 <evrardjp> sorry it was kinda my fault to ping then
16:08:32 <evrardjp> it's been laying around so I wanted some fresh eyes :)
16:08:33 <smcginnis> I didn't think we could do that in-repo due to secrets.
16:08:34 <dhellmann> yeah, we require the template so we don't have to know all of the other places jobs can be attached to projects
16:08:50 <smcginnis> dhellmann: Can you comment on https://review.openstack.org/#/c/638459/
16:08:51 <ttx> maybe our repo check is a bit too restrictive. Since we did have a light agenda I suggested we look into it and give them an answer :)
16:08:57 <dhellmann> the issue is that they have listed individual jobs in the release queue so ajaeger doesn't want to add the template
16:09:12 <dhellmann> commented
16:09:41 <dhellmann> I also restored the patch
16:09:53 <smcginnis> So we either need to expand our validation checking for whatever combination folks try to add, or we enforce that they are consistent with how this is done.
16:09:56 <smcginnis> I like consistency.
16:10:10 <ttx> expanding our validation is unfortunately not easy
16:10:14 <diablo_rojo> +2 for consistency
16:10:16 <smcginnis> But, they do still need to get their README fixed too.
16:10:17 <fungi> anybody know the backstory on why they aren't using the template today?
16:10:19 <ttx> without reimplementing the full Zuul logic for it
16:10:44 <fungi> like is it just historical baggage or are there jobs in the template they don't want to (or can't) run?
16:10:44 <dhellmann> fwiw, this hasn't been a problem since I made the decision to require using the templates at the very beginning of writing the validation
16:10:50 <smcginnis> Looks like that patch could be updated to also remove the individual jobs if they are adding the template.
16:11:04 <evrardjp> couldn't the freeze job details help here?
16:11:06 <dhellmann> there are some cross-dependencies listed there. maybe those are redundant?
16:11:28 <smcginnis> Those seem... odd.
16:11:38 <ttx> yes, that patch should drop the individual job def
16:12:53 <fungi> i've lost track apparently. where are you looking at cross-dependencies?
16:13:10 <dhellmann> https://review.openstack.org/#/c/638459/1/zuul.d/projects.yaml line 4523 and below
16:13:12 <ttx> "these have to dupe the publish-to-pypi jobs
16:13:14 <ttx> # because of the branch designation for the release job
16:13:16 <ttx> "
16:13:40 <dhellmann> not "cross" but dependencies
16:13:54 <fungi> oh, i didn't think to look at context further down below the diff
16:14:12 <dhellmann> ajaeger's objection to the patch seemed to be that the template was redundant
16:14:12 <ttx> so the duplication was intentional
16:14:48 <smcginnis> Which is valid. But our validation enforces that convention.
16:15:40 <dhellmann> I suppose this wouldn't be an issue if the master branch wasn't closed?
16:15:52 <smcginnis> I think that's the only gotcha in all that.
16:15:58 <ttx> The issue here is that the template does not really match what they have. So we should add another template to match what they have, and add that to the validation test
16:16:25 <fungi> why does it matter that they don't run jobs on master if they're not accepting patches for master anyway?
16:16:30 <ttx> some publish-tripleo template mabe
16:16:33 <dhellmann> ttx: in what way does it not match?
16:16:43 <fungi> dhellmann: the branch exclusions
16:16:55 <smcginnis> fungi: Yeah, there won't be any releases on master, so does it really matter?
16:17:02 <dhellmann> I don't see any branch exclusions in this patch, are those in the template?
16:17:16 <fungi> dhellmann: the lines just after the diff
16:17:41 <dhellmann> those are the check jobs
16:17:42 <dhellmann> I don't see any exclusions in the release jobs
16:17:45 <ttx> fungi: maybe the check/gate could stay
16:18:00 <fungi> yeah, those are the ones i was talking about, not the release jobs
16:18:19 <dhellmann> oh, I see, those are release test jobs
16:18:27 <fungi> just wondering why they even need to exclude master if they're not accepting changes for master
16:18:37 <dhellmann> I suppose they had to exclude the job from master in order to close it and change the readme in the first place
16:18:41 <dhellmann> they could probably remove all of that now
16:18:52 <fungi> right, cleanup
16:18:56 <dhellmann> wfm
16:19:12 <evrardjp> I am confused there
16:19:36 <ttx> ok, so remove all jobs and replace them with the template
16:19:38 <smcginnis> This is getting into zuul configuration, so confusion is normal. :)
16:19:48 <dhellmann> s/normal/required/
16:19:53 <evrardjp> I would hope not to me :p
16:19:57 <dhellmann> here's what I think happened:
16:20:06 <dhellmann> 1. one day, a long time ago, this was a normal project.
16:20:08 <evrardjp> but they don't want to run those jobs for master, which makes sense?
16:20:12 <fungi> evrardjp: the complexity seems to be that they've retired the instack-undercloud repo, but because it wasn't retired until the current cycle they had to leave testing in place to accept fixes on stable branches
16:20:17 <dhellmann> 2. they closed master to new development, but kept the stable branches open
16:20:27 <dhellmann> 3. to do that, they had to disable some jobs on master
16:20:37 <dhellmann> 4. they did that by listing them all out explicitly instead of using the template
16:20:47 <dhellmann> 5. now the release validation is complaining because the template is not there
16:20:52 <evrardjp> because they would need to override per branch in step 4
16:21:00 <fungi> right
16:21:02 <dhellmann> right
16:21:19 <fungi> i agree with dhellmann's probable timeline there
16:21:27 <evrardjp> me too, that's my understanding
16:21:36 <evrardjp> I don't understand how cleaning would help :p
16:21:38 <ttx> ok, should we propose the fix? Or go to write a lengthy explanation and expect someone else to push it?
16:21:39 <smcginnis> So since they are not accepting changes to master, now that they've gotten past that final merge to master there is no need to have special handling.
16:21:41 <dhellmann> the fix is to clean up the explicit job listings and apply the template
16:21:42 <dhellmann> the jobs will run on master and fail, but we don't care
16:21:49 <fungi> we're basically missing a step after retiring the master branch to put the jobs back the way they were
16:21:50 <dhellmann> evrardjp : because it removes ajeager's objection to having redundant information in the config
16:21:57 <evrardjp> smcginnis:  oh ok
16:22:01 <dhellmann> ttx: perhaps we should do it
16:22:05 <ttx> ++
16:22:10 <smcginnis> This is an odd case where a repo was retired but they still care about stable.
16:22:10 <ttx> I can push it if we agree
16:22:15 <dhellmann> ttx: ++
16:22:21 <evrardjp> thanks ttx
16:22:24 <smcginnis> ttx: That would be great.
16:22:29 <ttx> (I'll just remove lines 4506 and below)
16:22:40 <ttx> that will do it right
16:22:42 <dhellmann> ttx: perhaps with a nice big comment in there with the template explaining why it's ok
16:22:45 <dhellmann> yes
16:23:02 <ttx> I'll explain in commit message
16:23:14 <evrardjp> I still believe this is not the ultimate right way, but I am not sure we have the right tools now
16:23:39 <smcginnis> evrardjp: What is you suggestion?
16:23:41 <dhellmann> what would you do instead?
16:23:42 <evrardjp> but hey, we all agree :)
16:24:07 <evrardjp> I have nothing for now, but I would think changing the validation to use the future zuul's freeze api would help
16:24:31 <smcginnis> Future zuul fixes everything. :)
16:24:32 <evrardjp> because we could freeze the job info per branch, so it means for the release we are looking for we could know or not if that  job exists
16:24:44 <dhellmann> perhaps. the point of using the templates was to enforce an *interface* for releasing without enforcing details that tend to change a lot
16:24:58 <evrardjp> fair :)
16:25:05 <dhellmann> i.e., we added test jobs to that template this cycle
16:25:08 <evrardjp> in any way, that's only about the future :D
16:25:20 <ttx> fungi: should I keep a noop in check nd gate pipelines?
16:25:21 <fungi> evrardjp: i might agree with you if we even had consensus on the patch series for those zuul features yet
16:25:23 <evrardjp> sorry to digress here
16:25:32 <fungi> ttx: nope, the noop job can go away
16:25:32 <evrardjp> fungi: :D
16:25:34 <smcginnis> I see they are in the process of merging a README fix, so the other issue should be resolved shortly too.
16:25:41 <dhellmann> agreed
16:26:00 <smcginnis> Any more on these?
16:26:22 <smcginnis> #topic Tasks update
16:26:22 <fungi> if they're adjusting the readme on master, then we shouldn't remove the jobs in 638459 until that's done
16:26:37 <dhellmann> I hope they're only adjusting it in the releasable branches?
16:26:41 <smcginnis> fungi: No, master and stable/rocky are done. They just hadn't fixed queens.
16:26:45 <fungi> okay, in that case it's fine
16:27:15 <smcginnis> diablo_rojo: Did you follow up with QA?
16:27:24 <diablo_rojo> smcginnis, I did
16:27:37 <diablo_rojo> sounded like he had it handled and didn't need anything from us
16:27:38 <ttx> Done @ https://review.openstack.org/#/c/638459/
16:27:55 <smcginnis> diablo_rojo: Excellent, thanks.
16:27:58 <smcginnis> And thanks ttx
16:28:02 <diablo_rojo> smcginnis, no problem :)
16:28:12 <smcginnis> fungi: Any updates on signing key?
16:29:18 <smcginnis> The other task is for generating release requests for any missed c-w-i libs.
16:29:24 <fungi> #link https://sks-keyservers.net/pks/lookup?op=vindex&search=0xcdc08088c3cb45a9be08332b2354069e5b504663&fingerprint=on OpenStack Infra (Train Cycle) <infra-root@openstack.org>
16:29:27 <smcginnis> I can take that unless someone else want it.
16:29:42 <smcginnis> fungi: Great, thanks!
16:29:46 <fungi> i'm waiting for it to appear on the keyservers i'm pulling from so i can attest to it
16:30:11 <fungi> so hopefully later today you'll see a signature from me on it there
16:30:19 <fungi> but it's already signed by the stein cycle key
16:30:25 <ttx> fungi: wondering why we still need the queue defined in https://review.openstack.org/#/c/638459/ but hey I'll add it
16:31:33 <smcginnis> No one clamoring for doing generating the release patches, so I'll do it tomorrow once we're past the deadline and see what's left.
16:31:48 <fungi> ttx: that's so the tripleo jobs all share a named change queue
16:31:54 <ttx> ah ok
16:32:33 <smcginnis> Looking ahead to next week's tasks, same thing for generating the client libs that are missed.
16:32:42 <fungi> basically anything running those (so any changes for instack-undercloud) will end up in the tripleo queue even if they don't share jobs with other projects in that queue
16:33:01 <smcginnis> I've put prometheanfire down in there as a reminder for requirements freeze.
16:33:06 <dhellmann> smcginnis : perhaps we can sign up one of our new team members for that task next week, since they have more notice
16:33:19 <smcginnis> dhellmann: I like that idea. :)
16:33:40 * dhellmann looks meaningfully at evrardjp and diablo_rojo
16:34:09 <diablo_rojo> Ha ha so this is just generating another list?
16:34:15 <smcginnis> It's a matter of listing what deliverables of --type=client-lib have not been released yet and generating release patches for any that have any commits.
16:34:30 <dhellmann> right, not just a list but creating the patches to trigger the missing releases
16:34:31 <evrardjp> diablo_rojo: and updating the docs I would say
16:34:36 <smcginnis> Actually, at this point any regardless of commits as we need a stable/stein branchign point.
16:34:51 <dhellmann> there are scripts to do it all, I think, and if not we have the parts to build the scripts
16:35:00 <diablo_rojo> evrardjp, maybe we can work on this together again?
16:35:08 <evrardjp> I would be fine to sync with diablo_rojo on that
16:35:09 <evrardjp> yeah
16:35:09 <smcginnis> The parts are there, but I never had time to start to automate it.
16:35:10 <diablo_rojo> So we can fumble through it together?
16:35:28 <diablo_rojo> I think we can figure that out.
16:35:29 <evrardjp> fine for me
16:35:46 <ttx> At this point it's ok to approve stein branch creations for libraries, right?
16:35:48 <evrardjp> we'll ping for validation before sending a mass change
16:35:51 <smcginnis> Cool, thanks. I've added your nicks to the task in the etherpad.
16:35:52 <dhellmann> we'll help if you run into trouble, too
16:35:54 <dhellmann> thank you!
16:36:09 <dhellmann> ttx: it is, although I've been reminding folks that they can also wait to rc1
16:36:14 <smcginnis> It will be a good exercise.
16:36:23 <dhellmann> our docs say "allow ... but do not require" branches this week
16:36:39 <smcginnis> If they're pretty sure they are done, no reason not to branch.
16:36:54 <dhellmann> yeah, it's just a trade-off about potentially having to backport changes to fix bugs
16:37:01 <smcginnis> But if there's some concern there might still be an important bug fix, branching just causes extra work at this point.
16:37:06 <ttx> Since deadline is this week and we prefer not to release libs on Fridays...
16:37:06 <dhellmann> right
16:37:40 <smcginnis> Last task on the list is to run the aclissues checker.
16:37:41 <dhellmann> the constraints list protects us from bad releases going into the gate
16:37:42 <dhellmann> at least mostly
16:38:07 <ttx> I can do that, unless someone else wants to try that
16:38:47 <dhellmann> that feels like another case where it would be good to have someone "new" do it to verify the instructions
16:38:50 <smcginnis> Anyone want to walk through with ttx as he does it to learn? Like maybe someone in a similar timezone?
16:39:01 <smcginnis> :)
16:39:01 <ttx> (subtle hint)
16:39:02 <evrardjp> Wow that was targetted
16:39:08 <smcginnis> hehe
16:39:11 <dhellmann> I don't know, where is diablo_rojo this week?
16:39:20 <evrardjp> I am fine with doing a session with ttx
16:39:24 <ttx> I'm jetlagges so I'm currently UTC-3
16:39:26 <diablo_rojo> dhellmann, physically? or mentall?
16:39:29 <evrardjp> I think it's simply running a shell script
16:39:31 <diablo_rojo> *mentally
16:39:32 <evrardjp> iirc
16:39:42 <dhellmann> diablo_rojo : temporally
16:39:56 <smcginnis> evrardjp: Yeah, not a complicated task, but a few things to know.
16:39:57 <diablo_rojo> dhellmann, in between waking up and going to take a nap
16:40:06 <dhellmann> +1 for naps
16:40:33 <diablo_rojo> evrardjp, we could add that to the list too and verify against each other
16:40:40 <smcginnis> OK, I've put down ttx and evrardjp for that last task.
16:40:45 <diablo_rojo> since we will both need to know how to do all these things anyway
16:40:48 <diablo_rojo> Oh
16:40:51 * diablo_rojo shuts up
16:40:58 <smcginnis> diablo_rojo: If you can join too, that's great.
16:41:16 <diablo_rojo> smcginnis, might be a good idea
16:41:18 <evrardjp> the more the merrier
16:41:41 <smcginnis> Just trying to have a good transfer of knowledge where we can.
16:41:49 <dhellmann> ++
16:42:11 <smcginnis> The other tasks in the process doc are reminders that I've put in the countdown email.
16:42:25 <smcginnis> Just need to format them into actual content.
16:42:49 <smcginnis> That should be it for tasks.
16:42:53 <smcginnis> #topic Open
16:43:03 <smcginnis> Anything else we should discuss in-meeting?
16:43:37 <smcginnis> OK, we can get back to work then.
16:43:40 <smcginnis> Thanks everyone!
16:43:41 <dhellmann> there was some discussion of forum/ptg slots, how did that end up?
16:43:47 <smcginnis> Oh, right.
16:44:13 <smcginnis> bnemec: Had a topic submitted for the forum that will be a good time to try to get release change feedback from PTLs.
16:44:22 <fungi> thanks smcginnis!
16:44:26 <smcginnis> Anyone have a link for that?
16:44:39 <bnemec> I haven't actually submitted it yet.
16:45:43 <smcginnis> OK. Let me know if you need anything from us, otherwise we can watch for the session and plan to be there.
16:46:03 <bnemec> Yeah, I was kind of waiting to submit it to see if there was interest.
16:46:08 <bnemec> Since there is, I'll go ahead.
16:46:13 <smcginnis> ++
16:46:53 <smcginnis> OK, I guess that's it. Thanks everyone for participating in the release team.
16:46:59 <smcginnis> #endmeeting