16:00:19 <smcginnis> #startmeeting releaseteam
16:00:20 <ttx> meeting time!
16:00:20 <openstack> Meeting started Thu Mar 14 16:00:19 2019 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:24 <openstack> The meeting name has been set to 'releaseteam'
16:00:29 <smcginnis> Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong
16:00:33 <ttx> o/
16:00:37 <hberaud> o/
16:00:38 <smcginnis> #link https://etherpad.openstack.org/p/stein-relmgt-tracking
16:00:50 <dhellmann> o/
16:00:56 <diablo_rojo_phon> o/
16:01:05 <fungi> aloha
16:01:34 <smcginnis> Looks like ttx has us set with a long agenda today (thanks!) so I think we can get going.
16:01:42 <smcginnis> #topic Standing release automation issues
16:01:55 <ttx> So I approved a bunch of stuff this morning
16:02:00 <smcginnis> This is related to switching to bionic, right>
16:02:05 <ttx> and then was assaulted by release-job-failures
16:02:16 <ttx> smcginnis: partially yes
16:02:24 <ttx> Three issues
16:02:24 <fungi> you were our guinea pig!
16:02:31 <ttx> which I described in several ML posts
16:02:48 <fungi> can't say i didn't warn you to be on the lookout ;)
16:02:49 <ttx> First one is solved by https://review.openstack.org/#/c/643328/
16:03:00 <ttx> which should merge asap
16:03:20 <ttx> The governance one is likely transient, we'll reenqueue when https://review.openstack.org/#/c/643328/ lands
16:03:44 <ttx> The tripleo-ui one is due to a hardcoded host mention
16:03:58 <ttx> fungi: do we ahve a fix in flight for this one?
16:04:22 <fungi> ttx: yes, i replied to the ml with it
16:04:26 <ttx> Hmm https://review.openstack.org/#/c/643353/
16:04:28 <fungi> let me find that
16:04:33 <ttx> got it
16:04:42 <smcginnis> Doesn't look like that one is happy.
16:04:48 <fungi> eww, look at all the lint
16:05:15 <ttx> AttributeError: 'AnsibleUnicode' object has no attribute 'get' eeewwwww
16:05:37 <ttx> thanks for nothing ansible-lint
16:06:32 <smcginnis> Can we assume mordred is on that?
16:06:41 <ttx> maybe it's its way to complain about spaces in the task name
16:06:58 <ttx> if that is the case I recommend therapy
16:07:19 <fungi> yeah, i'm pinging him to help figure it out
16:07:27 <fungi> over in #openstack-infra
16:07:31 <ttx> anyway, that one is less urgent
16:07:39 <smcginnis> So assuming that all merges, are we pretty confident that is all the issues?
16:07:39 <ttx> since it only affects tripleo-ui
16:07:48 <ttx> which TripleO is not even sure to include in release
16:07:56 <ttx> yes
16:08:11 <smcginnis> OK, good.
16:08:19 <smcginnis> And nothing to reenqueue?
16:08:37 <ttx> the governance one should be reenqueued
16:08:46 <fungi> well, it may affect other javascript-based projects trying to release too
16:08:51 <ttx> openstack/governance 0.4.0
16:08:55 <fungi> so getting it ironed out sooner is better
16:09:42 <fungi> i'll reenqueue the governance tag and make sure we don't see any recurrence but i'm pretty sure that was a random provider issue
16:09:45 <smcginnis> We have a list of horizon projects on the agenda. I would assume then we shouldn't release any of those for now.
16:09:59 <ttx> ++
16:10:32 <smcginnis> Alright, sounds like everything is in hand. I'll move to the next topic.
16:10:45 <fungi> when we get a fix for the observed tripleo-ui release failure merged we can try approving one js-based project release and not a large batch initially, just to see what happens
16:10:46 <smcginnis> #topic Stein branch creation hold
16:10:56 <smcginnis> fungi: ++
16:11:11 <smcginnis> #link https://review.openstack.org/#/c/643268/
16:11:14 <ttx> yeah, just a quick point that while the patch has been created, apparently we should wait for tonyb to try something
16:11:39 <ttx> I think we should do it sooner rather than later though
16:11:49 <smcginnis> This is related to having the upper-constraints handling updated to simplify that transition, right?
16:11:50 <ttx> so do we have an ETA on that tonyb-thing?
16:11:58 <ttx> smcginnis: yes
16:12:03 <dhellmann> I don't know about an eta
16:12:05 <smcginnis> Really too bad this is in the middle of his night.
16:12:13 <smcginnis> I think we're going to have to switch times again.
16:12:19 <ttx> something he will have to fix as our new fearless leader
16:12:21 <smcginnis> Or send Tony a heck of a lot of coffee.
16:12:48 <ttx> the release management diet
16:13:35 <smcginnis> So nothing for us to do right now, just awareness that some branching things are on hold pending Tony's update, right?
16:13:38 <ttx> ok, let's make sure we communicate with him later in the day / earlier in his day to see how far it is and if it's worth waiting
16:14:04 <smcginnis> Does this mean we should not process any branch creation patches until then. (assuming we can release again soon)
16:15:18 <ttx> hmm, I'd say no.. but I'm not sure.. dhellmann ?
16:15:42 <dhellmann> this is the change we need: https://review.openstack.org/643387
16:16:27 <smcginnis> dhellmann: So we should merge that before processing any branch requests?
16:16:35 <ttx> ah. Probably wait then
16:16:50 <dhellmann> yes, that's the idea
16:17:01 <smcginnis> OK.
16:17:01 <dhellmann> I just did that very quickly, so check my spelling in that URL, please :-)
16:17:09 <ttx> dhellmann: note that we have a bunch that have already gone in
16:17:09 <smcginnis> I do see it redirects to master for now: https://releases.openstack.org/constraints/upper/stein
16:17:21 <ttx> (I list them on my commit message)
16:17:24 <dhellmann> ttx: it won't be the end of the world
16:17:44 <smcginnis> Do those ones that have branched need to be updated? Or they are fine for now and we eventually get to where we want to down the road?
16:17:50 <dhellmann> oh, we may need to update that template with the redirects after the opendev change
16:18:07 <dhellmann> they're fine for now, the patches just point to something that doesn't exist so they can't merge
16:18:39 <ttx> ok, I think we can safely move to next topic then
16:18:49 <dhellmann> or rather they can, but they will break local development if they do
16:18:51 <dhellmann> so they *shouldn't* merge
16:18:52 <dhellmann> this change was all about removing that window
16:19:33 <smcginnis> And that's no different than how we've been for at least the last few releases, so that's fine.
16:19:47 <dhellmann> oh, I wonder if that branch variable is going to contain "stable/stein" not just "stein"
16:20:38 <smcginnis> We better find out I guess.
16:20:41 <smcginnis> I'll move on for now though.
16:20:44 <smcginnis> #topic deliverables with no intermediary releases yet
16:20:49 <dhellmann> yeah
16:20:54 <ttx> This one is more FYI
16:21:09 <smcginnis> Do we need to add --type other and --type horizon-plugin to our process doc?
16:21:22 <ttx> We have a significant set of things that are cycle-with-intermediary and had no release yet
16:21:40 <ttx> so we have no fall back if they do not produce anything
16:22:11 <ttx> I guess we could have switched them to cycle-with-rc around FF like the service stuff
16:22:16 <smcginnis> Maybe add something that at RC1 they we force them if nothing from the team?
16:22:46 <smcginnis> Those types have always been a little different. I'm never sure if we should treat them as libs or services or what.
16:22:50 <ttx> but in a lot of cases (like the tempest-plugins) it's just that there is little interest in releasing them more than once (be it through RCs or intermediary releases)
16:22:59 <smcginnis> Which I guess is why they are called "other" in the first place. :)
16:23:18 <ttx> I wonder if we should not have one new release type called "automatic"
16:23:22 <smcginnis> Horizon ones kind of seem like they should be considered client libs though.
16:23:26 <dhellmann> the reason for making the tempest plugins -with-intermediary is that nothing can consume a pre-release of a python library in our CI system
16:23:43 <dhellmann> I don't know if anything does consume those from releases, though
16:23:47 <dhellmann> at least in CI
16:23:57 <ttx> dhellmann: making them "cycle-automatic" we would just automatically release them on RC weel for example
16:24:03 <ttx> week*
16:24:03 <dhellmann> sure, that works
16:24:22 <fungi> we certainly _could_ consume prereleases in our jobs if we want (there's a pip option to enable them)
16:24:40 <ttx> note that I would not encourage puting anything other than .. hmm.. "other" into that mode
16:24:43 <fungi> it's more been a philosophical choice to prefer testing against releases
16:24:50 <dhellmann> yeah
16:25:07 <ttx> horizon-plugins should proabbly be made cycle-with-rc
16:25:08 <smcginnis> "automatic" would make it explicit, but would we still want to give them the option to release as needed?
16:25:14 <dhellmann> ttx: maybe cycle-automatic is just cycle-with-intermediary && other then?
16:25:22 <ttx> maybe
16:25:28 <ttx> smcginnis: sure
16:25:36 <ttx> although...
16:25:53 <ttx> I'd say that -automatic things would be re-released at then end
16:25:56 <ttx> the*
16:26:03 <dhellmann> we would have to update list-deliverables to let us query for the absence of a value to build the list of intermediary things that need earlier releases, so a new model might be easier
16:26:07 <smcginnis> If commits merged since last release
16:26:21 <ttx> like if you do a release around milestone-1 we would still push a new release at the end of the cycle
16:26:47 <smcginnis> Or we just add to process.rst that we force release c-w-i --type other like we do for the other intermediary deliverables.
16:26:49 <dhellmann> yeah
16:27:07 <dhellmann> yeah, that works, too
16:27:34 <ttx> anyway, that's more for next cycle
16:27:43 <ttx> Adding a note at the bottom of the tracking doc to remember
16:27:48 <smcginnis> Yay, defer to the next PTL> :)
16:27:56 <ttx> For this cycle... I'd say force them at RC1
16:28:04 <ttx> or the week after
16:28:31 <smcginnis> I can add a note to this week's countdown email.
16:28:35 <ttx> see the task I added to R-2
16:28:58 <smcginnis> Great, was going to say that seems like the right timing to me.
16:29:24 <smcginnis> I will mention it to raise awareness now though.
16:29:33 <smcginnis> And we would create stable branches with that too?
16:29:33 <dhellmann> ++
16:29:55 <ttx> yes
16:30:13 <smcginnis> OK, moving on then.
16:30:28 <smcginnis> #topic mnaser's recent question on the IRC channel
16:30:42 <ttx> See 14:37 utc and down in the backlog
16:31:20 <smcginnis> #link http://eavesdrop.openstack.org/irclogs/%23openstack-release/%23openstack-release.2019-03-14.log.html#t2019-03-14T14:37:10
16:31:24 <ttx> wanting to create a stable/rocky branch on a recently-added deliverable
16:31:27 <mnaser> Hello
16:31:34 <ttx> asking how to proceed
16:31:39 <ttx> mnaser: oh hi sir
16:31:54 <mnaser> Yep. It’s necessary for us to get it to run and integrate with the rest of the repos
16:32:08 <smcginnis> It's independent, so I guess just add the branching request there?
16:32:29 <smcginnis> Odd since it wasn't actually part of that cycle, but these deployment projects are different than the other ones.
16:32:30 <ttx> is it independent?
16:32:34 <mnaser> Yep. All our roles are independent indeed so as far as I understood that should be okay?
16:32:38 <smcginnis> Maybe I read through too fast.
16:32:49 <dhellmann> yeah, just edit the deliverable file
16:32:50 <dhellmann> mnaser : do you need to branch the role? or openstack-ansible?
16:32:52 <mnaser> Yes I believe all our roles are independent. I’m on mobile so I can’t double check
16:33:11 <mnaser> No, the role: specifically OpenStack-ansible-os_mistral
16:33:31 <dhellmann> ok, there aren't any branches at all in the deliverable file right now
16:33:38 <dhellmann> I'm not sure what will happen if you try to just branch one repo
16:33:44 <dhellmann> so we might need to just do this by hand
16:33:48 <mnaser> Yep, we added it this cycle.. so it’s a bit tricky
16:33:50 <ttx> it's not cycle-independent right
16:34:15 <ttx> ok so
16:34:34 <ttx> One issue is that in theory we do NOT add deliverable files after a release is done
16:34:49 <dhellmann> oh, there are *older* deliverable files in the independent directory so I got confused
16:35:12 <smcginnis> So it isn't independent? I think both statements have been madre.
16:35:14 <smcginnis> *made
16:35:19 <dhellmann> smcginnis : it was, but is not now
16:35:23 <smcginnis> Ah
16:35:31 <ttx> It's an independent deliverable, as in separate
16:35:33 <mnaser> Yes. evrardjp worked on making that change
16:35:45 <ttx> but not cycle-independent model?
16:35:47 <mnaser> We realized roles don’t really need to be tied in
16:35:49 <mnaser> Let me verify
16:36:09 <dhellmann> I see deliverables/stein/openstack-ansible-roles.yaml and deliverables/rocky/openstack-ansible-roles.yaml
16:36:48 <dhellmann> and a couple of individual deliverable in _independent for 2 roles (chrony and redhat-subscription)
16:36:50 <dhellmann> and deliverables/_independent/openstack-ansible.yaml
16:37:09 <dhellmann> the change evrardjp made was to stop tagging every role every time there is an OSA release
16:37:19 <dhellmann> only the main openstack-ansible repo is tagged now
16:37:25 <dhellmann> they all still need to be branched, though
16:37:31 <dhellmann> hence the 2 separate deliverable files
16:37:39 <mnaser> ah, I miss understood then. chrony ans redhat one aren’t ansible ones
16:37:50 <dhellmann> oh, those may be tripleo
16:37:57 <mnaser> They are
16:38:03 <evrardjp> hey
16:38:07 <mnaser> ok, so within the “release” infrastructure, what we’re doing isn’t necessarily possible?
16:38:13 <evrardjp> I am in  a meeting, sorry I can't join this conversation
16:38:20 <mnaser> all good :)
16:38:21 <dhellmann> evrardjp : it's fine, I think we have it
16:38:26 <smcginnis> We can just blame it all on evrardjp
16:38:39 <mnaser> I’ll happily take the blame
16:38:46 <mnaser> It’s 8 degrees Celsius outside. I’m in a happy space.
16:38:49 <dhellmann> mnaser : you want to take a new deliverable and create a stable/rocky branch for it, right?
16:38:50 <dhellmann> and then add that to your rocky release for openstack-ansible?
16:38:53 <ttx> dhellmann: deliverables/_independent/openstack-ansible.yaml is not branched
16:39:04 <dhellmann> ttx: right, that's what threw me at first
16:39:10 <evrardjp> it's been a long time we haven't been independent :)
16:39:33 <mnaser> dhellmann: ideally.
16:39:33 <ttx> vi deliverables/rocky/openstack-ansible-roles.yaml
16:39:36 <ttx> oops
16:39:40 <mnaser> hah :)
16:39:45 <dhellmann> I think the thing to do here is to edit deliverables/rocky/openstack-ansible-roles.yaml
16:39:50 <evrardjp> yeah dhellmann is right, the idea was to still tag osa repo, and stop tagging the rest.
16:40:09 <evrardjp> so we had to move to a different release file, and defined branchless IIRC
16:40:17 <evrardjp> sorry, tagless*
16:40:20 <dhellmann> editing that file to add the new repo and a branch point for it should trigger a new branch in just that repo
16:40:23 <ttx> ok, so technically you are not adding a new deliverable file, so I think that should work
16:40:40 <ttx> trying to hole the principles line here
16:40:43 <ttx> hold even
16:41:09 <smcginnis> This seems OK. Again, it's a deployment project, so I think it's fine to add something to rocky.
16:41:09 <dhellmann> I'm willing to be pretty flexible on that for deployment tools
16:41:23 <evrardjp> I agree with adding the role to osa-roles, if necessary branch manually in gerrit?
16:41:26 <dhellmann> mnaser: is it clear what you need to do, now?
16:41:34 * ttx grabs a drink
16:41:35 <dhellmann> evrardjp : we shouldn't need to do anything by hand
16:41:43 <evrardjp> if necessary
16:41:48 <evrardjp> hoping it's not
16:41:50 <smcginnis> ttx: A California wine?
16:42:03 <ttx> smcginnis: served in a LFOSLS glass
16:42:11 <smcginnis> lol
16:42:14 <mnaser> dhellmann: it is and i wanted to get the “everything is alright by release team” sigh off which I seem to have.
16:42:22 <smcginnis> mnaser: ++
16:42:35 <mnaser> I’ll push the change later today. Thank you all.
16:42:39 <ttx> sigh-off is exact
16:42:48 <smcginnis> Thanks for bringing it up mnaser
16:42:49 <dhellmann> yep, +1 from me
16:42:58 <ttx> It's been a long week, and we are only Thursday
16:43:21 <smcginnis> Any other topics to cover?
16:43:22 <mnaser> I need a bigger phone screen. Typing means sigh offs
16:43:30 <mnaser> Thanks again
16:43:44 <smcginnis> #topic Open floor
16:43:46 <ttx> smcginnis: task review for next week?
16:43:55 <smcginnis> ttx: Ah, good call.
16:44:09 <smcginnis> #link https://releases.openstack.org/reference/process.html#rc1-week
16:44:33 <smcginnis> So we are generating the RC1 release patches?
16:44:33 <dhellmann> do we want to paste from the process doc into the etherpad?
16:44:39 <smcginnis> I'll do that.
16:44:43 <dhellmann> k
16:45:17 <dhellmann> some of those devstack-gate instructions are apparently out of date. maybe gmann and fungi can help with those details.
16:45:38 <smcginnis> I was going to raise that too.
16:45:52 <smcginnis> We should get the current requirements for that and get process.rst updated.
16:45:52 <ttx> points 3 and later , we can discuss at next week meeting
16:46:03 <ttx> since they won't be ready by Thursday morning
16:46:26 <ttx> I propose to push them to next week meeting agenda
16:46:40 <fungi> i think the only think devstack-gate needs these days is an entry in the version matrix gile
16:46:45 <fungi> s/gile/file/
16:47:41 <smcginnis> We can move those later, but leaving it there for now to make sure it's seen.
16:47:55 <ttx> Hmm, point 11 should be moved up
16:48:02 <ttx> I'll do that
16:48:13 <gmann> and grenade branch update also
16:48:32 <gmann> when we cut stable branch
16:48:34 <fungi> does grenade still rely on devstack-gate for that?
16:48:41 <gmann> yeah
16:48:48 <smcginnis> ttx: I do have the cycle-highlight mention in this weeks countdown email, but that could probably use another post of its own.
16:48:52 <ttx> I'll propose the corresponding process change
16:48:59 <gmann> zuulv3 job migration is in progress but not merged yet
16:49:01 <ttx> yeah, around Tuesday-ish
16:49:15 <ttx> I moved it up as point 2
16:49:24 <smcginnis> I'm wondering about our decision to propose the RC1 patches coming from us early in the week.
16:49:54 <smcginnis> There's usually a crunch for folks getting in those final patches, so us proposing one at the beginning of the week is just going to create extra zuul load that will probably need to get updated with a new hash later anyway.
16:49:57 <openstackgerrit> Duc Truong proposed openstack/releases master: Add Senlin release highlights  https://review.openstack.org/643396
16:50:23 <dhellmann> should we wait and do that later in the week?
16:50:48 <smcginnis> Maybe later on Wednesday?
16:51:03 <dhellmann> that doesn't give a lot of time for +1 from PTLs
16:51:17 <smcginnis> I would hope by that point they wouldn't need much time.
16:51:25 <ttx> done
16:51:27 <openstackgerrit> Thierry Carrez proposed openstack/releases master: Reorder steps on RC1 week process  https://review.openstack.org/643398
16:51:35 <fungi> gmann: so http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/roles/test-matrix/files/features.yaml and http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n302 need updating, but that's all for devstack-gate nowadays?
16:51:54 <ttx> smcginnis: it's a balance
16:52:11 <ttx> if the RC1s seem to be coming, then waiting is best
16:52:23 <gmann> fungi: i think so. other part of d-g where legacy jobs rely are static one
16:52:23 <smcginnis> I suppose we do have more projects now that could be ready already on Monday.
16:52:55 <ttx> also the announcements tricking down to teh ML are a great reminder
16:53:03 <ttx> trickling damn
16:53:03 <smcginnis> I guess that's what we decided and wrote down, so we could just stick with it and see how it goes, then readjust for train if we find it's not optimal.
16:53:31 <ttx> smcginnis: editing the existing review means less work for ptl too
16:53:43 <smcginnis> That's true. That could be nice.
16:54:08 <ttx> hmm.. did we update the ML address for RC announcements ?
16:54:24 <dhellmann> where?
16:54:33 <ttx> smcginnis: like... one less round for 0.0.0rc1 mistakes
16:54:35 <smcginnis> Oh, those go to discuss and not announce, right?
16:54:36 <dhellmann> oh, those go to the -dev/-discuss list
16:54:56 <ttx> yes²
16:54:56 <smcginnis> Is that in the job?
16:55:17 * ttx is looking
16:55:38 <smcginnis> Seems like one of the scripters using codesearch would have gotten that.
16:57:30 <dhellmann> 94a24d03dc818b4a23d0140863745ef9af590a88
16:57:44 <dhellmann> i think that will do it?
16:58:09 <ttx> yes
16:58:21 <smcginnis> Yep. Great.
16:58:34 <ttx> i was almost there down in that rabbit hole.
16:58:48 <ttx> dhellmann: you're fast. It's almost as if you wrote all those things.
16:58:50 <dhellmann> although I don't see anything at all sending to the release-announce list
16:58:54 <dhellmann> *cough* someone should have updated the comment block at the same time
16:59:09 * dhellmann is looking forward to having space in his head for other things
16:59:33 <smcginnis> OK, so let's go with generating RC requests early next week as we had previously decided. Anything else we should cover?
16:59:39 <ttx> sorry for the drill. But then it's always useful to check we still know where to look
16:59:46 <ttx> smcginnis: nope
16:59:47 <smcginnis> ++
17:00:00 <smcginnis> Nice full hour. Thanks everyone.
17:00:16 <fungi> thanks smcginnis!
17:00:18 <smcginnis> #endmeeting