16:01:00 <smcginnis> #startmeeting releaseteam
16:01:01 <openstack> Meeting started Thu Jun 18 16:01:00 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:04 <openstack> The meeting name has been set to 'releaseteam'
16:01:07 <smcginnis> #link https://etherpad.opendev.org/p/victoria-relmgt-tracking Agenda
16:01:11 <hberaud> o/
16:01:26 <dtantsur> o/
16:01:42 <smcginnis> Great, everyone is here. We can jump right to it.
16:01:45 <smcginnis> #topic Ironic release model discussion
16:02:00 <smcginnis> I saw the ping earlier, but haven't had a chance to look at anything.
16:02:14 <smcginnis> dtantsur: Did you have any questions for us or any updates we should know about?
16:02:16 <dtantsur> It's a follow-up to the ironic spec we've discussed
16:02:31 <dtantsur> my main question is whether we actually need to switch to the independent model
16:02:32 <diablo_rojo> o/
16:02:42 <ttx> dtantsur: unless your requirements evolved since last time we discussed it, you should not need to switch to independent, you just create additional branches in the cycle-with-intermediary model
16:02:44 <dtantsur> it seems that what we want is a superset of what we do now
16:03:03 <dtantsur> right, I mainly wanted to confirm this
16:03:07 <ttx> so it would be fully supported under that model
16:03:26 <ttx> I prefer to work on that rather than start including "independent" things in the cycle release
16:03:27 <dtantsur> like, in a week or two we may propose releases that will have branches like bugfix/X.Y with them
16:03:27 <smcginnis> dtantsur: I started making some release tooling changes to support the published ironic spec.
16:03:39 <dtantsur> sweet! which exactly?
16:03:46 <ttx> (Which would be super confusing)
16:03:57 <dtantsur> I guess reno may start freaking out on new branches?
16:04:31 <smcginnis> The tricky part yet will be updating the tooling to allow stable releases from those branches as we have a rule right now that all new releases need to be at the end of the tree. Or rather pole, as we don't support the multiple branches in a cycle today.
16:05:09 <smcginnis> #link https://review.opendev.org/#/c/732984/
16:05:10 <dtantsur> okay, so point releases from new branches may still be a problem?
16:05:13 <smcginnis> This was the start ^^
16:05:38 <smcginnis> At this point it is. Need some time to rework our tooling.
16:06:22 <ttx> dtantsur: worst case scenario we can push the tag directly, which should just work
16:06:37 <smcginnis> Good point.
16:06:40 <dtantsur> yeah, we want to avoid more work for everyone
16:06:43 <smcginnis> So we're not blocked for now.
16:06:59 <dtantsur> but honestly, I hope that not so many such releases will actually happen
16:07:00 <ttx> dtantsur: it's work to make our release models more flexible, so it's good work
16:07:05 <dtantsur> ++
16:07:37 <dtantsur> so, reno, will we need to update it to pick these new branches?
16:07:49 <smcginnis> If these are infrequent, might be good to stick with manual tagging for now.
16:08:10 <dtantsur> I *hope* they're infrequent since we'll only produced them for high and critical issues
16:08:18 <dtantsur> * produce
16:08:44 <smcginnis> Just don't have any bugs then. :)
16:08:54 <dtantsur> \o/
16:08:58 <dtantsur> problem solved, thanks everyone
16:09:01 <dtantsur> :D
16:09:10 <dtantsur> looking on https://review.opendev.org/#/c/732984 it assumes naming stable/X.Y?
16:09:15 <smcginnis> I feel better now. That gives us some time to work things out without holding anyone up.
16:09:33 <dtantsur> in our last discussion ttx (?) proposed using another name to avoid conflating intentions
16:09:42 <dtantsur> like bugfix/X.Y
16:09:51 <smcginnis> dtantsur: Yes. I think that's what the spec said, and we have more checks in place to make sure it's either stable/ or feature/, so that was the easier path.
16:10:04 <dtantsur> I don't have strong opinions on it, I can stick with stable
16:10:18 <ttx> I prefer bugfix/* personally
16:10:42 <smcginnis> OK, that is not allowed as things are today.
16:10:48 <smcginnis> So that will take a little more work.
16:10:49 <ttx> since stable brranch rules won;t apply to bugfix/x.y
16:11:03 <ttx> again, can be bypassed manually
16:11:43 <smcginnis> Anything more we need to discuss today?
16:11:43 <ttx> we can handle all that using manual branch creation and tag pushes until the validation catches up
16:11:50 <smcginnis> ++
16:11:53 <dtantsur> IIRC our first releases will happen on the week of Jun 29th
16:12:07 <dtantsur> we can meet again and see how it goes
16:12:10 <ttx> I'll be around :)
16:12:37 <dtantsur> the only unclear thing is reno. I guess we'll create the branch and see how we can fix it.
16:12:40 <smcginnis> Sounds like a plan.
16:12:43 <dtantsur> that's all from me, thanks folks!
16:12:52 <smcginnis> Thanks!
16:12:58 <dtantsur> I'll update our spec to mention bugfix/*
16:13:11 <ttx> sounds good
16:13:22 <smcginnis> OK, just let us know when it's needed. Happy to help.
16:13:31 <smcginnis> #topic Review task completion
16:13:49 <smcginnis> First one - auto releases for c-w-i.
16:13:54 <ttx> darn, I did not do the thing with hberaud
16:14:02 <smcginnis> I got those processed, so I think that one is all set.
16:14:12 <ttx> hberaud: would you be available tomorrow EU morning?
16:14:12 <hberaud> ttx: don't worry I just remember that too
16:14:14 <smcginnis> Yeah, next one is the acl check.
16:14:25 <hberaud> ttx: sure
16:14:28 <smcginnis> The week isn't over yet.
16:14:33 <ttx> ok will ping you around 10:00
16:14:41 <hberaud> excellent
16:15:02 <smcginnis> And I will send the countdown email tomorrow.
16:15:19 <smcginnis> Anything else on this weeks tasks?
16:15:57 <hberaud> nope
16:15:58 <ttx> I was holding on approvals.... should we process them now?
16:16:08 <fungi> looks like all's clear
16:16:24 <fungi> i'm working through what seems to be a python version mismatch in branch tip tarball jobs
16:16:26 <ttx> OK, I'll approve the stuffs
16:16:33 <smcginnis> I think we can process the ones that have received an ack now.
16:16:33 <fungi> but it doesn't affect the tagged release jobs
16:16:50 <smcginnis> We can wait a little longer on the unacknowledged patches.
16:17:14 <smcginnis> I noticed I didn't put the exact deadline in the commit this time, but we can do those tomorrow since today is the published deadline.
16:17:21 <openstackgerrit> Vishal Manchanda proposed openstack/releases master: Release horizon for victoria-1 milestone  https://review.opendev.org/736724
16:17:34 <smcginnis> And I will abandon those unack'd stable patches I had put up.
16:17:46 <smcginnis> But the ack'd ones of those can get processed too.
16:19:31 <smcginnis> #topic Release issues recap
16:19:49 <smcginnis> I thought we should at least bring this up in the meeting so it is captured.
16:20:06 <smcginnis> Summary is, pip and virtualenv were removed from the base images used in nodepool.
16:20:16 <smcginnis> This had some fallout for some of the release jobs that relied on these.
16:20:45 <smcginnis> So we needed to add the inclusion of some roles (and update or create new roles) to make sure everything our scripts expect to be present is there.
16:20:58 <smcginnis> We had a few jobs fail before we got it straightened out.
16:21:21 <smcginnis> Most were able to be reenqueued for the release failures. A few update-constraints had to be manually processed.
16:21:32 <smcginnis> Only thing that couldn't be rerun was the release-announcements.
16:21:49 <smcginnis> We could manually do those, but I don't think that's necessary. Unless someone else feels differently.
16:22:21 <smcginnis> Big thanks to fungi (and AJaeger, clarkb, ianw, mordred, and whoever else I'm missing) for helping get things back on track.
16:22:48 <smcginnis> fungi reenqueued the failing jobs earlier and they all seem to be working right now.
16:22:57 <smcginnis> So we should be in the clear to go ahead with processing new releases.
16:23:24 <smcginnis> #topic Validate countdown email content
16:23:30 <ttx> well good news, since I just approved 30
16:23:31 <smcginnis> #link https://etherpad.opendev.org/p/relmgmt-weekly-emails
16:23:34 <smcginnis> ttx: ;)
16:24:43 <smcginnis> I will send that out tomorrow.
16:24:52 <smcginnis> Looks like we have all the placeholders updated now.
16:24:57 <ttx> lgtm
16:25:02 <smcginnis> I will read through one more time before I send it out.
16:25:08 <smcginnis> #topic AOB
16:25:14 <smcginnis> Anything else for today?
16:25:33 <fungi> note it wasn't *just* pip and virtualenv we removed from images as preinstalled... also wheel, tox
16:25:46 <fungi> setuptools
16:26:01 <fungi> various jobs assumed them in different places
16:26:26 <smcginnis> Yes, good clarification.
16:26:49 <fungi> also for some of the failures i did have to manually copy sdist tarballs and wheels from pypi into afs (for tarballs.o.o) and manually used the victoria cycle openpgp key to create signatures for them
16:27:30 <fungi> but the upload job has been improved now to emit checksums and the text of the signatures in case we need them in the future, so we can more easily confirm the copies from pypi are unaltered and don't have to recreate sigs
16:27:45 <smcginnis> Oh nice.
16:28:04 <fungi> so if you watch the upload jobs, you'll see checksums and signatures embedded in the output now
16:28:19 <smcginnis> Should we look at reordering things? I think that was mentioned earlier, but not sure of the details.
16:28:29 <fungi> though there's some doubling of the stdout going on there, i'm not entirely sure what's causing it, doesn't hurt anythnig
16:28:50 <smcginnis> Sounds good.
16:28:51 <smcginnis> Sounds good.
16:28:53 <smcginnis> :PP
16:28:58 <fungi> yeah, i've got another change in progress to simplify how we're calling twine and then we should probably talk about tarballs site vs pypi ordering of uploads
16:29:24 <fungi> #link https://review.opendev.org/735932 Simplify twine invocation for PyPI uploads
16:29:46 <smcginnis> The advantage of doing tarballs first would be that we would then have all the artifacts in case we need to manually do the PyPi upload?
16:29:51 <fungi> right
16:30:06 <ttx> +1
16:30:17 <fungi> the downside is that if we reenqueue them we'll overwrite what got uploaded to the tarballs site
16:30:31 <fungi> before we  hit the duplicate file check on pypi
16:30:53 <fungi> but i don't think the risk from that is as high as the risk we have to redo a lot of work
16:31:16 <smcginnis> Yeah, seems pretty minor then.
16:31:49 <AJaeger> fungi: that twine change needs rework...
16:31:53 <fungi> i think the reason it was switched to pypi first is that we'd avoid uploading to tarballs.o.o if, say, the metadata included invalid trove classifiers or something which would cause pypi to reject the upload
16:32:03 <openstackgerrit> Merged openstack/releases master: Release python-masakariclient for victoria-1 milestone  https://review.opendev.org/735711
16:32:07 <openstackgerrit> Merged openstack/releases master: Release kuryr for victoria-1 milestone  https://review.opendev.org/735682
16:32:08 <fungi> AJaeger: yep, i need to get back around to it now that the other dust has settled
16:32:10 <openstackgerrit> Merged openstack/releases master: Release python-manilaclient for victoria-1 milestone  https://review.opendev.org/735710
16:32:18 <smcginnis> fungi: Not too hard to manually delete from tarballs though, right?
16:32:24 <fungi> right
16:32:29 <fungi> or overwrite
16:32:30 <smcginnis> OK, anything else?
16:32:36 <fungi> nothnig from me
16:32:41 <smcginnis> We should probably end the meeting before the merge deluge that ttx started. :)
16:32:43 <ttx> nope
16:32:52 <ttx> I want the deluge logged
16:32:52 <openstackgerrit> Merged openstack/releases master: Release os-win for victoria-1 milestone  https://review.opendev.org/735691
16:32:56 <smcginnis> OK, thank you everyone.
16:33:06 <smcginnis> ttx: You only get the IRC logs! :P
16:33:07 <diablo_rojo> Nothing from me
16:33:09 <smcginnis> #endmeeting