16:00:01 #startmeeting releaseteam 16:00:02 Meeting started Thu Apr 4 16:00:01 2019 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:06 The meeting name has been set to 'releaseteam' 16:00:08 ttx: You're jumping the gun. :) 16:00:14 * ttx might have to urgently pause in middle to drive kid somewhere 16:00:15 #link https://etherpad.openstack.org/p/stein-relmgt-tracking 16:00:24 that's why I'm starting early 16:00:24 They can walk. 16:00:30 :) 16:00:53 We're on line 518. 16:01:09 Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong 16:01:15 o/ 16:01:41 o/ 16:01:44 #topic Stein release 16:02:03 We had one unreleased that I took care of this morning, so we can skip that. 16:02:12 Missing stable branches: There's a few. 16:02:16 o/ 16:02:24 Looks like the proposed patches did not include any branching. 16:02:46 I figure we can branch those early tomorrow. Or should we do it today? 16:03:48 mmm 16:04:00 Or no one cares. :D 16:04:04 you mean for L523? 16:04:25 So it's normal to have missing stable 16:04:26 do we have anything other than tempest plugins that don't have branches? 16:04:33 Yeah, basically. 16:04:35 we are supposed to cut them this week 16:04:43 dhellmann: Yeah, there's a list of them in the etherpad in the meeting agenda. 16:04:45 A couple are very old releases though 16:04:56 so we might want to doublecheck recent changes 16:04:57 ec2-api is very low activity. 16:04:58 my question is what do you really want to branch :) 16:05:05 Not sure about manila-ui. 16:05:11 oh, sorry, I was looking too high up in that section of the pad 16:05:13 and shouldn't that be at least done by the PTLs? 16:05:26 It's all fine since they all have done a release during the cycle 16:05:37 evrardjp: Well, a lot should be done by the PTLs, but they don't. We need to make sure a stable branch is created. 16:05:45 and we said we would cut stable/stein from most recent release 16:06:08 Yep, we did announce we would do them if they were not proposed by now. 16:06:09 Ivan Kolodyazhny proposed openstack/releases master: Release Horizon Stein RC2 https://review.openstack.org/650055 16:06:14 smcginnis: agreed on general releases, tagging. But here it's to create a branch -- I am not so sure. Sorry to get cold feet 16:06:17 manila-ui has only test changes 16:06:28 so 2.17.0 probably ok 16:06:48 and 8.0.0 is current for ec2-api 16:06:49 evrardjp : part of the move we've made over the last couple of cycles is to do more of this stuff for the teams, instead of trying to chase down everyone to ensure they do it themselves 16:06:52 so all good 16:06:58 evrardjp: It's a good point that a stable branch can be created from the tag as needed. But good to have them created before they are needed to make sure. 16:07:17 dhellmann: agreed I like this, but I thought branching was a big thing due to the backport burden 16:07:29 no branching is cheap 16:07:31 but line 534 clarifies it's not a big thing 16:07:42 I can propose the branches today and push them through. Or we can approve them early tomorrow. Any preferences? 16:07:42 ttx: true for us :) 16:08:01 smcginnis: approve early tomorrow 16:08:04 smcginnis: to clarify, we did release stein with https://review.openstack.org/#/c/631776/ in manila-ui, no new changes were made, except gate stuff like ttx mentioned 16:08:18 gouthamr: Thanks! 16:08:19 smcginnis: no preference 16:08:21 gouthamr : if we branch without those changes, your new stable branch will be broken 16:08:21 gouthamr: +1 16:08:27 s/will/may 16:08:41 do you need to tag a more current point release so we can branch from that? 16:09:14 dhellmann: gouthamr: we can certainly do that 16:09:19 dhellmann: sure, yes.. tbarron is here, so we will do momentarily 16:09:31 sounds good, thanks 16:09:44 Great, thanks. That should help with the concern evrardjp raised of then making teams need to backport things. 16:10:30 OK, translations... 16:10:47 Horizon was the only one called out in the etherpad, but I see Ivan just proposed an RC2. 16:10:51 So we should be good there. 16:11:00 ++ 16:11:06 Late RCs 16:11:24 I saw a new nova RC proposed. 16:11:30 * smcginnis checks if it includes that bug. 16:11:38 there is a thread about that bug on the list 16:11:45 so I think we need to keep an eye on that one 16:11:48 Ah, missed it. 16:12:09 it's the only panic-potential thing I know of 16:12:24 Does look like something that's pretty critical. 16:12:34 But then I have been traveling a bit last week, did you all hear of anything else? 16:12:43 is that the oslo.service thing? 16:12:48 smcginnis: it's critical if it is a regression 16:12:48 Though Matt's last comment on the bug says it was there in rocky. 16:13:21 right, so now it's almost a feature 16:13:23 So not a regression. 16:13:30 * diablo_rojo sneaks in late becuase she didn't notice her IRCclient was disconnected 16:13:37 Undocumented feature. ;) 16:14:19 More generally, there is general flakiness in tests that point to a potential loss of quality 16:14:20 So I think we can continue with plans to do final release proposal. If that comes through in time, great. If not, looks like it will need to be a stable backport at least for rocky. 16:14:31 so we need to keep our ears on the ground 16:14:59 ++ 16:15:10 we could propose the nova final release on top of that rc2 and let eric block it until they're ready 16:15:20 * dhellmann doesn't see efried in channel 16:15:35 maybe melwitt is managing that instead? 16:15:48 It is still stein. 16:15:54 yeah 16:16:20 We can follow up and watch for that. 16:16:38 Let's move on to release week organization. 16:17:07 So after the final RC deadline today, we can propose the final release patch. 16:17:21 I can do that in the morning. We don't need to wait on the branching for that. 16:17:28 ++ 16:17:51 For the newer folks, we add all PTLs to that and request that they ack the patch. 16:18:07 Not strictly needed. We won't hold it if we don't get an ack. But gives us the warm fuzzies. 16:18:18 And ends up in the metadata that the PTLs signed off on it. 16:18:52 I don't think we did: "The week before final release test the release process using the openstack/release-test repository." 16:19:08 I saw one of those earlier today? or late yesterday? 16:19:12 and we've done releases this week 16:19:13 We did do a release-test release yesterday. 16:19:32 I used dhellmann's test patch to make sure things were working after the sphinx 2 issues. 16:19:50 s/yesterday/earlier this week/ 16:19:56 I've lost track of all time. 16:19:58 that's in the process because we don't expect to be doing releases this week, and once we had an unexpected failure in the process 16:20:05 cool 16:20:12 * ttx strikes out 16:20:14 We've had a LOT of releases this week this time around. 16:20:16 do we have a step to ask infra to hold off changing zuul config that might effect releases? 16:20:29 I don't think we call that out. 16:20:30 "On the morning of the deadline for final release candidates, check the list of unreleased changes for milestone projects and verify with the PTLs and liaisons that they are planning a release or that they do not need one." ?? 16:20:42 I haven't checked that ^ 16:21:38 I can run the script while we continue the meeting 16:21:43 tools/list_rc_updates.sh 16:21:44 dhellmann: thx! 16:21:44 OK, thanks dhellmann 16:22:14 Next week we should run the missing-releases script to look for any missing tarballs, pypi packages, etc. 16:22:27 Anyone want to volunteer for that? 16:22:30 ack, can do 16:22:36 yeah, exercising the automation lightly before pushing the big red button remains a good idea. the infra team is trying to limit changes which could impact release processes, but you never know what could slip through 16:22:44 Another easy "run script, report issues" task evrardjp and diablo_rojo_phon ;) 16:22:54 Thanks fungi 16:23:31 will have a look smcginnis 16:23:43 I suppose the ttx "ack" was for that, so I can do that with him 16:23:52 evrardjp: sure 16:23:53 I can look too 16:24:23 Thanks! 16:24:53 Next task - after release, we need to update series_status so releases.o.o shows it as released. 16:25:05 Pretty simple update with an example of previous times. 16:25:11 here's the unreleased rc report: http://paste.openstack.org/show/748878/ 16:25:12 I can look through that report for anything important 16:25:59 Good, mostly normal branching updates for most of them. 16:26:40 Any volunteers to update the series_status. It's an easy one. 16:26:51 I can take it if no volunteers. 16:26:58 But trying to "delegate". 16:27:31 OK, I'll put my name on that one. Just let me know if someone else wants to grab it though. 16:27:32 Confirming PR is officially going out at 10am CT on the 10th, so we should target finishing the release process around that 16:27:45 take it,and I will check if I can help you tomorrow 16:27:49 smcginnis: ^ 16:27:49 smcginnis, I would say go ahead and do it, but I will look at it too. 16:27:56 (it's also ok if we take more tiem, it's just a nice target to have) 16:27:59 diablo_rojo: haha :) 16:28:01 evrardjp: OK, thanks. 16:28:07 ttx: Thanks, was wondering about tat. 16:28:11 that 16:28:52 I'll get the announcement ready to send out Wednesday. 16:29:03 Do we usually wait for the PR to go out first, or doesn't it matter. 16:29:10 I can't remember now. Just around the same time? 16:29:26 We try to finish before 16:29:30 Doug Hellmann proposed openstack/releases master: heat 12.0.0.0rc2 https://review.openstack.org/650066 16:29:47 like 1-30min before the PR release 16:30:04 OK, then we should approve that final patch early and I will have the announcement queued up to go. 16:30:09 (the PR usually leaks as press articles want to be the first out) 16:30:36 final patch can be approved a long time before 16:30:46 in don't have anything going on that morning, so will make sure i'm on hand to keep tabs on the automation/ci bits 16:31:14 fungi: Thanks, that will be great to make sure nothing gets tripped up. 16:32:01 ttx: Do you want to hit the approval on the final release patch earlier in the day so it's done in time? 16:32:14 smcginnis: that is generally what we do 16:32:22 We can all +2 it by Tuesday, so it should be good to go. 16:32:41 re: unreleased changes, looks like we should ping a few teams for late RCs before doing teh final-release patch 16:32:44 Doug Hellmann proposed openstack/releases master: magnum 8.0.0.0rc2 https://review.openstack.org/650067 16:33:05 Hah, or just propose them I suppose. Thanks dhellmann 16:33:20 yeah, there are only a few, I'll just do them now 16:33:31 ++ 16:33:45 thanks dhellmann 16:33:45 Doug Hellmann proposed openstack/releases master: mistral 8.0.0.0rc2 https://review.openstack.org/650069 16:33:57 OK I think that is all 16:33:57 Let's get those through as soon as they pass today. Add branches and propose final release tomorrow morning. 16:34:18 Collect PTL +1's, add out own +2's, then ttx approves early on the 10th. 16:34:19 Doug Hellmann proposed openstack/releases master: murano 7.0.0.0rc2 https://review.openstack.org/650070 16:34:24 Love it when a plan comes together. 16:34:39 :) 16:34:58 Doug Hellmann proposed openstack/releases master: octavia 4.0.0.0rc2 https://review.openstack.org/650071 16:35:02 I think (knock on wood) we are in good shape. 16:35:12 ok, that's all of them 16:35:28 #topic Open floor 16:35:29 dhellmann I already have a release patch up 16:35:45 johnsom : ah, thanks, which project? 16:35:51 so... what time on wednesday should i be keeping an eye on things? 14:00z? earlier? 16:35:53 dhellmann https://review.openstack.org/649765 16:36:09 ah, your wip dropped it off of my dashboard 16:36:20 fungi: It sounds like earlier since we want it complete by 15:00 16:36:31 Yeah, just waiting for a patch or two to get through zuul 16:36:48 smcginnis: thanks, i mathed wrong ;) 16:37:06 ttx: You want to coordinate with fungi on timing there to have the fire brigade ready? 16:37:10 All set. Just need to find a stein to drink on release day 16:37:37 i have a wooden stein on hand 16:37:41 not lidded though 16:37:53 fungi: how much time do you expect the patch to run? 16:38:11 I have two of them not sure I can post one of the two in time. ttx :( 16:38:22 i was thinking on hitting W+1 around 9:00utc 16:38:25 judging from recent past releases, seems like it was something like 30 minutes from push to finish 16:38:42 * smcginnis is really disappointed no vendors used the opportunity to give out stein swag 16:38:44 ah, then yes, 13z? 16:38:54 smcginnis: :) 16:39:00 ttx: isn't that short on time? 16:39:04 i can certainly be around at 09:00z but might be a bit groggy 16:39:21 fungi: how about 12:00z 16:39:21 Hopefully you can sleep through things. 16:39:34 we'll probably have ianw and frickler more wide-eyed and bushy-tailed even if i'm not 16:39:38 that leaves us 3 hours to fix things on a 30-min run script 16:39:42 but sure, 12:00z is easy for me 16:40:07 hi, I just wanted to mention that the bug we're holding rc2 for is new in stein and is a security problem (leaks cell database and message queue credentials to the notifications message queue) 16:40:12 this is for wednesday, right? 16:40:16 also, not the end of the world if things are not ready by 15z 16:40:31 melwitt: i saw, thanks for prioritizing that one! 16:40:38 melwitt: Ah, I remember Matt mentioning that. How's that coming along? 16:41:01 ttx: I suppose it's not great to have PR on something not yet released? Or did I misunderstand the times? 16:41:09 smcginnis: master change is approved, we just had to recheck it a few min ago. so just have to babysit through the gate and then approve the stein backport after 16:41:37 OK, good. Then barring gate shenanigans, we should be good on timing there. 16:41:47 evrardjp: PR release time is 15z. If we push the patch around 12z that will be done by then 16:42:05 fungi: Maybe we should bump up that nova patch? 16:42:11 yeah but it was for your comment "not the end of the world if things are not ready by 15z" 16:42:16 Also PR is not really resulting in thousands of downloads. 16:42:25 ttx: fiar 16:42:27 fair* 16:42:29 i've blocked out 1200-1500z in my schedule on the 10th now 16:43:25 What we need to have up by 15z is the links that the PR points to (the landing page for stein release) 16:43:28 I'll be around early, but have to leave for a while at 1400 16:44:06 that in turn points to releases.o.o/stein which may still have RCs and nobody would notice :) 16:44:22 evrardjp: it's a double-edged sword, because it we release too far ahead the outlets who are syndicating the press release feel undercut because details will have already had time to circulate and steal their thunder 16:44:28 ttx: I see 16:44:38 fungi: indeed 16:45:00 I will plan on running the meeting next Thursday so we can debrief. But then tonyb will likely take over from there. Let's watch for updates from him on meeting details. 16:45:09 also, we usually get press articles BEFORE the release is out anyway 16:45:29 smcginnis: melwitt: is there urgency in getting the master security fix in before proposing/approving the backport for stein? 16:46:09 I hope the backport patch is already up? 16:46:12 Normally we want the later one to merge before cherry-picking, but probably safe to go ahead in this case. 16:46:14 we have the stein backport proposed 16:46:19 OK, good. 16:46:25 fungi: I guess if +w and merged while master is not merged, then it could be a pain . 16:46:28 it's just we don't +W until the master change merges 16:46:50 yeah, exactly evrardjp 16:46:53 fair enough. how long do we expect that to set back the last rc? 16:47:00 if the master change fails in the gate or something 16:47:01 then it's fine if it's just waiting for +w 16:47:22 I guess babysitting will matter :) 16:47:30 it always is! 16:47:35 yes, we are babysitting it 16:47:48 just had to recheck it for an ssh timeout a little while ago 16:48:14 Just wondering if it needs an infra magic kick to the top to make sure it makes it through in time to not impact final release activities. 16:48:16 hoping the gates gods are stable 16:48:17 Same here, we have two in the gates, but keep getting nova timeouts. 5 minutes to schedule?? ugh 16:48:38 ?? that doesn't sound normal 16:48:49 Agreed. That was limestone 16:48:59 oh, yeah ok 16:49:04 Vexxhost just had a cirros stuck in build 16:50:01 that's probably a discussion for infra :) 16:50:06 yeah, looks like there's a bit of a run on the gate for a variety of failures 16:50:35 per ttx's earlier comment about various sorts of failures on the rise suggesting openstack code quality slipping 16:51:19 I think what we are seeing is mostly high load on limited resources. These have not been happening over the course of the release. 16:52:00 I guess we'll watch and see what happens. 16:52:06 Anything else for the meeting today? 16:52:10 nope 16:52:11 none 16:52:13 yep, but remember these tests are running on openstack... so problems seen in the infrastructure we're using are still quite oftenn openstack problems 16:52:44 fungi++ 16:52:49 Thanks everyone! 16:52:51 #endmeeting