13:02:48 <ttx> #startmeeting releaseteam
13:02:49 <openstack> Meeting started Mon Oct 12 13:02:48 2015 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:02:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:02:53 <openstack> The meeting name has been set to 'releaseteam'
13:03:05 <ttx> Last of our Liberty meetings, i'll give you the chair baton for the next ones
13:03:18 <dhellmann> sounds good
13:03:18 <ttx> (probably when back from Tokyo)
13:03:27 <ttx> I don't have much
13:03:34 <ttx> https://wiki.openstack.org/wiki/Release_Cycle_Management#Meeting
13:03:51 <ttx> #topic Liberty release coordination
13:03:59 <ttx> https://etherpad.openstack.org/p/liberty-final-release
13:04:04 <ttx> We are on the last line
13:04:07 <ttx> 3 things
13:04:24 <ttx> 1/ Keep on track of things that may trigger a late respin
13:04:45 <ttx> I'll be in touch today with PTLs that have stable/liberty changes up, just in case
13:05:05 <ttx> but nothing has been mentioned yet
13:05:15 <dhellmann> ok, good
13:05:22 <ttx> There is https://review.openstack.org/#/c/232917/
13:05:34 <ttx> which if accepted would trigger requirements syncs all over
13:05:53 <dhellmann> do we still need it?
13:05:53 <ttx> My current take is to merge it first thing after release
13:05:59 <ttx> if still needed
13:06:08 <ttx> was that bumped ?
13:06:23 <dhellmann> no, that is still the most current version of requests, I thought it was possible they had fixed it
13:07:07 <ttx> not fixed yet. I'd say it's not a critical problem unless we have stuff to merge
13:07:24 <ttx> we /could/ merge it and ignore the syncs
13:07:29 <dhellmann> do you know if the distros are aware of the issue?
13:07:35 <dhellmann> yeah, that's another approach
13:07:50 <dhellmann> I guess at this point the main concern is to communicate that we know this version breaks us in some ways
13:07:55 <ttx> we already ignore a number of syncs, like translations
13:08:06 <ttx> We can make a release notes note
13:08:08 <dhellmann> but it's not clear to me whether we're really *broken* or just broken because of the images used in the gate
13:08:15 <dhellmann> ok
13:08:18 <ttx> that's what we did to avoid such late blacklisting in the past
13:08:36 * dhellmann nods
13:08:38 <ttx> maybe flaper87 can shine some light
13:08:58 <ttx> whether we're really *broken* or just broken because of the images used in the gate
13:09:32 <ttx> I'm fine approving the requirements blacklist pre-release, if we stand ready to -2 the resulting syncs
13:09:59 <ttx> since those are often blindly approved
13:10:10 <ttx> we need to catch them before they are
13:10:12 <dhellmann> yeah, do you think we would catch them all before someone approved one?
13:10:27 <ttx> we probably would, if we can predict when that triggers
13:10:51 <dhellmann> release day is this thursday, right?
13:10:55 <ttx> yes
13:11:03 <ttx> so let's go through the other last things first
13:11:22 <ttx> 2/ Release notes
13:11:32 <ttx> A number of people are chasing projects down so that they fill them
13:11:45 <dhellmann> good
13:11:49 <ttx> there is the weird question of scope, since there is no integrated release anymore
13:12:06 <ttx> We said we'd chase down release:managed ones and make sure those are done
13:12:12 <dhellmann> I'll hold off on the reno announcement until after the work on the wiki page has settled down
13:12:17 <ttx> and anyone else is fine to add
13:12:25 <ttx> 3/ Release day.
13:12:46 <ttx> On Thursday I consolidate the release pages on Launchpad, which takes nearly forever
13:12:52 <ttx> and retag
13:12:56 <ttx> (which doesn't take that long)
13:13:08 <ttx> so Thursday is basically too late for respins
13:13:20 <ttx> Wednesday is the very last moment for that
13:13:45 <ttx> I'll try to get most of the launchpad foo done in the EU morning
13:14:12 <ttx> I'll say that getting rid of that would remove a lot of work from release day.
13:14:20 <dhellmann> when you say "consolidate" what does that mean?
13:14:23 <ttx> but then I digged that hole
13:14:52 <ttx> well, it's about moving everything that's been done on milestones to the final release page, by retargeting bugs and blueprints
13:14:57 <dhellmann> ah
13:15:06 <ttx> so that "the release" shows everything that's been done, instead of nothing
13:15:33 <dhellmann> ah, right, I forgot you do that because we ended up not doing it for oslo
13:16:03 <ttx> let me find some pointers
13:16:14 <ttx> just in case
13:16:26 <ttx> https://wiki.openstack.org/wiki/ReleaseTeam/How_To_Release#Final_release_2
13:17:15 <ttx> Anything I'm missing ?
13:17:54 <dhellmann> I think that's it. I couldn't tell just from scrollback whether we were waiting for any critical issues other than the requests requirement block
13:17:58 <ttx> So.. back on the topic of the requests blacklist
13:18:02 * flaper87 reads backlog
13:18:03 <dhellmann> it sounds like not?
13:18:10 <ttx> two options
13:18:42 <ttx> Plan A is to freeze that until post-release, secretly hoping requests will update their release anyway
13:18:53 <ttx> Plan B is to approve the requierments change now and -2 the syncs
13:18:54 <dhellmann> flaper87: we were trying to tell how much trouble the requests 2.8 release is causing us
13:19:26 <ttx> Benefit of Plan B is that the requirements (and syncs) will be in, in case we urgently need them at the last minute for a late respin to pass tests
13:19:31 <flaper87> mmh, my current problem is this: https://review.openstack.org/#/c/232462/
13:19:45 <dhellmann> whether plan A can work depends on the nature of the issue -- if it's a packaging problem they may not fix it, because of the vendoring upstream vs. separate packages downstream
13:19:49 <flaper87> That backport for liberty is blocked by this requests issue
13:19:53 <ttx> Drawback of Plan B is we need to catch the syncs before some trigger-happy PTL approved them
13:19:57 <flaper87> it's probably not critical and we could wait, FWIW
13:20:13 <ttx> but even in that case we could retag the RC tag rather than HEAD, so it's not a critical issue
13:20:24 <dhellmann> flaper87: is the problem related to a change in the behavior of requests?
13:20:35 <flaper87> yes
13:20:42 <ttx> in both plans we would add a note to the RElease notes saying requests 2.8.0 sucks
13:20:45 <flaper87> There's a GH issue for it
13:20:48 <flaper87> lemme look it up
13:21:06 <flaper87> but, tl;dr: it is not behaving the same way
13:21:08 <dhellmann> flaper87: ok, whenever requests comes up I hear about the packaging stuff so I wasn't sure if this was a real code change
13:21:27 <ttx> flaper87: any other gate (other than python-glanceclient) broken by that ?
13:21:28 <flaper87> to be precise, the vendored urllib3 in requests is not behaving as it should
13:21:52 <flaper87> ttx: yup
13:21:54 <flaper87> 2s
13:22:26 <flaper87> https://bugs.launchpad.net/ospurge/+bug/1503768 ospurge
13:22:26 <openstack> Launchpad bug 1503768 in python-glanceclient "keystoneclient.exceptions.RequestTimeout with requests==2.8.0" [Undecided,New] - Assigned to Flavio Percoco (flaper87)
13:22:26 <ttx> dhellmann: I think we can do plan B if we know when to catch the syncs.
13:22:52 <flaper87> but the exception in that bug is from keystoneclient so, I'd expect keystoneclient to be broken by this too
13:23:16 <ttx> fungi: do you know when the requirements syncs are triggered ? Is it on merge or a periodic job ?
13:23:31 <fungi> on merge
13:23:32 <flaper87> I believe it's periodic
13:23:33 <dhellmann> ttx: I agree we can probably catch them. That won't solve flaper87's problem for glanceclient, but it will at least signal to distros that we have a known issue
13:23:37 <flaper87> u, I take that back
13:23:40 <flaper87> fungi: knows better
13:23:59 <ttx> flaper87: it's just that the merge is periodic
13:24:05 <flaper87> ttx: lol
13:24:09 <fungi> hah
13:24:20 <fungi> well, the constraints proposals are periodic
13:24:28 <ttx> fungi: does it take a while to trigger though, or is it relatively predictable ?
13:24:31 <fungi> (in an actual, non-joking sense)
13:24:44 <flaper87> dhellmann: if only we knew when the next requests release will go out, I think we could avoid this.
13:25:03 <ttx> I could approve it early tomorrow morning while the gate is calm and catch the syncs while most people are asleep
13:25:07 <dhellmann> flaper87: have you talked with sigmavirus24_awa about that? he's part of that team, isn't he?
13:25:07 <flaper87> to be safe, we should probably skip that versoin but the uncapped requirement should be enough to install the newest and be done with it
13:25:19 <fungi> ttx: it goes into the post pipeline, so if there's a resource starvation delaying post pipeline commits getting workers for their jobs then it will be however long that is plus something like 30 minutes for the job to run to completion
13:25:25 <flaper87> dhellmann: he's part of the team and I tried to reach out last week but TZs didn't work
13:25:25 <ttx> or we could even temporarily disable the stable/liberty sync
13:25:35 <dhellmann> flaper87: ack
13:25:41 <ttx> fungi: oh, that is why I thought it's periodic
13:25:53 <ttx> because when the gate is busy that can take quite a few hours
13:26:19 <fungi> i should also check that the job is currently running properly. i'll have a look at a recent job log for it
13:26:28 <ttx> one more reason to approve while gate is calm
13:26:48 <ttx> dhellmann: so.. plan A or plan B ?
13:26:50 <dhellmann> ttx: I do sort of like the idea of disabling the job for now, just to be safe, as long as we remember to turn it back on
13:27:01 * dimsum__ o/ (fashionably late to the party)
13:27:15 <dhellmann> ttx: B
13:27:26 <dhellmann> good morning, dims
13:27:41 <ttx> dhellmann: I think the risk profile is low enough to avoid the extra work in disabling/reenabling
13:27:58 <flaper87> #link https://github.com/kennethreitz/requests/issues/2811
13:27:59 <ttx> I'll approve it first thing tomorrow morning and block the syncs when they come through
13:28:02 <flaper87> That's the GH issue
13:28:03 <dhellmann> ttx: yeah, if you do it early tomorrow I think that will be true
13:28:13 <ttx> deal
13:28:35 <dhellmann> flaper87: so the patch is merged but not released, it seems
13:28:47 <ttx> #topic Open discussion
13:28:51 <flaper87> yup, I just double checked and requests' latest is still 2.8.0
13:28:51 <ttx> Anything else ?
13:29:01 * ttx is late with slide work
13:29:27 <dhellmann> I'm going to spend more time on that between now and thursday, too, since I'm delivering the talk thursday
13:29:52 <dhellmann> I'll leave comments with the parts where folks had questions, so we can figure out whether to add more info or leave space for q&a at the end of the talk
13:30:04 <fungi> fyi, log of most recent requirements update job:
13:30:05 <fungi> #link http://logs.openstack.org/f8/f82ab92ed8ff1a278391a196a39382fbb4211583/post/propose-requirements-updates/7347d46/console.html.gz
13:30:22 <fungi> most recent for stable/liberty i mean, of course
13:30:46 <dhellmann> fungi: ty
13:31:08 <fungi> that corresponds to the current branch tip of stable/liberty in the openstack/requirements repo, so barring more recent breakage than thursday when that ran, should be fine
13:31:42 <dhellmann> ttx: I was going to send email today or tomorrow to start communicating with ptls for mitaka about planning and process changes. I'll hold off on bringing up reno until after thursday so they don't confuse that with the current release notes process.
13:34:00 <dhellmann> ttx: it looks like there are some other library releases, too, so I'll probably handle those today
13:34:15 <ttx> ack
13:35:23 <Guest2821> dhellmann: ttx: ok with sending out a bunch of oslo library releases from master today? https://review.openstack.org/#/c/233326/
13:35:46 <dims__> dhellmann: ttx: ok with sending out a bunch of oslo library releases from master today? https://review.openstack.org/#/c/233326/
13:36:00 <dims__> dhellmann: ttx: https://review.openstack.org/#/c/232678/ and https://review.openstack.org/#/c/233256/
13:36:10 <dhellmann> dims__: yeah, that should be fine. I'll handle the others
13:36:21 <ttx> yep
13:36:22 <dims__> thanks i'll take care of those 3 items
13:37:19 <ttx> ok, anything lese before we close ?
13:37:22 <ttx> else*
13:37:27 <dhellmann> that's all I have
13:37:32 <ttx> alright then
13:37:35 <ttx> #endmeeting