15:01:25 #startmeeting stable 15:01:26 Meeting started Tue Jan 12 15:01:25 2016 UTC and is due to finish in 60 minutes. The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:27 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:29 o/ 15:01:29 The meeting name has been set to 'stable' 15:01:42 hi, who is all here (besides ttx)? 15:01:50 o/ 15:02:04 mtreinish: tonyb? 15:02:13 * ihrachys attends two meetings, can lag 15:02:28 i'll give it another minute or so 15:03:00 I can lurk if that is helpful to numbers 15:03:17 it does 15:03:20 yay 15:03:25 well let's just get started 15:03:26 * anteaya is useful today 15:03:33 #link agenda https://wiki.openstack.org/wiki/Meetings/StableTeam 15:03:40 #topic status 15:03:52 periodic-stable failures http://goo.gl/5qiw2U 15:03:59 there are really just 2 issues with those 15:04:15 1. neutron-*aas jobs aren't getting the branch set in the periodic-stable jobs properly 15:04:32 ihrachys: proposed https://review.openstack.org/#/c/245525/ and tonyb is picking it up 15:04:59 tonyb++ 15:05:01 2. ceilomer dsvm job is having issues setting the branch for gnocchi https://review.openstack.org/#/c/264769/ 15:05:12 gordc is playing d-g whack a mole with that one 15:05:32 basically, d-g is trying to clone stable/liberty gnocchi which doesn't exist, so he has to override the branch that d-g pulls for that job 15:05:39 i'm hoping sdague and mtreinish can review that one 15:05:58 that's it for the periodic-stable jobs 15:06:00 the dashboard shows horizon issues as well ? 15:06:12 the openstack-health one? 15:06:26 oh, that was last week 15:06:27 no the link you pointed us to 15:06:30 oh ok 15:06:36 yeah, last week was the kilo + liberty fubar 15:06:43 where horizon wouldn't install in dsvm on kilo 15:06:50 which broke all of kilo dsvm and liberty grenade jobs 15:06:55 that was fixed last week 15:07:31 i should probably adjust that logstash link to 2 days rather than 7 maybe 15:07:37 mriedem: I just approved it, I do think that the gnocchi story here tells a more important release concept that mixing independent release components with integrated release components just makes for pain 15:07:55 sing it 15:08:03 well 15:08:22 big tent everyone releases on their own schedule, so theoretically this should be fine right? 15:08:33 as long as they don't want to have linked jobs 15:08:44 gnocchi is a library isn't it? 15:08:45 it's the linked jobs that's the issue 15:08:47 yeah 15:09:03 i guess if ceilometer just pulled down a pip of gnocchi from pypi then it wouldn't be a problem 15:09:11 sdague: maybe we should make that more obvious in the doc where we explain both choices 15:09:14 so you can work like a library and install from pypi 15:09:19 and that's fine 15:09:21 like in the project team guide 15:09:21 I think this may be a gap where the theory of the big tent and the reality of what people with with it causes issues 15:09:24 but co-gating is not 15:09:44 anyone want an action for a doc update? 15:09:49 and even our libraries have stable/* branches 15:09:50 * mriedem is not sure where that is 15:09:51 anteaya: the consequences of choice are not always well explained 15:09:57 (yet) 15:10:04 or understood as well make up the rules 15:10:11 so I think anything out of openstack that has branches needs a stable/liberty branch 15:10:20 s/well/we 15:10:29 I think that's the thing that we missed before 15:10:55 sdague: or has to be installed from pypi 15:11:06 I don't mind taking an action item for a patch to the project team guide but it will need heavy review as I don't know what it should say 15:11:21 but I'll kick it off 15:11:21 sdague: interesting... would you mind raising a thread about that once you have it formalized ? I'd like us to discuss that a bit more 15:11:45 i will gladly assign an action to someone here 15:11:51 anteaya: we need to get to the bottom of what that means first 15:11:59 understood 15:12:09 I agree clarity is helpful 15:12:30 ttx: I've got a few other hard problems on my plate right now, this is just something I've been mulling as I've watched this gnocchi review hit various brick walls 15:12:43 #action mriedem to talk to ppl about co-gating and what that means for big tent and projects with branches and w/o, like ceilometer + gnocchi 15:13:28 i also wonder how that has implications for lifeless' spec that is part of a longer term goal to remove stable branches 15:13:31 or i thought it was 15:13:35 but we can get into that later 15:13:41 I mean, I'm open for limiting the choices there, I just want us to think a bit more about it before we announce anything 15:13:57 yeh, that's fair. I think there are 2 modes we know are good 15:14:05 the choices already are limited, we just need to inform folks 15:14:15 people can't test anything with everything else 15:14:20 you don't have branches, you release from master, and support all supported stable branches with your master 15:14:24 see: tempest 15:14:31 neutron drivers on independant release have run into this 15:14:51 nothing depends on tempest though, 15:14:51 you release with branches, and have a branch that's appropriate for the official release branches 15:14:57 so in this example, gnocchi is tempest 15:15:01 sdague: sounds fair 15:15:32 mriedem: lets say python-novaclient then 15:15:39 which has branches :) 15:15:45 it does now 15:16:13 part of the problem is we used to actually have this model, then capping forced a lot of new branches, then people thought they could do anything they wanted with branches 15:16:47 if ceilometer depends on gnocchi as a library, i'm not really sure why it's not just installing it via pypi 15:16:56 at whatever version it needs 15:17:07 gnocchi can have master and support all stable if it wants 15:17:15 yeh, I don't know 15:17:16 feels like a discussion for the release team anyway 15:17:27 i'll also follow up with gordc and jd i suppose 15:17:44 mriedem, it cannot b/c of deps on master might be too new for older branches 15:18:26 if it has too many common deps it should probably follow the cycle basically 15:18:29 apevec: i'm saying ceilometer wouldn't depend on master gnochi 15:18:51 ceilometer needs to depend on a version of gnocchi that is constrained by upper-constraints 15:18:56 in each stable branch of ceilometer 15:19:29 and if gnocchi does a backwards incompatible thing, then it's breaking backwards compat and we either have to cap it or revert that breaking change 15:19:39 which gets into lifeless' spec https://review.openstack.org/#/c/226157/ 15:20:10 that would also mean that the requirements repo changes would have to be gated on ceilometer, which they aren't today, but that gets into more details than we want here 15:20:29 shall we move off this topic? 15:20:34 yes 15:20:45 ok 15:20:49 ttx: Remaining stable/kilo (coordinated) point releases 15:20:56 So... Starting with liberty we do separated, on-demand stable point releases 15:21:03 But we still have a few coordinated point releases to do for kilo 15:21:05 gnocchi is actually a service, ceilo depends only on gnocchiclient 15:21:11 #link https://wiki.openstack.org/wiki/StableBranchRelease#Rinse.2C_Repeat 15:21:19 2015.1.3 January, 2016 15:21:23 2015.1.4 (eol) 2016-05-02 15:21:29 Daviey (in a private thread with apevec and me) proposed to handle both 15:21:41 way to be not open :) 15:21:49 sheesh 15:22:08 i'm fine with that 15:22:16 indeed, but since he volunteered I forgave him 15:22:24 It would be great to go through the release process and check what still applies 15:22:29 Also check if the needed tooling is still available, since we did remove some tools from release-tools recently 15:22:48 So I'll contact him and check we still have all ducks in row 15:22:58 #action ttx to go through the stable release process with Daviey to doublecheck tooling 15:23:21 cool 15:23:23 mriedem, my fault, I started that offlist thread from juno EOL announcement 15:23:47 ttx: anything else on that item? 15:23:50 ttx, couldn't we use part of new tooling, just for pushing tags? 15:23:52 kilo is pretty stable atm 15:23:52 It's good to have a volunteer for the last 2, so that nobody else has to dive into old process 15:24:07 apevec: that is what I want to figure out 15:24:10 yay kilo is ~stable 15:24:32 mriedem: I'm done with this topic 15:24:40 ttx: Stable team tags: moving from has-stable-branches to follows-stable-policy ? 15:24:40 unless there are questions 15:24:48 OK; so... We currently have one tag to describe /some/ stable branch support: 15:24:55 #link http://governance.openstack.org/reference/tags/release_has-stable-branches.html 15:25:07 The problem is... this tag is a bit overloaded and doesn't really mean anything useful anymore 15:25:17 Any project can set up stable branches now, they can even name them stable/$series 15:25:25 they don't have to follow the policy or even add us to their $project-stable-maint teams 15:25:35 So I'd like to propose we retire this now-useless tag and replace it by something more useful 15:25:45 The way to think about it is in terms of answering a question that downstream users have 15:25:55 In our domain I think what they want to know is which projects have "stable branches that they can rely on" 15:26:06 i.e. which projects follow the stable branch policy, as defined and checked by the stable team 15:26:17 do we have a tag that identifies if a project interacts with the release team, in a follows their guidance kind of way? 15:26:25 So I'd like us to propose a stable:follows-policy tag, which would be granted by the stable team: 15:26:54 anteaya: we have release:managed, but that means somethign different. It's also non-optional to release so basically all projects need to follow 15:27:08 okay 15:27:11 - as long as we are confident they are following the policy 15:27:15 - once we are added to their stable +2 team 15:27:30 then we would apply stable:follows-policy 15:27:34 basically using the tag as the carrot for following the common stable policy 15:27:44 and using tag removal as the stick to encourage outliers to better comply 15:27:56 how does that sound ? 15:27:56 what is the criteria for removal? 15:28:08 backporting and merging things that are against the stable policy 15:28:10 like features or api changes 15:28:12 it is the removal part we seem to have the hardest part with 15:28:13 anteaya: "project does not follow stable branch policy" ? 15:28:38 If that sounds like a good idea I can draft a tag definition 15:28:39 i have no problem with removal if someone continually gets caught breaking policy (like more than once as a mistake) 15:28:54 it's hard to police given there are so many 15:29:02 mriedem: that's the issue 15:29:02 that we would review as a team before proposing it to the tc 15:29:25 so really someone has to speak up and say project x backported and merged a thing which broke my production cloud - which rarely happens 15:29:31 Just wanted a temperature reading from the team before I get started 15:29:33 or some projects CI would have to start failing 15:29:34 mriedem: right 15:29:41 ttx: i'm all for this 15:29:47 NB: I would limit the tag to stable branches that track common stable series (like stable/liberty) 15:29:50 it's just the oversight that's hard 15:29:57 mriedem: yes 15:29:57 wow, that quite hard to decide 15:30:00 rather than also apply it to project-specific branches (like stable/1.0) 15:30:21 how often did we saw some bad backports, breaking compatibility? 15:30:41 mrunge: that's the thing, you don't really at a global level 15:30:49 especially if no one speaks up in irc or the ML if it happens 15:30:53 But yes, I agree, we should encourage folks to look at that 15:30:56 mrunge: some projects just don't backport bugfixes 15:31:14 Ideally, we would have more test coverage 15:31:16 it's fine, they just won't have the tag 15:31:30 since downstream users should know their stable branches are mostly worthless 15:31:35 i guess i wasn't thinking about whether or not they are backporting bug fixes 15:31:50 mrunge: I often catch those in neutron 15:32:00 I mean, I block them :) 15:32:02 since if you consider the number of bugs fixed in nova per master release vs how many get backported (which are applicable), it's a drop in the bucket 15:32:11 ihrachys, yes, that's greatly appreciated 15:32:17 I mean, it's a significant piece of information to communicate downstream, and currently it is quite hard to determine 15:32:28 yeah 15:32:36 and the current tag hides it completely 15:32:39 rather than bugs, it's probably just easier now to see if they are doing stable point releases 15:32:40 mriedem: in neutron, we backport everything applicable to N-1 15:32:43 One remaining question though is whether we would allow "late" stable branches to have the tag 15:32:53 things that are release:independent because they can't make the release date, but still want to have a branch called stable/liberty 15:33:17 like fuel 15:33:22 or can they make a stable/liberty branch in the middle of mitaka 15:33:28 or some neutron stadium things 15:33:31 which someone wanted to do the other day 15:33:49 we've created stable/juno branches on things after the fact, when we needed to backport something 15:33:51 why is it an issue to post date releases? 15:33:55 my gut feeling is that those should be outside policy but meh 15:34:00 chef cookbooks cut stable branches long after master is released too 15:34:15 cut or create? 15:34:28 they cut their mitaka release a few months after mitaka is released in the code projects 15:34:29 or does the difference matter? 15:34:30 sdague: we are basically communicating: "those projects have sane stable branches" 15:34:36 b/c the cookbooks are stabilizing the release 15:34:42 I think part of being sane is to have them created around the same time 15:34:44 yeh, I think we need to realize that stable/liberty is a convenience concept to users of stuff that is going to all work in liberty 15:34:49 so that you can follow through them as a user 15:34:56 mriedem: that makes sense due to their work 15:35:12 ttx: I agree, created dates need to be in the release in question 15:35:13 but as I said I don't feel strongly about this 15:35:14 i agree with sdague, the branch is there to indicate it works with the liberty releaes of the code 15:35:18 ttx: I guess, all the orgs I know that are deploying from stable don't do so until + 4 months or so 15:35:40 as long as they are following the stable branch policy, i don't really care when they create the branch 15:35:43 sdague: ok, I can leave that out of the definition 15:35:55 there are some projects that i think still have active stable/icehouse branches, fwiw :) 15:36:01 I think I have enough to start drafting 15:36:02 but it may affect their ability to test that branch 15:36:11 anteaya: sure, like stable/icehouse 15:36:15 right 15:36:20 but yeah, let's review in the review when that's up 15:36:23 I think that needs to be identified 15:36:31 moving on 15:36:33 it might be worth tagging projects that get a stable branch out at release date, those that hit it within 2 months, and those which show up later 15:36:37 ttx: Liberty stable point releases status: do we need to push for some releases ? 15:36:43 #action ttx to draft a stable:follows-policy tag definition for the team to discuss 15:36:48 So for liberty we said that projects would just request stable point releases themselves 15:36:55 Most did, but some may need some encouragement/reminder 15:37:02 I think regular releases are important to make our stable branches look better 15:37:03 yes! 15:37:09 so I think it's still the stable team responsibility to track that releases are regularly made 15:37:21 yes, i've been pushing on the nova one 15:37:23 For liberty the following projects did not make a release since liberty release: 15:37:27 we have a security fix that has to get in first 15:37:31 barbican, congress, designate, heat, horizon, mistral, nova, sahara, trove, zaqar 15:37:39 swift and aodh haven't released either but since they are doing intermediary releases that is not as needed imho 15:37:49 Some of them just didn't have that much stable activity, like barbican, congress, trove or zaqar 15:37:59 (might be worth checking if they really are following stable branch policy and backporting "enough" changes, but that is another topic) 15:38:10 Some of them have 34+ changes and should really be doing a point release: Heat, Mistral, Nova, Sahara 15:38:19 Some have 17+ and should consider one: Designate, Horizon 15:38:33 So in summary I think we should regularly review this and have someone assigned to the nagging 15:38:41 I'm happy to take the first round 15:38:45 horizon has probably more, including i18n updates, which are quite big 15:39:13 ttx: thanks for pulling the numbers, i agree we have to stay on top of it, i've been busy trying to get the nova one ready 15:39:23 probably some automated way of figuring out how much they need a release would be nice too 15:39:25 so i'd appreciate it if you want to own that first round of nagging 15:39:43 Yeah I'll own it, figure out what metrics we should track there 15:39:47 well i think we know after you've backported security fixes, you should release 15:40:03 then we can review that regularly, say once every ~ 6 weeks or so 15:40:08 in the case of nova, we really didn't start talking about it until some regression fixes were backported 15:40:33 from my pov, it's going to be hard to just look at what's been backported to e.g. zaqar and know if they need something 15:40:44 there could be 20 piddly backports or 2 critical backports 15:40:47 yeah that should not prevent releasing more aggressively 15:41:08 sure 15:41:22 we could probably have some tooling that scrapes the bug priority too 15:41:27 it's just that before liberty we had the safety net of the catch-all coordinated release 15:41:30 and flag if there are critical bugs backported that aren't yet released 15:41:33 ttx: I think oslo team should have some scripts they use to track unreleased changes 15:41:53 yeah, I was thinking going from that 15:42:01 anyway another action for me 15:42:12 ./list_unreleased_changes.sh in release-tools repo 15:42:20 #action ttx to do the first stable release nagging round 15:42:20 should CVE fix trigger point release? 15:42:26 apevec: yes 15:42:26 apevec: yeah 15:42:28 Nova had one right? 15:42:38 apevec: yes, the backports are being reviewed and then we'll do 12.0.1 15:42:45 (for stable/liberty only) 15:43:01 alright that is all I had 15:43:28 ttx: to get the list that haven't released yet, where did you come up with that? the releases repo? 15:43:54 ttx, there was a thread from zigo about stable horizon supporint latest django ? 15:44:03 mriedem: and some hand count on git logs 15:44:15 apevec: Hi! 15:44:35 zigo, hi - why do you want django 1.9 ? 15:44:38 1.8 is their LTS 15:44:40 I've wrote lots of patches for Django 1.9 in Horizon already. 15:44:41 apevec: not related to my topic, but yes 15:44:43 apevec: we had to cap django_compressor in stable/kilo horizon last week b/c the latest release drops support for django<1.8 15:44:48 apevec: Debian Sid already has Django 1.9. 15:44:56 So there's an RC bug filled. 15:44:59 I need to have it fixed. 15:45:23 mrunge said 1.8 is better choice 15:45:25 Also, Horizon in Mitaka will need to support Django 1.9, as most likely, it's going to be the version in the next LTS. 15:45:28 1.9 will be out of support soon 15:45:43 Better or not, we need to move forward and support the latest version. 15:45:52 django-1.9 won't be the next LTS version 15:45:57 it's 1.8 15:46:45 feels like debian bet on the wrong horse then 15:47:02 Well, anyway, I need to fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809575 15:47:03 Debian bug 809575 in src:horizon "horizon: FTBFS: ImportError: cannot import name importlib" [Serious,Open] 15:47:19 ttx: Debian Sid is always on the edge. 15:47:37 zigo: someone will have some fun backporting 1.8 sec fixes to 1.9 in 2017 15:47:47 Also, remember that the packages for Django are maintained in Debian, and just sync to Ubuntu. 15:47:59 zigo has a point, as we don't even support Django-1.9 right now in mitaka 15:48:20 but yeah, I think that's orthogonal to the issue at hand 15:48:26 mrunge: I really hope that my patches will be accepted when I send them (of course, without destroying 1.8 compat). 15:48:37 As much as I know, no problem with that so far. 15:48:59 zigo, I would prefer, if you'd file bugs in launchpad about that 15:49:00 I did couples of django.getversion() calls when needed. 15:49:00 we'll need to support >=1.8 anyway 15:49:22 I would like see fixes backported to liberty as well 15:49:36 just making horizon work in more environments 15:49:38 As I wrote to you in #openstack-horizon, I prefer to finish the work first, and make sure nothing is broken, before attempting to upstream my work. 15:49:51 Timur S. wrote to me he will try to help tomorrow as well. 15:50:07 I was just asking to document issues, just to be able to track them 15:50:25 Well, the issue is that when you fix one issue, it uncovers another one. 15:50:32 and I'm really glad you're trying to fix those issues 15:50:39 So it's impossible to document all until all is kind of fixed. 15:50:49 yes, sure. but still we can track it then 15:51:04 we should have a tracker for django-1.9 support anyways 15:51:10 makes sense? 15:51:14 can we move this to -horizon? 15:51:19 I'll do things properly and file bugs. 15:51:25 we already had the same discussion 15:51:26 It's just too early for that right now, IMO. 15:51:34 and agreed to disagree 15:51:38 ;-) 15:51:40 :) 15:51:47 #topic open discussion 15:51:53 there was nothing on the agenda 15:51:58 does anyone have anything else for today? 15:52:16 ok, well let's end the meeting, thanks everyone 15:52:19 #endmeeting