21:00:29 <mriedem> #startmeeting stable
21:00:30 <openstack> Meeting started Mon Jan  4 21:00:29 2016 UTC and is due to finish in 60 minutes.  The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:31 <mrunge> o/
21:00:32 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:34 <openstack> The meeting name has been set to 'stable'
21:00:44 <mriedem> hi, who's around?
21:00:56 <ihrachys> o/
21:01:02 <mrunge> o/
21:01:30 <mriedem> ok, will give it another minute or so, maybe i can lure mtreinish in
21:01:36 <mriedem> tonyb: are you around?
21:01:52 <tonyb> mriedem: yup
21:02:10 <tonyb> sorry should have waved
21:02:14 <mtreinish> mriedem: oh is something going on?
21:02:20 <mriedem> big big things are going on
21:02:25 <mtreinish> heh
21:02:39 <mriedem> let's just roll through this, there isn't much planned today since it's the first day back for a lot of people
21:02:40 <mtreinish> I guess I did say this time worked for me
21:02:50 <mriedem> the agenda is here https://wiki.openstack.org/wiki/Meetings/StableTeam#Agenda
21:02:51 <mriedem> #link https://wiki.openstack.org/wiki/Meetings/StableTeam#Agenda
21:03:04 <mriedem> pretty sparse, i think ttx said he had some items for next week
21:03:13 <mriedem> #topic status
21:03:32 <mriedem> so i put a logstash query in the agenda http://goo.gl/5qiw2U
21:03:39 <mriedem> that shows periodic-stable failures in the last 7 days
21:04:01 <ihrachys> well I guess that's mostly lbaas for a known reason
21:04:07 <mriedem> i've been out but just quickly looking over these this afternoon they are mostly neutron-lbaas and ceilomteer, which are known issues in https://etherpad.openstack.org/p/stable-tracker
21:04:09 <mriedem> yeah
21:04:17 <mtreinish> mriedem: there is also: http://status.openstack.org/openstack-health/#/g/build_queue/periodic-stable
21:04:19 <mriedem> kilo was also busted, but mkoderer fixed that
21:04:30 <mriedem> mtreinish: ooo pretty
21:04:36 <mriedem> #link http://goo.gl/5qiw2U
21:04:40 <tonyb> mriedem: is that the testrepositoiry issue?
21:04:41 <mriedem> #link http://status.openstack.org/openstack-health/#/g/build_queue/periodic-stable
21:04:45 <mtreinish> but that doesnt show the devstack failures
21:04:48 <mriedem> tonyb: yeah, pbr and tox
21:04:52 <mtreinish> tonyb: yeah, its actually a pbr thing
21:05:00 <tonyb> mriedem: cool.
21:05:05 <mriedem> https://review.openstack.org/#/c/262470/ fixed kilo
21:05:32 <tonyb> mriedem: cool.
21:05:47 <mriedem> ihrachys: on the neutron-lbaas thing, did you have any updates on that? it was bit rotting in project-config from what i remember
21:06:07 <mriedem> https://review.openstack.org/#/c/245525/
21:06:15 <ihrachys_> mriedem: yeah. I tried several isolated patchsets, but seems like infra wants a full blown rework
21:06:27 <ihrachys_> mriedem: introducing new macros just for periodic jobs
21:06:32 <mriedem> the alternative is hard-coding it in the in-tree setup file right?
21:06:43 <ihrachys_> which was not in line with my scarce time :-|
21:06:53 <mriedem> sure, but
21:07:03 <mriedem> you wouldn't have to work on it
21:07:12 <mriedem> it's not great since it's a thing that has to be done every release
21:07:12 <ihrachys_> mriedem: probably yeah, we could go with hardcoded way. Not that I like it much, and not that I get infra concerns enough.
21:07:46 <ihrachys_> mriedem: I haven't seen many volunteers to do it instead of me from infra side :)
21:08:07 <mriedem> i doubt anyone besides me knows about that infra patch
21:08:20 <ihrachys_> mriedem: you mean from infra folks?
21:08:26 <mriedem> well, anyone
21:08:35 <mriedem> are there neutron-lbaas mini PTLs that should be involved with this kind of thing?
21:08:55 <ihrachys_> mriedem: people were reviewing it, telling me I do it wrong, and that we need macros. not that we did not have discussion around it.
21:09:00 * mriedem is unclear on the complicated beurocracy that is the neutron stadium
21:09:06 <tonyb> ihrachys_: I'll have a look at it.
21:09:24 <tonyb> (where "it" == the new macros
21:09:28 <mriedem> #action tonyb to take a crack at the infra side of https://bugs.launchpad.net/neutron/+bug/1523241
21:09:30 <openstack> Launchpad bug 1523241 in neutron "neutron-lbaas failing in stable/liberty due to "ImportError: No module named embrane.common"" [Medium,In progress] - Assigned to Ihar Hrachyshka (ihar-hrachyshka)
21:09:46 <ihrachys_> mriedem: there are lbaas cores, not sure they would be of help here though. this tox_install.sh thing is hardly understood by enough folks in neutron team.
21:09:53 <ihrachys_> tonyb: thanks
21:09:56 <mriedem> ok, cool, thanks. if that doesn't get anywhere i'd say we hack the branch in the scripts for now if no one has time to work on this
21:10:08 <mriedem> ihrachys_: yeah
21:10:12 <ihrachys_> sounds like a plan
21:10:14 <mtreinish> ihrachys_: ugh, that script...
21:10:41 <ihrachys_> mtreinish: it's now all over neutron subprojects :D
21:11:03 <mriedem> funny how that works
21:11:12 <mriedem> anything else on this or should we move on?
21:11:14 <mtreinish> sigh, this is why I keep saying this is why you need a stable interface and work from releases
21:11:17 <mtreinish> but whatever
21:11:29 <ihrachys_> side note: problem with periodic jobs is they don't indicate that a patch will actually merge in gate.
21:11:50 <ihrachys_> mtreinish: we work on neutron-lib, it's ongoing.
21:12:03 <mriedem> ihrachys_: can you expand a bit on the periodic job thing?
21:12:17 <mriedem> they don't actually run on a patch
21:12:39 <ihrachys_> mriedem: well, the problem is that it covers unit tests, docs, prolly that's all? none of those devstack jobs that we run in real gate.
21:12:54 <mriedem> yeah
21:13:06 <mtreinish> ihrachys_: that's not true, there are periodic stable jobs for tempest jobs
21:13:07 <mriedem> mtreinish: did anyone ever propose running a dsvm tempest job in the periodic-stable queue?
21:13:15 <ihrachys_> and those jobs start to fail no less often than unit tests. so it's kinda not complete solution.
21:13:16 <mriedem> oh right,
21:13:28 <mriedem> stable tempest kilo was failing on that devstack thing
21:13:32 <mtreinish> right
21:13:40 <ihrachys_> mtreinish: really? ok. probably not of those from gate anyhow
21:13:55 <mtreinish> ihrachys_: jsut look at the link to the health dashboard
21:14:06 <mtreinish> those are the periodic stable tempest jobs we run
21:14:09 <mriedem> ihrachys_: e.g. http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-neutron-full-kilo/8de4dcd/console.html
21:14:20 <mtreinish> it's not exhaustive, but its the common ones
21:14:35 <ihrachys_> ack
21:14:35 <mriedem> which hits that devstack pbr failure http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-neutron-full-kilo/8de4dcd/logs/devstacklog.txt.gz#_2016-01-04_06_11_29_938
21:14:40 <mtreinish> full, postgres full, and neutron full for each branch iirc
21:14:43 <mriedem> so yeah, i guess i forget about that one
21:14:57 <mriedem> i'm surprsised we even run that many
21:15:25 <mriedem> let's move on
21:15:36 <mriedem> the other stable/liberty failure i was seeing was ceilometer, which was tracked at one point with https://bugs.launchpad.net/ceilometer/+bug/1526530
21:15:37 <openstack> Launchpad bug 1526530 in Ceilometer "stable/liberty integration tests failing" [Critical,In progress] - Assigned to gordon chung (chungg)
21:15:46 <mriedem> gordc had some things proposed but abandoned them, i need to follow up
21:15:47 <mtreinish> mriedem: we added them back in the day because I complained I was the only one looking at the failures, this was supposed to raise awareness of stable branch issues in the gate....
21:16:03 <mriedem> #action mriedem to follow up with gordc on https://bugs.launchpad.net/ceilometer/+bug/1526530
21:16:32 <mriedem> were there other stable branch issues we haven't covered?
21:16:34 <mtreinish> (those were the only 3 configs back when we added them)
21:16:46 <ihrachys> pbr thing? do we have a plan for that?
21:16:58 <mriedem> ihrachys: the kilo thing was fixed this morning
21:17:14 <ihrachys> mriedem: hm? what was the fix?
21:17:30 <mriedem> https://review.openstack.org/#/c/262470/
21:17:57 <mriedem> so i think by and large stable right now is okish
21:18:07 <mriedem> gordc has a patch up for the ceilometer issue https://review.openstack.org/#/c/258642/
21:18:13 <mrunge> I'm a bit astonished, that issue hit so different projects but not all branches...
21:18:19 <ihrachys> nice. thanks for the link.
21:18:25 <mriedem> mrunge: i think stable/liberty was already fixed
21:18:33 <mriedem> mrunge: the tox thing hit a bunch of projects
21:18:47 <mriedem> but dims and others were fixing before the vacation
21:19:24 <mrunge> heh, horizon was hit by the issue beginning with Dec 26th
21:19:47 <tonyb> mrunge: yeah really bad timing ;P
21:20:20 <mriedem> ok, moving on from status
21:20:32 <mriedem> #topic release-plans
21:20:52 <mriedem> the only thing i have on the agenda is nova doing a stable/liberty 12.0.1 release
21:21:04 <mriedem> which i brought up in the ML awhile back http://lists.openstack.org/pipermail/openstack-dev/2015-December/081872.html
21:21:13 <mriedem> these are the open reviews
21:21:15 <mriedem> #link https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/liberty
21:21:42 <mriedem> i haven't gone through all of those yet, but i want to get at least a couple in, like https://review.openstack.org/#/c/253224/
21:21:51 <mriedem> it's basically a point release to get some key bug fixes out
21:22:13 <mriedem> anyway, i plan on getting into more of those this week and then i'll be pinging nova stable cores to look
21:22:53 <mriedem> is anyone else aware of any projects that are looking at doing a stable branch release?
21:23:10 <tonyb> mriedem: FWIW I'll look through again and see if there is anything *I* think shoudl be in 12.0.1 (apart from the reno review)
21:23:18 <mriedem> tonyb: thanks
21:23:26 <mriedem> i didn't want to start pinging people today on reviews
21:23:26 <mtreinish> I dont know of any
21:23:37 <mriedem> there have been a couple of projects that have done a stable/liberty release already
21:23:39 * ihrachys wonders why we don't automate minor releases, and make them frequent. we were thinking of getting there for neutron for L.
21:23:54 <mriedem> ihrachys: well, there was that long thread in the ML on that
21:24:09 <mriedem> like, every commit being a release
21:24:30 <mriedem> it's really up to the project teams now as to how frequent they want to release
21:24:33 <dims_> mriedem : mordred had this review https://review.openstack.org/#/c/247677/ about documenting EOL and proposed release dates in openstack/releases repo. do we want to do that?
21:25:06 <mriedem> dims_: huh, hadn't seen it, i'll have to look
21:25:13 <mriedem> #action mriedem to look at https://review.openstack.org/#/c/247677/
21:25:37 <tonyb> it's merged
21:25:50 <mriedem> yeah
21:25:52 <mriedem> it's also confusing
21:25:58 <dims_> tonyb : i know, continue to use that place to document future releases
21:26:04 <mriedem> the commit message talks about when they will EOL but doesn't specify that
21:26:23 <mtreinish> mriedem: it does in the index
21:26:23 <tonyb> mriedem: yeah I just saw that
21:26:35 <mriedem> mtreinish: oh yeah
21:27:16 <mtreinish> we probably need to mark juno as EOL though
21:27:32 <mtreinish> instead of security supported (EOL 2015-12-03)
21:27:40 <mriedem> i guess that's implied
21:27:49 <tonyb> dims_: ok.
21:27:57 <mtreinish> heh, I was just going off the other branches where it says EOL
21:28:31 <mriedem> ok, i guess if there are things to address there let's take it to -stable after the meeting
21:28:44 <mriedem> #topic open discussion
21:28:55 <mriedem> there are 2 items on the agenda
21:29:10 <mriedem> 1. i want to get a review dashboard link in the #openstack-stable channel topic
21:29:11 <tonyb> FWIW it's now markjed as EOL
21:29:23 <mtreinish> tonyb: ah, ok
21:29:41 <tonyb> mtreinish: send me the link I'll work with @infra to get it in there.
21:29:43 <mriedem> sdague had an email to the ML on dashboards for new gerrit, i was thinking it'd be good to have something like that in the -stable channel topic
21:29:53 <mriedem> tonyb: was that meant for me?
21:29:58 <anteaya> nice idea
21:30:05 <tonyb> mriedem: yeah, bad tab-complete
21:30:17 <mtreinish> tonyb: ??
21:30:31 <ihrachys> mriedem: what will the board contain? all backports?
21:30:47 <mriedem> ihrachys: just open reviews in stable/branches, pretty simple to start
21:30:59 <mriedem> but sdague's had things broken down by what was passing jenkins, what has a +2, etc
21:31:13 <mriedem> makes prioritizing easier
21:31:31 <tonyb> Yeah we can modify the dashboard as we go
21:31:54 <ihrachys> mriedem: I am agnostic, but if you want to use it as a data point, personally I care little for backports outside neutron subprojects (because the latter are already more than enough for my taste)
21:31:54 <mriedem> so i was going to take a stab at that when i got a chance, but if others are interested feel free to propose some ideas and we can check them out
21:32:14 <mriedem> ihrachys: same for me and nova,
21:32:26 <mriedem> ihrachys: i just like to see what it looks like at first and we can iterate on it
21:32:37 <mriedem> swift has a nice dashboard in their channel topic and i'd like nova to do similar
21:32:42 <ihrachys> mriedem: for neutron stable stuff, I use this: https://github.com/openstack/gerrit-dash-creator/blob/master/dashboards/neutron-subprojects-stable.dash
21:32:48 <tonyb> I tend to focus on nova and whatever I see the get the requirements updates passing
21:33:19 <mriedem> #link neutron stable review dashboard example https://github.com/openstack/gerrit-dash-creator/blob/master/dashboards/neutron-subprojects-stable.dash
21:33:56 <mriedem> anyway, it's a thing, if people have input or want to propose something to infra, we can review there
21:34:10 <mriedem> the last item in open discussion,
21:34:23 <mriedem> 2. https://etherpad.openstack.org/p/stable-tracker has completed sections for icehouse and juno, which are now EOL
21:34:46 <mtreinish> just drop them
21:34:52 <mriedem> it's getting messy in there, and since i spent part of my vacation cleaning up and organizing my office, i'm in the mode
21:35:04 <mriedem> yeah, so was just wondering if anyone had reason not to drop those
21:35:17 <ihrachys> kill the witch
21:35:22 <mriedem> ok, will do
21:35:26 <tonyb> mriedem: nuke 'em
21:35:38 <mriedem> that's it for agenda items, did anyone else have anything?
21:35:56 <mriedem> #action mriedem to drop icehouse and juno sections from https://etherpad.openstack.org/p/stable-tracker
21:36:07 <mriedem> ok, not hearing anything else, so we can end early.
21:36:14 <mriedem> like i said, ttx said he had some items for next week
21:36:18 <ihrachys> one thing we should discuss is pushing forward constraints to more projects. next time.
21:36:20 <mriedem> so i await in eager anticipation
21:36:39 <mriedem> ihrachys: yeah, good point.
21:36:48 <mriedem> those jobs are passing but non-voting in nova stable/liberty right now
21:36:56 <ihrachys> we did it for neutron, and will backport remaining stuff to L
21:36:59 <mriedem> we should circle back with lifeless on the plan there
21:37:22 <mriedem> #action figure out what's going on with the -constraints jobs in stable/liberty and if/when they become voting
21:37:47 <mriedem> i'll follow up with lifeless and bring back what we talked about to this meeting next week
21:37:54 <ihrachys> I think L for neutron repo should be done, and we look into *aas and remaining gaps. hopefully more projects will get to it in L, so that we can reasonably consider extending life for the branch. but I suspect it will be Mitaka
21:38:27 <mriedem> i'm hoping the guys working on that are tracking that work somewhere
21:38:55 <ihrachys> let's hear from them for now.
21:39:03 <mriedem> ok, anything else?
21:39:21 <mriedem> taking that as a no, so we can end early
21:39:22 <mriedem> thanks everyone
21:39:24 <mriedem> #endmeeting