21:02:19 <ttx> #startmeeting project
21:02:20 <openstack> Meeting started Tue Nov 26 21:02:19 2013 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:24 <openstack> The meeting name has been set to 'project'
21:02:25 <ttx> #link http://wiki.openstack.org/Meetings/ProjectMeeting
21:02:35 <ttx> #topic Icehouse-1 progress
21:02:42 <markwash> o/
21:03:03 <ttx> icehouse-1 looks mostly on track from my 1-to-1s today, and not as empty as you'd think
21:03:08 <ttx> We should also have a Swift 1.11.0 next week
21:03:47 * russellb resists obligatory "why is swift different" question ...
21:04:08 <ttx> heat and horizon still have a lot to land
21:04:27 <ttx> but otherwise everyone else is on track
21:04:34 <ttx> #topic Icehouse cycle roadmapping
21:04:46 <ttx> we now have release status for icehouse at:
21:04:48 <lifeless> ttx: any release concerns about the proposed scheduler split out?
21:04:53 <ttx> #link http://status.openstack.org/release/
21:05:12 <ttx> lifeless: could you summarize your plans ?
21:05:31 <ttx> We have 172 tracked blueprints (Medium priority or above) in Icehouse so far
21:05:41 <ttx> I'd like to have this basic roadmap to be complete by the end of the week
21:05:47 <ttx> It doesn't have to be "final" in any way.
21:05:57 <ttx> It just have to be a fair reflection of what you know will be worked on and when you expect that work to hit
21:06:02 <lifeless> ttx: https://etherpad.openstack.org/p/icehouse-external-scheduler line 38 down for three sections; BP is coming later today
21:06:04 <ttx> (this is a coordination and communication tool)
21:06:16 <ttx> So please add missing blueprints to cover what you know should happen, and set an assignee and priority for them.
21:06:40 <russellb> since we're doing more in depth blueprint reviews for nova, there will still be a batch in the process, and not prioritized, but we'll have them all reviewed and at least under discussion
21:06:42 <ttx> lifeless: so this causes a number of issues
21:06:42 <russellb> (for nova)
21:06:56 <russellb> lifeless: i think we should just go off and do the technical bits that are right
21:07:07 <russellb> lifeless: and come back when we have something working, and a more well defined upgrade / deprecation plan
21:07:14 <lifeless> russellb: ack
21:07:20 <russellb> that we think are right, at least
21:07:25 <ttx> lifeless: in theory a new project can't be integrated until J, supposing you fast-track incubation
21:07:42 <russellb> ttx: ack
21:08:06 <russellb> looking at taking a cinder-like approach ... minimizing divergence from what it's replacing
21:08:13 <ttx> so marking it deprecated supposes you have the alternate project well lined up to replace it
21:08:13 <russellb> so that we can get to usable ASAP
21:08:34 <ttx> but I guess overall that's doable. Cinder-style
21:08:47 <russellb> early, but i think it's doable :)
21:09:05 <ttx> lifeless: i'll have to go deeper in that etherpad :)
21:09:47 <ttx> russellb: the trick is to avoid doing it ironic-style
21:09:54 <dansmith> oof
21:10:02 <russellb> well ironic-style is closer to neutron style
21:10:07 <russellb> IMO
21:10:07 <ttx> russellb: i.e. deprecating in icehouse and not replacing it in J
21:10:15 <russellb> but yes, with you on that
21:10:23 <russellb> won't deprecate until we know the replacement is well established
21:10:34 <ttx> right
21:10:36 <lifeless> so, if we start with it as a nova project in a new tree we'll have the technical room to do what we need.
21:10:50 <russellb> compute program project you mean?
21:10:58 <lifeless> *snarl*
21:11:02 <lifeless> :)
21:11:05 <russellb> IMO, yes, compute program until it starts growing real legs to do something more
21:11:11 <russellb> but it's a ways before it can get that far
21:11:18 <lifeless> agreed.
21:11:22 * annegentle sends lifeless some turkey and gravy
21:11:25 <ttx> lifeless: you should still file for incubation asap. There are some pretty harsh requirements for incub/graduation now :)
21:11:32 <russellb> ha
21:11:41 <russellb> there's not even a repo yet :)
21:11:47 <lifeless> so, lets move on
21:11:51 <lifeless> I have what I need
21:11:55 <russellb> cool
21:12:01 <ttx> other questions on icehouse-1 / icehouse roadmap before we discuss cross-project issues ?
21:12:22 <ttx> mriedem: around ?
21:12:28 <mriedem> ttx: yup
21:12:29 <ttx> #topic Oslo sync (mriedem)
21:12:31 <mriedem> powervm
21:12:32 <mriedem> oh
21:12:39 <bnemec> heh
21:12:40 <ttx> mriedem: not yet! i'll let you introduce the discussion
21:12:54 <mriedem> so i just brought up the oslo sync debate in the nova meeting last week,
21:13:01 <mriedem> russellb said it should come here since it's cross-project
21:13:07 <mriedem> since then the ML blew up a bit more but i haven't stayed up to date
21:13:20 <russellb> let's not boil the ocean on this ... but basically, i was saying it wasn't a nova issue specifically
21:13:24 <mriedem> that's why dhellmann, bnemec, lbragstad and lifeless are here
21:13:30 <russellb> and that we should engage with dhellmann, and other oslo folks
21:13:53 <mriedem> from what i gathered quickly from the ML was there is general consensus on improving update.py a bit?
21:14:27 <mriedem> russellb: right, that's what i meant to say :)
21:14:31 <russellb> k
21:14:33 <ttx> mriedem: that was my interpretation of that thread as well
21:14:34 <lifeless> so the thing I really want to see
21:14:46 <lifeless> is all the oslo bits that aren't evolving weekly
21:14:51 <lifeless> released as libraries
21:14:56 <david-lyle> +1
21:14:59 <lifeless> their special incubation period is over
21:15:11 <dhellmann> we are working on that
21:15:16 <lifeless> and we'd avoid a raft of overhead keeping up with hacking format changes and stuff
21:15:16 <markmc> mriedem, yeah, improving update.py would help
21:15:35 <lifeless> dhellmann: I know - not criticising, just saying that thats what I really want to see
21:15:37 <dhellmann> however, there are some unfortunate cross-dependencies
21:15:52 <markmc> lifeless, mostly agree, but there are some we'd regret taking that approach with when we struggle with future API changes
21:15:56 <lifeless> dhellmann: because the other angle I'd go is aiming at making the incubator /be/ a library.
21:15:58 <dhellmann> yes, I would prefer time spent on graduating code over gold-plating update.py
21:16:15 <markmc> lifeless, but I dig your point that if the APIs haven't changed for a long time, let's just bite the bullet
21:16:27 <lifeless> markmc: AIUI the point of oslo is to go from 'production in one project to production in > one'
21:16:31 <lifeless> markmc: the incubator I mean.
21:16:43 * markwash loves the improvements we've been making before promoting to libs
21:16:47 <markmc> lifeless, point of incubator is to evolve towards API stability
21:16:54 <dhellmann> lifeless: yes, but not "as is" -- we also try to make the code actually usable
21:16:55 <lifeless> markmc: How do we measure that?
21:16:57 <dhellmann> right, what markmc said
21:17:07 <markmc> lifeless, you feel the truthiness in your gut
21:17:09 <markwash> lifeless: gut check?
21:17:32 <russellb> and i think oslo generally attracts people with a good gut for such things
21:17:38 <ttx> one mornign I woke up and FELT that rootwrap was stable.
21:17:40 <lifeless> markmc: but if noone is evoling the thing
21:17:43 <markmc> https://github.com/openstack/oslo-incubator/blob/master/MAINTAINERS
21:17:45 * dhellmann thanks russellb for complimenting his gut
21:17:49 <russellb> has worked well so far i think
21:17:49 <markmc> "
21:17:50 <markmc> APIs in oslo-incubator are resting here temporarily until they have been
21:17:50 <markmc> cleaned up sufficiently so that we can make a commitment to backwards
21:17:50 <markmc> compatibility and release the API in a properly published library.
21:17:51 <markmc> "
21:17:55 <annegentle> dhellmann: snort
21:18:03 <lifeless> markmc: we're stuck with manual integration rather than continual integration
21:18:33 <markmc> lifeless, incompatible API churn is painful too
21:18:44 <russellb> was afraid of this same discussion again, i feel like we've had it 100 times
21:18:52 <markmc> lifeless, basically, I think the incubation process is evolving, we'll see libraries released more quickly now
21:19:02 <ttx> the update.py pain is a good motivator for cleaning up APIs
21:19:06 <markmc> lifeless, but the principle of evolving towards API stability still makes sense
21:19:09 <dhellmann> right, I think we figured out how to do it with oslo.config and oslo.messaging
21:19:14 <dhellmann> that's going to make the next few easier
21:19:25 <lifeless> I'm not disputing the principle - thats a different discussion.
21:19:46 <lifeless> I'm worried that we're not executing effectively, and I'd like to know how I and the community can help that happen.
21:19:46 <ttx> OK, we've been havgin that discussion at least 10 times already, and I don't want to spend all this meeting on it
21:20:00 <markmc> one thing that can help get libraries out more quickly
21:20:03 <markmc> more people helping!
21:20:12 <ttx> especially since we all know that markmc always wins the argument in the end
21:20:34 <markmc> lifeless, seriously, pick a random API, turn it into a library ... see if anyone objects :)
21:20:44 <bnemec> Have we decided what to do with the pieces of Oslo that don't fit nicely into a library?
21:20:53 <dhellmann> bnemec: which are those?
21:21:06 <dhellmann> bnemec: maybe we should take that up after the meeting
21:21:10 <markmc> ttx, I dunno, lifeless does alright on that front :)
21:21:23 <lifeless> :P
21:21:24 <ttx> yes, anyone needing convincing can talk to dhellmann and markmc off-meeting, let's move on
21:21:28 <bnemec> I don't have any off the top of my head, but I recall from previous discussions that there were things that didn't fit nicely.
21:21:32 <bnemec> But +1 to after the meeting.
21:21:33 <lifeless> -> list
21:21:37 <ttx> #topic Nova PowerVM Driver Removal (mriedem)
21:21:38 <lifeless> 'how can we move it faster'
21:21:40 * jd__ agrees with ttx
21:21:41 <ttx> You again :)
21:21:56 <mriedem> so i don't have much else to say about the powervm driver removal
21:22:00 <ttx> So, on this one... I'm OK with a fast deprecation path if we don't screw any user in the process
21:22:06 <ttx> We got some reassuring comments from the User committee that this driver may not be in use
21:22:13 <mriedem> ttx: russellb's idea was just move it to stackforge
21:22:14 <russellb> no users have shown up, one possible future user
21:22:16 <ttx> But then we had this guy: http://lists.openstack.org/pipermail/openstack/2013-November/003370.html
21:22:16 <mriedem> which i think we're OK with
21:22:27 <ttx> that said, if he is in POC -> Prod mode, choosing the PowerVM driver at this point might not be the best option, if IBM loses interest in maintaining it
21:22:33 <mrodden> i am in favor of the stackforge move
21:22:50 <dims> +1 to stackforge move
21:22:51 <ttx> so we might just save him by encouraging him to not go on tat road
21:22:53 <mriedem> ttx: russellb: yeah, i think what the one user was saying is as long as it's freely available (stackforge), he's ok with it
21:23:05 <dhellmann> what is the point of keeping the code without any maintainers?
21:23:10 <russellb> yep ... but honestly, you guys have already said you're not interested
21:23:10 <ttx> mriedem: I think he said (ubuntu) but meh :)
21:23:13 <russellb> so i'm not sure why anyone would
21:23:36 <mriedem> russellb: not interested in what? stackforge?
21:23:43 <russellb> not interested in maintaining it long term
21:23:43 <ttx> mriedem: you should probably engage with that specific person and make sure he moves to powervc when ready
21:23:53 <russellb> so it seems like a dead end for someone to be looking to deploy in the next year
21:24:06 <russellb> and let's not confuse this with powervc
21:24:12 <russellb> none of this means we'd accept a powervc driver necessarily
21:24:12 <mriedem> ttx: yeah, that's what we plan on doing, there are some key people out to be in that discussion though this week (thanksgiving holiday)
21:24:37 <mriedem> russellb: right, that's understood
21:24:37 <russellb> we may or may not, but it would be evaluated based on its own merits
21:24:39 <russellb> ok
21:24:46 <dims> russellb, what's your thought here? delete from nova and not move to it to stackforge?
21:24:47 <russellb> it keeps getting brought up, and i really want to separate it
21:24:54 <mrodden> we will be maintain the powervm driver internally for some time until the powervc one is 'ready'ish
21:24:58 <russellb> dims: delete from nova, and then you guys (IBM) do what you want
21:24:59 <ttx> mriedem: so at this point I'm fine with fast deprecation + stackforge if there is some personal advice given to that one user who has mentioned wanting to use it
21:25:00 <russellb> but stackforge is an option
21:25:41 <mriedem> my only question about stackforge is if we can use the same gerrit and CI for running unit tests? i wasn't sure how that works.
21:25:47 <russellb> yes
21:25:50 <russellb> same infrastructure
21:25:54 <mriedem> ok, good to know
21:25:57 <russellb> so actions
21:25:58 <mriedem> as a fallback plan basically
21:26:00 <russellb> 1) approve the removal
21:26:01 <dims> russellb, cool let's stick with that then "Delete from Nova, Move to Stackforge"
21:26:12 <lifeless> so whats the difference between powervm and powervc
21:26:15 <russellb> 2) mriedem respond to the openstack post giving some recommendation to the one guy
21:26:20 <lifeless> is one free with the hardware and one commercial ?
21:26:21 <mriedem> lifeless: powervc is vcenter on power
21:26:28 <dhellmann> as a note, there is currently a hold on creating new stackforge projects because of a zuul issue
21:26:28 <russellb> 3) move to stackforge
21:26:34 <russellb> mriedem: um, but not vcenter at all right?
21:26:36 <dhellmann> so be aware of that when making scheduling plans
21:26:36 <russellb> it's your own thing
21:26:41 <ttx> russellb: +1
21:26:45 <mriedem> russellb: right
21:26:55 <lifeless> dhellmann: is that a short or long hold?
21:26:57 <mriedem> russellb: i agree with your action plan
21:27:08 <russellb> ok thanks
21:27:09 <dhellmann> lifeless: there's a bug they need to fix, so it depends on that
21:27:11 <russellb> i think we can move on then
21:27:28 <dhellmann> #link https://bugs.launchpad.net/openstack-ci/+bug/1242569
21:27:31 <uvirtbot> Launchpad bug 1242569 in openstack-ci "manage-projects error on new project creation" [Critical,Triaged]
21:27:43 <ttx> jeblair: do we have an ETA for the fix on this one ?
21:27:55 * ttx has a icehouse-1 blueprint blokced on it)
21:28:04 <jeblair> ttx: mordred is working on it
21:28:09 <jeblair> also, totally not a zuul issue :)
21:28:15 <dhellmann> ttx, no, yours went through because it's not a stackforge repo
21:28:17 <jeblair> it is a manage-projects issue
21:28:19 <russellb> [infra magic] issue
21:28:20 <mordred> yes. I will fix this in a few hours
21:28:23 <dhellmann> jeblair: sorry, didn't mean to mischaracterize it
21:28:40 <ttx> dhellmann: oh. nice. stops complaining
21:28:44 <mordred> the fix is well identified - I just need about an hour to actually do it
21:28:58 * dhellmann would give mordred and hour if he had one
21:29:09 <mordred> as soon as I'm done with these powerpoint slides...
21:29:27 <markmc> hehe
21:29:28 <ttx> ok, let's move on
21:29:35 <ttx> russell did summarize the plan quite well
21:29:50 <ttx> #topic flake8 (notmyname)
21:29:53 <notmyname> hi
21:29:58 <ttx> notmyname: you mentioned getting pushback that forces you to ignore a check ?
21:30:08 <ttx> but then you were using a phone keyboard
21:30:13 <notmyname> (maybe there has been process since)
21:30:18 <notmyname> *progress
21:30:49 <notmyname> so swift has a few cases where a bare except is needed. so we offered a patch to hacking to support #noqa on bare excepts
21:31:06 <jeblair> #link https://review.openstack.org/#/c/57334/
21:31:15 <lifeless> notmyname: thats surprising. Do you run on python2.4 ?
21:31:18 <notmyname> it got some pushpack (https://review.openstack.org/#/c/57334/) which caused us to simply ignore the check https://review.openstack.org/#/c/56712/
21:31:57 <jeblair> notmyname: i don't see pushback
21:32:09 <jeblair> notmyname: except for a suggestion that the commit message have an explanation
21:32:12 <jog0> notmyname: neither do I
21:32:25 <jog0> in fact this hasn't got approved yet just do to review backlog
21:32:26 <lifeless> notmyname: I realise thats a side issue, but bare except: only difference from except BaseException in that it catches string exceptions
21:32:38 <lifeless> notmyname: which are thorughly deprecated and dead upstream
21:32:52 <notmyname> like I said, seems perhaps there was movement yesterday that I hadn't seen
21:33:25 <notmyname> so perhaps a non issue today (not that I have more info than when I was on my phone with ttx this morning)
21:33:27 <jog0> so no feedback in a few days is pushback?
21:33:31 <dolphm> notmyname: is there a discussion somewhere regarding the need for bare excepts?
21:33:51 <lifeless> notmyname: on a purely technical basis I'm fascinated :)
21:33:58 * dhellmann wonders why each hacking rule has to look for noqa independently
21:34:07 <notmyname> dhellmann: great question :-)
21:34:38 <dolphm> notmyname: (just found clay's comment on Nov 19 https://review.openstack.org/#/c/57334/ citing PEP8; reading)
21:34:42 <notmyname> there was other discussions in IRC earlier in the week that caused us to believe the hacking patch wouldn't land.
21:34:46 <notmyname> were*
21:34:50 <ttx> notmyname: I think that patch is not blocked, so you should be able to remove your workaround
21:34:59 <notmyname> ttx: then we can move on
21:35:01 <ttx> (when it lands)
21:35:08 <mordred> dhellmann: each hacking rule parses the line using a few different methods - and there is not a global line parser for a line
21:35:12 <notmyname> I don't want to be blocked on that
21:35:40 <notmyname> and simply wanted to raise it here to ensure it goes (based on IRC conversations more than review comments)
21:35:47 <ttx> ok, looks like this is a non-issue, next topic
21:35:58 <ttx> #topic Proper tracking for non-deterministic bugs affecting gate
21:36:04 <ttx> That one should be more bikeshedding
21:36:18 <ttx> lifeless: want to introduce this one ?
21:36:26 <lifeless> oh yay
21:36:34 <russellb> btw, cross project topic for the queue ... checking in on neutron vs nova-network status
21:36:39 <lifeless> so 40 minutes ago we were gathered here
21:36:47 <lifeless> and talking about how to get gate bugs attention
21:36:59 <jeblair> ttx: NICE time estimate, btw!
21:36:59 <lifeless> and the various tradeoffs between using the well known hammer of critical
21:37:08 <ttx> jeblair: I'm a pro.
21:37:18 <lifeless> (actually 38, but 40 was a h/t to ttx)
21:37:23 <lifeless> anyhoo
21:37:28 <lifeless> critical
21:37:43 <lifeless> or some gate-thing-specific search (e.g. /rechecks, a canned bug search - e.g. tag, etc etc)
21:37:44 <ttx> lifeless: I thought notmyname's topic would create more flames.
21:37:46 <lifeless> go to it
21:37:55 <lifeless> ttx: well if my technical question was answered, maybe:P
21:38:08 <markmc> how about we start by figuring out how to decide which non-deterministic failures are really critical?
21:38:21 <lifeless> lets not use the word critical - overloaded.
21:38:24 <ttx> markmc: I think sdague's point is that they all are
21:38:30 <lifeless> Lets say - things we want 24 hour fast focus on
21:38:33 <ttx> markmc: as they just add up
21:38:35 <sdague> markmc: so up until this point it's been the judgement of the group of us that flag it
21:38:36 <lifeless> and things we want ahead of other bug fixes
21:38:47 <lifeless> and things we want in the general mix of bugfixes
21:38:56 <lifeless> because those are I think the three effective categories we have
21:39:11 <lifeless> 'drop everything', 'front of the regular queue', 'whenever'
21:39:27 <sdague> and I guess the question is if we're going to continue with a small group of folks with that power (either explicitly or by cultural construct), do we trust those folks to make good judgements there
21:39:27 <markmc> they sound like they should be marked as critical, yes :)
21:39:46 <markmc> however, if we have say 30 non-deterministic bugs right now ... they don't all qualify under that criteria
21:39:48 <ttx> lifeless: some busy projects only have two ('drop everything' and 'whenever')
21:40:01 <dolphm> are we also proposing that 'critical' bug fixes get priority in the gate queue?
21:40:02 <sdague> because the reality is, until the race shows up a few times it doesn't pop up
21:40:06 <ttx> but that's arguably a failure
21:40:20 <lifeless> ttx: degenerate case of front of the queue ;P
21:40:21 <sdague> markmc: so here's the thing, it's not 2 giant races
21:40:30 <russellb> dolphm: have to be careful with doing that kind of thing, if it's automatic it'll get abused :(
21:40:31 <lifeless> from collections import set as deque
21:40:31 <sdague> it's actually 30 little races that all interact
21:40:37 <markmc> sdague, "trust to make good judgement" does not start with "all non-deterministic bugs, no matter how rare, are automatically Critcial" IMHO :)
21:40:44 <russellb> markmc: big +1
21:40:51 <dolphm> russellb: understood, that's why i'm asking! might affect my opinion on the larger issue
21:41:12 <lifeless> so perhaps, if its the set of things that is the problem
21:41:22 <sdague> markmc: ok, well I've been staring at this one for a while. And the problem is we're actually getting killed by a thousand paper cuts.
21:41:25 <lifeless> we can talk about the criticality of making that set be below some threshold
21:41:44 <sdague> however, it will take some more time to be able to easily visualize that fact
21:41:53 <lifeless> and if we agree about that, then we can say - whenever it's over that threshold, *all the members* count as critical until it's below the threshold.
21:41:54 <sdague> which is basically my top dev priority at this point
21:41:58 <markmc> sdague, making them a thousand Critical bugs won't fix it :)
21:42:16 <dolphm> markmc: ++
21:42:16 <lifeless> markmc: what do you think will fix it?
21:42:19 <ttx> lifeless: I think we could use Tag + High/Critical (as a way to prioritize amongst them)
21:42:28 <dhellmann> how many of these things do we actually have open right now?
21:42:36 <lifeless> markmc: also it's 100 bugs spread over a dozen code bases.
21:42:36 <russellb> i think we should focus on the goal ... making people want to work on this
21:42:38 <sdague> markmc: well let me reverse it. Do you think that what myself, jog0, and clarkb have done so far has made things better or worse?
21:42:38 <ttx> i.e. if all critical, it might be hard to prioritize which one should be fixed first
21:42:39 <markmc> lifeless, the kind of work and awareness raising going on this week
21:42:43 <lifeless> markmc: thats what - 10 per code base.
21:42:47 <jeblair> for those that haven't seen it, i imagine these bugs would be a subset of the bugs here: http://status.openstack.org/elastic-recheck/
21:42:52 <jeblair> i don't think it's 100 bugs...
21:42:53 <lifeless> So lets not extrapolate out to infinity on the data we have
21:42:54 <jog0> right now we are tracking 9 gate bugs with e-r http://status.openstack.org/elastic-recheck/
21:42:55 <dolphm> not all bugs on /rechecks/ are actually transient issues at all
21:43:07 <lifeless> jog0: we have 110 tracked rechecks today, if I counted right
21:43:09 <russellb> what gets more people interested in helping with these things?
21:43:11 <dolphm> (maybe they are at the moment, but it's not always true)
21:43:16 <sdague> dolphm: these aren't trusting the user provided bugs
21:43:19 <jog0> lifeless: recehck vs elastic-recheck
21:43:20 <dhellmann> jeblair: I count about 18
21:43:21 <jgriffith> TBF it's not just 3 or 4 individuals making things better
21:43:23 <jeblair> lifeless: the rechecks page has a lot of garbage data
21:43:27 <lifeless> jeblair: ok!
21:43:34 <jgriffith> and it's not just 3 or 4 that are going to address it
21:43:35 <markwash> if they are critical, is there a good way to get help understanding and duplicating the bugs? I often cannot make heads or tails of the bug reports for things in the rechecks list
21:43:35 <lifeless> so elastic-recheck is our source of truth?
21:43:37 <jgriffith> it's a scaling problem
21:43:39 <sdague> this is elastic recheck
21:43:39 <jeblair> dhellmann: and at looks like 5 of them are ready to be removed
21:43:46 <jgriffith> tags, priorities etc aside
21:43:48 <dhellmann> jeblair: progress!
21:43:58 <markmc> sdague, oh, far better ... totally - my input is basically that bumping a big number of bugs to Critical will hurt your progress
21:43:58 <sdague> which means that a set of us have found a fingerprint, and found that it shows up more than once
21:44:07 <dhellmann> markwash: +1
21:44:12 <jeblair> so that puts us at about a dozen or so on the e-r page, but if jog0  says 9 i believe him :)
21:44:45 <jeblair> lifeless: for this conversation, i think so.  elastic-recheck is made up of bugs that have gone through human review and we know are gate-affecting
21:45:24 <russellb> is there a good dashboard of the e-r bugs?
21:45:27 <jeblair> lifeless: /recheck is sadly full of people rechecking things against nonsense bugs.  it will eventually be replaced with the e-r page
21:45:32 <russellb> nm i see the URL now
21:45:33 <sdague> markmc: ok, fair. :) But we are talking about a finite set here - http://status.openstack.org/elastic-recheck/
21:45:49 <jeblair> russellb: that's sdague's big project for the next bit, to improve http://status.openstack.org/elastic-recheck/ and make it more dashboardy
21:45:50 <dolphm> sdague: cool! this is new to me
21:45:54 <jd__> so problem number one is people, noted. :)
21:46:06 <mriedem> sdague: jog0: the elastic-recheck page is only for those that are in the yaml query file though right?
21:46:08 <russellb> i think the improved dashboard will make a *much* bigger impact than changing bug priorities
21:46:18 <sdague> russellb: I 100% agree
21:46:21 <russellb> k :)
21:46:35 <sdague> unfortunately, had these other things take the last two weeks of my time
21:46:46 <sdague> so diving back in right after thanksgiving ont hat
21:46:49 <jeblair> russellb: here's the problem i want to solve -- we are able to fix these bugs only when we get people aware of them and working on them.  i'm trying to find the best process to do that.
21:46:50 <russellb> there was just a whole lot of focus on bug priority ...
21:47:13 <russellb> jeblair: a good dashboard, and people regularly reporting status to the -dev list, and in meetings
21:47:14 <russellb> IMO
21:47:18 <mriedem> jeblair: i think it might help to show people how easy it is to get a bug into the e-r query.yaml file
21:47:37 <mriedem> jeblair: once the bug is in the query and e-r is checking and reporting on it, you should stop seeing so many 'recheck no bug'
21:47:43 <russellb> like jog0's report recently ... would love to see that regularly
21:47:44 <mriedem> and it's pretty easy to push patches to add queries to -er
21:47:47 <russellb> here are the current issues and their status
21:47:47 <dhellmann> russellb: I think the point is giving devs a whole other place to look when deciding what to work on might be less effective than just setting the priority so they show up at the top of the list in the usual place
21:47:50 <sdague> russellb: agreed, honestly, the sequence would have been better to hit this first
21:47:51 <russellb> and which ones need more attention
21:47:57 <sdague> russellb: yes, that's completely the intention
21:47:57 <jeblair> russellb: i'm not sure we should wait for a good dashboard.  i anticipate that's a while out yet, and it depends on people incorporating 'wake up and check the dashboard' into their workflow
21:48:03 <lifeless> dhellmann: +1
21:48:08 <lifeless> dhellmann: thats precisely my point
21:48:24 <ttx> frankly, I think we are getting better at this, not worse.
21:48:24 <lifeless> folk are bad at pulling together widely disparate information sources in their inner loop of 'what to work on next'
21:48:25 <russellb> a weekly email then
21:48:30 <dolphm> could elastic-recheck leave comments on bugs as a starting point?
21:48:43 <salv-orlando> I think it already does?
21:48:44 <russellb> or more frequent when there are severe issues
21:48:44 <sdague> dolphm: it already does
21:48:44 <markmc> sdague, cool stuff - that page looks far more useful to people want to help with this than bug priorities
21:48:47 <jog0> russellb: we plan on it
21:48:50 <dhellmann> I like the weekly email and I like a tag ("gate"?) and I like setting the priority
21:48:51 <russellb> ok all good stuff
21:48:54 <ttx> the recent events have been quite well communicated imho
21:49:01 <dolphm> i guess i'm lucky in that i haven't experience that yet lol
21:49:02 <russellb> ttx: yep
21:49:17 <markwash> critical isn't quite "where I look for stuff to work on" its more like "oh crap what is ttx gonna yell at me about?"
21:49:20 <jeblair> markmc: thank you.
21:49:24 <sdague> dolphm: yeh, honestly, keystone isn't really subject to this much by not having an rpc bus
21:49:25 <dhellmann> markwash: +1
21:49:35 <lifeless> markmc: I don't understand why you (seem to) think bug priorities aren't helpful
21:49:49 <lifeless> markmc: but we can sidebar that if you like
21:49:52 <jog0> so the recent gate issues have shown that we were just two bugs away from a massive gate wedge -- something we never want to have again
21:49:53 <ttx> markwash: that's one benefit/problem of using "critical" for that. That reuses my nagging
21:50:01 <russellb> i think markmc's point is to be careful not to dilute the value of bug priorities
21:50:04 <ttx> nagging-as-a-service
21:50:10 <russellb> marking a ton of stuff critical is counter-productive
21:50:11 <sdague> so, honestly, I'm completely ok with the resolution of "go make the dashboard better first" and if it's not working talk about bug priorities later
21:50:14 <jeblair> and we've only been able to fix these issues by getting people on it.
21:50:20 <russellb> so just want some reasonable line
21:50:21 <lifeless> so there are 18 such bugs
21:50:23 <russellb> and then i'd be Ok with it
21:50:25 <markmc> lifeless, just that marking obviously not-obviously-critical bugs as critical will be (yeah, as russellb says) counter-productive
21:50:34 <jeblair> sdague: i don't see how getting a better dashboard will get people to commit to fixing bugs
21:50:42 <lifeless> maybe 14 once fixed ones are gc'd
21:50:44 <ttx> how about we use tag + high / critical depending on how urgent the fix is ?
21:50:45 <markwash> ttx: lol :-)
21:50:47 <russellb> i don't see how marking bugs will either on its own
21:50:56 <lifeless> markmc: russellb: ok, understood on dilution.
21:50:58 <jeblair> sdague: right now, sdague and jog0 go and bother people until someone starts working on it
21:51:01 <ttx> (that's already what we do, btw)
21:51:08 <jeblair> there has to be a better way to communicate
21:51:15 <markmc> sdague, easy to use the dashboard as data to ask bug triage teams to bump the priority of bugs
21:51:22 <lifeless> so the bug database is our shared understanding of defects
21:51:26 <russellb> i think we'll always need people making this a higher priority on their list to look afer
21:51:27 <sdague> I guess here's the question - what is the right and consistent signaling mechanism to the PTLs to get people to look at bugs
21:51:29 <dolphm> ttx: and i think that's a sane approach!
21:51:34 <russellb> and then report status to the rest of the community
21:51:35 <lifeless> my point is that storing stuff elsewhere is a bit crazy
21:51:46 <jeblair> sdague: exactly
21:51:47 <russellb> sdague: more regular reports
21:51:49 <russellb> list posts
21:51:50 <russellb> meeting topics
21:51:51 <lifeless> because it dminishes the authority of the shared database of bugs.
21:51:57 <ttx> sdague: how about raising them at this meeting weekly ?
21:52:01 <sdague> and the answer can't involve me or jog0 having to ping people regularly, because it doesn't scale
21:52:07 <lifeless> how about each project has an item to touch on them ?
21:52:12 <russellb> need to scale a team that makes this a priority to look after
21:52:13 <jeblair> russellb: these can't wait a week.  they need people working on them quickly.
21:52:25 <russellb> list post for the critical ones
21:52:38 <sdague> russellb: so I disagree, I think we need folks from the core teams that make it a priority
21:52:39 <markmc> mark critical ones as critical :)
21:52:42 <russellb> someone needs to take ownership over tracking it and pushing toward the solution, sort of like a VMT member does for a vulnerability
21:52:43 <ttx> sdague: (if setting them High/Critical didn't trigger the right response)
21:52:45 <lifeless> jeblair: all of them? AIUI there are 'zomg this alone is a killer' + 'we have a background level of noise that is a real problem'
21:52:46 <stevebaker> lifeless: a sharp pointy item to touch on them? ;)
21:52:49 <mriedem> is it fair to say that all bugs on http://status.openstack.org/rechecks/ should have checks on http://status.openstack.org/elastic-recheck/ ?
21:53:09 <sdague> mriedem: /rechecks/ is kind of broken right now
21:53:16 <lifeless> jeblair: It seems to me that 'zomg X' is drop-everything, and handled already, and 'background noise' is what we're trying to solve here
21:53:17 <ttx> OK, we ahve a couple of other points to touch in this meeting. There is a calm thread on the ML, please hit it now
21:53:33 <jeblair> lifeless: a lot of them, certainly.  i mostly want to indicate that many can't wait a week.
21:53:35 <russellb> wish i had time for all the threads i'm interested in :(
21:53:36 <mriedem> sdague: ok, point is, i think people have gotten used to finding a bug or opening one to recheck against, but not to adding e-r queries on them
21:53:38 <lifeless> jeblair: ack
21:53:39 <russellb> tough to keep up
21:53:58 <ttx> moving on
21:54:01 <sdague> mriedem: yes, that's true, we'll get better at this.
21:54:03 <mriedem> if we have e-r queries on these things, they show up in patches and the e-r page
21:54:08 <russellb> so each project could have a gate czar that is making this a priority?
21:54:08 <ttx> #topic Red Flag District / Blocked blueprints
21:54:14 <sdague> russellb: +1
21:54:20 <russellb> what a naughty topic this is.
21:54:26 <ttx> We have no blocked blueprints that I know about
21:54:28 <ttx> Any blocked work that this meeting could help unblock ?
21:54:32 <russellb> (is what i think when ttx says red flag district)
21:54:59 <russellb> hrm, could consider deprecating nova-network as a blocked thing ... but that could be deserving of its own meeting topic each week
21:55:40 <ttx> russellb, salv-orlando: I haven't looked carefully into the neutron roadmap, do we have anything there to hope for icehouse ?
21:56:09 <russellb> summary of blockers: feature parity (there are icehouse plans, but then again, there were havana plans, and grizzly plans ...)
21:56:20 <russellb> and CI / general quality parity
21:56:27 <russellb> not sure on the progress there
21:56:49 <salv-orlando> on the deprecation issues, that goes far beyond my reach. On the specific parity items
21:56:54 <ttx> neutron suffers a bit from too much tactical, not enough strategic contributions yet
21:57:14 <russellb> and to be clear, i've started saying that i'd like to re-evaluate nova-network's status after icehouse-2
21:57:18 <ttx> and the very few that do strategic contributions are totally overwhelmed
21:57:23 <russellb> depending on progress, i may propose un-freezing nova-network
21:57:29 <ttx> and get a lot of blame
21:57:33 <russellb> ttx: yep ...
21:57:46 <salv-orlando> ttx: don't understand what you mean by tactical and strategic. Can you explain?
21:57:48 <ttx> so that do not encourage other people to go all in
21:58:21 <ttx> salv-orlando: tactical = feature in a vendor plugin
21:58:32 <ttx> strategic = test coverage, feature gap
21:58:35 <russellb> strategic = anything beneficial to the project overall
21:59:18 <salv-orlando> cool I get it - I think we're all aware of this issue. Thankfully the strategic contributor team is gathering new members
21:59:32 <ttx> oldie but still relevant: http://fnords.wordpress.com/2011/09/28/the-next-step-for-openstack/
21:59:51 <ttx> #topic Open discussion
22:00:07 <ttx> markwash: wanted to quickly raise the client lib thread ?
22:00:12 <markwash> #link http://lists.openstack.org/pipermail/openstack-dev/2013-November/019911.html
22:00:15 <ttx> sorry not much time left
22:00:16 <russellb> so no updates on that?  :(
22:00:29 <markwash> just wanted to mention, we've got some rough consensus approaching there I think
22:00:37 <markwash> in case folks want to (struggle to) review the thread
22:00:50 <ttx> russellb: we might want to make it a full topic at another meeting
22:01:01 <markwash> I can summarize what it seems to me later today
22:01:06 <ttx> russellb: although markmcclain might be traveling a lot on next Tuesdays
22:01:19 <russellb> ok
22:01:33 <ttx> markwash: yeah, maybe summarize what you intend to do basd on the outcome of that thread... that might trigger new answers
22:01:41 <markwash> ttx: deal
22:01:52 <ttx> markwash: "WAT! defintely DONT do that"
22:01:57 <markwash> yeah
22:01:58 <markwash> haha
22:02:14 <ttx> And we are out of time
22:02:16 <dolphm> regarding QA team having triaging abilities on LP, is there someway PTL's can opt into that short of making all those people *-core?
22:02:20 <dolphm> boo
22:02:37 <jeblair> open bug teams
22:02:37 <ttx> #endmeeting