21:02:48 <ttx> #startmeeting project
21:02:49 <openstack> Meeting started Tue Jan 28 21:02:48 2014 UTC and is due to finish in 60 minutes.  The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:53 <openstack> The meeting name has been set to 'project'
21:02:55 <ttx> #link http://wiki.openstack.org/Meetings/ProjectMeeting
21:03:00 <stevebaker> \o
21:03:02 <ttx> #topic icehouse-2 / 1.12.0 postmortem
21:03:19 <ttx> So... last weeks gate issues created delays for Swift 1.12.0 and complicated the delivery of a sane icehouse-2 milestone
21:03:32 <ttx> In particular we shipped heat with a milestone-critical issue, because (1) that issue wasn't really tested in gate yet so it ended up in master
21:03:37 <ttx> and (2) milestone-critical issues do not get fast-tracked at the gate
21:03:38 * russellb dreams of synchronized releases
21:03:50 <ttx> stevebaker: do we have a bug to track the absence of integration tests in that specific area ?
21:04:36 <stevebaker> ttx: the tests exist, I need to raise a bug to enable vm -> heat on port 8000 in the gate infra
21:04:57 <ttx> stevebaker: ok, when you have bug number, let me know
21:05:03 <ttx> would like to tie loose ends
21:05:03 <stevebaker> something like this, but which works https://review.openstack.org/#/c/69276/ <-- advice welcome
21:05:19 <ttx> I think that we'll have the same issues again if we can't keep the top of gate entry under 12 hours of age
21:05:27 <ttx> So we'll see how good we are at keeping it below that
21:05:53 <ttx> but otherwise we may need to relax gate-jumping rules to include release-critical issues
21:06:18 <ttx> I think I can deal with 12 hours
21:06:39 <ttx> but 27 or 34... difficult to stick to release dates then
21:06:50 <ttx> icehouse-3 should be a nice test :)
21:07:05 <hub_cap> heh
21:07:19 <russellb> still a lot of work to do to make icehouse-3 not blow up
21:07:22 <ttx> any other post-mortem thoughts on i2 / 1.12.0 ?
21:07:28 <ttx> russellb: yes, next topic
21:07:29 <russellb> but i think the steps to get there are clear (enough)
21:07:49 <russellb> holidays really hurt i2 velocity, too, i think
21:07:55 <russellb> but i guess we can't cancel those
21:08:02 <ttx> russellb: yeah, not sure that would fly
21:08:09 <ttx> #topic master gate status
21:08:19 <ttx> Things definitely improved over the last 7 days
21:08:25 <russellb> yeah, lots has changed
21:08:29 <russellb> lots and lots of good bug fixes
21:08:32 <ttx> As far as I can tell the recent improvement is not due to implementation of Sean's suggestions yet
21:08:35 <russellb> from a bunch of people
21:08:37 <ttx> (from http://lists.openstack.org/pipermail/openstack-dev/2014-January/025140.html )
21:08:40 <russellb> ttx: right htat isn't done yet
21:08:44 <ttx> So we still have room for improvement
21:08:45 <sdague> ttx: correct, we're not there yet
21:08:49 <ttx> which is good news
21:08:51 <russellb> it's mainly bug fixes, improvements to zuul, and increased node capacity
21:09:08 <sdague> yep, lots of good bug fixes from people, increased capacity, and the sliding window on the gate queue all helped
21:09:15 <ttx> sdague: notmyname wanted to ask about progress being made on reducing the overlapping tests -- i suspect that effort is not started yet ?
21:09:38 <sdague> ttx: right, the first step is the zuul logic to handle requiring recent good check
21:09:46 <ttx> ack
21:09:48 <stevebaker> how did tempest day go yesterday?
21:09:50 <sdague> which jeblair has mostly ready, but it tickled a gerrit bug last night
21:10:02 <ttx> stevebaker: pretty good I think. sdague may have more data
21:10:11 <russellb> gate bug day was productive i think
21:10:18 <sdague> stevebaker: pretty well I think, we're now about 95% on categorization of issues
21:10:28 <russellb> link: http://status.openstack.org/elastic-recheck/data/uncategorized.html
21:10:30 <ttx> at the very least it familiarized myself with some of the e-r tooling
21:10:38 <sdague> and we made some good progress on some of the top ones around cinder and neutron
21:10:39 <russellb> i think that's the biggest concrete achievement, better categorization
21:10:54 <sdague> ttx: yeh, I think we got a lot more people familiar with that, which is really good
21:11:01 <russellb> patch in progress related to top neutron failure: https://review.openstack.org/#/c/69445/
21:11:10 <russellb> a patch merged that squashed the top cinder bug (thanks jgriffith!)
21:11:32 <ttx> hah
21:11:43 <ttx> that was my next point
21:11:48 <sdague> russellb: well we might be declaring victory too fast on that one :)
21:11:58 <russellb> sdague: not victory, just progress :)
21:12:01 <sdague> heh
21:12:02 <ttx> sounds a pomising workaround at least
21:12:05 <ttx> promising
21:12:24 <ttx> Good to see a number of people working on it, including on Ubuntu's side
21:12:37 <sdague> yeh our throughput on friday & sat was over 100 patches merged / 24 hrs
21:12:52 <sdague> just to give a sense that we're kind of back to business in the master gate
21:13:26 <russellb> but let's not get comfortable
21:13:32 <sdague> agreed
21:13:32 <ttx> jamespage told me they might have narrowed it on their side too
21:13:34 <russellb> i still feel like the list of active bugs could use more eyes
21:13:41 <sdague> yep, definitely
21:13:43 <russellb> ttx: very good to hear.
21:14:13 <sdague> 62 bugs being tracked by elastic recheck now - http://status.openstack.org/elastic-recheck/
21:14:26 <ttx> the gate queue currently climbs but not to stratospheric heights
21:14:57 <ttx> so it seems we are back at pre-crisis levels
21:15:11 <ttx> (but then, neutron is not really gating those days)
21:15:15 <russellb> we have a huge nova patch series we're trying to merge for nova-network performance
21:15:24 <russellb> once that's all in, i'd like to try to increase tempest concurrency again
21:15:29 <russellb> which should be a big speedup on test runtime
21:15:37 <ttx> ack
21:15:46 <ttx> anything else on that topic ?
21:15:47 <russellb> most of that is in the gate now
21:15:57 <sdague> oh, we also need to try to get the better image on rax perf nodes
21:16:09 <russellb> sdague: what's the better image?
21:16:11 <sdague> the current image is part of the slowdown
21:16:24 <sdague> russellb: one with paravirt drivers configured
21:16:28 <russellb> ah
21:16:31 <sdague> jnoller is helping on that
21:16:41 <russellb> cool, just a tweak to nodepool configs right?
21:16:44 <sdague> yes
21:16:50 <sdague> need to make sure it works reliably first
21:16:54 <russellb> pfft
21:17:19 <sdague> ttx: done on this topic I think
21:17:24 <ttx> #topic Code proposal deadline (russellb)
21:17:35 <russellb> yeah, so we had one of these deadlines last cycle
21:17:35 <ttx> russellb: how is that proposal doing so far ?
21:17:42 <russellb> 5 projects had a deadline, across 3 dates
21:17:55 <russellb> i'm proposing it again for nova, but wanted to see if others wanted to coordinate on a single date
21:17:58 <russellb> to make our schedule less confusing
21:18:06 <russellb> i'm proposing 2 weeks ahead of feature freeze, so feb 18
21:18:23 <markmcclain> We're planning to use Feb 18th too
21:18:24 <dhellmann> we should probably just build this into the schedule when we plan it at the next summit
21:18:24 <russellb> proposal on ML has seen some feedback ... acked by markmcclain and hub_cap
21:18:25 <ttx> yes, that can be opt-in but a single date would be less confusing
21:18:41 <ttx> jd__ told me he would not follow it for ceilometer
21:18:50 <ttx> he has a good grip on incoming proposals
21:18:51 <russellb> opt-in seems fine
21:19:06 <russellb> it's a bigger deal when you're overwhelmed by the incoming wave
21:19:07 <sdague> yeh, honestly spreading out the freezes also probably helps on gate load
21:19:23 <sdague> so a few projects going later is good and fine
21:19:23 <jgriffith> sdague: true dat
21:19:24 <russellb> sdague: well, i thought about that, but remember, this isn't *merge* deadline
21:19:26 <russellb> so it's not that big of a deal
21:19:27 <jd__> (not that I would't be against it if we wanted to do that for _all_ projects though)
21:19:35 <jd__> s/not/note/
21:19:43 <sdague> russellb: true
21:19:46 <jgriffith> russellb: ahhh... exccellent point
21:19:55 <ttx> I think the key thing is to avoid having a different date for each project
21:19:57 <russellb> it'll be a big rush on check
21:20:06 <russellb> which may be a nice warmup for feature freeze :)
21:20:19 <russellb> ttx: right, that's what i was hoping
21:20:27 <ttx> So it's opt-in, on Feb 18. I'll document it on the release schedule
21:20:28 <russellb> i like having a nice coordinated schedule
21:20:32 <russellb> perfect
21:20:50 <ttx> #action ttx to document FPF on Feb 18 on icehouse schedule
21:21:04 <ttx> other comments on that ?
21:21:58 <ttx> #topic Logging standards (sdague)
21:22:05 <ttx> sdague: floor is yours
21:22:09 <sdague> great
21:22:26 <sdague> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/025542.html
21:22:59 <sdague> this is actually an idea I floated early in the cycle, but I'd like to revisit to see if we could get a few things sorted for Icehouse
21:23:28 <sdague> in staring at logs reading test fails, we've definitely got a bunch of inconsistency challenges, in single projects and across them
21:23:51 <sdague> so my current thinking is for icehouse if we can get some basic guidelines for the INFO log level
21:23:55 * ttx wonders how sdague manages to have so many fishing lines at the same time in the ocean
21:24:23 <sdague> and see which projects want to buy in, we could do a bunch in terms of overall debugability of OpenStack, for ourselves, and operators
21:24:56 <lifeless> logging standards ++
21:24:59 <sdague> so mostly this is socialization, to figure out which projects want to take part, and if anyone objects to the current list of guidelines I put up there
21:25:14 <russellb> i think it's a sane idea
21:25:15 <sdague> I expect this is going to be a multi cycle effort
21:25:19 <lifeless> I'd really like to move to something like kafka and make all our logs machine data rather than human
21:25:26 <lifeless> but *thats* a different idea :)
21:25:33 <ttx> sdague: I forwarded the effort intro to reed, as he was looking for areas where beginners could help. Those commits do not require that much deep knowledge of stuff
21:25:34 <sdague> but I think INFO sanity is doable in icehouse
21:25:35 <russellb> question like many things, how does it stack up against all the other things we need to be doing
21:25:51 <ttx> sdague: as long as the base principles are well documented
21:26:00 <russellb> if there are volunteers, sure, i'm fine with it ...
21:26:13 <sdague> ttx: thanks, yeh this is actually a good place to bring in volunteers
21:26:54 <sdague> russellb: yeh, I think we'll mostly be pulling in newer folks on this, it will impact review load on core teams, which is part of trying to get some buy in on "standards" up front, to hopefully make the reviews easy
21:27:52 <sdague> from my interacting with some of the larger operators, this is a pretty high impact to them, because dealing with our logs today is .... *interesting*
21:27:55 <ttx> do youhave a link to the proposed guidelines ?
21:28:08 <sdague> #link https://wiki.openstack.org/wiki/LoggingStandards
21:28:14 <russellb> sdague: review load at this point in the cycle is quite painful
21:28:14 <dolphm_afk> is there anyway we can realistically write gate tests against the guidelines?
21:28:17 <ttx> cool
21:28:19 <russellb> we're already incredibly overloaded
21:28:33 <sdague> russellb: agreed.
21:28:46 <sdague> I just cringe on the current state of things
21:29:17 <russellb> i cringe at a lot of things
21:29:20 <sdague> :)
21:29:22 <IanGovett1> I haven't looked over the logging standards, but wonder if the logging standards enable the possibility of doing some programmatic analysis of logs when trying to do post mortems ?
21:29:34 <russellb> do i cringe at our log formatting more than the number of other blueprints already ready for review?  probably not
21:29:36 <ttx> "Lifecycle event 1 on VM b1b8e5c7-12f0-4092-84f6-297fe7642070"
21:29:38 <sdague> I think we're a long way away from thsoe things
21:29:39 <ttx> nice
21:29:49 <hub_cap> IanGovett1: i think lifeless mentioned that (but likely for another conversation)
21:29:57 <sdague> we're going to take this one step at a time
21:30:05 <hub_cap> ++
21:30:18 <jgriffith> sdague: gotta start somewhere
21:30:23 * russellb nods
21:30:37 <devananda> Ironic doesn't have, IMHO, enough INFO logged today. So I'm ++ to having a common standard to point volunteers to
21:30:44 <devananda> and taht's a good place to start
21:30:45 <ttx> sdague: not sure we can fully nail the INFO by the icehouse release, but having convergence on the standards and a model project would be nice
21:30:54 <sdague> ttx: sure
21:31:09 <sdague> honestly if every project logged wsgi requests exactly once at INFO
21:31:12 <ttx> sdague: then we can use it as model to encourage people to converge INFO in J
21:31:16 <sdague> we'd be a huge step forward
21:31:22 <sdague> because... we very much don't today
21:31:53 <sdague> ttx: agreed, I'm hoping to tackle ERROR in J as well
21:31:59 <ttx> yep. I used to complain at Eucalyptus logs but ours don't really look better those days.
21:32:28 <sdague> anyway, we can do most of this on the list I think, but it made sense to open it up here as well
21:32:40 <sdague> especially if anyone has comments on the wiki page so far
21:32:58 <ttx> so in summary... good idea, can't do that much with icehouse-" review load but at least defien standards and push what we can ?
21:33:04 <ttx> icehouse-3
21:33:21 <sdague> ttx: I think that's fair
21:33:24 <russellb> +1
21:33:36 <SergeyLukjanov> fwiw +1, it'll be a good start
21:34:05 <russellb> would hate to see complex blueprint patches go into conflict over log message shuffling :-/
21:34:11 <ttx> fwiw current standards sound sane to me
21:34:13 <russellb> so yeah, would rather mass patches show up early juno
21:34:31 <ttx> sdague: we could also point -operators to that wiki page for feedback
21:34:58 <sdague> ttx: sure, or get them on the dev thread. We discourage cross posting.
21:35:03 <ttx> sounds like an area where they could be interested in participating
21:35:10 <sdague> a few have already
21:35:32 <ttx> yeah, just a pointer on -operators to make them aware of the -dev thread, no cross-posting
21:35:41 <sdague> russellb: realistically, I think the # of patches to handle info will be relatively low
21:36:15 <ttx> sdague: anything else on that topic ?
21:36:27 <sdague> nope
21:36:33 <IanGovett1> Sorry for what may seem a dumb question (I'm a newbie),  but do the log messages contain a timestamp in 'cut' time so that log events across multiple servers can construct a sequence of events.
21:37:37 <ttx> IanGovett1: they have timestamps yes, and if you use synced clocks you should be fine enough
21:37:49 <ttx> #topic Red Flag District / Blocked blueprints
21:37:49 <IanGovett1> ok.  thanks
21:37:55 <russellb> if you don't have synced clocks, more is broken than logs
21:37:58 <ttx> Any inter-project blocked work that this meeting could help unblock ?
21:38:22 <ttx> in particular, I'm interested by critical work that depends on some other project completing their stuff
21:38:40 <ttx> Like Horizon waiting for a feature to land to make a panel about it available
21:38:57 <jd__> we would need more eyes oslo.messaging review because that's blocking several bp of ceilometer FWIW
21:39:14 <ttx> jd__: link to review ?
21:39:18 <jd__> I'm feeling like stating the obvious, but well. :)
21:39:34 <ttx> did you get the oslo.messaging peeps on it already ?
21:39:35 <jd__> https://review.openstack.org/#/q/status:open+project:openstack/oslo.messaging+branch:master+topic:bp/notification-subscriber-server,n,z    mainly
21:39:49 <dhellmann> jd__: today's my review day so I'll try to get to them this afternoon
21:39:54 <jd__> cool
21:40:06 <jd__> just saying' really, I know markmc is also trying is best to take a look
21:40:29 <dhellmann> jd__: there are a few in that series with -1 comments already
21:40:49 <ttx> jd__: that BP is high, so I pester dhellmann with it regularly
21:40:52 <jd__> sileht: ^
21:41:00 <jd__> hehe :)
21:41:20 <ttx> jd__: thx for the pointer though, that's what I want to hear in that section of the meeting
21:41:40 <ttx> prefer to hear about those early, rather than too late
21:42:13 <ttx> anything else with inter-project dependency that you'd like to make sure is prioritized correctly ?
21:42:58 <ttx> I guess not
21:43:05 <ttx> #topic Incubated projects
21:43:12 <SergeyLukjanov> o/
21:43:12 <devananda> o/
21:43:15 <ttx> hi guys
21:43:22 <ttx> https://launchpad.net/savanna/+milestone/icehouse-3
21:43:48 <ttx> doesn't look too bad -- you might want to have assignees on all of those
21:44:06 <SergeyLukjanov> ttx, yup, working on it
21:44:06 <ttx> https://launchpad.net/ironic/+milestone/icehouse-3
21:44:24 <ttx> devananda: I suspect those "Unknwon" are actually "Not started", right ?
21:44:49 <SergeyLukjanov> ttx, additionally, the good news is that we're ready to setup async gate - https://review.openstack.org/#/c/68066/
21:44:54 <devananda> actually, one of those needs to be updated to Ready for Review
21:44:57 <ttx> SergeyLukjanov: awesome
21:45:07 <devananda> the other 3 are either Not Started or ~vendor hasn't shared the code yet~ :)
21:45:10 <ttx> dendrobates: romcheg's ?
21:45:15 <ttx> oosp
21:45:20 <ttx> devananda: romcheg's ?
21:45:56 <ttx> devananda: I see migration-from-nova is not started ?
21:46:09 <devananda> he's been working on it but i need to follow up and see where the code is at
21:46:22 <ttx> ok, wil mark started
21:46:39 <devananda> k
21:46:43 <ttx> kgriffs: around?
21:47:56 <ttx> kgriffs: if you read this, you may have too much, and also you should get people assigned to the unassigned BPs
21:48:05 <ttx> too much essential BPs, I mean
21:48:31 <ttx> SergeyLukjanov, devananda: questions ?
21:48:47 <devananda> no questions here
21:48:48 <SergeyLukjanov> ttx, nope, thx
21:49:11 <ttx> #topic Open discussion
21:49:16 <ttx> anything else anyone ?
21:49:53 <hub_cap> just hugs
21:50:04 <ttx> ok then
21:50:10 <ttx> #endmeeting