21:00:32 <russellb> #startmeeting nova
21:00:33 <openstack> Meeting started Thu Dec 19 21:00:32 2013 UTC and is due to finish in 60 minutes.  The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:36 <openstack> The meeting name has been set to 'nova'
21:00:39 <beagles> o/
21:00:43 <russellb> hello, everyone!
21:00:49 <alaski> o/
21:00:51 <n0ano> o/
21:00:53 <bpokorny> o/
21:01:00 <dansmith> \o
21:01:01 <russellb> #link https://wiki.openstack.org/wiki/Meetings/Nova
21:01:03 <hartsocks> \o
21:01:14 <russellb> #topic general
21:01:17 <russellb> first, a meeting about meetings
21:01:29 <russellb> let's skip next week, i'm sure many people will be out anyway
21:01:42 <russellb> once we get back, we'll start a new schedule where we alternate each week
21:01:57 <russellb> hoping to offer more people a chance to join in every other week
21:02:01 <russellb> wiki page has the times
21:02:11 <russellb> we'll do 2100 UTC when we first start back
21:02:15 <russellb> then 1400 UTC after that, then alternate
21:02:18 <russellb> we'll see how it goes ...
21:02:44 <russellb> alternative approach as suggested by ttx is to keep this every week, and occasionally (monthly?) hold other time slots to check in with other time zones
21:02:58 <russellb> but seems a number of folks are happy to try the alternating bit first
21:03:27 <russellb> that was the only general thing, on to the real stuff
21:03:29 <russellb> #topic sub-teams
21:03:46 * hartsocks waves
21:03:48 <n0ano> o/ scheduler
21:03:59 <russellb> melwitt had a conflict, but sent this for novaclient
21:04:02 <russellb> <melwitt> open bugs, 122 !fix released, 78 !fix committed and !fix released
21:04:02 <russellb> <melwitt> 20 new bugs
21:04:02 <russellb> <melwitt> 0 high bugs
21:04:02 <russellb> <melwitt> 46 patches up, 6 WIP, increasing activity, number of patches continually increasing
21:04:20 <russellb> so, new bug count steady, but a growing review queue
21:04:34 <russellb> so, don't forget to take a look at those when doing nova reviews
21:04:49 <russellb> n0ano: you're up
21:05:21 <n0ano> no_db scheduler - got slowed down by devstack failure, that should be resolved and they can make some more progress soon...
21:06:08 <russellb> cool, haven't had a chance to look at those patches yet myself
21:06:15 <n0ano> gantt (code forklift) - new tree up, I'm in the process of trying to get it to pass Jenkins so we can actually submit patches (lots of detailed name changes to go throught)
21:06:28 <russellb> #link http://git.openstack.org/cgit/openstack/gantt
21:06:35 <russellb> n0ano: so one thing we could do is switch all jobs to non-voting
21:06:45 <russellb> while we work to land a series of changes to get jobs to pass
21:06:54 <russellb> instead of having to do one giant "make it work" commit
21:07:07 <russellb> if "make it work" is getting really big anyway
21:07:10 <n0ano> russellb, yeah but I'd at least like to get things like tox -epep8 to work first
21:07:30 <russellb> "make it pass" i mean
21:07:32 <russellb> not necessarily work
21:08:12 <n0ano> anyway, going forward my idea is I'll keep the gantt tree in sync with any scheduler changes to the nova tree, hopefully not too many changes until we can cut over
21:08:23 <russellb> related: please add openstack/gantt to your review queue queries.  right now gantt-core == nova-core + lifeless
21:08:30 <russellb> since he was helping bootstrap the thing
21:08:43 <russellb> n0ano: great
21:08:49 <lifeless> russellb: ack, think it's there already but I'll double check
21:09:02 <russellb> we're going to have to revisit the deprecation plan once it's working
21:09:08 <russellb> we sort of punted that discussion on openstack-dev
21:09:11 <russellb> pending it working
21:09:21 <lifeless> as soon as gantt is passing tests I'm going to ping the list and the volunteers to do the actual work
21:09:29 <russellb> cool
21:09:38 <n0ano> my goal is to get it working as soon as possible, then we can start some serious discussions about futures
21:09:50 <russellb> though making the repo and getting all the tests to pass is quite a bit of actual work itself :)
21:09:57 <russellb> but probably mostly a one person thing for now
21:10:00 <russellb> or a small group anyway
21:10:08 <russellb> n0ano: sounds great to me
21:10:17 * n0ano is getting exhausted already :-)
21:10:28 <russellb> hope you're having some fun though
21:10:33 <russellb> it's exciting bootstrapping something new
21:10:44 <n0ano> that's about it for now, we've cancelled the next 2 meetings, start up again in Jan.
21:10:55 * n0ano fun, yeah that's it, fun
21:11:04 <russellb> alright, thanks!
21:11:09 <russellb> let me know what i can do along the way
21:11:12 <russellb> hartsocks: you're up
21:11:16 <hartsocks> hey
21:11:25 <hartsocks> Just wanted to update on the Minesweeper.
21:11:35 <russellb> ooh
21:11:39 <hartsocks> We're still on track for Jan 26th
21:11:42 <russellb> still love that name
21:11:53 <russellb> on track for?
21:12:01 <russellb> running against all changes?
21:12:02 <hartsocks> Our current plan is to have the Minesweeper vote +1 and 0
21:12:13 <hartsocks> on changes in our driver dir and above.
21:12:25 <jog0> above?
21:12:32 <hartsocks> So that's all patches except for patches affecting the other hypervisors.
21:12:46 <jog0> neato
21:13:04 <hartsocks> Yeah. So since in the tree the other hypervisors are siblings… you know… test all above and around.
21:13:05 <russellb> so ... everything except something that only changes nova/virt/someotherdriver/*.py (and tests) ?
21:13:17 <hartsocks> yeah. Not sure how we *could* test that.
21:13:29 <russellb> it's just like changing anything else
21:13:33 <dansmith> well,
21:13:41 <russellb> testing anything else i mean
21:13:43 <dansmith> yeah, it's just a patch that you run and make sure everything is cool
21:13:53 <russellb> you're just running against that state of the nova tree
21:14:02 <hartsocks> right.
21:14:23 <dansmith> we've had cross-driver dependent code before, which isn't cool, but...
21:14:42 <hartsocks> So the current logic is, if it only touches the other drivers then we're not going to be interested… or qualified to test it anyway.
21:15:10 <dansmith> and the logic should be: it's a patch against nova, so you run your tests against it
21:15:24 <russellb> another benefit ... running tests more often helps find bugs that only surface some of the time
21:15:26 <dansmith> if it only changes virt/libvirt code, then it should pass
21:15:32 <dansmith> yeah
21:15:40 <hartsocks> Well, we wouldn't be exercising a libvirt patch anyhow.
21:15:47 <dansmith> right
21:15:47 <hartsocks> It would just be a noop.
21:15:49 <dansmith> yep
21:15:53 <hartsocks> So we're ignoring them.
21:15:54 <russellb> noop in a perfect world
21:15:57 <hartsocks> But…
21:16:10 <russellb> in the gate world today, we're running "no-ops" over and over and over, but we find bugs that way
21:16:15 <hartsocks> if it touches libvirt *and* some other code in the tree outside that… then yeah we have to test.
21:16:52 <russellb> if it's a matter of you can keep up without those changes, but can keep up filtering them out, that may be OK short-term
21:17:00 <russellb> long term, running against everything seems best overall
21:17:14 <russellb> i didn't say that right
21:17:19 <hartsocks> Okay. This is why I wanted to bring it up. It didn't seem right to have Minesweeper voting on other hypervisors since it can't really say anything about them anyway.
21:17:37 <dansmith> it absolutely should be voting on every patch, IMHO
21:17:38 <russellb> sure, and i appreciate it
21:17:48 <russellb> just trying to share feedback
21:17:51 <hartsocks> I'll push the feeback back to the testing guys.
21:17:52 <dansmith> https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan
21:17:56 <russellb> you guys are still ahead of the pack on all of this :)
21:18:00 <jog0> ++ to vote on all patches but babysteps are good too
21:18:01 <dansmith> says: " feedback is being reported on every proposed change to nova,"
21:18:05 <russellb> jog0: ++
21:18:15 <hartsocks> our testing guys rock and I'm honored to be their receptionist in IRC.
21:18:20 <hartsocks> :-)
21:18:23 <russellb> heh
21:18:36 <hartsocks> The other bit I *might mention is*
21:18:43 <hartsocks> https://blueprints.launchpad.net/oslo/+spec/vmware-api
21:19:01 <hartsocks> But that's a longer discussion and not really necessarily for IceHouse.
21:19:24 <hartsocks> That's all I wanted to bring up this session.
21:19:26 <russellb> so here's a question
21:19:38 <russellb> would this vmware python library be useful outside of openstack?
21:19:47 <russellb> or do you really see it as openstack specific
21:20:00 <russellb> just something to think about WRT that blueprint
21:20:07 <lifeless> how many machines did you end up needing to keep up?
21:20:16 <hartsocks> That *is* a good question… *and* what we *should* be contributing is only the bits useful for OpenStack.
21:20:18 <lifeless> thats useful data for other orgs, I think
21:21:23 <hartsocks> lifeless: I don't have that stat in front of me, it changed recently.
21:21:45 <hartsocks> lifeless: I can tell you we "burned out" one set of older infrastructure and had to move to new machines.
21:21:53 <lifeless> hartsocks: I'd like to know how far off the estimates in https://etherpad.openstack.org/tripleo-test-cluster are basically
21:22:16 <russellb> 10s of servers?  100s?
21:22:18 <hartsocks> lifeless: and I'll deliver a message to the testing guys about that :-)
21:22:21 <lifeless> hartsocks: thanks!
21:22:38 <hartsocks> russellb: part of it is in an internal IaaS thing that makes that hard to answer.
21:22:40 <russellb> heh, anyway, yeah, would love to know more if you can share
21:22:42 <russellb> fair enough
21:23:00 <russellb> ah, it's like you're using a cloud for what it's good for
21:23:12 <hartsocks> russellb: yep.
21:23:13 <russellb> alright, i think that was all the sub-teams that raised hands
21:23:28 <russellb> #topic bugs
21:23:30 <lifeless> russellb: the estimates I have are ~20 concurrent vm's for an exact devstack-gate setup, which is to say 60 cores, 320GB of RAM. mumble.
21:23:36 <lifeless> oh, my fav topic
21:23:41 <lifeless> so I started the thread
21:23:42 <russellb> lifeless: cool
21:23:44 <lifeless> it seemed to peter out
21:23:53 <russellb> definition of triage?
21:24:05 <lifeless> yeah, you asked the q in there that the other thread was about
21:24:09 <lifeless> so I answered it there
21:24:26 <russellb> was there another thread too that i missed?
21:24:32 <lifeless> no
21:24:40 <russellb> ok
21:25:17 <russellb> guess i need to take another look
21:25:24 <russellb> so, we have 8 critical bugs open right now for gate issues
21:25:26 <russellb> https://bugs.launchpad.net/nova/+bugs?search=Search&field.importance=Critical&field.status=New&field.status=Incomplete&field.status=Confirmed&field.status=Triaged&field.status=In+Progress&field.status=Fix+Committed
21:25:35 <russellb> could really use some more eyes on these things
21:25:45 <russellb> we need to help improve reliability, as it's been a little rough lately
21:26:18 <russellb> notmyname came up with a nice graph that shows probability of patches to pass the gate ... http://not.mn/gate_status.html
21:26:30 <russellb> just to help demonstrate the importance of looking at these bugs
21:26:47 <jog0> russellb: https://bugs.launchpad.net/bugs/1253896
21:26:49 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress]
21:26:50 <russellb> not *all* of that is openstack bugs .... some of it is external events (sphinx API, etc)
21:26:54 <jog0> is a msassive one
21:27:09 <russellb> is it just with neutron?
21:27:20 <russellb> not trying to pass it off on the neutron team
21:27:20 <jog0> no, confirming now though
21:27:40 <jog0> AFAIK it was a nova-network issue
21:27:44 <russellb> ok
21:27:57 <jog0> hmm it my be neutron
21:28:24 <jog0> yeah its neutron.  but neutron == nuetron+nova
21:28:29 <russellb> has anyone set up an env to reproduce?
21:28:32 <russellb> right
21:28:47 <russellb> i'd like a devstack VM I can log in to, run the test that triggers it over and over and see it happen
21:28:48 <sdague> the neutron team has been banging on that one a lot
21:28:50 <russellb> and then poke at the env
21:29:04 <russellb> i know i could make that, just haven't yet :)
21:29:06 <sdague> they found a bunch of races, which made it better, but there are still issues falling out
21:30:19 <russellb> sorry, looking at it
21:31:10 <russellb> well, if anyone has some time to put in, eyes on that one seems to be the most important
21:31:17 <markmcclain> yeah.. we've found and fixed a bunch of other items while digging into this one
21:31:22 <russellb> markmcclain: nice
21:31:53 <jog0> markmcclain: any nova patches from neutron folks we need  to look at
21:32:08 <russellb> so one thing we talked about earlier this week is, at what point do we have to limit other dev activities to force more focus on the gate bugs
21:32:18 <russellb> i'm not sure what the answer is, and would just prefer we don't let it get to that point :)
21:32:33 <markmcclain> jog0: none that I know of
21:32:52 <russellb> markmcclain: please ping me, or #openstack-nova, if any come up
21:32:59 <markmcclain> russellb: will do
21:33:19 <russellb> jog0: any other bugs worthy of special attention today you think?
21:33:33 <jog0> https://bugs.launchpad.net/nova/+bug/1254890
21:33:35 <uvirtbot> Launchpad bug 1254890 in tempest ""Timed out waiting for thing" causes tempest-dsvm-neutron-* failures" [Low,Confirmed]
21:34:07 <jog0> https://bugs.launchpad.net/tempest/+bug/1258682
21:34:10 <uvirtbot> Launchpad bug 1258682 in tempest "timeout causing gate-tempest-dsvm-full to fail" [Undecided,Invalid]
21:34:26 <jog0> and it would be good to try to rule out nova on https://bugs.launchpad.net/tempest/+bug/1253896
21:34:30 <uvirtbot> Launchpad bug 1253896 in neutron "Attempts to verify guests are running via SSH fails. SSH connection to guest does not work." [Critical,In progress]
21:35:23 <russellb> OK, cool, thanks for the pointers
21:35:28 <russellb> we can come back to these in open discussion if we have time
21:35:30 <russellb> #topic blueprints
21:35:39 <jog0> those are the top 4 bugs on http://status.openstack.org/elastic-recheck/
21:35:46 <russellb> #link https://launchpad.net/nova/+milestone/icehouse-2
21:36:01 <russellb> i posted to the ML last week and said that today is the deadline to have blueprints approved for icehouse-2
21:36:12 <russellb> and after today, we'll be pushing the rest off to icehouse-3
21:36:27 <russellb> for any that are waiting on review (in Pending Approval or New), I'd like to review them before deferral, though
21:36:30 <russellb> so I could use some help on that part
21:36:55 <russellb> unfortunately that status doesn't show up there, so it's easier to find those on this page: https://blueprints.launchpad.net/nova/icehouse
21:37:09 <russellb> any specific blueprints anyone wants to discuss today?
21:37:42 <hartsocks> I have https://blueprints.launchpad.net/nova/+spec/autowsdl-repair I'm trying to finish tonight. It's only low priority though.
21:38:29 <russellb> OK, looks like it's probably fine
21:38:35 <russellb> but will review before deferring
21:39:04 <russellb> looks like we have 11 that need review
21:40:01 <russellb> #topic open discussion
21:40:26 <russellb> anything else?
21:41:12 <russellb> alright, thanks everyeone!  we'll meet again in 2 weeks
21:41:17 <russellb> #endmeeting