21:01:20 <russellb> #startmeeting nova
21:01:21 <openstack> Meeting started Thu Jul 18 21:01:20 2013 UTC.  The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:24 <openstack> The meeting name has been set to 'nova'
21:01:26 <russellb> hello, everyone!
21:01:28 <hemna> hello
21:01:31 <dansmith> yo
21:01:31 <cyeoh> hi
21:01:33 <hartsocks> \o/
21:01:36 <n0ano> o/
21:01:36 <jog0> o/
21:01:41 <alaski> o/
21:01:56 <russellb> #topic havana-2 and havana-3
21:02:04 <russellb> #link https://launchpad.net/nova/+milestone/havana-2
21:02:11 <russellb> we released havana-2 !  \o/
21:02:16 <hemna> :)
21:02:16 <russellb> 25 blueprints implemented
21:02:18 <russellb> 201 bugs fixed
21:02:23 <mikal> heya
21:02:36 <russellb> for reference, 16 blueprints and 214 bugs for havana-1
21:02:48 <russellb> so a pretty similar velocity
21:03:07 <russellb> but now comes the "fun" part ... havana-3 needs a reality check
21:03:09 <russellb> #link https://launchpad.net/nova/+milestone/havana-3
21:03:16 <russellb> we have 100 blueprints targeted for havana-3
21:03:39 <dansmith> ouch, the one essential at the top seems the farthest from reality
21:03:52 <russellb> yeah.
21:04:12 <russellb> so, we need to start aggressively combing through this and moving stuff that won't make it to the "next" milestone
21:04:15 <jog0> deprecate nova-network  I can't see that happening
21:04:39 <russellb> even if half of this was finished, there's no way we have the review bandwidth for this much
21:04:47 <russellb> on a related note ...
21:04:51 <russellb> #link http://lists.openstack.org/pipermail/openstack-dev/2013-April/007823.html
21:05:11 <russellb> back at the beginning of the cycle, we discussed a deadline for proposing code to even have a chance of going in ... 2 weeks in advance of the feature freeze
21:05:15 <russellb> still seem reasonable to everyone?
21:05:38 <cburgess> yes
21:05:38 <geekinutah> seems reasonable given the amount of review that will need to be done
21:05:44 <mikal> Agreed
21:05:46 <dansmith> at least two weeks, but yeah
21:06:00 <devananda> ++
21:06:06 <cyeoh> yes
21:06:06 <russellb> #agreed we will still be enforcing a proposal deadline of at least 2 weeks before feature freeze.  http://lists.openstack.org/pipermail/openstack-dev/2013-April/007823.html
21:06:08 <russellb> cool.
21:06:10 <jog0> ++
21:06:18 <alaski> +1
21:06:30 <russellb> so ... please look over these blueprints and update ones that are yours
21:06:34 <russellb> any particular ones we should talk about?
21:06:49 <dansmith> I think we can keep compute-api-objects for h3, but move the parent out to -next
21:06:51 <russellb> re: nova-network ... vishy: are we at the point where we should just punt deprecating nova-network to Icehouse?
21:07:05 <russellb> dansmith: ok, you should have permissions to update all blueprints
21:07:10 <dansmith> yep, will do
21:07:11 <hemna> I'm not sure if I'm too late for a new BP or not, but I just created a new one that I'd like to work on for H3
21:07:18 <russellb> hemna: which
21:07:20 <vishy> russellb: i think so
21:07:24 <hemna> https://blueprints.launchpad.net/nova/+spec/refactor-iscsi-fc-brick
21:07:30 <cyeoh> what state do I set a BP that is longed required? https://blueprints.launchpad.net/nova/+spec/v3-multi-status-support
21:07:32 <devananda> there's several db-related ones. is the priority of them reasonable?
21:07:35 <russellb> vishy: ok, that's what i figured, i'll move it now
21:07:54 <dansmith> russellb: I got it
21:08:26 <jog0> how is the Direct MYSQLdb impl coming?
21:08:33 <devananda> direct mysql-db, db slave reads, and then several bp's that improve db test coverage, test on multiple backends, etc
21:08:37 <russellb> cyeoh: good question ... i suppose we untarget from havana and then change the definition to obsolete
21:08:50 <cyeoh> russellb: ok, thx
21:09:28 <dansmith> got that one
21:09:43 <russellb> regarding priority ... Essential means we *can't* release without it
21:09:48 <russellb> High is we *really* want to have it
21:10:02 <russellb> Medium is we'd like it, and are still tracking it weekly for release status
21:10:23 <russellb> Low is effectively on the list, but not tracked by release management
21:10:42 <russellb> joearnold: regarding direct mysql, that's comstud, and i think he's out
21:10:46 <russellb> err, jog0 ^
21:10:59 <russellb> jog0: he said "yes, i'll make it happen somehow"  heh
21:11:02 <russellb> we'll see :-)
21:11:25 <dansmith> what about the high, not-started baremetal one near the top?
21:11:26 <jog0> \o/ I hope that one makes it in
21:11:27 <russellb> hemna: that seems reasonable, you guys already using this in cinder?
21:11:32 <russellb> devananda: ^
21:11:45 <hemna> russellb, yes, it just landed in Cinder before H2
21:12:05 <russellb> hemna: ok cool, i'll put it on havana-3 ... ideally up for review *ASAP*
21:12:12 <hemna> ok cool thanks
21:12:22 <russellb> because we're going to get killed with havana-3 reviews i think
21:12:24 <dansmith> russellb: I can do the changes if you want, while you make the hard decisions here
21:12:33 <hemna> not unlike G3 :P
21:12:34 <russellb> dansmith: k!  set that one to h3
21:12:43 <russellb> hemna: heh, like g3, but worse :-)
21:12:56 <hemna> d'oh, lets hope not!
21:13:01 <mrodden> around now, would like to talk about user-locale-api if we get a chance...
21:13:05 <dansmith> set to h3, approved, medium
21:13:08 <hemna> thanks
21:13:09 <russellb> dansmith: ack
21:13:12 <russellb> mrodden: ok
21:13:20 <dansmith> mrodden: that's low
21:13:40 <dansmith> untracked and rather far under the concern bar, IMHO, unless that needs to change
21:13:54 <russellb> i know that one has been out there for a long time
21:13:58 <mrodden> dansmith: correct, but i was wondering if we could bump the priority... its mostly complete
21:14:04 <russellb> anyone else want to lobby for it to be bumped up?
21:14:05 <mrodden> actually it is complete
21:14:10 <russellb> i don't have strong feelings about it
21:14:11 <devananda> given the crunch, I feel that https://blueprints.launchpad.net/nova/+spec/db-slave-handle shouldn't be completed until the other "test the db better" BPs are also done, especially the "test with real back ends"
21:14:11 <dansmith> does it need bumping then?
21:14:30 <russellb> priority is independent of implementation status
21:14:39 <russellb> it's mostly about prioritizing our resources
21:14:48 <russellb> and unfortunately, we do have to stack things somehow
21:14:53 <dansmith> right
21:14:58 <mrodden> it just seems that it gets buried by other higher BP priorities... idk
21:15:05 <mikal> We should also beat the drum a bit lounder and get more cores reviewing regularly
21:15:07 <russellb> devananda: but it's different people right?
21:15:19 <devananda> russellb: yes
21:15:24 <russellb> mikal: yeah ... suggestions on how to do that without being a jerk?  :-)
21:15:35 <mikal> russellb: I got nothing
21:15:37 <russellb> devananda: so does it need to be blocked?
21:15:42 <russellb> mikal: yeah, same, pretty much
21:15:54 <russellb> mikal: i try to send out friendly reminders
21:15:59 <russellb> and i've dropped 2-3 people this cycle ...
21:16:03 <devananda> russellb: i'm curious how others feel about it
21:16:04 <russellb> while also adding a few that are more active
21:16:08 <mikal> russellb: goons, hired goons
21:16:43 <dansmith> geekinutah: comments on the db-slave-handle one?
21:16:47 <russellb> mikal: one thing i'd like to do is write up a wiki page on core team expectations to make it more clear what is expected in terms of review involvement
21:16:54 <russellb> mikal: it's a bit unspoken right now
21:16:58 <mrodden> +1
21:16:59 <dansmith> russellb: good call
21:17:02 <mikal> russellb: that's a good idea
21:17:13 <geekinutah> just a sec, lemme get off the phone
21:17:26 <russellb> #action russellb to write up a nova-core team expectations wiki page and publish to the mailing list next week
21:17:37 <jog0> russellb: what about what is expected from a patch author?
21:17:42 <russellb> #action russellb to post about the need for a havana-3 reality check to the ML
21:17:45 <jog0> to make the reviewers life easier
21:17:49 <russellb> jog0: yeah, that too :-)
21:17:50 <mikal> *action russellb to personally review all patchsets
21:17:58 <dansmith> mikal: he kinda is :)
21:18:01 <russellb> mikal: i tried that for a while, about burned out
21:18:09 <devananda> unified-migrations is High, but has no code up for review or landed yet
21:18:15 <devananda> dansmith: ^ :)
21:18:18 <dansmith> devananda: not true
21:18:22 <russellb> it's a parent blueprint
21:18:43 <devananda> ah! i see
21:18:43 <dansmith> devananda: what about the baremetal migrations one? that seems not started
21:19:02 <devananda> dansmith: indeed. I bumped that to -next
21:19:12 <yjiang5> not sure if pci passthrough can be high, it has been discussed for so long and people really want it.
21:19:13 <russellb> mikal: libvirt console logging?
21:19:15 <dansmith> okay, cool
21:19:53 <mikal> russellb: I actually started that one yesterday
21:19:57 <russellb> mikal: oh, nice
21:20:03 <russellb> dansmith: set that one to started :-)  ^^
21:20:04 <dansmith> I'm okay with PCI passthrough being medium, personally
21:20:07 <russellb> dansmith: same
21:20:14 <mikal> Having a six week family thing in the middle of the cycle has been quite disruptive. I don't recommend it.
21:20:24 <russellb> mikal: all good, family is more important, good sir
21:20:25 <yjiang5> dansmith: ok.
21:20:30 <dansmith> russellb: hmm, I can't make it started
21:20:35 <mikal> russellb: you haven't met my family... :P
21:20:50 <dansmith> no, I can
21:20:52 <dansmith> nevermind
21:21:17 <russellb> cyeoh: we still feel good about v3?
21:21:34 <cyeoh> russellb: overall yes. Most of the extensions are in
21:21:35 <russellb> i think the parts i'm most concerned about there are docs, and novaclient support
21:21:50 <dansmith> is the "create v3 api" something we should defer past havana?
21:21:51 <cyeoh> I agree about docs and novaclient as no one has volunteered to do that bit
21:21:53 <russellb> how'd the docs discussion go?
21:21:58 <annegentle_> docs yes. +10
21:22:08 <mriedem1> annegentle_: you docs team?
21:22:16 <russellb> ok, well, i'd rather not have v3 finished in havana, than not have the docs we need, or not have novaclient
21:22:24 <annegentle_> mriedem1: I lead the team
21:22:39 <dansmith> I don't think v3 is going to be "finished" until it's stable and maybe default,
21:22:40 <russellb> we can have it disabled by default, and when enabled, exposed as "BETA" or whatever we tag things not "STABLE" in the API
21:22:41 <jog0> russellb: ++ its not finished without docs and client
21:22:45 <dansmith> which isn't going to happen in havana, I expect
21:22:45 <cyeoh> russellb: had a good talk with annegentle_ and others about docs. I think I've found some some extra help with docs
21:23:06 <mriedem1> cyeoh: we should talk about docs later, i might be able to sway some resources
21:23:08 <russellb> would another cycle be good to give us more time to think through changes we want to make?
21:23:10 <mriedem1> not promising
21:23:18 <cyeoh> but getting the docs completely finished by the h3 deadline is going to be very hard
21:23:20 <russellb> deferring makes me less happy about all the copied code ...
21:23:23 <russellb> but it is what it is
21:23:34 <cyeoh> mriedem1: thanks
21:23:48 <sdague> russellb: we can also carry v3 as experimental, and enable it. As long as nova-pythonclient picks v2 correctly by default
21:23:57 <russellb> sdague: that's true
21:23:58 <cyeoh> There's going to be a lot of patches just for the api samples in h3
21:23:58 <sdague> the api mechanism does support multi versions
21:24:08 <russellb> yep
21:24:19 <dansmith> sdague, russellb: then the blueprint should have a sub-blueprint of making it not experimental, right?
21:24:21 <cyeoh> I do have one request for reviewers for api related code -
21:24:26 <russellb> so it seems we may need to go that route, and tighen up in Icehouse
21:24:28 <dansmith> and v3 itself should be targeted to -next?
21:24:31 <sdague> dansmith: yes, I think so
21:24:32 <russellb> let's keep discussing each week in this meeting
21:24:45 <cyeoh> and that is to not let anything through that only makes changes to the v2 api now
21:24:50 <dansmith> so leave it for now?
21:24:58 <russellb> well, you guys sound confident
21:25:08 <russellb> basically no chance at this point of having a stable complete documented APi with client support?
21:25:20 <russellb> if so, move to -next
21:25:42 <mriedem1> russellb: if it's docs holding it back, we might have some help
21:25:46 <dansmith> I vote we move the parent to -next, leave the docs and tests for h3 and move those if/when we need
21:26:09 <cyeoh> dansmith: yes that sounds about right.
21:26:25 <russellb> so, we're saying, v3 will be experimental in Havana
21:26:25 <russellb> yes?
21:26:34 <dansmith> IMHO, yes
21:26:38 <russellb> cyeoh: ^ ?
21:26:46 <russellb> just want to be very clear is all
21:26:53 <cyeoh> russellb: yes, in practice its going to need a lot more testing than it has anyway
21:27:05 <russellb> ok, move it!
21:27:10 <russellb> thanks guys.
21:27:11 <dansmith> moved!
21:27:15 <cyeoh> russellb: we ramping up the tempest tests this cycle in an effort to help, but needs more real world testing too
21:27:41 <dansmith> cyeoh: is "unittests for v3" really "api_samples" ?
21:27:42 <russellb> annegentle_: we'll make sure we have solid docs :-)  moving off to Icehouse for now
21:27:57 <annegentle_> russellb: go go get 'em
21:28:19 <cyeoh> dansmith: no api samples is under docs, the unittests bp was to make sure we have better unittest coverage for the extensions
21:28:30 <dansmith> cyeoh: okay
21:28:41 <cyeoh> dansmith: more like a check where the gaps are and fill them.
21:28:57 <dansmith> cyeoh: okay, I'm just looking at all the high bps assigned to you
21:29:08 <dansmith> cyeoh: there are a lot of them.. are they all still justified in remaining for h3?
21:29:20 <russellb> they kind of seem like they're mostly done in parallel
21:29:49 <cyeoh> dansmith, russellb: yea most of them are still there because they apply to all extensions ported
21:29:53 <dansmith> okay
21:29:55 <cyeoh> and we have 5-6 left to merge
21:29:58 <russellb> k
21:30:18 <dansmith> I assume ndipanov_gone 's is still on track as he seems to be making progress on that
21:30:23 <russellb> belliott: around?  graceful service shutdown status?
21:30:54 <russellb> looks like some code made it in to oslo, but needs to be synced over and used by nova
21:31:08 <russellb> i'd say "good progress" dansmith ^
21:31:25 <dansmith> the belliott one?
21:31:28 <russellb> yeah
21:31:38 <dansmith> done
21:32:04 <russellb> joel-coffman: hey, around?
21:32:15 <joel-coffman> yep
21:32:22 <russellb> looking at encrypt-ephemeral-storage
21:32:31 <russellb> i know you have patches out there, but it seems they're not linked to the blueprint
21:32:44 <joel-coffman> two blueprints actually
21:32:47 <russellb> oh
21:32:58 <russellb> ah yes
21:32:59 <joel-coffman> encrypt Cinder volumes: https://blueprints.launchpad.net/nova/+spec/encrypt-cinder-volumes
21:33:01 <russellb> the other is for cinder
21:33:07 <belliott> russellb: yeah some progress :)  it took some time to get oslo reviews through
21:33:14 <russellb> belliott: *nod*
21:33:26 <russellb> joel-coffman: ok, so it's encrypt-cinder-volumes we have patches for
21:33:30 <joel-coffman> yes, ephemeral storage encryption: https://blueprints.launchpad.net/nova/+spec/encrypt-ephemeral-storage
21:33:34 <joel-coffman> so far
21:33:46 <russellb> joel-coffman: so are you basically just needing reviews on these?
21:34:02 <russellb> (encrypt-cinder)
21:34:08 <joel-coffman> yes, although there's some plumbing issues with Cinder that we're resolving
21:34:12 <russellb> ok
21:34:22 <dansmith> that one is already "needs review"
21:34:23 <joel-coffman> should have updated patches in a week or two tops
21:34:25 <russellb> i'd definitely like to see the cinder guys weigh in
21:34:43 <russellb> joel-coffman: the other one, for ephemeral storage, do you still feel like that's realistic for havana-3?
21:34:50 <joel-coffman> I think so
21:35:04 <joel-coffman> we have a beta available internally
21:35:09 <russellb> ok, just beware of the incoming merge proposal storm
21:35:17 <joel-coffman> hope to refine it and submit by early August
21:35:23 <russellb> ok
21:35:49 <joel-coffman> russellb: thanks for the warning -- I took detailed notes at the beginning
21:35:53 <russellb> k :-)
21:35:54 <joel-coffman> :)
21:36:02 <russellb> so we have some medium ones not started ...
21:36:10 <russellb> alaski: soft-errors ?
21:36:45 <alaski> russellb: most likely won't get done, though some work probably will
21:36:54 <russellb> alaski: ok, so let's move it to "next" then
21:37:00 <alaski> works for me
21:37:00 <russellb> alaski: thanks for being realistic :-)
21:37:10 <geekinutah> okay, my phone call is over, sorry folks
21:37:14 <geekinutah> I can talk about db-slave whenever
21:37:17 <russellb> use-db-time should be easy, so we can leave it
21:37:23 <russellb> geekinutah: ok, we can talk about it now
21:37:43 <russellb> devananda: had some thoughts about whether it should be blocked by some others i think
21:37:46 <russellb> devananda: ^ ?
21:38:15 <devananda> geekinutah: hi! i'm curious your (and others) thoughts on the priority / order of landing
21:38:27 <geekinutah> I'm not sure why it would need to be blocked.... I have everything I need to complete it now that oslo-config thing is resolved
21:38:32 <devananda> geekinutah: between teh db-slave patch and the other patches which improve db test coverage (like test-all-backends)
21:38:44 <russellb> has test-all-backends been worked on at all?
21:39:03 <russellb> db-slave seems separate to me
21:39:08 <russellb> it's off by default, so fairly low risk
21:39:21 <dansmith> is it really a high review priority?
21:39:29 <russellb> which, db-slave?  yeah it's high
21:39:34 <dansmith> okay
21:39:40 <devananda> hm, ok
21:39:56 <russellb> i think most of the other db ones are close to being completed IIRC
21:40:01 <russellb> other than test-all-backends ...
21:40:08 <geekinutah> I think it's realistic to get it done in H3 also, there's not a ton of work that needs doing for the BP
21:40:11 <russellb> but the session cleanup, and unit test additions
21:40:16 <devananda> off by default != low risk, IMHO.
21:40:22 <russellb> devananda: fair enough
21:40:27 <devananda> geekinutah: landing the support for it looks easily doable
21:40:43 <devananda> geekinutah: and i'm totally good with that in H3. my concern is the last TODO item: decorate all the things
21:40:58 <geekinutah> that decorate all the things probably doesn't belong in the BP
21:41:06 <russellb> devananda: so are you basically worried about our ability to provide good enough test coverage of it?
21:41:11 <devananda> geekinutah: it's a little late right now to start changing behavior of various methods, IMHO
21:41:15 <devananda> russellb: yes
21:41:16 <russellb> ok
21:41:25 <russellb> so, we could pull that out of the scope
21:41:32 <russellb> make decorate all the things part of Icehouse
21:41:36 <devananda> works for me
21:41:50 <russellb> would that alleviate your concerns?
21:41:59 <devananda> yep
21:42:07 <geekinutah> I was thinking I would decorate one or two very low risk targets... but I'm okay not to do that I guess
21:42:28 <dansmith> so that is just a change that geekinutah can make and then a new bp to file, rihgt?
21:42:50 <russellb> yeah, i think so
21:42:56 <geekinutah> sounds right
21:42:56 <dansmith> sweet
21:42:59 * dansmith closes that tab
21:43:01 <russellb> geekinutah: feel free to propose it, and then we can discuss it
21:43:07 <geekinutah> k
21:43:10 <russellb> separate from the base support
21:43:34 <russellb> eharney: around?
21:43:39 <eharney> hi
21:43:48 <russellb> eharney: are you optimistic about native qemu suppot for glusterfs?
21:43:49 <dansmith> man, the h3 bp list is getting a smackdown by russellb today
21:44:00 <russellb> eharney: given the snapshot work you're doing now
21:44:10 <russellb> dansmith: 'tis my job, i think :-)
21:44:25 <dansmith> russellb: just commenting on the show :)
21:44:35 <eharney> russellb: yes, i think it should get in there, as a follow-up to the snapshot work
21:44:48 <russellb> eharney: ok, good luck sir :-)
21:44:59 <eharney> :)
21:45:00 <dansmith> status change for that?
21:45:05 <dansmith> it's not started currently
21:45:08 <russellb> dansmith: sounds like not started is still accurate
21:45:18 <russellb> eharney: yes?
21:45:20 <dansmith> oh, as a followup to snapshot, got it
21:45:43 <eharney> yes, that's accurate
21:45:50 <russellb> ok
21:46:07 <russellb> dragondm: ping
21:46:17 <russellb> dragondm: i haven't heard anything about versioning notifications, status?
21:46:42 <dansmith> I wonder if that's a "use objects for notifications" thing now?
21:46:56 <dansmith> instead of versioning something else independently
21:46:58 <russellb> ummm ... maybe
21:47:07 <russellb> notifications go outside of nova though
21:47:11 <russellb> and even outside of openstack completely
21:47:14 <russellb> so it's tricky
21:47:14 <dansmith> ah, right
21:47:28 <russellb> entrypoints-plugins - i think we should consider a scope reduction there
21:47:35 <russellb> instead of "entrypoints all the things"
21:47:50 <russellb> entrypoints these specific things that we know will be done in havana (perhaps what is already completed) and call it a day
21:48:26 <russellb> i'll follow up on that, it's not a quick status change ...
21:48:40 <russellb> alright, let's jump to open discussion while we still have some time
21:48:45 <russellb> i suspect next week we'll do this again
21:48:52 <mrodden> +!
21:48:57 <mrodden> err, +1
21:49:02 <russellb> #topic open discussion
21:49:16 <dansmith> The current set of objects patches need some +As:
21:49:17 <russellb> any other topics?
21:49:19 <dansmith> https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/compute-api-objects,n,z
21:49:20 <jog0> rootwrap
21:49:24 <dansmith> and some reviews
21:49:32 <cburgess> rootwrap +1
21:49:33 <dansmith> oh yeah, rootwrap.
21:49:40 <russellb> rootwrap is slow.  :-/
21:49:48 <russellb> it's bad enough that we should do something.
21:50:05 <jog0> TL;DR:  'time sudo nova-rootwrap /etc/nova/rootwrap.conf ls' takes 0.254 seconds
21:50:09 <russellb> step 1) determine if there's anything in the code we can make more efficient to make an impact, or if it's just Python startup pain
21:50:27 <russellb> status on step 1 ?
21:50:57 <russellb> (context, rootwrap adds over 2 minutes to the time it takes to run 30 instances vs. when not using rootwrap)
21:50:58 <dansmith> I haven't looked into rootwrap internals
21:51:06 <jog0> russellb: haven't dug into it too far but pyton looks fast to start up
21:51:22 <russellb> do we have a bug for it yet?
21:51:24 <lifeless> does rootwrap import nova?
21:51:28 <russellb> documenting the data you have so far?
21:51:28 <lifeless> cause that would be super slow
21:51:44 <dansmith> jog0: well, faster than a quarter second, but not fast like sudo, right?
21:51:52 <dansmith> russellb: yeah
21:51:59 <russellb> lifeless: nah, it runs it as a separate process
21:52:03 <jog0> running a one line python file takes 0m0.018s
21:52:03 <lifeless> python itself should be about 20ms
21:52:09 <lifeless> russellb: I know its a separate process
21:52:18 <lifeless> russellb: but if it sucks in bits of nova for extensions
21:52:21 <sdague> jog0: with no imports?
21:52:24 <lifeless> which rootwrap is designed to do...
21:52:32 <jog0> sdague: yes
21:52:33 <russellb> oh, does rootwrap import nova, i read it the other way around, sorry
21:52:52 <russellb> so maybe it's rootwrap reading all of its config files every time?
21:53:11 <russellb> could probably get clever with that.
21:53:13 <jog0> the config takes time and some regex take time
21:53:23 <russellb> ... or make rootwrap a daemon ...
21:53:32 <jog0> ran python -m cprofile on it yesterday
21:53:37 <dansmith> as ttx said, trimming the config to just network things would make it a little faster
21:53:39 <lifeless> from nova.openstack.common.rootwrap import wrapper
21:53:43 <lifeless> it's importing nova
21:53:48 <lifeless> so everything that nova.__init__ imports
21:53:56 <lifeless> and everything that nova.openstack.__ imports
21:54:10 <lifeless> and everything that nova.openstack.common.__init__ imports
21:54:26 <lifeless> e.g. huge
21:54:34 <jog0> speeding up rootwrap by 3x wouldn't be enough o fa fix
21:54:45 <lifeless> it should be able to run in ~30ms
21:54:49 <lifeless> would that be fast enough ?
21:54:52 <russellb> lifeless: it looks like all of those __init__.py files import nothing
21:55:00 <dansmith> IMHO, no
21:55:09 <dansmith> we run ten commands back to back when setting up things like networks
21:55:15 <jog0> err 25
21:55:15 <dansmith> 10 * 30 == 300ms
21:55:28 <dansmith> jog0: was choosing ten as an order of magnitude :)
21:55:46 <dansmith> we can trim some of what we're doing
21:55:47 <lifeless> jog0: please apply this patch http://paste.ubuntu.com/5888945/
21:55:49 <russellb> certainly better than 250ms per command.
21:55:55 <lifeless> that will give us a good idea
21:56:08 <jog0> http://paste.openstack.org/show/40715/
21:56:11 <jog0> to give you an idea
21:56:40 <jog0> and this is the result: long waits for iptables locks http://paste.openstack.org/show/40747/
21:57:02 <russellb> can someone also paste the bug link for this?
21:57:07 <russellb> just want to make sure we have it in the log
21:57:32 <jog0> ip route w/o rootwrap is 0.007 seonds
21:57:58 <jog0> with 0.158
21:58:11 <jog0> russellb: sorry https://bugs.launchpad.net/oslo/+bug/1199433
21:58:14 <uvirtbot> Launchpad bug 1199433 in nova "nova boot --num-instances=50 times out " [High,New]
21:58:20 <russellb> #link https://bugs.launchpad.net/oslo/+bug/1199433
21:58:35 <russellb> jog0: can you try the patch lifeless posted?
21:58:42 <jog0> lifeless: my dev env is down at the moment
21:58:46 <russellb> ah, k
21:58:48 <jog0> anyone else have devstack running?
21:58:49 <russellb> well something to look at
21:59:28 <dansmith> 20 things
21:59:37 <dansmith> no
21:59:42 <dansmith> 268
21:59:45 <jog0> lol
21:59:48 <mrodden> :(
22:00:00 <dansmith> >>> print("imported modules %s" % len(sys.modules.keys(),))
22:00:01 <dansmith> imported modules 268
22:00:01 <cburgess> That seems like it *might* be an issue.
22:00:22 <russellb> print them?
22:00:29 <russellb> see what the heck it is
22:00:44 <dansmith> it's not bad though
22:00:58 <dansmith> python startup + that import == 28ms
22:01:10 <dansmith> well, 44ms wall clock time
22:01:17 <dansmith> dan@guaranine:~/nova$ time python -c "from nova.openstack.common.rootwrap import wrapper"
22:01:17 <dansmith> real	0m0.044s
22:01:17 <dansmith> user	0m0.028s
22:01:17 <dansmith> sys	0m0.012s
22:01:43 <russellb> ok, well, more investigation to do here
22:01:45 <russellb> we're out of time
22:01:47 <russellb> thanks everyone
22:01:53 <russellb> #endmeeting