21:01:20 #startmeeting nova 21:01:21 Meeting started Thu Jul 18 21:01:20 2013 UTC. The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:22 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:24 The meeting name has been set to 'nova' 21:01:26 hello, everyone! 21:01:28 hello 21:01:31 yo 21:01:31 hi 21:01:33 \o/ 21:01:36 o/ 21:01:36 o/ 21:01:41 o/ 21:01:56 #topic havana-2 and havana-3 21:02:04 #link https://launchpad.net/nova/+milestone/havana-2 21:02:11 we released havana-2 ! \o/ 21:02:16 :) 21:02:16 25 blueprints implemented 21:02:18 201 bugs fixed 21:02:23 heya 21:02:36 for reference, 16 blueprints and 214 bugs for havana-1 21:02:48 so a pretty similar velocity 21:03:07 but now comes the "fun" part ... havana-3 needs a reality check 21:03:09 #link https://launchpad.net/nova/+milestone/havana-3 21:03:16 we have 100 blueprints targeted for havana-3 21:03:39 ouch, the one essential at the top seems the farthest from reality 21:03:52 yeah. 21:04:12 so, we need to start aggressively combing through this and moving stuff that won't make it to the "next" milestone 21:04:15 deprecate nova-network I can't see that happening 21:04:39 even if half of this was finished, there's no way we have the review bandwidth for this much 21:04:47 on a related note ... 21:04:51 #link http://lists.openstack.org/pipermail/openstack-dev/2013-April/007823.html 21:05:11 back at the beginning of the cycle, we discussed a deadline for proposing code to even have a chance of going in ... 2 weeks in advance of the feature freeze 21:05:15 still seem reasonable to everyone? 21:05:38 yes 21:05:38 seems reasonable given the amount of review that will need to be done 21:05:44 Agreed 21:05:46 at least two weeks, but yeah 21:06:00 ++ 21:06:06 yes 21:06:06 #agreed we will still be enforcing a proposal deadline of at least 2 weeks before feature freeze. http://lists.openstack.org/pipermail/openstack-dev/2013-April/007823.html 21:06:08 cool. 21:06:10 ++ 21:06:18 +1 21:06:30 so ... please look over these blueprints and update ones that are yours 21:06:34 any particular ones we should talk about? 21:06:49 I think we can keep compute-api-objects for h3, but move the parent out to -next 21:06:51 re: nova-network ... vishy: are we at the point where we should just punt deprecating nova-network to Icehouse? 21:07:05 dansmith: ok, you should have permissions to update all blueprints 21:07:10 yep, will do 21:07:11 I'm not sure if I'm too late for a new BP or not, but I just created a new one that I'd like to work on for H3 21:07:18 hemna: which 21:07:20 russellb: i think so 21:07:24 https://blueprints.launchpad.net/nova/+spec/refactor-iscsi-fc-brick 21:07:30 what state do I set a BP that is longed required? https://blueprints.launchpad.net/nova/+spec/v3-multi-status-support 21:07:32 there's several db-related ones. is the priority of them reasonable? 21:07:35 vishy: ok, that's what i figured, i'll move it now 21:07:54 russellb: I got it 21:08:26 how is the Direct MYSQLdb impl coming? 21:08:33 direct mysql-db, db slave reads, and then several bp's that improve db test coverage, test on multiple backends, etc 21:08:37 cyeoh: good question ... i suppose we untarget from havana and then change the definition to obsolete 21:08:50 russellb: ok, thx 21:09:28 got that one 21:09:43 regarding priority ... Essential means we *can't* release without it 21:09:48 High is we *really* want to have it 21:10:02 Medium is we'd like it, and are still tracking it weekly for release status 21:10:23 Low is effectively on the list, but not tracked by release management 21:10:42 joearnold: regarding direct mysql, that's comstud, and i think he's out 21:10:46 err, jog0 ^ 21:10:59 jog0: he said "yes, i'll make it happen somehow" heh 21:11:02 we'll see :-) 21:11:25 what about the high, not-started baremetal one near the top? 21:11:26 \o/ I hope that one makes it in 21:11:27 hemna: that seems reasonable, you guys already using this in cinder? 21:11:32 devananda: ^ 21:11:45 russellb, yes, it just landed in Cinder before H2 21:12:05 hemna: ok cool, i'll put it on havana-3 ... ideally up for review *ASAP* 21:12:12 ok cool thanks 21:12:22 because we're going to get killed with havana-3 reviews i think 21:12:24 russellb: I can do the changes if you want, while you make the hard decisions here 21:12:33 not unlike G3 :P 21:12:34 dansmith: k! set that one to h3 21:12:43 hemna: heh, like g3, but worse :-) 21:12:56 d'oh, lets hope not! 21:13:01 around now, would like to talk about user-locale-api if we get a chance... 21:13:05 set to h3, approved, medium 21:13:08 thanks 21:13:09 dansmith: ack 21:13:12 mrodden: ok 21:13:20 mrodden: that's low 21:13:40 untracked and rather far under the concern bar, IMHO, unless that needs to change 21:13:54 i know that one has been out there for a long time 21:13:58 dansmith: correct, but i was wondering if we could bump the priority... its mostly complete 21:14:04 anyone else want to lobby for it to be bumped up? 21:14:05 actually it is complete 21:14:10 i don't have strong feelings about it 21:14:11 given the crunch, I feel that https://blueprints.launchpad.net/nova/+spec/db-slave-handle shouldn't be completed until the other "test the db better" BPs are also done, especially the "test with real back ends" 21:14:11 does it need bumping then? 21:14:30 priority is independent of implementation status 21:14:39 it's mostly about prioritizing our resources 21:14:48 and unfortunately, we do have to stack things somehow 21:14:53 right 21:14:58 it just seems that it gets buried by other higher BP priorities... idk 21:15:05 We should also beat the drum a bit lounder and get more cores reviewing regularly 21:15:07 devananda: but it's different people right? 21:15:19 russellb: yes 21:15:24 mikal: yeah ... suggestions on how to do that without being a jerk? :-) 21:15:35 russellb: I got nothing 21:15:37 devananda: so does it need to be blocked? 21:15:42 mikal: yeah, same, pretty much 21:15:54 mikal: i try to send out friendly reminders 21:15:59 and i've dropped 2-3 people this cycle ... 21:16:03 russellb: i'm curious how others feel about it 21:16:04 while also adding a few that are more active 21:16:08 russellb: goons, hired goons 21:16:43 geekinutah: comments on the db-slave-handle one? 21:16:47 mikal: one thing i'd like to do is write up a wiki page on core team expectations to make it more clear what is expected in terms of review involvement 21:16:54 mikal: it's a bit unspoken right now 21:16:58 +1 21:16:59 russellb: good call 21:17:02 russellb: that's a good idea 21:17:13 just a sec, lemme get off the phone 21:17:26 #action russellb to write up a nova-core team expectations wiki page and publish to the mailing list next week 21:17:37 russellb: what about what is expected from a patch author? 21:17:42 #action russellb to post about the need for a havana-3 reality check to the ML 21:17:45 to make the reviewers life easier 21:17:49 jog0: yeah, that too :-) 21:17:50 *action russellb to personally review all patchsets 21:17:58 mikal: he kinda is :) 21:18:01 mikal: i tried that for a while, about burned out 21:18:09 unified-migrations is High, but has no code up for review or landed yet 21:18:15 dansmith: ^ :) 21:18:18 devananda: not true 21:18:22 it's a parent blueprint 21:18:43 ah! i see 21:18:43 devananda: what about the baremetal migrations one? that seems not started 21:19:02 dansmith: indeed. I bumped that to -next 21:19:12 not sure if pci passthrough can be high, it has been discussed for so long and people really want it. 21:19:13 mikal: libvirt console logging? 21:19:15 okay, cool 21:19:53 russellb: I actually started that one yesterday 21:19:57 mikal: oh, nice 21:20:03 dansmith: set that one to started :-) ^^ 21:20:04 I'm okay with PCI passthrough being medium, personally 21:20:07 dansmith: same 21:20:14 Having a six week family thing in the middle of the cycle has been quite disruptive. I don't recommend it. 21:20:24 mikal: all good, family is more important, good sir 21:20:25 dansmith: ok. 21:20:30 russellb: hmm, I can't make it started 21:20:35 russellb: you haven't met my family... :P 21:20:50 no, I can 21:20:52 nevermind 21:21:17 cyeoh: we still feel good about v3? 21:21:34 russellb: overall yes. Most of the extensions are in 21:21:35 i think the parts i'm most concerned about there are docs, and novaclient support 21:21:50 is the "create v3 api" something we should defer past havana? 21:21:51 I agree about docs and novaclient as no one has volunteered to do that bit 21:21:53 how'd the docs discussion go? 21:21:58 docs yes. +10 21:22:08 annegentle_: you docs team? 21:22:16 ok, well, i'd rather not have v3 finished in havana, than not have the docs we need, or not have novaclient 21:22:24 mriedem1: I lead the team 21:22:39 I don't think v3 is going to be "finished" until it's stable and maybe default, 21:22:40 we can have it disabled by default, and when enabled, exposed as "BETA" or whatever we tag things not "STABLE" in the API 21:22:41 russellb: ++ its not finished without docs and client 21:22:45 which isn't going to happen in havana, I expect 21:22:45 russellb: had a good talk with annegentle_ and others about docs. I think I've found some some extra help with docs 21:23:06 cyeoh: we should talk about docs later, i might be able to sway some resources 21:23:08 would another cycle be good to give us more time to think through changes we want to make? 21:23:10 not promising 21:23:18 but getting the docs completely finished by the h3 deadline is going to be very hard 21:23:20 deferring makes me less happy about all the copied code ... 21:23:23 but it is what it is 21:23:34 mriedem1: thanks 21:23:48 russellb: we can also carry v3 as experimental, and enable it. As long as nova-pythonclient picks v2 correctly by default 21:23:57 sdague: that's true 21:23:58 There's going to be a lot of patches just for the api samples in h3 21:23:58 the api mechanism does support multi versions 21:24:08 yep 21:24:19 sdague, russellb: then the blueprint should have a sub-blueprint of making it not experimental, right? 21:24:21 I do have one request for reviewers for api related code - 21:24:26 so it seems we may need to go that route, and tighen up in Icehouse 21:24:28 and v3 itself should be targeted to -next? 21:24:31 dansmith: yes, I think so 21:24:32 let's keep discussing each week in this meeting 21:24:45 and that is to not let anything through that only makes changes to the v2 api now 21:24:50 so leave it for now? 21:24:58 well, you guys sound confident 21:25:08 basically no chance at this point of having a stable complete documented APi with client support? 21:25:20 if so, move to -next 21:25:42 russellb: if it's docs holding it back, we might have some help 21:25:46 I vote we move the parent to -next, leave the docs and tests for h3 and move those if/when we need 21:26:09 dansmith: yes that sounds about right. 21:26:25 so, we're saying, v3 will be experimental in Havana 21:26:25 yes? 21:26:34 IMHO, yes 21:26:38 cyeoh: ^ ? 21:26:46 just want to be very clear is all 21:26:53 russellb: yes, in practice its going to need a lot more testing than it has anyway 21:27:05 ok, move it! 21:27:10 thanks guys. 21:27:11 moved! 21:27:15 russellb: we ramping up the tempest tests this cycle in an effort to help, but needs more real world testing too 21:27:41 cyeoh: is "unittests for v3" really "api_samples" ? 21:27:42 annegentle_: we'll make sure we have solid docs :-) moving off to Icehouse for now 21:27:57 russellb: go go get 'em 21:28:19 dansmith: no api samples is under docs, the unittests bp was to make sure we have better unittest coverage for the extensions 21:28:30 cyeoh: okay 21:28:41 dansmith: more like a check where the gaps are and fill them. 21:28:57 cyeoh: okay, I'm just looking at all the high bps assigned to you 21:29:08 cyeoh: there are a lot of them.. are they all still justified in remaining for h3? 21:29:20 they kind of seem like they're mostly done in parallel 21:29:49 dansmith, russellb: yea most of them are still there because they apply to all extensions ported 21:29:53 okay 21:29:55 and we have 5-6 left to merge 21:29:58 k 21:30:18 I assume ndipanov_gone 's is still on track as he seems to be making progress on that 21:30:23 belliott: around? graceful service shutdown status? 21:30:54 looks like some code made it in to oslo, but needs to be synced over and used by nova 21:31:08 i'd say "good progress" dansmith ^ 21:31:25 the belliott one? 21:31:28 yeah 21:31:38 done 21:32:04 joel-coffman: hey, around? 21:32:15 yep 21:32:22 looking at encrypt-ephemeral-storage 21:32:31 i know you have patches out there, but it seems they're not linked to the blueprint 21:32:44 two blueprints actually 21:32:47 oh 21:32:58 ah yes 21:32:59 encrypt Cinder volumes: https://blueprints.launchpad.net/nova/+spec/encrypt-cinder-volumes 21:33:01 the other is for cinder 21:33:07 russellb: yeah some progress :) it took some time to get oslo reviews through 21:33:14 belliott: *nod* 21:33:26 joel-coffman: ok, so it's encrypt-cinder-volumes we have patches for 21:33:30 yes, ephemeral storage encryption: https://blueprints.launchpad.net/nova/+spec/encrypt-ephemeral-storage 21:33:34 so far 21:33:46 joel-coffman: so are you basically just needing reviews on these? 21:34:02 (encrypt-cinder) 21:34:08 yes, although there's some plumbing issues with Cinder that we're resolving 21:34:12 ok 21:34:22 that one is already "needs review" 21:34:23 should have updated patches in a week or two tops 21:34:25 i'd definitely like to see the cinder guys weigh in 21:34:43 joel-coffman: the other one, for ephemeral storage, do you still feel like that's realistic for havana-3? 21:34:50 I think so 21:35:04 we have a beta available internally 21:35:09 ok, just beware of the incoming merge proposal storm 21:35:17 hope to refine it and submit by early August 21:35:23 ok 21:35:49 russellb: thanks for the warning -- I took detailed notes at the beginning 21:35:53 k :-) 21:35:54 :) 21:36:02 so we have some medium ones not started ... 21:36:10 alaski: soft-errors ? 21:36:45 russellb: most likely won't get done, though some work probably will 21:36:54 alaski: ok, so let's move it to "next" then 21:37:00 works for me 21:37:00 alaski: thanks for being realistic :-) 21:37:10 okay, my phone call is over, sorry folks 21:37:14 I can talk about db-slave whenever 21:37:17 use-db-time should be easy, so we can leave it 21:37:23 geekinutah: ok, we can talk about it now 21:37:43 devananda: had some thoughts about whether it should be blocked by some others i think 21:37:46 devananda: ^ ? 21:38:15 geekinutah: hi! i'm curious your (and others) thoughts on the priority / order of landing 21:38:27 I'm not sure why it would need to be blocked.... I have everything I need to complete it now that oslo-config thing is resolved 21:38:32 geekinutah: between teh db-slave patch and the other patches which improve db test coverage (like test-all-backends) 21:38:44 has test-all-backends been worked on at all? 21:39:03 db-slave seems separate to me 21:39:08 it's off by default, so fairly low risk 21:39:21 is it really a high review priority? 21:39:29 which, db-slave? yeah it's high 21:39:34 okay 21:39:40 hm, ok 21:39:56 i think most of the other db ones are close to being completed IIRC 21:40:01 other than test-all-backends ... 21:40:08 I think it's realistic to get it done in H3 also, there's not a ton of work that needs doing for the BP 21:40:11 but the session cleanup, and unit test additions 21:40:16 off by default != low risk, IMHO. 21:40:22 devananda: fair enough 21:40:27 geekinutah: landing the support for it looks easily doable 21:40:43 geekinutah: and i'm totally good with that in H3. my concern is the last TODO item: decorate all the things 21:40:58 that decorate all the things probably doesn't belong in the BP 21:41:06 devananda: so are you basically worried about our ability to provide good enough test coverage of it? 21:41:11 geekinutah: it's a little late right now to start changing behavior of various methods, IMHO 21:41:15 russellb: yes 21:41:16 ok 21:41:25 so, we could pull that out of the scope 21:41:32 make decorate all the things part of Icehouse 21:41:36 works for me 21:41:50 would that alleviate your concerns? 21:41:59 yep 21:42:07 I was thinking I would decorate one or two very low risk targets... but I'm okay not to do that I guess 21:42:28 so that is just a change that geekinutah can make and then a new bp to file, rihgt? 21:42:50 yeah, i think so 21:42:56 sounds right 21:42:56 sweet 21:42:59 * dansmith closes that tab 21:43:01 geekinutah: feel free to propose it, and then we can discuss it 21:43:07 k 21:43:10 separate from the base support 21:43:34 eharney: around? 21:43:39 hi 21:43:48 eharney: are you optimistic about native qemu suppot for glusterfs? 21:43:49 man, the h3 bp list is getting a smackdown by russellb today 21:44:00 eharney: given the snapshot work you're doing now 21:44:10 dansmith: 'tis my job, i think :-) 21:44:25 russellb: just commenting on the show :) 21:44:35 russellb: yes, i think it should get in there, as a follow-up to the snapshot work 21:44:48 eharney: ok, good luck sir :-) 21:44:59 :) 21:45:00 status change for that? 21:45:05 it's not started currently 21:45:08 dansmith: sounds like not started is still accurate 21:45:18 eharney: yes? 21:45:20 oh, as a followup to snapshot, got it 21:45:43 yes, that's accurate 21:45:50 ok 21:46:07 dragondm: ping 21:46:17 dragondm: i haven't heard anything about versioning notifications, status? 21:46:42 I wonder if that's a "use objects for notifications" thing now? 21:46:56 instead of versioning something else independently 21:46:58 ummm ... maybe 21:47:07 notifications go outside of nova though 21:47:11 and even outside of openstack completely 21:47:14 so it's tricky 21:47:14 ah, right 21:47:28 entrypoints-plugins - i think we should consider a scope reduction there 21:47:35 instead of "entrypoints all the things" 21:47:50 entrypoints these specific things that we know will be done in havana (perhaps what is already completed) and call it a day 21:48:26 i'll follow up on that, it's not a quick status change ... 21:48:40 alright, let's jump to open discussion while we still have some time 21:48:45 i suspect next week we'll do this again 21:48:52 +! 21:48:57 err, +1 21:49:02 #topic open discussion 21:49:16 The current set of objects patches need some +As: 21:49:17 any other topics? 21:49:19 https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/compute-api-objects,n,z 21:49:20 rootwrap 21:49:24 and some reviews 21:49:32 rootwrap +1 21:49:33 oh yeah, rootwrap. 21:49:40 rootwrap is slow. :-/ 21:49:48 it's bad enough that we should do something. 21:50:05 TL;DR: 'time sudo nova-rootwrap /etc/nova/rootwrap.conf ls' takes 0.254 seconds 21:50:09 step 1) determine if there's anything in the code we can make more efficient to make an impact, or if it's just Python startup pain 21:50:27 status on step 1 ? 21:50:57 (context, rootwrap adds over 2 minutes to the time it takes to run 30 instances vs. when not using rootwrap) 21:50:58 I haven't looked into rootwrap internals 21:51:06 russellb: haven't dug into it too far but pyton looks fast to start up 21:51:22 do we have a bug for it yet? 21:51:24 does rootwrap import nova? 21:51:28 documenting the data you have so far? 21:51:28 cause that would be super slow 21:51:44 jog0: well, faster than a quarter second, but not fast like sudo, right? 21:51:52 russellb: yeah 21:51:59 lifeless: nah, it runs it as a separate process 21:52:03 running a one line python file takes 0m0.018s 21:52:03 python itself should be about 20ms 21:52:09 russellb: I know its a separate process 21:52:18 russellb: but if it sucks in bits of nova for extensions 21:52:21 jog0: with no imports? 21:52:24 which rootwrap is designed to do... 21:52:32 sdague: yes 21:52:33 oh, does rootwrap import nova, i read it the other way around, sorry 21:52:52 so maybe it's rootwrap reading all of its config files every time? 21:53:11 could probably get clever with that. 21:53:13 the config takes time and some regex take time 21:53:23 ... or make rootwrap a daemon ... 21:53:32 ran python -m cprofile on it yesterday 21:53:37 as ttx said, trimming the config to just network things would make it a little faster 21:53:39 from nova.openstack.common.rootwrap import wrapper 21:53:43 it's importing nova 21:53:48 so everything that nova.__init__ imports 21:53:56 and everything that nova.openstack.__ imports 21:54:10 and everything that nova.openstack.common.__init__ imports 21:54:26 e.g. huge 21:54:34 speeding up rootwrap by 3x wouldn't be enough o fa fix 21:54:45 it should be able to run in ~30ms 21:54:49 would that be fast enough ? 21:54:52 lifeless: it looks like all of those __init__.py files import nothing 21:55:00 IMHO, no 21:55:09 we run ten commands back to back when setting up things like networks 21:55:15 err 25 21:55:15 10 * 30 == 300ms 21:55:28 jog0: was choosing ten as an order of magnitude :) 21:55:46 we can trim some of what we're doing 21:55:47 jog0: please apply this patch http://paste.ubuntu.com/5888945/ 21:55:49 certainly better than 250ms per command. 21:55:55 that will give us a good idea 21:56:08 http://paste.openstack.org/show/40715/ 21:56:11 to give you an idea 21:56:40 and this is the result: long waits for iptables locks http://paste.openstack.org/show/40747/ 21:57:02 can someone also paste the bug link for this? 21:57:07 just want to make sure we have it in the log 21:57:32 ip route w/o rootwrap is 0.007 seonds 21:57:58 with 0.158 21:58:11 russellb: sorry https://bugs.launchpad.net/oslo/+bug/1199433 21:58:14 Launchpad bug 1199433 in nova "nova boot --num-instances=50 times out " [High,New] 21:58:20 #link https://bugs.launchpad.net/oslo/+bug/1199433 21:58:35