21:00:16 #startmeeting nova 21:00:17 Meeting started Thu Nov 8 21:00:16 2012 UTC. The chair is vishy. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:18 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:19 The meeting name has been set to 'nova' 21:00:25 is anyone here today? 21:00:31 I'm here 21:00:37 me 21:00:39 * Vek waves 21:00:40 hi 21:00:41 o/ 21:00:55 well hello everyone! 21:00:56 hello 21:01:20 #topic Agenda 21:01:24 #link http://wiki.openstack.org/Meetings/Nova 21:01:32 * devananda waves 21:01:43 comstud: are you here? 21:01:58 skipping the first topic to give him time to show up 21:02:07 we may have a DST casualty 21:02:15 #topic Baremetal Driver 21:02:39 here 21:02:43 so devananda let me know the other day that hp is putting a group of people specifically on bare-metal 21:02:49 sort of here I should say 21:03:00 yep, we are 21:03:05 so he was wondering about just merging the existing code and cleaning it up afterwards 21:03:20 big -1 to that idea ... 21:03:22 so collaboration is easier 21:03:31 couple of points 21:03:32 vishy: here 21:03:47 first, i don't think we should sacrifice review quality, every, especially for something really big 21:03:54 agreed 21:03:58 and second, why can't collaboration happen outside of trunk? 21:03:59 +1 21:04:04 thing is, it's not that easy for _us_ to clean up someone else's review 21:04:06 Why not just use a separate github repo until we're closer to the point of needing reviews? 21:04:07 s/every/ever/ 21:04:18 devananda: have you tried to talk to them? 21:04:24 russellb: yep 21:04:32 they've accepted 3 pull requests fo far for small things 21:04:38 and squashed them into their reviews w/o any attribution 21:04:49 not that that's a big deal for small things 21:05:00 but when we throw 4 or 5 devs full time at this, it's kinda not so good 21:05:24 so definitely a collaboration challenge, but throwing it into trunk before it passes code review can't be the answer ... 21:05:49 as long as the review is owned by not-us, how do we develop/improve it at a reasonable pace and using the tools we're all used to? 21:06:18 one option would be put it on stackforge 21:06:25 sure, could do that 21:06:29 and bring it back in once it is clean 21:06:38 I think that's a much better idea 21:06:58 though someone is going to have to do some squashing if you go that route ... 21:06:58 agree 21:07:05 didn't we once upon a time have a discussion about long-running branches? 21:07:25 so anyway, stackforge, or just a github repo 21:07:32 no matter the tool, the collaboration problem is going to still exist 21:07:37 o/ 21:07:51 how does stackforge / squashing work w.r.t retaining commit history / authorship? 21:08:00 when N people at M companies are collaborating on something? 21:08:18 looking at out-of-trunk forks of things like reddwarf, where >1 company work on it, it looks like fail to me 21:08:27 we need to standardize on a header like "Co-Authored-By" or whatever 21:08:36 yeah, that's a different problem, IMHO 21:08:42 solved in other communities without much trouble 21:08:46 so a 10k line patch becomes co-authord by 10 ppl? 21:08:47 indeed 21:09:03 honestly, 21:09:17 I think we should shoot for smaller chunks anyway, shouldn't we? 21:09:29 10k is ridiculous :) 21:09:33 devananda: given our gating, I just don't see any other way of doing it, and +1 to dansmith re smaller chunks 21:09:35 dansmith: the crrent NTT reviews are around 1 - 2k each, i think 21:09:47 devananda: yeah, which are still too large, IMHO 21:09:48 granted, none of them actually _work_ without the others 21:09:55 dansmith: smaller chunks doesn't really help if a lot of collaborating is happening on each chenk 21:10:01 * chunk 21:10:12 functionally, we can't test or develop any of these pieces independently of eachother 21:10:18 vishy: they can be smaller chunks before merge and larger for collab, no? 21:10:33 so, fork it if you have to. if they won't take your pull requests, and you honestly produce a better end result, we merge that? 21:10:45 i'd really like to see everyone work together and play nicely though :) 21:10:48 :( 21:10:58 russellb: boooo 21:11:08 NTT and ISI have been very eager to work with us 21:11:26 so heres the problem 21:11:26 there is a large code change 21:11:27 with lots of moving parts 21:11:30 is the problem that they won't attribute, or that their attribution isn't enough for you? i.e. you need git authorship 21:11:31 comstud: boo? 21:11:48 playing nice removes drama i like to watch 21:11:48 it - frankly - needs a massive overhaul, to reduce duplication with nova's core, make better use of e.g. quantum. 21:11:56 heh 21:12:00 the folk working on it are playing nice with each other but: 21:12:07 dansmith: no. attribution is a part but not the whole issue. dev pace is the bigger problem IMO 21:12:07 comstud: heh, well as long as we agree that drama is not a reason to merge a huge amount of code that isn't ready 21:12:09 - we have no place to record bugs. 21:12:13 haha 21:12:18 ^ ++ to bugs on LP 21:12:18 devananda: Error: "++" is not a valid command. 21:12:28 gah 21:12:34 - attribution is going to be a PITA when it lands: if we flatten everything, a dozen or more folk will show as one commit 21:12:38 tons of projects seem to have good pace without openstack infrastructure :) 21:13:05 so, I've been mostly ignoring the patches, 21:13:06 - if we don't, its going to be many hundreds, if not thousands of patches landing as a single rebased set on trunk 21:13:10 but I agree with lifeless that they seem to need an overhaul 21:13:28 there are scaling and design issues that happened because the core thing was written off in a silo 21:13:29 that has been my impression as well 21:13:35 rather than a bunch of incremental work on trunk. 21:13:35 it definitely needs an overhaul, but us forking NTT's code and doing that is the wrong way to collaborate with them 21:13:45 hence my concern on just merging it and sorting out the survivors later 21:13:47 lifeless / devananda is it possible to implement it with a small driver portion in trunk and the majority a separate install? 21:13:53 The longer we keep it out of trunk *the worse the problem gets* 21:14:16 vishy: i'll take another look at it from that angle, but off-hand, no. 21:14:18 So, my suggestion here is to figure out how to mitigate the fallout on *non-baremetal-stuff* in trunk, then land it wholesale 21:14:24 is there any way to handle this using branching, i.e., branch to a monster topic branch, handle all the collaboration there, then once it passes tests in situ and conflicts have been resolved, then we just merge it into core? 21:14:31 and fix in situ once the structural problem has been resolved. 21:14:51 we could do a special case ff-merge i supose. 21:14:56 vishy: we might end up with that, my concern as above is to remove the structural thing that is exacerbating the problem every day 21:15:17 vishy: like - I want to totally delete the baremetal-deploy-helper 21:15:28 yep. that's big on our plan 21:15:32 vishy: but to do that requires work to get baremetal w/quantum in play 21:16:06 ok so it sounds like there is opposition to the jfmi (just freaking merge it) plan 21:16:12 vishy: thats tricky at best with baremetal something that you need a branch of devstack, adhoc sql etc to get going : and we're pushing that stuff upstream as fast as its bits become relevant to the various projects 21:16:14 huge opposition from me 21:16:21 fyi the 1st patch is about to co in 21:16:26 which will help a bit. 21:16:43 scheduler one? 21:16:44 vishy: you mean the scheduler change? 21:16:50 devananda: yes 21:16:58 https://review.openstack.org/#/c/13920/ 21:17:51 so, one thing - would you guys be happy with bugs in the baremetal branch being in main nova launchpad bugs? tagged baremetal ? 21:17:59 That would remove one part of the issue around collaboration 21:18:12 lifeless: I don't mind that, although why not use github issues? 21:18:15 (and give data for folk concerned about performance and design) 21:18:30 vishy: because we'd need to migrate the bugs to LP as each commit gets merged. 21:18:33 vishy: thats pretty wasteful 21:18:52 LP bugs are fine, as long as they tagged well enough 21:18:52 vishy: I have much better things to do than to write bug migration tools between disparate systems ;) 21:18:53 lifeless: if we have a long running branch as part of main repo then using main nova LP bugs makes sense too IMHO 21:18:56 hopefully it's not too high volume 21:19:13 russellb: you can exclude baremetal tagged bugs from any subscription you have 21:19:15 we have enough bugs to go through as it is ... so hopefully those involved triage them and keep them in good shape 21:19:16 russellb: and then ignore them 21:19:25 sounds like work :) 21:19:28 russellb: I will commit to doing that if you like 21:19:33 k 21:19:39 i'm fine seeing them 21:19:44 i have trouble seeing how it will make a huge difference but if it helps i don't mind 21:19:48 i just don't want 20 baremetal bugs in my queue of stuff that hasn't been triaged 21:20:14 I've just setup a subscription just for them 21:20:31 As long as the nova PTL etc are happy with me getting in there, consider them triaged. 21:20:42 devananda: you guys are from the same company as mtaylor so you could try to convince him to set up another branch in nova where you can collaborate. 21:20:53 if you really feel like it is worth it over github 21:21:09 at least for now it'll probably be largely us (me/lifeless) managing the barmetal bugs 21:21:09 russellb: i feel like this is another case for a more general nova-compute (splitting at driver layer) 21:21:10 vishy: we can do that, but the question is a nova project one not CI plumbing :) 21:21:13 so who gets +2 rights on the branch? 21:21:30 it would need a new group nova-baremetal 21:21:30 vishy: not quite sure i follow what having another branch gets us 21:21:37 honestly I think it makes it more complicated 21:21:46 vishy: who approves who goes into nova-baremetal ? :-) 21:21:46 besides that :) 21:21:50 +1 on it being complicated 21:21:51 you could use all of the normal openstack tools. 21:22:15 if i were doing it though i would just start a shared org on github and put the code there 21:22:19 and do pull requests 21:22:24 same 21:22:33 I'm thinking this approach might apply to cells and whatever the NBT is.. perhaps it's worth documenting the process, 21:22:33 let me repeat the problem statement 21:22:43 and the conditions for being considered for this special treatment? 21:22:59 the problem is that there is a combination of A) lots of code, B) lots of organisations, C) *not* getting review by the reviewers of its final destination. 21:23:10 that's easy enough for us _IF_ we wanted to fork nova and maintain our own project outside of trunk. but we think that's a bad idea 21:23:24 C is the crux: any work we do without C being inverted, means review debt for when the final merge happens. 21:23:49 so it sounds like we need some core volunteers 21:23:52 to join the effort 21:24:06 yep 21:24:12 I'd be delighted to work towards being a nova core 21:24:17 same 21:24:18 but you guys don't know me yet :) 21:24:26 the obvious candidates are markmc and myself 21:24:38 since we've reviewed the code quite a bit 21:24:38 right, you guys are already providing feedback 21:24:49 i think devananda has been doing a great job with reviews also 21:24:55 so i will volunteer markmc 21:24:57 :) 21:25:06 heh, who's not here to decline 21:25:10 but he has been involved a good bit already 21:25:15 perfect it is decided 21:25:20 seriously though 21:25:39 devananda/lifeless, can you guys ping me if you need help on specific points? 21:25:44 and markmc as well 21:26:03 I think if you guys attack it from a cleanup perspective we will be happier with it in general 21:26:15 but i don't think we've solved the collaboration problem 21:26:22 russellb: exactly 21:26:27 why not 21:26:34 shared org on github isn't good enough? 21:26:40 oh, i think it is 21:26:50 i just don't know that i heard anyone agree with you that it was good (other than me) 21:27:31 vishy: perhaps i'm just missing the end part 21:27:41 let's say we do the shared github 21:27:50 with some nova-core involvement hopefully 21:28:01 lifeless, myself, and other hp folk do lots of work, and NTT/ISI guys do some work, 21:28:07 and a few core people kinda watch what we do 21:28:20 and eventually it has to be reviewed again 21:28:23 a) there's no gerrit to realy get people to review things 21:28:37 github has code review stuff in pull requests 21:28:57 eh, ok. with human gate? 21:29:03 what about the CI part? 21:29:12 right. that was going to be my (b) 21:29:28 no CI. easy for anyone to break it and not know 21:29:40 persumably you can still have a system run periodic test runs on the branch 21:29:45 lastly, what's the final path to merging it into trunk in, say, a few months 21:29:46 smokestack can do that on arbitrary git repos 21:29:58 it has to get reviewed just like everything else IMO 21:30:13 russellb: yes, it does. i just hesitate to force us to use different tools 21:30:14 devananda: so it sounds like you are pushing for another branch in ci 21:30:31 which i thought we just decided wasn't worth it... 21:30:51 * devananda scrolls back 21:30:59 so, remind me: now that this is all done, it still doesn't seem like something that can be broken up into much smaller pieces and submitted normally, is that right? 21:31:39 dansmith: it would be hard to do so correct 21:32:05 dansmith: but if the code is only new files we could just ff merge the whole thing 21:32:21 hopefully things that affect other parts of core code are submitted in pieces along the way 21:32:25 right, so, 21:32:30 and then in the end it's a fairly self-contained review 21:32:35 (even if big) 21:32:38 such is life 21:32:39 if we could get the core changes submitted normally, either before or after the whole-file bits, 21:33:04 i think that is the way to go, do core changes against trunk 21:33:11 then it seems preferable to do that over this other approach, 21:33:13 other stuff separately 21:33:17 dansmith: so I think it *can* be split out, but the fundamental problem there is that what exists today is kindof shortest-path-to-solution rather than best-path 21:33:21 which will end up with a lot less visibility into all of the changes, I think 21:33:35 having climbed through it 21:33:36 lifeless: yeah, understand, but people do this all the time 21:33:51 lifeless: they write it one way, learn a bunch, and then have to make it into something that's actually acceptable :) 21:33:55 (or reviewable) 21:34:04 +1 :) 21:34:16 dansmith: sure; the scale here is perhaps the root of the issue ;) 21:34:35 the scale makes it more important, IMHO :) 21:34:55 so i don't thinke we have really come up with a solution yet 21:35:21 well, nothing is easy, but the way I see it ... 21:35:27 dansmith: right 21:35:27 1) no, it can't go into trunk now, it's not ready 21:35:32 lifeless / devananda: i think you should just focus on cleaning up each individual review already in queue 21:35:38 in fact, it would be almost better to start over:) 21:35:46 2) keep making it better, submitting core changes as pieces as you can 21:35:51 lifeless: sometimes it is! :) 21:35:59 get them up to snuff, then you can refactor to your hearts content using the normal review process. 21:36:02 [not kidding - identify each problem that needs solving, which we have a good list of, and do targeted work to solve it] 21:36:51 lifeless / devananda: It sounds like anything else will really slow things down 21:36:57 you guys could start poaching bits out of the existing reviews, 21:37:07 fixing them and submitting them against core with proper attribution, right? 21:37:12 vishy: that route means we do our testing/devel off of their fork, giving them pull requests, and waiting for them to update the reviews 21:37:20 then the NTT folks would remove that bit from their patch and re-post 21:37:34 wash, rinse, repeat until the actual patches are small and specific 21:37:40 vishy: unless i am misunderstanding 21:37:50 devananda: not really, you can cleanup and push over them if they dont' mind 21:38:24 vishy: i'll ask. might be ok, and would simplify things 21:38:26 devananda: but I'm suggesting not to do the long term "right" cleanup 21:38:34 but the cleanup to make it mergable 21:38:41 vishy: what is the bar for that ? 21:38:49 i.e. a) it works b) doesn't break other core functionality 21:38:52 ok 21:39:00 so thats what I was proposing we do. 21:39:04 c) doesn't do really silly ugly python etc. 21:39:05 vishy: is it not there right now? jenkins is passing, isn't it? :) 21:39:10 gah 21:39:12 (c) :P 21:39:27 so we can do c) easily enough, and a and b are in play already I believe. 21:39:36 honestly I don't have huge issues with the other patches in the queue 21:39:41 russellb: ^ is this sufficient for you? 21:39:46 I'd like to minimize them touching core code 21:39:55 which is why we've spent so long on the first one 21:39:59 maybe ... I haven't looked at the code very closely. 21:40:10 obviously the stuff that touches code outside the driver should be reviewed the most strictly 21:40:46 but even the driver itself, there should be a quality bar ... 21:40:46 russellb, vishy, afaik the only patch touching nova core is the first one, which you said was about to merge anyway 21:40:50 beyond "it works" 21:40:58 the others are all driver specific, or extra helpers in /bin/ 21:41:08 yeah, but even the rest, have to consider the maintenance burden 21:41:18 having all these new people committed to working on it helps that 21:41:23 russellb: I think we have a ton of volunteers helps 21:41:30 yep 21:41:35 devananda has been very helpful to nova in general 21:41:41 i'm not going to say "sounds good" because i haven't looked at the code enough yet 21:41:44 and it sounds like lifeless is getting up to speed 21:41:55 aratu has been doing good work as well 21:42:11 they have a lot of people that will be valuable to nova in general imo 21:42:12 but in general, as long as we're not merging something before it meets the same quality standards everything else is held to, then I'm good with it 21:42:20 that was my #1 concern this meeting 21:42:35 russellb: so thats what confuses me :) 21:42:36 russellb: fair enough. I think we should treat it as we do all other drivers though 21:43:00 I can tell you that at a macro scale its quality is poor, but we can fix the micro scale easily. 21:43:03 as long as it is tested and has people working on it we don't need to get every inner part up to code and design quality standards. 21:43:12 Micro scale quality is a poor indicator for macro quality. 21:43:32 ok lets move on for now 21:43:38 we will revisit next week 21:44:00 lets try to get patch 1 in this week and then we can see how easy the others are 21:44:29 #topic Cells Status 21:44:36 oh hi 21:44:38 similar type of topic 21:44:40 :) 21:44:42 haha 21:44:56 I have updated the blueprint spec with additional information (config options and so forth) 21:44:56 unfortunately i haven't had time to look at this code 21:45:02 I'm working address some of the current concerns 21:45:04 how are we doing with it so far? 21:45:15 I have enough feedback that I'll be busy for a couple days yet 21:45:20 I am out tomorrow and this weekend, though. 21:45:23 i wanted to review this week, got sidetracked by a security release and a frustrating qpid+eventlet bug ... 21:45:30 will try again this week 21:45:54 i'm guessing i'll have it updated Monday 21:46:01 might want to wait until then 21:46:21 ok 21:46:22 Should have some additional info in doc strings to help reviewing. 21:46:39 oh eventlet, how I love thee 21:46:52 comstud: ok cool, so we are waiting on you for the moment 21:46:58 #topic Nova bugs 21:46:59 in any case, I'm not hurting for any more feedback at the moment 21:46:59 :) 21:47:08 heh 21:47:15 ah, cool, then i don't feel as bad :-p 21:47:28 ya, no worries. i'm still working on splitting the 4.5K line patch up also 21:47:29 doing pretty well 21:47:31 #link http://webnumbr.com/untouched-nova-bugs 21:47:38 russelb is the big winner this week! 21:47:44 \o/ 21:47:48 hah 21:47:51 #link http://www.stillhq.com/openstack/nova/triage-20121109.txt 21:48:10 i did it for the stats, no other reason 21:48:16 any important notes on bugs? 21:48:17 and the chicks 21:48:31 vishy: haven't come across any new big ones worth noting here 21:48:38 i beat vish, at least. 21:48:47 oh, there was one weird security thing that nobody has been able to reproduce 21:48:48 comstud: self-triage doesn't count! 21:48:52 haha 21:48:53 :p 21:49:05 #topic grizzly-1 planning 21:49:10 i find em and I fix em! 21:49:15 https://bugs.launchpad.net/nova/+bug/1074343 21:49:16 Launchpad bug 1074343 in nova "ec2 describe instances does not filter by project_id" [Undecided,Incomplete] 21:49:22 that was the potential security thing worth looking at 21:49:35 https://launchpad.net/nova/+milestone/grizzly-1 21:49:56 #link https://launchpad.net/nova/+milestone/grizzly-1 21:50:07 so some stuff probably needs to move out of grizzly 1 21:50:25 I had a question: 21:50:31 don't we need to break up no-db-compute a bit? 21:50:41 into some pieces we can actually mark as done at some point? :) 21:50:51 dansmith: sure, that'd be good 21:50:54 dansmith: we could 21:51:09 dansmith: or we could just target it to g-3 :) 21:51:36 vishy: well, it's still just a huge chunk of work, seemingly too large to be one bp 21:51:39 besides, 21:51:49 I'm jealous that russellb gets to have his name all over it :) 21:52:03 and launchpad keeps mocking that I "have no assigned blueprints" ;) 21:52:03 seems reasonable to have some sub-bps 21:52:27 isolate-virt-drivers-from-db could be one ... final-bits-of-no-db-messaging another ... 21:52:31 from there it gets complicated :) 21:52:38 next is figure-out-the-end-of-no-db-compute 21:52:42 split-nova-compute 21:52:48 there ya go.. 21:53:01 want to file those? 21:53:11 sure 21:53:21 ok i moved out the ones that i'm pretty sure aren't g-1 21:53:21 ping me and i can approve and such 21:53:25 vish gave me magic powers 21:53:32 heh 21:53:35 jog0: ping 21:53:42 vishy: pong 21:53:47 jog0: I targetted teh remove nova volume one to you 21:53:55 vishy: I just saw, thanks 21:53:56 since you have been submitting the remaining work 21:54:22 jog0: are you still doing aggregate based availability zones? 21:54:28 vishy: yes 21:54:31 cool 21:54:37 haven't had a chance to dust off the code yet though 21:55:03 i don't know what happened to mtaylor's entry points stuff 21:55:20 but aside from cells and bare metal which we discussed everything else is looking decent. 21:55:22 it all expired; guess he hasn't had time to work on it 21:55:30 performance problem stalled it 21:55:47 ya 21:55:50 still waiting on code for https://blueprints.launchpad.net/nova/+spec/server-count-for-nova-flavors 21:56:07 i think hp has that somewhere 21:56:35 anyway i don't think there is anything horribly out of line 21:56:42 # topic General Discussion 21:56:49 4 minutes! 21:56:52 #topic General Discussion 21:57:27 nova is busy. 21:57:39 fun times. 21:57:40 *nod* 21:59:22 i think i have to stop getting every review email 21:59:30 heh 21:59:33 that's intense 21:59:36 takes me most of my day just keeping up with email 21:59:43 vishy: you don't have to read them all :) 21:59:50 * Vek has had days with 100+ nova review emails, right after having done a review day 22:00:06 can't imagine what it would be like to get 'em all 22:00:12 I get them all 22:00:22 you just have to be selective :) 22:00:24 mark all as read is your friend 22:00:34 scoring/tagging is your friend 22:00:46 killfile is your friend :) 22:00:52 woah now, that's advanced 22:01:00 hehe :) 22:01:46 vishy: #endmeeting? 22:01:55 seconded. 22:02:04 thanks a lot everybody :) 22:02:09 #endmeeting