18:04:15 <aignatov> #startmeeting sahara
18:04:16 <openstack> Meeting started Thu Jun 19 18:04:15 2014 UTC and is due to finish in 60 minutes.  The chair is aignatov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:04:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:04:20 <openstack> The meeting name has been set to 'sahara'
18:04:28 <aignatov> ok, let
18:04:33 <aignatov> lets start
18:04:51 <aignatov> #link http://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:05:10 <aignatov> #topic News/Updates
18:05:19 <aignatov> please guys :)
18:06:20 <aignatov> nothing to update? dmitry, elmiko, tosky, mattf, crobertsrh tmckay ?
18:06:31 <dmitryme> not much from my side, just a few fixes here and there. Most I was busy with other stuff
18:06:43 <crobertsrh> Dashboard merge has picked-up some review momentum.  Sadly, it's largely another round of mostly nit-picky sorts of things.  My plan is to address the current set of -1s and let the horizon people know of our desire to 1) get the merge done timely and 2) take care of minor rafactorings post-merge 3) continue to add functionality.
18:06:57 <elmiko> i've been working on a solution to the swift/auth issue. i'm close to getting a mock up of some ideas.
18:07:26 <dmitryme> elmiko: can you share details on the approach you’ve picked?
18:07:29 <aignatov> crobertsrh: is there any chance to be merged your horizon work till juno-2 ? :)
18:07:56 <crobertsrh> The current plan is to either get merged by j-2 or "not at all"
18:08:23 <elmiko> dmitryme: i'm proposing that we use keystone trusts and delegation
18:08:52 <elmiko> dmitryme: i would like to put together a full spec though, so that we can debate it
18:08:53 <mattf> i've been busy w/ extra-sahara things. if folks are in san fran in a week, come by spark summit and we can talk spark + sahara.
18:09:07 <elmiko> i'm curious what the next step should be. i have a solid idea but i think we need to work out some details.
18:09:18 <themistymay> mattf: aww im in pasadena this week and next.  so close but so far
18:09:23 <dmitryme> elmiko: sure, I’d like to see the spec
18:09:37 <aignatov> wow! what happends with irc?
18:09:52 <themistymay> aignatov: lol, mass kick
18:09:53 <elmiko> super-split
18:09:56 <aignatov> so many guys left chat room
18:10:02 <dmitryme> aignatov: split-brain :-)
18:10:16 <aignatov> lol
18:10:29 <dmitryme> hope we didn’t lost somebody of our folks
18:11:00 <aignatov> yes, I hope too :)
18:11:03 <crobertsrh> I *think* I'm still here
18:11:05 <aignatov> bah
18:11:06 <themistymay> AAAAAAAAAND they are back
18:11:10 <tmckay> I'm here
18:11:17 <aignatov> cool :)
18:11:31 <mattf> spammy mc spamster
18:12:26 <aignatov> not so much news from me, I’m busy right now by other stuff at mirantis, only updated some pep8 erros in new hacking version…
18:12:40 <tosky> so netsplit? But the topic is wrong
18:12:50 <dmitryme> aignatov: seems like the meet bot (or what is its name?) gone mad: take a look at current topic
18:12:54 <tosky> maybe the old meeting is still open
18:13:04 <aignatov> # News ? Updates
18:13:06 <dmitryme> aignatov: try to set the topic
18:13:14 <aignatov> #topic Action items from the last meeting
18:13:23 <tosky> hyakuhei: could you please try again to close the meeting?
18:13:24 <dmitryme> cool, it works
18:13:24 <aignatov> yes, it works :)
18:13:44 <tosky> oh, nevermind
18:13:48 <aignatov> tosky: hyakuhei no, it set for sahara tight now
18:13:52 <aignatov> *right
18:14:03 <tosky> yes, sorry
18:14:17 <aignatov> so the items were
18:14:19 <aignatov> aignatov create bp re moving/updating rest samples docs and do it
18:14:21 <aignatov> SergeyLukjanov to create bp with steps to enable heat be default
18:14:22 <aignatov> SergeyLukjanov to create bp about removing/hiding username@image for heat based provisioning
18:14:22 <aignatov> tmckay to create bp re moving edp samples to sahara and make test jobs examples
18:14:31 <tmckay> done
18:14:36 <tmckay> just did it :)
18:15:08 <aignatov> I’m partially done, bp is created but code not yet :)
18:15:08 <aignatov> oh, nooooo
18:15:17 <themistymay> *facepalm*
18:16:10 <aignatov> I’m not sure if SergeyLukjanov finished his AIs
18:16:32 <aignatov> let’s move them to next meeting
18:16:39 <elmiko> if i get a spec together for the swift/auth issue, do we need to make it private?
18:16:49 <aignatov> #action aignatov create bp re moving/updating rest samples docs and do it
18:17:03 <tmckay> aignatov, okay, I did mine
18:17:10 <aignatov> #action SergeyLukjanov to create bp with steps to enable heat be default
18:17:14 <tmckay> don't know if you saw that before the split
18:17:26 <aignatov> #action SergeyLukjanov to create bp about removing/hiding username@image for heat based provisioning
18:17:36 <aignatov> tmckay: I saw it :)
18:18:02 <dmitryme> elmiko: good question. I’d say let’s keep it private until somebody makes a good point why it should be disclosed. It seems to be part of the bug discussion
18:18:05 <aignatov> #info DONE: tmckay to create bp re moving edp samples to sahara and make test jobs examples
18:18:13 <tmckay> update from me -- PTO starting tomorrow until June 30th.  Please everyone, look at my reviews for refactoring the job manager and moving edp engine choice to plugins
18:18:42 <elmiko> dmitryme: i will start by making a thorough explanation in the bug report, then we can debate from there
18:18:54 <dmitryme> elmiko: ok
18:19:10 <aignatov> #topic Review of backward compat - https://review.openstack.org/#/c/98260/
18:19:37 <aignatov> so, I think the main concern was from alazarev but he is not here :)
18:20:14 <aignatov> personally I think that plugin vendor should dictate his upgrade policy
18:20:27 <aignatov> so I’m mostly agree with jspiedel
18:20:34 <tmckay> +1, me too
18:20:52 <RobLevas> +1, me too
18:21:47 <aignatov>  I think alazarev and SergeyLukjanov have their own opinions but they are not here...
18:22:04 <aignatov> probably we need to reraise this question again on the next meetings
18:22:37 <aignatov> #action alazarev reraise discussions about backward compat https://review.openstack.org/#/c/98260/
18:22:46 <aignatov> let’s move on
18:22:47 <mattf> +1 ^^ reraise
18:22:52 <tmckay> +1 next meeting, I won't be here then but that's okay
18:23:15 <aignatov> #topic Pilot sahara-specs
18:23:30 <aignatov> do anyone know about this topic? :)
18:24:15 <aignatov> actually I missed something about that but saw patches on the review
18:24:26 <crobertsrh> I think tmckay was going to be writing up a spec to try the new system, right?
18:24:49 <tmckay> yes, I created 2
18:25:13 <aignatov> are you about edp pluggable model?
18:25:15 <tmckay> Very easy.  The template.rst in sahara-specs tells you what to do
18:25:24 <dmitryme> aignatov: frankly I am expecting an intro from SergeyLukjanov
18:25:30 <aignatov> ok
18:25:37 <tmckay> aignatov, yes, I made 1 about edp refactor and one about examples moving
18:25:51 <elmiko> doesn't the nova project have an intro about using the -spec system?
18:26:03 <aignatov> I think current topic is about something else but not about edp :)
18:26:16 <dmitryme> ok, I will read the template.rst as tmckay suggest … eventually :-)
18:27:04 <aignatov> move on to talk about edp or any  more thoughts abut specs?
18:27:13 <dmitryme> aignatov: I think you misunderstood tmckay - he says he already have experience with sahara-specs
18:27:22 <tmckay> yes, that's what I mean
18:27:29 <aignatov> oh, ok, tmckay, sorry
18:28:25 <aignatov> well lets move on
18:28:38 <aignatov> #topic Roadmap update / cleanup
18:28:50 <aignatov> any points aout this?
18:28:57 <aignatov> *about
18:29:31 <aignatov> #topic Open discussion
18:30:38 <aignatov> HW guys around?
18:30:44 <RobLevas> Here
18:31:00 <crobertsrh> Just a reminder to keep up with dashboard merge reviews in horizon.  There should be another round coming up soon.
18:31:19 <RobLevas> I will make sure Erik knows
18:31:41 <aignatov> I have questions about https://review.openstack.org/#/c/101050/
18:31:54 <aignatov> not actually questions but some thoughts
18:31:57 <dmitryme> tmckay: re https://etherpad.openstack.org/p/sahara_spark_edp. I have a feeling that we started design of determining which JobEngine to run too early
18:31:58 <RobLevas> I am working on that...
18:32:14 <aignatov> I think you should split patch on to several ones
18:32:17 <RobLevas> aignatov: I am working on that - sorry new to the workflow here
18:32:23 <dmitryme> I mean we can pick the dumbiest option right now and it will work
18:32:37 <aignatov> it’s really hard to review about 1000 lines of code :)
18:32:56 <RobLevas> Did I add that much?
18:32:56 <dmitryme> on the other side, from your implementation I can see that it is extracted into a separate function
18:33:01 <tmckay> dmitryme, true.  I had an idea I thought would work, so I tried it out
18:33:37 <tmckay> dmitryme, we can go with a simple option for now, but the plugin call with defaults is almost as simple I think
18:33:38 <RobLevas> aignatov: the bulk of the code is in a new class
18:33:44 <dmitryme> also, later, when we will have a more specific use cases, we will have a better vision on what is best
18:33:47 <aignatov> roblevas: in general your patch looks good, but splitting it you’ll make review process more faster :)
18:33:52 <tmckay> dmitryme, I'm open to other solutions
18:34:10 <tmckay> but I'm leaving on PTO so I wanted to get something down :) dmitryme
18:34:17 <RobLevas> aignatov: I am open to suggestions. :)
18:35:42 <aignatov> roblevas: can I make it more later, in comments to your patch, is it ok for you?
18:35:50 <RobLevas> aignatov: there are really 3 large chunks of code.. the class I added to implement the feature, the metadata and config options for the feature, and test code
18:36:01 <RobLevas> aignatov: sure
18:36:46 <RobLevas> aignatov: the number of lines would be smaller if we could have more than 79 characters per line.. ;)
18:37:14 <aignatov> roblevas: indeed, but policy is policy :)
18:37:34 <dmitryme> tmckay: ok, I just wanted to say that I don’t really have an opinion what is good there. I believe that later, when we have use cases, it will be more clear. And by ‘later’, I mean at least after 3 month, or maybe even 6
18:37:57 <RobLevas> aignatov: I know... just kidding... I am a fan of coding conventions and will follow blindly
18:37:58 <aignatov> “..implement the feature, the metadata and config options..” - that’s already 3 patches :)
18:38:51 <RobLevas> aignatov: I guess you can work it that way.. but would you split out the feature code and it's tests?
18:39:42 <RobLevas> aignatov: or maybe I misunderstand what you mean by patch.
18:39:51 <aignatov> feature code should go with tests, yes
18:40:16 <RobLevas> aignatov: I can submit multiple patches for a single blueprint?
18:40:33 <aignatov> I meant 3 new smaller change requests instead of largest one
18:40:46 <aignatov> roblevas: sure!! that’s the best approach
18:40:56 <RobLevas> aignatov: sorry, newbie here.
18:41:28 <RobLevas> aignatov: so I should retract my patch, and create 3 smaller ones
18:42:40 <aignatov> roblevas: you don’t need to say sorry, everething is ok, you already mage a huge work as new sahara contributor :)
18:43:02 <RobLevas> :)
18:43:45 <aignatov> you could use current patch as one patch in the smallest new 3 patches
18:44:12 <RobLevas> i dont understant
18:45:36 <aignatov> lets say you have current patch A with (feature 1, feature 2, feature 3), you decided to split it into 3 new pathces
18:46:00 <aignatov> so you could reuse patch A as implementation of feature 1 for example
18:47:07 <aignatov> dmitryme: tmckay so what would be our suggestion for implementing new edp model?
18:47:50 <dmitryme> aignatov: sorry, I don’t understand your question
18:47:52 <tmckay> aignatov, dmitryme, well, I am hoping to get simple spark execution working, using shell commands
18:48:05 <tmckay> to use with the spark provisioning plugin
18:48:13 <tmckay> as a first step for spark edp
18:48:25 <tmckay> Then maybe we can look at ooyala later on, or something else
18:48:59 <aignatov> dmitryme:  I meant approach how to determine job engine during strating job exeution :)
18:49:40 <aignatov> tmckay: seems to be a very good plan :)
18:49:57 <dmitryme> aignatov: ah, then as I said, right now any approach works for me. The only thing I think is important is to decouple engine selection from the rest of the code and it is already done actually
18:50:43 <dmitryme> that will enable us to choose another approach if we consider current as not suitable
18:50:54 <tmckay> aignatov, yes, for now we can just do by job type.  I think later, for example when HDP does oozie and spark both, it will be more complicated in the hdp plugin.  "Is the oozie server installed? etc"
18:51:43 <tmckay> vanilla right now is "always oozie".  But, if we add spark to vanilla as an option, we'll have to check if spark is installed on a cluster, for example
18:51:56 <tmckay> for spark plugin, oozie is never an option (currently).
18:52:07 <aignatov> tmckay: yep, we already have validate_edp for that
18:52:14 <aignatov> in plugin spi
18:52:22 <tmckay> unless provisioning plugin for spark adds mapred dep as an option (but I don't think so)
18:52:55 <tmckay> so basically, default choice is by job type now, but it is overloadable in the plugin with the patch I put up
18:53:03 <aignatov> my idea was to crate new field in cluster or in job execution object where we could define job engine
18:53:12 <aignatov> and just validate it on starting
18:53:15 <aignatov> job
18:53:22 <tmckay> we could do that too
18:54:01 <tmckay> cluster makes more sense to me.  User shouldn't be forced to pick the engine
18:54:19 <aignatov> tmckay: agree
18:55:22 <tmckay> I have to go.  Bye!
18:55:33 <aignatov> actually it’s just an idea, and in fact you are the main architector of pluggable edp right now, I will agree with any decisions, as for start I’m ok with current your idea :)
18:55:42 <aignatov> tmckay: bye!
18:56:04 <dmitryme> tmckay: have a good vacation!
18:57:03 <aignatov> roblevas: I think you didn’t undesrstand and I want to explain you what I meant about splitting patches :)
18:57:15 <elmiko> tmckay: later
18:57:19 <aignatov> lets talk about it in sahara channel
18:57:40 <RobLevas> aignatov: thanks... I can also ask some of my coworkers so we dont have to go over it here
18:57:43 <aignatov> so, guys, anything else to discuss today?
18:58:16 <aignatov> I think we can finish :)
18:58:29 <aignatov> thank you all! bye!
18:58:45 <aignatov> #endmeeting