14:01:00 <SergeyLukjanov> #startmeeting sahara
14:01:01 <openstack> Meeting started Thu Jun  4 14:01:00 2015 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:04 <openstack> The meeting name has been set to 'sahara'
14:01:14 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
14:01:26 <SergeyLukjanov> #topic sahara@horizon status (crobertsrh, NikitaKonovalov)
14:01:32 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-reviews-in-horizon
14:01:39 <SergeyLukjanov> is it actual list?
14:02:12 <NikitaKonovalov> SergeyLukjanov: the changes on review a still there
14:02:21 <crobertsrh> A few bug fixes and enhancements I've added, on review of course.
14:02:31 <crobertsrh> I did talk with the horizon folks.....
14:02:31 <tmckay> I mean, hi folks :) (wrong window)
14:02:52 <crobertsrh> They are thinking that for L, a single core +2 combined with enough +1s from us will get a patch merged.
14:03:25 <SergeyLukjanov> crobertsrh, it'll be great IMO
14:03:26 <elmiko> nice
14:03:27 <NikitaKonovalov> crobertsrh: that's gread
14:03:32 <NikitaKonovalov> great*
14:03:35 <SergeyLukjanov> crobertsrh, and add you to the core team ;)
14:03:45 <crobertsrh> heh
14:04:05 <elmiko> +1
14:04:22 <kchen> hi all, these days I found the gate-sahara-neutron-direct-spark-aio test always failed, is it a bug? was it resolved now?
14:04:48 <elmiko> alazarev linked to a bug patch for that issue, in one of the reviews
14:05:15 <alazarev> https://review.openstack.org/#/c/187155/
14:06:42 <SergeyLukjanov> #topic News / updates
14:06:55 * SergeyLukjanov recovering from the vacation prev. week
14:07:23 <tmckay> SergeyLukjanov, any update on Sahara-Ironic integration blog post / patches?
14:07:24 <crobertsrh> That is what work is for, right?  Recovering from vacation?
14:07:49 <SergeyLukjanov> tmckay, blog post is on review for the blog :(
14:07:55 <elmiko> i've been researching the improved secret spec more, and also working on a spec for keystone session objects. i have a few questions about the latter, for the group.
14:08:12 <tmckay> SergeyLukjanov, well, review is good! :) Almost done then.
14:08:23 <SergeyLukjanov> tmckay, we've not yet worked on the patches, we hope to attach volumes using cloud-init
14:08:45 <alazarev> rewrote heat engine to use ResourceGroup, please review https://review.openstack.org/#/c/186543/
14:08:47 <vgridnev> still working with auto-tune hadoop configs, for vanilla ready for review
14:08:54 <vgridnev> #link https://review.openstack.org/#/c/177280/
14:08:54 <SergeyLukjanov> I'm now working on a few specs we were discussing on summit, hopefully will start to publish them soon
14:09:33 <tmckay> There are some NetApp guys near me physically, we are going to meet face to face and brainstorm about Sahara/Manila integration.  There was a Manila talk at Summit that mentioned Sahara integration, so I followed up with them.  I'll let you know how it turns out.
14:09:47 <tellesnobrega> i just started working on the multiple clusters change, we need to discuss some issues before i can start implementing
14:09:58 <SergeyLukjanov> tmckay, cool, looking forward for info
14:10:13 <tmckay> I am wondering if we can get some security benefits from it (multi-tenancy, file shares scoped to users maybe) and also some file transfer effiiciency (job binaries mounted in the cluster)
14:10:52 <tmckay> and maybe, hdfs shares (intel wrote a Manila hdfs driver I think) can provide a simple way for users to set up shared libraries for Hadoop and host them in a long-running hdfs share
14:11:03 <tmckay> that could be linked into multiple clusters.  Things like that.
14:11:20 <tellesnobrega> tmckay, cool
14:11:26 <SergeyLukjanov> #topic Open discussion
14:11:33 <SergeyLukjanov> tmckay, sounds interesting
14:11:42 <elmiko> can we talk about keystone session objects for a few minutes?
14:12:01 <weiting> Yes, we did hdfs share in Manila.
14:12:30 <SergeyLukjanov> elmiko, I don't see sreshetnyak and apavlov online now
14:12:30 <egafford> Working on interface map; pretty much down to tests and review revisions @ this point (client and Horizon changes to follow).
14:12:55 <elmiko> SergeyLukjanov: ok, i'll keep working on the spec. maybe we can talk next week about it
14:13:04 <tmckay> egafford, I am trying to run against that but I hit a snag with my neutron based stack :(
14:13:15 <tmckay> but, I am looking at it.  Looks good so far
14:13:23 <weiting> And we can discuss it how to use it in sahara.
14:13:24 <egafford> tmckay: Interesting; I'm also using Neutron. We should talk.
14:14:05 <egafford> tmckay: (Also, I'm sure there are bugs yet, as I posted a working happy path WIP without even vaguely adequate testing, and knew it.)
14:14:10 <tmckay> weiting, certainly
14:14:27 <tmckay> egafford, ack on WIP, I understand
14:17:11 <tmckay> SergeyLukjanov, are you working on a spec for moving plugins out-of-tree?
14:17:18 <SergeyLukjanov> tmckay, yup
14:18:02 <SergeyLukjanov> in my list now - resources ACL, extract plugins, extract scenario tests,
14:18:36 <tmckay> okay. We talked about that a little more, and what impact multiple versions of plugins supported against multiple versions of Sahara might do to the CI matrix.  How do we test it all?  Who tests it?
14:18:41 <SergeyLukjanov> I've got the most powerful jetlag this time after a few days at home :(
14:18:45 <tmckay> heh
14:19:07 <tmckay> but we can ask such questions on review ^^
14:19:48 <SergeyLukjanov> tmckay, it's one of the most important questions, I'd like to start spec with a basic ideas and grow it to the state it cover all questions
14:19:50 <elmiko> yea, i would imagine those will be addressed in the spec
14:19:55 <SergeyLukjanov> because it's very huge and important thing
14:20:02 <elmiko> SergeyLukjanov: +1
14:20:16 <SergeyLukjanov> so far, I think it could be done be min set of tests vs not all plugins
14:20:23 <SergeyLukjanov> for the stable branches
14:20:43 <SergeyLukjanov> we don't need to test all plugins, we need to check that api isn't changed
14:22:56 <SergeyLukjanov> probably it's a good time to chat about multiple cluster creation
14:23:00 <SergeyLukjanov> tellesnobrega, ^^
14:23:04 <tellesnobrega> sure
14:23:06 <tmckay> okay.  Looking forward to the spec :) thanks
14:23:22 <tellesnobrega> we need to define a couple things
14:23:46 <tellesnobrega> first is to decide if we implement the change on the sahara side or horizon and client side
14:24:29 <tmckay> tellensnobrega, sorry, what exactly does "multiple cluster creation" mean? I can already create multiple clusters :)
14:24:38 <crobertsrh> I think I need more details
14:24:52 <tellesnobrega> ok
14:24:57 <crobertsrh> Any feature that we have should eventually be mirrored in the client lib and horizon
14:25:16 <tellesnobrega> the idea is to put a field to allow the user to select how many cluster with that configuration he wants to create
14:25:55 <tmckay> ah, ok.  So it's like pushing the launch button multiple times.  We need a naming convention
14:26:04 <crobertsrh> Is there a real use case for such a feature?
14:26:22 <alazarev> in nova this is implemented on server side, REST has a field "count", as I understand the proposal is to make the same in Sahara
14:26:23 <tellesnobrega> i can think on research labs (like the one i work)
14:26:35 <tellesnobrega> alazarev, exactly
14:26:49 <tmckay> crobertsrh, the motivation would be fewer clicks and forms
14:26:59 <tellesnobrega> sometime we need to create more than one cluster to process different sets of data and so on
14:27:24 <SergeyLukjanov> tellesnobrega, and clusters will be exactly the same>
14:27:26 <tmckay> same image, same ssh key, etc.  just a different name
14:27:32 <tellesnobrega> yes
14:27:53 <crobertsrh> Possible to do this with just a horizon change, if desired.
14:27:54 <tmckay> I wonder if one big cluster, with multi-tenancy (I know we don't have it) would work as well
14:28:24 <alazarev> it is easy to implement, the only thing I concirned is use case, I can hardly imagine situation when multiple clusters with the same parameters are needed and this can't be automated via script
14:28:24 <elmiko> tmckay: i think we might need some sort of job queueing for that though?
14:28:37 <SergeyLukjanov> if do on client side - there is an issue how to check quotas
14:28:46 <weiting> tellesnobrega, how many cluster you would like to create at the same time in your case?
14:29:58 <tellesnobrega> that depends, we have different labs that use our cloud, one of them is just analytics stuff, they have lots of students needing clusters there, so sometimes plenty, sometimes 2, most of the times just 1 for sure
14:30:54 <SergeyLukjanov> so, the good news that it's already partially implemented - case when count==1
14:31:08 <crobertsrh> very true
14:31:11 <tellesnobrega> yes
14:31:11 <tmckay> :) yay, partly done
14:31:16 <tellesnobrega> lol
14:31:31 <alazarev> count==0 is implemented too
14:31:40 <tmckay> alazarev, that was POC
14:31:58 <egafford> You could make an argument for count==-1 too.
14:32:21 <SergeyLukjanov> egafford, -inf as well, yay
14:32:23 <tellesnobrega> will we implement for count > 1?
14:32:29 <SergeyLukjanov> so, the last case is count==2
14:32:35 <SergeyLukjanov> :)
14:33:09 <SergeyLukjanov> I'm ok with this feature, the only question I have - what's the correct place to do it
14:33:36 <SergeyLukjanov> if we'd like to check quotas then we need to do it on server side somewhere after intelligent quotas check
14:33:38 <tmckay> and do we just add "-N" to the name given to the cluster?
14:34:13 <alazarev> tmckay, I think so
14:34:15 <tellesnobrega> tmckay, i was thinkinh uuid, but was suggested the -N
14:34:24 <tellesnobrega> so the name isn't too big
14:34:34 <elmiko> -N is nice so you know they're in a series
14:34:41 <tellesnobrega> +1
14:34:43 <elmiko> uuid could be confusing
14:34:54 <tmckay> what about network settings?  do we want the clusters segregated? Do we need to change neutron settings for that?  (not sure)
14:35:08 <elmiko> good question
14:35:21 <tmckay> just wondering if anything else on the cluster launch form has to be tweaked besides name
14:35:33 <tmckay> (well, the cluster JSON really)
14:36:21 <alazarev> tmckay, it looks like nothing, you can create clusters with the same parameters
14:36:53 <tellesnobrega> not sure, btu i think the network can keep the same settings for now, if we detect problems we improve it
14:37:42 <SergeyLukjanov> +1 for -N (in the best case impl exactly the same as it done in nova)
14:38:00 <tellesnobrega> +1
14:38:18 <elmiko> sounds like we are assuming that when creating multiple clusters they will all belong to the same keystone project?
14:38:43 <egafford> I might suggest that we pad with as many zeros as needed for the last number in the sequence.
14:39:05 <tellesnobrega> elmiko, yes
14:39:09 <tmckay> elmiko, yeah.  All within the same tenant.  otherwise you're going to have to log out and log in again in the UI.
14:39:10 <egafford> (So that an alphabetic sort by name will yield the expected order, in case anyone cares.)
14:39:14 <tellesnobrega> egafford, like that
14:39:20 <elmiko> k, just wanted to confirm
14:39:25 <elmiko> egafford: +1
14:39:45 <elmiko> although, i can't imagine needing to pad with more than 1 or *maybe* 2 0's lol
14:40:07 <elmiko> but, who knows...
14:40:18 <tellesnobrega> i think 1 is enough, but 2 would be a safer choice
14:40:20 <tellesnobrega> you never know
14:40:22 <tmckay> you can have up to 1000 clusters with one button push.  If you need more, you can push the button again
14:40:25 <elmiko> yea, 2 for safety
14:40:32 <elmiko> tmckay: lol!
14:40:44 <tellesnobrega> tmckay, +1
14:40:56 <egafford> Well, I just figure, okay, you're asking for 10 clusters? 1 x "0". 349? You're crazy, but okay, you get 2 zeros. 400,000? That's ambitious. 5 zeros.
14:41:31 <elmiko> oh, are you saying dynamically create the padding 0s?
14:41:42 <egafford> But just padding 2 might be easier, and yeah, it's hard to imagine a case for 100,000 individual clusters.
14:42:04 <elmiko> sorry, this is serious bike-shedding at this point
14:42:05 <tellesnobrega> i'm ok with dynamic as well
14:42:09 <sreshetnyak> what about API request and response? It's breaks backward compatible with older sahara versions?
14:42:11 <egafford> elmiko: That was my thought, but auto-padding with a rational number also meets the use case in huge numbers of cases.
14:42:19 <egafford> True.
14:42:43 <elmiko> sreshetnyak: not sure, sounds like it will just be an extra param in the json
14:42:55 <tmckay> We could use random words from a dictionary, with localization
14:43:11 <tellesnobrega> sreshetnyak, if we implement in the sahara side, that might be a point where we need to return a list of clusters
14:43:26 <elmiko> hey, now that sreshetnyak is here could we talk about keystone sessions for a minute? =)
14:44:03 <sreshetnyak> elmiko, hi :)
14:44:10 <elmiko> hey!
14:44:17 <tellesnobrega> we need to decide where to implement it, but we can talk later about this
14:45:00 <elmiko> so, i'm working through the spec for replacing keystone Client with Session based auth. i'm thinking the first step will be to replace our usage of keystone clients, but as a follow up we should replace all the service clients with Session usage. does that make sense?
14:45:00 <sreshetnyak> elmiko, apavlov wants works on keystone sessions
14:45:11 <elmiko> cool!
14:45:40 <sreshetnyak> elmilo, i think you need to talk to him
14:45:51 <sreshetnyak> apavlov, here?
14:47:09 <elmiko> oh well, i'll try and send something to the ML
14:47:25 <elmiko> sreshetnyak: do you know if he's working on a poc or spec yet?
14:47:56 <elmiko> also, should i add both of you to the spec as co-assignees?
14:48:59 <sreshetnyak> elmiko, no, but he wants to start
14:49:16 <elmiko> sreshetnyak: ok, i'll have to talk with him
14:49:22 <elmiko> is he in saratov timezone?
14:49:30 <sreshetnyak> yes
14:49:32 <elmiko> k
14:49:36 <elmiko> thanks =)
14:51:11 <SergeyLukjanov> 9 mins left
14:51:18 <SergeyLukjanov> anything else to chat?
14:51:27 <crobertsrh> Nothing from me
14:51:58 <alazarev> quotas is a separate topic to discuss, now we are retrieving the whole neutron DB for each cluster creation, horizon does the same thing
14:52:18 <hdd> just a quick question: what's the status of HDP 2.2 plugin?
14:53:03 <SergeyLukjanov> hdd, the first parts are already on review
14:53:07 <SergeyLukjanov> by sreshetnyak
14:53:25 <egafford> We discussed the idea of using an agent and the possibility of Zaqar adoption at Summit. It was extremely brief, and I don't think anything was decided, but I'd like to get a notion of where we are on the question of trying to move away from straight SSH for communication.
14:53:48 <hdd> ok, cool, I'll check it out. I'm eager to test
14:53:58 <tellesnobrega> the multiple clusters questions that are still open we can discuss later
14:54:27 <egafford> (This could be a worth discussing next week as a larger topic.)
14:54:34 * flaper87 drops something here: http://specs.openstack.org/openstack/zaqar-specs/specs/liberty/pre-signed-url.html
14:54:46 <egafford> flaper87: :D
14:54:48 <SergeyLukjanov> flaper87, thx :)
14:55:13 <SergeyLukjanov> egafford, so, it's the base for the potential guest agent impl
14:56:23 <egafford> SergeyLukjanov: Right; I just didn't come away with a firm picture of whether the guest agent impl is potential or planned.
14:57:11 <SergeyLukjanov> egafford, it's a potential implementation option, but now it looks like the best option
14:57:39 <SergeyLukjanov> egafford, with zaqar queues per cluster and signed urls it'll be the full isolation for the control traffic
14:57:46 <SergeyLukjanov> egafford, and no need for SSH :)
14:58:08 <egafford> I see. So we are moving toward a guest agent, and Zaqar is a potential route toward it. Cool; that's what I needed to know. Thanks.
14:58:20 <egafford> (Potential and likely.)
14:59:09 <SergeyLukjanov> egafford, yup, in real life - someday, hopefully sooner :)
14:59:20 <SergeyLukjanov> we're going out of time
14:59:23 <SergeyLukjanov> thanks folks!
14:59:26 <SergeyLukjanov> #endmeeting