18:03:22 <SergeyLukjanov> #startmeeting sahara
18:03:23 <openstack> Meeting started Thu Jul 24 18:03:22 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:03:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:03:26 <openstack> The meeting name has been set to 'sahara'
18:03:30 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:03:37 <SergeyLukjanov> #topic News / updates
18:03:50 <SergeyLukjanov> crobertsrh, you have a separated topic ;)
18:03:53 <ylobankov> hello
18:04:04 <crobertsrh> aww, ok
18:04:07 <mattf> hiya
18:04:20 <SergeyLukjanov> #info Juno 2 released
18:04:36 <SergeyLukjanov> tag to sahara has been pushed yesterday
18:04:51 <SergeyLukjanov> and I've pushed tags to -dashboard, elements and extra right before the meeting
18:05:06 <SergeyLukjanov> #link https://launchpad.net/sahara/+milestone/juno-2
18:05:14 <SergeyLukjanov> so, congratulations folks!
18:05:17 <mattf> yay!
18:06:08 <tmckay> here
18:06:17 <aignatov> hurrah!
18:06:22 <SergeyLukjanov> any news/updates folks?
18:06:29 <tmckay> working on my spec so much I forgot the meeting
18:06:32 <tmckay> yes, update:
18:06:41 <sreshetnyak> I'm working on CDH plugin
18:06:45 <alazarev> I was busy implementing https://blueprints.launchpad.net/sahara/+spec/cluster-secgroups
18:07:08 <tmckay> spark edp initial impl is done, just needs unit tests (and maybe integration tests?)  Working on a detailed spec now, should be posted today
18:07:08 <alazarev> patch is ready, will commit today
18:07:13 <aignatov> I’ve started working on moving rest samples to the docs but realised that all of them are in the rest api v1.0 v1.1 pages
18:07:15 <tosky> I have a fix for sahara-image-elements blocked in savanna-ci because of failing jobs (I think mostly timeouts)
18:07:25 <ylobankov> I am working on tests in tempest style
18:07:35 <aignatov> so i’ve decided just to updated current rest api items in all docs
18:07:39 <tosky> also, anyone for HWX around? One of the ambari repository used by dib elements is still not working
18:07:45 <aignatov> and add new ones woth more examples
18:07:45 <sreshetnyak> also, I'm working on various minor changes
18:07:47 <tmckay> I have a question about integration tests for Spark.  Essentially, should we have them? :)  There are no spark cluster tests yet, afaik
18:08:11 <SergeyLukjanov> ylobankov, could you please help tosky with searching the reason of failing jobs on his commit?
18:08:52 <SergeyLukjanov> tmckay, we need to have itests for spark too
18:09:00 <ylobankov> yes, of course
18:09:06 <SergeyLukjanov> tmckay, otherwise we'll break it :)
18:09:08 <tosky> ylobankov: thanks
18:09:14 <tmckay> SergeyLukjanov, okay, I'll make a blueprint for it
18:09:32 <tmckay> news, I have a flight and hotel for OSS :)
18:09:41 <SergeyLukjanov> tmckay, cool!
18:09:56 <SergeyLukjanov> tmckay, and I have France VISA that ends Nov 1 ;)
18:10:11 <tmckay> hmm, seems too short
18:10:23 <crobertsrh> I guess we'll miss you there SergeyLukjanov :)
18:10:51 <tmckay> is mattf here?
18:10:58 <mattf> hi hi
18:11:02 <mattf> you rang?
18:11:17 <tmckay> hey.  I was going to raise your spec suggestion, but if you're here ...
18:11:42 <mattf> oh, i'll take that off-line for a bit
18:11:47 <tmckay> ok
18:11:52 <mattf> tmckay, thanks for the reminder tho
18:11:56 <tmckay> np
18:12:05 <SergeyLukjanov> oops :)
18:12:19 <SergeyLukjanov> #topic sahara-dashboard @ horizon status (croberts)
18:12:23 <SergeyLukjanov> crobertsrh, please :)
18:12:30 * mattf gets ready to cheer
18:12:32 <crobertsrh> Good news....about 27 min ago, the last of the horizon merge patches finally merged!
18:12:39 <mattf> w00t!
18:12:44 <SergeyLukjanov> [21:44:58]  <openstackgerrit>	 A change was merged to openstack/horizon: Adding Job Binaries panel for Sahara  https://review.openstack.org/91091
18:12:44 <SergeyLukjanov> [21:45:03]  <openstackgerrit>	 A change was merged to openstack/horizon: Adding Jobs and Job Executions panels for Sahara  https://review.openstack.org/91118
18:12:47 <SergeyLukjanov> yay!
18:12:52 <crobertsrh> Is now a bad time to propose pulling our dashboard from horizon?? :)
18:12:55 <aignatov> wohoooo!
18:13:05 <aignatov> awesome news
18:13:12 <mattf> now's a great time to discuss pulling our dashboard
18:13:16 <mattf> -1 from me
18:13:27 <tmckay> good time to add Spark job type form
18:13:42 <crobertsrh> absolutely, tmckay
18:13:57 <SergeyLukjanov> yeah, I think that we'll be able to land all rest fixes to horizon before J release
18:14:02 <tmckay> crobertsrh, I should have something written up about that soon
18:14:06 <SergeyLukjanov> and some additional features like Spark support
18:14:28 <mattf> crobertsrh, please send our thanks to the horizon folks for consuming the large merge
18:14:34 <crobertsrh> Yes.  All the fixes should make it.  They are small and mostly +1 or +2 already.  I've got some other items on my list too.
18:14:53 <crobertsrh> mattf:  Yes....will do.  I suspect that I'll have a few drinks to buy in Paris.
18:15:26 <tmckay> do they drink there?
18:15:35 <mattf> tmckay, wine only, le sigh
18:15:41 <tmckay> lol
18:16:00 <crobertsrh> Side note for dashboard stuff.....I'm slated to have a few weeks off starting around August 15th (baby coming).  Is there anyone else that is planning to do dashboard work for juno?
18:16:05 <SergeyLukjanov> crobertsrh, thank you for continuously pushing sahara dash to horizon!
18:16:40 <tmckay> not me, I'd likely break it
18:17:08 <SergeyLukjanov> crobertsrh, I think NikitaKonovalov could take it for a few weeks
18:17:31 <crobertsrh> I am striving to have bugs/specs/blueprints written up as much as possible.  Hopefully, there won't be too much to do other than write a bunch of simple UI code :)
18:17:37 <alazarev> crobertsrh: I’ll take care of my staff and coming security groups
18:17:52 <crobertsrh> Great, thanks alazarev!
18:18:43 <SergeyLukjanov> okay, anything else re dashboard?
18:19:02 <crobertsrh> Nothing from me.  I'm going to have a celebratory beverage now :)
18:19:08 <SergeyLukjanov> #info crobertsrh will have a few weeks off starting around August 15th
18:19:21 <SergeyLukjanov> crobertsrh, :)
18:19:35 <SergeyLukjanov> #topic Action items from the last meeting
18:19:42 <SergeyLukjanov> #action SergeyLukjanov to create bp with steps to enable heat be default
18:19:50 <SergeyLukjanov> #action SergeyLukjanov to create bp about removing/hiding username@image for heat based provisioning
18:20:24 <SergeyLukjanov> any other big topics to discuss today?
18:20:55 <elmiko> i have a few minor questions about the JobExecution data model
18:21:23 <SergeyLukjanov> #topic Open discussion
18:21:28 <SergeyLukjanov> elmiko, you're welcome
18:21:33 <tmckay> ah yes, crobertsrh, we wanted to look into making job name available from job execution, remember?
18:21:53 <crobertsrh> Yes, that is desired for:  https://bugs.launchpad.net/horizon/+bug/1342293
18:21:54 <uvirtbot> Launchpad bug 1342293 in horizon "[data processing] Make the job executions page more useful" [Wishlist,New]
18:21:56 <tmckay> and maybe cluster name?  or some other stuff, so you don't have to make double requests
18:22:17 <crobertsrh> oooh, fancy uvirtbot...thanks
18:22:19 <tmckay> easiest way is to add relation in job execution to job object
18:22:35 <tmckay> but that's a data model change
18:22:49 <elmiko> currently, we use the job_configs column in JobExecution to store the swift credentials, i was going to add a column for the trust id associated with a JobExecution, would it be appropriate to use job_configs instead?
18:23:17 <tmckay> elmiko, we've put a lot of stuff in job_configs or extra to save migrations
18:23:28 <tmckay> I think it's fine to put it in configs or extra
18:23:29 <elmiko> tmckay: that's what i figured
18:23:38 <tmckay> fewer migrations is better
18:23:43 <elmiko> it actually makes my job easier :)
18:24:28 <tmckay> ack, I just crammed some stuff in extra for spark ;-)
18:25:01 <alazarev> do we really want to use extra?
18:25:04 <elmiko> sounds like a good place to stuff the job/cluster name?
18:25:10 <alazarev> I don’t like how we use it now
18:25:18 * mattf grins
18:25:26 <elmiko> alazarev: should i create a new column?
18:25:53 <alazarev> in cluster extra is reserved for plugins data, in jobs engine uses it, need more consistency
18:26:29 <alazarev> elmiko: for cluster I was told to add new columns for all engine data
18:26:35 <tmckay> extra is used for neutron info in job executions currently
18:26:36 <SergeyLukjanov> alazarev, but often it's the only way to support different engines/plugins
18:26:43 <tmckay> I stored a spark path there
18:26:52 <elmiko> alazarev: i need a place to store the trust id associated with each job execution, would it be preferable to make this a column?
18:27:05 <tmckay> for edp engine stuff, we could go extra.engine_name.stuff
18:27:09 <tmckay> and make it formal
18:27:41 <tmckay> elmiko, only execs that use swift will need it, so it at the very least needs to be nullable
18:27:51 <elmiko> tmckay: definitely
18:28:02 <SergeyLukjanov> elmiko, for trusts it sounds like it doesn't depend on any engine/plugin
18:28:18 <elmiko> SergeyLukjanov: correct, only depends on swift usage
18:28:22 <tmckay> true.  We can add a column, I'm okay with that.
18:28:41 <alazarev> +1 on column
18:28:51 <SergeyLukjanov> but w/o swift it's not needed
18:28:53 <SergeyLukjanov> at least atm
18:28:55 <aignatov> SergeyLukjanov: it depends on data source actually as tmckay said above
18:28:59 <tmckay> question about compat -> if you upgrade, and you have old objects in the db with swift credentials, will we still run them? or reject them?
18:29:12 <alazarev> about columns - we really need to orginize them, number of columns in each objest grows
18:29:52 <elmiko> tmckay: that's a really good question, and it goes towards backward compatibility. i made an explicit note in the spec about ensuring there are no job executions when the database migration is performed
18:30:12 <SergeyLukjanov> tmckay, we should save backward compat here
18:30:43 <tmckay> elmiko, I guess the question is, will you be able to use  old data source objects that have swift creds in them?  Just use a trust id, and ignore them?
18:30:49 <aignatov> elmiko: old ones job executions shoudl be supported aftre upgrade
18:30:53 <elmiko> tmckay: yes
18:31:09 <tmckay> ok.  The only way to use old job executions is with relaunch
18:31:16 <elmiko> aignatov, SergeyLukjanov, supporting old JobExecution's means keeping credentials stored inthe db
18:31:24 <tmckay> and that's a feature of the UI, not Sahara api itself
18:31:38 <aignatov> elmiko: obly for old ones, we cann’t do anything with it
18:31:41 <SergeyLukjanov> elmiko, only for old job executions
18:31:51 <aignatov> if we wants to keep backward
18:31:57 <aignatov> compat
18:32:05 <SergeyLukjanov> elmiko, upgrading with removing entites from db is a very bad solution
18:32:07 <tmckay> I think old job executions are not an issue.
18:32:14 <SergeyLukjanov> tmckay, yup
18:32:18 <sreshetnyak> cluster object has already trust_id column
18:32:32 <elmiko> sreshetnyak: this is for JobExecution
18:32:44 <elmiko> we need a new trust for each job that includes swift sources
18:33:23 <SergeyLukjanov> elmiko, hm, what's about case when we'll have two different swift data sources for input and output?
18:33:26 <aignatov> do a trusts have expiration timeouts?
18:33:32 <SergeyLukjanov> elmiko, will we need to store two trust_ids
18:33:42 <elmiko> aignatov: they can
18:34:10 <elmiko> SergeyLukjanov: correct, we need the cluster trust id for transient stuff _and_ we will need trusts for any job that requires swift access
18:34:27 <aignatov> so, for example cluster has infinite expiration date for trust…. could be reusable for job_executions?
18:34:39 <elmiko> it makes more sense to me if we store the job execution trusts with those objects
18:34:44 <tmckay> elmiko, but potentially 1 trust for input and 1 for output if they are from different swifts
18:35:25 <elmiko> tmckay: not necessarily, it depends how the swift objects are scoped in keystone
18:35:37 <tmckay> ok.
18:35:38 <SergeyLukjanov> elmiko, they aren't scoped AFAIK
18:35:46 <elmiko> aignatov: the trusts we establish might be different tenants
18:36:15 <SergeyLukjanov> elmiko, yup
18:36:21 <elmiko> SergeyLukjanov: well, for example, if a user wants to start a new job with swift objects, and the swift objects exist in a different tenant, then the trust would need to be scoped to that tenant
18:36:25 <alazarev> tmckay: we don’t support different swifts
18:37:18 <elmiko> by moving to the trust based system it's possible that we could support swift containers in other tenants
18:37:43 <elmiko> mainly becuase we will need to be using swift's storageURL to access those objects
18:40:01 <elmiko> so... what's the group opinion on adding a new column to JobExecution for the swift trust id?
18:41:01 <tmckay> okay with me
18:41:09 <SergeyLukjanov> I'm mostly ok with it
18:41:29 <SergeyLukjanov> elmiko, but I have concern that it could be several trust_id
18:41:42 <SergeyLukjanov> elmiko, like one for input and another for output
18:42:00 <tmckay> could be a json dictionary type, like configs
18:42:09 <tmckay> then you can store a whole bunch with keys
18:42:15 <crobertsrh> I think it seems like the right thing to do, even if there winds-up being multiple values in there.
18:42:31 <elmiko> ok, so maybe more like what we've done with job_configs?
18:42:45 <tmckay> yeah, just a single level dictionary
18:42:48 <tmckay> no nesting
18:43:07 <tmckay> "whatever_id": "value"
18:43:31 <elmiko> it could get really complicated to interleave the trust ids with the source objects if we start to get more than 1 entry for input and 1 for output
18:43:58 <tmckay> hmmm
18:44:05 <SergeyLukjanov> tmckay, it sounds like adding the third extra field to job_exec
18:44:53 <elmiko> currently we only allow for 1 input and 1 output object?
18:44:53 <aignatov> I have an idea, what if elmiko starts adding trust objects to current extra as a temporary approach
18:44:54 <tmckay> SergeyLukjanov, almost.  It could be a list, but then how do you know order?
18:45:13 <tmckay> elmiko, yes
18:45:19 <aignatov> we will see how it works and then move it to new column(s) if need
18:45:44 <SergeyLukjanov> aignatov, or we could use job_configs for it too
18:45:51 <elmiko> aignatov: i'm ok with that, it sounded like there was some difference of opinion on that though
18:46:18 <elmiko> tmckay: so, currently, max we would need to store would be 2 ids
18:46:19 <SergeyLukjanov> in job_configs we currently have the following structure - https://github.com/openstack/sahara/blob/b2c393f752f7bdd4c789bb3a25e4b639869beeec/etc/rest-api-samples/edp/job_execute.json
18:46:54 <tmckay> but configs, args, and params all have specific meanings
18:46:58 <tmckay> they're different
18:47:04 <tmckay> not so crazy
18:47:05 <SergeyLukjanov> what's about adding some upper level field like "trusts" that will store trust ids for all data sources?
18:47:21 <elmiko> SergeyLukjanov: within job_configs?
18:47:22 <tmckay> I'd be okay with that
18:47:32 <SergeyLukjanov> elmiko, yup
18:47:39 <elmiko> ack
18:47:43 <SergeyLukjanov> because in fact this trusts are part of the job config :)
18:47:58 <elmiko> agreed
18:48:25 <aignatov> SergeyLukjanov: we could add separate table eventually like pool of trusts for different tenants
18:48:31 <SergeyLukjanov> any objections?
18:48:54 <aignatov> for whatever needs - killink transient cluster, or access to switft objects
18:48:55 <SergeyLukjanov> aignatov, we'll need to re-work the whole model eventually
18:49:22 <aignatov> killnk -> killing :)
18:49:38 <SergeyLukjanov> #agreed add trusts under the upper level field to the job_configs
18:49:45 <elmiko> ok, sounds good
18:49:59 <sreshetnyak> agreed with SergeyLukjanov
18:50:39 <aignatov> lgtm
18:50:48 <alazarev> +1
18:50:50 <elmiko> and i'm assuming we will not print the trust ids on the rest calls?
18:50:51 <tmckay> me too
18:51:07 <elmiko> much like the username/password currently
18:51:09 <tmckay> elmiko, is there a security reason not too?
18:51:10 <SergeyLukjanov> elmiko, tmckay, yup
18:51:31 <tmckay> elmiko, okay, easy to filter.
18:51:38 <tmckay> I can help you with that.
18:51:56 <elmiko> tmckay: off the top of my head, i don't think there is a risk, but i also don't see a reason why we would print them
18:52:05 <tmckay> gotcha
18:53:00 <SergeyLukjanov> 7 mins left
18:53:03 <elmiko> the trusts will be scoped to the proxy user, so you would need at minimum the trustee or trustor's auth token to do something useful with the trust id
18:53:06 <SergeyLukjanov> anything else to dicsuss?
18:53:33 <SergeyLukjanov> how will attend the paris summit this november?
18:53:42 <elmiko> o/
18:53:51 <SergeyLukjanov> \o/
18:53:59 <aignatov> how or who?
18:54:04 <SergeyLukjanov> who*
18:54:05 <tmckay> me
18:54:42 <tmckay> crobertsrh too
18:54:54 <crobertsrh> I will be there, indeed
18:55:17 <crobertsrh> Hopefully, everyone can make it for all 5 days this time :)
18:55:49 <SergeyLukjanov> yeah, I hope so
18:57:25 <SergeyLukjanov> 2 mins left
18:57:39 <SergeyLukjanov> if there are no more q. I'm ending the meeting
18:57:39 <crobertsrh> Nothing else from me
18:57:40 <tosky> SergeyLukjanov: may I suggest to introduce caching in your gate? It seems sahara-image-elements jobs are slowed because of slow download times
18:57:42 <alazarev> I think I’ll attend too
18:58:33 <SergeyLukjanov> tosky, I'm not sure how it's actually working atm, I'll ping our folks tomorrow about it
18:59:04 <SergeyLukjanov> #endmeeting