18:02:33 <SergeyLukjanov> #startmeeting sahara
18:02:33 <openstack> Meeting started Thu Aug  7 18:02:33 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:36 <openstack> The meeting name has been set to 'sahara'
18:02:47 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:03:05 * SergeyLukjanov on PTO, so I'd like to end meeting in 20-30 mins :)
18:03:12 <SergeyLukjanov> #topic sahara@horizon status (croberts)
18:03:18 * mattf grins
18:03:19 <SergeyLukjanov> let's start with this
18:03:24 <SergeyLukjanov> crobertsrh, please
18:03:48 <crobertsrh> Things are going pretty well after the merge....
18:03:57 <crobertsrh> Several of our bug fixes are already merged as well
18:04:00 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes
18:04:20 <crobertsrh> I've been trying to keep that etherpad up to date in the event that I disappear for awhile
18:04:33 <SergeyLukjanov> crobertsrh, could you please ack that you'll be in PTO for the two weeks (Aug 11-24) ?
18:04:47 <crobertsrh> I have -1 workflow on a few patches that will be used for Spark, Storm and swift trusts
18:05:00 <crobertsrh> Currently, I plan to start PTO on Aug 15
18:05:10 <crobertsrh> If the baby has other plans, I will leave for PTO sooner
18:05:10 <alazarev> crobertsrh: did you see https://bugs.launchpad.net/horizon/+bug/1352590 ?
18:05:11 <uvirtbot> Launchpad bug 1352590 in horizon "[Sahara] A lot of things don't work in new horizon design" [Undecided,Confirmed]
18:05:42 <SergeyLukjanov> crobertsrh, ack, NikitaKonovalov will work on dashboard stuff starting from the next week two
18:05:46 <crobertsrh> alazarev:  That is a new one for me.  Probably bootstrap 3 related :)
18:05:51 <SergeyLukjanov> crobertsrh, for a few weeks I think
18:05:51 <crobertsrh> Great
18:05:53 <aignatov> interesting bug I also found two days ago
18:05:54 <aignatov> https://bugs.launchpad.net/sahara/+bug/1352812
18:05:55 <uvirtbot> Launchpad bug 1352812 in sahara "[UI] NodeGroupCreate first frame looks incorrect" [High,Confirmed]
18:05:55 <alazarev> crobertsrh: it looks like horizon.modals.addModalInitFunction stopped to work
18:06:15 <SergeyLukjanov> heh
18:06:37 <crobertsrh> aignatov:  Yes, that is definitely related to bootstrap 3....my patch should fix that one.
18:06:47 <crobertsrh> https://review.openstack.org/#/c/110680/
18:06:49 <aignatov> crobertsrh: cool
18:07:13 <crobertsrh> The problem is that bootstrap 3 changed from "control-group" to "form-group" as a class name
18:07:39 <crobertsrh> Our JS still referenced "control-group" in many places, resulting in ugly UI with extra, incorrect fields that we normally hide.
18:08:12 <SergeyLukjanov> crobertsrh, great, I hope it'll be merged soon
18:08:20 <crobertsrh> If things are still behaving badly after my above patch, we might need additional tweaks.
18:08:29 <crobertsrh> Yeah, me too
18:09:23 <SergeyLukjanov> anything else re dashboard?
18:09:36 <SergeyLukjanov> #topic News / updates
18:09:37 <crobertsrh> Nothing else from me
18:09:39 <SergeyLukjanov> non-dashboard time
18:09:42 <SergeyLukjanov> crobertsrh, thx!
18:10:11 * SergeyLukjanov working on moving sahara to the new just released oslo.* libs
18:10:38 <SergeyLukjanov> and NikitaKonovalov is near to finish Sahara-Rally integration to easily perform perf testing of sahara itself
18:10:51 <tmckay> I've been working on moving edp-examples and merging with integration tests, in prep for spark integration tests
18:10:53 <elmiko> swift/auth is coming along, i am working on acquiring/revoking trusts related to data objects and job binaries, i have a few design related questions though.
18:11:26 <tmckay> Also, on https://review.openstack.org/#/c/110576/, I've discovered today that extra "addResource" call for Java actions is not necessary in Oozie 4
18:11:41 <tmckay> they fixed it :)
18:11:49 <reshetnyaks> I'm working on CDH integration tests and parallel EDP integration tests
18:12:08 <tmckay> only outstanding issue for https://review.openstack.org/#/c/110576/ is System.exit, hopefully they fixed that too, looking into it
18:12:11 <SergeyLukjanov> tmckay, nice
18:12:15 <tmckay> if so, the spec request can go away
18:13:28 <tmckay> maybe it should go away anyway, because I'm not sure Sahara should be responsible for covering up "System.exit".  If you have to change a job and recompile, well ...
18:13:52 <tmckay> SergeyLukjanov, oh, I'm bad, I did not do my "action" yet -- spec for cancel/delete semantics
18:14:42 <SergeyLukjanov> tmckay, me too :)
18:14:47 <SergeyLukjanov> #action tmckay finishes spec and blueprint for job execution cancel/delete semantics
18:14:54 <SergeyLukjanov> #action SergeyLukjanov to talk with Horizon folks about features merge deadline
18:15:31 <alazarev> I fixed anti-affinity for heat engine, but it will work with new heat only because support of server groups was added in July
18:15:36 <tmckay> so, can we say that Sahara expects Oozie version 4?
18:15:51 <tmckay> I believe the image builder pulls in most recent anyway...
18:16:20 <tmckay> I think we were using version 3 when I made the original Java action with swift support, which is what required the conf section
18:17:06 <tmckay> maybe a "if you are using earlier than version 4, you need to ...."
18:17:06 <aignatov> tmckay: I think we always used oozie 4
18:17:28 <tmckay> hmm, okay.  I must have had something strange way back when
18:17:37 <SergeyLukjanov> aignatov, ++
18:18:54 <SergeyLukjanov> okay, lets' move on
18:19:09 <SergeyLukjanov> #topic stable/icehouse: 2014.1.2 (Aug 7)
18:19:25 <SergeyLukjanov> so, the new OpenStack stable/icehouse release is today
18:19:38 <SergeyLukjanov> for Sahara it'll be a bit later - Monday I think
18:19:49 <SergeyLukjanov> so, call for backports is open
18:20:03 <SergeyLukjanov> if you think that we should backport smth - please ping me or do it
18:20:12 <SergeyLukjanov> #topic Juno 3 (Sept 4)
18:20:20 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Juno_Release_Schedule
18:20:33 <SergeyLukjanov> #link https://launchpad.net/sahara/+milestone/juno-3
18:21:22 <SergeyLukjanov> it looks nice, but I think that we're already busy for j3 for new bps
18:21:31 <SergeyLukjanov> and there is a bunch of un-assigned issues
18:21:54 <SergeyLukjanov> #topic Open discussion
18:23:00 <elmiko> for the swift/auth stuff, it is looking like we will only need a single trust to access storage objects within the same project, i'm going to adjust the JobExecution.config accordingly, but try to leave rooom for multi-tenancy in the future. does this sound acceptable?
18:23:05 <tmckay> so spark will work with hdfs, but I haven't tried swift
18:23:44 <tmckay> at the very least, even if spark rides on top of hadoop, the configs need to be set.  Maybe when elmiko swift auth changes are done, it will magically work
18:24:10 <elmiko> also, i'm curious about how the swift-fs plugin for hadoop gets deployed
18:24:23 <tmckay> elmiko, maybe we can get together and test this standalone, without all the changes being in sahara ... set up a trust, etc and try to verify by hand
18:24:24 <elmiko> because once i start to make these changes, the old one won't work
18:24:54 <elmiko> tmckay: test the access from spark?
18:25:04 <tmckay> yeah
18:25:16 <elmiko> i'm game to investigate with you
18:26:03 <elmiko> another design type question i have is, will it be a problem if i store the JobBinary and DataSource urls using the storageURL from Swift, and then filter out that url when displaying?
18:27:19 <elmiko> for example, now we store something like "swift://contianer.sahara/object", i would like to store "swift://storageURL.sahara/container/object" and then filter out the storageURL when we display the value. is this acceptable?
18:27:21 <tmckay> oops, I might have been wrong about addResource ... darn
18:28:29 <elmiko> i'll take silence as approval :)
18:28:50 <crobertsrh> I'm a little fuzzy there...
18:29:00 <crobertsrh> What will need to be entered for the swift URL?
18:29:24 <tmckay> elmiko, depends whether or not a cut and paste gives a valid url for a swift client, say the user has the proper auth
18:29:28 <elmiko> i'd like to make it so that the user doesn't see a change, but internally sahara will filter out the swift URL
18:29:37 <crobertsrh> Ah, got it
18:29:39 <tmckay> elmiko, if the user can cut and paste, I'm fine with the filter
18:29:47 <elmiko> tmckay: cut/paste from where?
18:29:59 <tmckay> elmiko, "display"
18:30:18 <crobertsrh> "display" won't have it, if I understand correctly
18:30:20 <tmckay> if we display a value, it should be usable from the swift client, imho
18:30:25 <elmiko> not sure i follow, but i think it will look no different to the user
18:31:13 <elmiko> well that's the issue, if you use the swift client with a username/password you can access containers/objects in the same project by their names. if you use an authentication token you must know the root storageURL for the Swift object store in that project.
18:31:19 <tmckay> elmiko, when you say "filter out" do you mean leave an object field out completely, or change the value of a field
18:31:40 <elmiko> tmckay: change the value before display
18:31:52 <elmiko> tmckay: also, the workflow.xml would still contain the full url
18:32:05 <tmckay> okay, so all I'm saying is that the "changed" value should not mislead the user
18:32:52 <elmiko> the difference would be, user sees "container.sahara/object", sahara sees "swift://<ip>:<port>/<swift auth id>/container.sahara/object"
18:33:04 <tmckay> k
18:33:12 <tmckay> I think that's fine
18:33:35 <elmiko> it's more filtering on the sahara side, also the "sahara sees" value will be in the workflow.xml as well
18:34:06 <elmiko> ok
18:34:36 <aignatov> elmiko: I agree with this approach
18:34:48 <elmiko> also, as i get closer to making this work, we will need to plan how we roll it out as it will make backward compat with the swift-fs stuff not possible.
18:34:53 <aignatov> if it’ll solve original problem :)
18:35:03 <elmiko> it will :)
18:35:20 <elmiko> i'm confident we can eliminate sahara storing any credentials
18:36:18 <elmiko> thanks guys, that answer my questions... for now ;)
18:36:52 <SergeyLukjanov> any other topics?
18:37:16 <aignatov> elmiko: we need to solve this problem too, not sure how right now, but should…. I mean about changes in swift-fs
18:37:33 <mattf> none from me
18:37:41 <elmiko> aignatov: agreed, i need to know more about how that swift-fs plugin is deployed
18:38:28 <aignatov> elmiko: it’s just a jar file which is intrgrated to hadoop classpasth
18:38:44 <elmiko> aignatov: does sahara deploy it?
18:39:02 <aignatov> sahara-image-elements script soes it
18:39:06 <aignatov> *does
18:39:28 <tmckay> no more topics for me
18:39:35 <elmiko> ok, so i need to make sure that it gets upstream for sahara-image-elements. also the hwx guys will need to deploy it to their download location.
18:40:00 <elmiko> this is gonna be complicated to coordinate before the end of juno freeze
18:40:12 <aignatov> elmiko: here https://github.com/openstack/sahara-image-elements/blob/master/elements/swift_hadoop/post-install.d/81-add-jar
18:40:18 <elmiko> aignatov: thanks!
18:40:20 <SergeyLukjanov> elmiko, yeah, sounds like very huge task
18:40:51 <elmiko> SergeyLukjanov: that's my largest concern at this point, how we will integrate to get all the pieces in place.
18:41:27 <elmiko> we can talk more next week though
18:41:40 <SergeyLukjanov> elmiko, the j3 is deadline for such feature, so, probably it makes sence to preverntually move it to the K release but I really like to see it done in J
18:42:13 <elmiko> SergeyLukjanov: i'd like to see it done for J as well. i think the coding will be done, we will need to be aggressive about integration though.
18:42:35 <elmiko> this is also why i'm trying to push changes now that won't break things, but that will soon change.
18:42:46 <SergeyLukjanov> elmiko, yeah
18:43:43 <elmiko> SergeyLukjanov: if it would make things easier i can post an email to the openstack-dev list detailing the issues so we have good visibility on what needs to happen.
18:44:14 <SergeyLukjanov> elmiko, it's a good option, so, yes, please
18:44:21 <elmiko> ok, will do
18:44:34 <alazarev> elmiko: +1 on email to openstack-dev
18:47:42 <SergeyLukjanov> time to end meeting?
18:47:51 <SergeyLukjanov> #endmeeting