Monday, 2014-05-05

*** shakamunyi has joined #openstack-sahara00:23
*** zhiyan_ is now known as zhiyan00:26
*** shakamunyi has quit IRC00:48
*** mojorison has quit IRC01:00
*** shakamunyi has joined #openstack-sahara01:13
*** IlyaE has quit IRC01:19
*** ErikB1 has joined #openstack-sahara01:28
*** IlyaE has joined #openstack-sahara01:30
*** shakamunyi has quit IRC02:57
*** ErikB1 has quit IRC03:02
*** ErikB1 has joined #openstack-sahara03:07
*** ErikB1 has quit IRC03:22
*** IlyaE has quit IRC05:04
*** Ch00k has joined #openstack-sahara05:07
*** IlyaE has joined #openstack-sahara05:39
*** Ch00k has quit IRC05:48
*** Ch00k has joined #openstack-sahara05:49
*** IlyaE has quit IRC05:53
*** bogdando has quit IRC05:59
*** bogdando has joined #openstack-sahara06:00
*** IlyaE has joined #openstack-sahara06:13
*** Ch00k has quit IRC06:16
*** akuznets_ has quit IRC06:24
*** akuznetsov has joined #openstack-sahara06:26
*** skolekonov has joined #openstack-sahara06:35
*** akuznets_ has joined #openstack-sahara06:48
*** akuznetsov has quit IRC06:48
openstackgerritDaniele Venzano proposed a change to openstack/sahara-image-elements: Update Spark element  https://review.openstack.org/9134006:52
*** RockKuo_Office has joined #openstack-sahara07:26
*** Ch00k has joined #openstack-sahara07:28
*** IlyaE has quit IRC07:35
*** IlyaE has joined #openstack-sahara07:35
*** IlyaE has quit IRC07:38
*** IlyaE has joined #openstack-sahara07:43
*** openstackgerrit has quit IRC07:53
*** Ch00k has quit IRC08:07
*** Ch00k has joined #openstack-sahara08:07
*** IlyaE has quit IRC08:13
*** IvanBerezovskiy has joined #openstack-sahara09:07
*** IvanBerezovskiy has left #openstack-sahara09:17
*** tosky has joined #openstack-sahara09:58
*** bogdando has quit IRC10:38
*** bogdando has joined #openstack-sahara10:51
*** RockKuo_Office has quit IRC11:04
*** akuznets_ has quit IRC11:13
*** akuznetsov has joined #openstack-sahara11:15
*** _crobertsrh has quit IRC11:28
*** IvanBerezovskiy has joined #openstack-sahara11:36
*** crobertsrh has joined #openstack-sahara11:46
*** Ch00k has quit IRC12:22
*** Ch00k has joined #openstack-sahara12:22
*** crobertsrh has quit IRC12:34
*** sballe has joined #openstack-sahara12:36
*** Ch00k has quit IRC12:37
*** crobertsrh has joined #openstack-sahara12:39
*** _mattf is now known as mattf12:51
*** mattf has quit IRC12:51
*** mattf has joined #openstack-sahara12:51
*** elmiko has joined #openstack-sahara12:53
*** Ch00k has joined #openstack-sahara13:02
*** ErikB1 has joined #openstack-sahara13:08
*** openstackgerrit has joined #openstack-sahara13:36
*** openstackgerrit has quit IRC13:51
*** tmckay has joined #openstack-sahara13:51
*** Ch00k has quit IRC14:17
*** Ch00k has joined #openstack-sahara14:19
*** bradd1 has joined #openstack-sahara14:25
*** skolekonov has quit IRC14:51
*** IlyaE has joined #openstack-sahara15:01
*** ErikB1 has quit IRC15:03
*** ErikB1 has joined #openstack-sahara15:10
*** zhiyan is now known as zhiyan_15:25
*** openstackgerrit has joined #openstack-sahara15:42
*** mattf is now known as _mattf15:48
*** Ch00k has quit IRC15:50
*** IlyaE has quit IRC15:50
*** Ch00k has joined #openstack-sahara15:54
*** IlyaE has joined #openstack-sahara15:56
*** Ch00k has quit IRC16:15
*** _mattf is now known as mattf16:27
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Extended plugin SPI with methods to communicate with EDP  https://review.openstack.org/7372916:32
*** IlyaE has quit IRC16:59
*** IlyaE has joined #openstack-sahara17:01
*** ylobankov_ has joined #openstack-sahara17:20
*** openstackgerrit has quit IRC17:20
*** ylobankov has quit IRC17:21
*** openstackgerrit has joined #openstack-sahara17:24
*** ErikB1 has quit IRC17:25
*** ErikB1 has joined #openstack-sahara17:25
tmckayhas anyone been thinking about how to use trusts with swift from Hadoop?  We have a problem currently with swift credentials exposed in workflows.  If I understand trusts correctly, however, the hadoop user on a cluster would need it's own keystone authentication in order to have a user id that could be matched to a trustee id for swift access (and of course the hadoop credentials would have to be guarded)17:28
tmckayI am also wondering if trust tokens would give us a way to encode the endpoint with the token, so that we don't need ".sahara" in the data source swift url17:30
tmckaydmitryme, do you have knowledge of this stuff? ^^17:31
tmckayaignatov, maybe you? ^^ Looking for Summit design fodder, and also usability improvements (tellesnobrega gave us a good run through last week from someone relatively new, and there were some pitfalls)17:35
aignatovtmckay: not sure that trusts could solve “.sahara” problem17:50
tmckaydarn.  Why not?17:50
aignatovbecause that’s separate things as I understand17:50
aignatovah, sorry17:51
aignatovdidn’t observe your idea completely17:51
dmitrymetmckay: I pretty sure trusts will not remove need for .sahara prefix. The endpoint is not encoded in trust in any way, as far as I know17:52
aignatovbut to solve this we could just hide ‘.sahara’ postfix in the urls17:52
aignatovand append it only in sahara internaly17:52
tmckayaignatov, dmitryme, what about "phase 2" with arbitrary attributes in a token? https://wiki.openstack.org/wiki/Keystone/Trusts17:53
dmitrymetmckay: sorry, need to go, will come back later17:53
tmckayaignatov, dimitryme, "but to solve this we could just hide ‘.sahara’ postfix in the urls", exactly, I did that this morning :)  But, Java actions are the exception :(17:53
tmckayaignatov, Java actions do not use data sources.  They take only args.  If you pass a swift url as an arg to a Java action, you have to "fix" it yourself17:54
tmckayI don't like the "specialness" of that.17:54
SergeyLukjanovtmckay, hey17:55
aignatovtmckay: I see, yep, that’s a problem and point to discuss at the Summit :)17:55
tmckayaigntov, the only other idea I had was to add <property>fs.swift.service</property> to the job config and have the service key specified there, instead of extracted from the url17:55
* SergeyLukjanov not sure that we could solve swift suffix issue17:55
tmckayaignatov, SergeyLukjanov, tellesnobrega was bitten by the suffix issue on Thurs/Friday, which is why I added the validation checks17:56
tmckayI would love to see it disappear :)  I suppose it's possible to add a warning to scan Java action args, and if we see a swift:// url we could at least generate a warning about it.  Or, even make a configurable setting that applies the validation check to Java args that are swift urls, and generate an error17:58
tmckay(default to True, because you are probably wrong)17:58
tmckayMaybe I can add that to my CR and throw up as a draft17:58
tmckayaignatov, ping re Summit, just want to confirm that we want to use bigpetstore for the demo.  Only a few days left :)  Agreed?  If we have other examples we can link to them from a slide.18:00
aignatovtmckay: agreed, +2, I’ve starting preparing our presentation sectin for EDP18:00
tmckaygreat.18:01
aignatovwill share once I’ll get some18:01
aignatov:)18:01
tmckayaignatov, great, thank you18:03
aignatovtmckay: we have an example, I’ve shared this pig scripts with you last week18:04
aignatovbut I’m afraid that data preparation will take too long18:04
tmckayaignatov, ah right, my gmail account!  I will have to look.18:05
tmckayaignatov, I think we we leave attendees with a link to another example, that's okay.  We can commit it to sahara-extra before Summit18:05
aignatovyes, I think so18:10
*** IlyaE has quit IRC18:12
*** IlyaE has joined #openstack-sahara18:14
tmckaycrobertsrh, do you have any facility in the UI for printing a note on success?18:25
tmckaycrobertsrh, so when there are errors there is a red box, and when something works there is a green box, but can I control what's in the green box?18:25
* tmckay hasn't looked closely18:25
*** IlyaE has quit IRC18:32
crobertsrhYeah, you can do notes on success18:40
crobertsrhgreen box is up for whatever you want it to do18:40
crobertsrhsorry for the lag, tmckay18:41
*** IlyaE has joined #openstack-sahara18:41
*** bogdando has quit IRC18:41
tmckaycrobertsrh, that is wicked cool.  I am mucking with Java args if they are swift://, I think I am obligated to say so :)18:42
crobertsrhIf the response from the api gives a hint, the UI should be able to tell the user that you did what was best for them.18:44
openstackgerritA change was merged to stackforge/sahara-ci-config: Change sahara-api to sahara-all  https://review.openstack.org/9213918:44
tmckayok, I'll give it a try.  Icing on the cake18:44
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Updating the puppet module to reference Sahara  https://review.openstack.org/9211118:46
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Updating the puppet module to reference Sahara  https://review.openstack.org/9211118:50
*** IlyaE has quit IRC18:52
*** IlyaE has joined #openstack-sahara18:55
*** tosky has quit IRC18:59
dmitrymetmckay: review what you guys were talking about here. Indeed seems like there is no simple solution to remove .sahara prefix19:00
dmitrymeas for Hadoop user in keystone to hold trust, right, it is needed19:01
dmitrymeHeat had similar problems and they solved that by creating a users per stack (as far as I understand) in a special ‘service’ domain19:02
mattfthat's kinda scary19:02
mattfis that something that the keystone folks endorse?19:03
tmckaydmitryme, I am working on fixing up all the swift:// paths in jobs, even Java actions.  At least we can make it transparent, that is an improvement I think.19:15
dmitrymemattf: frankly, I don’t know19:15
dmitrymetmckay: could you clarify what do you mean by ’transparent’?19:16
mattfdmitryme, outlook hazy (http://en.wikipedia.org/wiki/Magic_8-Ball)19:16
dmitrymemattf: I don’t think it is that bad, we just need to take a look on discussions around that stuff in Heat. Possibly ask Heat folks on summit19:19
tmckaydmitryme, maybe bad word choice.  I want users to be able to use "swift://container/object" for input/output paths in all places, same for data sources as for job binaries19:19
tmckaysahara will insert ".sahara" suffix if it's not there19:20
tmckayI've got it working in every case but one (explicit user args to Pig jobs, edge case_19:20
tmckaydmitryme, one of our friends was troubled by this last week, and I want to be more user friendly19:21
dmitrymetmckay: yea, I see. Such approach still leaves an issue: these edge cases will cause the same troubles19:23
mattfkinda begs the question, if all swift urls coming in are effectively .sahara, why do we need .sahara at all?19:23
mattfswift plugin using a ".sahara" persona seems more like a matter of configuration instead of payload metadata19:24
dmitrymetmckay: right now I see the only way to avoid trouble: it is to do validation instead of transparency: i.e. if user supplied url without .sahara, show him an error. That will work in those places, where we can add validation (like DataSources) and will serve as education and prepare user to work with functionality without validation (pig script input)19:27
tmckaymattf, yes, I've been thinking about that too.  dmitryme, what if the hadoop swift implementation read the service_name from the "conf" object instead of parsing it from the swift:// url?  It could still be set per job, I think.19:27
mattfentering a swift:// could prompt for credentials instead of throwing an error19:28
tmckaydmitryme, that is a another option.  I added some validation like that last week.19:28
mattfwhich is what tripped my wtf alarm19:29
tmckaymattf, the UI at least already requires credentials.  The trouble is getting the credentials into the hands of hadoop when it tries to dereference the url and get some data.  Here is the original jira https://issues.apache.org/jira/browse/HADOOP-854519:31
tmckaybe back soon19:31
tmckaymaybe we should kick this around at Summit19:32
dmitrymetmckay: getting service name from config sounds like not a bad idea, though I am not sure how easy will it be to persuade Hadoop community19:33
dmitrymeand we will have to leave support for service name in URL at least for backward compatibility in hadoop-swift driver19:33
tmckaydmitryme, yes.  I looked at the patch this morning, looked like maybe it was viable, but it does mean diving in to hadoop again19:34
dmitrymeit has one more advantage: it allows to use different swifts for input and output19:34
mattfdmitryme, is that the motivating use case?19:35
dmitrymebut we since we do not support the later anyway (we use only fixed .sahara suffix), we can do without it just fine19:35
*** tmckay is now known as tmckay_errand19:35
tmckay_errandI'll read back when I return :)  Hot topic.19:35
*** IlyaE has quit IRC19:35
dmitrymemattf: maybe, I don’t remember discussions which occurred at that time. Besides it was another guy from Mirantis who drove the driver ( :-) ) from our side19:36
mattfcall me crazy, but since sahara controls the config for the job, it could inject a config w/ a glob. if url == xyz use creds A, if url == prq use creds B19:38
mattfbut that's kinda ugly19:39
mattfthe url should actually be swift:/// <- 3 slashes. i think we can use the first field for a tenant19:39
dmitrymemattf: but there is a single config for input and output, so you can’t make them pointing to different swifts :-)19:40
*** crobertsrh has quit IRC19:40
* mattf googles around for a good example19:41
*** crobertsrh has joined #openstack-sahara19:41
mattfhttp://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/FileSystemShell.html - All FS shell commands take path URIs as arguments. The URI format is scheme://authority/path. For HDFS the scheme is hdfs, and for the Local FS the scheme is file. The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used. An HDFS file or directory such as /parent/child can be specified as hdfs://19:42
mattfnamenodehost/parent/child or simply as /parent/child (given that your configuration is set to point to hdfs://namenodehost).19:42
dmitrymemattf: there is no need for tenant_id field. It could be get from keystone using the provided token19:42
mattfthe authority part19:42
mattfhttp://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileContext.html too19:43
mattfor http://hadoop.apache.org/docs/r2.4.0/api/org/apache/hadoop/fs/FileContext.html19:43
mattfimho, that's a better place to stick the ".sahara"19:43
aignatovdmitryme: tmckay_errand prabably we could add sahara suffix as General config over the whole cluster like “Swift Enable” and put it even to the UI page, also we could add the possibility to change it and add more suffixes if user’s swift works over external swift19:49
dmitrymemattf, aignatov: to late for my timezone, I’d rather review it tomorrow19:49
openstackgerritRobert Nettleton proposed a change to openstack/sahara-image-elements: Fixes image creation script to work on CentOS  https://review.openstack.org/9219219:50
mattfdmitryme, nite!19:51
openstackgerritA change was merged to openstack/sahara-image-elements: Update Spark element  https://review.openstack.org/9134019:57
* mattf feels ill wrt what it takes to run diskimage-create.sh20:00
* mattf recommends never running it on a machine you care about20:00
elmikolol20:00
elmikomattf: are you suggesting i should not have run it on my work laptop?20:01
* mattf hands elmiko lighter fluid and a match20:01
mattfbob's cr enables epel on your machine if it happens to be centos20:02
elmikoyea, i saw that20:02
mattfi'd like to -1, but we've kinda opened the flood gate here and i don't feel like it's good to close them just on this one review20:02
mattfi have python-argparse because it comes w/ rhel osp and rdo20:03
elmikotough call, i'm not sure what the correct solution is. if we don't add epel, we need to hack around arparse, or pull just 1 rpm, or ?20:04
mattfargparse is in dib since jan 2013, so...ship sailed there20:04
elmikoseems like no good option20:05
mattfyum install url to argparse rpm is fragile too20:05
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Updating the puppet module to reference Sahara  https://review.openstack.org/9211120:09
aignatovtmckay_errand: approved hdfs simplifications, nice patch and simple :)20:17
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Updating the puppet module to reference Sahara  https://review.openstack.org/9211120:22
openstackgerritA change was merged to openstack/sahara-image-elements: Fixes image creation script to work on CentOS  https://review.openstack.org/9219220:23
mattfaignatov, did anyone response to the retrospective?20:24
mattfi completely forgot20:24
openstackgerritChad Roberts proposed a change to stackforge/puppet-sahara: Updating the puppet module to reference Sahara  https://review.openstack.org/9211120:25
aignatovmatff, only Daniele Venzano, man who is working on Spark stuff20:26
mattfwoman20:27
mattfaignatov, so...small sample, doh!20:27
aignatovmattf, it’s just a sample :)20:36
elmikogeneral question, if i am doing some sahara debugging is it safe to insert pdb.set_trace commands if i am running sahara-api from the cli?20:39
mattfaignatov, hehe20:41
mattfelmiko, i'd hope so. tmckay_errand and crobertsrh may have done that in the past.20:41
*** mattf is now known as _mattf20:42
crobertsrhI believe so.  I usually just do my debugging in eclipse.20:42
crobertsrhtmckay_errand would know for sure20:43
elmikocrobertsrh: never done python work in eclipse, does it have the py stuff out of the box or did you install something?20:43
crobertsrhThere is a plugin, PyDev, that I use.  I shouldn't say "eclipse" too loudly.  It doesn't seem to be too popular with anyone besides me.20:44
elmikolol, use what you like that's my philosophy20:44
elmikoalthough i have had enough eclipse, i wrote an rcp application several years ago, so yea...20:45
crobertsrhI came from a job using Eclipse for Java, so I was familiar with it.  I am just too lazy to learn something else20:45
elmikolol20:45
elmikoeclipse+java seems like a total win20:45
crobertsrhI'm liking it better without the java part involved as much :)20:46
elmikohaha, is that more about eclipse or java though ;)20:46
*** tmckay_errand is now known as tmckay20:49
elmikodo i need to have sources keystone_adminrc for the term where i'm running sahara-api?20:50
tmckayelmiko, yes, pdb.set_trace() is fine20:52
elmikotmckay: cool, thanks20:52
tmckayelmiko, the only time it messes things up that I know of is when you insert it in tests, then do tox -e py2720:53
elmikook20:53
tmckaytox can't handle it.  That's why you have to run tests directly with python -m testtools blah blah blah in that case20:53
elmikoi'm just doing some debugging and i want to inspect the output of some stuff in the hdp plugin20:53
tmckayyeah, should be fine.  I embed it and run sahara-api from the command line all the time20:54
elmikoalthough, now when i hit the sahara urls i'm getting weird authentication errors :(20:54
elmikodo i need to source the keystonerc for sahara-api?20:54
tmckayI think so20:54
tmckayI always do20:54
elmikook20:55
*** IlyaE has joined #openstack-sahara20:58
elmikotmckay: do you know if i need to have keystonerc sourced for running horizon from cli?20:58
tmckayalso yes I think20:59
elmikok20:59
*** bradd1 has quit IRC20:59
*** crobertsrh has quit IRC21:02
*** boris-42 has quit IRC21:15
*** boris-42 has joined #openstack-sahara21:16
tmckayhmm, does new gerrit have a work-in-progress feature?21:16
tmckayah, yes21:16
elmikotmckay: when you run local, do you use the sahara.conf.sample ?21:23
openstackgerritA change was merged to openstack/sahara: Allow HDFS data source paths without the hdfs:// scheme  https://review.openstack.org/9166421:23
tmckayelmiko, yes, I copy it and modify.  The full sample21:24
elmikook, i tried to get away with using devstack's sample and it's causing hell!21:24
elmikoi keep getting crazy auth errors when hitting the sahara endpoints21:25
elmikotmckay: have you ever seen something like this from horizon? http://paste.openstack.org/show/79029/21:28
tmckayelmiko, yes, usually that means something went wrong on the Sahara side.  If you look through the Sahara output you may see it21:30
openstackgerritTrevor McKay proposed a change to openstack/sahara: Add ".sahara" suffix automatically to swift URLs in workflows  https://review.openstack.org/9221021:31
elmikohrm21:34
elmikoso many configurations... lol21:36
elmikotheoretically it's possible that i could run an openstack on another machine and then have a local horizon/sahara pointing to that machine?21:37
tmckayyes21:39
tmckaydimitryme, aignatov, SergeyLukjanov, mattf, elmiko, I put up https://review.openstack.org/91664 to help start a discussion exploring what we might do to smooth the edges on Hadoop swift URLs.  It would be nice if the URLs for binaries vs data were uniform.  Not sure if hiding the difference this way is a good idea or not.21:42
tmckaysee you all tomorrow!21:42
elmikohave a good one21:42
aignatovtmckay: see you21:43
tmckayaigantov, bye!21:43
*** ErikB1 has quit IRC21:45
*** tmckay has quit IRC21:46
*** IlyaE has quit IRC21:56
*** IlyaE has joined #openstack-sahara22:09
*** ErikB1 has joined #openstack-sahara22:21
*** _mattf is now known as mattf22:21
*** elmiko is now known as elmiko_22:23
*** bradd1 has joined #openstack-sahara22:31
*** ErikB1 has quit IRC23:01
*** IlyaE has quit IRC23:12
*** IlyaE has joined #openstack-sahara23:16
*** elmiko_ has quit IRC23:27
*** bradd1 has quit IRC23:31
*** IlyaE has quit IRC23:45

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!