Monday, 2014-08-18

*** Networkn3rd has joined #openstack-sahara00:10
*** Networkn3rd has quit IRC00:12
*** witlessb has quit IRC00:14
*** xianghui has joined #openstack-sahara00:55
*** shakayumi is now known as shakamunyi00:59
*** Networkn3rd has joined #openstack-sahara01:00
*** shakayumi has joined #openstack-sahara01:46
*** shakamunyi has quit IRC01:49
*** witlessb has joined #openstack-sahara01:56
*** witlessb has quit IRC02:01
*** elmiko has joined #openstack-sahara02:33
*** elmiko is now known as _elmiko02:34
*** Networkn3rd has quit IRC03:25
*** akuznetsov has joined #openstack-sahara03:44
*** akuznetsov has quit IRC03:44
*** Krast has joined #openstack-sahara03:56
*** akuznetsov has joined #openstack-sahara04:58
*** ViswaV has joined #openstack-sahara05:25
*** ViswaV_ has joined #openstack-sahara05:31
*** witlessb has joined #openstack-sahara05:32
*** ViswaV has quit IRC05:34
*** witlessb has quit IRC05:37
*** ViswaV_ has quit IRC07:21
*** Longgeek has joined #openstack-sahara07:22
*** skolekonov has joined #openstack-sahara07:28
*** Longgeek has quit IRC07:41
*** Longgeek has joined #openstack-sahara07:42
*** Longgeek has quit IRC07:42
*** Longgeek has joined #openstack-sahara07:44
*** Longgeek has quit IRC07:44
*** Longgeek has joined #openstack-sahara07:45
*** ViswaV has joined #openstack-sahara07:57
*** Longgeek_ has joined #openstack-sahara07:58
*** Longgeek has quit IRC08:00
*** IvanBerezovskiy has joined #openstack-sahara08:04
*** witlessb has joined #openstack-sahara08:21
*** xianghui has quit IRC08:27
*** ViswaV has quit IRC08:31
*** xianghui has joined #openstack-sahara08:31
*** Timotey has joined #openstack-sahara09:00
*** Longgeek_ has quit IRC09:02
*** tnovacik has joined #openstack-sahara09:35
*** tosky has joined #openstack-sahara09:43
*** Longgeek has joined #openstack-sahara10:03
*** oikawa has quit IRC10:05
*** oikawa has joined #openstack-sahara10:06
*** Longgeek has quit IRC10:08
openstackgerritA change was merged to openstack/python-saharaclient: Fix duplicate help strings  https://review.openstack.org/11435510:39
*** xianghui has quit IRC11:07
*** xianghui has joined #openstack-sahara11:08
*** Longgeek has joined #openstack-sahara11:19
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Parallel testing EDP jobs  https://review.openstack.org/11096911:20
*** Longgeek has quit IRC11:24
*** Longgeek has joined #openstack-sahara11:28
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Fix starting instances after scaling for CDH  https://review.openstack.org/11287711:37
*** Longgeek has quit IRC11:44
*** Longgeek has joined #openstack-sahara11:45
*** Longgeek_ has joined #openstack-sahara12:02
*** Longgeek has quit IRC12:04
*** miqui has quit IRC12:07
*** DenisEgorenko has joined #openstack-sahara12:55
*** _elmiko is now known as elmiko13:22
*** Networkn3rd has joined #openstack-sahara13:22
openstackgerritMichael McCune proposed a change to openstack/sahara-specs: Updating EDP Swift trust authentication spec  https://review.openstack.org/11359113:37
*** ylobankov has joined #openstack-sahara13:37
*** tmckay has joined #openstack-sahara13:41
*** tmckay has left #openstack-sahara13:43
*** tellesnobrega has joined #openstack-sahara13:44
*** miqui has joined #openstack-sahara13:45
tellesnobregahi, i asked here a while ago about how to spin up cluster via cli, because i need to test a new plugin and someone sent me a link with some examples on how to do it, i can't find it in the logs (can't remember when this happened) if anyone knows about a link that has examples on how to do this i will be grateful if you post it here13:47
elmikotellesnobrega: are you looking to use the sahara cli app?13:51
tellesnobregamaybe13:53
tellesnobregaother option is to do as trevor suggested to create a test file13:54
tellesnobregabut first i would like to test via cli to see it really working13:54
elmikotellesnobrega: ok, `sahara --help` from the cli will show you the available options13:55
tellesnobregaok13:55
elmikoyou basically follow the same workflow as the ui, create node groups, then create a cluster template, then create the cluster13:55
elmikoon the other hand, if you want to use the rest api(with httpie or curl), then start here http://docs.openstack.org/developer/sahara/restapi/index.html13:56
toskyelmiko: maybe Matt's blog, or did you have some updated example on your blog?13:57
toskytellesnobrega: maybe this one, but it requires some updates on few commands http://spinningmatt.wordpress.com/2014/01/29/hadoop-on-openstack-with-a-cli-creating-a-cluster/13:58
elmikotosky: that's probably correct. i haven't done a blog on using the sahara cli13:58
tellesnobregathanks guys13:59
toskyelmiko: I think it was about installation only then13:59
elmikoi think the other place to check is tmckay's integration tests for the cli. that's buried in the code somewhere.13:59
toskybut you did write something :)13:59
elmikotosky: yea, my stuff was mostly about installation13:59
* elmiko makes a note to write a post about cli tools14:00
*** Krast_ has joined #openstack-sahara14:04
*** miqui_ has joined #openstack-sahara14:04
*** xianghuihui has joined #openstack-sahara14:05
*** miqui has quit IRC14:11
*** xianghui has quit IRC14:11
*** Krast has quit IRC14:11
*** tmckay has joined #openstack-sahara14:13
openstackgerritDenis Egorenko proposed a change to stackforge/sahara-ci-config: Refactoring localrc  https://review.openstack.org/11498214:14
*** miqui_ is now known as miqui14:21
*** tnovacik has quit IRC14:35
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Parallel testing EDP jobs  https://review.openstack.org/11096914:51
*** rustlebee is now known as russellb14:52
openstackgerritTrevor McKay proposed a change to openstack/sahara: Add Spark integration test  https://review.openstack.org/11269815:03
openstackgerritDenis Egorenko proposed a change to stackforge/sahara-ci-config: Refactoring localrc  https://review.openstack.org/11498215:13
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: [HDP1.3.2] Fixed bug with decommissioning cluster  https://review.openstack.org/11418715:18
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: [HDP1.3.2] Fixed bug with decommissioning cluster  https://review.openstack.org/11418715:23
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: [HDP1.3.2] Fixed bug with decommissioning cluster  https://review.openstack.org/11418715:26
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: Add notifications to Sahara  https://review.openstack.org/10898215:45
*** akuznetsov has quit IRC15:53
*** Networkn3rd has quit IRC16:01
*** ViswaV has joined #openstack-sahara16:04
*** skolekonov has quit IRC16:04
*** ViswaV has quit IRC16:06
*** akuznetsov has joined #openstack-sahara16:30
*** Networkn3rd has joined #openstack-sahara16:46
openstackgerritSergey Reshetnyak proposed a change to stackforge/sahara-ci-config: Enable pig job for Hadoop 2.4.1  https://review.openstack.org/11502616:54
openstackgerritYaroslav Lobankov proposed a change to openstack/sahara: The first bunch of tests in Tempest style  https://review.openstack.org/11503317:31
*** Longgeek_ has quit IRC17:32
openstackgerritA change was merged to openstack/sahara: Unit tests for CDH plugin  https://review.openstack.org/11351317:35
openstackgerritYaroslav Lobankov proposed a change to openstack/sahara: The first bunch of tests in Tempest style  https://review.openstack.org/11503317:38
*** Networkn3rd has quit IRC18:11
*** Networkn3rd has joined #openstack-sahara18:13
openstackgerritA change was merged to openstack/sahara: Remove unused parameter from CDH IT  https://review.openstack.org/11325818:24
*** tosky has quit IRC18:45
*** tnovacik has joined #openstack-sahara18:55
tellesnobregaelmiko: im trying here to create a cluster template via cli19:09
tellesnobregaas i see hadoop_version is a required field right?19:09
elmikotellesnobrega: i would imagine so19:10
tellesnobregayeah, that wont work. We need to remove that for storm19:10
tellesnobregastorm doesnt use hadoop19:10
elmikohmm, what about spark does it use hadoop?19:11
elmikoi would think we would have needed to remove it for that as well19:11
tellesnobregaspark uses hadoop19:11
tellesnobregafrom what i know it run on top of hadoop19:11
elmikoahh ok19:12
elmikoare you running `sahara cluster-template-create` ?19:12
tellesnobregayes19:12
elmikoi'd say make a note of these things and create bugs on launchpad. we'll need to keep track of what needs to be changed for storm.19:13
elmikotmckay: didn't you write integration tests for the cli?19:13
tmckayyes19:13
elmikowhere are those?19:14
tmckaysaharaclient repo19:14
elmikogot it, thanks19:14
tmckayI use them to launch clusters and run EDP all the time19:14
tellesnobregaelmiko: ok19:15
elmikotmckay: do you know if that hadoop_version parameter is required in the json?19:16
elmikotellesnobrega: np, i think we will have more changes to make for storm. in the UI that hadoop version is all over the place.19:16
tellesnobregayes19:16
tmckayelmiko, I believe the plugins check that value19:16
elmikoack19:17
elmikotellesnobrega: then yea, best to start opening bugs about these issues19:17
elmikotellesnobrega: and if you have ideas about how to fix them, just assign the bug to yourself and then propose a patch19:17
tmckaynot sure if it's technically required, though.  The validation stuff should say19:18
elmikotmckay: sounds like tellesnobrega is running into issues creating storm cluster from cli19:18
elmikoit might be requiring hadoop version, when for storm it isn't even applicable19:19
tmckayfor spark, hadoop version is just a string that the spark plugin is expecting19:19
tmckayso, for my spark runs, it's set to 1.0.019:19
tmckayIt gets matched against the tags on the image19:19
elmikoright19:19
elmikodoes spark actually require a certain hadoop version?19:19
tmckaywell, it's a misnomer.  In this case it really means "spark version"19:20
tmckayThe name should probably change at some point19:20
elmikotellesnobrega: ^^19:20
tmckaythe vanilla plugins, for instance, use it to fork between the vanilla 1 and the vanilla 2 plugin19:21
elmikoso that hadoop version stuff should probably be looked into again19:21
tmckayhold, on, I have a paste for how spark uses it I think19:21
elmikosounds like we need plugin_type and plugin_version or something19:22
tellesnobregaelmiko: ++19:22
tmckaytellesnobrega, elmiko, okay, if you look in sahara/plugins/spark/plugin.py spark just ignores the value, basically19:23
tmckaythe cli tests use it to find the image19:23
tmckayso, storm can effectively ignore it19:23
elmikocool, thanks for digging that up19:23
tmckayWe need to revisit hadoop_version at some point, now that we have "non" hadoop paradigms19:24
tmckay(even though spark still uses hdfs)19:24
elmikoyea, that totally what tellesnobrega is running up against19:24
tmckayStorm can dictate whatever value it wants there.  I would use the storm version19:24
tellesnobregatmckay: that is what i was thinking19:25
*** Networkn3rd has quit IRC19:25
tmckayyeah, I think that's fine19:26
tellesnobregawe would rewrite the creating method to make this hadoop_version option flexible? and accept other options like storm_version?19:30
elmikoimo we probably need to look at separting verion from plugin, and have some way to specify the plugin type and the version19:31
*** Networkn3rd has joined #openstack-sahara19:33
elmikols19:34
elmikomt19:34
tellesnobregasounds good19:51
*** shakayumi has quit IRC19:53
*** mikehn_ has joined #openstack-sahara19:57
*** mikehn has quit IRC19:58
tellesnobregai was able to create a cluster template now20:02
tellesnobregathe problem was my get_config that was just a pss20:02
tellesnobregapass20:02
tellesnobregasince storm, for now, is not getting any configs. I plan to improve that later20:03
tellesnobregaso i changed to an empty dictionary so it could pass the validation process20:03
elmikotellesnobrega: nice20:03
*** Networkn3rd has quit IRC20:22
elmikodo we have any guidelines for making sahara-all exit due to error?20:39
elmikoor should i just let the exception raise and allow that to stop the process?20:39
elmiko(this is for domain detection btw, so sahara will need to exit if a domain is requested but can't be found)20:40
elmikotmckay: any thoughts on that ^^20:51
tmckayhmm, I don't think I've run into that before.20:52
tmckayelmiko, the domain stuff is for swift jobs, right?20:53
elmikoso, like on start, if the user has requested sahara to use a proxy domain, but sahara can't connect to that domain, should it just blow up?20:53
elmikoyea20:53
elmikoi'm not finding a universal exit command for sahara20:53
tmckaywell, if it only uses the domain in the case of jobs that reference swift objects, you could still launch a cluster with it, or run jobs with hdfs data, right?20:54
tmckayso maybe not a total kill20:54
tmckayI don't know if a universal exit20:54
tmckayI think Sahara's philosophy is "continue to accept requests and maybe I'll be able to do something" :)20:54
elmikoi'm only setting it up to universal exit if the user explicitly called for a proxy domain and it can't find it.20:55
tmckayis there a ctrl-c handler anywhere?20:55
*** miqui has quit IRC20:55
elmikoi don't think so, but i'll hunt around20:55
tmckayelmiko, the fact that ctrl-c seems to just bail, without any structured shutdown, makes me thing that throwing a fatal exception early in the setup code would be fine20:57
elmikoi guess, i'll just add a log message for now and let it continue to operate20:57
elmikotmckay: ok, thanks for looking20:57
tmckayyeah, log message, followed by failures on EDP jobs that try to access swift..  That wouldn't be terrible.20:58
elmikotmckay: given that we are leaving backward compat in place, it's easy enough to let sahara continue to operate. but all swift access with then fail without username/pass being entered.20:58
elmikook20:58
tmckayack.  Makes sense to me20:59
elmikotmckay: could be that flask implements a ctrl-c handler21:06
tmckayhmm, good point21:06
openstackgerritTrevor McKay proposed a change to openstack/sahara-specs: Add a config option to block the Oozie server port  https://review.openstack.org/11511321:10
tmckayelmiko, when you get a chance ^^21:10
elmikowill do21:10
tmckayneed to think about the best way to express config options ...21:10
tmckayconcise, intuitive, powerful21:10
tmckaylike Shazam!21:10
elmikolol, just like Shazam!21:11
elmikoyou should be aware that i added you as an "Other Contributor" to the swift-auth spec =)21:11
tmckayglad to help21:13
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Added create_hdfs_dir method to oozie edp engine  https://review.openstack.org/11511521:17
*** alazarev has joined #openstack-sahara21:17
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Removed one round trip to server for HDFS put  https://review.openstack.org/11512021:34
openstackgerritAndrew Lazarev proposed a change to openstack/sahara-image-elements: Added check on parameters without leading dash  https://review.openstack.org/11370621:38
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Fix some of tests that rely on hash ordering  https://review.openstack.org/11513222:21
*** elmiko has quit IRC22:28
*** elmiko has joined #openstack-sahara22:30
alazarevtmckay: hi, have a minute to talk about EDP refactoring?22:32
tmckaysure22:33
alazarevtmckay: I’m thinking what to do with get_job_config_hints, we need plugin info there22:33
tmckayyep22:33
alazarevnow we have only job_type22:34
tmckaybiggest problem I see is what is available from the UI22:34
tmckaywe need cluster, probably22:34
alazarevbut this is in job creation screen, no cluster at that point22:34
tmckayI *think* that when that call is made, we know which cluster we're going to launch on22:34
tmckaybecause that stuff is for the job_execution22:34
tmckayoh ... hmm22:35
tmckayare you sure?22:35
alazarevno :)22:35
tmckayI thought configs were only on job_execution22:35
tmckay:)22:36
alazarevyeah, probably it is in job_execution22:36
* tmckay where's chad? Oh, yeah22:36
alazarevbut current API has nothing about cluster22:36
tmckayI think so.  So, it might be a change to the Rest API, which is why I skipped over it before, but it really needs to be fixed22:36
tmckayright22:36
tmckaywithout the cluster arg, I see no way to do it.22:37
alazarevwhat is status of separate job type for spark?22:38
alazarevI saw this somewhere22:38
tmckayoh, that should be merged already22:38
tmckayyes, Auguest 622:38
tmckayheh, Au-guest.  August.22:39
tmckaynot sure how far Chad was with the UI, though22:39
alazarevbut Java is still supportable type, right?22:39
tmckayyes.  It will work with Java.  The validation is slightly different22:39
tmckayFor the spark type, it requires "mains"22:39
alazarevI’m thinking about strict mapping from type to “base” class of engine22:40
tmckayfor Java, you have to put it in "libs".  But EDP will put them all in the same dir, so it works :)22:40
alazarevcan we separate spark java type from oozie java type?22:40
tmckayI think that could work.  spark == spark , everything else is oozie at this point22:40
tmckayStorm will be storm :)  I think22:41
alazarevthis will help to solve problem without changing API22:41
tmckaywe could drop it22:41
tmckayOnce crobertsrh fixes up the UI, we don't need Java for spark22:41
tmckayWe can just use Spark22:41
tmckayI have no problem with only "Spark" for spark, it will work fine through the client22:42
tmckayOnly UI might be broken for spark, temporarily during development22:42
tmckayalazarev, have to run an errand, I'll be back in half an hour or 40 minutes or so22:43
alazarevok, so that looks like solution22:43
alazarevwill change CR accordingly22:44
elmikoalazarev: you did the security groups patch?22:56
alazarevelmiko: security group patch is merged, auto security group patch is on review22:57
elmikoalazarev: does that error look familar to you? http://paste.openstack.org/show/97063/22:59
alazarevelmiko: yeap, need to upgrade db schema22:59
elmikoalazarev: ok, i ran `sahara-db-manage upgrade head`, do i need something else?23:00
alazarevelmiko: probably not23:01
elmikoalazarev: is it possible to clear the db with sahara-db-manage?23:01
alazarevelmiko: I don’t know, I use sqlite (despite the fact it is forbidden :) )23:03
elmikoalazarev: lol, i'm trying to switch from sqlite!23:04
elmikoi'll trying deleting it through the mysql interface23:04
elmikothanks!23:04
alazarevreturn in 35th line of 007_increase_status_description_size.py works fine for now :)23:05
openstackgerritLawrence Davison proposed a change to openstack/sahara: Documentation correction for RESTAPI's convert-config  https://review.openstack.org/11514423:05
elmikoalazarev: ok, if needed i'll try that23:05
*** tmckay has quit IRC23:06
elmikoalazarev: ok, i created a new database and i still get that error. do i need to add something to config?23:11
alazarevelmiko: I think everything should work with new db23:13
elmikoalazarev: hmm, this may be a bug then23:14
elmikoi'll keep digging23:14
*** tmckay has joined #openstack-sahara23:27
tmckaybah, I forgot to plug my laptop back in :)23:27
tmckayIRC went missing for about 20 minutes23:27
elmikowb23:30
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Made EDP engine plugin specific  https://review.openstack.org/11472123:31
tmckayhopefully I didn't miss much :)  My corporate channels haven't recovered yet, weird -- pidgin seems to always do that23:34
elmikotmckay: have you ever seen this http://paste.openstack.org/show/97063/ coming from the transient cluster removal task?23:34
tmckayusually when I see that, it means that my database ( which I have in /tmp) doesn't exist :)23:35
tmckaywell, something like that23:35
tmckayI see operationalerrors when I boot fresh and start up Sahara, and forget to create the db.  I really need to get off sqlite, and get it out of /tmp23:36
tmckayIf the DB is there, then obviously there is something wrong with the query, but I don't know what23:36
elmikotmckay: i'm switching to mysql, made the new db, ran the migrate, but i'm getting this on a fresh install23:36
tmckayhmm.  Do you get similar stuff when you log in to horizon and for instance try to create a job binary?23:37
elmikoit looks like this is coming from a cluster_get_all call23:37
elmikothat's my next step23:37
elmikoi was just testing some stuff with sahara, so i hadn't fired up horizon23:37
tmckayif that fails, then the db is probably not there, or not readable, or Sahara can't connect to it or something23:37
elmikook, i'll give it a run23:37
tmckayYou'll have to let me know how it goes, getting off sqlite has been on my short list for a while23:38
elmikotmckay: switching to mysql has been easy, but this migration is screwing something up23:39
elmikogonna try re-installing some stuff. looks like the security_groups didn't make it in or something.23:39
elmikoit's like my db doesn't have that field23:41
elmikook, got it. stale .pyc files .... /grumble23:42
elmikodinner time, back soon23:43
tmckayk23:43
tmckaydoh23:43
tmckaypyc23:43
*** witlessb has quit IRC23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!