Tuesday, 2016-02-16

« Monday, 2016-02-15 Index Wednesday, 2016-02-17 »
*** witlessb has quit IRC01:06
rickflareanyone seen this from the sahara element generator01:23
rickflareip6tables: Saving firewall rules to /etc/sysconfig/ip6table[FAILED]01:23
*** david-lyle has joined #openstack-sahara01:23
*** esikachev has joined #openstack-sahara01:45
*** esikachev has quit IRC01:49
*** plieb has joined #openstack-sahara01:50
*** plieb has left #openstack-sahara01:50
*** itisha has quit IRC02:37
*** tellesnobrega is now known as tellesnobrega_af03:31
*** esikachev has joined #openstack-sahara03:45
*** esikachev has quit IRC03:49
*** david-lyle has quit IRC04:03
*** tellesnobrega_af is now known as tellesnobrega04:09
*** tellesnobrega is now known as tellesnobrega_af04:25
*** hid-kanetoshi has joined #openstack-sahara04:34
*** hid-kanetoshi has quit IRC04:35
*** n-anzen has joined #openstack-sahara04:35
*** dave-mccowan has quit IRC04:47
*** david-lyle has joined #openstack-sahara04:55
openstackgerritAkihiro Motoki proposed openstack/sahara-dashboard: Translation setup for sahara-dashboard  https://review.openstack.org/28049004:59
*** Poornima has joined #openstack-sahara05:01
*** sgotliv has joined #openstack-sahara05:24
*** AndreyPavlov has joined #openstack-sahara05:44
*** coolsvap|away is now known as coolsvap05:56
*** vgridnev has joined #openstack-sahara06:18
*** vgridnev has joined #openstack-sahara06:20
*** vgridnev has quit IRC06:21
*** vgridnev has joined #openstack-sahara06:22
*** vgridnev_ has joined #openstack-sahara06:22
*** vgridnev has quit IRC06:26
*** david-lyle has quit IRC06:39
*** david-lyle has joined #openstack-sahara06:40
*** coolsvap is now known as coolsvap|away06:58
*** Erming__ has joined #openstack-sahara07:06
*** Erming_ has quit IRC07:07
*** rcernin has joined #openstack-sahara07:37
*** ekarlso has quit IRC07:44
*** ruhe has quit IRC07:44
*** ruhe has joined #openstack-sahara07:45
*** AndreyPavlov has quit IRC07:51
openstackgerritGeorgy Dyuldin proposed openstack/sahara-dashboard: Add testcase 'Run EDP jobs with many types'  https://review.openstack.org/28053707:56
*** ekarlso has joined #openstack-sahara07:57
*** sgotliv has quit IRC07:57
*** coolsvap|away is now known as coolsvap08:09
*** witlessb has joined #openstack-sahara08:11
*** vgridnev has joined #openstack-sahara08:26
*** coolsvap is now known as coolsvap|away08:26
*** vgridnev has quit IRC08:27
*** vgridnev has joined #openstack-sahara08:29
*** vgridnev has quit IRC08:31
openstackgerritGeorgy Dyuldin proposed openstack/sahara-dashboard: Add testcase 'Run EDP jobs with many types'  https://review.openstack.org/28053708:32
*** tmckay_dinner has quit IRC08:33
*** vgridnev_ has quit IRC08:33
*** vgridnev has joined #openstack-sahara08:34
vgridnevtest08:35
pino|worktest passed08:38
*** vgridnev has quit IRC08:38
*** vgridnev_ has joined #openstack-sahara08:39
*** vgridnev_ has quit IRC08:42
*** vgridnev has joined #openstack-sahara08:43
*** vgridnev has quit IRC08:44
*** tmckay has joined #openstack-sahara08:50
*** AndreyPavlov has joined #openstack-sahara08:59
*** al_indig_ has joined #openstack-sahara09:27
*** al_indigo has quit IRC09:29
*** sgotliv has joined #openstack-sahara09:52
*** vgridnev has joined #openstack-sahara10:14
*** vgridnev has quit IRC10:14
*** vgridnev has joined #openstack-sahara10:15
vgridnevtest10:15
*** vgridnev has quit IRC10:26
*** vgridnev has joined #openstack-sahara10:34
*** tellesnobrega_af is now known as tellesnobrega10:34
*** tellesnobrega is now known as tellesnobrega_af10:56
*** dave-mccowan has joined #openstack-sahara11:04
*** witlessb has quit IRC11:08
*** vgridnev_ has joined #openstack-sahara11:16
*** vgridnev_ has quit IRC11:16
*** esikachev has joined #openstack-sahara11:17
*** vgridnev has quit IRC11:19
*** vgridnev has joined #openstack-sahara11:19
*** witlessb has joined #openstack-sahara11:20
*** witlessb has quit IRC11:20
*** witlessb has joined #openstack-sahara11:21
*** vgridnev has quit IRC11:25
*** vgridnev has joined #openstack-sahara11:27
openstackgerritVitaly Gridnev proposed openstack/sahara: cloudera health checks implementation  https://review.openstack.org/27900711:28
openstackgerritVitaly Gridnev proposed openstack/sahara: base cluster verifications implementation  https://review.openstack.org/27358711:28
openstackgerritVitaly Gridnev proposed openstack/sahara: ambari health check implementation  https://review.openstack.org/28020311:29
*** esikachev has quit IRC11:34
*** esikachev has joined #openstack-sahara11:35
*** _degorenko|afk is now known as degorenko11:39
*** AndreyPavlov has quit IRC11:57
*** raildo-afk is now known as raildo12:11
*** raildo is now known as raildo-afk12:16
*** pcaruana has joined #openstack-sahara12:18
*** raildo-afk is now known as raildo12:20
*** raildo is now known as raildo-afk12:25
*** raildo-afk is now known as raildo12:31
*** Poornima has quit IRC12:51
*** vgridnev has quit IRC13:05
*** vgridnev has joined #openstack-sahara13:10
*** _crobertsrh is now known as crobertsrh13:24
*** sgotliv_ has joined #openstack-sahara13:24
*** sgotliv has quit IRC13:25
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: [WIP]Fix scenario tests for correct output to swift  https://review.openstack.org/27662413:26
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: [WIP]Fix scenario tests for correct output to swift  https://review.openstack.org/27662413:26
*** tmckay has quit IRC13:28
*** egafford has joined #openstack-sahara13:28
*** vgridnev has quit IRC13:30
*** vgridnev has joined #openstack-sahara13:31
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix using proxy node for checks  https://review.openstack.org/27944713:35
*** tmckay has joined #openstack-sahara13:39
*** tellesnobrega_af is now known as tellesnobrega13:46
*** raildo is now known as raildo-afk14:03
*** raildo-afk is now known as raildo14:11
vgridnevelmiko, tmckay could you please take one more looks at base cluster verifications things? https://review.openstack.org/#/q/status:open+branch:master+topic:bp/cluster-verification14:17
elmikovgridnev: i'll try to get to today, but i've got an internal project that needs attention14:18
vgridnevelmiko, thanks!14:18
crobertsrhI need to take a look too....it's on my list.14:19
*** itisha has joined #openstack-sahara14:25
openstackgerritting wang proposed openstack/python-saharaclient: Add debug testenv in tox  https://review.openstack.org/26801114:27
*** vgridnev has quit IRC14:37
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix READMEs location for sahara_tests  https://review.openstack.org/28073014:37
*** vgridnev has joined #openstack-sahara14:38
tmckayvgridnev, sure14:44
*** vgridnev has quit IRC14:47
openstackgerritEvgeny Sikachev proposed openstack/sahara-tests: Fix using proxy node for checks  https://review.openstack.org/27944714:49
*** vgridnev has joined #openstack-sahara14:49
openstackgerritSergey Reshetnyak proposed openstack/sahara: Start RPC service before waiting  https://review.openstack.org/28074114:51
*** vgridnev has quit IRC14:52
*** nkrinner has joined #openstack-sahara14:54
*** vgridnev has joined #openstack-sahara14:54
*** nkrinner has quit IRC14:54
*** AndreyPavlov has joined #openstack-sahara14:56
*** akuznetsov has joined #openstack-sahara14:57
*** araji has joined #openstack-sahara14:57
*** coolsvap|away has quit IRC14:57
*** nkrinner has joined #openstack-sahara14:59
*** nkrinner has quit IRC15:00
*** nkrinner has joined #openstack-sahara15:00
*** coolsvap|away has joined #openstack-sahara15:01
tmckayvgridnev, do you know if anyone is working on the gate? very recently, swift became very troublesome.  PUT fails all the time in the scenario tests, for example https://sahara.mirantis.com/logs/87/273587/17/check/gate-sahara-neutron-heat-vanilla_2.7.1-u14/39bbf96/console.html15:06
tmckayI noticed some changes related to swift in the sahara-scenarios repo (one to do with auth, one to do with requirements) but it's unclear to me what the issue is (maybe the swift instance itself is broken in CI?)15:07
tmckayit's hard to debug without access to a CI lab :)15:08
tmckayI've seen "503 service unavailable", I've also seen 404 I think15:09
vgridnevesikachev working on it, it looks like swift itself was broken, it's probably fixed right now15:09
tmckaythat is fantastic news, thanks :)15:09
esikachevit's working now, I hope)15:10
tmckayesikachev, thanks! I'll put a recheck on vitaly's patch and see ...15:10
*** vgridnev_ has joined #openstack-sahara15:17
*** vgridnev has quit IRC15:21
*** vgridnev_ has quit IRC15:32
*** vgridnev has joined #openstack-sahara15:32
*** nkrinner has quit IRC15:45
*** dave-mccowan has quit IRC15:53
*** esikachev has quit IRC16:03
openstackgerritWang Bo proposed openstack/sahara: Replace assertNotEqual(None,) with assertIsNotNone  https://review.openstack.org/28078816:08
*** dave-mccowan has joined #openstack-sahara16:12
*** tellesnobrega is now known as tellesnobrega_af16:13
rickflaremorning folks16:18
*** pcaruana has quit IRC16:18
tmckaymorning16:19
vgridnevtmckay, tests added!16:21
rickflareman thank you so much for the help yesterday16:22
rickflareI got my ten node spark cluster up and running16:22
rickflarei was shocked that the centos 7 builds still fail16:22
rickflarebut the ubunutu spark image had no issues16:22
tmckayvgridnev, thanks!16:30
*** agireud has quit IRC16:31
tmckayrickflare, good news. Hmm, centos 7 builds fail, huh?  For the spark plugin? I built one recently for vanilla16:31
tmckayrickflare, if centos7 builds are failing we should have a bug for that and fix it ASAP.16:31
*** esikachev has joined #openstack-sahara16:32
* tmckay checks launchpad16:32
*** agireud has joined #openstack-sahara16:32
*** tellesnobrega_af is now known as tellesnobrega16:32
tmckayrickflare, does your case sound like this or something else? https://bugs.launchpad.net/sahara/+bug/150053016:34
openstackLaunchpad bug 1500530 in Sahara "Sahara-Image-Elements Fails to generate a Centos image that will boot" [High,Triaged]16:34
openstackgerritVitaly Gridnev proposed openstack/sahara: cloudera health checks implementation  https://review.openstack.org/27900716:34
openstackgerritVitaly Gridnev proposed openstack/sahara: ambari health check implementation  https://review.openstack.org/28020316:34
openstackgerritMerged openstack/sahara-tests: Update MapReduce job  https://review.openstack.org/27453216:35
vgridnevtmckay, we don't build spark images for cents16:36
vgridnevtmckay, we don't build spark images for centos716:36
tmckayah. haven't tried for a while16:37
*** rcernin has quit IRC16:37
rickflarelooking now16:40
rickflaretmckay16:41
rickflarethat is my bug16:41
rickflareI submitted that16:41
rickflareand yes16:41
rickflareI tried Spark and centos and fedora and neither appeared to be supported16:42
rickflareI also tried hadoop 2.7.1 centos 716:42
rickflareand it fails16:43
*** AndreyPavlov has quit IRC16:43
tmckayyes, spark is only cdh on ubuntu, vgridnev is right16:43
tmckayI had forgotten16:43
rickflarebut I also tried a native hadoop16:44
rickflareusing centos 716:44
rickflareand the build failed16:44
openstackgerritVitaly Gridnev proposed openstack/sahara-dashboard: implement health checks in sahara-dashboard  https://review.openstack.org/27606716:45
*** esikachev has quit IRC16:49
*** vgridnev has quit IRC16:49
*** akuznetsov has quit IRC16:54
tmckaycrobertsrh, remember my idea of adding the '*' to label names for filter fields in sahara dashboard that will use regex? Is that just a simple cosmetic change?17:01
tmckaycrobertsrh, low-hanging fruit for rickflare17:02
*** rcernin has joined #openstack-sahara17:09
openstackgerritlu huichun proposed openstack/sahara: [EDP] Add suspend_job() for sahara edp engine(oozie implementation)  https://review.openstack.org/20144817:14
*** tellesnobrega is now known as tellesnobrega_af17:22
*** tellesnobrega_af is now known as tellesnobrega17:24
openstackgerritAkihiro Motoki proposed openstack/sahara-dashboard: Install horizon more efficiently  https://review.openstack.org/28025417:35
openstackgerritDenis Egorenko proposed openstack/sahara-ci-config: Rewrite local.sh with openstack client  https://review.openstack.org/24404417:43
*** degorenko is now known as _degorenko|afk17:45
*** esikachev has joined #openstack-sahara17:46
elmikotmckay: if you don't come up with anything today, i should be able to have another bandit related change that rickflare could take over18:00
elmikobut it won't be till tomorrow most likely18:01
*** vgridnev has joined #openstack-sahara18:03
*** AndreyPavlov has joined #openstack-sahara18:14
tmckayelmiko, awesome, thanks18:16
rickflaresweet!18:17
rickflarehave you guys droped any binary jobs into spark18:18
rickflarefrom horizon18:18
tmckayyes18:19
rickflareare these test jobs?18:19
rickflareor stuff you wrote?18:19
tmckayMostly test jobs, or something really simple ripped off from wordcount, or something like that.18:20
tmckayfor me, nothing approaching any kind of real data science18:20
tmckayrickflare, I assume you mean uploading a jar?18:21
tmckayto create a job binary18:21
*** sgotliv_ has quit IRC18:25
tmckayrickflare, okay, here's a patch for you -- if you check out the sahara project and run tox  -e pep8, you'll see pep8 errors in sahara/tests/tempest/scenario/data_processing/client_tests18:25
tmckayimport order violates the convention.18:25
tmckayNot sure how that snuck through gerrit, but anyway, that's a simple patch. If you don't fix it, I will because it is annoying me when I run pep8 on my own changes (rickflare)18:26
tmckay:)18:26
*** david-lyle has quit IRC18:27
rickflaretmckay yea if I had a test jar to use that would be great18:28
rickflaretmckay18:28
rickflarei will check it out now18:28
rickflaregithub correct?18:29
tmckaygit://github.com/openstack/sahara.git18:29
rickflarerunning on my mac now18:30
rickflarethe tox command18:31
rickflarelol18:32
rickflareguess I need to do this on linux18:32
tmckaymaybe so. http://docs.openstack.org/developer/sahara/devref/development.environment.html, might be something useful here18:33
tmckayrickflare, okay, for some test jars, you can checkout the sahara-scenario repo (https://github.com/openstack/sahara-scenario) and look in etc/edp-examples18:34
*** david-lyle has joined #openstack-sahara18:35
tmckaythere is a spark-wordcount.jar in there with instructions in the README, that ought to work18:36
rickflareok18:37
rickflaresweet18:37
rickflareim attempting to see if I can get my environment setup on osx18:37
*** esikachev has quit IRC18:52
*** esikachev has joined #openstack-sahara19:00
*** crobertsrh1 has joined #openstack-sahara19:01
*** vgridnev has quit IRC19:14
*** esikachev has quit IRC19:15
*** esikachev has joined #openstack-sahara19:18
tmckaycrobertsrh, did you see my earlier note on '*' added to filter labels?19:22
tmckayis that cosmetic, or does the name appearing in that box get passed around and compared to stuff?19:23
crobertsrh1tmckay:  it shouldn't be anything beyond cosmetic19:30
crobertsrh1Somewhere though, we should note what the heck * means19:30
crobertsrh1I have to update the user guide, so I'll certainly add a note in there19:30
tmckayoh, yeah, in a doc. but it will help folks remember19:30
tmckaycrobertsrh1, I wanted to do it anyway, and rickflare wants low-hanging fruit to get his feet wet and become an ATC :)19:31
crobertsrh1awesome....then he can get you that beer he promised :)19:31
tmckayI almost never drink -- but pizza, ice-cream, cheeseburgers, there you og19:32
tmckaygo19:32
crobertsrh1I wouldn't recommend drinking pizza, ice cream, or cheeseburgers19:33
tmckaysmoothie ...19:36
*** vgridnev has joined #openstack-sahara19:54
*** vgridnev has quit IRC20:01
*** tellesnobrega is now known as tellesnobrega_af20:03
*** tellesnobrega_af is now known as tellesnobrega20:06
*** apavlov has joined #openstack-sahara20:06
*** AndreyPavlov has quit IRC20:07
*** david-lyle has quit IRC20:18
*** esikachev has quit IRC20:21
*** sgotliv_ has joined #openstack-sahara20:25
tmckaycrobertsrh1, well, in my latest horizon, I don't see any '*' for nova search ... maybe I dreamed it, or it's been removed20:25
tmckayhmm, oh, I ran the wrong one.  There's still hope20:26
tmckayno, darn. Still wrong.20:28
crobertsrh1Hmm20:32
crobertsrh1I sorta remember seeing a * for something somewhere.  I can't confirm that it was for regex fields though.20:32
tmckayyeah, I'll poke some more. I'm pretty sure I saw it. Maybe it was old skool and got changed ...20:35
rickflareok guys I am attempting to use the job guide to test the spark word count jar20:47
tmckaycrobertsrh1, ^^ job guide :) That's you20:52
* tmckay brb20:52
crobertsrh1oh, that is totally me20:54
crobertsrh1rickflare:  I'm sure it works 100% awesome20:54
rickflareim uploading the wordcount.jar as the binary20:55
rickflarenow20:55
crobertsrh1hmmm....uploading to swift, right? :)20:55
rickflareewww20:55
rickflareno20:55
crobertsrh1ok :)20:55
rickflarei did database20:55
rickflareshould I have?20:55
crobertsrh1that's the easiest way to try it out20:55
crobertsrh1In a "production" setup, "internal database" probably wouldn't be your best choice though20:56
rickflare10-420:56
rickflareI am trying to keep it prod like20:56
rickflaremight want to note that20:57
rickflarewhen creating them20:57
crobertsrh1not a major difference, but if you had a lot of large binaries, database storage isn't the most efficient20:57
crobertsrh1no difference functionality-wise though20:57
rickflarenow that I think about it20:57
rickflareI dont even know if I have swift running20:58
rickflarein my packstack20:58
crobertsrh1heh20:58
rickflarei have not had a need for it yet20:58
rickflaresighes20:58
rickflaretbh20:58
crobertsrh1one thing at a time :)20:58
rickflarei dont know what swift is really used for20:58
crobertsrh1swift is an object store20:58
rickflareok20:58
rickflareso like if you had SANs20:59
rickflareluns etc20:59
rickflareyou could write data to them as objects20:59
crobertsrh1Manila is also a possibility (it lets us mount NFS shares and use them in our clusters)20:59
tmckayyou can put anything in there, it just stores blobs of data grouped in containers20:59
crobertsrh1not to derail you from your upcoming great success of running spark wordcount21:00
rickflareok21:00
rickflareit failed21:00
rickflareDone with Error21:00
rickflarereported21:00
crobertsrh1tmckay broke it most likely21:00
tmckayhey now21:01
*** raildo is now known as raildo-afk21:01
crobertsrh1rickflare:  Ok, here's a slight rough edge of Sahara [for now...solutions in the works]....debugging job runs21:01
tmckayokay, not to worry. now rickflare, in the case of spark, I did something very helpful here21:01
tmckaycrobertsrh1, do you want to lead him through this?21:02
crobertsrh1what I meant to say was that tmckay will save the day21:02
tmckayheh21:02
crobertsrh1Go for it21:02
crobertsrh1My method is to ssh to the master and dig in /tmp21:02
tmckaycrobertsrh, yep, that's it, but in the case of spark it's designed to be really easy.21:03
rickflareim in tmp21:03
tmckayrickflare, okay, so log into your spark master node via ssh and the key you told sahara to use, and look for /tmp/spark-edp21:03
tmckaysee it?21:03
tmckayrickflare, under /tmp/spark-edp you should see a directory named for your job, and under that there will be dirs for each run of the job21:04
tmckayin that last dir, you will see the command wrapper script that was used to launch the job, and the log21:05
tmckayyou can even launch the job by hand from there by copying the line out of the log (it should have been echoed)21:05
crobertsrh1There are plans to grab these logs and make them somehow available in the UI (or CLI)21:05
tmckaysomething in there should be an error from spark telling you what it was upset about21:05
rickflareim there21:05
rickflaredont see much21:05
rickflarein the launch_commnd.log21:06
tmckaycan you paste an ls?21:06
tmckaycheck stdout or stderr21:06
rickflareArrayIndexOutofBounds error21:06
tmckaycan you paste the whole thing?21:07
rickflareyea21:08
tmckaystdout and stderr would be good, along with the launch_command.log so I can see args21:08
crobertsrh1Fair chance of the args being wrong....not the easiest thing to know on the first run21:08
rickflarehttp://pastebin.com/UypqPnbc21:08
tmckaycrobertsrh1, I'll owe you a virtual cheeseburger smoothie if you're right :)21:09
crobertsrh1Did you provide any args?21:09
* crobertsrh1 is almost hoping to be wrong21:09
crobertsrh1I'd hate to gum up my ethernet with cheeseburger smoothie21:10
rickflarehttp://pastebin.com/7FYsh5vG21:10
rickflareno args21:10
tmckayso we need an input data source and an output data source for wordcount. crobertsrh1, did you mention that?21:10
crobertsrh1I didn't mention that, but I may have in the user guide once upon a time.21:11
tmckaylol21:11
openstackgerritMerged openstack/sahara-dashboard: Translation setup for sahara-dashboard  https://review.openstack.org/28049021:12
tmckayrickflare, okay, so we need to give it input and output data. The easiest way to do that is put some stuff in swift.21:12
crobertsrh1he mentioned that he may or may not have swift installed21:12
rickflaresooo yea21:13
rickflareswift21:13
rickflarelol21:13
rickflarehow do i know if I installed it21:13
rickflarelet me check my21:13
tmckaywell, we should be able to use hdfs from the cluster itself too21:13
crobertsrh1you can look at the service catalog21:13
tmckaykeystone service-list is the easiest way to tell21:13
rickflarewhere is that21:13
tmckayjinx21:13
crobertsrh1Or, if you have "object store" showing up in horizon somewhere, that would also indicate that swift is running.21:14
rickflarekeystone service-list didnt do anything21:14
rickflareoh wait21:15
rickflareits there21:15
rickflareand its running21:15
rickflareso swift is there21:16
tmckayokay, so you should have a keystonerc file somewhere that you can source to set env vars for os_username, os_password, etc21:16
rickflarei have done a source keystone_admin21:16
tmckayif you source that, you should be able to do "swift list"21:16
rickflareim locked and loaded21:16
tmckayit should come back empty21:16
rickflareit id21:17
rickflareit did21:17
crobertsrh1rickflare:  If anything in the UI gives you fits, you can take a look at:  http://docs.openstack.org/developer/sahara/horizon/dashboard.user.guide.html21:17
tmckaythen you can do "swift upload container_name stuff" where container_name is whatever you want to call it (rickflare maybe) and stuff is a file you want to wordcount21:17
tmckaythen if you do swift list you should see the container, and I think you can do "swift list container_name" and see your file object sitting there21:18
crobertsrh1Purely for information, but you can also upload objects into swift via Horizon if you wanted to avoid any CLI interaction (since you're looking to show-off for higher-ups, they might prefer to see fancy UI stuffs)21:19
rickflareyea21:19
rickflarethey will21:19
tmckayheh21:19
crobertsrh1They are suckers for that kind of stuff21:19
tmckayI just did exactly that :) Trying to run alongside here so I don't miss anything21:20
rickflarewhat is a container21:20
rickflareand what should I name it21:20
rickflareim in the object store of horizon21:20
tmckaycontainer is a swift word for a name space21:20
crobertsrh1container ~= folder21:20
tmckayyeah21:20
rickflareok21:20
tmckayyou can name it "sheep" if you like, or any other animal, or ...21:21
rickflareok created21:21
rickflarenow to upload to it21:21
rickflareok21:21
rickflarenow the wordcount jar21:22
rickflareis in there21:22
rickflarenow what21:22
crobertsrh1You'll also want to upload a file that you'd like to do wordcounting upon21:23
rickflaredoes it matter the type21:23
crobertsrh1upload monstertextfile.txt21:23
rickflarehas to be a txt?21:23
crobertsrh1extension doesn't matter21:23
crobertsrh1as long as the file is wordcountable21:23
crobertsrh1you can name it whatever you'd like21:24
rickflaredownloading a huge txt21:26
crobertsrh1Ok21:27
*** apavlov has quit IRC21:27
crobertsrh1after you upload it, the next step will be to create a data source that references the file you uploaded to swift21:28
rickflarewill a rtf work?21:29
rickflareok21:30
rickflarefile is there21:30
rickflarei hope the rtf is ok21:30
rickflarehow do I find the swift url?21:32
rickflareill wait because I have no idea what im doing21:32
tmckayokay, the swift url should just be swift://container/file21:32
tmckayso you have a spark job already that you can launch, which references the jar, correct?21:33
tmckayYou want to do "launch on existing cluster"21:33
rickflarehttp://memesvault.com/wp-content/uploads/Dog-Meme-I-Have-No-Idea-What-Im-Doing-03.jpg21:33
*** crobertsrh1 has quit IRC21:33
rickflareim at the guided job execution page21:33
rickflareive selected the type21:34
tmckayoh, okay21:34
rickflareand the job template21:34
rickflareis now asking for the switch URL21:34
rickflareusername and pass21:34
rickflareI have no idea what to put there21:34
tmckayokay, swift url should be "swift://containername/filename"21:35
tmckayand username/pass should be the stuff from the RC file. Now, there is a more secure way to do this with proxy users but let's ignore that for now21:35
tmckayhadoop is going to use the username and password to call back to swift and retrieve the file at runtine from cluster21:36
tmckayfor input, it will be the one you created21:36
rickflarethe filename should just be the jar right21:37
rickflarenot the word file?21:37
tmckayno, this should be the word file.  The jar should have been specified as the main lib when the job template was created21:38
tmckayI'm trying to do this alongside you, but I'm semi hosed at the moment :)21:39
rickflareok21:39
rickflareuser name and pass21:40
rickflareare the same as what I use for horizon?21:40
rickflarein the keystone file/21:40
rickflare?21:40
tmckayyes21:40
rickflarek21:40
rickflareim not sure21:41
rickflarebut under libs21:41
rickflaresomething was already there21:41
rickflareso im assuming its the jar21:41
tmckayokay, on the libs tab you don't need anything21:42
tmckaynot for spark21:42
rickflarelaunching the job21:42
rickflarewait21:42
tmckaybut you do for "choose a main binary"21:42
rickflareso21:43
rickflarefor the main binary21:43
rickflarewhat should that be21:43
rickflarethe jar?21:43
tmckayyes21:43
tmckaybah, trying to launch a spark cluster but it's failing on me21:44
rickflareok21:44
rickflarenow21:44
rickflarewhere does the wordcount file21:44
rickflarego21:45
rickflarethe text file21:45
tmckayokay, so you have uploaded a job binary (the jar file) and you have made a job template that has the job binary set as the main binary21:46
tmckaynow, you launch the job, and on the configure tab you specify the main class (unavoidable, but a pain) and then add 2 arguments for the input path and the output path21:47
tmckayinput path is the swift://container/file for the text file21:47
tmckayoutput path is swift://container/someoutputname21:47
*** apavlov has joined #openstack-sahara21:47
* tmckay just ran a job that got done with error, checks to see21:47
rickflareall I see21:49
rickflareis main class21:49
rickflareno input or output21:49
tmckayyeah, you see where it says "Arguments" ?21:51
tmckayThere is a little button under that which says "Add"21:51
rickflareyea21:52
rickflarei clicked it21:52
rickflareis is just the swift lines21:52
rickflareor is it21:52
rickflareinput path = blah blah21:53
tmckayhmm, can you cancel that and check something for me?21:53
tmckayUnder the "Data Sources" tab, do you have anything?21:54
tmckayData Processing -> Data Sources21:54
rickflareno21:54
rickflarenothing21:54
tmckayokay. We'll come back to that.  So the input and output args are just going to be the paths21:56
tmckaySo add one arg, "swift://container/filename"21:56
tmckayand one arg "swift://container/outputname"21:56
tmckayof course based on your actual container name, etc21:56
rickflareok21:56
tmckaynow two other things we have to add because of swift and because we didn't use data sources ...21:57
tmckaywe need to add 2 configs too21:57
tmckaySo above arguments, there is configurations21:57
rickflarewait21:58
rickflareshould I launch this?21:58
tmckaynot yet21:58
rickflareok21:58
tmckayit's missing the swift username and password for the input output paths21:58
tmckaypain, I know, but they have to be there21:58
rickflarek21:58
tmckayso we want a config "fs.swift.service.sahara.username" set to the username from the keystone rc21:59
tmckayand another one "fs.swift.service.sahara.password" set to the password.  Then I think you can launch21:59
tmckayOne you have a job setup, it's relatively easy to use the "relaunch" and run it again, but the first time there is a bunch of stuff to set up22:00
rickflarelaunching22:00
rickflarefailed like a mofo22:00
rickflarelol22:00
tmckayheh, okay, back to /tmp/spark-edp to see what it says this time22:01
rickflarewtf22:03
tmckayoooo, I think mine is working22:03
rickflaremy master is now refusing22:03
rickflaressh22:03
rickflarestrang22:03
rickflareok22:03
rickflareim in22:03
tmckayyay, I ran it successfully. Okay, so we have a template to compare too22:04
tmckaywhat's your stderr say?22:04
* tmckay wishes there was a good way to autodetect main class in a jar22:04
tmckaymaybe there is, I'm not a java guy22:05
tmckaybut that is something I always mess up22:05
rickflarethere is nothing22:06
rickflarein the folder at all22:06
tmckayin /tmp/spark-edp, there is a uuid-named folder that is empty?22:06
tmckayor no new uuid? (just the first one we ran)22:07
tmckaycause that probably means it errored on the sahara side, so we look in the sahara log (which I think is probably /var/log/sahara for you)22:07
tmckayOh, yeah, what's the status on horizon for that last run?22:08
tmckayFailed, or Done with Error?22:08
tmckayFailed means it crashed before it got to the cluster22:08
rickflare57960e6-47a9-4c76-877b-c645ec28aaf822:08
rickflareSparkwordcount22:08
rickflareSparkTest22:08
rickflareFailed22:08
tmckayokay, so we check /var/log/sahara22:09
tmckaythis seems harder when I talk someone through it. As a noob, with no preconceived notions, any feedback you have on this would be awesome22:09
rickflarethe api?22:09
rickflareor the engine22:10
rickflarewell22:10
tmckayah, engine I think. you can grep for the uuid above22:10
rickflarethe troubleshooting is a lot man22:10
tmckayyeah22:10
rickflarenewb admins22:10
rickflareare not going to be able to do this22:10
rickflareat all22:10
rickflareeven good ones22:10
tmckaywe've been bemoaning that for cycles and cycles22:10
rickflarewill struggle22:10
rickflarebut22:10
rickflaretbh22:10
rickflarelot is not on you guys22:10
rickflarehadoop22:11
rickflarecassandra22:11
rickflareand spark22:11
rickflareare PITA's22:11
tmckayyeah, we talk about greasing the rails for debug all the time, but it's tough. Maybe the N cycle, we can make that our focus22:11
rickflarehttp://pastebin.com/7FymykGc22:12
rickflareno idea what the hell is going on22:12
tmckayokay, just something wrong in the submission. Let's rewind a bit.22:13
tmckayWhen you look at the "Job binaries" panel, how many do you have?22:14
*** apavlov has quit IRC22:14
rickflarethree22:15
tmckayrickflare, ah, okay. So, did you successfully upload that one to the database via horizon?22:15
tmckaybefore crobertsrh1 mentioned swift?22:16
tmckayI think we should start there, to eliminate a variable, and give you some success22:16
rickflareyes22:16
tmckayI think this went wrong trying to retrieve the job binary from swift. We can dodge that for the moment by running the db one22:16
rickflareit also wont let me delete any of these22:17
tmckayso, we need to create a job template that refences the job binary that is in the db22:17
rickflareError: Unable to delete job binary: WordCountfile22:17
tmckayyeah, because they are referenced by job templates22:17
tmckayif you delete the job templates, then you can delete the other job binaries, and we can run through again. Should be quicker this time22:18
tmckayrickflare, actually, wait.22:18
tmckayyou must already have a legit job, because you launched it successfully once22:18
tmckayCan you find the job on the "Jobs" tab that says "Done with Error?"22:19
tmckaywe can do a relaunch on that one22:19
tmckaywhen you relaunch, you have an opportunity to correct stuff on the configure tab22:21
rickflarefailed again22:21
tmckayright, because we have to change stuff :)22:21
tmckayrickflare, okay, do relaunch on existing cluster, go to the configure tab, but don't launch it22:22
tmckaythis is the one that originally said "Done With Error", correct?22:22
rickflarei still can delete these binaries22:22
rickflareand i have no job templates now22:23
tmckaytry deleting all the jobs22:23
rickflarethey are all gone22:23
tmckayokay, so no job templates, and no jobs22:24
rickflareright22:24
rickflarethe stuff in swift22:24
rickflarethose job binaries22:24
rickflarei can not delete22:24
tmckaythat is strange. What's it say?22:25
*** dave-mccowan has quit IRC22:26
tmckayrickflare, that object in sahara should just be a record holding a path, so if it's not referenced by a completed job or a job template it should be deletable. Strange.22:27
tmckayrickflare, oh, is it marked "protected" ?22:29
tmckayyou can click on the name link and get info, there is a protected field22:29
rickflarewhere is that at22:30
tmckayon the job binaries page, click on the hyperlink in the name column22:30
tmckayShould have boolean values for public and protected in the output there22:30
rickflareName22:31
rickflareWordCountfile22:31
rickflareID22:31
rickflare53ea5aef-ddd0-43b4-9464-32b69e04f4a922:31
rickflareURL22:31
rickflareswift://swift://sparkstuff/sparkwordcountbinary22:31
rickflareDescription22:31
rickflareNone22:31
rickflareCreate time22:31
rickflareFeb. 16, 2016, 9:40 p.m.22:31
rickflaredont see that22:31
rickflareand I cant edit it22:32
tmckayrickflare, okay, we can come back to this and drill the sahara-api and engine log to find out why delete is failed.22:33
tmckayBut I think I see the problem22:33
tmckaythe url is swift://swift://22:33
rickflareoh shit22:34
rickflareLMAO22:34
rickflareit is22:34
tmckayso, we may have a tweak to horizon for that. It may have been horizon's fault, sahara's fault, or your fault (we'll have to figure that out)22:34
tmckayBut let's try again.22:34
rickflareim trying22:35
rickflareit was my fault22:35
rickflarei entered swift://22:35
rickflarethen the stuff22:35
rickflareit should error check for that though22:35
tmckayon the horizon page, yeah, it should be smart enough to not add if you already did22:35
tmckayexactly22:35
tmckaytrying to be helpful, it messed you up22:35
tmckayokay, so if you have the job binary, we create the job template again, then launch22:36
tmckayconfigure tab on the launch needs 5 things22:36
tmckaymain class, user and password configuration values for swift, and input and output arguments22:37
rickflarealunched22:37
rickflarerunning!22:37
tmckayawesome22:37
tmckaythat extra swift add is definitely a bug, well done22:37
rickflaredammit22:38
rickflareerror22:38
tmckaythat should make mitaka22:38
tmckayokay, done with error? or Failed22:38
rickflaredone with error22:38
tmckayyay! better22:38
tmckayk, back to /tmp/spark-edp22:38
tmckayyou will be an expert at this in no time :)22:38
* tmckay bets you got the class name wrong22:38
tmckaycuz I do, all the time22:39
rickflareException in thread "main" java.lang.ClassNotFoundException: swift://sparkstuff.sahara/monstertextfile22:39
tmckayokay, weird, I need to see more. can you paste stderr and the launch log?22:40
tmckayrickflare, ^^ that makes me thing an arg was missing or out of place22:41
rickflarehttp://pastebin.com/L5TMqjae22:41
tmckayyeah, the wrapper class tried to take the input path as the class to run22:42
tmckayyou have the launch log?22:43
rickflarehttp://pastebin.com/HAFaBM9822:43
tmckaythank you :)22:43
* tmckay compares to my successful run22:43
tmckayman, I wish I could see your screen :) so much easier to show someone in person22:43
rickflarepm me22:45
tmckayrickflare, okay, looks to me like you didn's set the main class value22:45
rickflareahh22:45
rickflareok22:45
tmckayhold, should be exactly this ...22:46
tmckaysahara.edp.spark.SparkWordCount22:46
tmckaynow, the only other potential problem that I might anticipate is that Oozie has problems with jar files that do not end in ".jar". I don't know if spark does, but you named your binary wordcountreal22:47
tmckaysahara uses the name of the object as the filename22:47
tmckayso if it fails again, we'll try making one called wordcountreal.jar  If that's an issue, it should be a bug too (sahara should force .jar on the name)22:48
rickflarei need the same args22:48
rickflareand configs22:48
rickflareright22:48
tmckayyeah, this is where relanuch is great22:48
tmckayrickflare, so go to the Done with Error job on the jobs page and select "relaunch on existing cluster" from the menu to the right22:49
tmckaymost of your args should still be there, username and password are probably blank22:49
tmckaywe just need to add the class22:49
rickflarepending22:50
rickflarerunning22:50
* tmckay come sahara, don't embarass me22:50
tmckayyou can log in and watch it in /tmp/spark as it goes ...22:50
rickflarerunning longer22:50
rickflaredone with error22:50
tmckayokay, let's see what it says this time :) must be close22:51
tmckayI may have to go here soon22:51
tmckaybut I want to see this work22:51
rickflarehttp://pastebin.com/PNZbyKfH22:51
rickflarelots more22:52
rickflarein stderr22:52
rickflarehttp://pastebin.com/5fwcNems22:52
tmckayhmm, so, I wonder. if you do "swift list sparkstuff" is there an outputfile there?22:54
tmckayI don't see any failure here. anything in stdout?22:54
tmckaymight have failed trying to write the result22:55
rickflareoh wait22:55
rickflarei have to make a outputfile?22:56
rickflarei did not make one22:56
rickflarein the object sotre22:56
tmckaydid you send me the whole stderr? I only have 41 lines22:56
tmckayno, it should auto create22:56
tmckayI think it will actually fail if it already exists. We have a fancy url fix for that (we can talk about that later)22:56
rickflarehttp://pastebin.com/6Fni64GV22:56
rickflarewhich i know i set22:57
rickflarebtw22:57
tmckayk, what does your spark.xml say?22:58
tmckayform the job run dir on master22:58
tmckayrickflare, mine looks like this22:59
tmckayhttp://pastebin.com/eZnecEW422:59
tmckaywe are so close ... another way to do this is put input/output in hdfs, and then give the hdfs://blah/blah paths as input and output23:00
* tmckay is going to save this irc log and mine it for user experience improvements23:01
rickflarehttp://pastebin.com/8AJKV7yh23:02
rickflareis mine23:02
*** araji has quit IRC23:03
tmckaystrange23:04
rickflareyea23:04
rickflareim going to have to run soon23:05
tmckayrickflare, this could be a classpath issue. What version of spark are you using?23:05
rickflarewe can pick this up tomorrow23:05
tmckayrickflare, yeah, me too23:05
rickflaregood question23:05
rickflarei think its 1.6.023:05
tmckayyeah, iirc I had to tweak this for 1.3.1, and it changed for 1.6.023:05
tmckayrecently23:05
*** sgotliv_ has quit IRC23:05
rickflare 1.6.023:06
tmckayk, we'll poke tomorrow. good news is, you did everything right23:06
rickflaresweet23:06
rickflareand i learned23:06
rickflare a lot23:06
tmckayif we use hdfs paths, it will work23:06
rickflareabout how the jobs are invoked23:06
tmckayyeah, we got pretty far. I'll have to research, a change about this came through in recent memory23:06
tmckayI think23:06
* rickflare tips his hat to tmckay23:06
tmckaytalk to you tomorrow23:06
rickflareok take care23:07
rickflareand thanks again23:07
tmckayyou too23:07
tmckaynp, I want you to be successful23:07
tmckaywe all do :)23:07
rickflarethat means a great deal to me23:07
tmckayit may be as easy as patching your sahara binaries on your controller23:07
tmckaycuz you're not running master23:08
tmckayalright, see you tomorrow23:08
rickflaretake care!23:10
*** sgotliv has joined #openstack-sahara23:17
*** rcernin has quit IRC23:18
*** DuncanT_ has joined #openstack-sahara23:30
*** dave-mccowan has joined #openstack-sahara23:31
*** rickflare2 has joined #openstack-sahara23:31
*** kgalanov_ has joined #openstack-sahara23:33
*** ruhe2 has joined #openstack-sahara23:35
*** ekarlso has quit IRC23:37
*** ruhe has quit IRC23:37
*** rickflare has quit IRC23:37
*** DuncanT has quit IRC23:37
*** kgalanov has quit IRC23:37
*** ruhe2 is now known as ruhe23:37
*** rickflare2 is now known as rickflare23:37
*** kgalanov_ is now known as kgalanov23:38
*** DuncanT_ is now known as DuncanT23:43
*** ekarlso has joined #openstack-sahara23:47
*** ekarlso has quit IRC23:48
*** ekarlso has joined #openstack-sahara23:49
« Monday, 2016-02-15 Index Wednesday, 2016-02-17 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!