| *** IlyaE has joined #openstack-sahara | 00:18 | |
| *** crobertsrh has joined #openstack-sahara | 00:32 | |
| *** crobertsrh has quit IRC | 00:38 | |
| *** shakamunyi has joined #openstack-sahara | 01:05 | |
| *** zhiyan_ is now known as zhiyan | 01:15 | |
| *** shakayumi has joined #openstack-sahara | 01:15 | |
| *** shakamunyi has quit IRC | 01:18 | |
| *** bradd1 has joined #openstack-sahara | 01:47 | |
| *** zhiyan is now known as zhiyan_ | 02:37 | |
| *** Ch00k has joined #openstack-sahara | 02:56 | |
| *** shakayumi has quit IRC | 02:58 | |
| *** IlyaE has quit IRC | 03:08 | |
| *** bradd1 has quit IRC | 03:27 | |
| *** IlyaE has joined #openstack-sahara | 03:29 | |
| *** bradd1 has joined #openstack-sahara | 03:32 | |
| *** IlyaE has quit IRC | 03:55 | |
| *** bradd1 has quit IRC | 03:56 | |
| *** akuznetsov has quit IRC | 03:59 | |
| *** Ch00k has quit IRC | 04:15 | |
| *** Ch00k has joined #openstack-sahara | 04:23 | |
| *** IlyaE has joined #openstack-sahara | 04:25 | |
| *** akuznetsov has joined #openstack-sahara | 04:31 | |
| *** IlyaE has quit IRC | 05:02 | |
| *** IlyaE has joined #openstack-sahara | 05:25 | |
| *** akuznetsov has quit IRC | 05:27 | |
| *** Ch00k has quit IRC | 05:28 | |
| *** akuznetsov has joined #openstack-sahara | 05:28 | |
| *** akuznetsov has quit IRC | 06:03 | |
| *** akuznetsov has joined #openstack-sahara | 06:10 | |
| *** akuznetsov has quit IRC | 06:14 | |
| *** akuznetsov has joined #openstack-sahara | 06:15 | |
| *** akuznetsov has quit IRC | 06:16 | |
| *** skolekonov has joined #openstack-sahara | 06:18 | |
| *** Ch00k has joined #openstack-sahara | 06:34 | |
| *** Ch00k has quit IRC | 06:37 | |
| *** Ch00k has joined #openstack-sahara | 06:37 | |
| *** bogdando has joined #openstack-sahara | 07:07 | |
| *** IlyaE has quit IRC | 07:23 | |
| *** IlyaE has joined #openstack-sahara | 07:28 | |
| *** IlyaE has quit IRC | 07:43 | |
| *** IlyaE has joined #openstack-sahara | 07:49 | |
| *** witlessb has joined #openstack-sahara | 07:53 | |
| *** IlyaE has quit IRC | 07:54 | |
| *** Ch00k has quit IRC | 08:14 | |
| *** Ch00k has joined #openstack-sahara | 08:15 | |
| *** ekarlso has quit IRC | 08:29 | |
| *** ekarlso has joined #openstack-sahara | 08:29 | |
| *** Ch00k has quit IRC | 08:45 | |
| *** Ch00k has joined #openstack-sahara | 08:46 | |
| *** Ch00k has quit IRC | 08:47 | |
| *** Ch00k has joined #openstack-sahara | 08:48 | |
| *** akuznetsov has joined #openstack-sahara | 09:23 | |
| *** tosky has joined #openstack-sahara | 09:45 | |
| *** Ch00k has quit IRC | 10:17 | |
| *** Ch00k has joined #openstack-sahara | 10:20 | |
| *** skolekonov has quit IRC | 11:11 | |
| *** tnovacik has joined #openstack-sahara | 11:27 | |
| *** themistymay has joined #openstack-sahara | 11:34 | |
| *** crobertsrh has joined #openstack-sahara | 11:41 | |
| *** themisty_ has joined #openstack-sahara | 11:47 | |
| *** themisty_ has quit IRC | 11:48 | |
| *** IvanBerezovskiy1 has joined #openstack-sahara | 11:48 | |
| *** IvanBerezovskiy has quit IRC | 11:49 | |
| *** themistymay has quit IRC | 11:49 | |
| *** tosky has quit IRC | 11:50 | |
| *** ErikB2 has joined #openstack-sahara | 11:53 | |
| *** Ch00k has quit IRC | 12:07 | |
| *** themistymay has joined #openstack-sahara | 12:14 | |
| *** ErikB2 has quit IRC | 12:15 | |
| *** Ch00k has joined #openstack-sahara | 12:20 | |
| *** akuznetsov has quit IRC | 12:26 | |
| *** ErikB2 has joined #openstack-sahara | 12:30 | |
| *** Ch00k has quit IRC | 12:39 | |
| *** themistymay has quit IRC | 12:45 | |
| *** Ch00k has joined #openstack-sahara | 12:47 | |
| *** ErikB2 has quit IRC | 12:48 | |
| *** tosky has joined #openstack-sahara | 12:51 | |
| *** elmiko has joined #openstack-sahara | 12:52 | |
| *** ErikB has joined #openstack-sahara | 12:55 | |
| *** themistymay has joined #openstack-sahara | 12:59 | |
| *** Ch00k has quit IRC | 13:04 | |
| *** ErikB has quit IRC | 13:12 | |
| *** ErikB1 has joined #openstack-sahara | 13:25 | |
| openstackgerrit | Chad Roberts proposed a change to stackforge/puppet-sahara: Adding RHEL support for puppet-sahara https://review.openstack.org/92368 | 13:25 |
|---|---|---|
| *** Ch00k has joined #openstack-sahara | 13:44 | |
| *** Ch00k has quit IRC | 13:44 | |
| *** Ch00k has joined #openstack-sahara | 13:45 | |
| *** Ch00k has quit IRC | 13:45 | |
| *** ylobankov_ has quit IRC | 13:56 | |
| *** ylobankov has joined #openstack-sahara | 13:56 | |
| *** tmckay has joined #openstack-sahara | 14:06 | |
| *** akuznetsov has joined #openstack-sahara | 14:07 | |
| *** bradd1 has joined #openstack-sahara | 14:23 | |
| *** mattf is now known as _mattf | 14:33 | |
| *** akuznetsov has quit IRC | 14:41 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Add ".sahara" suffix automatically to swift URLs in workflows https://review.openstack.org/92210 | 14:41 |
| *** themistymay has quit IRC | 14:45 | |
| crobertsrh | tmckay: are there any examples out there where other services use swift in a similar way? Maybe someone has already invented the wheel. | 14:48 |
| *** shakayumi has joined #openstack-sahara | 14:51 | |
| *** akuznetsov has joined #openstack-sahara | 14:55 | |
| tmckay | crobertsrh, I don't know. It seems natural to me that hadoop would authenticate via keystone and access swift with a token. That's how OpenStack is designed. | 14:56 |
| crobertsrh | That does seem like the OS way to do it. | 14:56 |
| tmckay | crobertsrh, kind of what we're doing, the trouble is just indicating the credentials to use from hadoop | 14:56 |
| tmckay | that's what all the fuss is about. Mapping from the url to the credentials, so the data can be retrieved. | 14:57 |
| crobertsrh | Ah, I get it now. I think I missed part of the conversation yesterday. | 14:57 |
| tmckay | I think some combination of trusts and configs might work to secure the credentials and normalize the URLs | 14:58 |
| tmckay | crobertsrh, my CR is an attempt to smooth it out while we invent something else :) | 14:58 |
| *** IlyaE has joined #openstack-sahara | 14:59 | |
| crobertsrh | Ah, ok. I was looking at the CR, and it seems ok, then I saw your comments which made me want to think harder. | 14:59 |
| tmckay | yeah, me too. | 15:00 |
| *** zhiyan_ is now known as zhiyan | 15:01 | |
| *** ErikB1 has quit IRC | 15:02 | |
| *** _mattf is now known as mattf | 15:04 | |
| *** ylobankov has quit IRC | 15:07 | |
| tmckay | crobertsrh, aignatov, so for Summit, should we use the stable release, or master? We have fixed some bugs. Maybe latest/greatest is best. And we have ci ;-) | 15:09 |
| crobertsrh | Whatever works :) | 15:10 |
| crobertsrh | If you're comfortable with master, that might be fine. I suppose most of your recent work/testing has probably been with master, right? | 15:11 |
| tmckay | crobertsrh, mostly. I've also been using your sahara VM. | 15:14 |
| tmckay | haven't done a git pull there, how up to date is it? | 15:15 |
| crobertsrh | Ah, ok. I haven't touched it since you started using it. It's probably about 3 weeks behind at the moment. | 15:15 |
| tmckay | k. I'll do a pull and restart | 15:15 |
| crobertsrh | Ok. The UI probably hasn't changed much at all. | 15:18 |
| crobertsrh | Sort of on hold while we merge | 15:18 |
| tmckay | alright, gotta pick up a kid, I'll bring it back up when I return (and process some pets) | 15:20 |
| *** tmckay is now known as _tmckay | 15:20 | |
| *** bradd1 has quit IRC | 15:23 | |
| *** bradd1 has joined #openstack-sahara | 15:23 | |
| *** _tmckay has quit IRC | 15:25 | |
| *** IvanBerezovskiy1 has left #openstack-sahara | 15:27 | |
| *** shakayumi has quit IRC | 15:31 | |
| *** shakamunyi has joined #openstack-sahara | 15:32 | |
| crobertsrh | Don't forget to do reviews for our UI merge CRs... https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z | 15:34 |
| *** tmckay has joined #openstack-sahara | 15:34 | |
| *** tnovacik has quit IRC | 15:34 | |
| crobertsrh | Also, https://review.openstack.org/#/c/90799/2 if you'd like something smaller to review | 15:35 |
| *** ErikB has joined #openstack-sahara | 15:37 | |
| *** tmckay has quit IRC | 15:38 | |
| *** tmckay has joined #openstack-sahara | 15:38 | |
| *** tmckay has quit IRC | 15:44 | |
| *** tmckay has joined #openstack-sahara | 15:46 | |
| *** zhiyan is now known as zhiyan_ | 15:47 | |
| *** tmckay has quit IRC | 15:52 | |
| *** tmckay has joined #openstack-sahara | 16:05 | |
| aignatov | tmckay: yep, I think we can use working latest Sahara :) | 16:06 |
| aignatov | each demo I’ve demonstrated was on master :) | 16:07 |
| tosky | do you have any idea about this? It seems an upstream issue, not packaging one: https://www.redhat.com/archives/rdo-list/2014-May/msg00018.html | 16:11 |
| *** tmckay has quit IRC | 16:13 | |
| *** tmckay has joined #openstack-sahara | 16:13 | |
| SergeyLukjanov | ++ for using latest sahara | 16:13 |
| SergeyLukjanov | but on well tested commit id :) | 16:13 |
| *** tmckay has quit IRC | 16:14 | |
| *** tmckay has joined #openstack-sahara | 16:14 | |
| tosky | tmckay, elmiko: you too, as you were running edp jobs lately ^^ | 16:15 |
| tmckay | tosky, the rhel hdp issue? I am unfamiliar | 16:16 |
| tosky | uhm, but does it make sense what he wrote? | 16:17 |
| * tmckay reads more | 16:17 | |
| aignatov | seems to be packaging issue, sources are not up-to-date there, hdp directories are 2_0 only, but should be 2_0_6 | 16:17 |
| tosky | aignatov: the error he gets is on 1.3.2 | 16:18 |
| aignatov | probably you should try http://tarballs.openstack.org/sahara/sahara-stable-icehouse.tar.gz | 16:25 |
| *** themistymay has joined #openstack-sahara | 16:25 | |
| tosky | aignatov: thanks - is it the final final? What was the other package then, a previous spin? | 16:30 |
| tmckay | tosky, I think he's right, for what that's worth. I just installed that rpm on a RHEL6. Indeed, there is no resources directory under 1_3_2 for hdp, but the version handler for hdp calls for the path | 16:30 |
| elmiko | tosky, tmckay, it's also possible that those files aren't getting pulled in from the spec file. i'm gonna start to investigate the differences | 16:32 |
| elmiko | i can also adjust the upstream source if needed | 16:33 |
| tmckay | tosky, elmiko, and sure enough, the resources dir is in the tarball. So this seems like a package build problem, I agree, not a Sahara code problem | 16:33 |
| aignatov | tosky: I think that’s final stable super icehouse package :) also stable-icehouse are being created from the separate branch stable/icehouse | 16:33 |
| aignatov | and could be updated once we decide to backport some needed and critical fixes | 16:34 |
| aignatov | 2014.1 is git tag | 16:34 |
| tmckay | elmiko, yeah, I grab the srpm, or koji project, whatever, and spin a package | 16:34 |
| tosky | aignatov: the changelog is sahara-stable-icehouse says also "* Opening stable/icehouse" - wasn't it released after the stable version? | 16:34 |
| aignatov | and will never been updated | 16:34 |
| tmckay | I *would* grab | 16:34 |
| aignatov | tosky: yes it was released after official release announcment, but this’s branch for backports and should be used, as for example, we are using it in Mirantis OpenStack | 16:40 |
| *** zhiyan_ is now known as zhiyan | 16:42 | |
| *** shakamunyi has quit IRC | 16:42 | |
| *** akuznetsov has quit IRC | 16:43 | |
| aignatov | but there are no significant defferences betwen stable and 2014.1 though | 16:44 |
| tosky | just one patch for floating IPs | 16:44 |
| tosky | it seems | 16:44 |
| aignatov | yes | 16:45 |
| SergeyLukjanov | stable/icehouse will be developed by backports | 16:46 |
| elmiko | so, is http://tarballs.openstack.org/sahara/sahara-2014.1.tar.gz the authoritative upstream for icehouse? | 16:46 |
| SergeyLukjanov | and than we'll release 2014.1.1 if needed | 16:46 |
| SergeyLukjanov | and 2014.1 is the final tag release that will not be updated | 16:46 |
| SergeyLukjanov | elmiko, it's a first release in stable/icehouse | 16:47 |
| SergeyLukjanov | elmiko, the next one will be 2014.1.1 | 16:47 |
| elmiko | SergeyLukjanov: ok, i'm just trying to get these package sources straight for the rdo beta release | 16:47 |
| SergeyLukjanov | elmiko, so, you should use http://tarballs.openstack.org/sahara/sahara-2014.1.tar.gz if you want tarball that'll not be changed | 16:47 |
| elmiko | SergeyLukjanov: thanks | 16:47 |
| *** zhiyan is now known as zhiyan_ | 16:53 | |
| SergeyLukjanov | elmiko, np | 17:36 |
| *** IlyaE has quit IRC | 17:39 | |
| *** bradd1 has quit IRC | 17:43 | |
| *** IlyaE has joined #openstack-sahara | 18:02 | |
| elmiko | SergeyLukjanov, aignatov, do all the files in sahara/hdp/versions/version_1_3_2/resources need to exist for a working sahara? | 18:03 |
| elmiko | do i need to package the .json, .template, and .sh? | 18:04 |
| tmckay | crobertsrh, ping | 18:07 |
| crobertsrh | yes? | 18:07 |
| crobertsrh | tmckay: yes? | 18:09 |
| tmckay | I've noticed something, practicing for the demo :) It seems like the Sahara dashboard doesn't update until I hit the refresh on the browser. I had a job go all the way from Pending to Succeeded when I hit refresh, never saw it "Running". Is this expected? Are there automatic updates? What's the refresh rate? Is there a way to turn it up? | 18:09 |
| crobertsrh | ah | 18:09 |
| elmiko | i've seen that too | 18:09 |
| crobertsrh | Let me check | 18:09 |
| tmckay | wonder if I'll have to keep clicking refresh in the demo. Bah. ;-) | 18:09 |
| crobertsrh | Looks like it is an "UpdateRow", which is supposed to use ajax to do updates | 18:11 |
| crobertsrh | Let me see if I can find the interval | 18:11 |
| crobertsrh | Looks like 25 seconds is the default interval (which we are using) | 18:12 |
| tmckay | hmm, maybe a bug then | 18:12 |
| tmckay | I should make sure everything is updated | 18:13 |
| crobertsrh | Do you see your browser sending the ajax requests every 25 seconds? | 18:14 |
| crobertsrh | ctrl-shift-Q will bring up the networking pane in firefox | 18:14 |
| tmckay | looking now with firebug | 18:15 |
| crobertsrh | I'm not seeing any ajax requests. | 18:16 |
| tmckay | k, I just relaunched a job, I expect it to go "killed" because I didn't delete the output dir | 18:16 |
| tmckay | "Pending" | 18:16 |
| tmckay | no requests yet | 18:16 |
| tmckay | Maybe the job execution page isn't sending updates? | 18:17 |
| tmckay | oozie has it running | 18:17 |
| tmckay | still pending | 18:17 |
| tmckay | well, that's definitely longer than 25 seconds, judging from the Pidgin timestamps :) | 18:18 |
| tmckay | ooo, oozie has succeeded. Java action must not care about the output dir | 18:19 |
| crobertsrh | Yeah, probably a bug | 18:19 |
| tmckay | crobertsrh, well, darn. Next question is, regression, or did it never work? And should we try to fix it this week? | 18:20 |
| tmckay | I could do the whole demo from the cli :-D | 18:20 |
| crobertsrh | I think I remember it working when we first put it in there. | 18:21 |
| tmckay | k. I'm going to see if I can get the 3rd leg of bigpetstore to run as a standalone Pig job. I'll leave the udpates to you. | 18:23 |
| crobertsrh | oh...and the default refresh is 2.5 sec :) | 18:23 |
| crobertsrh | stinking off by 22.5 errors | 18:23 |
| tmckay | lol | 18:23 |
| tmckay | those always get me | 18:24 |
| tmckay | base 2.5 was never natural for me | 18:24 |
| tmckay | all the other kids laughed | 18:24 |
| *** tosky has quit IRC | 18:44 | |
| tmckay | crobertsrh, the other embarassing thing I'm seeing often is when I try to click on a cluster from Sahara, I get error output, and if I refresh (most of the time) I get a cluster display | 18:46 |
| crobertsrh | That's not good. I don't think I've ever seen that running locally. | 18:47 |
| crobertsrh | anything in the logs when that happens? | 18:47 |
| crobertsrh | oh wait....probably no logs on that instance :) | 18:47 |
| tmckay | yeah, I'm posting a trace. Again, could be something whacky with our os instance | 18:48 |
| tmckay | I'm still logged in so luckily I got the trace from the background | 18:48 |
| tmckay | http://www.fpaste.org/99641/94021011/ | 18:49 |
| tmckay | crobertsrh, I suppose it could be my home setup. I'm going to try to be in the office again this week, hopefully | 18:51 |
| crobertsrh | ConnectionReset seems like it could be some sort of transient networking thing | 18:52 |
| tmckay | yeah. Well, if you're not seeing it on os uno then maybe we can disregard | 18:53 |
| crobertsrh | Will keep an eye on it. I'll try to do some testing in the next day or so | 18:55 |
| * crobertsrh reboots | 18:55 | |
| *** crobertsrh has left #openstack-sahara | 18:55 | |
| *** crobertsrh has joined #openstack-sahara | 18:57 | |
| tmckay | welcome back | 19:00 |
| tmckay | how was the extranet? | 19:00 |
| crobertsrh | fantastic | 19:02 |
| *** Ch00k has joined #openstack-sahara | 19:03 | |
| crobertsrh | I was going to check out the new bugs on my own instance, but realized that my devstack was beyond repair | 19:03 |
| tmckay | I think I'm on to something. I think the reason I couldn't make bigpetstore work as 3 standalones was that the oozie share lib (which contains the pig jars) isn't set up right | 19:03 |
| tmckay | devstack turns to devpile | 19:03 |
| crobertsrh | heh | 19:04 |
| *** IlyaE has quit IRC | 19:08 | |
| tmckay | yep, if I bundle a copy of the piglib with my Java job, it can run. If I don't, it fails with class not found on org/apache/pig/ExecType which is in the piglib. So the oozie sharelib is not being read, apparently | 19:16 |
| *** mattf is now known as _mattf | 19:17 | |
| *** Ch00k has quit IRC | 19:17 | |
| elmiko | tmckay, crobertsrh, did the rsa key for review.openstack.org change? | 19:23 |
| crobertsrh | Yeah | 19:23 |
| elmiko | k, just wanted to make sure :) | 19:23 |
| crobertsrh | they upgraded their bits a week or 2 back | 19:23 |
| elmiko | i remember that(still getting used to the ui), but i didn't realize the key changed too | 19:24 |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: correcting the MANIFEST.in paths https://review.openstack.org/92441 | 19:24 |
| tmckay | oh yeah, I had to futz with knownhosts. forgot about that | 19:24 |
| elmiko | shows you how long it's been since i've had a review... | 19:26 |
| crobertsrh | tmckay: for what it's worth, I think that the cluster launching might also be failing to do ajax updates too. | 19:39 |
| crobertsrh | I wonder if something the horizon base code changed and we never noticed/adapted. | 19:39 |
| tmckay | could be | 19:39 |
| *** IlyaE has joined #openstack-sahara | 19:46 | |
| *** ErikB has quit IRC | 20:36 | |
| elmiko | tmckay, crobertsrh, do you guys know if it's possible to turn off horizon in devstack? | 20:44 |
| crobertsrh | yeah, should be | 20:44 |
| elmiko | it just seems redundant to run the devstack/horizon and my custom one | 20:45 |
| tmckay | never tried | 20:45 |
| crobertsrh | should be "disable_service horizon" or something like that | 20:45 |
| elmiko | crobertsrh: i'll give it a run | 20:45 |
| elmiko | just not sure what else might need to be turned off | 20:45 |
| crobertsrh | good luck | 20:45 |
| elmiko | lol thanks | 20:46 |
| elmiko | do you guys know much about the infrastructure_engine option, i'm curious about the differences between "direct" and "heat" | 20:50 |
| elmiko | aside from the obvious... | 20:50 |
| tmckay | elmiko, I think aignatov can tell you a lot about that | 20:52 |
| elmiko | ok, maybe i'll poke him tomorrow | 20:53 |
| elmiko | aignatov: you around? | 20:53 |
| aignatov | elmiko, yes, what do you want to know? :) | 20:54 |
| elmiko | i'm mainly curious about the side-effects of running direct over heat | 20:55 |
| elmiko | or i should say, direct instead of heat | 20:55 |
| aignatov | the main difference that in the direct mode sahara uses resourse provisinioning by direct calls to nova,cinder, neutron etc | 20:58 |
| aignatov | in the heat engine all these ops are handed off to heat component | 20:58 |
| elmiko | ok, do i need any extra authentication to use the heat pathway? | 20:59 |
| aignatov | to use heat you should configure it properly (in case if heat is not in devstack) | 21:00 |
| aignatov | nothing special in authentification is needed | 21:00 |
| elmiko | thanks :) | 21:01 |
| aignatov | do you use heat in devstack? | 21:01 |
| elmiko | it looks like devstack uses heat by default | 21:01 |
| elmiko | i'm doing some testing and running devstack with my personal sahara along side it, and i want to try and match the devstack config | 21:01 |
| elmiko | i just noticed that was a difference between the 2 configs | 21:02 |
| aignatov | in the sahara conf nothing special except engine name :) | 21:02 |
| elmiko | cool | 21:02 |
| tmckay | I want to scream | 21:14 |
| elmiko | ok, go for it! | 21:15 |
| tmckay | What's the default yarn log dir? Oh, that's easy -- it's ${yarn.log.dir}/logs | 21:15 |
| tmckay | FOR THE LOVE OF ALL OPENSTACK, PEOPLE, ANSWER THE BLEEDIN QUESTION | 21:16 |
| elmiko | oh man... | 21:16 |
| tmckay | I finally thought to hack /opt/hadoop/bin/yarn and print it out | 21:16 |
| tmckay | /mnt/yarn/logs | 21:16 |
| tmckay | been looking for this for an hour or two | 21:16 |
| tmckay | circles on circles on circles. I think it's ultimately set from HADOOP_YARN_HOME | 21:17 |
| elmiko | that's sounds pretty frustrating, to say the least | 21:17 |
| tmckay | it can never be easy | 21:17 |
| elmiko | it can when you live in a van down by the river ;) | 21:18 |
| tmckay | lol | 21:18 |
| tmckay | I gotta dredge up some old Farley | 21:19 |
| elmiko | take the edge off | 21:19 |
| tmckay | elmiko, excuse me, I was wrong. It's ${yarn.log.dir}/userlogs | 21:26 |
| tmckay | have to be precise | 21:26 |
| elmiko | well, if you want to find the logs you do ;) | 21:28 |
| tmckay | lol | 21:29 |
| aignatov | tmckay: you could always ask sreshetn1ak about hadoop 2 configs/infra in Sahara | 21:29 |
| *** crobertsrh has quit IRC | 21:29 | |
| tmckay | aignatov, good suggestion. fyi, here is what I am seeing | 21:29 |
| tmckay | I can run bigpetstore jobs that invoke Pig directly, but only if I bundle the piglib with the Sahara job | 21:30 |
| tmckay | it should be available from oozie share lib, however. Which I can see with hadoop fs -ls | 21:30 |
| tmckay | but, it's not found | 21:30 |
| tmckay | aignatov, trying to find enough info in the logs to figure out why the oozie share lib (including the pig jar) is not copied to the job dir | 21:31 |
| tmckay | I can make it work.... but I shouldn't have to push the piglib as a job binary myself | 21:32 |
| aignatov | I see, so the problemm is that shared piglib by oozie is not in the classpath when you are triyng to run pig job, right? | 21:33 |
| tmckay | well, I should double check "Pig". This is actually a Java action using the Pig api, on hadoop 2.3.0 | 21:34 |
| tmckay | but, the sharelib I would still expect to be there. Maybe Java actions are special somehow | 21:34 |
| aignatov | btw, I can suggest a tip how to run oozie jobs in Hadoop cluster without Sahara | 21:36 |
| aignatov | so it speeds up debugging of job executions :) | 21:36 |
| tmckay | yes, I've run lots of oozie jobs from the command line :) | 21:36 |
| tmckay | aignatov, current problem summary: 1) why do I have to bundle the piglib with my Java job? 2) why was I not able to get the 3rd part of bigpetstore to run successfully as a standalone Pig job | 21:37 |
| tmckay | the 3rd part can be run as a script argument to the 2nd part | 21:38 |
| tmckay | But, I would like to see it run as 3 jobs, because it should be able to | 21:38 |
| aignatov | 1) the answer could be that Oozie don’t not include all libraries for each job type | 21:38 |
| aignatov | *doesn’t | 21:38 |
| tmckay | aignatov, could be | 21:39 |
| aignatov | I mean if you run Pig action than Oozie append all needed libraries only | 21:39 |
| tmckay | that's why I want to find the detailed job log that shows the files being copied | 21:39 |
| aignatov | does it work on the hadoop 1.0? | 21:40 |
| tmckay | aignatov, haven't tried to compile it for 1.0. jayunit100 wrote it against 2.3.0 | 21:40 |
| tmckay | that is another possibility | 21:40 |
| aignatov | ok, so I probably missed domething but you shared only screencast with me, do you have full script/scenario how to run bigpetstore? | 21:42 |
| tmckay | well, I have instructions in an email | 21:43 |
| tmckay | that I sent to matt | 21:43 |
| aignatov | and all binaries/scripts needed, I’ll try to help you :) | 21:43 |
| tmckay | okay. Time to eat :) I send you something this evening or tomorrow | 21:43 |
| elmiko | later tmckay | 21:44 |
| aignatov | ok, thx :) have a nice lunch | 21:45 |
| aignatov | or dinner :) | 21:45 |
| *** elmiko is now known as elmiko_ | 22:11 | |
| *** IlyaE has quit IRC | 22:12 | |
| *** IlyaE has joined #openstack-sahara | 22:47 | |
| *** IlyaE has quit IRC | 23:16 | |
| *** IlyaE has joined #openstack-sahara | 23:58 | |
| tmckay | aignatov, putting together my instructions for you. Looks like bigpetstore moved to apache bigtop, so I have to re-pull and make sure I can build it ... | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!