| *** shakamunyi has quit IRC | 00:05 | |
| *** mattf is now known as _mattf | 00:05 | |
| *** IlyaE has joined #openstack-sahara | 00:15 | |
| *** ErikB has quit IRC | 00:16 | |
| *** witlessb has joined #openstack-sahara | 01:56 | |
| *** witlessb has quit IRC | 02:00 | |
| *** openstackgerrit has quit IRC | 02:04 | |
| *** openstackgerrit has joined #openstack-sahara | 02:05 | |
| *** IlyaE has quit IRC | 02:10 | |
| *** IlyaE has joined #openstack-sahara | 03:14 | |
| *** Ch00k has joined #openstack-sahara | 03:18 | |
| *** Ch00k has quit IRC | 04:01 | |
| *** _mattf has quit IRC | 04:02 | |
| *** Ch00k has joined #openstack-sahara | 04:31 | |
| *** Ch00k has quit IRC | 04:44 | |
| *** Ch00k has joined #openstack-sahara | 04:44 | |
| *** IlyaE has quit IRC | 04:45 | |
| *** Ch00k has quit IRC | 04:45 | |
| *** IlyaE has joined #openstack-sahara | 05:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/91517 | 06:06 |
|---|---|---|
| *** IlyaE has quit IRC | 06:09 | |
| *** IlyaE has joined #openstack-sahara | 06:19 | |
| *** openstackgerrit has quit IRC | 06:57 | |
| *** IlyaE has quit IRC | 07:27 | |
| *** IlyaE has joined #openstack-sahara | 07:48 | |
| *** witlessb has joined #openstack-sahara | 07:50 | |
| *** akuznetsov has joined #openstack-sahara | 07:58 | |
| *** IlyaE has quit IRC | 08:00 | |
| *** sreshetn1ak has joined #openstack-sahara | 08:27 | |
| *** akuznetsov has quit IRC | 08:34 | |
| *** sreshetnyak has quit IRC | 08:34 | |
| *** akuznetsov has joined #openstack-sahara | 08:53 | |
| *** tosky has joined #openstack-sahara | 09:35 | |
| *** akuznetsov has quit IRC | 10:29 | |
| *** akuznetsov has joined #openstack-sahara | 10:30 | |
| *** witlessb has quit IRC | 11:58 | |
| *** witlessb has joined #openstack-sahara | 12:02 | |
| *** shakamunyi has joined #openstack-sahara | 12:36 | |
| *** _crobertsrh is now known as crobertsrh | 12:40 | |
| *** shakamunyi has quit IRC | 12:41 | |
| *** elmiko has joined #openstack-sahara | 12:52 | |
| *** tmckay has joined #openstack-sahara | 13:03 | |
| *** ErikB1 has joined #openstack-sahara | 13:17 | |
| *** shakamunyi has joined #openstack-sahara | 13:37 | |
| *** shakamunyi has quit IRC | 13:41 | |
| *** zhiyan_ is now known as zhiyan | 13:45 | |
| *** shakamunyi has joined #openstack-sahara | 13:52 | |
| *** mattf has joined #openstack-sahara | 13:58 | |
| *** openstackgerrit has joined #openstack-sahara | 14:09 | |
| *** zhiyan is now known as zhiyan_ | 14:13 | |
| *** shakamunyi has quit IRC | 14:31 | |
| *** shakamunyi has joined #openstack-sahara | 14:31 | |
| *** shakamunyi has quit IRC | 14:32 | |
| *** shakamunyi has joined #openstack-sahara | 14:32 | |
| *** bradd1 has joined #openstack-sahara | 14:33 | |
| *** ErikB1 has quit IRC | 14:49 | |
| *** shakamunyi has quit IRC | 14:53 | |
| *** ErikB1 has joined #openstack-sahara | 14:54 | |
| *** shakamunyi has joined #openstack-sahara | 14:54 | |
| *** shakamunyi has quit IRC | 14:56 | |
| *** shakamunyi has joined #openstack-sahara | 14:56 | |
| *** shakayumi has joined #openstack-sahara | 14:59 | |
| *** shakamunyi has quit IRC | 15:01 | |
| *** shakayumi has quit IRC | 15:03 | |
| *** shakamunyi has joined #openstack-sahara | 15:03 | |
| *** shakamunyi has quit IRC | 15:03 | |
| *** shakamunyi has joined #openstack-sahara | 15:18 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Allow HDFS data source paths without the hdfs:// scheme https://review.openstack.org/91664 | 15:30 |
| openstackgerrit | A change was merged to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/91517 | 15:35 |
| *** shakayumi has joined #openstack-sahara | 15:37 | |
| *** shakamunyi has quit IRC | 15:38 | |
| *** IlyaE has joined #openstack-sahara | 16:04 | |
| tellesnobrega | tmckay: hi, im back | 16:06 |
| tmckay | hi | 16:06 |
| tellesnobrega | i tried to find the userlogs but it doesnt exists | 16:07 |
| tellesnobrega | the only logs i found stated sucess on the job | 16:07 |
| tellesnobrega | maybe the worker has more info | 16:07 |
| tellesnobrega | i was looking in the master node | 16:07 |
| tmckay | hmm, I haven't seen a case where /mnt/log/hadoop/hadoop/userlogs doesn't exist on the worker in 1.2.1 | 16:08 |
| *** IlyaE has quit IRC | 16:10 | |
| tmckay | tellesnobrega, oh, try the worker | 16:11 |
| tmckay | mine looks like this (from this morning) | 16:11 |
| tmckay | /mnt/log/hadoop/hadoop/userlogs | 16:11 |
| tmckay | ls -l | 16:11 |
| tmckay | total 24 | 16:11 |
| tmckay | drwx--x--- 2 hadoop hadoop 4096 May 2 15:13 job_201405021509_0001 | 16:11 |
| tmckay | drwx--x--- 2 hadoop hadoop 4096 May 2 15:15 job_201405021509_0003 | 16:11 |
| tmckay | drwx--x--- 2 hadoop hadoop 4096 May 2 15:15 job_201405021509_0004 | 16:11 |
| tellesnobrega | found it | 16:12 |
| tmckay | okay great | 16:12 |
| tmckay | tellesnobrega, so there should be a job (or jobs) there, and other each job some attempts. On a failed job, one of the attempts should have a non-zero stderr and maybe a stdout | 16:13 |
| tmckay | other -> under | 16:13 |
| tellesnobrega | this is the log.index stdout:0 -1 | 16:15 |
| tellesnobrega | stderr:0 -1 | 16:15 |
| tellesnobrega | syslog:0 -1 | 16:15 |
| tellesnobrega | stdout is empty and also stderr | 16:15 |
| tellesnobrega | the only file that has info is the syslog | 16:16 |
| tmckay | hmm, okay, anything interesting in the syslog? | 16:17 |
| tellesnobrega | not much http://paste.openstack.org/show/78485/ | 16:18 |
| tellesnobrega | wait | 16:19 |
| tellesnobrega | found something here | 16:19 |
| tellesnobrega | on the other job | 16:19 |
| tellesnobrega | Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.MapReduceMain], main() threw exception, Invalid swift hostname 'null': hostname must in form container.service org.apache.hadoop.fs.swift.exceptions.SwiftConfigurationException: Invalid swift hostname 'null': hostname must in form container.service | 16:20 |
| tmckay | okay, that's promising. | 16:20 |
| tmckay | we need the command from yesterday that lists the oozie jobs | 16:21 |
| * tmckay is reconstructing on current node | 16:22 | |
| tellesnobrega | ok | 16:23 |
| tellesnobrega | i found more info in the stdout | 16:23 |
| tmckay | /opt/oozie/bin/oozie jobs -oozie http://1.2.3.4:11000/oozie so something like this | 16:24 |
| tmckay | Then this with the ID of the failed job, which will show the workflow.xml that was used: | 16:25 |
| tmckay | /opt/oozie/bin/oozie job -oozie http://1.2.3.4:11000/oozie -definition 0000003-140502151059962-oozie-hado-W | 16:25 |
| tmckay | I'm guessing maybe your swift path does not have ".sahara" in it | 16:26 |
| tmckay | swift://me.sahara/somepath | 16:26 |
| tmckay | where "me" is the container, and somepath is the object | 16:27 |
| tellesnobrega | it doesn't | 16:27 |
| tmckay | mmm, sahara should check this in the validation, maybe. | 16:27 |
| tellesnobrega | i created just demo | 16:27 |
| tellesnobrega | for example | 16:27 |
| tellesnobrega | i will try running with demo.sahara swift | 16:27 |
| tellesnobrega | container | 16:27 |
| tmckay | So, hadoop has been fixed up to pull the ".sahara" string out and match it against the fs.swift.service.sahara.xxxx configs to find user/password | 16:28 |
| tmckay | it could be anything, doesn't have to be ".sahara", but they all have to match, and Sahara is expecting ".sahara" when it makes the user/passw configs | 16:28 |
| tmckay | tellesnobrega, I'll look to see if there is validation for this, if not, it should raise an error when the data source is created | 16:29 |
| tmckay | that would have saved you the mystery | 16:29 |
| tellesnobrega | ok | 16:29 |
| tmckay | yeah, it's not checking -- I just made one. That's an oversight, .sahara should be required on swift data sources | 16:30 |
| tmckay | (not on binaries, though) | 16:30 |
| tmckay | that's because the binaries are accessed on the Sahara side, not from hadoop at runtime | 16:31 |
| *** akuznetsov has quit IRC | 16:31 | |
| tellesnobrega | wiat im confused, it must or not have the .sahara name? | 16:32 |
| *** akuznetsov has joined #openstack-sahara | 16:32 | |
| tmckay | tellesnobrega, so we have swift paths used in 2 different ways | 16:33 |
| tmckay | 1) data sources | 16:33 |
| tmckay | 2) job binaries | 16:33 |
| tellesnobrega | ok | 16:33 |
| tmckay | for data sources, swift paths must be of the form "swift://container.sahara/object" | 16:33 |
| tmckay | for job binaries it's just "swift://container/object" | 16:34 |
| tellesnobrega | i got it | 16:34 |
| tmckay | tellesnobrega, note that is just "sugar". In your actual swift, your data source will still be "swift://container/object" | 16:34 |
| tmckay | I don't like it :) | 16:34 |
| *** IlyaE has joined #openstack-sahara | 16:35 | |
| tellesnobrega | so if i have a container named demo, when i create data source the path i should pass is demo.sahara/input, is that it? | 16:35 |
| tmckay | yes | 16:35 |
| tellesnobrega | tmckay: thats weird | 16:36 |
| tmckay | tellesnobrega, yeah, we should revisit that, I think. It was a way to let hadoop map paths to configs that hold credentials. | 16:37 |
| tmckay | it involves changes in the hadoop jars, too | 16:37 |
| tmckay | I | 16:37 |
| tmckay | I'm not familiar with all the issues, I didn't work on it but maybe we can make something better | 16:38 |
| tmckay | brb | 16:38 |
| tellesnobrega | ok | 16:38 |
| tellesnobrega | tmckay: thanks for the help, it worked now :D | 16:53 |
| tmckay | tellesnobrega, you're welcome, thanks for finding the bug! https://bugs.launchpad.net/sahara/+bug/1315448 | 16:54 |
| tmckay | going to patch this right now | 16:54 |
| tmckay | it's a nasty one | 16:54 |
| tellesnobrega | tmckay: cool | 16:54 |
| tmckay | doesn't make new users feel loved ;-) | 16:54 |
| tellesnobrega | tmckay: thanks again. Maybe we get to talk in atlanta, and I can try to explain my ideas for sahara | 17:00 |
| tmckay | tellesnobrega, absolutely, we've got some design sessions Thursday afternoon and Friday morning | 17:01 |
| tellesnobrega | tmckay: i've already added to my schedule | 17:01 |
| tmckay | I think we can improve EDP | 17:01 |
| tellesnobrega | tmckay: certainly. | 17:05 |
| tellesnobrega | tmckay: i think that having edp pluggable is needed, at least for what i'm thinking. i would love to have sahara processing streams | 17:06 |
| tmckay | tellesnobrega, yes. I've been thinking too that we need to review and make sure that "Oozie-ness" has not seeped through too much. We need to be more general than that. Maybe we have a deeper hierarchy of job types, depending on the facility to used to run them. "Oozie jobs (Java, MapReduce, Pig, etc)" okay these can be oozie-like in configs, etc. But maybe Spark, or something else, should have its own abstractions. | 17:08 |
| tmckay | Data Source, Job Binary, Job, okay -- those are general | 17:08 |
| tellesnobrega | +1 | 17:09 |
| tmckay | we developed initial EDP pretty rapidly, now it's time for review and rework, I think | 17:09 |
| tellesnobrega | hmm, i will try to understand the code better by the summit, so i can try and help with some improvements | 17:12 |
| tmckay | great, thanks | 17:17 |
| tmckay | as always, it takes a while using something to see how it can be made better | 17:26 |
| *** IlyaE has quit IRC | 17:34 | |
| *** IlyaE has joined #openstack-sahara | 17:35 | |
| *** shakayumi has quit IRC | 17:52 | |
| *** IlyaE has quit IRC | 18:09 | |
| *** IlyaE has joined #openstack-sahara | 18:09 | |
| *** IlyaE has quit IRC | 18:10 | |
| *** tosky has quit IRC | 18:10 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Improve validation for swift data source URLs https://review.openstack.org/91848 | 18:18 |
| tellesnobrega | tmckay: i think you missed a t in swift in the tests | 18:24 |
| *** akuznets_ has joined #openstack-sahara | 18:27 | |
| *** akuznetsov has quit IRC | 18:28 | |
| tellesnobrega | i was going to review it, but i think you already fixed it right? | 18:32 |
| tmckay | tellesnobrega, one is on purpose, the others are copy/paste | 18:34 |
| tellesnobrega | yes | 18:34 |
| tmckay | yes, I'm about to fix it | 18:35 |
| tellesnobrega | ok, i won't -1 there | 18:35 |
| tellesnobrega | i will wait for the fixed patch | 18:35 |
| tmckay | for some reason, it's easier to see on the UI than with git diff | 18:35 |
| tellesnobrega | i get that | 18:41 |
| tellesnobrega | tmckay: to run the wordcount job as java action. It doesn't show the input and output path when starting the job. how can i pass the path to swift | 18:45 |
| tellesnobrega | ?/ | 18:45 |
| tellesnobrega | just swift://demo.sahara/input as argument? | 18:46 |
| tellesnobrega | and the same for output? | 18:46 |
| tmckay | tellesnobrega, Java actions are a little different, there is no input/output required, so yes, you pass arguments | 18:46 |
| tellesnobrega | the main classe is WordCount or WordCount.class? | 18:47 |
| tellesnobrega | and java opt is there any? | 18:47 |
| tmckay | tellesnobrega, also, you have to add fs.swift.service.sahara.username and fs.swift.service.sahara.password as config params | 18:47 |
| tellesnobrega | ok | 18:48 |
| tellesnobrega | tmckay: i will look for some bugs in sahara, this weekend so i can start messing around a little with the code | 18:49 |
| tmckay | main class should be "org.openstack.sahara.examples.WordCount" assuming you've got the latest jar | 18:49 |
| tellesnobrega | if you have any suggestions would be good | 18:49 |
| tmckay | suggestions for bugs? | 18:50 |
| tellesnobrega | yes | 18:51 |
| tmckay | (no java opts that are required) | 18:51 |
| tmckay | well, you can look at what is open in launchpad | 18:51 |
| tellesnobrega | ok, that was going for | 18:51 |
| tmckay | https://bugs.launchpad.net/sahara | 18:51 |
| tellesnobrega | i will try to have something fixed | 18:51 |
| tmckay | tellesnobrega, there's one that I can think of | 18:53 |
| tmckay | some of the other stuff there looks complicated | 18:53 |
| tmckay | when you launch a job, sometimes the connection to Oozie fails before the job is delivered | 18:53 |
| tmckay | this happens in job_manager.py | 18:53 |
| tellesnobrega | hum | 18:54 |
| tmckay | run_job needs an exception handler, I think, to catch the case where the job never makes it to Oozie | 18:54 |
| tmckay | it stays Pending forever | 18:54 |
| tellesnobrega | i see | 18:54 |
| tellesnobrega | sounds pretty easy | 18:54 |
| tellesnobrega | do you have the link for the bug? | 18:54 |
| tmckay | It's a subcase of https://bugs.launchpad.net/sahara/+bug/1265068, the exception handler would be a partial fix. | 18:55 |
| tmckay | But not a complete fix. I have seen it fail in that particular spot more than once | 18:55 |
| tellesnobrega | ok | 18:57 |
| tellesnobrega | i will take a look this weekend if i have the chance | 18:57 |
| tellesnobrega | i gotta go now. Have a nice weekend. thanks again for the help | 18:57 |
| tmckay | tellesnobrega, oh, see my comment #10 | 18:57 |
| tmckay | k, bye! | 18:57 |
| tellesnobrega | bye | 18:59 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Improve validation for swift data source URLs https://review.openstack.org/91848 | 19:00 |
| crobertsrh | Anyone have experience with using mysql for sahara? | 19:07 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara-extra: Change class name in example workflow https://review.openstack.org/91871 | 19:17 |
| tmckay | crobertsrh, still haven't used it | 19:17 |
| crobertsrh | Ok. I've got a puppet module *mostly* working with my foreman-based openstack installation (It's pretty neat). Hopefully, my last sticking point is getting the database access straightened out. | 19:18 |
| *** witlessb has quit IRC | 20:25 | |
| *** witlessb has joined #openstack-sahara | 20:27 | |
| *** IlyaE has joined #openstack-sahara | 20:29 | |
| crobertsrh | Something unexpected with sahara-db-manage..... | 20:43 |
| crobertsrh | http://fpaste.org/98701/90633771/ | 20:43 |
| *** witlessb has quit IRC | 20:43 | |
| crobertsrh | Any ideas why sahara-db-manage current fails to produce anything useful? | 20:43 |
| crobertsrh | 4:44 on Friday is probably a poor time for such a question :) | 20:44 |
| dmitryme | crobertsrh: hello! Did you call | 20:45 |
| dmitryme | sahara-db-manage upgrade head | 20:45 |
| crobertsrh | bah, that's it :) | 20:46 |
| crobertsrh | my memory betrayed me. Thanks | 20:46 |
| dmitryme | :-) | 20:46 |
| crobertsrh | any chance you know about the puppet module? | 20:46 |
| dmitryme | If it is sahara-puppet then nope | 20:47 |
| crobertsrh | Yeah. I'm working on updating it for RHEL (and making it sahara rather than savanna) | 20:47 |
| crobertsrh | "nope" is probably a good answer :) | 20:48 |
| aignatov | crobertsrh: hi, not sure what you are looking for in puppet module but here is a puppet stuff for Sahara in Fuel library | 20:49 |
| aignatov | https://github.com/stackforge/fuel-library/search?q=sahara&ref=cmdform | 20:49 |
| crobertsrh | aignatov: Thanks. That might save me some work (or might just let me throw away what work I've done!) | 20:50 |
| crobertsrh | Is it worth updating puppet-sahara? | 20:52 |
| crobertsrh | or should the fuel stuff be a separate entity? | 20:52 |
| aignatov | fuel team wants someday to use only pupet-sahara | 20:54 |
| aignatov | but now it’s separate | 20:54 |
| crobertsrh | Hmm, ok. | 20:55 |
| *** witlessb has joined #openstack-sahara | 20:58 | |
| *** ErikB1 has quit IRC | 21:00 | |
| *** tmckay has quit IRC | 21:06 | |
| *** crobertsrh is now known as _crobertsrh | 21:09 | |
| *** ErikB1 has joined #openstack-sahara | 21:11 | |
| *** IlyaE has quit IRC | 21:15 | |
| *** ErikB1 has quit IRC | 21:30 | |
| *** witlessb has quit IRC | 21:37 | |
| *** ErikB1 has joined #openstack-sahara | 21:41 | |
| *** IlyaE has joined #openstack-sahara | 21:47 | |
| *** IlyaE has quit IRC | 22:00 | |
| *** elmiko is now known as elmiko_afk | 22:24 | |
| *** IlyaE has joined #openstack-sahara | 22:25 | |
| *** ErikB1 has quit IRC | 22:50 | |
| *** IlyaE has quit IRC | 23:13 | |
| *** IlyaE has joined #openstack-sahara | 23:35 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!