| *** hoonetorg has quit IRC | 00:56 | |
| *** hoonetorg has joined #openstack-sahara | 01:10 | |
| *** hoonetorg has quit IRC | 01:49 | |
| *** hoonetorg has joined #openstack-sahara | 02:01 | |
| *** dave-mccowan has quit IRC | 03:07 | |
| *** pcaruana has joined #openstack-sahara | 04:36 | |
| *** pcaruana has quit IRC | 04:43 | |
| *** hoonetorg has quit IRC | 06:36 | |
| *** rcernin has quit IRC | 07:01 | |
| *** pcaruana has joined #openstack-sahara | 07:01 | |
| *** tosky has joined #openstack-sahara | 07:47 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: DNM Test print python version https://review.openstack.org/607145 | 10:14 |
|---|---|---|
| *** Bhujay has joined #openstack-sahara | 12:36 | |
| guimaluf | tosky, tellesnobrega: how could I run spark plugin using vanilla image since hadoop location expected by spark is different then in vanilla? | 12:48 |
| tellesnobrega | to run spark plugin you need a spark image | 12:49 |
| guimaluf | my goal is to run spark-bench[https://github.com/CODAIT/spark-bench/] project with sahara, but spark images doesn't fill the requirements for it... | 12:49 |
| guimaluf | tellesnobrega, so the best approuch is to build a spark image with my requirements? java8, scala, etc? | 12:50 |
| tellesnobrega | guimaluf, that would be best for sure | 12:50 |
| tellesnobrega | it shouldn't be too hard to add those requirements to the image | 12:50 |
| *** pcaruana has quit IRC | 12:57 | |
| guimaluf | tellesnobrega, thanks :D | 13:05 |
| *** pcaruana has joined #openstack-sahara | 13:27 | |
| Gaasmann | you have spark on vanilla image | 13:48 |
| tellesnobrega | guimaluf, that is true. You can run spark on vanilla images, but this is different than starting spark plugin using vanilla image | 13:57 |
| guimaluf | tellesnobrega, I know there is spark on vanilla, and I tried to run spark on it. acctualy I tried spark-bench, but I'm getting this error: http://paste.openstack.org/show/731047/ | 14:13 |
| tellesnobrega | probably scala is not installed on the image | 14:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/607254 | 14:54 |
| guimaluf | tellesnobrega, I've included scala into the image, but even thou I can't run it :/ | 15:11 |
| tellesnobrega | same error? | 15:11 |
| guimaluf | I'll try to build spark image with the requirements I need | 15:11 |
| guimaluf | tellesnobrega, yes | 15:11 |
| guimaluf | tellesnobrega, with scala 2.12 and 2.11 and without scala, I'm getting the same error | 15:11 |
| guimaluf | which points me to something related to java not finding scala libs(jars?) | 15:11 |
| tellesnobrega | https://stackoverflow.com/questions/14417814/why-do-i-get-java-lang-noclassdeffounderror-when-i-trying-to-run-this-code | 15:12 |
| tellesnobrega | you may need to add scala lib to the runtime classpath | 15:12 |
| guimaluf | tellesnobrega, I'm using prebuilt spark bench, not building it... | 15:17 |
| tellesnobrega | but I think here you can edit java_classpath | 15:18 |
| *** dave-mccowan has joined #openstack-sahara | 15:29 | |
| guimaluf | tellesnobrega, I'd tried to adjust classpath but with no success... but now I could make it work | 16:09 |
| tellesnobrega | how? | 16:09 |
| *** Bhujay has quit IRC | 16:12 | |
| *** openstackgerrit has quit IRC | 17:51 | |
| guimaluf | tellesnobrega, java -cp /opt/spark-bench/lib/*:/opt/spark/jars/* com.ibm.sparktc.sparkbench.sparklaunch.SparkLaunch /opt/spark-bench/examples/minimal-example.conf | 17:56 |
| tellesnobrega | I see | 17:56 |
| *** pcaruana has quit IRC | 18:02 | |
| *** dave-mccowan has quit IRC | 19:41 | |
| *** openstackgerrit has joined #openstack-sahara | 19:49 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: sahara-image-pack: use curl for tarballs.openstack.org https://review.openstack.org/602507 | 19:50 |
| *** dave-mccowan has joined #openstack-sahara | 20:07 | |
| *** dave-mccowan has quit IRC | 20:30 | |
| tellesnobrega | tosky, are you around? | 20:38 |
| tosky | tellesnobrega: hi | 20:39 |
| tellesnobrega | we defined the name for the plugins repo to be sahara-<plugin>-plugin right? | 20:39 |
| tosky | uhm, I don't remember; wouldn't it be more consistent to use sahara-plugin-<plugin> ? | 20:40 |
| tellesnobrega | could be | 20:41 |
| tellesnobrega | makes more sense | 20:41 |
| tellesnobrega | about the folder structure should I keep sahara_plugins/plugins/<plugin> for each of them? | 20:41 |
| tellesnobrega | that eases my work with changing imports | 20:42 |
| tellesnobrega | they would stay the same actually | 20:42 |
| tosky | the internal import inside each plugin, I guess? | 20:42 |
| tellesnobrega | yes | 20:42 |
| tosky | up to you - remember that it will be like that forever also for new plugins :) | 20:43 |
| tellesnobrega | since sahara doesn't import from the plugins, I don't this will cause any conflicts | 20:43 |
| tosky | yep | 20:43 |
| tellesnobrega | I can remove the plugins subfolder to makes things simpler | 20:44 |
| tosky | but there is sahara_plugins/plugin and sahara_plugin/tests/unit, right ? | 20:44 |
| tellesnobrega | yes | 20:44 |
| tellesnobrega | maybe best to keep the plugins folder as well | 20:45 |
| tellesnobrega | similar to what we have just change the repo name | 20:45 |
| Gaasmann | I'm trying to use EDP with vanilla plugin (2.7.1 on spike). And all the jobs fail with this error | 20:50 |
| Gaasmann | java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.Credentials | 20:50 |
| Gaasmann | seems to come from oozie | 20:50 |
| tellesnobrega | weird, the logs you are looking are from hadoop? | 20:52 |
| Gaasmann | they are on sahara-engine.log | 20:53 |
| Gaasmann | actually, the oozie web console is broken too | 20:53 |
| Gaasmann | http://paste.openstack.org/show/731300/ | 20:54 |
| Gaasmann | the error on sahara-engine.log: http://paste.openstack.org/show/731301/ | 20:56 |
| tellesnobrega | Gaasmann, I really don't know how to fix this. what kind of job are you trying to run? | 21:08 |
| Gaasmann | java. In fact I'm trying to run wordcount from hadoop-mapreduce-examples-2.7.5.jar | 21:28 |
| tellesnobrega | maybe the 2.7.5 added some requirements that 2.7.1 do not support | 21:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/607254 | 22:35 |
| *** tosky has quit IRC | 22:41 | |
| *** rcernin has joined #openstack-sahara | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!