| *** alyonapy has joined #openstack-sahara | 00:27 | |
| *** shuyingya has joined #openstack-sahara | 00:34 | |
| *** shuyingya has quit IRC | 00:49 | |
| *** alyonapy has quit IRC | 01:06 | |
| *** alyonapy has joined #openstack-sahara | 01:06 | |
| *** alyonapy has quit IRC | 01:07 | |
| *** shuyingya has joined #openstack-sahara | 01:36 | |
| *** shuyingya has quit IRC | 01:51 | |
| *** abalutoiu has joined #openstack-sahara | 02:11 | |
| *** skp has quit IRC | 02:21 | |
| *** abalutoiu has quit IRC | 02:26 | |
| *** sahithirv has quit IRC | 02:47 | |
| *** tuanluong has joined #openstack-sahara | 02:48 | |
| *** dave-mccowan has quit IRC | 02:58 | |
| *** sudipto has joined #openstack-sahara | 03:01 | |
| *** sudipto_ has joined #openstack-sahara | 03:01 | |
| *** chlong has joined #openstack-sahara | 04:03 | |
| *** skp has joined #openstack-sahara | 04:03 | |
| *** sahithirv has joined #openstack-sahara | 04:20 | |
| *** sudipto_ has quit IRC | 04:32 | |
| *** sudipto has quit IRC | 04:32 | |
| *** sudipto has joined #openstack-sahara | 05:20 | |
| *** sudipto_ has joined #openstack-sahara | 05:20 | |
| *** gokhan has joined #openstack-sahara | 05:44 | |
| *** nkrinner_afk is now known as nkrinner | 06:18 | |
| sahithirv | hello!! When I try to register an image to sahara image registry with apt username, I get "Object ID not found" error. How to resolve it? | 06:26 |
|---|---|---|
| sahithirv | the image visibility is public. | 06:26 |
| sahithirv | The dataprocessing tab is not visible in dashboard. | 06:27 |
| frippe75 | Finally (almost) I was able to deploy a 4 node hadoop cluster using the ambari 2.4 plugin. Used the node templates in the Sahara repo. Now the final stage of the plugin (Start cluster) fails. Kafka , Ambari metrics and HDFS complain about not reaching services on the master node (port 50090,6667,61310). Security group looks good. | 06:32 |
| frippe75 | In ambari 3 services are in the stopped state, those listed above. Its the HDFS SNameNode that is failing to start. Tried restarting. | 06:34 |
| sahithirv | frippe75:I use devstack with sahara, ceilometer and heat enabled. I tried to register sahara-newton-vanilla-ubuntu image. | 06:37 |
| frippe75 | Sorry, I'm on a packstack all-in-one with sahara deployed manually. | 06:38 |
| frippe75 | newton release. | 06:38 |
| sahithirv | I get error while registering image itself. I haven't created node yet. | 06:38 |
| frippe75 | looked at the sahara quickstart for vanilla? That guide worked perfectly for me. Having issues with the slightly more complex ambari setup. | 06:41 |
| frippe75 | sahithirv, https://docs.openstack.org/developer/sahara/devref/quickstart.html | 06:42 |
| sahithirv | I am following that. I manually installed sahara client. | 06:43 |
| sahithirv | That could be the reason why I don't find it on the dashboard? | 06:43 |
| sahithirv | But why do I get object id not found, I can see the image in the image list. | 06:44 |
| frippe75 | are u only using the dashborad?? | 06:44 |
| frippe75 | ..board.. | 06:44 |
| sahithirv | no. I tried to run the dataprocessing image register command from command line and I get this error. | 06:45 |
| frippe75 | as admin? | 06:45 |
| sahithirv | yes | 06:45 |
| sahithirv | frippe75: How do i ensure it is referring to admin project? | 06:46 |
| sahithirv | Do I ssh first? | 06:47 |
| frippe75 | did you source the auto-generated .openrc file for admin (not sure what the name is in devstack) | 06:48 |
| sahithirv | no. I haven't | 06:48 |
| sahithirv | Any reference? | 06:49 |
| frippe75 | source openrc admin admin | 06:49 |
| sahithirv | Is openrc similar to local.conf | 06:52 |
| frippe75 | think I solved my issue. Tried to restart the SNameNode. stdout was quite clear... "failed; error='Cannot allocate memory' (errno=12)" | 06:53 |
| frippe75 | So bumping up the size and re-deploy again... | 06:53 |
| frippe75 | sahithirv, when you source the openrc file a few env vars are set OS_ which sets endpoint, user,pass and the project (you should point to the admin one) | 06:54 |
| frippe75 | sahithirv, "env | grep ^OS" should give you the currently setup which goes for all openstack clients. | 06:55 |
| sahithirv | ok | 07:01 |
| *** tesseract has joined #openstack-sahara | 07:25 | |
| *** openstackgerrit has quit IRC | 07:32 | |
| *** pcaruana has joined #openstack-sahara | 07:37 | |
| *** sahithirv has quit IRC | 07:51 | |
| *** frippe75 has quit IRC | 07:52 | |
| *** anshul has quit IRC | 07:57 | |
| *** frippe75 has joined #openstack-sahara | 08:35 | |
| *** tosky has joined #openstack-sahara | 08:53 | |
| *** kavithahr has joined #openstack-sahara | 09:06 | |
| kavithahr | hi.. | 09:06 |
| tosky | hi | 09:08 |
| tosky | frippe75: happy to see (from the channel logs) that you solved the problem (and yes, resources are the relevant thing here) | 09:09 |
| frippe75 | tosky, created my own flavors and now a new cluster is in the oven. Dedicated to much of my 500GB SSD for cinderstorage so I'm struggling with the diskfilter filtering out my host. Increased disk_allocation_ratio in nova. Lets hope things work out. But I'm at least getting a good working experience. I'm changing my mind around improving doc. There should he enough barrier to get started. Good thing actually :-) | 09:12 |
| tosky | :D | 09:12 |
| tosky | it's a complicated set of tools; we try to simplify the common use cases, but there are some requirements that should be known nevertheless | 09:14 |
| kavithahr | @tosky hi i am new to sahara... can you please explain me how the sahara clusters are different from creating many nodes in heat | 09:15 |
| tosky | kavithahr: which kind of nodes? Sahara knows that some contains some specific hadoop- or other big-data- related services and enables and checks them | 09:16 |
| *** logan- has quit IRC | 09:16 | |
| kavithahr | @tosky using heat component of openstack we can launch many instances with network connection.. if we provide the image with hadoop installed then that also works for bid data right? | 09:18 |
| tosky | kavithahr: sure, if you write the code which configures the various services | 09:19 |
| kavithahr | what is specific use of sahara? | 09:19 |
| tosky | for example out ambari plugin contains only ambari and deploys only the required hadoop services | 09:19 |
| frippe75 | kavithahr, think there are some puppet modules that could assist u if you want to go that route. | 09:20 |
| kavithahr | and one more question... hadoop itself is able to do data analytics or what? | 09:20 |
| tosky | it saves you from building your own custom hadoop image and change it whenever you need to change the settings; this is handled by Sahara | 09:20 |
| kavithahr | ok thank you.. | 09:20 |
| tosky | sure, tools like Hadoop, Spark and Storm were born for data analytics (but this is really not a Sahara-specific question :) | 09:22 |
| kavithahr | yeah but i wanted to know.... because i got confused with big data processing and big data analytics... | 09:23 |
| tosky | also Sahara provides an higher-level interface for job submission (called EDP in our documentation), even if it may not necessarily useful for everyone (you may have an existing application which talks directly with the spawned cluster) | 09:23 |
| tosky | well, let's say better this way, and hoping to not mess the definitions: with data processing you prepare your data for the analysis | 09:25 |
| tosky | if you can reorder and filter your data, you will have an hard time to analyze them | 09:26 |
| tosky | data processing tools prepare the data | 09:26 |
| kavithahr | according | 09:33 |
| kavithahr | according to me big data is huge data with comple | 09:40 |
| *** skp has quit IRC | 09:41 | |
| frippe75 | think of bigdata when hitting any of the 3(4) v's | 09:42 |
| frippe75 | volume,velocity,variety | 09:43 |
| kavithahr | @tosky according to me big data is huge data with complexity... big data processing is retrieving required information from stored big data... big data analytics is finding some patterns in that stored big data... is my understanding is correct? | 09:43 |
| tosky | yes, that should be it; I would say that you find big data processing for the analytics part (kind of first step) | 09:44 |
| frippe75 | also schema-on-read vs. schema-on-write. | 09:44 |
| frippe75 | you "trow in" any data (any format) in your "datalake"... The depending on what you want to do ... You apply your schema to the data and process it. | 09:45 |
| frippe75 | then depending... | 09:45 |
| kavithahr | The Sahara project provides a simple means to provision a data-intensive application cluster (Hadoop or Spark) on top of OpenStack.. i got this sentence from internet.. | 09:46 |
| kavithahr | sahara does only provisioning or it does any other service.. | 09:46 |
| frippe75 | hadoop also differs on moving the processing to the data instead of the other way around. | 09:46 |
| frippe75 | exactly | 09:47 |
| tosky | sahara also provides an interface for the jobs | 09:47 |
| kavithahr | interface for job means? | 09:48 |
| frippe75 | Think of hadoop as a multiple projects collections. Then you have distributions of theese (like Ubuntu and CentOS for Linux). Horton works has their dist and then there is MapR and Cloudera.... And the vanilla. | 09:49 |
| frippe75 | It's how you assemble the projects.... | 09:49 |
| tosky | interface for jobs (look for EDP in the documentation, Elastic Data Processing) | 09:49 |
| kavithahr | ok.. | 09:50 |
| kavithahr | if i install basic hadoop on my system will it be able to do big data analytics or any specific project (like mapr cloudera hive) i need to install? | 09:52 |
| *** frippe75 has quit IRC | 09:54 | |
| kavithahr | you told that think of hadoop as a multiple project collection... my question is if install hadoop all collection will come together or what? | 09:54 |
| tosky | you need to create your application which uses the frameworks, so whether you use hive or pig or some other component of the ecosystem it's up to you | 09:54 |
| kavithahr | thank you both of you.. you solved my doubts.. | 09:58 |
| *** sudipto has quit IRC | 10:17 | |
| *** sudipto_ has quit IRC | 10:17 | |
| *** degorenko has quit IRC | 10:30 | |
| *** degorenko has joined #openstack-sahara | 10:32 | |
| *** magicboiz has joined #openstack-sahara | 10:34 | |
| magicboiz | Hi, I have a problem launching my first spark test with sahara. Can I ask someone here to get some help?? Thanks | 10:41 |
| *** degorenko has quit IRC | 10:43 | |
| *** degorenko has joined #openstack-sahara | 10:45 | |
| tosky | magicboiz: not my best known part of the framework, but...spark test? Does it mean that you have a working spark cluster? | 10:49 |
| magicboiz | tosky: yeah, I have my own spark cluster running into my openstack cloud. | 10:50 |
| raissa | and what's the problem you're facing? | 10:51 |
| magicboiz | I launch an spark job but I get "Done with errors" | 10:55 |
| magicboiz | I'd like to know how to debug that error. | 10:55 |
| magicboiz | I'm testing with spark-wordcount.jar and https://github.com/openstack/sahara-tests/tree/master/sahara_tests/scenario/defaults/edp-examples/edp-spark | 10:56 |
| tosky | and nothing from the logs on the spark machines? You can access the webui as well | 10:58 |
| *** tuanluong has quit IRC | 10:59 | |
| magicboiz | I get logs like these: http://paste.openstack.org/show/599790/ | 11:00 |
| tosky | did you properly define the output datasource? | 11:01 |
| magicboiz | well, I think so: http://paste.openstack.org/show/599794/ | 11:08 |
| tosky | iirc the output should be an empty directory | 11:08 |
| tosky | (that's why he can use placeholders http://specs.openstack.org/openstack/sahara-specs/specs/kilo/edp-datasource-placeholders.html ) | 11:09 |
| magicboiz | tosky: empty directory? something like hdfs://spark16cluster-spark16ngtmasternodegroup-0:8020/user/ubuntu/<OUTPUT_DIR>/?? | 11:12 |
| tosky | yes | 11:12 |
| magicboiz | but, how could I configure spark daemon to debug/trace these errors? | 11:13 |
| *** frippe75 has joined #openstack-sahara | 11:13 | |
| tosky | uhm, not sure; this is on the boundary between spark and sahara | 11:13 |
| tosky | sparks completes and then the output is copied | 11:14 |
| frippe75 | tosky, still resource issues probably. it's stuck in the cluster creation phase... for 2 hours.. | 11:14 |
| frippe75 | waiting for instance accessibility.. instances are up. with floating ip's | 11:14 |
| frippe75 | well.. no floating ip's.. | 11:15 |
| tosky | uh | 11:15 |
| magicboiz | <tosky>: Done with Error again :( | 11:16 |
| tosky | did you specify the pool in the node template? (if you use them; if not, the configuration requires some tuning as sahara would need access through the private ip) | 11:16 |
| tosky | magicboiz: just to be sure, did <OUTPUT_DIR> exist? | 11:17 |
| frippe75 | you mean via the ssh-proxy? | 11:17 |
| tosky | frippe75: yeah, I always mess up that but it should be documented | 11:17 |
| magicboiz | <tosky>: yes it does exists and it's empty: | 11:18 |
| magicboiz | openstack dataprocessing data source show output | 11:18 |
| magicboiz | +--------------+-------------------------------------------------------------------------------+ | 11:18 |
| magicboiz | | Field | Value | | 11:18 |
| magicboiz | +--------------+-------------------------------------------------------------------------------+ | 11:18 |
| magicboiz | | Description | | | 11:18 |
| magicboiz | | Id | 67f8c794-3476-47b4-9c40-be9e515abb20 | | 11:18 |
| magicboiz | | Is protected | False | | 11:18 |
| magicboiz | | Is public | False | | 11:18 |
| magicboiz | | Name | output | | 11:18 |
| magicboiz | | Type | hdfs | | 11:18 |
| magicboiz | | Url | hdfs://spark16cluster-spark16ngtmasternodegroup-0:8020/user/ubuntu/outputdir/ | | 11:18 |
| magicboiz | +--------------+-------------------------------------------------------------------------------+ | 11:18 |
| magicboiz | ubuntu@spark16cluster-spark16ngtmasternodegroup-0:~$ hdfs dfs -ls /user/ubuntu/outputdir/ | 11:18 |
| magicboiz | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". | 11:18 |
| magicboiz | SLF4J: Defaulting to no-operation (NOP) logger implementation | 11:18 |
| magicboiz | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. | 11:18 |
| magicboiz | ubuntu@spark16cluster-spark16ngtmasternodegroup-0:~$ | 11:18 |
| tosky | ehm, when pasting more than two lines, please use a pastebin | 11:18 |
| magicboiz | sorry, you're right! | 11:19 |
| frippe75 | The only thing I've changed from being almost successful (last few services failing to start due to memory shortage). Is the flavors. And they look almost the same. Just some more memory on the master/secondary nodes.... Will go over all config again :-) | 11:19 |
| tosky | frippe75: if it's network and waiting on nodes, it's not on ambari yet | 11:19 |
| frippe75 | ok thanks! | 11:19 |
| tosky | frippe75: when did you say "no floating ip's" did you mean that you configured it to not use them, or do you expect to use them but they were not associated with the instances? | 11:20 |
| tosky | magicboiz: do you happen to have swift around just for a quick test with a swift datasource? | 11:20 |
| frippe75 | they were not associated after two hours. Maybe that come later? | 11:20 |
| tosky | no, it's the first thing | 11:21 |
| frippe75 | ?? ok | 11:21 |
| magicboiz | tosky, no sorry, swift not deployed right now | 11:21 |
| tosky | frippe75: when they are available, sahara access the nodes through ssh and start the real configuration (in your case, by driving ambari) | 11:22 |
| frippe75 | really strange. I'm working on a script that sets the sahara bits (images,register them and so on). So I'm really sure I didn't change anything but the flavor_id part. | 11:22 |
| frippe75 | Ok sound fairly uncomplicated. Must have made a change somewhere by mistake. | 11:22 |
| tosky | magicboiz: uhm, so, the issue is definitely related with the lack of the output in the expected place: http://git.openstack.org/cgit/openstack/sahara/tree/sahara/service/edp/spark/engine.py#n107 | 11:27 |
| frippe75 | tosky, fired up an ubuntu with a floating ip. project quotas are 37 available. Must be in the node group template after all.. | 11:27 |
| magicboiz | tosky: any idea on how to enable debug/tracing on the spark daemons? | 11:28 |
| tosky | magicboiz: no | 11:31 |
| tosky | magicboiz: check if the /tmp/spark-edp<something> files are still available | 11:31 |
| magicboiz | tosky: http://paste.openstack.org/show/599798/ this is a log from one of the workers | 11:31 |
| frippe75 | tosky, sure u were correct. No ip association in the node group template for any of them.. How that got that way is beyond me that's why I didn't even bother to check... | 11:31 |
| frippe75 | So... Now I think it will work.... 5bucks on that? :-) | 11:32 |
| tosky | magicboiz: and I may be wrong, it could be an error when the tmp file is read, not when it is uploaded | 11:32 |
| frippe75 | floating assigned.... puh | 11:36 |
| magicboiz | toskey: it might be related with https://issues.apache.org/jira/browse/SPARK-9844?? | 11:36 |
| tosky | magicboiz: uhm, it's not clear if it was fixed in 1.6.1 or later | 11:41 |
| tosky | but we generally deploy spark 1.6.0 unfortunately (only in mapr there is 1.6.1) | 11:42 |
| tosky | on the other side, shouldn't that bug appear with both hdfs and swift? | 11:42 |
| tosky | on the gates that spark jobs writes to swift | 11:43 |
| tosky | but yeah, race conditions | 11:46 |
| tosky | which spark image did you use? The basic one (vanilla-based), or cdh, mapr...? | 11:49 |
| tosky | you may try to fix sahara-image-elements so that it accepts 1.6.1 and rebuild the image | 11:49 |
| magicboiz | I'm running sahara-newton-spark-1.6.0-ubuntu.qcow2 | 11:50 |
| magicboiz | I'm going to try to update that image and test again | 11:51 |
| tosky | the last comment on the bug says that it should be mostly harmless though | 11:53 |
| tosky | maybe a red herring | 11:53 |
| frippe75 | tusky, THANKS A BUNCH, no all services have started and I can continue playing on my own hadoop instead of the HDInsight on Azure ... Woho. I do realize this is more of a dev-department-hangaround so really thanks for giving me the support! | 11:59 |
| frippe75 | now all the... | 11:59 |
| tosky | good! | 12:06 |
| tosky | at least one problem solved today | 12:06 |
| *** logan- has joined #openstack-sahara | 12:35 | |
| magicboiz | tosky: with spark image 1.3.1, almost the same error: http://paste.openstack.org/show/599812/ | 12:37 |
| tosky | magicboiz: well, if it is that error, according the bug report it was definitely in 1.3.1 | 12:38 |
| magicboiz | solved in 1.3.1? | 12:38 |
| tosky | found in 1.3.1, up to 1.6.0 | 12:39 |
| tosky | solved in 1.6.1 | 12:39 |
| tosky | this is what I read from https://issues.apache.org/jira/browse/SPARK-9844 | 12:39 |
| tosky | can you try to prepare a longer example? Maybe the race goes away if the job takes more time | 12:39 |
| magicboiz | do you have a spark binary that you're fully sure that works? Right now, I'm only working with spark-wordcount.jar | 12:41 |
| magicboiz | spark-wordcount.jar from https://github.com/openstack/sahara-tests/tree/master/sahara_tests/scenario/defaults/edp-examples/edp-spark | 12:42 |
| tosky | as I said, that works at least on the gates with swift as output datasource | 12:42 |
| magicboiz | could I setup a AWS S3/swift store for the data sources? | 12:43 |
| tosky | I can't test right now on a proper environment (which version of openstack are you using, by the way?) but one thing I would try is fix sahara-image-elements to accept spark 1.6.1 and rebuild that image | 12:43 |
| magicboiz | newton | 12:43 |
| tosky | or that too, if you have a swift setup you can try as well | 12:44 |
| tosky | now why the review that I sent did not show here? | 13:09 |
| tosky | uhm, no openstackgerri | 13:09 |
| tosky | openstackgerrit* | 13:09 |
| *** afazekas is now known as afazekas|broken_ | 13:20 | |
| *** slashme has quit IRC | 13:33 | |
| *** slashme has joined #openstack-sahara | 13:35 | |
| *** dave-mccowan has joined #openstack-sahara | 13:52 | |
| *** dave-mcc_ has joined #openstack-sahara | 13:55 | |
| *** dave-mccowan has quit IRC | 13:57 | |
| *** jamielennox|away is now known as jamielennox | 14:09 | |
| *** dave-mcc_ has quit IRC | 14:10 | |
| *** david-lyle has joined #openstack-sahara | 14:12 | |
| *** abalutoiu has joined #openstack-sahara | 14:14 | |
| *** abalutoiu has quit IRC | 14:16 | |
| *** abalutoiu has joined #openstack-sahara | 14:16 | |
| *** jperry has joined #openstack-sahara | 14:21 | |
| *** nkrinner is now known as nkrinner_afk | 14:22 | |
| *** openstackgerrit has joined #openstack-sahara | 14:31 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: Default test templates: add missing, remove obsolete https://review.openstack.org/436116 | 14:31 |
| tosky | oh, welcome back openstackgerrit | 14:32 |
| raissa | it's alive | 14:33 |
| *** dave-mccowan has joined #openstack-sahara | 14:33 | |
| *** david-lyle has quit IRC | 14:45 | |
| *** david-lyle has joined #openstack-sahara | 14:46 | |
| *** david-lyle has quit IRC | 14:46 | |
| *** david-lyle has joined #openstack-sahara | 14:53 | |
| *** david-lyle has quit IRC | 14:57 | |
| *** abalutoiu_ has joined #openstack-sahara | 15:01 | |
| *** abalutoiu has quit IRC | 15:04 | |
| *** sahithirv has joined #openstack-sahara | 15:06 | |
| *** abalutoiu_ is now known as abalutoiu | 15:12 | |
| *** jiasen has joined #openstack-sahara | 15:13 | |
| *** jiasen has quit IRC | 15:16 | |
| *** david-lyle_ has joined #openstack-sahara | 15:29 | |
| *** skp has joined #openstack-sahara | 15:59 | |
| *** jamielennox is now known as jamielennox|away | 15:59 | |
| *** jamielennox|away is now known as jamielennox | 15:59 | |
| magicboiz | <tosky>: I managed to upgrade spark 1.6.3, race condition fixed, but spark job still is interrupted | 16:00 |
| tosky | uh | 16:01 |
| tosky | so maybe it was really a red herring | 16:01 |
| tosky | can you check the files /tmp/spark-edp<something> on the nodes, if not removed? | 16:01 |
| *** david-lyle has joined #openstack-sahara | 16:01 | |
| *** david-lyle_ has quit IRC | 16:02 | |
| magicboiz | http://paste.openstack.org/show/599859/ | 16:02 |
| magicboiz | http://paste.openstack.org/show/599860/ | 16:03 |
| magicboiz | tmp dir looks empty | 16:03 |
| tosky | magicboiz: which node is that? (see https://docs.openstack.org/developer/sahara/horizon/dashboard.user.guide.html#example-jobs ) | 16:18 |
| tosky | quoting: "The stdout and stderr files of the command used for executing your job are located at /tmp/spark-edp/<name of job template>/<job id> on Spark master node in case of Spark clusters, or on Spark JobHistory node in other cases like Vanilla, CDH and so on." | 16:18 |
| magicboiz | sorry, it was a worker node. | 16:20 |
| tosky | oh, np | 16:22 |
| tosky | I didn't remember that at all, so good thing for me too :) | 16:22 |
| magicboiz | ummm..... "Exception in thread "main" java.io.IOException: No FileSystem for scheme: datasource" | 16:22 |
| tosky | uh | 16:24 |
| tosky | can you please repost the configuration of the data source? | 16:25 |
| *** david-lyle has quit IRC | 16:27 | |
| magicboiz | here they are: http://paste.openstack.org/show/599863/ | 16:29 |
| magicboiz | but it looks like the job definition is mixing "hdfs" with "datasource" | 16:30 |
| *** david-lyle has joined #openstack-sahara | 16:34 | |
| tosky | magicboiz: how did you specify the data sources in the job? | 16:35 |
| magicboiz | http://paste.openstack.org/show/599865/ | 16:36 |
| magicboiz | I think I miss edp.substitute_data_source_for_name = False..... | 16:36 |
| tosky | iirc that's the default | 16:37 |
| tosky | do you mean True? | 16:38 |
| tosky | yeah, maybe | 16:38 |
| magicboiz | yes, I mean, I'm starting to think that I need to set edp.substitute_data_source_for_name = True | 16:39 |
| tosky | I would try, easier than check | 16:39 |
| tosky | :) | 16:39 |
| *** jamielennox is now known as jamielennox|away | 16:39 | |
| magicboiz | got it!!! | 16:40 |
| tosky | did it work? | 16:41 |
| magicboiz | it looks like it works. When " edp.substitute_data_source_for_name = False" Args must look like "hdfs://<server>:8020/full/path" | 16:43 |
| *** abalutoiu has quit IRC | 16:43 | |
| magicboiz | for example: "hdfs://spark16clusterv2-spark16ngtmasternodegroup-0:8020/user/ubuntu/README.md" | 16:43 |
| magicboiz | Succeeded :) | 16:44 |
| magicboiz | Exit status 0 | 16:44 |
| magicboiz | I love it! | 16:44 |
| magicboiz | output dir must not exist, btw | 16:45 |
| *** sahithirv has quit IRC | 16:45 | |
| tosky | \o/ \o/ \o/ | 16:46 |
| tosky | ah, ok | 16:46 |
| magicboiz | tosky: THANKS | 16:49 |
| magicboiz | :) | 16:49 |
| tosky | :) | 16:49 |
| *** david-lyle has quit IRC | 16:58 | |
| magicboiz | tosky: Can I ask you what are the basic steps to transform a spark-apache app (like WordCount.java) into a EDP.Spark-apache app? thanks in advance :) | 16:59 |
| tosky | eh, uh, isn't it what you already did? I mean, the spark example does not use anything Sahara-specific | 17:02 |
| *** dave-mccowan has quit IRC | 17:08 | |
| magicboiz | no, jajajaja, I Just made use of work already done by others. In this case, I made use of https://github.com/openstack/sahara-tests/blob/master/sahara_tests/scenario/defaults/edp-examples/edp-spark/spark-wordcount.jar | 17:08 |
| magicboiz | but I'd like to know how to transfor my apache spark binaries into edp-sahara-apache-spark binary....just importing sahara.edp.spark? | 17:10 |
| *** pcaruana has quit IRC | 17:12 | |
| *** david-lyle has joined #openstack-sahara | 17:15 | |
| *** david-lyle has quit IRC | 17:16 | |
| tosky | if you mean the name of the package, I don't think that it's relevant | 17:17 |
| tosky | you just need to specify the correct class with edp.java.main_class: | 17:17 |
| tosky | the example is in the package sahara.edp.spark because it comes from us (sahara), but it's just an example | 17:18 |
| tosky | like the edp-jave example is org.openstack.sahara.examples, or the example used for the pig job is org.openstack.sahara.examples.pig | 17:18 |
| tosky | but it's because we created them | 17:18 |
| tosky | you are not bound by that | 17:18 |
| magicboiz | ok, thx again | 17:19 |
| tosky | as a generic note, this week most of OpenStack developers (including Sahara, with few exceptions like me) are at the Project Team Gathering in Atlanta | 17:33 |
| tosky | so expect less answers than usual | 17:33 |
| *** tesseract has quit IRC | 17:41 | |
| *** abalutoiu has joined #openstack-sahara | 17:47 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara master: Configure the publicURL instead of adminURL in devstack https://review.openstack.org/436567 | 17:48 |
| *** sudipto has joined #openstack-sahara | 17:54 | |
| *** sudipto_ has joined #openstack-sahara | 17:54 | |
| *** dave-mccowan has joined #openstack-sahara | 18:04 | |
| *** sudipto has quit IRC | 18:05 | |
| *** sudipto_ has quit IRC | 18:05 | |
| *** sudipto_ has joined #openstack-sahara | 18:05 | |
| *** sudipto has joined #openstack-sahara | 18:06 | |
| *** sudipto_ has quit IRC | 18:07 | |
| *** sudipto has quit IRC | 18:07 | |
| frippe75 | where can I find the json version of "openstack dataprocessing node group create --autoconfig", autoconfig and auto_config return unexpected keyword? | 18:08 |
| *** openstack has joined #openstack-sahara | 18:15 | |
| *** abalutoiu_ has joined #openstack-sahara | 18:18 | |
| frippe75 | now trying to find the volume_type value for cinder storage. Tempest tests suggest it's not "big" I guess... | 18:19 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: Default test templates: add missing, remove obsolete https://review.openstack.org/436116 | 18:21 |
| *** abalutoiu has quit IRC | 18:21 | |
| *** dave-mccowan has quit IRC | 18:32 | |
| *** mkoderer___ has quit IRC | 18:33 | |
| *** dave-mccowan has joined #openstack-sahara | 18:33 | |
| *** david-lyle has joined #openstack-sahara | 18:46 | |
| *** abalutoiu_ has quit IRC | 18:48 | |
| *** david-lyle has quit IRC | 18:57 | |
| *** dave-mccowan has quit IRC | 19:19 | |
| *** dave-mccowan has joined #openstack-sahara | 19:21 | |
| *** dave-mccowan has quit IRC | 19:39 | |
| *** dave-mccowan has joined #openstack-sahara | 19:41 | |
| *** chlong has quit IRC | 20:01 | |
| *** david-lyle has joined #openstack-sahara | 20:01 | |
| *** chlong has joined #openstack-sahara | 20:08 | |
| *** dave-mccowan has quit IRC | 20:12 | |
| openstackgerrit | Marianne Linhares Monteiro proposed openstack/sahara master: Code integration with the abstractions https://review.openstack.org/427480 | 20:18 |
| *** david-lyle has quit IRC | 20:23 | |
| *** david-lyle has joined #openstack-sahara | 20:38 | |
| *** david-lyle has quit IRC | 20:49 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: Default test templates: add missing, remove obsolete https://review.openstack.org/436116 | 20:50 |
| *** david-lyle has joined #openstack-sahara | 20:55 | |
| *** david-lyle has quit IRC | 20:59 | |
| *** esikachev has joined #openstack-sahara | 21:01 | |
| *** itisha has joined #openstack-sahara | 21:13 | |
| *** dave-mccowan has joined #openstack-sahara | 21:22 | |
| *** dave-mccowan has quit IRC | 21:29 | |
| *** skp has quit IRC | 22:11 | |
| *** dave-mccowan has joined #openstack-sahara | 22:13 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: Default test templates: add missing, remove obsolete https://review.openstack.org/436116 | 22:20 |
| *** tosky has quit IRC | 22:20 | |
| *** dave-mccowan has quit IRC | 22:24 | |
| *** esikachev has quit IRC | 22:48 | |
| *** frippe75 has quit IRC | 23:07 | |
| *** jperry has quit IRC | 23:16 | |
| *** esikachev has joined #openstack-sahara | 23:28 | |
| *** itisha has quit IRC | 23:32 | |
| *** esikachev has quit IRC | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!