| *** alazarev has quit IRC | 00:04 | |
| *** alazarev has joined #openstack-sahara | 00:18 | |
| *** bradd1 has joined #openstack-sahara | 00:28 | |
| *** zhiyan_ is now known as zhiyan | 00:28 | |
| *** IlyaE has joined #openstack-sahara | 00:31 | |
| *** shakayumi has joined #openstack-sahara | 00:32 | |
| *** shakayumi has quit IRC | 00:45 | |
| *** ViswaV has quit IRC | 01:10 | |
| *** alazarev has quit IRC | 01:26 | |
| *** ViswaV has joined #openstack-sahara | 01:28 | |
| *** ViswaV_ has joined #openstack-sahara | 02:02 | |
| *** ViswaV has quit IRC | 02:05 | |
| *** IlyaE has quit IRC | 02:10 | |
| *** IlyaE has joined #openstack-sahara | 02:10 | |
| *** ViswaV_ has quit IRC | 02:15 | |
| *** sballe has joined #openstack-sahara | 03:47 | |
| *** zhiyan is now known as zhiyan_ | 04:15 | |
| *** zhiyan_ is now known as zhiyan | 04:17 | |
| *** shakamunyi has joined #openstack-sahara | 05:04 | |
| *** bandarji has joined #openstack-sahara | 05:10 | |
| *** bandarji has quit IRC | 05:18 | |
| *** bandarji has joined #openstack-sahara | 05:18 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Rework keystone auth_token middleware configs https://review.openstack.org/74688 | 05:31 |
|---|---|---|
| *** alazarev has joined #openstack-sahara | 05:36 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/97006 | 06:01 |
| openstackgerrit | Sergey Kolekonov proposed a change to stackforge/sahara-ci-config: Add HDP2 job https://review.openstack.org/96746 | 06:04 |
| *** bandarji has quit IRC | 06:35 | |
| *** alazarev has quit IRC | 07:01 | |
| *** skolekonov has joined #openstack-sahara | 07:16 | |
| openstackgerrit | A change was merged to openstack/sahara: Create trusts for admin user with correct tenant name https://review.openstack.org/93322 | 07:19 |
| yanndegat | dmitryme: no | 07:19 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/python-saharaclient: Fix network passing for cluster-template-create https://review.openstack.org/97255 | 07:21 |
| openstackgerrit | Sergey Kolekonov proposed a change to stackforge/sahara-ci-config: disable Sahara pypi mirror in all jobs https://review.openstack.org/97700 | 07:33 |
| *** zhiyan is now known as zhiyan_ | 07:34 | |
| openstackgerrit | Sergey Kolekonov proposed a change to stackforge/sahara-ci-config: Add HDP2 job https://review.openstack.org/96746 | 07:51 |
| *** IlyaE has quit IRC | 07:55 | |
| *** jpich has joined #openstack-sahara | 07:55 | |
| *** julienvey has joined #openstack-sahara | 08:07 | |
| openstackgerrit | Daniele Venzano proposed a change to openstack/sahara: Documentation for the Spark plugin https://review.openstack.org/97498 | 08:10 |
| openstackgerrit | Daniele Venzano proposed a change to openstack/sahara: Add Spark to overview and feature matrix https://review.openstack.org/97709 | 08:10 |
| *** shakamunyi has quit IRC | 08:10 | |
| *** tosky has joined #openstack-sahara | 08:23 | |
| *** yanndegat has quit IRC | 08:30 | |
| *** julienvey has quit IRC | 08:35 | |
| *** IvanBerezovskiy1 has joined #openstack-sahara | 08:35 | |
| *** sballe has quit IRC | 08:48 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** IvanBerezovskiy1 has quit IRC | 08:48 | |
| *** liusheng has quit IRC | 08:48 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 08:48 | |
| *** IvanBerezovskiy has left #openstack-sahara | 08:49 | |
| *** sballe has joined #openstack-sahara | 08:49 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 08:49 | |
| *** liusheng has joined #openstack-sahara | 08:49 | |
| *** openstackgerrit has joined #openstack-sahara | 08:50 | |
| *** julienvey has joined #openstack-sahara | 09:03 | |
| *** witlessb has joined #openstack-sahara | 09:04 | |
| *** julienvey has quit IRC | 09:08 | |
| *** julienvey has joined #openstack-sahara | 09:17 | |
| *** julienvey has quit IRC | 09:42 | |
| *** julienvey has joined #openstack-sahara | 09:43 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Rework keystone auth_token middleware configs https://review.openstack.org/74688 | 09:44 |
| *** julienvey has quit IRC | 09:45 | |
| *** vrovachev has joined #openstack-sahara | 09:57 | |
| *** Timotey has joined #openstack-sahara | 10:02 | |
| dmitryme | yanndegat: we’ve uploaded the image to the sahara-files, here is the link: http://sahara-files.mirantis.com/centos-6_4-64-hdp-2-0.qcow2 | 10:20 |
| *** sballe has quit IRC | 10:35 | |
| *** shakamunyi has joined #openstack-sahara | 10:37 | |
| *** sballe has joined #openstack-sahara | 10:40 | |
| *** shakamunyi has quit IRC | 10:41 | |
| *** sballe_ has joined #openstack-sahara | 10:42 | |
| *** sballe has quit IRC | 10:45 | |
| *** sballe_ has quit IRC | 10:48 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Rework keystone auth_token middleware configs https://review.openstack.org/74688 | 11:02 |
| *** witlessb has quit IRC | 11:02 | |
| *** witlessb has joined #openstack-sahara | 11:03 | |
| *** julienvey has joined #openstack-sahara | 11:10 | |
| *** julienvey has quit IRC | 11:15 | |
| *** julienvey has joined #openstack-sahara | 11:17 | |
| openstackgerrit | Daniele Venzano proposed a change to openstack/sahara: Documentation for the Spark plugin https://review.openstack.org/97498 | 11:22 |
| openstackgerrit | Daniele Venzano proposed a change to openstack/sahara: Add Spark to overview and feature matrix https://review.openstack.org/97709 | 11:22 |
| *** shakamunyi has joined #openstack-sahara | 11:38 | |
| openstackgerrit | afazekas proposed a change to openstack/sahara: Allow HDFS data source paths without the hdfs:// scheme https://review.openstack.org/97762 | 11:39 |
| *** shakamunyi has quit IRC | 11:43 | |
| *** yanndegat has joined #openstack-sahara | 11:55 | |
| dmitryme | yanndegat: we’ve uploaded the image to the sahara-files, here is the link: http://sahara-files.mirantis.com/centos-6_4-64-hdp-2-0.qcow2 | 12:24 |
| yanndegat | ok | 12:25 |
| yanndegat | thank you | 12:25 |
| *** yanndegat has left #openstack-sahara | 12:37 | |
| *** _elmiko is now known as elmiko | 13:02 | |
| *** _crobertsrh is now known as crobertsrh | 13:07 | |
| *** tmckay has joined #openstack-sahara | 13:07 | |
| *** julienvey has quit IRC | 13:08 | |
| *** shakamunyi has joined #openstack-sahara | 13:16 | |
| *** palar has joined #openstack-sahara | 13:27 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Make deleting transient clusters safe https://review.openstack.org/97167 | 13:38 |
| *** julienvey has joined #openstack-sahara | 13:39 | |
| *** julienve_ has joined #openstack-sahara | 13:41 | |
| *** julienvey has quit IRC | 13:44 | |
| *** tellesnobrega_ has joined #openstack-sahara | 13:52 | |
| *** bradd1 has quit IRC | 14:05 | |
| *** bradd1 has joined #openstack-sahara | 14:06 | |
| *** skolekonov has quit IRC | 14:18 | |
| *** jpich has quit IRC | 14:50 | |
| *** sballe has joined #openstack-sahara | 14:50 | |
| *** sballe_ has joined #openstack-sahara | 14:52 | |
| *** sballe has quit IRC | 14:56 | |
| *** sballe__ has joined #openstack-sahara | 14:57 | |
| *** jpich has joined #openstack-sahara | 14:57 | |
| *** IlyaE has joined #openstack-sahara | 15:00 | |
| *** sballe_ has quit IRC | 15:00 | |
| *** ghenriks has quit IRC | 15:04 | |
| *** IlyaE has quit IRC | 15:05 | |
| *** sballe__ has quit IRC | 15:06 | |
| *** yanndegat has joined #openstack-sahara | 15:13 | |
| yanndegat | dmitryme: hi again dmitry | 15:13 |
| yanndegat | tried the new image | 15:13 |
| yanndegat | i think there must be a bug somewhere | 15:14 |
| yanndegat | and i think it's not related to the image | 15:14 |
| yanndegat | when i deploy my cluster | 15:14 |
| yanndegat | my slaves ( DATANODE | 15:15 |
| yanndegat | HDFS_CLIENT | 15:15 |
| yanndegat | MAPREDUCE2_CLIENT | 15:15 |
| yanndegat | YARN_CLIENT | 15:15 |
| yanndegat | NODEMANAGER ) | 15:15 |
| yanndegat | miss a /volumes/disk1/haddop/yarn directory | 15:15 |
| yanndegat | we have to manually mkdir the yarn/local dir | 15:15 |
| yanndegat | and chown it to the yarn user | 15:15 |
| yanndegat | have you ever encounter this pb ? | 15:16 |
| *** tellesnobrega_ has quit IRC | 15:17 | |
| *** ylobankov has joined #openstack-sahara | 15:23 | |
| dmitryme | yanndegat: nope, we didn’t. With the new image it just worked | 15:26 |
| dmitryme | on the other side we used different processes set on node groups | 15:27 |
| dmitryme | see here: https://review.openstack.org/#/c/95739/5/sahara/tests/integration/configs/itest.conf.sample-full | 15:27 |
| dmitryme | MASTER_NODE_PROCESSES and WORKER_NODE_PROCESSES | 15:27 |
| *** IvanBerezovskiy has left #openstack-sahara | 15:27 | |
| yanndegat | how can i know if it is due to the version of sahara we use ? | 15:29 |
| dmitryme | and which version do you use? | 15:29 |
| dmitryme | if it is Icehouse or master, I think it is expected to just work fine | 15:30 |
| yanndegat | openstack-sahara.noarch 2014.1.0-13.el6 @openstack-icehouse | 15:30 |
| yanndegat | python-django-sahara.noarch 2014.1.0-3.el6 @openstack-icehouse | 15:30 |
| yanndegat | python-saharaclient.noarch 0.7.0-3.el6 @openstack-icehouse | 15:30 |
| dmitryme | yep, that is Icehouse | 15:30 |
| yanndegat | well, the yarn local directory is not present | 15:31 |
| dmitryme | try setting processes in the node groups the same way as in config above ^^ | 15:31 |
| dmitryme | it might help | 15:32 |
| *** alazarev has joined #openstack-sahara | 15:33 | |
| *** ViswaV has joined #openstack-sahara | 15:37 | |
| yanndegat | alright | 15:37 |
| yanndegat | we'll try this config, but wouldn't be strange ? the only difference between our config and the one you send is | 15:41 |
| yanndegat | on the master : yours have theses processes GANGLIA_SERVER, NAGIOS_SERVER, OOZIE_SERVER | 15:42 |
| yanndegat | on the slave : yyours have these processes : ZOOKEEPER_CLIENT, PIG, OOZIE_CLIENT | 15:42 |
| yanndegat | this would mean that either PIG, OOZIE or ZOOKEEPER would create the yarn directory ? | 15:43 |
| openstackgerrit | A change was merged to openstack/python-saharaclient: removed unused methods from nova.utils https://review.openstack.org/95932 | 15:46 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/96828 | 15:52 |
| dmitryme | yanndegat: I don’t know HDP plugin well, but I remember that when I worked with it long ago setting the right process set was essential | 15:57 |
| dmitryme | maybe Hortonworks guys fixed that | 15:57 |
| dmitryme | but it is worth trying | 15:57 |
| yanndegat | i'm trying | 15:57 |
| yanndegat | ;) | 15:57 |
| yanndegat | hmmm | 16:03 |
| yanndegat | maybe it is because | 16:03 |
| yanndegat | i have 2 images | 16:03 |
| yanndegat | registered with the tags "hdp" "2.0.6" ;) | 16:03 |
| dmitryme | yanndegat: nope, it does not matter, you select just one image while launching the cluster from the UI | 16:05 |
| dmitryme | just make sure you’ve picked the new one | 16:05 |
| openstackgerrit | Matthew Farrellee proposed a change to openstack/python-saharaclient: Remove excutils from openstack-common.conf https://review.openstack.org/97843 | 16:09 |
| yanndegat | i've picked up the good onbe | 16:09 |
| yanndegat | but it is said in the docs that hdp needs absolutely the correct tags to work properly | 16:10 |
| yanndegat | so ... | 16:10 |
| yanndegat | my cluster is still in the "starting" phase | 16:10 |
| dmitryme | don’t worry about tags if cluster reached the ‘starting’ phase | 16:12 |
| dmitryme | check sahara-api log for exceptions if it is ‘starting’ for too long | 16:12 |
| yanndegat | alright | 16:13 |
| yanndegat | it is active | 16:13 |
| yanndegat | wait a minute.... | 16:13 |
| *** tellesnobrega_ has joined #openstack-sahara | 16:16 | |
| yanndegat | ok | 16:17 |
| yanndegat | so the yarn directory is not present | 16:17 |
| yanndegat | even with the same processes as listed in your conf | 16:17 |
| dmitryme | do you say that cluster entered ‘active’ state? | 16:18 |
| *** alazarev has quit IRC | 16:19 | |
| yanndegat | i recreated a complete cluster | 16:20 |
| *** julienve_ has quit IRC | 16:20 | |
| dmitryme | are you sure it is actually needed? | 16:22 |
| yanndegat | with the correct image and the correct processes | 16:22 |
| yanndegat | the cluster is active | 16:22 |
| yanndegat | when i ssh in a slave node | 16:22 |
| dmitryme | s / it / the directory / | 16:22 |
| yanndegat | the yarn directory is not preset | 16:22 |
| yanndegat | yes | 16:23 |
| yanndegat | because in the resource manager, the node is marked as unhealthy | 16:23 |
| dmitryme | sorry, have really small knowledge in the area | 16:24 |
| yanndegat | ok | 16:24 |
| yanndegat | nevermind, we'll figure it out | 16:25 |
| yanndegat | thanks | 16:25 |
| dmitryme | and there seems to be no guys from the Hortonworks around | 16:25 |
| dmitryme | no problem | 16:25 |
| *** jpich has quit IRC | 16:28 | |
| *** Timotey has quit IRC | 16:35 | |
| *** alazarev has joined #openstack-sahara | 16:50 | |
| *** ylobankov has quit IRC | 16:55 | |
| tmckay | SergeyLukjanov, aignatov, ping | 17:07 |
| tmckay | alazarev, too | 17:07 |
| alazarev | tmckay: here | 17:08 |
| *** tosky__ has joined #openstack-sahara | 17:08 | |
| *** tosky has quit IRC | 17:08 | |
| tmckay | alazarev, hi. Looking to talk about spark stuff, I should probably start a pad. And we can talk about it tomorrow at the meeting. But, here is what I have found so far ... | 17:09 |
| tmckay | Cloudera initially implemented spark job submission in Hue using an Oozie shell action that invoked pyspark to launch a job. Very minimal wrapper. I found the code from late 2013 in git. | 17:10 |
| tmckay | But, this means that hadoop-mapreduce is required in order to run spark jobs. Seems bad. It shouldn't be required. | 17:10 |
| tmckay | So, by Christmas they moved to an opensource spark job server from a company called ooyala | 17:10 |
| tmckay | This seems to be what everyone wants to use on the spark mail lists for a central job server | 17:11 |
| tmckay | So I think the answer to our investigation of spark via Oozie is probably "No" or at the very least "Well, okay, but we're going to turn around and replace it almost immediately" | 17:12 |
| *** shakamunyi has quit IRC | 17:13 | |
| tmckay | alazarev, so this brings up a few questions. How in openstack can we pull in an opensource job server from a non-openstack project? If we did that, would we just be responsible for maintaining it as part of Sahara? | 17:13 |
| alazarev | we can start with Oozie and think how to make edp logic engine-independant | 17:13 |
| tmckay | well, that's one option | 17:13 |
| tmckay | It means that the spark plugin has to be extended to install hadoop mapreduce, just to be able to run the shell action | 17:14 |
| tmckay | seems like a big dependency, which I don't like | 17:14 |
| alazarev | because we need at least two engines to perform such work, and facing issues with spark itself is not something we want to do during such work | 17:14 |
| alazarev | tmckay: agree, but this dependancy is big mostly for customers, in terms of code it is pretty easy | 17:16 |
| tmckay | alazarev, I almost think it would be preferable to just issue remote commands to run spark jobs from the command line, instead of going through Oozie | 17:16 |
| tmckay | alazarev, yes. It might get us a quick proof of concept but I don't think it's a good long term solution. | 17:17 |
| tmckay | I should start the pad and we can discuss it | 17:17 |
| tmckay | It concerns me that cloudera moved away from the oozie imple immediately -- October to Dec. | 17:17 |
| alazarev | tmckay: agree, long term we need to support spark directly | 17:17 |
| tmckay | they obviously saw no future there | 17:18 |
| alazarev | probably they had all infrastructure to make such move | 17:18 |
| alazarev | currently we don’t | 17:18 |
| alazarev | job_manager is hardly tied on oozie | 17:19 |
| tmckay | alazarev, do you mean hardwired? | 17:19 |
| alazarev | tmckay: | 17:20 |
| alazarev | tmckay: yes | 17:20 |
| alazarev | I tried to structure the code a little bit: https://review.openstack.org/#/c/97633/ | 17:20 |
| tmckay | alazarev, yes, I agree. I'm wondering if it is worth looking at the refactoring first, or make spark work via oozie first and then refactor | 17:20 |
| tmckay | oozie is quick, maybe, but then we throw the work away | 17:21 |
| alazarev | without throwing code away it could take even more effort | 17:22 |
| alazarev | incremental refactoring is much better than complete rewriting of job_manager | 17:23 |
| *** tosky__ is now known as tosky | 17:38 | |
| *** julienvey has joined #openstack-sahara | 17:47 | |
| *** akuznetsov has joined #openstack-sahara | 18:09 | |
| *** shakayumi has joined #openstack-sahara | 18:18 | |
| *** elmiko has quit IRC | 18:39 | |
| *** elmiko has joined #openstack-sahara | 18:42 | |
| *** akuznetsov has quit IRC | 18:48 | |
| *** akuznetsov has joined #openstack-sahara | 18:49 | |
| *** elmiko has quit IRC | 18:50 | |
| *** elmiko has joined #openstack-sahara | 18:52 | |
| *** tellesnobrega_ has quit IRC | 19:12 | |
| *** tellesnobrega_ has joined #openstack-sahara | 19:19 | |
| *** elmiko has quit IRC | 19:22 | |
| *** akuznetsov has quit IRC | 19:23 | |
| tellesnobrega_ | alazarev: hi, i’m working on the quotas check bug | 19:26 |
| tellesnobrega_ | i was thinking the following, on the manager.py i would call novaclient to get the quotas and verify if there is enough based on the templates that are passed on the values dict, does that sound like a good option? | 19:27 |
| alazarev | tellesnobrega_: yes, that sounds good | 19:27 |
| tellesnobrega_ | alazarev: ok | 19:27 |
| *** alazarev has quit IRC | 19:28 | |
| SergeyLukjanov | tmckay, hey, aignatov is on vacations till the end of week and I'm too tired to be useful now | 19:29 |
| SergeyLukjanov | but I see non-opensource job manager - -2 | 19:29 |
| *** elmiko has joined #openstack-sahara | 19:34 | |
| *** bradd1 has quit IRC | 19:38 | |
| *** alazarev has joined #openstack-sahara | 19:40 | |
| *** palar_ has joined #openstack-sahara | 19:40 | |
| *** palar has quit IRC | 19:43 | |
| *** palar_ has quit IRC | 19:44 | |
| *** elmiko has quit IRC | 19:52 | |
| openstackgerrit | A change was merged to openstack/sahara: [HDP] Changed test tag for HDP1 plugin https://review.openstack.org/97489 | 19:54 |
| *** elmiko has joined #openstack-sahara | 19:58 | |
| tmckay | SergeyLukjanov, well, it is *opensource*, just not *openstack* :) | 20:13 |
| tmckay | SergeyLukjanov, I'm going to start a pad for discussion on how we approach this. Whether to do something quick with Oozie, and introduce hadoop-mapreduce dependency for spark clusters initially, or do something better that takes more work and skip the oozie step | 20:14 |
| tmckay | I lean toward "skip". | 20:14 |
| tmckay | but we can talk about it tomorrow | 20:14 |
| *** bandarji has joined #openstack-sahara | 20:27 | |
| *** tellesnobrega_ has quit IRC | 20:32 | |
| *** bandarji has quit IRC | 20:50 | |
| *** crobertsrh is now known as _crobertsrh | 21:01 | |
| *** bandarji has joined #openstack-sahara | 21:05 | |
| *** bandarji has quit IRC | 21:12 | |
| *** alazarev has quit IRC | 21:23 | |
| *** alazarev has joined #openstack-sahara | 21:24 | |
| *** witlessb has quit IRC | 21:34 | |
| *** elmiko is now known as _elmiko | 21:37 | |
| *** ghenriks has joined #openstack-sahara | 22:04 | |
| *** bradd1 has joined #openstack-sahara | 22:08 | |
| *** julienvey has quit IRC | 22:30 | |
| *** akuznetsov has joined #openstack-sahara | 22:31 | |
| *** tosky has quit IRC | 22:42 | |
| *** shakayumi has quit IRC | 22:55 | |
| *** julienvey has joined #openstack-sahara | 23:04 | |
| *** julienvey has quit IRC | 23:09 | |
| *** bradd1 has quit IRC | 23:19 | |
| *** tmckay has left #openstack-sahara | 23:23 | |
| *** bandarji has joined #openstack-sahara | 23:36 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara-dashboard: Fixed help text for new cluster dialog https://review.openstack.org/97989 | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!