| *** esikache1 has joined #openstack-sahara | 00:10 | |
| *** esikache1 has quit IRC | 00:15 | |
| *** esikache1 has joined #openstack-sahara | 00:25 | |
| *** esikache1 has quit IRC | 00:29 | |
| *** masber has quit IRC | 00:32 | |
| *** masber has joined #openstack-sahara | 00:32 | |
| *** jeremyfreudberg has quit IRC | 00:39 | |
| *** jeremyfreudberg has joined #openstack-sahara | 00:55 | |
| *** stanchan has joined #openstack-sahara | 01:03 | |
| *** stanchan has quit IRC | 01:08 | |
| *** shuyingya has joined #openstack-sahara | 01:32 | |
| *** shuyingya has quit IRC | 01:32 | |
| *** shuyingya has joined #openstack-sahara | 01:33 | |
| *** shuyingya has quit IRC | 01:46 | |
| *** shuyingya has joined #openstack-sahara | 01:47 | |
| *** shuyingya has quit IRC | 02:01 | |
| *** jamielennox is now known as jamielennox|away | 02:04 | |
| *** jamielennox|away is now known as jamielennox | 02:15 | |
| *** jeremyfreudberg has quit IRC | 02:23 | |
| *** links has joined #openstack-sahara | 03:34 | |
| *** dave-mccowan has quit IRC | 04:26 | |
| *** pcaruana has joined #openstack-sahara | 04:31 | |
| *** aselius has quit IRC | 04:32 | |
| *** pcaruana has quit IRC | 04:57 | |
| *** jamielennox is now known as jamielennox|away | 05:12 | |
| *** Poornima has joined #openstack-sahara | 05:13 | |
| *** jamielennox|away is now known as jamielennox | 05:19 | |
| *** shuyingya has joined #openstack-sahara | 05:38 | |
| openstackgerrit | zhongshengping proposed openstack/puppet-sahara master: Update global requirements https://review.openstack.org/472071 | 05:56 |
|---|---|---|
| *** jamielennox is now known as jamielennox|away | 05:56 | |
| *** jamielennox|away is now known as jamielennox | 06:03 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara master: Updated from global requirements https://review.openstack.org/472096 | 06:11 |
| *** esikache1 has joined #openstack-sahara | 06:11 | |
| *** rcernin has joined #openstack-sahara | 06:15 | |
| *** esikache1 has quit IRC | 06:18 | |
| *** esikache1 has joined #openstack-sahara | 06:23 | |
| *** esikache1 has quit IRC | 06:28 | |
| *** shuyingya has quit IRC | 06:31 | |
| *** Poornima_K has joined #openstack-sahara | 06:35 | |
| *** Poornima has quit IRC | 06:36 | |
| *** shuyingy_ has joined #openstack-sahara | 06:38 | |
| *** pcaruana has joined #openstack-sahara | 06:57 | |
| *** Poornima_K has quit IRC | 07:04 | |
| *** tesseract has joined #openstack-sahara | 07:12 | |
| *** esikache1 has joined #openstack-sahara | 07:16 | |
| *** esikache1 has quit IRC | 07:20 | |
| *** esikache1 has joined #openstack-sahara | 07:26 | |
| *** esikache1 has quit IRC | 07:30 | |
| *** esikache1 has joined #openstack-sahara | 07:35 | |
| *** Poornima has joined #openstack-sahara | 07:40 | |
| *** esikache1 has quit IRC | 07:41 | |
| *** anshul has joined #openstack-sahara | 07:46 | |
| *** esikache1 has joined #openstack-sahara | 07:47 | |
| *** esikache1 has quit IRC | 07:51 | |
| *** tuanluong has joined #openstack-sahara | 07:57 | |
| *** esikache1 has joined #openstack-sahara | 08:07 | |
| *** esikache1 has quit IRC | 08:11 | |
| *** esikache1 has joined #openstack-sahara | 08:14 | |
| *** qwertyco has joined #openstack-sahara | 08:17 | |
| *** esikache1 has quit IRC | 08:19 | |
| *** links has quit IRC | 08:25 | |
| *** esikache1 has joined #openstack-sahara | 08:26 | |
| *** esikache1 has quit IRC | 08:35 | |
| *** tosky has joined #openstack-sahara | 08:38 | |
| *** links has joined #openstack-sahara | 08:38 | |
| *** shuyingy_ has quit IRC | 08:38 | |
| *** shuyingya has joined #openstack-sahara | 08:38 | |
| *** shuying__ has joined #openstack-sahara | 08:43 | |
| *** shuyingya has quit IRC | 08:47 | |
| *** Poornima has quit IRC | 08:56 | |
| *** Poornima has joined #openstack-sahara | 09:04 | |
| *** esikache1 has joined #openstack-sahara | 09:11 | |
| openstackgerrit | Merged openstack/sahara-image-elements master: Fixing spark and CDH refinements https://review.openstack.org/471690 | 09:12 |
| openstackgerrit | Merged openstack/sahara-image-elements master: Fix an outdated link https://review.openstack.org/469229 | 09:14 |
| *** esikache1 has quit IRC | 09:19 | |
| *** esikache1 has joined #openstack-sahara | 09:21 | |
| *** qwertyco has quit IRC | 09:23 | |
| *** esikache1 has quit IRC | 09:33 | |
| *** esikache1 has joined #openstack-sahara | 09:46 | |
| *** esikache1 has quit IRC | 09:52 | |
| *** esikache1 has joined #openstack-sahara | 09:54 | |
| *** esikache1 has quit IRC | 09:59 | |
| *** shuying__ has quit IRC | 10:10 | |
| *** shuyingy_ has joined #openstack-sahara | 10:10 | |
| *** esikache1 has joined #openstack-sahara | 10:15 | |
| *** esikache1 has quit IRC | 10:20 | |
| *** anshul has quit IRC | 10:36 | |
| *** anshul has joined #openstack-sahara | 10:36 | |
| *** shuying__ has joined #openstack-sahara | 11:00 | |
| *** shuyingy_ has quit IRC | 11:03 | |
| openstackgerrit | Merged openstack/sahara master: Updated from global requirements https://review.openstack.org/472096 | 11:07 |
| *** shuying__ has quit IRC | 11:24 | |
| *** shuyingy_ has joined #openstack-sahara | 11:24 | |
| *** pgadiya has joined #openstack-sahara | 11:30 | |
| *** pgadiya has quit IRC | 11:30 | |
| *** shuying__ has joined #openstack-sahara | 11:35 | |
| *** shuying__ has quit IRC | 11:36 | |
| *** shuyingy_ has quit IRC | 11:37 | |
| *** shuying__ has joined #openstack-sahara | 11:37 | |
| *** shuyingya has joined #openstack-sahara | 11:38 | |
| *** shuying__ has quit IRC | 11:41 | |
| openstackgerrit | Merged openstack/sahara master: Support cinder API version 3 https://review.openstack.org/465804 | 11:43 |
| *** esikache1 has joined #openstack-sahara | 11:44 | |
| *** esikache1 has quit IRC | 11:48 | |
| *** shuyingya has quit IRC | 11:49 | |
| *** shuyingya has joined #openstack-sahara | 11:50 | |
| *** shuying__ has joined #openstack-sahara | 11:51 | |
| *** abalutoiu has joined #openstack-sahara | 11:52 | |
| *** shuying__ has quit IRC | 11:53 | |
| *** shuyingy_ has joined #openstack-sahara | 11:53 | |
| *** shuyingya has quit IRC | 11:54 | |
| *** esikache1 has joined #openstack-sahara | 11:56 | |
| openstackgerrit | Merged openstack/sahara master: Raise better exception for Spark master validation https://review.openstack.org/453856 | 12:00 |
| *** shuyin___ has joined #openstack-sahara | 12:03 | |
| *** shuyin___ has quit IRC | 12:06 | |
| *** shuying__ has joined #openstack-sahara | 12:06 | |
| *** shuyingy_ has quit IRC | 12:06 | |
| *** esikache1 has quit IRC | 12:14 | |
| *** esikachev has joined #openstack-sahara | 12:14 | |
| *** tuanluong has quit IRC | 12:23 | |
| *** anshul has quit IRC | 12:25 | |
| *** shuyingy_ has joined #openstack-sahara | 12:26 | |
| *** shuying__ has quit IRC | 12:29 | |
| *** shuyingy_ has quit IRC | 12:34 | |
| *** shuyingya has joined #openstack-sahara | 12:35 | |
| *** shuyingya has quit IRC | 12:35 | |
| *** shuyingya has joined #openstack-sahara | 12:36 | |
| *** shuyingya has quit IRC | 12:37 | |
| *** anshul has joined #openstack-sahara | 12:37 | |
| *** shuyingya has joined #openstack-sahara | 12:37 | |
| *** shuyingya has quit IRC | 12:42 | |
| *** esikachev has quit IRC | 12:58 | |
| openstackgerrit | Merged openstack/python-saharaclient master: Updated from global requirements https://review.openstack.org/465373 | 12:59 |
| openstackgerrit | Merged openstack/python-saharaclient master: Fix html_last_updated_fmt for Python3 https://review.openstack.org/470692 | 12:59 |
| tosky | few nice commits just in time for P-2 | 13:01 |
| *** esikachev has joined #openstack-sahara | 13:02 | |
| *** esikachev has quit IRC | 13:07 | |
| *** esikachev has joined #openstack-sahara | 13:10 | |
| *** dave-mccowan has joined #openstack-sahara | 13:12 | |
| *** esikachev has quit IRC | 13:14 | |
| *** links has quit IRC | 13:15 | |
| *** lucasxu has joined #openstack-sahara | 13:16 | |
| *** esikachev has joined #openstack-sahara | 13:24 | |
| *** esikachev has quit IRC | 13:28 | |
| *** egafford has joined #openstack-sahara | 13:29 | |
| *** esikachev has joined #openstack-sahara | 13:33 | |
| *** esikachev has quit IRC | 13:37 | |
| *** esikachev has joined #openstack-sahara | 13:38 | |
| *** lucasxu has quit IRC | 13:38 | |
| *** lucasxu has joined #openstack-sahara | 13:40 | |
| *** esikachev has quit IRC | 13:43 | |
| *** Poornima has quit IRC | 13:47 | |
| *** lucasxu has quit IRC | 13:51 | |
| *** lucasxu has joined #openstack-sahara | 14:02 | |
| *** chlong has joined #openstack-sahara | 14:02 | |
| *** mgoddard has joined #openstack-sahara | 14:03 | |
| mgoddard | hi all, I'd like to try to get OSU's high performance RDMA-enabled spark (http://hibd.cse.ohio-state.edu/#spark) running under sahara | 14:05 |
| mgoddard | I can see a number of issues in the way and would like to try to classify which will be easiest to tackle | 14:05 |
| mgoddard | 1. the sahara spark plugin only supports ubuntu. Our experience is largely with centos/rhel/fedora, and we have recipes for installing the fiddly mellanox OFED RDMA dependencies on RPM distros only. Does anyone know why spark is not supported on other distros? | 14:07 |
| mgoddard | 2. the sahara spark plugin as of ocata supports spark 1.31. and 1.6.0. Ideally we'd like to use spark 2.1.0 as the latest HiBD packages are based on this. Has anyone looked into what would be required to support spark 2.1.0 on sahara? | 14:09 |
| mgoddard | I can't think of a 3 currently but I'm sure there is one :) | 14:09 |
| mgoddard | any pointers would be much appreciated | 14:10 |
| tosky | mgoddard: you can use spark under CDH | 14:13 |
| mgoddard | tosky: ok. Is CDH open or does it require a license? | 14:16 |
| *** tenobreg has joined #openstack-sahara | 14:16 | |
| tenobreg | vgrivned, on the patch https://review.openstack.org/#/c/471371/ | 14:17 |
| tenobreg | where do you the problem with heat and floating ips? | 14:17 |
| mgoddard | tosky: looks like CDH will run on RPM distros but is still limited to spark 1.6.0 in CDH 5.9 | 14:19 |
| *** jeremyfreudberg has joined #openstack-sahara | 14:20 | |
| *** jeremyfreudberg has left #openstack-sahara | 14:20 | |
| *** jeremyfreudberg has joined #openstack-sahara | 14:20 | |
| tosky | mgoddard: it's what provided by CDH | 14:21 |
| mgoddard | tosky: sure. thanks for the help | 14:22 |
| tosky | mgoddard: not too expert about CDH; the manager (more or less same role as ambari) is not free/libre software, but the basic version can be used gratis | 14:23 |
| tosky | see https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_faqs.html#id_iyv_2qr_qj__d29624e81 | 14:23 |
| tosky | about spark, centos and 2.1, I think that tenobreg can answer better than me | 14:23 |
| mgoddard | tosky: that's useful, thanks | 14:24 |
| tenobreg | mgoddard, can I be of any help? | 14:24 |
| mgoddard | tenobreg: do you know the reasons why the spark plugin is only supported on ubuntu | 14:26 |
| *** shuyingya has joined #openstack-sahara | 14:26 | |
| mgoddard | tenobreg: and also if you have any idea what might be involved in making it support spark 2.1.0 that would be useful | 14:26 |
| jeremyfreudberg | mgoddard, the reason for ubuntu only is that (at least at the time we created the Spark plugin) the CDH fork of HDFS we use for the Spark plugin is/was only available for Ubuntu | 14:33 |
| jeremyfreudberg | and spark 2.1.0 support is planned for pike release | 14:35 |
| mgoddard | jeremyfreudberg: thanks, that's useful. why was the CDH fork used over vanilla HDFS? | 14:35 |
| mgoddard | jeremyfreudberg: is there a blueprint/spec for spark 2.1.0? | 14:35 |
| mgoddard | jeremyfreudberg: looks like the cloudera plugin supports centos & centos7 | 14:37 |
| mgoddard | jeremyfreudberg: perhaps that's a recent change | 14:37 |
| jeremyfreudberg | mgoddard, let me start with the easy questions... ;) | 14:39 |
| *** shuyingya has quit IRC | 14:39 | |
| mgoddard | jeremyfreudberg: sorry, one step at a time :) | 14:40 |
| *** shuyingya has joined #openstack-sahara | 14:40 | |
| jeremyfreudberg | ok, I haven't seen an actual blueprint for tracking spark 2.1.0 status, but I can link you some gerrit reviews surrounding the upgrade | 14:41 |
| jeremyfreudberg | and I am trying to find the actual reasons for using Cloudera HDFS on Spark plugin, I can't find the exact reason, but it has something to do with early versions of Spark | 14:43 |
| *** aselius has joined #openstack-sahara | 14:46 | |
| jeremyfreudberg | So, if you look at how we build the images for Sahara, both the Spark and CDH plugins use some element called "hadoop-cloudera", so I am guessing the issue with non-Ubuntu OSes is no longer relevant | 14:46 |
| *** shuyingya has quit IRC | 14:46 | |
| *** shuyingya has joined #openstack-sahara | 14:47 | |
| mgoddard | jeremyfreudberg: cool. I'll hack some changes into sahara-image-create and try building spark for centos7 | 14:47 |
| *** mgoddard has quit IRC | 14:49 | |
| *** mgoddard has joined #openstack-sahara | 14:49 | |
| jeremyfreudberg | mgoddard, cool, let us know if it works, it would be nice if we could introduce it officially | 14:49 |
| mgoddard | jeremyfreudberg: absolutely. is every permutation tested? | 14:50 |
| *** shuyingya has quit IRC | 14:51 | |
| tenobreg | mgoddard, I don't think spark on centos7 is difficult to make, we might just not got around to it yet | 14:52 |
| *** abalutoiu has quit IRC | 14:53 | |
| tenobreg | we plan to add storm as well on centos, but just haven't had the time to do so | 14:53 |
| tenobreg | mgoddard, you plan to use spark on its on or only with CDH? | 14:53 |
| *** logan- has joined #openstack-sahara | 14:53 | |
| jeremyfreudberg | exactly, I was going to bring up Storm too, I don't think it would be an issue either | 14:53 |
| jeremyfreudberg | (which also reminds me, tenobreg, I should try to get spark on vanilla upgraded too when I upgrade vanilla plugin) | 14:54 |
| tenobreg | jeremyfreudberg, sure, we should try to keep the same version all around where we can | 14:55 |
| tenobreg | CDH, Ambari and MapR is not really up to us, we have to respect what they support, but vanilla we can and should keep with the current version of the standalone plugins | 14:55 |
| mgoddard | tenobreg: I'm going to continue with the standalone spark plugin | 14:58 |
| tenobreg | mgoddard, great. Let me know if we need anything or help at all | 14:58 |
| mgoddard | tenobreg: thanks, will do. | 14:59 |
| mgoddard | thanks for the help tenobreg, jeremyfreudberg and tosky. | 15:01 |
| tenobreg | mgoddard, no problem at all | 15:01 |
| jeremyfreudberg | mgoddard, no problem | 15:01 |
| jeremyfreudberg | hey, tenobreg, my coworker lucasxu and I are trying to launch storm cluster, actually we cannot find the correct way to assign the node processes for that plugin, do you know? | 15:09 |
| jeremyfreudberg | (I see in sahara-tests we do a separate node for nimbus, supervisor, and zookeeper, but are 3 separate instances really needed?) | 15:10 |
| lucasxu | we looked through this page: https://docs.openstack.org/developer/sahara/userdoc/plugins.html | 15:10 |
| *** rcernin has quit IRC | 15:16 | |
| *** stanchan has joined #openstack-sahara | 15:17 | |
| *** anshul has quit IRC | 15:36 | |
| *** esikachev has joined #openstack-sahara | 16:22 | |
| *** esikachev has quit IRC | 16:26 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara master: Updated from global requirements https://review.openstack.org/472329 | 16:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara-dashboard master: Updated from global requirements https://review.openstack.org/472330 | 16:29 |
| *** esikachev has joined #openstack-sahara | 16:29 | |
| *** tesseract has quit IRC | 16:32 | |
| *** jeremyfreudberg has quit IRC | 16:33 | |
| *** jeremyfreudberg has joined #openstack-sahara | 16:33 | |
| *** esikachev has quit IRC | 16:38 | |
| tosky | I won't be likely around for the meeting, but again, great group work fixing the issues! | 16:38 |
| jeremyfreudberg | thanks tosky! | 16:41 |
| *** pcaruana has quit IRC | 17:00 | |
| *** esikachev has joined #openstack-sahara | 17:06 | |
| *** lucasxu has quit IRC | 17:14 | |
| tenobreg | jeremyfreudberg, I usually do them with 3 nodes | 17:15 |
| tenobreg | but I think if you get a large flavor you can put nimbus and zookeeper together | 17:15 |
| jeremyfreudberg | yeah i was just reading nimbus and zookeeper are the ones to put together | 17:15 |
| jeremyfreudberg | (I personally find the name "supervisor" very confusing since that is for the workers) | 17:15 |
| jeremyfreudberg | thanks, tenobreg | 17:16 |
| *** esikachev has quit IRC | 17:16 | |
| tenobreg | I think I put them apart because I was using small flavors, and with small one zookeeper was having some issues | 17:16 |
| tenobreg | can't really remember right now | 17:16 |
| tenobreg | its been a while | 17:16 |
| tenobreg | let me know if you can do with two and the flavors sizes, I can update documentation | 17:16 |
| jeremyfreudberg | (well, me and lucasxu could not even find the storm doc) | 17:17 |
| jeremyfreudberg | https://github.com/openstack/sahara/search?l=reStructuredText&q=storm&type=&utf8=%E2%9C%93 says we don't have any doc on storm plugin | 17:18 |
| tenobreg | jeremyfreudberg, that is good to know. I should probably write that up | 17:18 |
| tenobreg | jeremyfreudberg, thanks | 17:19 |
| jeremyfreudberg | no rush with the docs, i know we are all working on a million things right now | 17:20 |
| jeremyfreudberg | but it would be nice | 17:20 |
| openstackgerrit | Merged openstack/sahara-dashboard master: Updated from global requirements https://review.openstack.org/472330 | 17:28 |
| tenobreg | yes | 17:30 |
| *** lucasxu has joined #openstack-sahara | 18:00 | |
| jeremyfreudberg | meeting? | 18:01 |
| tenobreg | iwonka, we starting our meeting at #openstack-meeting-alt | 18:04 |
| tenobreg | if you want to join | 18:04 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara master: Grenade fix https://review.openstack.org/471371 | 18:12 |
| *** tellesnobrega has joined #openstack-sahara | 18:49 | |
| *** mgoddard has quit IRC | 18:50 | |
| *** tenobreg has quit IRC | 18:52 | |
| *** tellesnobrega is now known as tenobreg | 18:52 | |
| *** jeremyfreudberg has quit IRC | 18:55 | |
| *** dave-mccowan has quit IRC | 18:58 | |
| *** dave-mccowan has joined #openstack-sahara | 19:02 | |
| openstackgerrit | Merged openstack/sahara master: Updated from global requirements https://review.openstack.org/472329 | 19:08 |
| *** mgoddard has joined #openstack-sahara | 19:10 | |
| *** esikachev has joined #openstack-sahara | 20:40 | |
| *** esikachev has quit IRC | 20:45 | |
| *** esikachev has joined #openstack-sahara | 20:48 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara master: Grenade fix https://review.openstack.org/471371 | 20:52 |
| *** esikachev has quit IRC | 20:54 | |
| *** mgoddard has quit IRC | 20:59 | |
| *** lucasxu has quit IRC | 21:11 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara master: Grenade fix https://review.openstack.org/471371 | 21:18 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-sahara master: Bump version for Pike M3 Dev https://review.openstack.org/472440 | 21:34 |
| *** mgoddard has joined #openstack-sahara | 21:50 | |
| *** chlong has quit IRC | 21:53 | |
| *** mgoddard has quit IRC | 21:58 | |
| *** dave-mccowan has quit IRC | 22:05 | |
| *** tosky has quit IRC | 22:18 | |
| openstackgerrit | Merged openstack/sahara master: Changing reconcile to test_only https://review.openstack.org/468956 | 22:21 |
| openstackgerrit | Merged openstack/puppet-sahara master: Update global requirements https://review.openstack.org/472071 | 22:44 |
| *** esikachev has joined #openstack-sahara | 22:48 | |
| *** esikachev has quit IRC | 22:52 | |
| *** jamielennox is now known as jamielennox|away | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!