| *** links has joined #openstack-sahara | 01:33 | |
| *** brad[] has joined #openstack-sahara | 01:55 | |
| *** rcernin has quit IRC | 02:52 | |
| *** rcernin has joined #openstack-sahara | 03:10 | |
| *** rcernin has quit IRC | 03:55 | |
| *** rcernin has joined #openstack-sahara | 04:14 | |
| *** prasen has quit IRC | 04:22 | |
| *** pgadiya has joined #openstack-sahara | 04:40 | |
| *** pgadiya has quit IRC | 04:40 | |
| *** prasen has joined #openstack-sahara | 04:55 | |
| *** links has quit IRC | 05:32 | |
| *** links has joined #openstack-sahara | 05:56 | |
| *** zhuli__ has quit IRC | 06:49 | |
| *** Bhujay has joined #openstack-sahara | 07:03 | |
| *** zhuli__ has joined #openstack-sahara | 07:04 | |
| *** tesseract has joined #openstack-sahara | 07:11 | |
| *** whooliganface has joined #openstack-sahara | 07:17 | |
| *** links has quit IRC | 07:21 | |
| *** whooligan has quit IRC | 07:21 | |
| *** links has joined #openstack-sahara | 07:21 | |
| *** pcaruana has joined #openstack-sahara | 07:22 | |
| *** tosky has joined #openstack-sahara | 07:33 | |
| *** links has quit IRC | 07:42 | |
| *** rcernin has quit IRC | 07:47 | |
| *** links has joined #openstack-sahara | 07:55 | |
| *** Bhujay has quit IRC | 08:23 | |
| *** Bhujay has joined #openstack-sahara | 09:14 | |
| openstackgerrit | sunqingliang proposed openstack/sahara-dashboard master: add a link to release notes in README file https://review.openstack.org/577104 | 09:21 |
|---|---|---|
| *** rcernin has joined #openstack-sahara | 09:39 | |
| *** rcernin has quit IRC | 09:53 | |
| *** rcernin has joined #openstack-sahara | 10:55 | |
| *** rcernin has quit IRC | 11:19 | |
| *** rcernin has joined #openstack-sahara | 11:39 | |
| *** rcernin has quit IRC | 11:44 | |
| openstackgerrit | Merged openstack/sahara-dashboard master: add a link to release notes in README file https://review.openstack.org/577104 | 11:46 |
| *** prasen has quit IRC | 12:16 | |
| *** links has quit IRC | 12:43 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: DNM Experimental scenario job based on RadosGW (not Swift) https://review.openstack.org/576929 | 13:00 |
| *** zchkun has joined #openstack-sahara | 13:47 | |
| *** jeremyfreudberg has joined #openstack-sahara | 13:56 | |
| *** links has joined #openstack-sahara | 14:17 | |
| *** links has quit IRC | 14:23 | |
| openstackgerrit | wangqi proposed openstack/puppet-sahara master: add release note link in readme https://review.openstack.org/577201 | 14:50 |
| *** pcaruana has quit IRC | 14:56 | |
| tellesnobrega | jeremyfreudberg, tosky please continue | 15:00 |
| tosky | sooo | 15:00 |
| tosky | we were at (more or less) cloudera jar (and the S3 support that they advertise) is not enough | 15:01 |
| jeremyfreudberg | yes, let me collect my thoughts for a minute | 15:01 |
| jeremyfreudberg | so, let's talk about one issue, which is the problems with hadoop-aws.jar: | 15:02 |
| jeremyfreudberg | background knowledge, there are two ways of addressing the real S3 hosted by amazon | 15:02 |
| jeremyfreudberg | bucket.host.com | 15:03 |
| jeremyfreudberg | or host.com/bucket | 15:03 |
| jeremyfreudberg | radosgw and all other object store solutions which are not amazon itself only support host.com/bucket | 15:03 |
| jeremyfreudberg | however, in hadoop-aws.jar bundled with hadoop before 2.8, it defaults to bucket.host.com and cannot be configured (it will fallback to host.com/bucket only if the endpoint is an ip address and not a domain name) | 15:04 |
| jeremyfreudberg | so, we need to "backport" the support for host.com/bucket to hadoop-aws for hadoop 2.6 and 2.7 | 15:05 |
| jeremyfreudberg | every plugin except Vanilla 2.8.2 uses this old jar | 15:06 |
| tosky | I see | 15:06 |
| tosky | and I guess that vendor plugins didn't patch this themselves | 15:07 |
| jeremyfreudberg | ooh, well that's interesting, i didn't think to check if the vendors did that | 15:07 |
| tosky | given that "speed" that they move, I would not bet too much | 15:07 |
| tosky | but still | 15:07 |
| tosky | worth checking | 15:07 |
| tellesnobrega | yes | 15:08 |
| tellesnobrega | I'm heading out for lunch, I will read the docs when I'm back | 15:09 |
| tellesnobrega | s/docs/logs | 15:10 |
| jeremyfreudberg | no problem, enjoy the lunch | 15:10 |
| tellesnobrega | thanks | 15:10 |
| tellesnobrega | just an update, storm 1.2.0 image just created | 15:10 |
| tosky | nice! | 15:10 |
| tellesnobrega | need to test the cluster | 15:10 |
| tellesnobrega | but that is one step closer | 15:10 |
| tellesnobrega | please continue with the hard issues | 15:11 |
| tosky | jeremyfreudberg: another question: if I remember correctly, you mentioned that we use s3a, but isn't that only for Hadoop 2.7+? | 15:11 |
| jeremyfreudberg | hadoop 2.6 was the initial implementation | 15:11 |
| tosky | but not s3a, do we use the other one when using the older hadoop? | 15:14 |
| jeremyfreudberg | no, 2.6 has s3a still, i just peeked in the upstream and cdh jars | 15:15 |
| jeremyfreudberg | maybe it's not very good, but its there | 15:15 |
| jeremyfreudberg | tosky: the cdh jar has the necesssary patch already, from my inspection! | 15:15 |
| tosky | I see | 15:15 |
| tosky | oh! | 15:15 |
| tosky | nuce | 15:16 |
| tosky | nice* | 15:16 |
| jeremyfreudberg | i will look in the other vendor jars later | 15:17 |
| jeremyfreudberg | let me just touch on one other issue now | 15:17 |
| jeremyfreudberg | so, you'll recall i just said, the hadoop-aws.jar in hadoop>=2.8 fixes the bucket-in-path problem | 15:17 |
| jeremyfreudberg | BUT | 15:17 |
| jeremyfreudberg | that creates a problem for spark | 15:18 |
| jeremyfreudberg | on vanilla | 15:18 |
| jeremyfreudberg | because spark manages its own installation of the hadoop common libraries | 15:19 |
| jeremyfreudberg | and when running a spark job hadoop-aws will look in Spark's installation of those | 15:19 |
| jeremyfreudberg | the problem is, spark is distrubited only with hadoop common 2.6 or 2.7 | 15:19 |
| jeremyfreudberg | see: http://archive.apache.org/dist/spark/spark-2.2.0/ | 15:19 |
| tosky | so another patch? | 15:20 |
| jeremyfreudberg | yes | 15:20 |
| jeremyfreudberg | because hadoop aws 2.8, which is what vanilla 2.8, provides, is incompatible with hadoop-common 2.7 | 15:20 |
| jeremyfreudberg | due to a one word change in the code | 15:20 |
| jeremyfreudberg | java private -> public | 15:20 |
| tosky | but can't spark 2.2/2.3 work with hadoop 2.8? | 15:21 |
| tosky | are there other incompatibilities? Just untested by ustream? | 15:21 |
| jeremyfreudberg | it should work, but nothing is there upstream | 15:21 |
| tosky | at least someone tried: https://gist.github.com/momijiame/fefb9831e9f032ef264d8d517df57cb4 | 15:21 |
| jeremyfreudberg | it is possible to download spark "without hadoop", and copy in hadoop 2.8 ourselves | 15:22 |
| tosky | I know someone who is working with spark a lot | 15:22 |
| * tosky summons elmiko as usual | 15:22 | |
| jeremyfreudberg | indeed | 15:22 |
| tosky | any other issues that we can discuss about before the summoning triggers properly? :) | 15:24 |
| jeremyfreudberg | just that testing all this stuff is going to be a pain | 15:24 |
| jeremyfreudberg | we have so many job types, and so many plugins | 15:24 |
| tosky | oh, ok, that's the "usual" complicated part | 15:25 |
| jeremyfreudberg | and setting up the CI on my hardware is so far down my backlog... | 15:25 |
| tosky | depending on how difficult is to patch and fix S3, it may go up :) | 15:25 |
| tosky | let's see | 15:25 |
| jeremyfreudberg | that's a bit of tangent | 15:26 |
| jeremyfreudberg | oh, i remember what the other problem is now | 15:27 |
| jeremyfreudberg | or not "the other", "another" | 15:27 |
| jeremyfreudberg | we will still have oozie classpath problems on mapr and hdp | 15:28 |
| jeremyfreudberg | actually, we might not | 15:28 |
| jeremyfreudberg | depends how dumb oozie is and how smart those plugins are | 15:29 |
| jeremyfreudberg | regardless, that's enough of a reason to maybe push those plugins till later | 15:29 |
| *** Bhujay has quit IRC | 15:29 | |
| tosky | uh, what was the oozie classpath issue? | 15:30 |
| jeremyfreudberg | oozie has trouble finding hadoop-aws | 15:30 |
| jeremyfreudberg | i had actually addressed this in my first s3_hadoop patch, but it was totally lost in the s3_hadoop revision | 15:31 |
| elmiko | hello | 15:31 |
| jeremyfreudberg | basically oozie does not care what the hadoop classpath really is | 15:31 |
| tosky | oki | 15:31 |
| jeremyfreudberg | and has its own ideas | 15:31 |
| tosky | elmiko: hi! Do you (or one of your spark friends :) know if Spark (any latest version) can work with Hadoop 2.8 | 15:31 |
| tosky | despite being shipped only with Hadoop 2.6 and 2.7? | 15:31 |
| elmiko | i _think_ it can, but you have to load the 2.8 jars manually. someone was just looking at this, let me ask him | 15:32 |
| tosky | I don't care too much how, as long as it's possible (without patching stuff) | 15:32 |
| tosky | thanks! | 15:32 |
| jeremyfreudberg | yes, i just care if someone has already tested it with success | 15:33 |
| elmiko | not successfully yet | 15:33 |
| elmiko | someone in our chat is working on it, but they have not been able to make it work yet | 15:33 |
| tosky | uh, interesting | 15:33 |
| elmiko | so, it's /theoretically/ possible | 15:34 |
| jeremyfreudberg | elmiko: okay ,good to know | 15:34 |
| elmiko | i can keep you all updated when i hear more | 15:34 |
| elmiko | it might require a recompile of spark though to use the new dependencies | 15:34 |
| tosky | sure, thanks, it would be really useful | 15:34 |
| jeremyfreudberg | yes: thanks elmiko! | 15:34 |
| tosky | so not a simple drop-in? | 15:34 |
| tosky | even without using new features | 15:34 |
| elmiko | i'm not sure about the full extent, it sounded like a simple drop-in but then the person working on it ran into trouble | 15:35 |
| jeremyfreudberg | tosky: it looks like hdp might have the patch too, but i think i'll have to actually boot an image to be sure | 15:44 |
| tosky | ack | 15:45 |
| tosky | still, good news | 15:45 |
| tosky | which version of HDP? Or Ambari? Or both? | 15:45 |
| jeremyfreudberg | i did a spot check on a few of these: http://repo.hortonworks.com/content/repositories/releases/org/apache/hadoop/hadoop-aws/ | 15:46 |
| *** zchkun has quit IRC | 15:54 | |
| *** zchkun has joined #openstack-sahara | 16:01 | |
| *** tesseract has quit IRC | 16:09 | |
| *** jeremyfreudberg has quit IRC | 16:21 | |
| *** raopajay has joined #openstack-sahara | 16:45 | |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: DNM Experimental scenario job based on RadosGW (not Swift) https://review.openstack.org/576929 | 16:50 |
| tellesnobrega | just read the logs | 17:04 |
| tellesnobrega | hdp has the patches to work with s3? is that correct? spark may be a pain | 17:05 |
| tellesnobrega | did I get it all correct? | 17:05 |
| tosky | and cdh may have the patches too | 17:05 |
| tellesnobrega | cool | 17:06 |
| tellesnobrega | mapr issue is only with oozie? | 17:06 |
| *** jeremyfreudberg has joined #openstack-sahara | 17:23 | |
| jeremyfreudberg | mapr remains a mystery | 17:24 |
| jeremyfreudberg | tosky, actually, for spark, forget about hadoop 2.8 | 17:24 |
| jeremyfreudberg | i can just patch the hadoop 2.7.3 aws jar, instead patching hadoop-common | 17:24 |
| jeremyfreudberg | what i mean is, the "hadoop2.7" distro here http://archive.apache.org/dist/spark/spark-2.2.0/ is 2.7.3 | 17:25 |
| tosky | I see | 17:26 |
| jeremyfreudberg | probably somebody (tosky) will think, wouldn't it be nice with we had hadoop 2.8 in spark anyway, to match the vanilla version | 17:28 |
| jeremyfreudberg | s/if/with | 17:28 |
| jeremyfreudberg | argh | 17:28 |
| jeremyfreudberg | s/with/if | 17:28 |
| tosky | we have an active watch on someone working on that (thanks to elmiko :) | 17:30 |
| jeremyfreudberg | here's an unrelated question | 17:31 |
| jeremyfreudberg | if i did one of those "Submitted Together" things on gerrit, what's the best way to edit the earlier commit? | 17:31 |
| tellesnobrega | I think you need to update the first and update the second all together | 17:32 |
| tosky | multiple patches? Heavy use of git interactive rebase | 17:32 |
| jeremyfreudberg | yeah, i guess it's interactive rebase | 17:33 |
| tosky | depending on how the fix touches the later patches, you can simply apply the fix on top and the rebase interactivly changing the order and merging two patches, | 17:33 |
| tosky | or interactivly rebase by stopping on the first patch and manually editing it | 17:33 |
| tosky | etc | 17:33 |
| jeremyfreudberg | tosky: the patch series here is the keystone thing | 17:33 |
| tosky | the fun part of git, where you can shot yourself in the foot and lose your work :D | 17:34 |
| elmiko | tosky: i'll poke the person more on monday | 18:58 |
| *** jeremyfreudberg has quit IRC | 19:19 | |
| tosky | thanks! | 19:22 |
| openstackgerrit | Luigi Toscano proposed openstack/sahara-tests master: DNM Experimental scenario job based on RadosGW (not Swift) https://review.openstack.org/576929 | 22:16 |
| *** rcernin has joined #openstack-sahara | 22:26 | |
| *** tosky has quit IRC | 23:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!