| jodah | Hash anyone discovered any hacks for learning which racks hosts reside on, as opposed to providing the topology files? | 00:10 |
|---|---|---|
| jodah | Hash->has | 00:10 |
| *** miqui_ has joined #openstack-sahara | 00:12 | |
| *** witlessb has quit IRC | 00:15 | |
| *** egafford has quit IRC | 00:16 | |
| *** rodrigod` is now known as rodrigods | 00:41 | |
| *** tellesnobrega_ has joined #openstack-sahara | 00:50 | |
| *** macjack has joined #openstack-sahara | 01:26 | |
| *** openstack has joined #openstack-sahara | 01:34 | |
| *** hdd_ has quit IRC | 01:50 | |
| *** tellesnobrega_ has quit IRC | 02:38 | |
| *** witlessb has joined #openstack-sahara | 02:56 | |
| *** witlessb has quit IRC | 03:01 | |
| *** hdd_ has joined #openstack-sahara | 03:01 | |
| *** Longgeek has joined #openstack-sahara | 03:08 | |
| openstackgerrit | lu huichun proposed openstack/sahara-specs: Add check service test in integration test https://review.openstack.org/147424 | 03:38 |
| *** Longgeek has quit IRC | 03:41 | |
| *** Longgeek has joined #openstack-sahara | 03:42 | |
| *** Longgeek has quit IRC | 04:05 | |
| *** Longgeek has joined #openstack-sahara | 04:07 | |
| *** macjack has quit IRC | 04:18 | |
| *** Networkn_ has quit IRC | 04:19 | |
| *** Networkn3rd has joined #openstack-sahara | 04:19 | |
| *** hdd_ has quit IRC | 04:44 | |
| *** hdd_ has joined #openstack-sahara | 05:15 | |
| *** saurabhs has left #openstack-sahara | 05:24 | |
| *** macjack has joined #openstack-sahara | 05:47 | |
| *** macjack has quit IRC | 06:21 | |
| *** macjack has joined #openstack-sahara | 06:32 | |
| *** hdd_ has quit IRC | 06:59 | |
| *** Longgeek has quit IRC | 07:18 | |
| *** chlong has quit IRC | 07:20 | |
| *** Longgeek has joined #openstack-sahara | 07:20 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-saharaclient: Remove cyclic dependency https://review.openstack.org/147771 | 07:41 |
| openstackgerrit | Jamie Lennox proposed openstack/python-saharaclient: Use requests-mock for mocking https://review.openstack.org/147772 | 07:41 |
| openstackgerrit | Jamie Lennox proposed openstack/python-saharaclient: Try and rework auth https://review.openstack.org/147773 | 07:41 |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Fixing log messages to avoid information duplication https://review.openstack.org/147504 | 07:59 |
| openstackgerrit | Vitaly Gridnev proposed openstack/python-saharaclient: Add ability to get events from saharaclient https://review.openstack.org/136005 | 08:33 |
| *** witlessb has joined #openstack-sahara | 08:51 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-saharaclient: Rework authentication https://review.openstack.org/147773 | 08:51 |
| *** coolsvap|afk is now known as coolsvap | 09:03 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-saharaclient: Rework authentication https://review.openstack.org/147773 | 09:03 |
| *** IvanBerezovskiy has joined #openstack-sahara | 09:07 | |
| *** Longgeek has quit IRC | 09:48 | |
| *** Longgeek has joined #openstack-sahara | 09:48 | |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Transient test job improvements https://review.openstack.org/147795 | 09:49 |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding hive support for vanilla 2.6 https://review.openstack.org/147796 | 09:51 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Fixed issues in docs https://review.openstack.org/147797 | 09:51 |
| *** Longgeek_ has joined #openstack-sahara | 09:52 | |
| *** Longgeek has quit IRC | 09:55 | |
| *** Longgeek_ has quit IRC | 09:56 | |
| *** Longgeek has joined #openstack-sahara | 09:56 | |
| *** macjack has quit IRC | 10:15 | |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Move Saharaclient tempest tests to Sahara https://review.openstack.org/143686 | 10:24 |
| *** Longgeek_ has joined #openstack-sahara | 10:25 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add integration test for Hive on vanilla2 https://review.openstack.org/143950 | 10:26 |
| *** Longgeek_ has quit IRC | 10:27 | |
| *** Longgeek_ has joined #openstack-sahara | 10:27 | |
| *** Longgeek has quit IRC | 10:28 | |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Move Saharaclient tempest tests to Sahara https://review.openstack.org/143686 | 10:28 |
| *** Longgeek has joined #openstack-sahara | 10:36 | |
| *** Longgeek_ has quit IRC | 10:38 | |
| *** tosky has joined #openstack-sahara | 11:06 | |
| openstackgerrit | Merged stackforge/sahara-ci-config: Transient test job improvements https://review.openstack.org/147795 | 11:16 |
| openstackgerrit | Merged stackforge/sahara-ci-config: Back ubuntu_cdh_latest image for CDH integration tests https://review.openstack.org/147222 | 11:16 |
| openstackgerrit | Merged stackforge/sahara-ci-config: devstack automation improvements https://review.openstack.org/147100 | 11:16 |
| *** sgotliv has joined #openstack-sahara | 11:31 | |
| *** tellesnobrega_ has joined #openstack-sahara | 11:33 | |
| *** Longgeek has quit IRC | 11:37 | |
| openstackgerrit | Sergey Kolekonov proposed stackforge/sahara-ci-config: Fix image name for transient clusters https://review.openstack.org/147814 | 11:45 |
| *** tellesnobrega_ has quit IRC | 11:52 | |
| openstackgerrit | Merged stackforge/sahara-ci-config: Fix image name for transient clusters https://review.openstack.org/147814 | 12:14 |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Move Saharaclient tempest tests to Sahara https://review.openstack.org/143686 | 12:15 |
| openstackgerrit | Merged openstack/sahara: Adding usage of "openstack.common.log" instead of "logging" https://review.openstack.org/146804 | 12:19 |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Move Saharaclient tempest tests to Sahara https://review.openstack.org/143686 | 12:22 |
| *** Longgeek has joined #openstack-sahara | 12:29 | |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Move Saharaclient tempest tests to Sahara https://review.openstack.org/143686 | 12:40 |
| openstackgerrit | Merged stackforge/sahara-ci-config: Update image names to latest https://review.openstack.org/141796 | 12:42 |
| openstackgerrit | Merged stackforge/sahara-ci-config: Fix old name for hdp plugins https://review.openstack.org/146812 | 12:49 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [WIP] Apply event log feature to HDP plugins https://review.openstack.org/147830 | 12:56 |
| *** _tmckay has quit IRC | 12:58 | |
| *** Longgeek has quit IRC | 13:04 | |
| *** tosky_ has joined #openstack-sahara | 13:05 | |
| *** tosky has quit IRC | 13:06 | |
| *** Longgeek has joined #openstack-sahara | 13:11 | |
| *** stannie42 has joined #openstack-sahara | 13:21 | |
| openstackgerrit | Sebastien Badia proposed stackforge/puppet-sahara: Migrate postgresql backend to use openstacklib::db::postgresql https://review.openstack.org/145511 | 13:24 |
| openstackgerrit | Sebastien Badia proposed stackforge/puppet-sahara: fixtures: Unpin puppetlabs-mysql module https://review.openstack.org/145510 | 13:24 |
| openstackgerrit | Sebastien Badia proposed stackforge/puppet-sahara: Rewrite puppet-sahara from scratch https://review.openstack.org/145509 | 13:24 |
| *** Networkn3rd has quit IRC | 13:35 | |
| *** hdd_ has joined #openstack-sahara | 13:38 | |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Fix image name for Vanilla 2.6 for transient https://review.openstack.org/147847 | 13:41 |
| *** ylobankov has quit IRC | 13:44 | |
| *** tosky_ has quit IRC | 13:45 | |
| *** ylobankov has joined #openstack-sahara | 13:47 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/python-saharaclient: Fix output of supported api versions https://review.openstack.org/147850 | 13:48 |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Devstack and nodepool improvements https://review.openstack.org/147855 | 13:55 |
| *** tmckay has joined #openstack-sahara | 13:58 | |
| *** tosky has joined #openstack-sahara | 14:03 | |
| openstackgerrit | Merged stackforge/sahara-ci-config: Fix image name for Vanilla 2.6 for transient https://review.openstack.org/147847 | 14:04 |
| *** egafford has joined #openstack-sahara | 14:07 | |
| *** hdd_ has quit IRC | 14:22 | |
| *** hdd_ has joined #openstack-sahara | 14:22 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara-specs: New integration tests for Sahara https://review.openstack.org/147219 | 14:40 |
| *** hdd_ has quit IRC | 14:47 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara-specs: New integration tests for Sahara https://review.openstack.org/147219 | 14:51 |
| *** coolsvap is now known as coolsvap|afk | 14:55 | |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Refactor MapR plugin for Sahara Implements: blueprint mapr-refactor Change-Id: Ib4385845a00aab2506dddccb9e6a6f3297a5622d https://review.openstack.org/147872 | 14:57 |
| *** hdd_ has joined #openstack-sahara | 15:15 | |
| *** Networkn3rd has joined #openstack-sahara | 15:37 | |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Refactor MapR plugin for Sahara https://review.openstack.org/147872 | 15:38 |
| openstackgerrit | Trevor McKay proposed openstack/sahara: Initial work on spark-swift integration https://review.openstack.org/146659 | 15:42 |
| *** IvanBerezovskiy has left #openstack-sahara | 16:01 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara: Initial work on spark-swift integration https://review.openstack.org/146659 | 16:23 |
| *** vDruids has quit IRC | 16:36 | |
| *** Networkn3rd has quit IRC | 16:38 | |
| *** Networkn3rd has joined #openstack-sahara | 16:38 | |
| *** Longgeek has quit IRC | 16:41 | |
| *** Networkn3rd has quit IRC | 16:43 | |
| *** Networkn3rd has joined #openstack-sahara | 16:44 | |
| *** Longgeek has joined #openstack-sahara | 16:45 | |
| *** tellesnobrega_ has joined #openstack-sahara | 16:46 | |
| *** Longgeek has quit IRC | 16:54 | |
| *** Longgeek has joined #openstack-sahara | 16:55 | |
| *** tellesnobrega_ has quit IRC | 16:59 | |
| *** DenisEgorenko has quit IRC | 17:05 | |
| jodah | Has anyone looked into alternative ways of dealing with data locality without having to provide topology files? | 17:18 |
| jodah | ...such as attempting to learn about rack locality via Nova/Neutron/etc? | 17:18 |
| elmiko | jodah: i think this may have been discussed at a summit, but i don't think i've seen any specs or blueprints | 17:27 |
| elmiko | SergeyLukjanov: ^^ | 17:28 |
| jodah | Aside from ideas specific to Sahara, I'm also curious if anyone has talked to other teams such as Nova to try and solve this more generally? | 17:31 |
| *** ghenriks has quit IRC | 17:31 | |
| elmiko | that's a good question. i feel like i've heard this talked about before, but i can't point to any specific sources. | 17:31 |
| elmiko | and i know you've brought it up | 17:32 |
| *** ghenriks has joined #openstack-sahara | 17:32 | |
| jodah | I actually just hit on this problem this week - I've been working on clustering in Trove (DB service) and the ability to place database nodes on the same rack is crucial - similar to Sahara's ability to place data nodes on the same rack. | 17:36 |
| elmiko | right, makes sense | 17:36 |
| jodah | This problem is pretty general and is going to keep popping up in OpenStack. I think the solution should be general as well, and probably belongs in Nova. I'd like to start that discussion but just wanted to check with you guys to see if anything has already been started. | 17:36 |
| elmiko | let me look a little deeper, 1 sec | 17:37 |
| elmiko | so, i can see we have a bunch of data locality options in the codebase. but, to be honest i haven't worked on those areas and i'm not 100% clear on how they work. | 17:38 |
| elmiko | they might just be options we pass on to hadoop/spark/etc | 17:39 |
| jodah | I haven't tried the stuff out either - just reading the docs and looking at the code briefly, Sahara requires that users describe which racks hosts are on | 17:39 |
| elmiko | i'd have to dig a bit deeper for a better explanation | 17:39 |
| elmiko | yea | 17:39 |
| elmiko | have you talked with the nova folks about this? | 17:39 |
| jodah | I plan to, but I wanted to start by learning from the teams who need this capability what they might have done | 17:40 |
| jodah | Pretty much all of the clustered PaaS services could use this - and some like Sahara really need it | 17:41 |
| elmiko | i'll try to ask some of the team about this, maybe i can come up with a better answer | 17:41 |
| elmiko | definitely | 17:41 |
| jodah | Ok. Ping me anytime today if you hear anything and hopefully my chat window will flash :) | 17:41 |
| elmiko | jodah: http://docs.openstack.org/developer/sahara/userdoc/features.html has a little more info | 17:42 |
| elmiko | but i gather you've seen that | 17:42 |
| jodah | Yea - I think that shows why native Nova support for rack aware VM placement is needed | 17:43 |
| jodah | ...the fact that Sahara had to build a workaround :) | 17:43 |
| elmiko | definitely | 17:43 |
| elmiko | also, try pinging SergeyLukjanov, i'm sure he has more info about this | 17:43 |
| jodah | SergeyLukjanov: Let me know if/when you're free to chat on this | 17:44 |
| *** Networkn3rd has quit IRC | 17:44 | |
| *** Networkn3rd has joined #openstack-sahara | 17:44 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Separate the codes of CDH5 and CDH5.3.0 https://review.openstack.org/147933 | 17:55 |
| *** shakamunyi has joined #openstack-sahara | 18:02 | |
| *** tellesnobrega_ has joined #openstack-sahara | 18:17 | |
| openstackgerrit | Merged openstack/sahara: Fixed issues in docs https://review.openstack.org/147797 | 18:22 |
| *** tellesnobrega_ has quit IRC | 18:43 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: Enable Spark jobs to access Swift URLs https://review.openstack.org/147949 | 18:44 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: Enable Spark jobs to access Swift URLs https://review.openstack.org/147949 | 18:45 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: Enable Spark jobs to access Swift URLs https://review.openstack.org/147949 | 18:46 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: Enable Spark jobs to access Swift URLs https://review.openstack.org/147949 | 18:48 |
| *** tellesnobrega_ has joined #openstack-sahara | 18:58 | |
| *** tosky has quit IRC | 19:12 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-extra: Add a SparkWrapper class https://review.openstack.org/147955 | 19:13 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-extra: Add a SparkWrapper class https://review.openstack.org/147955 | 19:15 |
| *** ghenriks has quit IRC | 19:33 | |
| *** ghenriks has joined #openstack-sahara | 19:35 | |
| *** tellesnobrega_ has quit IRC | 19:35 | |
| *** Longgeek has quit IRC | 20:42 | |
| *** Longgeek has joined #openstack-sahara | 20:43 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-image-elements: Add the hadoop-swift element to Spark images https://review.openstack.org/147985 | 20:44 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-image-elements: Add the swift_hadoop element to Spark images https://review.openstack.org/147985 | 20:49 |
| egafford | tmckay: Question: why is the HDFS_LIB_DIR default in 81-add-jar different than the diskimage-create.sh's export? There's probably a good reason; just curious. | 21:03 |
| *** Longgeek_ has joined #openstack-sahara | 21:06 | |
| *** Longgeek has quit IRC | 21:06 | |
| crobertsrh | SergeyLukjanov and others: Here is the etherpad of sahara-related CRs open in horizon. Feel free to tweak if you know of some that I haven't listed. https://etherpad.openstack.org/p/sahara-reviews-in-horizon | 21:06 |
| openstackgerrit | Michael McCune proposed openstack/sahara: Refactoring swift binary retrievers to allow context auth https://review.openstack.org/145960 | 21:06 |
| elmiko | crobertsrh: wow, didn't realize so many were open | 21:07 |
| crobertsrh | Some of them are open, but waiting for sahara api changes (template editing) | 21:07 |
| elmiko | ahh ok | 21:07 |
| crobertsrh | at least 4 with workflow -1 for one reason or another | 21:08 |
| tmckay | egafford, I was adding the element for Spark. The previous code had the element at /usr/share/hadoop/lib for all images -- Spark wants it at /usr/lib/hadoop | 21:14 |
| tmckay | So Spark is the only image that is now setting DIB_HDFS_LIB_DIR | 21:15 |
| tmckay | for all others, it will be unset, and so the original value will be used | 21:15 |
| *** tellesnobrega has quit IRC | 21:15 | |
| egafford | Ah, I see; you're adding an override to the pre-existent default. | 21:16 |
| tmckay | right | 21:16 |
| tmckay | kind of a pain, really | 21:17 |
| egafford | Testing DIB is time consuming, yeah. | 21:17 |
| egafford | The changes are usually tiny, but they take a whie. | 21:17 |
| egafford | /s/whie/while | 21:18 |
| tmckay | yeah, I made a spark image, that worked, checking a vanilla 1 just to be sure | 21:18 |
| tmckay | and a lot of the hadoop2 images don't use the Sahara element at all :) | 21:18 |
| egafford | It'd be kind of mystical if vanilla1 didn't still work. | 21:18 |
| tmckay | I'm just mounting with guestfish to check for the file in the right location | 21:19 |
| *** Longgeek_ has quit IRC | 21:22 | |
| *** tellesnobrega has joined #openstack-sahara | 21:28 | |
| *** tellesnobrega has quit IRC | 21:50 | |
| *** tellesnobrega has joined #openstack-sahara | 22:01 | |
| *** tmckay has quit IRC | 22:12 | |
| *** Networkn3rd has quit IRC | 22:31 | |
| *** hdd_ is now known as hdd | 22:38 | |
| *** shakamunyi has quit IRC | 22:40 | |
| *** crobertsrh is now known as _crobertsrh | 23:05 | |
| *** openstackgerrit has quit IRC | 23:19 | |
| *** openstackgerrit has joined #openstack-sahara | 23:19 | |
| *** egafford has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!