| *** IlyaE has joined #openstack-sahara | 00:02 | |
| *** shakamunyi has joined #openstack-sahara | 00:50 | |
| *** shakamunyi has quit IRC | 00:54 | |
| *** IlyaE has quit IRC | 01:24 | |
| *** IlyaE has joined #openstack-sahara | 02:22 | |
| *** IlyaE has quit IRC | 02:41 | |
| *** zhiyan_ is now known as zhiyan | 02:47 | |
| *** shakamunyi has joined #openstack-sahara | 02:51 | |
| *** shakamunyi has quit IRC | 02:56 | |
| *** RockKuo_Office has joined #openstack-sahara | 03:12 | |
| *** IlyaE has joined #openstack-sahara | 03:45 | |
| *** shakamunyi has joined #openstack-sahara | 03:52 | |
| *** shakamunyi has quit IRC | 03:57 | |
| *** martsen_ has joined #openstack-sahara | 04:35 | |
| *** shakamunyi has joined #openstack-sahara | 04:53 | |
| *** shakamunyi has quit IRC | 04:57 | |
| *** akuznetsov has joined #openstack-sahara | 05:05 | |
| *** IlyaE has quit IRC | 05:13 | |
| *** IlyaE has joined #openstack-sahara | 05:20 | |
| *** zhiyan is now known as zhiyan_ | 05:26 | |
| *** IlyaE has quit IRC | 05:31 | |
| *** zhiyan_ is now known as zhiyan | 05:33 | |
| *** martsen_ has quit IRC | 05:42 | |
| *** shakamunyi has joined #openstack-sahara | 05:53 | |
| *** shakamunyi has quit IRC | 05:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/90649 | 06:07 |
|---|---|---|
| *** akuznetsov has quit IRC | 06:10 | |
| *** akuznetsov has joined #openstack-sahara | 06:32 | |
| *** IlyaE has joined #openstack-sahara | 06:47 | |
| *** skolekonov has joined #openstack-sahara | 06:51 | |
| *** shakamunyi has joined #openstack-sahara | 06:54 | |
| *** shakamunyi has quit IRC | 06:58 | |
| *** shakamunyi has joined #openstack-sahara | 07:55 | |
| *** witlessb has joined #openstack-sahara | 07:59 | |
| *** shakamunyi has quit IRC | 08:00 | |
| *** IlyaE has quit IRC | 08:05 | |
| *** zhiyan is now known as zhiyan_ | 08:11 | |
| *** zhiyan_ is now known as zhiyan | 08:16 | |
| *** shakamunyi has joined #openstack-sahara | 08:56 | |
| *** shakamunyi has quit IRC | 09:01 | |
| *** akuznetsov has quit IRC | 09:42 | |
| *** akuznetsov has joined #openstack-sahara | 09:49 | |
| *** shakamunyi has joined #openstack-sahara | 09:57 | |
| *** tosky has joined #openstack-sahara | 10:00 | |
| *** shakamunyi has quit IRC | 10:01 | |
| *** zhiyan is now known as zhiyan_ | 10:41 | |
| *** shakamunyi has joined #openstack-sahara | 10:57 | |
| *** shakamunyi has quit IRC | 11:02 | |
| *** RockKuo_Office has quit IRC | 11:27 | |
| *** mattf is now known as _mattf | 11:46 | |
| *** shakamunyi has joined #openstack-sahara | 11:58 | |
| *** shakamunyi has quit IRC | 12:04 | |
| openstackgerrit | A change was merged to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/90649 | 12:26 |
| *** _mattf is now known as mattf | 12:29 | |
| *** ErikB1 has joined #openstack-sahara | 12:53 | |
| *** zhiyan_ is now known as zhiyan | 12:56 | |
| *** shakamunyi has joined #openstack-sahara | 12:59 | |
| openstackgerrit | Yaroslav Lobankov proposed a change to openstack/sahara: [IT] More coverage of EDP in tests https://review.openstack.org/90717 | 13:01 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Add sahara-all binary https://review.openstack.org/90720 | 13:03 |
| *** shakamunyi has quit IRC | 13:04 | |
| mattf | dmitryme, ^^ smart, my first objection to the api/engine split was going to be on the lines of complexity (1 service -> 2 services + message bus), my second objection was going to be along the lines of "when does a deployment need a separate api & engine" -- having a sahara-all eases my concerns | 13:06 |
| *** elmiko has joined #openstack-sahara | 13:07 | |
| mattf | that said, the bp for the api/engine split needs to contain criteria for when the split is necessary -- at what level of cluster scale, openstack scale, network layout, multi-tenancy do we need to stop using sahara-all and start using sahara-api + n sahara-engines | 13:07 |
| dmitryme | mattf: the idea to add sahara-all alias came from SergeyLukjanov | 13:17 |
| mattf | dmitryme, what's the criteria where a sahara user would want to split the api/engine? | 13:18 |
| dmitryme | regarding the rationale on why to split sahara into api and engine, I see the main gain in splitting user-interacting part from provisioning one. That way whatever happens during provisioning, the API and UI will be responsive. The monolithic sahara-all might get stuck on provisioning and stop responding to the user's requests, or do this very slowly. | 13:20 |
| mattf | i'm 100% with you there, but doing the split adds deployment complexity. so i'd like to know when it is advised to do the split. | 13:21 |
| mattf | for instance, we can define "very slowly" to mean it takes > 5 minutes to start a 32 instance cluster | 13:22 |
| mattf | then we figure out what may cause >5 min provisioning time, and say if you're a deployment in said environment then we recommend you use sahara-api+engine instead of sahara-all | 13:22 |
| mattf | another tack is to not talk about performance, but about user interaction. if you have X users of sahara & cluster churn rate of Y, then you'll experience delays as the sahara-all engine is going to serialize operations, in such an environment you should deploy api+engine | 13:24 |
| mattf | with motivations of ^^ form that are concrete, doing the api+engine split is nice engineering and architecture, but doesn't address a concrete need and may decrease usability (reliability & increase deployment complexity) of sahara | 13:25 |
| * mattf hops off soapbox for a bit | 13:25 | |
| *** tmckay has joined #openstack-sahara | 13:26 | |
| dmitryme | mattf: I pick option #2 of those proposed by you: do not talk about performance, but about user interaction. As for the threshold when to deploy split Sahara services, I think the right answer is 'whenever you go in production'. Running two services is not complicated in OpenStack in general. Nova and Neutron have more than 5 each. And they do this for the very same reason: to increase manageability of the service. Also remember that split is only | 13:33 |
| dmitryme | part of overall plan. Did you see the spec link in the blueprint? https://wiki.openstack.org/wiki/Savanna/NextGenArchitecture | 13:33 |
| openstackgerrit | A change was merged to openstack/sahara-extra: Change the package name of the example to org.openstack.sahara.examples https://review.openstack.org/90443 | 13:37 |
| mattf | dmitryme, i'm skimming it again. it looked out of date to me before. still does actually, there's an s-conductor in there, and i thought that idea died on the vine. | 13:39 |
| mattf | ...and was chopped off | 13:39 |
| openstackgerrit | A change was merged to openstack/sahara-image-elements: Better setup for custom Java installation https://review.openstack.org/89515 | 13:42 |
| dmitryme | mattf: we figured we don't really need DB access as a separate service. The only reason Nova does that is because it needs to run agent on compute nodes, which are considered to be 'untrusted'. On the other side, with the local conductor we've reached another goal: we incapsulated all the DB logic inside the conductor, while before that DB model objects were scattered all over the source code. | 13:50 |
| tellesnobrega | hi, when using CentOS vanilla image for sahara, is there any password needed? | 13:52 |
| mattf | dmitryme, will you be updating the bp? | 13:55 |
| dmitryme | tellesnobrega: nope, you should get access to the instance via keypair you've specified during boot | 13:55 |
| tellesnobrega | dmitryme: ok | 13:56 |
| dmitryme | mattf: ok. I will discuss that with SergeyLukjanov and either me or he will update the blueprint | 13:57 |
| tmckay | crobertsrh, I started thinking about ways to streamline job running -- I wonder if we can create/destroy data source objects on the fly and skip a step for the user (via CLI or UI) | 13:57 |
| SergeyLukjanov | dmitryme, you, please :) | 13:57 |
| crobertsrh | tmckay: Hmm, that is probably do-able. | 13:57 |
| tmckay | crobertsrh, currently, data source objects are only 1) path and 2) credentials for swift. They could be passed directly as arguments in job executions | 13:58 |
| dmitryme | SergeyLukjanov: ok :-) | 13:58 |
| mattf | dmitryme, thx | 13:58 |
| crobertsrh | Is there any use case for still allowing the more permanent versions of data sources? | 13:58 |
| tmckay | for hdfs data sources, the job execution just needs 2 url strings. For the current swift impl, we could extend the url a la qpid and have swift://user@password/container.sahara/object | 13:59 |
| crobertsrh | I suppose there might be some benefit to defining the DS once and reusing it | 13:59 |
| tmckay | crobertsrh, well, there could be | 13:59 |
| tmckay | I was thinking that we make the treatment context sensitive. If you pass on ID, you get data source objects. If you pass url strings, you get transient objects created/destroyed for you under the hood | 14:00 |
| *** shakamunyi has joined #openstack-sahara | 14:00 | |
| crobertsrh | That makes sense to me | 14:00 |
| tmckay | crobertsrh, that lets someone launch a job just by defining the binaries, grouping them in the job, and launching | 14:00 |
| tmckay | brb | 14:01 |
| crobertsrh | I do like the potential for the minimalist path to execution. | 14:01 |
| SergeyLukjanov | folks, I'm mostly offline today, will read scrollback tomorrow | 14:01 |
| SergeyLukjanov | if you need something - please, ping me directly in private msg | 14:02 |
| *** shakamunyi has quit IRC | 14:04 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Split sahara into sahara-api and sahara-engine https://review.openstack.org/90350 | 14:05 |
| tmckay | crobertsrh, all we need is a hook to mark transient data sources and blow them away when the job completes, or alternatively pipe the path info straight in to the job execution if it's not too much trouble for the job manager (currently built around data sources) | 14:15 |
| dmitryme | tmckay: re swift://user@password/container.sahara/object . Credentials are not enough, you might need to supply tenant name or region name as well for Swift | 14:15 |
| tmckay | crobertsrh, the other question is what to do about advanced swift authentication | 14:15 |
| crobertsrh | right, yhat might be trickier | 14:16 |
| tmckay | dmitryme, could be. We don't do that currently. | 14:16 |
| tmckay | worth thinking about, though. crobertsrh, dimitryme, when I was running a bunch of jobs for testing/playing with demos, it seemed to me that having to create the data objects as a separate step slowed me down. | 14:17 |
| tmckay | unless I was running the same job with the same in put over and over .... | 14:17 |
| dmitryme | tmckay: right now I don't see how we can improve it. We can provide user with ability to specify data sources inline in Job Creation dialog. But that will not be much faster IMHO | 14:21 |
| dmitryme | hmm, but maybe it can be viable and faster, I am not sure | 14:22 |
| *** bradd1 has joined #openstack-sahara | 14:27 | |
| *** shakamunyi has joined #openstack-sahara | 14:27 | |
| tmckay | dmitryme, agreed. I'm not sure if it's a win or not. We might want to think about how to combine it with better authentication -- we've already raised issues about passing swift credentials around (storing in the database, putting in the oozie flows in plain text, etc) | 14:27 |
| *** skolekonov has quit IRC | 14:36 | |
| *** shakamunyi has quit IRC | 14:44 | |
| *** IlyaE has joined #openstack-sahara | 15:00 | |
| *** shakamunyi has joined #openstack-sahara | 15:03 | |
| *** mattf is now known as _mattf | 15:13 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/python-saharaclient: Add EDP jobs to cli integration tests https://review.openstack.org/89692 | 15:19 |
| openstackgerrit | Trevor McKay proposed a change to openstack/python-saharaclient: Stub integration tests for vanilla, vanilla2 and hdp plugins https://review.openstack.org/89371 | 15:33 |
| openstackgerrit | A change was merged to openstack/sahara: Change the package name of the example to org.openstack.sahara.examples https://review.openstack.org/90445 | 15:40 |
| *** IlyaE has quit IRC | 15:42 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/python-saharaclient: Add cli cluster creation and teardown test https://review.openstack.org/89585 | 15:45 |
| *** elmiko has quit IRC | 15:51 | |
| *** elmiko has joined #openstack-sahara | 15:53 | |
| *** bradd1 has quit IRC | 15:54 | |
| *** Ch00k has joined #openstack-sahara | 16:01 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/python-saharaclient: Add EDP jobs to cli integration tests https://review.openstack.org/89692 | 16:02 |
| *** _mattf is now known as mattf | 16:04 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/python-saharaclient: Add EDP jobs to cli integration tests https://review.openstack.org/89692 | 16:04 |
| tmckay | mattf, changed default skip_all to True for hdp/vanilla2 on those CLI CRs | 16:08 |
| * tmckay looks for a Hive example to play with | 16:09 | |
| mattf | kinda bummer. i'd hope vanilla2/hdp2 would be default | 16:09 |
| *** IlyaE has joined #openstack-sahara | 16:11 | |
| *** IlyaE has quit IRC | 16:11 | |
| tmckay | hmm, right, vanilla is kind of old news | 16:13 |
| *** martsen has joined #openstack-sahara | 16:16 | |
| tmckay | hmm, I'm getting Service Temporarily Unavailable on review.openstack.org. Anyone else? | 16:45 |
| *** shakamunyi has quit IRC | 16:46 | |
| dmitryme | tmckay: Gerrit is down for upgrade | 16:46 |
| dmitryme | http://lists.openstack.org/pipermail/openstack-dev/2014-April/033070.html | 16:47 |
| tmckay | dmitryme, oh, thanks | 16:48 |
| elmiko | tmckay: you broke it | 16:48 |
| tmckay | elmiko, probably :-D | 16:49 |
| elmiko | you've abused your gerrit privileges and angered it ;) | 16:50 |
| tmckay | too many rebases | 16:50 |
| elmiko | lol, that would do it | 16:50 |
| crobertsrh | Ah...small heads-up. Gerrit is currently down. https://wiki.openstack.org/wiki/Infrastructure_Status | 16:58 |
| elmiko | yea, tmckay broke it | 16:58 |
| crobertsrh | heh, dmitryme beat me to it :) | 16:59 |
| crobertsrh | and yes, tmckay broke it | 16:59 |
| tmckay | lol, I'm looking at a bagger for my mower, maybe I should take up landscaping | 17:00 |
| *** IlyaE has joined #openstack-sahara | 17:00 | |
| *** shakamunyi has joined #openstack-sahara | 17:12 | |
| *** IlyaE has quit IRC | 17:17 | |
| *** zhiyan is now known as zhiyan_ | 17:19 | |
| *** IlyaE has joined #openstack-sahara | 17:24 | |
| *** tmckay is now known as _tmckay | 17:25 | |
| *** shakamunyi has quit IRC | 17:35 | |
| *** martsen has quit IRC | 17:44 | |
| *** IlyaE has quit IRC | 17:57 | |
| *** IlyaE has joined #openstack-sahara | 18:15 | |
| *** ErikB1 has quit IRC | 18:17 | |
| *** ErikB1 has joined #openstack-sahara | 18:23 | |
| *** Ch00k has quit IRC | 18:33 | |
| *** IlyaE has quit IRC | 18:36 | |
| *** tosky has quit IRC | 18:45 | |
| *** martsen_ has joined #openstack-sahara | 18:50 | |
| *** _tmckay is now known as tmckay | 18:55 | |
| *** openstackgerrit has quit IRC | 19:04 | |
| elmiko | ErikB1: hey Erik, did you or Bob get a chance to look at that email I sent last week? | 19:07 |
| *** IlyaE has joined #openstack-sahara | 19:17 | |
| *** akuznetsov has quit IRC | 20:07 | |
| *** mattf is now known as _mattf | 20:12 | |
| *** shakamunyi has joined #openstack-sahara | 20:17 | |
| elmiko | if i'm using use_neutron=true, i should use_floating_ips=false and use_namespaces=true? | 20:18 |
| tmckay | elmiko, you can use floating ips with neutron if you want to, but you have to tell horizon, too that you're using neutron, and you have to supply a floating ip pool value with the node template | 20:34 |
| tmckay | but, I think "false" is fine | 20:34 |
| elmiko | do i need false if i turn namespaces on though? | 20:34 |
| tmckay | I don't know | 20:35 |
| tmckay | :) | 20:35 |
| elmiko | also, it looks like devstack sets sahara up for neutron and floatingips. i'm just trying to get a little deeper into the configs | 20:35 |
| *** martsen_ has quit IRC | 20:38 | |
| elmiko | cool, and i just figured out how to manipulate the conf from local.conf. | 20:38 |
| *** crobertsrh is now known as _crobertsrh | 21:08 | |
| tmckay | hmm, well, I got a hive job to run on my vanilla node, invoking hive directly. I wonder if I can make it work from oozie now :) | 21:13 |
| tmckay | bbl | 21:13 |
| *** tmckay is now known as tmckay_bbl | 21:13 | |
| *** witlessb has quit IRC | 21:21 | |
| *** witlessb has joined #openstack-sahara | 21:23 | |
| *** IlyaE has quit IRC | 21:31 | |
| *** IlyaE has joined #openstack-sahara | 21:32 | |
| *** IlyaE has quit IRC | 21:53 | |
| *** openstackgerrit has joined #openstack-sahara | 21:53 | |
| *** witlessb has quit IRC | 21:55 | |
| *** IlyaE has joined #openstack-sahara | 22:05 | |
| *** elmiko has quit IRC | 22:14 | |
| *** IlyaE has quit IRC | 23:10 | |
| *** IlyaE has joined #openstack-sahara | 23:39 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!