| *** saurabhs has left #openstack-sahara | 00:00 | |
| *** mahito has joined #openstack-sahara | 00:03 | |
| *** stanchan has quit IRC | 00:16 | |
| *** stanchan has joined #openstack-sahara | 00:25 | |
| *** stanchan has quit IRC | 00:27 | |
| *** mahito has quit IRC | 01:01 | |
| *** mahito has joined #openstack-sahara | 01:06 | |
| *** fvollero has quit IRC | 01:45 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add Sample Versioned Objects for Node Group Template https://review.openstack.org/173654 | 02:26 |
|---|---|---|
| openstackgerrit | Ken Chen proposed openstack/sahara: Add Sample Versioned Objects for Node Group Template https://review.openstack.org/173654 | 02:28 |
| *** mahito has quit IRC | 03:03 | |
| *** Longgeek has joined #openstack-sahara | 03:18 | |
| *** _crobertsrh has quit IRC | 03:34 | |
| *** Longgeek has quit IRC | 03:41 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add Sample Versioned Objects for Node Group Template https://review.openstack.org/173654 | 03:48 |
| *** stanchan has joined #openstack-sahara | 04:08 | |
| *** mahito has joined #openstack-sahara | 04:09 | |
| *** Longgeek has joined #openstack-sahara | 04:37 | |
| *** Longgeek has quit IRC | 04:44 | |
| *** tnovacik has joined #openstack-sahara | 05:17 | |
| *** sgotliv has joined #openstack-sahara | 05:36 | |
| *** Nikolay_St has quit IRC | 05:44 | |
| *** Longgeek has joined #openstack-sahara | 05:48 | |
| *** hdd has joined #openstack-sahara | 06:09 | |
| *** sreshetn1 has joined #openstack-sahara | 06:35 | |
| *** sreshetn1 has quit IRC | 06:48 | |
| *** sgotliv has quit IRC | 06:55 | |
| *** sreshetn1 has joined #openstack-sahara | 07:19 | |
| *** pino|work_ has joined #openstack-sahara | 07:31 | |
| *** chlong has quit IRC | 07:35 | |
| *** Nikolay_St has joined #openstack-sahara | 07:45 | |
| *** witlessb has joined #openstack-sahara | 08:04 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add Sample Versioned Objects for Node Group Template https://review.openstack.org/173654 | 08:27 |
| *** mahito has quit IRC | 08:34 | |
| *** sreshetn1 has quit IRC | 08:53 | |
| *** IBerezovskiy has joined #openstack-sahara | 09:00 | |
| *** esikachev has joined #openstack-sahara | 09:02 | |
| *** sgotliv has joined #openstack-sahara | 09:11 | |
| *** Suprith has quit IRC | 09:16 | |
| *** sreshetn1 has joined #openstack-sahara | 09:21 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Deprecate the Direct Engine https://review.openstack.org/173740 | 09:40 |
| *** jamielennox is now known as jamielennox|away | 09:52 | |
| *** tosky has joined #openstack-sahara | 09:59 | |
| *** Longgeek has quit IRC | 10:02 | |
| *** Longgeek_ has joined #openstack-sahara | 10:02 | |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Install Oozie UI on MapR clusters https://review.openstack.org/172475 | 10:29 |
| *** tnovacik has quit IRC | 10:29 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Move cluster deletion to common engine module https://review.openstack.org/173762 | 10:36 |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Install Oozie UI on MapR clusters https://review.openstack.org/172475 | 10:38 |
| openstackgerrit | Merged openstack/sahara: Updated from global requirements https://review.openstack.org/173490 | 10:55 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Support x-openstack-request-id https://review.openstack.org/172426 | 10:55 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Support x-openstack-request-id https://review.openstack.org/172426 | 10:55 |
| openstackgerrit | Merged openstack/sahara: Adding config hints for HDP plugin https://review.openstack.org/166494 | 10:58 |
| *** sgotliv has quit IRC | 11:00 | |
| openstackgerrit | Merged openstack/sahara: Adding config hints for vanilla plugin https://review.openstack.org/167420 | 11:00 |
| openstackgerrit | Merged openstack/sahara: Restrict cluster to have at most one secondary namenode https://review.openstack.org/171064 | 11:04 |
| openstackgerrit | Merged openstack/python-saharaclient: Add regions support to saharaclient https://review.openstack.org/171734 | 11:14 |
| *** sgotliv has joined #openstack-sahara | 11:15 | |
| *** Longgeek has joined #openstack-sahara | 11:29 | |
| *** Longgeek_ has quit IRC | 11:31 | |
| *** crobertsrh has joined #openstack-sahara | 11:54 | |
| *** tnovacik has joined #openstack-sahara | 11:54 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara-extra: Add script for building oozie https://review.openstack.org/173786 | 11:58 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara-specs: Spec for deprecation of direct engine https://review.openstack.org/173251 | 12:01 |
| *** pino|work_ is now known as pino|work | 12:32 | |
| *** sreshetn1 has quit IRC | 12:36 | |
| openstackgerrit | Merged openstack/sahara: Add CDH template for the scenario integration test https://review.openstack.org/171909 | 12:42 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara-specs: Spec for deprecation of direct engine https://review.openstack.org/173251 | 12:43 |
| *** sreshetn1 has joined #openstack-sahara | 12:43 | |
| *** sgotliv has quit IRC | 12:50 | |
| *** sgotliv has joined #openstack-sahara | 12:51 | |
| *** sgotliv has quit IRC | 12:51 | |
| *** sgotliv has joined #openstack-sahara | 12:51 | |
| *** tmckay has joined #openstack-sahara | 12:56 | |
| *** elmiko has quit IRC | 13:01 | |
| *** elmiko has joined #openstack-sahara | 13:01 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add Sample Versioned Objects for Node Group Template https://review.openstack.org/173654 | 13:31 |
| openstackgerrit | Merged stackforge/sahara-ci-config: Fix tempest job https://review.openstack.org/173324 | 13:40 |
| openstackgerrit | Merged stackforge/sahara-ci-config: Zuul layout.yaml improvements https://review.openstack.org/166209 | 13:41 |
| *** hdd has quit IRC | 13:46 | |
| *** esikachev has quit IRC | 13:46 | |
| *** hdd has joined #openstack-sahara | 13:50 | |
| *** esikachev has joined #openstack-sahara | 13:53 | |
| *** Nikolay_St has quit IRC | 13:54 | |
| *** Nikolay_St has joined #openstack-sahara | 14:04 | |
| openstackgerrit | Gael Chamoulaud proposed stackforge/puppet-sahara: Move rabbit/kombu settings to oslo_messaging_rabbit section https://review.openstack.org/173891 | 14:05 |
| *** hdd has quit IRC | 14:39 | |
| *** hdd has joined #openstack-sahara | 14:52 | |
| *** stanchan has quit IRC | 14:59 | |
| *** DuncanT has joined #openstack-sahara | 15:11 | |
| *** Nikolay_St has quit IRC | 15:20 | |
| *** hdd has quit IRC | 15:20 | |
| *** stanchan has joined #openstack-sahara | 15:35 | |
| *** macjack has joined #openstack-sahara | 15:52 | |
| *** tosky has quit IRC | 15:55 | |
| *** sreshetn1 has quit IRC | 16:09 | |
| *** stanchan has quit IRC | 16:10 | |
| *** tosky has joined #openstack-sahara | 16:21 | |
| *** sgotliv has quit IRC | 16:25 | |
| *** sreshetn1 has joined #openstack-sahara | 16:27 | |
| *** sreshetn1 has quit IRC | 16:35 | |
| *** sreshetn1 has joined #openstack-sahara | 16:38 | |
| *** winston-d_ has joined #openstack-sahara | 16:39 | |
| winston-d_ | greetings | 16:39 |
| elmiko | hi | 16:39 |
| winston-d_ | Does anyone here use/know that any deployment in production use Sahara with Cinder block device driver? | 16:40 |
| winston-d_ | Or simply Hadoop + Cinder block device if that's the case. | 16:41 |
| elmiko | i don't have a lot of experience with cinder, but i've heard from people using it | 16:41 |
| winston-d_ | I am asking because we have Hadoop customer want to try that out. But the driver in Cinder hasn't been updated (bug fixes/feature improvement, etc) after first commit, so I have some concern. | 16:42 |
| winston-d_ | Either the driver is super stable/reliable to simply because it is not widely used. | 16:43 |
| winston-d_ | s/to/ or | 16:43 |
| elmiko | good question, i am probably not the best source on that. | 16:44 |
| winston-d_ | So it'd be nice to learn that somebody has been using Hadoop with it in a relatively large & active deployment | 16:44 |
| elmiko | yea | 16:44 |
| elmiko | it's a question that has come up on the mailing list in the past, but i haven't seen a clear definitive answer | 16:45 |
| winston-d_ | dump question for sahara, please forgive my ignorance, what is the default/preferred/reference approach to handle the disk for Hadoop? | 16:46 |
| winston-d_ | Cinder may not even on the list, I guess? | 16:46 |
| elmiko | is this for input/output data i assume? | 16:47 |
| winston-d_ | elmiko: yes | 16:47 |
| elmiko | you can use swift, or hdfs through configuration, and i think you can use cinder for the hdfs backing | 16:48 |
| *** macjack has quit IRC | 16:48 | |
| elmiko | i've seen folks questioning the performance under cinder though | 16:48 |
| elmiko | although, i've also seen recent questions about swift with large data sets | 16:49 |
| winston-d_ | ok, what's the default/preferred hdfs backing in Sahara? | 16:50 |
| *** esikachev has quit IRC | 16:50 | |
| winston-d_ | elmiko: are you saying the Swift backed hdfs is even better than Cinder backed hdfs? | 16:51 |
| elmiko | no, i don't think so | 16:51 |
| elmiko | i think it would require more testing to confirm though | 16:51 |
| elmiko | afaik, the default backing for hdfs will be whatever the orchestration uses to provision the volumes for an instance | 16:52 |
| elmiko | or, you could use an external hdfs not created by openstack and use it's url when creating the data sources for a job | 16:53 |
| winston-d_ | I thought the extra (multiple) layer of Swift would introduce quite a bit of overhead in terms of disk throughput and latency. | 16:53 |
| winston-d_ | elmiko: ok, all good info. | 16:53 |
| elmiko | i think you are correct, plus there seems to be some issue about the number of REST calls that the hadoop-swift connector is making | 16:53 |
| *** IBerezovskiy has quit IRC | 16:55 | |
| winston-d_ | elmiko: thx for all the info, really appreicate your help! | 16:58 |
| elmiko | winston-d_: glad to help, wish i had better info | 16:59 |
| elmiko | winston-d_: this bug might provide an intersting read as well https://bugs.launchpad.net/sahara/+bug/1371360 | 17:00 |
| openstack | Launchpad bug 1371360 in Sahara "[DOC] Features overview suggests Cinders to increase reliability" [Low,Confirmed] - Assigned to plieb (jliberma) | 17:00 |
| *** sreshetn1 has quit IRC | 17:02 | |
| tmckay | elmiko, in the new era of API documentation with wadl, how does that actually happen? For instance, if I'm writing a bp for datasource object edit, the template guidelines say add REST stuff. But what REST stuff do I add? | 17:08 |
| tmckay | do I just add the same old expected return codes, errors, and sample json objects? | 17:08 |
| tmckay | along with the endpoint | 17:08 |
| elmiko | tmckay: i imagine you'd need to update the docs at the api-ref site. this is kinda why i don't like removing all the api stuff from our docs. a topic i've marked for discussion at summit. | 17:10 |
| tmckay | alright. But at least for the spec, we're supposed to include content that describes the change if any to REST. So I suppose I should include that just the same | 17:11 |
| elmiko | and yea, you will add expected returns, maybe some example json | 17:11 |
| tmckay | k, thanks. | 17:11 |
| tmckay | just wondered if that had changed | 17:11 |
| elmiko | oh yea, for the spec i would add whatever if necessary to demonstrate | 17:12 |
| elmiko | s/if/is | 17:12 |
| *** tnovacik has quit IRC | 17:28 | |
| *** sreshetn1 has joined #openstack-sahara | 17:32 | |
| *** hdd has joined #openstack-sahara | 17:32 | |
| *** pino|work has quit IRC | 17:34 | |
| *** Longgeek has quit IRC | 17:51 | |
| *** openstackstatus has quit IRC | 17:58 | |
| *** openstackstatus has joined #openstack-sahara | 18:00 | |
| *** ChanServ sets mode: +v openstackstatus | 18:00 | |
| -openstackstatus- NOTICE: Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem. | 18:04 | |
| *** ChanServ changes topic to "Gerrit has stopped emitting events so Zuul is not alerted to changes. We will restart Gerrit shortly to correct the problem." | 18:04 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Added check of scaling for Spark plugin https://review.openstack.org/172842 | 18:24 |
| tmckay | crobertsrh, hey guess what? There are no ng or cluster template update operations in python-saharaclient CLI. Was that on purpose? | 18:26 |
| tmckay | the client method is there, but the CLI isn't | 18:26 |
| *** ChanServ changes topic to "launchpad openid is down. login to openstack services will fail until launchpad openid is happy again" | 18:26 | |
| -openstackstatus- NOTICE: Gerrit has been restarted. New patches, approvals, and rechecks between 17:30 and 18:20 UTC may have been missed by Zuul and will need rechecks or new approvals added. | 18:27 | |
| crobertsrh | Not intentional | 18:27 |
| crobertsrh | Well, maybe semi-intentional | 18:27 |
| crobertsrh | But mostly forgotten. | 18:28 |
| elmiko | lol | 18:28 |
| tmckay | hmm. Okay. More bugs for Liberty :) | 18:28 |
| *** sreshetn1 has quit IRC | 18:28 | |
| *** Nikolay_St has joined #openstack-sahara | 18:30 | |
| *** winston-d_ has quit IRC | 18:34 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: [EDP] Allow editing datasource objects https://review.openstack.org/174019 | 18:36 |
| *** esikachev has joined #openstack-sahara | 18:41 | |
| *** esikachev has quit IRC | 18:44 | |
| *** tosky has quit IRC | 18:44 | |
| *** hdd has quit IRC | 18:53 | |
| *** tnovacik has joined #openstack-sahara | 18:55 | |
| *** stanchan has joined #openstack-sahara | 18:59 | |
| *** sreshetn1 has joined #openstack-sahara | 19:14 | |
| *** sgotliv has joined #openstack-sahara | 19:24 | |
| *** Nikolay_St has quit IRC | 19:27 | |
| *** Nikolay_St has joined #openstack-sahara | 19:29 | |
| *** sreshetn1 has quit IRC | 19:29 | |
| *** winston-d_ has joined #openstack-sahara | 19:30 | |
| *** winston-d_ has quit IRC | 19:35 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: [EDP] Allow editing job binaries https://review.openstack.org/174045 | 19:43 |
| *** sgotliv has quit IRC | 19:56 | |
| tmckay | crobertsrh, ping | 20:07 |
| crobertsrh | yes? | 20:07 |
| crobertsrh | oh :) ^^^ | 20:08 |
| crobertsrh | gotcha | 20:08 |
| tmckay | hey. I think I've asked this before -- but you can display any success message we want on an object create in the UI, right? I'm thinking about the early-warning system on invalid swift paths. If I grab a swift connection and use the head_object() method I can return a success with a warning if I can't dereference the path | 20:09 |
| tmckay | crobertsrh, that patch wasn't it, I was just typing slowly :) | 20:09 |
| crobertsrh | Yep, I can make just about any message happen | 20:10 |
| tmckay | that gives the user a clue that his swift path does not exist. Either they typed it incorrectly, or gave the wrong creds, or they know that and just haven't made it yet. | 20:10 |
| tmckay | crobertsrh, do you key off an error code, or grab some message that I would send back? | 20:10 |
| tmckay | how do I return "success" and some other message? | 20:11 |
| tmckay | not sure I've ever looked at that | 20:11 |
| crobertsrh | Could use error code. I think right now it's pretty much only one message per call, either good or bad | 20:11 |
| crobertsrh | The code might have to be part of the returned object on "success" | 20:12 |
| crobertsrh | or, there is probably a specific code for this sort of thing | 20:12 |
| tmckay | gotcha. you mean an http response code? Okay, I'll take a look. I think it's a nice-to-have feature | 20:13 |
| crobertsrh | yeah | 20:13 |
| crobertsrh | "strange" codes might generate bizarre results in hoirzon, not sure I've seen many other than 404, 200, 204, 202 | 20:14 |
| crobertsrh | Not sure which ones would be handled nicely by default | 20:15 |
| tmckay | yeah, I'm a little worried about that. 404 on a create might communicate it, for a data source :) Path not found | 20:15 |
| crobertsrh | It would be a slight bending of the codes, that's for sure | 20:24 |
| tmckay | crobertsrh, so when you get an object back, what do you do with it? | 20:24 |
| crobertsrh | Or maybe a separate call that can be made against the object after we create it? | 20:25 |
| crobertsrh | It's less automatic, but clears some of the mud | 20:25 |
| tmckay | yeah, I was just thinking that. Choices seem to be a validation error of some sort (but that would be weird, if we created the object anyway), or add a message in the json object sent back | 20:26 |
| tmckay | or a separate endpoint | 20:26 |
| tmckay | crobertsrh, the other idea I have is that it is an optional validation check to ensure the path is accessible. On by default, probably, and you can turn it off. | 20:27 |
| crobertsrh | That might be good. If validation is ON and it's not valid, create fails? | 20:28 |
| tmckay | yep | 20:28 |
| crobertsrh | I like that | 20:28 |
| tmckay | then if you really want to because you know better, uncheck the box and try again | 20:28 |
| *** hdd has joined #openstack-sahara | 20:28 | |
| crobertsrh | checkboxes are cheap too | 20:28 |
| crobertsrh | I have a pile of them just sitting here | 20:28 |
| tmckay | heh. And it's not like the data source form has a lot of stuff on it | 20:29 |
| crobertsrh | very true | 20:29 |
| tmckay | well, if it meets your UI guy sniff test, it's good enough for me | 20:29 |
| crobertsrh | I'm +1 for validation flag, but we might want to call it something other than "validation" | 20:30 |
| tmckay | I just know personally I've been bitten by mistyped swift and hdfs paths on any number of occasions | 20:30 |
| crobertsrh | "Verify path exists" or something like that | 20:30 |
| crobertsrh | me too | 20:30 |
| tmckay | yeah, I'll think of a flashy name | 20:30 |
| crobertsrh | Maybe on the UI it should be, "I'd like for my job to actually run" | 20:31 |
| tmckay | llol | 20:31 |
| tmckay | good, but too general | 20:31 |
| tmckay | might give them false hope | 20:31 |
| crobertsrh | or, "Uncheck this to poke yourself in the eye" | 20:31 |
| tmckay | also good | 20:32 |
| crobertsrh | Maybe I should have a checkbox, checked by default and grayed out | 20:32 |
| crobertsrh | it gives the illusion of choice | 20:32 |
| tmckay | you're full of great ideas today :) | 20:32 |
| crobertsrh | Defender of the users | 20:33 |
| crobertsrh | I gotta look out for my peeps | 20:33 |
| tmckay | so I found a swiftclient method for checking those, I need to come up with a slick check for external hdfs paths | 20:33 |
| crobertsrh | seems like something that would have been done elsewhere | 20:34 |
| tmckay | yeah, could be | 20:34 |
| tmckay | I suppose I don't need that to write the spec. Exercise for the implementer | 20:34 |
| *** crobertsrh is now known as _crobertsrh | 21:04 | |
| *** jamielennox|away is now known as jamielennox | 21:46 | |
| *** jodah has left #openstack-sahara | 21:55 | |
| openstackgerrit | Merged openstack/sahara: Use jsonutils from oslo.serialization https://review.openstack.org/172083 | 22:02 |
| *** hdd has quit IRC | 22:10 | |
| *** winston-d_ has joined #openstack-sahara | 23:03 | |
| *** winston-1_ has joined #openstack-sahara | 23:08 | |
| *** winston-d_ has quit IRC | 23:08 | |
| *** chlong has joined #openstack-sahara | 23:10 | |
| *** ylobankov_ has joined #openstack-sahara | 23:33 | |
| *** skolekonov_ has joined #openstack-sahara | 23:38 | |
| *** skolekonov has quit IRC | 23:38 | |
| *** ylobankov has quit IRC | 23:39 | |
| *** stanchan has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!