| *** egafford has joined #openstack-sahara | 00:08 | |
| *** shakamunyi has joined #openstack-sahara | 00:12 | |
| *** barra204 has joined #openstack-sahara | 00:12 | |
| *** egafford has quit IRC | 00:27 | |
| *** superfly_ has joined #openstack-sahara | 01:12 | |
| *** superfly_ is now known as superflyy | 01:14 | |
| *** barra204 has quit IRC | 01:24 | |
| *** shakamunyi has quit IRC | 01:24 | |
| *** superflyy has quit IRC | 01:24 | |
| *** hdd has joined #openstack-sahara | 01:55 | |
| *** Networkn3rd has joined #openstack-sahara | 01:58 | |
| *** robsparker has joined #openstack-sahara | 02:23 | |
| *** Networkn3rd has quit IRC | 02:40 | |
| *** egafford has joined #openstack-sahara | 03:17 | |
| *** robsparker has quit IRC | 03:26 | |
| *** hdd has quit IRC | 03:39 | |
| *** Longgeek_ has quit IRC | 03:53 | |
| *** Longgeek has joined #openstack-sahara | 03:54 | |
| *** sgotliv has joined #openstack-sahara | 03:55 | |
| *** Longgeek_ has joined #openstack-sahara | 03:57 | |
| *** Longgeek has quit IRC | 04:00 | |
| *** egafford has quit IRC | 04:01 | |
| *** Longgeek has joined #openstack-sahara | 04:02 | |
| *** Longgeek_ has quit IRC | 04:04 | |
| *** Longgeek_ has joined #openstack-sahara | 04:08 | |
| *** Longgeek has quit IRC | 04:10 | |
| *** Longgeek_ has quit IRC | 04:12 | |
| *** Longgeek has joined #openstack-sahara | 04:18 | |
| *** Longgeek has quit IRC | 04:18 | |
| *** Longgeek has joined #openstack-sahara | 04:19 | |
| *** Longgeek has quit IRC | 04:20 | |
| *** Longgeek has joined #openstack-sahara | 04:20 | |
| *** Longgeek_ has joined #openstack-sahara | 04:23 | |
| *** Longgee__ has joined #openstack-sahara | 04:25 | |
| *** Longgeek has quit IRC | 04:25 | |
| *** Longgeek_ has quit IRC | 04:26 | |
| *** Longgeek has joined #openstack-sahara | 04:26 | |
| *** hdd has joined #openstack-sahara | 04:26 | |
| *** Longgeek_ has joined #openstack-sahara | 04:28 | |
| *** Longge___ has joined #openstack-sahara | 04:29 | |
| *** Longgee__ has quit IRC | 04:30 | |
| *** coolsvap|afk is now known as coolsvap | 04:30 | |
| *** coolsvap is now known as coolsvap|afk | 04:31 | |
| *** Longgeek has quit IRC | 04:32 | |
| *** coolsvap|afk is now known as coolsvap | 04:32 | |
| *** Longgeek_ has quit IRC | 04:32 | |
| *** nkrinner has joined #openstack-sahara | 05:03 | |
| *** robsparker has joined #openstack-sahara | 05:25 | |
| *** robsparker has quit IRC | 05:29 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Add CDH5.4 support in sahara https://review.openstack.org/177660 | 05:32 |
|---|---|---|
| *** hdd has quit IRC | 05:34 | |
| *** sgotliv has quit IRC | 06:15 | |
| *** shakamunyi has joined #openstack-sahara | 06:16 | |
| *** barra204 has joined #openstack-sahara | 06:17 | |
| openstackgerrit | Ken Chen proposed openstack/sahara: Transform configuration values into int or float when needed https://review.openstack.org/187444 | 06:34 |
| *** Longgeek has joined #openstack-sahara | 06:35 | |
| *** Longge___ has quit IRC | 06:37 | |
| openstackgerrit | lu huichun proposed openstack/sahara-specs: Add scheduling and recurring oozie workflow to EDP jobs https://review.openstack.org/175719 | 07:17 |
| openstackgerrit | lu huichun proposed openstack/sahara-specs: Add scheduling and recurring oozie workflow to EDP jobs https://review.openstack.org/175719 | 07:22 |
| *** chlong has quit IRC | 07:37 | |
| *** coolsvap is now known as coolsvap|afk | 07:41 | |
| *** coolsvap|afk is now known as coolsvap | 07:46 | |
| *** sgotliv has joined #openstack-sahara | 07:47 | |
| *** witlessb has joined #openstack-sahara | 07:52 | |
| *** esikachev has joined #openstack-sahara | 08:50 | |
| *** nkrinner has quit IRC | 08:52 | |
| *** nkrinner has joined #openstack-sahara | 08:57 | |
| *** pino|work has joined #openstack-sahara | 08:57 | |
| *** tellesnobrega has quit IRC | 09:07 | |
| *** tosky_ has joined #openstack-sahara | 09:09 | |
| *** witlessb has quit IRC | 09:38 | |
| *** witlessb has joined #openstack-sahara | 09:40 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Added method for connect to node and run command https://review.openstack.org/185419 | 09:51 |
| *** tellesnobrega has joined #openstack-sahara | 10:02 | |
| *** robsparker has joined #openstack-sahara | 10:25 | |
| *** robsparker has quit IRC | 10:31 | |
| *** sudipto has joined #openstack-sahara | 10:41 | |
| *** sudipto has quit IRC | 10:42 | |
| *** sudipto has joined #openstack-sahara | 10:43 | |
| *** sudipto has quit IRC | 10:44 | |
| *** esikachev has quit IRC | 11:01 | |
| *** sudipto has joined #openstack-sahara | 11:07 | |
| sudipto | Hi, can anyone please tell me if libguestfs is a must for sahara to inject ssh keys? | 11:08 |
| SergeyLukjanov | sudipto, hi, which version of sahara are you using? | 11:16 |
| sudipto | SergeyLukjanov, I am using the Kilo one. | 11:17 |
| *** esikachev has joined #openstack-sahara | 11:18 | |
| SergeyLukjanov | sudipto, we're using user data script to push keys // https://github.com/openstack/sahara/blob/master/sahara/service/engine.py#L166 | 11:19 |
| SergeyLukjanov | sudipto, so, I think guestfs is not needed for it | 11:19 |
| SergeyLukjanov | and user key is passed as a parameter for instance creation https://github.com/openstack/sahara/blob/master/sahara/service/direct_engine.py#L363 | 11:21 |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Added method for connect to node and run command https://review.openstack.org/185419 | 11:22 |
| sudipto | SergeyLukjanov, Thanks. During my deployments - Sahara would not be able to login to the VMs at all, I observed the guestfs errors in the logs. So i thought it might be dependent on that. Libguestfs had issues on my nodes. | 11:24 |
| sudipto | guestfs errors in the nova logs to be precise. | 11:24 |
| SergeyLukjanov | sreshetnyak, ^^ any thoughts? | 11:24 |
| tosky_ | sudipto: which kind of errors? | 11:26 |
| tosky_ | sudipto: can you run instances from those images directly from nova? I think it may be unrelated to Sahara | 11:26 |
| sudipto | tosky_, Let me get the exact details. I have destroyed my environment and re-building it again. | 11:27 |
| tosky_ | sudipto: also, which platform is running on the OS, and which distribution on images? | 11:27 |
| sudipto | tosky_, If i enable the config drive in nova - I am able to work on a standalone deploy using the ssh key | 11:27 |
| sudipto | tosky_, but when i deploy the VMs using sahara...the sahara logs show that it's unable to login to the VMs. | 11:27 |
| sudipto | tosky_, the platform is Power and the distribution is Fedora. | 11:28 |
| sudipto | tosky_, Let me re-run my setup again and get back with all the details needed... | 11:28 |
| SergeyLukjanov | tosky_, I'd like to collect your feedback on extracting scenario tests to separate repo to make installable, packageable, add CLI and release it separately with backward compat for at least 1-2 OpenStack releases | 11:34 |
| SergeyLukjanov | tosky_, I'm now working on a detailed spec, but it's the main changes I'm thinking about so far | 11:35 |
| *** robsparker has joined #openstack-sahara | 11:41 | |
| *** sudipto has quit IRC | 11:58 | |
| tosky_ | SergeyLukjanov: I think splitting is a good idea; regarding the supported versions, I would consider at least the same deadlines that Tempest considers, if no more | 12:03 |
| SergeyLukjanov | tosky_, yup, most probably it should be the same as tempest from the support PoV | 12:04 |
| *** sgotliv has quit IRC | 12:09 | |
| *** sgotliv has joined #openstack-sahara | 12:12 | |
| *** tmckay has quit IRC | 12:18 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Changing log level inside execute_with_retries method https://review.openstack.org/187978 | 12:20 |
| *** robsparker has quit IRC | 12:34 | |
| *** _crobertsrh is now known as crobertsrh | 12:54 | |
| *** nkrinner has quit IRC | 13:15 | |
| *** tmckay has joined #openstack-sahara | 13:19 | |
| *** tmckay has quit IRC | 13:19 | |
| *** tmckay has joined #openstack-sahara | 13:20 | |
| *** robsparker has joined #openstack-sahara | 13:22 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Added method for connect to node and run command https://review.openstack.org/185419 | 13:28 |
| *** hdd has joined #openstack-sahara | 13:30 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Implement recommendations for vanilla 2.6.0 https://review.openstack.org/177280 | 13:35 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Implement recommendations for vanilla 2.6.0 https://review.openstack.org/177280 | 13:39 |
| *** rnowling has joined #openstack-sahara | 13:39 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara-specs: Allow multiple clusters creation simultaneously https://review.openstack.org/188011 | 13:53 |
| *** egafford has joined #openstack-sahara | 13:58 | |
| *** egafford has quit IRC | 13:58 | |
| *** egafford1 has joined #openstack-sahara | 13:58 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [Spark] Add recommendation about dfs.replication https://review.openstack.org/187590 | 13:59 |
| *** coolsvap is now known as coolsvap|afk | 14:00 | |
| *** hdd has quit IRC | 14:05 | |
| *** tmckay has quit IRC | 14:07 | |
| *** tmckay has joined #openstack-sahara | 14:07 | |
| *** esikachev has quit IRC | 14:08 | |
| *** tmckay has joined #openstack-sahara | 14:11 | |
| *** tmckay has joined #openstack-sahara | 14:12 | |
| *** tmckay has left #openstack-sahara | 14:12 | |
| *** tmckay has joined #openstack-sahara | 14:13 | |
| *** tmckay has quit IRC | 14:17 | |
| *** tmckay has joined #openstack-sahara | 14:17 | |
| *** tmckay has quit IRC | 14:18 | |
| *** Networkn3rd has joined #openstack-sahara | 14:19 | |
| *** tmckay has joined #openstack-sahara | 14:19 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [CDH] Load missed configs from yarn-gateway.json https://review.openstack.org/187179 | 14:20 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Support recommendations in CDH plugin https://review.openstack.org/185371 | 14:20 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: [CDH] Load missed configs from yarn-gateway.json https://review.openstack.org/187179 | 14:22 |
| *** barra204 has quit IRC | 14:36 | |
| *** shakamunyi has quit IRC | 14:36 | |
| *** raildo has joined #openstack-sahara | 14:39 | |
| *** hdd has joined #openstack-sahara | 14:41 | |
| egafford1 | crobertsrh, NikitaKonovalov: ping | 14:50 |
| crobertsrh | yes? | 14:50 |
| egafford1 | Hi Chad. I was wondering first about whether you'd thought at all about how you'd like the UI for the Job Interface Map story to work, and secondly, whether you see a purpose for storing the execution arguments as data rows. In implementation, I've found that the JobExecutionArguments table is pretty vestigial. | 14:52 |
| egafford1 | Don't want to remove it, though, without talking to UI folks, in case you think there's a solid point in favor from a presentation standpoint. I can scrounge you links on the subject if you need a refresher, or just answer questions. | 14:53 |
| crobertsrh | It's been on my todo list for awhile, but I haven't yet had any strokes of genius. I need to go through that spec again. Do you have any ideas? | 14:53 |
| egafford1 | Well, on the job side, it seems reasonably straightforward if not easy to implement; we need some manner of table with little + buttons to add more rows, little - buttons to remove them, etc. | 14:54 |
| *** robsparker has quit IRC | 14:54 | |
| crobertsrh | To answer the other question though, I'm not sure I can see a reason for having execution arguments stored as data rows. | 14:56 |
| egafford1 | On the execution side, though, because the execution object is frozen when it gets to the EDP engine, there's not a clean way to merge in place when the job is ready to run without updating the object, so we don't really gain any record keeping power (as I was hoping) by having the execution arguments table and I'm thinking of dropping it for simplicity. | 14:56 |
| egafford1 | Yeah, the execution arguments can always be reconstituted from the merged job_configs json and the interface in a pinch. | 14:56 |
| crobertsrh | Right. I think that makes sense. Simplicity is probably the way to go there. | 14:57 |
| egafford1 | Agreed. Thanks crobertsrh. | 14:57 |
| crobertsrh | Well...for job relaunch, we do need to be able to get the arguments | 14:58 |
| crobertsrh | So, I suppose there needs to be a reasonably efficient way to make that happen. | 14:58 |
| crobertsrh | egafford1: Does that change anything? ^^^ | 15:03 |
| egafford1 | crobertsrh: Well, we'll have the merged, eventual argument set (that the job actually ran with,) and we'll still have the interface itself, so we can throw the form back up if we want to. | 15:05 |
| egafford1 | That feels to me like enough, but whether that is enough is very precisely the question. :) | 15:05 |
| crobertsrh | Ok, that question came mostly from me not having read through things in awhile. | 15:05 |
| egafford1 | No, it's a reasonable question. | 15:05 |
| *** esikachev has joined #openstack-sahara | 15:06 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Added method for connect to node and run command https://review.openstack.org/185419 | 15:09 |
| egafford1 | elmiko: API question: I'm strongly considering dropping the JobExecutionArguments table in the unified job interface map story, which would naturally result in dropping that from the GET for job executions (and just representing the final configuration state.) Many folks have many beliefs about what makes an API RESTful, but one such belief I've heard from time to time is that the resource that you GET is the resource that | 15:13 |
| elmiko | egafford1: resource that....? | 15:15 |
| egafford1 | elmiko: ...you GET is the resource that you PUT. Do we have such scruples here, or is it perfectly fine by the API WG's standards to drop some input fields from the output when necessary (just as we inevitably add new fields to the output?) | 15:16 |
| elmiko | hmm | 15:16 |
| egafford1 | Just trying to assess which flavor of RESTafarians we are. | 15:16 |
| elmiko | so far, the api-wg has mainly addressed request methods and the proper usage of them. there aren't any guidelines yet as to content. i think that is solidly in the developers hands at this point. | 15:17 |
| egafford1 | The data that you provide will come back to you under the new model; it'll just be munged into the normal job_configs json. Okay, cool. If it's solidly in my hands, I'll run with it. Thanks elmiko. | 15:17 |
| elmiko | in general though, i think folks would agree with you. i would expect the resrouce i PUT to be returned in the response, but i don't think that is a hard rule. | 15:17 |
| egafford1 | elmiko: Right, I've thought of it as a good guideline to follow in almost all cases. Here, though, on review, I don't think any purpose is served by not just returning the job_configs. | 15:19 |
| elmiko | egafford1: i think the JobExecutionArguments table makes a great deal of sense if we consider the possibility of an endpoint just for that resource | 15:20 |
| elmiko | otherwise, i think it would be ok to return what the final job config looks like. i dunno, this is a tough question. | 15:20 |
| egafford1 | The execution is frozen by the time it gets to the edp job manager (where I was hoping to do the merge to avoid committing the merge to the execution object,) so I have to update in order to prevent the edp api from becoming more complex. | 15:21 |
| egafford1 | Well, I can definitely see a case for a JobInterfaceArgument resource, easy. I have trouble seeing a JobExecutionArgument resource unless we just go crazy with the resources. | 15:22 |
| elmiko | i don't remember, because i need to read the spec again, but it might be worthwhile to have an /job-execution/{j-e id}/arguments resource that could be manipulated? | 15:22 |
| egafford1 | I don't think so; job-executions are the one resource that can't be edited. | 15:22 |
| elmiko | ok, maybe i just misinterpretted. JobInterfaceArgument makes sense | 15:22 |
| elmiko | i'm a little off on the specifics, but i think you get my gist | 15:23 |
| egafford1 | (And really shouldn't be.) They can be replayed, but not edited. Yeah, the interface args are ABSOLUTELY a resource-ready model. I do; I think we're in broad agreement on both the ideal and the reasonable. | 15:23 |
| elmiko | ok, cool | 15:23 |
| elmiko | sorry for munging the details so badly | 15:24 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara-specs: Removing JobExecutionArgument Table https://review.openstack.org/188047 | 15:25 |
| egafford1 | elmiko: It is utterly not your job to know the intimate details of everything I'm working on right right now. :) Thanks for the counsel; it's great to have an OS API authority on tap. | 15:34 |
| egafford1 | I think the terminology change around job_templates/job_executions/jobs probably can't have helped recollection of the details either, but, hey, that's what v2 is for. | 15:35 |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Rename all plugins to full plugin version https://review.openstack.org/188053 | 15:36 |
| *** Networkn3rd has quit IRC | 15:37 | |
| elmiko | egafford1: yea, i feel like i should have remembered more details. and yea, those route changes are in my first pass lineup for v2. | 15:38 |
| egafford1 | elmiko: Well, if you had remembered all the details, I'd be moderately amazed at you. And +1 for those changes. | 15:40 |
| elmiko | lol! | 15:40 |
| openstackgerrit | Denis Egorenko proposed stackforge/sahara-ci-config: Rename all plugins to full plugin version https://review.openstack.org/188053 | 15:43 |
| *** esikachev has quit IRC | 16:01 | |
| *** jamielennox is now known as jamielennox|away | 16:21 | |
| *** barra204 has joined #openstack-sahara | 16:21 | |
| *** barra204 has quit IRC | 16:28 | |
| openstackgerrit | Shilla Saebi proposed openstack/sahara: made a number of changes to sahara docs https://review.openstack.org/177523 | 16:50 |
| *** Longgeek has quit IRC | 16:55 | |
| openstackgerrit | Merged openstack/sahara: [CDH] Load missed configs from yarn-gateway.json https://review.openstack.org/187179 | 17:00 |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara: Fix problem with removing PID from list https://review.openstack.org/187155 | 17:09 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: [WIP][EDP] Unified Map to Define Job Interface https://review.openstack.org/187809 | 17:13 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara-specs: Allow multiple clusters creation simultaneously https://review.openstack.org/188011 | 17:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements https://review.openstack.org/188102 | 17:39 |
| *** Networkn3rd has joined #openstack-sahara | 17:44 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed openstack/sahara-specs: Allow multiple clusters creation simultaneously https://review.openstack.org/188011 | 17:47 |
| tmckay | egafford1, setting up a stack to give your job stuff a whirl | 17:49 |
| elmiko | tmckay: i take it the grid0/1 stuff worked? | 17:49 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sahara: Updated from global requirements https://review.openstack.org/188102 | 17:49 |
| tmckay | elmiko, heh, still in progress | 17:49 |
| elmiko | ah ok, just getting my hopes up =) | 17:50 |
| tmckay | I just tossed up an allinone on one of the machines so I could whack at ethan's patch | 17:50 |
| tmckay | elmiko, I am still hoping for this week, but I've been remiss on reviews and the job mapping seems like a good one to focus on | 17:50 |
| tmckay | it's all about balance ;-) | 17:50 |
| elmiko | oh yea, no questions from me. i was just curious given our pre-lunch conversation. | 17:53 |
| elmiko | i thought you might be close to a "eureka" moment ;) | 17:54 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Implement recommendations for vanilla 2.6.0 https://review.openstack.org/177280 | 17:56 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Support recommendations in CDH plugin https://review.openstack.org/185371 | 17:56 |
| elmiko | SergeyLukjanov: ping | 17:58 |
| *** sgotliv has quit IRC | 18:02 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Remove WritableLogger wrapper https://review.openstack.org/188130 | 18:05 |
| *** hdd has quit IRC | 18:06 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Remove deprecated group name of option https://review.openstack.org/188132 | 18:11 |
| openstackgerrit | Michael McCune proposed openstack/sahara-specs: Reworking improved secret storage spec https://review.openstack.org/179393 | 18:14 |
| *** hdd has joined #openstack-sahara | 18:21 | |
| *** Networkn3rd has quit IRC | 18:37 | |
| *** stanchan has joined #openstack-sahara | 18:40 | |
| *** witlessb has quit IRC | 18:50 | |
| *** witlessb has joined #openstack-sahara | 18:52 | |
| elmiko | egafford1: i just want to understand https://review.openstack.org/#/c/188047 a little better | 18:54 |
| *** hdd has quit IRC | 18:54 | |
| elmiko | egafford1: so, the actual execution arguments won't be stored in the db? | 18:54 |
| elmiko | they will be parsed/aggregated and stored as the true configs? | 18:55 |
| *** Networkn3rd has joined #openstack-sahara | 18:56 | |
| *** barra204 has joined #openstack-sahara | 18:57 | |
| *** Networkn3rd has quit IRC | 19:00 | |
| *** Networkn3rd has joined #openstack-sahara | 19:00 | |
| *** barra204 has quit IRC | 19:02 | |
| *** hdd has joined #openstack-sahara | 19:03 | |
| egafford1 | elmiko: That's correct; the execution args will be folded into their final form in the config map for storage. | 19:11 |
| egafford1 | The arguments can always be reconstituted from their locations if desired. We'll lose whether the user passed in the argument via the interface or the map, but that's all we lose. | 19:12 |
| elmiko | egafford1: ok, thanks. just wanted to make sure i understood | 19:13 |
| egafford1 | elmiko: In order to fold relatively seamlessly into the current edp job runner, the configs had to all be updated into the execution object's job_configs attribute anyway, so there didn't seem to be much upside in storing them all twice. | 19:14 |
| egafford1 | elmiko: So while I was planning on merging just pre-run, I ended up pulling the merge of the interface back to the initial execution storage. Do you have concerns about removing the execution args table? | 19:16 |
| egafford1 | elmiko: Apparently not. :) | 19:17 |
| *** Networkn3rd has quit IRC | 19:17 | |
| elmiko | egafford1: yea, not really. i just want to better understand the problem space | 19:19 |
| egafford1 | elmiko: Solid. | 19:22 |
| egafford1 | Basically, I ran into "the job execution is frozen when it gets to the executor, so I can either jump through a bunch of hoops and clutter the executor's interface or pre-merge the configs". Pre-merging the configs was a lot cleaner, and once I did that, I was well below my indifference curve on the existence of the execution_args table (which was already kind of pushing it, as without a join the presentation was just a bu | 19:24 |
| elmiko | yea, makes total sense | 19:24 |
| elmiko | no need to clutter up the db with more stuff when it's a wash in the ned | 19:24 |
| elmiko | end | 19:24 |
| egafford1 | And the ned. | 19:25 |
| egafford1 | They both work. | 19:25 |
| elmiko | lol | 19:25 |
| elmiko | ahh ned | 19:25 |
| egafford1 | It's always a wash in that guy. | 19:25 |
| elmiko | lol | 19:25 |
| *** Networkn3rd has joined #openstack-sahara | 19:51 | |
| *** barra204 has joined #openstack-sahara | 19:58 | |
| *** barra204 has quit IRC | 20:03 | |
| htruta | hi guys! do you know something about this bp https://blueprints.launchpad.net/sahara/+spec/ceilometer-integration ? | 20:08 |
| htruta | I mean, it is marked as complete, but looks like we don't have any sahara metrics collected by ceilometer... only the cluster events | 20:09 |
| elmiko | htruta: hmm, i dunno. looks like the patches were abandonned | 20:11 |
| htruta | elmiko: so, that's all sahara sends to ceilometer? | 20:14 |
| elmiko | htruta: not sure, i don't know much about what sahara sends to sahara | 20:15 |
| elmiko | looks like this patch, https://review.openstack.org/#/c/108982/ , made it through | 20:15 |
| htruta | elmiko: yes... that's the one which sends the cluster events | 20:16 |
| elmiko | that might be all, i'm not 100% sure on our ceil. integration | 20:16 |
| htruta | elmiko: ok | 20:23 |
| htruta | what about some sort of internal cluster performance monitoring | 20:23 |
| htruta | does sahara do that? | 20:23 |
| elmiko | there are some reviews up for cluster health checks, | 20:23 |
| elmiko | but i don't think they touch on performance | 20:23 |
| elmiko | https://review.openstack.org/#/c/175363/ | 20:24 |
| elmiko | that's the spec | 20:24 |
| elmiko | htruta: what kind of performance data are you thinking about? | 20:24 |
| htruta | elmiko: I'll take a look at the spec | 20:29 |
| htruta | I'm talking about some plugin specific metrics... | 20:30 |
| htruta | map/reduce tasks, throughput... | 20:30 |
| htruta | in the hadoop example | 20:30 |
| elmiko | ok, yea. i don't think we have much in sahara for that. it might be that ambari and cloudera manager expose that type of info | 20:30 |
| elmiko | certainly sounds cool though =) | 20:31 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Support recommendations in CDH plugin https://review.openstack.org/185371 | 20:35 |
| htruta | elmiko: cool. I need to take a look in Ambari, then | 20:40 |
| elmiko | htruta: i think only the hdp plugin supports ambari, but i might be wrong about that. | 20:40 |
| htruta | just made a quick grep here, and I guess you're not wrong | 20:41 |
| elmiko | i think we might be able to build some simple metrics based on the start/stop time for jobs | 20:42 |
| *** egafford1 has quit IRC | 20:46 | |
| htruta | elmiko: wouldn't it be the job execution finish time? | 20:52 |
| htruta | my big picture is looking at the job at runtime and get some metrics to see if we can make it faster | 20:54 |
| elmiko | htruta: yea, my bad, job execution | 20:54 |
| htruta | by using, for example, opportunistic instances (cc tellesnobrega ) | 20:54 |
| htruta | or in the future, vm migrations | 20:54 |
| elmiko | i think the topic has rich possibilities for features we could implement | 20:55 |
| tellesnobrega | elmiko, +1 | 20:55 |
| tellesnobrega | if we push this through we have to detail a roadmap and where we want to arrive (what we need in the end) | 20:57 |
| tellesnobrega | so we can decide milestones and work an implementation plan | 20:58 |
| elmiko | definitely | 20:59 |
| *** Networkn3rd has quit IRC | 21:00 | |
| *** Networkn3rd has joined #openstack-sahara | 21:01 | |
| *** stanchan has quit IRC | 21:04 | |
| alazarev | tmckay, hi, are you here? | 21:06 |
| tmckay | alazarev, hi | 21:06 |
| alazarev | tmckay, I'm trying to make a demo with spark EDP | 21:07 |
| alazarev | tmckay, and it looks like all examples print to stdout which is lost | 21:07 |
| alazarev | tmckay, what did you use to test spark-swift integration (you did it, right?) | 21:07 |
| tmckay | alazarev, yeah, hmm. I think I read something in, don't remember if I output something. I'll take a look. And, if I can't find it, I have some colleagues doing spark stuff, I'm sure I can get some help from them. | 21:09 |
| tmckay | alazarev, I understand you want output for the demo. For your own debug, you can see stdout if you go to the spark master node in the cluster and look in /tmp/spark-edp | 21:09 |
| tmckay | all the files for you job will be there. Including a log file showing the command, so you can rerun by hand :) | 21:10 |
| tmckay | alazarev, I have to run, I' | 21:10 |
| alazarev | tmckay, I can make custom job (e.g. using "Word Count" from https://spark.apache.org/examples.html), but I thought there is something standard | 21:10 |
| tmckay | I'll look for my spark example tomorrow. Shouldn't be too hard to write something to swift | 21:10 |
| tmckay | alazarev, not sure we have a standard example -- first example was sparkPi, which pre-dated swift integration | 21:11 |
| tmckay | we probably should add a swift example | 21:11 |
| alazarev | tmckay, I'm surprized that all spark examples from jar file with spark don't write to file | 21:13 |
| alazarev | tmckay, the only exeption is web page PageRank which is a little bit overkill for demo | 21:13 |
| tmckay | alazarev, ah, found some notes. I used org.apache.spark.examples.SparkHdfsLR and created some silly data, put it in swift://tmckay.sahara/data | 21:15 |
| tmckay | I think I also used WordCount | 21:15 |
| tmckay | since the swift integration depended on authentication and use of the hadoop openstack jar, I didn't test output | 21:15 |
| tmckay | Just a read. Should be the same case. | 21:16 |
| tmckay | alazarev, you're right, though, we really should have an example with input/output | 21:16 |
| tmckay | even modifying WordCount to write results to a path instead of stdout would be good | 21:16 |
| *** rnowling has quit IRC | 21:17 | |
| *** tmckay has left #openstack-sahara | 21:17 | |
| *** jamielennox|away is now known as jamielennox | 22:20 | |
| *** barra204 has joined #openstack-sahara | 22:48 | |
| *** htruta_ has joined #openstack-sahara | 22:49 | |
| *** barra204 has quit IRC | 22:53 | |
| *** witlessb has quit IRC | 23:09 | |
| *** pino|work has quit IRC | 23:11 | |
| *** chlong has joined #openstack-sahara | 23:19 | |
| *** hdd has quit IRC | 23:26 | |
| *** hdd has joined #openstack-sahara | 23:33 | |
| *** tellesnobrega has quit IRC | 23:40 | |
| *** hdd has quit IRC | 23:55 | |
| *** tosky_ has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!