| *** bandarji has quit IRC | 00:06 | |
| openstackgerrit | A change was merged to openstack/sahara: Update documentation for Spark 1.0.0 https://review.openstack.org/98972 | 01:08 |
|---|---|---|
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/python-saharaclient: Updated from global requirements https://review.openstack.org/99094 | 01:37 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/99096 | 01:38 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara-dashboard: Updated from global requirements https://review.openstack.org/99097 | 01:38 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara-extra: Updated from global requirements https://review.openstack.org/99098 | 01:38 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara-image-elements: Updated from global requirements https://review.openstack.org/99099 | 01:38 |
| *** miqui has joined #openstack-sahara | 01:48 | |
| *** abonilla has joined #openstack-sahara | 01:55 | |
| openstackgerrit | A change was merged to openstack/sahara: Adding disconnected mode fixes to hdp plugin https://review.openstack.org/94460 | 02:07 |
| *** bandarji has joined #openstack-sahara | 02:13 | |
| *** bandarji has quit IRC | 04:56 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Make deleting transient clusters safe https://review.openstack.org/97167 | 06:25 |
| *** zhiyan_ is now known as zhiyan | 06:37 | |
| *** julienvey has joined #openstack-sahara | 06:42 | |
| *** jpich has joined #openstack-sahara | 07:01 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Remove usage of remote from HDP Instance constructor https://review.openstack.org/97502 | 07:01 |
| *** julienvey has quit IRC | 07:05 | |
| *** vrovachev has joined #openstack-sahara | 07:25 | |
| *** witlessb has joined #openstack-sahara | 07:31 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 08:08 | |
| *** julienvey has joined #openstack-sahara | 08:17 | |
| *** akuznetsov has quit IRC | 08:17 | |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 08:25 |
| openstackgerrit | Daniele Venzano proposed a change to openstack/sahara: Implement scaling for Spark clusters https://review.openstack.org/99295 | 08:47 |
| *** akuznetsov has joined #openstack-sahara | 08:57 | |
| *** akuznetsov has quit IRC | 09:01 | |
| *** akuznetsov has joined #openstack-sahara | 09:07 | |
| *** tosky has joined #openstack-sahara | 09:17 | |
| *** akuznetsov has quit IRC | 09:19 | |
| *** akuznetsov has joined #openstack-sahara | 09:42 | |
| *** julienvey has quit IRC | 09:58 | |
| *** zhiyan is now known as zhiyan_ | 10:00 | |
| *** akuznetsov has quit IRC | 10:02 | |
| *** zhiyan_ is now known as zhiyan | 10:03 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Modify integration-cleanup script to cleanup diskimage jobs https://review.openstack.org/99322 | 10:19 |
| *** zhiyan is now known as zhiyan_ | 10:28 | |
| *** themistymay has joined #openstack-sahara | 10:48 | |
| *** akuznetsov has joined #openstack-sahara | 11:00 | |
| *** themistymay has quit IRC | 11:03 | |
| *** themistymay has joined #openstack-sahara | 11:07 | |
| *** themistymay has left #openstack-sahara | 11:08 | |
| *** IvanBerezovskiy1 has joined #openstack-sahara | 11:10 | |
| *** IvanBerezovskiy has quit IRC | 11:14 | |
| *** _mattf has quit IRC | 11:14 | |
| *** _mattf has joined #openstack-sahara | 11:14 | |
| *** _mattf has quit IRC | 11:14 | |
| *** _mattf has joined #openstack-sahara | 11:14 | |
| *** julienvey has joined #openstack-sahara | 11:16 | |
| *** themistymay has joined #openstack-sahara | 11:47 | |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 11:53 |
| *** anteaya has quit IRC | 11:58 | |
| *** witlessb has quit IRC | 12:08 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Modify integration-cleanup script to cleanup diskimage jobs https://review.openstack.org/99322 | 12:16 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 12:16 |
| *** sballe has quit IRC | 12:31 | |
| tellesnobrega | _crobertsrh: hey | 12:32 |
| tellesnobrega | can you help me out with a problem regarding the patch that adds get_quotas to nova.py | 12:32 |
| tellesnobrega | ?? | 12:32 |
| *** witlessb has joined #openstack-sahara | 12:33 | |
| *** anteaya has joined #openstack-sahara | 12:40 | |
| *** _mattf is now known as mattf | 12:49 | |
| *** mattf has quit IRC | 12:50 | |
| *** mattf has joined #openstack-sahara | 12:50 | |
| *** mattf has quit IRC | 12:57 | |
| *** mattf has joined #openstack-sahara | 12:57 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Modify integration-cleanup script to cleanup diskimage jobs https://review.openstack.org/99322 | 12:57 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Add new jobs for sahara-image-elements https://review.openstack.org/98763 | 12:59 |
| *** zhiyan_ is now known as zhiyan | 13:00 | |
| *** _crobertsrh is now known as crobertsrh | 13:03 | |
| crobertsrh | Good morning tellesnobrega | 13:04 |
| tellesnobrega | crobertsrh: good morning | 13:04 |
| tellesnobrega | i think we are on a different time, sorry to bother you this early | 13:04 |
| *** tmckay has left #openstack-sahara | 13:04 | |
| crobertsrh | No problem. I'm US Eastern time (9:00 am here right now) | 13:05 |
| tellesnobrega | it pretty much the same | 13:05 |
| tellesnobrega | 10am here | 13:05 |
| tellesnobrega | about the patch, i added to option to load compute_api_v3 but there is a problem with it | 13:07 |
| crobertsrh | Ok, what is the problem that you're seeing? | 13:07 |
| tellesnobrega | when trying to get images it doesn't work, i dont know if its a novaclient bug that it doesnt recover images using api v3 | 13:08 |
| tellesnobrega | but im looking into it | 13:08 |
| crobertsrh | Hmm, that's weird. I have never tried anything with v3, so I might not be much help | 13:09 |
| tellesnobrega | ok, i will figure this out | 13:14 |
| *** shakayumi has joined #openstack-sahara | 13:28 | |
| *** abonilla has quit IRC | 13:31 | |
| *** tmckay has joined #openstack-sahara | 13:54 | |
| tellesnobrega | tmckay: every job that is created today goes through the job_manager.py? | 13:54 |
| tmckay | yes | 13:54 |
| tellesnobrega | so everything is created as an edp job | 13:55 |
| tmckay | tellesnobrega, I'm continuing to work on a refactoring, should have something soon | 13:55 |
| tmckay | yes, anything you can define as a job and launch on a cluster goes through the job manager | 13:55 |
| tellesnobrega | ok | 13:56 |
| tmckay | tellesnobrega, but I am imagining that based on information in the cluster object, the "run", "status", and "kill" methods (not the exact names) will be handed off to a different object | 13:57 |
| tmckay | at first, it can be as simple as plugin name -- "spark" will be one object, everything else will be another. Hopefully, a storm implementation fits that as well | 13:58 |
| *** _elmiko is now known as elmiko | 13:58 | |
| tmckay | later, it can be refined. Maybe the hadoop version matters, maybe the processes present on the cluster matter, etc | 13:58 |
| tellesnobrega | so by changing this object we would be able to define a specific job for a different types of processors | 13:59 |
| tmckay | Yes. It's essentially a kind of edp plugin. It will be rough at first :) | 14:00 |
| tellesnobrega | the idea is good | 14:01 |
| tmckay | so, implement "run", "status", "kill" and associate it with a cluster type and/or a job type, and we should be fine. For example, if storm doesn't have it's own plugin, but can run on vanilla 2 hadooop for instance, we might just have a Storm job type. | 14:01 |
| tellesnobrega | the code can be improved as it goes | 14:01 |
| tmckay | I don't have an idea what job types might be for Storm. | 14:02 |
| tmckay | just thinking out loud. | 14:02 |
| tellesnobrega | Storm processes real time data | 14:02 |
| tmckay | I got bogged down with a bug, and I've been reading some more about spark to make an intelligent choice, but I hope to have something posted today or tomorrow | 14:02 |
| tellesnobrega | so for storm to work we need to have a topology with necessary configurations | 14:03 |
| tellesnobrega | and data needs to be submitted to this workers continuously | 14:03 |
| tmckay | tellesnobrega, okay, so the cluster setup sounds different. What is a Storm job? Java? What executes it, some kind of Storm server? | 14:03 |
| * tmckay has a head full of spark | 14:04 | |
| tellesnobrega | there is a storm server and workers | 14:04 |
| tellesnobrega | they work like master and worker from hadoop | 14:04 |
| tmckay | okay, so it sounds like it will fit the model. We can identify a storm cluster, and we can submit jobs to the storm server in the "run" method | 14:04 |
| tmckay | tellesnobrega, so, we will need a storm plugin to launch a cluster I think. Have you thought about a provisioning plugin yet? | 14:05 |
| tmckay | That has to happen before EDP. | 14:05 |
| tmckay | Or, at the very least, one of the current plugins has to be extended to allow the installation of storm processes. | 14:06 |
| tellesnobrega | Yes. We would need this plugin to start storm, submit the code of the processors unit and the units that get the data | 14:06 |
| tellesnobrega | and start them so the processing can start | 14:07 |
| tellesnobrega | haven't thought about how the plugin would work | 14:07 |
| tellesnobrega | can you explain me a little how they work now | 14:07 |
| tellesnobrega | what they must have | 14:07 |
| tmckay | tellesnobrega, if you look in the "plugins" directory, you can find the provisioning plugin API, and you can also find implementations for vanilla, vanilla2, hdp, and now spark. They all use the API. Also, there are docs ... | 14:08 |
| * tmckay looks | 14:08 | |
| tellesnobrega | great | 14:09 |
| tellesnobrega | i will look at it now | 14:09 |
| tmckay | http://docs.openstack.org/developer/sahara/devref/plugin.spi.html | 14:09 |
| tmckay | http://docs.openstack.org/developer/sahara/userdoc/plugins.html | 14:09 |
| tmckay | tellesnobrega, that ought to give you a bunch to get started with. I have not worked on the provisioning plugins directly, but I'm learning about them | 14:10 |
| tellesnobrega | tmckay: great. I will take a look at them, but judging from what you are saying, we might have a setup that will work for storm | 14:11 |
| tellesnobrega | at least from my point of view looks like it | 14:11 |
| *** akuznetsov has quit IRC | 14:16 | |
| *** akuznetsov has joined #openstack-sahara | 14:19 | |
| *** zhiyan is now known as zhiyan_ | 14:26 | |
| *** bradd1 has quit IRC | 14:26 | |
| *** sballe has joined #openstack-sahara | 14:51 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/sahara-ci-config: Modify integration-cleanup script to cleanup diskimage jobs https://review.openstack.org/99322 | 14:52 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Remove IDH tests https://review.openstack.org/98993 | 14:54 |
| *** shakayumi has quit IRC | 14:55 | |
| crobertsrh | Howdy Sahara people! As part of the merge to horizon process, I'm getting some feedback that the term "Plugin" is undesirable. Does anyone have a better word/phrase we can use in the user interface? | 15:05 |
| elmiko | computation modules? | 15:08 |
| *** bandarji has joined #openstack-sahara | 15:09 | |
| *** bradd1 has joined #openstack-sahara | 15:15 | |
| *** bradd1 has quit IRC | 15:22 | |
| *** jpich has quit IRC | 15:23 | |
| *** themistymay has quit IRC | 15:32 | |
| *** bradd1 has joined #openstack-sahara | 15:38 | |
| *** sballe_ has joined #openstack-sahara | 15:40 | |
| *** sballe has quit IRC | 15:43 | |
| *** akuznetsov has quit IRC | 15:53 | |
| *** akuznetsov has joined #openstack-sahara | 15:59 | |
| *** IvanBerezovskiy1 has left #openstack-sahara | 16:09 | |
| *** bradd1 has quit IRC | 16:18 | |
| *** themistymay has joined #openstack-sahara | 16:19 | |
| *** julienvey has quit IRC | 16:32 | |
| *** themistymay1 has joined #openstack-sahara | 16:37 | |
| *** themistymay has quit IRC | 16:39 | |
| *** bradd1 has joined #openstack-sahara | 16:54 | |
| *** mattf is now known as _mattf | 17:27 | |
| *** _mattf is now known as mattf | 17:42 | |
| *** mattf has quit IRC | 17:43 | |
| *** mattf has joined #openstack-sahara | 17:43 | |
| *** themistymay1 has quit IRC | 17:48 | |
| *** themistymay has joined #openstack-sahara | 17:59 | |
| *** themistymay has quit IRC | 18:01 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/python-saharaclient: Don't set json content type for non-json data https://review.openstack.org/97642 | 18:13 |
| *** tosky has quit IRC | 18:34 | |
| tmckay | crobertsrh, is there a UI walkthrough guide anywhere? | 19:03 |
| crobertsrh | There is a userguide in the sahara docs | 19:03 |
| crobertsrh | possibly slightly out of date | 19:03 |
| tmckay | yeah, I saw that. I thought I remembered a step through of creating data sources or something like that. Didn't see it currently | 19:04 |
| tmckay | crobertsrh, I ask because I wonder if for https://review.openstack.org/#/c/92833 we want to mention the swift/hdfs prefix stuff in the guide. Or, maybe in augmented help text on the screen? | 19:05 |
| crobertsrh | Maybe another bug to add some text in both places? | 19:05 |
| tmckay | I added my comment. Do you think it needs words? I could go either way, but extra words couldn't hurt | 19:06 |
| tmckay | if so, another bug is not a bad idea | 19:06 |
| crobertsrh | Another bug is my preference. I think that we could probably use a good overhaul of all the help and UI docs | 19:09 |
| tmckay | ack. I just don't know if users would be confused. The hdfs paths are the tricky ones, because the semantics are different | 19:11 |
| elmiko | +1 to making a new bug report | 19:12 |
| dmitryme | tmckay: http://sahara.readthedocs.org/en/latest/horizon/dashboard.user.guide.html#data-sources | 19:15 |
| *** ylobankov has quit IRC | 19:18 | |
| tmckay | dmitryme, ah, dashboard stuff. I was looking at docs.openstack.org/developer/sahara | 19:19 |
| tmckay | crobertsrh, looks like this stuff could just be updated ^^ | 19:20 |
| crobertsrh | Yeah, that part needs a bit of love | 19:22 |
| *** bradd1 has quit IRC | 19:26 | |
| *** bradd1 has joined #openstack-sahara | 19:28 | |
| *** mattf is now known as _mattf | 19:40 | |
| *** alazarev has joined #openstack-sahara | 19:45 | |
| *** _mattf is now known as mattf | 19:55 | |
| *** crobertsrh is now known as _crobertsrh | 20:48 | |
| *** tmckay has quit IRC | 21:23 | |
| *** elmiko is now known as _elmiko | 21:27 | |
| *** bradd1 has quit IRC | 21:35 | |
| *** sballe has joined #openstack-sahara | 21:52 | |
| *** sballe_ has quit IRC | 21:52 | |
| *** akuznetsov has quit IRC | 22:07 | |
| *** bradd1 has joined #openstack-sahara | 22:09 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Fixed remote call in external HDFS configuration method https://review.openstack.org/99491 | 22:15 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Fixed /etc/hosts update for external hdfs https://review.openstack.org/99492 | 22:15 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Minor improvements in provisioning engine https://review.openstack.org/99497 | 22:35 |
| *** witlessb has quit IRC | 22:37 | |
| *** bandarji has quit IRC | 22:46 | |
| *** bradd1 has quit IRC | 22:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!