| *** openstack has joined #openstack-sahara | 00:19 | |
| *** IlyaE has joined #openstack-sahara | 00:20 | |
| *** IlyaE has quit IRC | 00:26 | |
| *** alazarev has quit IRC | 00:33 | |
| *** bradd1 has joined #openstack-sahara | 01:47 | |
| *** elmiko has quit IRC | 02:51 | |
| *** witlessb has joined #openstack-sahara | 02:56 | |
| *** alazarev has joined #openstack-sahara | 04:01 | |
| *** alazarev has quit IRC | 04:48 | |
| *** Ch00k has joined #openstack-sahara | 05:24 | |
| *** IlyaE has joined #openstack-sahara | 05:31 | |
| *** alazarev has joined #openstack-sahara | 05:43 | |
| openstackgerrit | Jenkins proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/82719 | 06:07 |
|---|---|---|
| *** alazarev has quit IRC | 06:14 | |
| *** Ch00k has quit IRC | 06:38 | |
| *** bogdando has quit IRC | 07:05 | |
| openstackgerrit | A change was merged to openstack/sahara: Rename missed env variables in oslo code https://review.openstack.org/82582 | 07:07 |
| *** IlyaE has quit IRC | 07:14 | |
| *** Ch00k has joined #openstack-sahara | 07:53 | |
| *** bogdando has joined #openstack-sahara | 08:03 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: WIP Add EDP integration tests for vanilla 2 plugin https://review.openstack.org/82733 | 08:05 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add EDP support for Vanilla 2 plugin https://review.openstack.org/82474 | 08:05 |
| *** dmitryme has joined #openstack-sahara | 08:23 | |
| *** tmckay has quit IRC | 08:34 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add integration tests for vanilla 2 plugin https://review.openstack.org/77089 | 08:35 |
| *** witlessb has joined #openstack-sahara | 08:56 | |
| *** tmckay has joined #openstack-sahara | 09:15 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/sahara: Don't autocreate db on start https://review.openstack.org/82750 | 09:29 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add EDP support for Vanilla 2 plugin https://review.openstack.org/82474 | 09:35 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Remove agent remote https://review.openstack.org/82757 | 09:49 |
| *** ruhe has left #openstack-sahara | 10:02 | |
| *** tosky has joined #openstack-sahara | 10:10 | |
| *** Ch00k has quit IRC | 10:11 | |
| *** Ch00k has joined #openstack-sahara | 10:13 | |
| *** eanxgeek|log is now known as eanxgeek | 10:52 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add EDP support for Vanilla 2 plugin https://review.openstack.org/82474 | 11:35 |
| *** _crobertsrh has quit IRC | 11:38 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 12:11 | |
| *** tmckay has left #openstack-sahara | 12:30 | |
| *** tmckay has joined #openstack-sahara | 12:31 | |
| *** dmitryme has quit IRC | 12:39 | |
| *** tosky has quit IRC | 12:47 | |
| *** _mattf is now known as mattf | 12:55 | |
| *** elmiko has joined #openstack-sahara | 12:59 | |
| *** crobertsrh has joined #openstack-sahara | 13:06 | |
| *** dmitryme has joined #openstack-sahara | 13:11 | |
| SergeyLukjanov | tmckay, ping | 13:14 |
| tmckay | hi | 13:14 |
| tmckay | SergeyLukjanov, pong | 13:16 |
| SergeyLukjanov | tmckay, morning! | 13:19 |
| tmckay | morning | 13:19 |
| SergeyLukjanov | tmckay, are you planning to fix https://bugs.launchpad.net/sahara/+bug/1273661 and https://bugs.launchpad.net/sahara/+bug/1272452 this week? | 13:19 |
| SergeyLukjanov | if not - I'm ready to take a look on them / reassign to someone else | 13:20 |
| SergeyLukjanov | RC1 will be next week, so, we need to fix such stuff | 13:20 |
| SergeyLukjanov | tmckay ^^ | 13:20 |
| mattf | there's a deeper issue there, that the credentials are being stored in the first place | 13:22 |
| tmckay | SergeyLukjanov, reassigning them is probably a good idea. btw, mattf had concerns about having the credentials stored in the database, and suggested that fixing the returns from REST really doesn't address the issue fully | 13:22 |
| tmckay | what he said :) | 13:22 |
| mattf | simply filtering them is really just papering over the issue | 13:22 |
| mattf | truly, someone should have smacked me harder when i deleted them from the cli output | 13:23 |
| mattf | those bugs just hide that we have a deeper security issue to address | 13:23 |
| * mattf end rant | 13:23 | |
| tmckay | well, you could argue for a database on a server where nobody has access but root | 13:23 |
| tmckay | but that counts on deployment to cover it | 13:23 |
| SergeyLukjanov | tmckay, it fixes these issues, but it's not fixing architecture issue | 13:24 |
| mattf | i've never, has anyone used ssl to communicate w/ the service? | 13:24 |
| tmckay | I don't think I have | 13:24 |
| SergeyLukjanov | mattf, agreed with that it's a bigger problem | 13:24 |
| crobertsrh | I have not | 13:24 |
| SergeyLukjanov | mattf, I think you've created blueprint to not store credentials | 13:25 |
| SergeyLukjanov | mattf, /me not | 13:25 |
| SergeyLukjanov | (re ssl) | 13:25 |
| mattf | it's on my list of big things for juno: horizon merge, security, stable api | 13:25 |
| SergeyLukjanov | mattf, I've talked with David Lyle (ptl of Horizon) yesterday and he said that it's quite common approach for merging our plugin to horizon | 13:26 |
| SergeyLukjanov | and we could start doing it right after the horizon master will be open for Juno dev | 13:27 |
| SergeyLukjanov | (next week I think) | 13:27 |
| tmckay | SergeyLukjanov, okay, I'll fix those this week. One is basically a copy of one already done | 13:27 |
| SergeyLukjanov | tmckay, yup, thank you | 13:27 |
| tmckay | That should take just a few minutes. | 13:27 |
| SergeyLukjanov | tmckay, yeah | 13:27 |
| tmckay | The other is slightly harder, but not much I think. | 13:27 |
| mattf | crobertsrh, pls toss filing a horizon merge bp on your queue | 13:28 |
| crobertsrh | ack | 13:28 |
| SergeyLukjanov | crobertsrh, please, add me to this bp | 13:28 |
| crobertsrh | will do | 13:28 |
| SergeyLukjanov | anyway it's low prio comparing to releasing Icehouse ;) | 13:29 |
| SergeyLukjanov | heh, renaming was quite difficult in the end of cycle but we've integrated and it's really great... | 13:29 |
| mattf | looks like we just have ci and puppet remaining | 13:32 |
| *** bradd1 has quit IRC | 13:33 | |
| mattf | it was a great cross team effort and example of collaboration | 13:33 |
| mattf | s/was/is/ since it's not done yet! | 13:33 |
| mattf | bbiab | 13:33 |
| *** mattf is now known as _mattf | 13:33 | |
| crobertsrh | SergeyLukjanov: For https://review.openstack.org/#/c/82539/ in the compatibility module, there is some support for a folsom url. Should I clean that up as part of this compatibility change, or file another CR to clean it up separately? | 13:37 |
| SergeyLukjanov | crobertsrh, could you please cleanup < havana compat separately? | 13:38 |
| crobertsrh | no problem | 13:38 |
| SergeyLukjanov | crobertsrh, thx! | 13:38 |
| SergeyLukjanov | _mattf, yup, it was awesome work done (mostly) | 13:39 |
| openstackgerrit | Chad Roberts proposed a change to openstack/sahara-dashboard: Fixing sahara image registry links https://review.openstack.org/82539 | 13:39 |
| *** Ch00k has quit IRC | 13:41 | |
| *** Ch00k has joined #openstack-sahara | 13:42 | |
| *** rhodgin has joined #openstack-sahara | 13:43 | |
| crobertsrh | SergeyLukjanov: For filing the dashboard merge bp, should I file it against sahara or horizon? I'm guessing horizon, but I don't want to be wrong. | 13:44 |
| SergeyLukjanov | crobertsrh, against horizon | 13:45 |
| crobertsrh | thx | 13:45 |
| SergeyLukjanov | crobertsrh, but you can duplicate it to sahara too, because it means removing sahara-dashboard | 13:45 |
| crobertsrh | Ok, I'll do both | 13:45 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Remove agent remote https://review.openstack.org/82757 | 13:49 |
| *** tosky has joined #openstack-sahara | 14:08 | |
| *** bradd1 has joined #openstack-sahara | 14:09 | |
| *** alazarev has joined #openstack-sahara | 14:32 | |
| *** IlyaE has joined #openstack-sahara | 15:00 | |
| *** _mattf is now known as mattf | 15:11 | |
| *** ErikB has joined #openstack-sahara | 15:13 | |
| *** alazarev has quit IRC | 15:18 | |
| *** Ch00k_ has joined #openstack-sahara | 15:27 | |
| *** Ch00k has quit IRC | 15:30 | |
| *** alazarev has joined #openstack-sahara | 16:21 | |
| *** alazarev has quit IRC | 16:35 | |
| SergeyLukjanov | crobertsrh, I've asked David to triage dashboard merging blueprint, thank you | 16:41 |
| crobertsrh | great! | 16:42 |
| SergeyLukjanov | crobertsrh, heh, it's already high prio for the next cycle :) | 16:42 |
| SergeyLukjanov | yay! we can block them | 16:43 |
| crobertsrh | wow, now the pressure is on! | 16:43 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Filter credentials in jobs returned from REST api https://review.openstack.org/82863 | 16:44 |
| *** alazarev has joined #openstack-sahara | 16:48 | |
| mattf | crobertsrh, what are you going to do w/ your newfound power over horizon? | 16:58 |
| crobertsrh | I'm looking to add more obscure JavaScript to everything. | 17:00 |
| crobertsrh | Of course, I'm always looking to do that | 17:01 |
| elmiko | lol | 17:01 |
| mattf | lol | 17:02 |
| SergeyLukjanov | great intention! | 17:04 |
| *** Ch00k_ has quit IRC | 17:07 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Filter 'fields' from JobExecutions returned from REST api https://review.openstack.org/82883 | 17:35 |
| tmckay | SergeyLukjanov, mattf, ^^, no mechanism currently in Sahara to really do anything but leave out job_configs completely on wrapped JobExecutions. Acceptable? Possible alternative is to add a Resource type for job_configs, then add a mechanism for filtering callbacks so that specific configs can be screened out while leaving the rest. Sounds like overkill to me. | 17:37 |
| mattf | sounds like bending over backward to open a paper bag | 17:39 |
| tmckay | Not sure why you would need job_configs reported in a job | 17:41 |
| tmckay | At least for Oozie, you can find all the info you want from the Oozie console anyway :) | 17:42 |
| tmckay | which is maybe another problem | 17:42 |
| dmitryme | tmkay: we can add a _custom_filter() function to resource which we can call in _to_dict() | 17:42 |
| dmitryme | we can then override it in JobExecution to filter only sensitive configs | 17:43 |
| dmitryme | s/resource/Resource/ | 17:43 |
| dmitryme | I meant the class | 17:43 |
| tmckay | dmitryme, yes, wasn't sure if we wanted to invent filtering features at this point ... I see that as a feature. | 17:43 |
| tmckay | And we're frozen ;-) | 17:44 |
| dmitryme | tmkay: if you mean feature freeze, then I think it does not count here: I would not call a feature something which is not user-facing | 17:45 |
| tmckay | dmitryme, okay, feel free to review and comment the above 82883. I'll wait for feedback before doing anything else | 17:46 |
| dmitryme | tmkay: ok, sure | 17:47 |
| tmckay | dmitryme, I think of "feature" as anything that could destabilize the code | 17:47 |
| tmckay | always a judgement call on where to draw the line with bug fixes | 17:47 |
| dmitryme | tmckay: probably you are right | 17:48 |
| dmitryme | that Resource class is a nasty thing, so probably it is better not to touch it until we release Icehouse | 17:49 |
| *** crobertsrh has quit IRC | 17:50 | |
| tmckay | dmitryme, :) and as mattf pointed out we need an overall security review, I think. How we store credentials, does it matter that credentials are visible in the Oozie console for swift access, is there a way we can get around that (maybe encrypt values with a keypair or something) | 17:53 |
| dmitryme | tmckay: agree, that would be a good topic for design summit | 17:54 |
| tmckay | Good idea. I think Sergey sent an email about topics ... | 17:55 |
| SergeyLukjanov | http://summit.openstack.org | 17:56 |
| SergeyLukjanov | for proposing topics for summit | 17:56 |
| SergeyLukjanov | sahara topic :) | 17:56 |
| * mattf is waiting for the email from SergeyLukjanov that says "ok folks, deadline is tomorrow" | 17:56 | |
| mattf | SergeyLukjanov, how many slots do we have? | 17:56 |
| SergeyLukjanov | mattf, design summit layout is under discussion now | 17:57 |
| SergeyLukjanov | mattf, heh, I'm waiting for such email from myself too :) | 17:57 |
| mattf | lol | 17:57 |
| SergeyLukjanov | mattf, we'll have more slots than in HKG | 17:57 |
| SergeyLukjanov | I think that the exact number will be known in a few days | 17:58 |
| mattf | all your slot are belong to us? | 17:58 |
| SergeyLukjanov | what do you mean? | 17:58 |
| mattf | wow, bowie was young on the young americans album | 17:59 |
| * mattf googles | 17:59 | |
| mattf | oh man, it's an old meme | 17:59 |
| mattf | http://en.wikipedia.org/wiki/All_your_base_are_belong_to_us | 18:00 |
| * mattf needs to update his material | 18:00 | |
| * SergeyLukjanov reading | 18:00 | |
| elmiko | well played mattf | 18:00 |
| * tmckay chuckles | 18:01 | |
| mattf | i'll take your pity chuckle | 18:01 |
| SergeyLukjanov | re slots - I think 6-7 | 18:02 |
| SergeyLukjanov | currently, 7 in draft layout | 18:02 |
| tmckay | ooo, alazarev put up the error reporting topic, +++1 | 18:02 |
| tmckay | I was just ranting about that yesterday in another chat | 18:02 |
| alazarev | another big topic is workflow management | 18:03 |
| alazarev | now the only thing you can do with hanged (or error) cluster is to delete it | 18:04 |
| alazarev | this is not something users expect | 18:04 |
| tmckay | alazarev, could be the same idea, I was thinking of "What now?" with EDP. For instance, I think we need at least raw Oozie workflows if users are going to submit through Sahara. Otherwise, to do something complicated, I think they will go straight to the cluster command line on a node | 18:04 |
| tmckay | oh, okay, different topic | 18:04 |
| tmckay | brb | 18:04 |
| alazarev | +1 on oozie workflow templates | 18:05 |
| mattf | i'm more of a get-better-insight than go-more-general, they often conflict | 18:06 |
| mattf | well, at least for edp, which is to streamline/simplify the user experience | 18:06 |
| mattf | imho, the general tools are good for experts, but our audience is non-experts | 18:07 |
| *** Ch00k has joined #openstack-sahara | 18:07 | |
| mattf | we could also simplify the template creation process. right now all the validation information happens at the end of the process. it could happen at the beginning | 18:08 |
| mattf | "make master node" -> "make worker node" -> how many workers do you want -> done ? | 18:09 |
| mattf | i bet that covers 95% of the template creations | 18:09 |
| mattf | no exposure to processes, just high level roles | 18:10 |
| mattf | experts say "nah, forget this, i'll work w/ processes and make ha etc etc" | 18:10 |
| mattf | a user shouldn't have to wonder if they need to add a mapreduce client to all nodes | 18:10 |
| alazarev | mattf: now we have mixed approach and this leads to user confusion. cluster template is "full template for cluster" in UI and "partially filled cluster info" for backend | 18:10 |
| mattf | it's like we've exposed the implementation to the user. we can keep it around for the expert, but should have a simpler workflow for everyone else (most of the users) | 18:11 |
| alazarev | we should either allow user to overwrite values in templates during cluster creation (more complex way) or add validation to templates and simplify the whole process (your suggestion) | 18:13 |
| mattf | the fewer decisions we ask the user to make the better | 18:14 |
| mattf | an example: "make master node" -> "make worker node" -> how many workers do you want -> done ? | 18:14 |
| tosky | simplified workflow in the UI with a potential for separate "advanced options" if the user really want to tune (a small button :) | 18:14 |
| mattf | why do we even ask if htey want to make a master or a worker... | 18:14 |
| alazarev | probably it is possible to keep both approaches | 18:14 |
| mattf | why not ask "how much memory and disk do you want?" | 18:15 |
| alazarev | mattf: I like the idea, wizard with one screen that creates the whole cluster | 18:16 |
| mattf | each plugin knows it's default config for master & worker. we can start by evenly splitting across instance sizes, and optimize sizes in the future. | 18:16 |
| mattf | also "how much memory and disk" shouldn't be an open ended question, that's cognitive effort on the user's part | 18:17 |
| mattf | we can get the user's quota and present them w/ sizes: small, medium, large | 18:17 |
| mattf | true, your medium != my medium, but what's that matter? | 18:18 |
| alazarev | mattf: quota often doesn't correlate with reality | 18:19 |
| *** Ch00k has quit IRC | 18:19 | |
| mattf | that's a different problem tho, right? quota does constrain what the user can do. | 18:19 |
| alazarev | mattf: but we could ask flavor and required characteristics, that's all sahara need to create cluster | 18:19 |
| mattf | i'm suggesting we guess flavor based on available quota and desired size of the cluster/job | 18:20 |
| mattf | mguess == evenly divide | 18:21 |
| mattf | maybe we can do better than that in the future, like making master > worker if # worker over a threshhold | 18:21 |
| mattf | or at least different shape for master & workers | 18:21 |
| alazarev | mattf: it's implementation details and could be discussed, I like the idea in general: one screen with few questions = running cluster | 18:22 |
| mattf | we'll get that spark plugin (or similar) sometime and if you're running a spark cluster the shape of disk would be different than for mapreduce | 18:22 |
| *** Ch00k has joined #openstack-sahara | 18:25 | |
| tmckay | mattf, alazarev, so maybe this is one big topic after all, workflow for cluster and job creation, and maybe some subtopics like how do we expose the full power of things we've wrapped (like Oozie) | 18:35 |
| *** alazarev has quit IRC | 18:35 | |
| tmckay | I haven't been to one of these before, is there a scope limit on a topic? | 18:35 |
| SergeyLukjanov | ++ for super simple flows for non-expert users | 18:36 |
| tmckay | What is the session duration? | 18:36 |
| SergeyLukjanov | tmckay, topics are 40 min long | 18:36 |
| aignatov | tmckay: 40-50 mins, but we can discuss it after and before session as well :) | 18:36 |
| * mattf expects SergeyLukjanov is going to stop putting him on his reviews soon | 18:37 | |
| tmckay | can a topic run over multiple sessions, or is it always 1 to 1? | 18:37 |
| *** alazarev has joined #openstack-sahara | 18:37 | |
| aignatov | I think it can | 18:37 |
| aignatov | few possible topics from me - security in Hadoop & OpenStack | 18:39 |
| SergeyLukjanov | mattf, :) | 18:39 |
| aignatov | and pluggable mechanism of EDP ;) | 18:40 |
| aignatov | because I think in the future EDP can't be dependent only on Oozie | 18:40 |
| mattf | aignatov, couldn't we teach oozie about spark? | 18:41 |
| aignatov | actually of spark plugin will be landed I don't know how edp will work on it :) | 18:41 |
| mattf | it won't. having a spark plugin is not really the right arch imho. | 18:41 |
| mattf | it's an ok stopgap, but ultimately spark is another process, like mapreduce, that runs on a hadoop cluster | 18:42 |
| mattf | it should be something that a distro plugin provides/deploys | 18:42 |
| aignatov | mattf: yes, so some changes are needed in EDP to support new data processing flows in non-hadoop plugins | 18:44 |
| aignatov | non-pure-hadoop plugins :) | 18:44 |
| mattf | assuming we actually get the spark plugin in, we couldn't really claim we support spark until it's exposed through edp, imho | 18:45 |
| *** alazarev has quit IRC | 18:45 | |
| mattf | did you have something else in mind when oyu mentioned oozie might not be our only interface into a cluster? | 18:45 |
| tmckay | mattf, aignatov, alazarev, SergeyLukjanov, http://summit.openstack.org/cfp/details/143, tried to capture the earlier discussion | 18:46 |
| aignatov | mattf: I think we could claim about plugin support if we can deploy its processes :) | 18:46 |
| tmckay | let me know if you think I should edit the description | 18:46 |
| tmckay | oh my, all one line | 18:46 |
| aignatov | tmckay: thx | 18:47 |
| SergeyLukjanov | I think that we'll need to have some additional meetings after the summit to discuss topics for summit and prepare base for them | 18:47 |
| SergeyLukjanov | after the release* | 18:47 |
| aignatov | after the summit? | 18:47 |
| aignatov | ok | 18:48 |
| aignatov | yes | 18:48 |
| tmckay | k, that looks a little better | 18:48 |
| aignatov | but we can brainstorm here, in irc :) | 18:48 |
| aignatov | as for security, the some initial idea that we should think not only for hiding credentials but some tenant isolations because now anyone who know public url of some ooze deployed can have access to it :) | 18:49 |
| tmckay | yes | 18:50 |
| SergeyLukjanov | aignatov, sure, right now for example :) | 18:50 |
| SergeyLukjanov | aignatov, yeah, that's an issue too | 18:51 |
| aignatov | some who has its own ooze client can run job on another's Oozie xD | 18:51 |
| *** tosky has quit IRC | 18:55 | |
| *** crobertsrh has joined #openstack-sahara | 18:56 | |
| aignatov | one more topic: "do we really need 'use_neutron' option? :)" | 18:57 |
| mattf | aignatov, nah, i'll put a patch up for that before we cut I | 18:59 |
| * mattf kids at SergeyLukjanov | 18:59 | |
| SergeyLukjanov | aignatov, nope, we could just auto discover it in catalog | 18:59 |
| SergeyLukjanov | mattf, heh | 19:00 |
| mattf | aignatov, we probably want a use_nova_networking | 19:00 |
| aignatov | mattf: I think the same as SergeyLukjanov said, we can auto discover networking | 19:01 |
| mattf | that sounds cool, kinda hurts my head a bit tho | 19:02 |
| aignatov | one more topic, actually old theme: "Supporting new data sources and edp over it - Ceph, HBase, Cassandra, GlusterFS" | 19:04 |
| mattf | oldie by goodie | 19:05 |
| mattf | err but goodie | 19:05 |
| aignatov | I don't understand these expressions :) | 19:06 |
| aignatov | is it kinda one more meme? | 19:06 |
| dmitryme | I think mattf means that we already talked about it | 19:06 |
| dmitryme | right? | 19:06 |
| mattf | you should use some russian idioms | 19:06 |
| * mattf will google them and learn | 19:07 | |
| mattf | dmitryme, mostly. it may be an old topic but it's a good topic | 19:07 |
| aignatov | dmitryme, mattf, got it, yes, we talked about it right after 0.2 version released :) | 19:08 |
| SergeyLukjanov | it'll be quite weird if we'll start writing them on russian :) | 19:08 |
| dmitryme | mattf, regarding 'google them and learn', you can't even imaging how often we use that approach :-) | 19:08 |
| aignatov | I don't want write these idioms because most of them will contain "*******" :) | 19:09 |
| mattf | lol | 19:09 |
| dmitryme | aignatov: not true | 19:10 |
| aignatov | dmitryme: I'm kidding | 19:10 |
| mattf | aignatov, nice | 19:11 |
| aignatov | dmitryme: right, during irc conversation I use google 1 time per 10-20 lines | 19:11 |
| aignatov | avg | 19:11 |
| *** ErikB has quit IRC | 19:12 | |
| SergeyLukjanov | I'm using three finder click for dictionary/wiki | 19:15 |
| *** alazarev has joined #openstack-sahara | 19:16 | |
| mattf | aignatov, SergeyLukjanov tells me that you might have the hong kong demo pig script. can i have it? | 19:17 |
| mattf | i'm demo'ing sahara at devnation (red hat summit), and haven't had time to come up w/ my own | 19:17 |
| SergeyLukjanov | mattf, you should torture aignatov for it | 19:17 |
| mattf | i'd like to use yours as backup | 19:17 |
| *** openstackgerrit has quit IRC | 19:18 | |
| mattf | hmm, not sure if torture is the right word | 19:18 |
| aignatov | mattf: sure | 19:18 |
| *** openstackgerrit has joined #openstack-sahara | 19:18 | |
| tmckay | lol | 19:18 |
| tmckay | I don't often torture my colleagues, but when I do ... | 19:19 |
| mattf | lol | 19:19 |
| SergeyLukjanov | mattf, it was a joke, not so bad as I think, but looks like nope | 19:19 |
| tmckay | SergeyLukjanov, no, it was quite funny | 19:19 |
| *** ErikB2 has joined #openstack-sahara | 19:20 | |
| * mattf just missed it | 19:20 | |
| aignatov | mattf: http://paste.openstack.org/show/74285/ ^^ | 19:22 |
| aignatov | btw, mattf, tmckay SergeyLukjanov we need to create new edp example for summit's demo | 19:23 |
| aignatov | more impressive or/and funny as it was before :) | 19:25 |
| SergeyLukjanov | yup, something that could be visualized | 19:25 |
| aignatov | SergeyLukjanov: exactly | 19:25 |
| mattf | i agree | 19:26 |
| mattf | my talk is on 16 apr | 19:26 |
| mattf | so we need that new demo soon! | 19:26 |
| mattf | s/we/i/ | 19:26 |
| aignatov | so everyone are welcome for ideas | 19:26 |
| aignatov | so I can start implementing this asap in Pig, Jar, Java action, Streaming MapReduce or Hive, tmckay, ok, you know these things :) | 19:28 |
| tmckay | aignatov, a little bit | 19:29 |
| tmckay | aignatov, my Pig and Hive is pretty thin | 19:29 |
| mattf | aignatov, i was thinking an interactive spark demo would be slick, but...that's tricky atm | 19:29 |
| aignatov | tmckay: the same here :) | 19:29 |
| aignatov | mattf: yes, because we don't have spark in sahara | 19:30 |
| tmckay | I have an HDP big data book set from a training course at the office, there are some examples in there of predictor algorithms. You know, people who bought this item also bought this other one. Or you might like this movie, since you rented that one | 19:31 |
| *** mattf is now known as _mattf | 19:32 | |
| tmckay | I think those were all map reduce jobs | 19:33 |
| tmckay | aignatov, does mirantis have a big data training arm too? I wonder if we could get some stuff from the HWX or mirantis training people. I'm no data scientist :) | 19:36 |
| tmckay | just like I wasn't a chemist or a physicist working on NMRs and CT scanners | 19:36 |
| aignatov | tmckay: no, there are no bigdata courses at Mirantis. It is totally OpenStack oriented atm, so we are rest "Big data scientists" | 19:38 |
| tmckay | ah, okay. Well, maybe our HWX friends can help. | 19:39 |
| tmckay | just wondering if there is something that already exists that we can just turn into a Sahara job | 19:39 |
| aignatov | hm, good idea, HW runs its own Hadoop summit, so I guess there are tons of great demos :) | 19:40 |
| tmckay | and then maybe do something cool with the output to render it | 19:40 |
| aignatov | so we can "torture" them xD | 19:40 |
| tmckay | lol | 19:40 |
| tmckay | The new open source way | 19:40 |
| tmckay | Open it .. .or else | 19:40 |
| SergeyLukjanov | I like this idea | 19:45 |
| tmckay | the torture? ;-) | 19:46 |
| elmiko | lol | 19:48 |
| aignatov | let's rename sahara to... | 19:49 |
| aignatov | torture | 19:49 |
| tmckay | llol, now that would get a lot of users | 19:49 |
| tmckay | TortureStack | 19:49 |
| *** IlyaE has quit IRC | 19:49 | |
| aignatov | ahah, lol | 19:49 |
| tmckay | "Running a job never felt so bad" | 19:49 |
| aignatov | tmckay: haha, stop making me laughing | 19:51 |
| tmckay | aignatov, yeah, I think we've all gone a bit crazy. Probably the renaming... | 19:51 |
| ErikB2 | For demos - I am trying to think of something visually pleasing... | 19:53 |
| tmckay | ErikB2, in the HWX hadoop training class, we did movie selection predictions | 19:54 |
| elmiko | is tmckay ready for another renaming so soon? | 19:54 |
| ErikB2 | Hadoop is not the most exciting thing when it comes to glitz per se. How about showing how to run something like the HiBench benchmark on a Sahara provisioned cluster | 19:54 |
| tmckay | elmiko, no, once is enough. Next time I say we do global search and replace and cross our fingers | 19:55 |
| elmiko | HA! | 19:55 |
| elmiko | can we do that on a friday at like 4:45pm? | 19:55 |
| aignatov | ErikB2: I guess that we will no have such resources to show benchmarking in the live demo | 19:55 |
| ErikB2 | tmckay, I am not familiar with this, but this is likely something we could pull together (or out of the training class perhaps) | 19:56 |
| tmckay | ErikB2, hmm, it might have been in the 'R' part of the data science portion... I'll have to look in the book | 19:57 |
| ErikB2 | What do we want to demo/show - that we have a functioning hadoop cluster that we can do stuff on? It should definitely be Hadoop 2 based demo. | 19:57 |
| ErikB2 | tmckay, ok. If you have the name of a lab or otherwise I can chase down. I have not had the luxury of taking our training. ;-) | 19:58 |
| tmckay | hmm, actually there's another shadowman guy who might have something... I'll try to drag him in here | 19:58 |
| aignatov | It should be like calculating some useful or funny things or analysing text and then visualise it somehow | 19:59 |
| tmckay | jvyas, I think he's in here sometimes | 19:59 |
| tmckay | ErikB2, ack, I'll take a look through the book this evening or tomorrow morning | 20:00 |
| ErikB2 | I will try to come back with some concrete ideas for demos tomorrow as well. | 20:00 |
| aignatov | I think we don't needed overload openstack users with any difficult calculations demo or benchmarking, because OS is not so close to hadoop, I'm afraid that typical OS users will not or will not want deeply understand such things | 20:01 |
| SergeyLukjanov | ErikB2, thanks for helping | 20:02 |
| SergeyLukjanov | aignatov, yup, some EDP job + nice visualization will be great | 20:02 |
| aignatov | ErikB2: yes, that's would be great! | 20:02 |
| ErikB2 | I don't want to demo EDP... :-) | 20:03 |
| SergeyLukjanov | ErikB2, why not? | 20:03 |
| aignatov | ErikB2: I will, I just need an example which will be loved by all in our summit talk | 20:04 |
| ErikB2 | just joshing, although I do find EDP somewhat prickly on very easy to use. I know that we will discuss in ATL. | 20:07 |
| *** IlyaE has joined #openstack-sahara | 20:07 | |
| ErikB2 | on=and not | 20:07 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/sahara: Validate data sources reference different resources https://review.openstack.org/82910 | 20:16 |
| *** dmitryme has quit IRC | 20:16 | |
| *** dmitryme has joined #openstack-sahara | 20:16 | |
| aignatov | regardless edp,we just need a good mapreduce example for demo on summit :) because pi job and my last demo are not in the trend ;) | 20:17 |
| *** alazarev has quit IRC | 20:20 | |
| *** dmitryme has quit IRC | 20:21 | |
| *** eanxgeek is now known as eanxgeek|log | 20:23 | |
| *** alazarev has joined #openstack-sahara | 20:37 | |
| *** ErikB2 has quit IRC | 20:46 | |
| elmiko | crobertsrh: if i am using neutron do i need to specify AUTO_ASSIGNMENT_ENABLED in my local_settings? | 20:48 |
| crobertsrh | elmiko: I have never used neutron, so I can't say for sure | 20:49 |
| elmiko | k | 20:49 |
| elmiko | i'm guessing that i don't, but who knows :) | 20:49 |
| *** dmitryme has joined #openstack-sahara | 20:51 | |
| crobertsrh | Looks like it defaults to true if you don't set it | 20:51 |
| crobertsrh | I really should try out neutron at some point | 20:52 |
| crobertsrh | Let me know what you find | 20:54 |
| elmiko | i'm getting very close to having trunk sahara+dashboard+horizon running on the rdo icehouse stack | 20:55 |
| elmiko | just working out a few last kinks | 20:55 |
| crobertsrh | Nice! | 20:55 |
| elmiko | and i'll have a doc to share when i get it working | 20:55 |
| crobertsrh | Even better | 20:55 |
| elmiko | i've got the dashboard into horizon, now i'm getting some flack about the /sahara endpoint not connecting, but i think i might have an issue with the ip address or something | 20:56 |
| elmiko | yay, working :) | 20:57 |
| elmiko | now to start pushing images | 20:57 |
| *** IlyaE has quit IRC | 21:01 | |
| *** ErikB2 has joined #openstack-sahara | 21:03 | |
| openstackgerrit | Alexander Ignatov proposed a change to openstack/sahara: Add fixed and floating IPs discovery via neutron https://review.openstack.org/82919 | 21:06 |
| *** crobertsrh has quit IRC | 21:10 | |
| elmiko | tmckay: ping | 21:19 |
| elmiko | why doesn't sahara-image-elements put diskimage-create.sh into the path during `python setup.py install` ? | 21:32 |
| *** dmitryme has quit IRC | 21:34 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: [IDH] Fixed cluster scale down https://review.openstack.org/78876 | 21:34 |
| *** witlessb has quit IRC | 21:35 | |
| *** Ch00k has quit IRC | 21:36 | |
| *** IlyaE has joined #openstack-sahara | 21:43 | |
| *** ErikB2 has quit IRC | 22:01 | |
| *** ErikB2 has joined #openstack-sahara | 22:08 | |
| *** tmckay has left #openstack-sahara | 22:12 | |
| *** alazarev has quit IRC | 22:16 | |
| *** alazarev has joined #openstack-sahara | 22:18 | |
| *** alazarev has quit IRC | 22:18 | |
| *** alazarev has joined #openstack-sahara | 22:20 | |
| *** alazarev has quit IRC | 22:22 | |
| *** IlyaE has quit IRC | 22:24 | |
| *** ErikB2 has quit IRC | 22:38 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!