14:01:37 <elmiko> #startmeeting sahara
14:01:37 <openstack> Meeting started Thu Jan 14 14:01:37 2016 UTC and is due to finish in 60 minutes.  The chair is elmiko. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:40 <openstack> The meeting name has been set to 'sahara'
14:02:03 <elmiko> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
14:02:11 <sreshetnyak> o/
14:02:39 <elmiko> question, do we still need the sahara@horizon agenda item?
14:02:43 <SergeyLukjanov> oops, 2 mins late
14:02:50 <elmiko> (it looked like most of the links on the etherpad are completed)
14:02:54 <elmiko> #chair SergeyLukjanov
14:02:54 <openstack> Current chairs: SergeyLukjanov elmiko
14:02:57 <elmiko> np SergeyLukjanov
14:02:59 <SergeyLukjanov> seems like time to remove it elmiko
14:03:04 <elmiko> the floor is yours =)
14:03:09 <elmiko> ack, i'll cut it
14:03:11 <SergeyLukjanov> elmiko thx :)
14:03:20 <SergeyLukjanov> there should be no more merging issues ;)
14:03:24 <SergeyLukjanov> any objections?
14:03:44 <elmiko> none here
14:03:45 <sreshetnyak> nothing
14:04:06 <SergeyLukjanov> #agreed to remove separated dashboard discussion item from meeting regular agenda
14:04:14 <SergeyLukjanov> #agree to remove separated dashboard discussion item from meeting regular agenda
14:04:23 <vgridnev> hi
14:04:29 <SergeyLukjanov> #topic News / updates
14:04:34 <SergeyLukjanov> (including dashboard :) )
14:05:14 <sreshetnyak> I'm working on support running sahara-api via apache
14:05:27 <AndreyPavlov> i've been working on distributed periodic tasks implementation
14:05:28 <elmiko> i've got the last part of the improved-secret-store spec up for review (cdh passwords), and am bringing the api v2 wiki into shape. i've also been investigating some security issues with sahara related to our bandit gate.
14:05:36 <SergeyLukjanov> I'll be in CA for the next 2-3 month starting this weekends, so, will be available in PST time zone
14:05:52 <vgridnev> made some researches for health checks, writing spec is in the progress
14:06:02 <esikachev> i am working on moving sahara scenario tests
14:06:05 <huichun> elmiko: EDP log enhancement spec updated need your review https://review.openstack.org/#/c/245571/
14:06:07 <SergeyLukjanov> esikachev is currently actively working on the scenario tests separation and hopefully we'll get it soon
14:06:09 <elmiko> SergeyLukjanov: wow, nice long stay. getting away from the snow?
14:06:21 <elmiko> huichun: ack, i will take a look today
14:06:37 <SergeyLukjanov> elmiko, yeah, it was -20 C few days ago
14:06:43 <elmiko> yeeeouch!
14:07:31 <SergeyLukjanov> esikachev could you please send a link to the CR for tests?
14:07:43 <NikitaKonovalov> I'm almost finished with internal tests, will be moving to dashboard activities soon
14:07:50 <huichun> SergeyLukjanov EDP scheduler patch has been long time reviewed, and the integration tests has been passed. Ready to merge https://review.openstack.org/#/c/182310/
14:07:53 <SergeyLukjanov> #info we'll have Mitaka-2 next week
14:08:01 <esikachev> #link https://review.openstack.org/#/c/267543/
14:08:05 <esikachev> SergeyLukjanov: ^^
14:08:40 <SergeyLukjanov> huichun ack
14:08:56 <esikachev> SergeyLukjanov: other patches merged
14:09:51 <huichun> elmiko:  suspend edp patch has been updated, long time reviewed, need your review too ^ ^ https://review.openstack.org/#/c/201448/
14:09:58 <tosky> esikachev: thanks for the work on moving scenario tests (and thanks more for keeping the history)
14:10:11 <elmiko> huichun: thanks, i'll add it to my queue ;)
14:10:16 <esikachev> tosky: np)
14:11:32 <SergeyLukjanov> actually regarding the tests move
14:11:55 <SergeyLukjanov> I've pushed the cleaned-up by esikachev base code to the sahara-scenario repo
14:12:08 <SergeyLukjanov> so, please, do not merge into the scenario tests in the sahara repo
14:12:18 <esikachev> SergeyLukjanov: thanks
14:12:39 <SergeyLukjanov> and we're planning to rename repo on the next maintainence window from sahara-scenario to sahara-tests
14:13:06 <elmiko> cool
14:13:18 <tosky> when is it planned more or less?
14:13:23 <SergeyLukjanov> rename?
14:13:32 <tosky> yes; one week, two weeks... ?
14:13:42 <SergeyLukjanov> no idea, hopefully no more then few weeks
14:13:46 <tosky> ok
14:14:04 <SergeyLukjanov> we have a request for one more rename, so, probably earlier than later, will decide next Tue on the infra meeting
14:14:32 <SergeyLukjanov> #topic Action items from the last meeting
14:14:34 <SergeyLukjanov> we have one
14:14:39 <SergeyLukjanov> tosky to comment on the scenario separation spec about git grafting the history
14:14:53 <SergeyLukjanov> but I think it's fully resolved ;)
14:15:06 <SergeyLukjanov> tosky could you please ack if you're ok with how it looks like now?
14:15:22 <tosky> SergeyLukjanov: yes, it's good, thanks
14:15:32 <tosky> ack here or on the review?
14:15:54 <SergeyLukjanov> here is enough
14:15:55 <SergeyLukjanov> thx
14:16:04 <SergeyLukjanov> #topic Separated launchpad for sahara-dashboard
14:16:25 <SergeyLukjanov> so, I remember there were questions about where to manage sahara-dashboard blueprints and etc.
14:16:52 <SergeyLukjanov> IMO for now it's better to keep it in sahara project, because we have the same release cycle, specs and etc.
14:16:56 <SergeyLukjanov> any thoughts?
14:17:19 <NikitaKonovalov> are there any examples of other projects tracking their dashboard separately?
14:17:20 <tosky> I agree to keep them into sahara; you can easily search by tags, as it was done up to this point
14:17:35 <elmiko> i think trove just decided to make a separate lp for their dashboard
14:17:46 <vgridnev> I agree with SergeyLukjanov position
14:17:57 <elmiko> and iirc, there are a few others that manage their dashboard through a separate lp
14:18:06 <elmiko> but, i'm ok with keeping it in the same
14:18:24 <SergeyLukjanov> NikitaKonovalov, nope, I've never seen such
14:18:38 <elmiko> the one nice point about having the dashboard stuff in a separate lp is that we can add horizon cores to that project
14:18:38 <vgridnev> It will be friendly for users in most cases, bug manager will decide correct scope of the bug
14:18:41 <SergeyLukjanov> elmiko, oh, I've missed this (re trove)
14:18:41 <elmiko> SergeyLukjanov: look at trove
14:18:55 <elmiko> SergeyLukjanov: they just decided that last week
14:18:59 <SergeyLukjanov> ok
14:19:13 <huichun> vgridnev: https://review.openstack.org/#/c/201448/ suspend EDP patch updated according to your review comments
14:19:34 <vgridnev> huichun, ok
14:19:43 <SergeyLukjanov> elmiko, what's the profit of having horizon-cores added at launchpad?
14:19:55 <SergeyLukjanov> we can have them at gerrit and it's not related
14:20:13 <tosky> elmiko: add horizon core to the project, does it change something?
14:20:25 <tosky> ups, too slow to type :)
14:20:52 <elmiko> tosky: i meant, add them to the lp project for just the dashboard. (i'm not even sure they'd want to, but it was an interesting topic brought up in the trove discussion)
14:21:11 <tosky> ok, but as SergeyLukjanov  pointed out, gerrit permissions are not related
14:21:20 <elmiko> right
14:21:38 <elmiko> and don't get me wrong, i'm not arguing to make our dashboard separate. i think it's fine how it is
14:22:04 <tosky> no, I was really trying to understand if there is some difference for some ACL, for example
14:22:20 <elmiko> SergeyLukjanov: the point that was made by the trove team was that having a separate lp for dashboard would give them more granularity to allow the horizon cores to manage the dashboard lp, if necessary
14:22:29 <SergeyLukjanov> tosky only bugs and bp management
14:22:29 <elmiko> tosky: i'm not exactly sure on that point
14:22:54 <elmiko> yea, i think they mainly were interested in bug mgmt by horizon cores, if necessary
14:23:10 <SergeyLukjanov> elmiko unfortunately I don't know why horizon folks will want to manage lp of some external dashboard :)
14:23:19 <elmiko> SergeyLukjanov: me either ;)
14:23:21 <tosky> which I don't think they (horizon core) would ever do
14:23:23 <tosky> exactly
14:23:27 <elmiko> tosky: right...
14:23:38 <SergeyLukjanov> I think they a lot of things to do in their project :)
14:23:45 <elmiko> so, i'm +1 for keeping it in the same lp
14:23:53 <elmiko> could be
14:24:11 <SergeyLukjanov> so, seems like we all don't see any reasons for separating right now
14:24:30 <SergeyLukjanov> let's keep this item for the next meeting if some one will find out why we should do it
14:24:46 <elmiko> sounds good
14:24:58 <SergeyLukjanov> okay
14:25:02 <SergeyLukjanov> #topic clouds.yaml support, #link https://review.openstack.org/#/c/236712/
14:25:07 <SergeyLukjanov> #link https://review.openstack.org/#/c/236712/
14:25:26 <SergeyLukjanov> I don't know who added this item, but it's the thing I was planning to discuss too ;)
14:25:32 <elmiko> i added it =)
14:25:49 <elmiko> and i know you saw the review, i was just curious if we had someone who will take up this work?
14:26:32 <SergeyLukjanov> I think apavlov can do it
14:26:36 <tosky> elmiko: is it just for the CLI? Given that we are migrating to openstackclient, does it make sense to add support to the old CLI client?
14:26:39 <SergeyLukjanov> AndreyPavlov ^^
14:26:40 <tosky> SergeyLukjanov: ^
14:27:01 <elmiko> tosky: right... it may not even be worth it
14:27:11 <SergeyLukjanov> I think only for the new CLI
14:27:15 <elmiko> right
14:27:30 <elmiko> so, if we are planning to fully integrate with openstackclient then we might not need to
14:27:40 <SergeyLukjanov> and as I understand it should be easy to do
14:27:57 <elmiko> yea, the patch looked relatively small
14:28:06 <tosky> the line "Many projects provide openstackclient extensions rather than their own client, so are covered already." sounds like you don't need to do anything, but I don't know the internal there
14:28:20 <elmiko> on this topic though, will we be planning to integrate with the openstacksdk project?
14:28:36 <elmiko> tosky: yea, it would mainly be if we planned to port it into the sahara cli tool
14:29:55 <SergeyLukjanov> tosky, I have a feeling that it should be anyway checked
14:30:15 <tosky> sure
14:30:17 <elmiko> it's a nice convenience for our users to have
14:30:18 <tosky> just wondering
14:30:28 <tosky> I totally agree it's useful
14:32:33 <SergeyLukjanov> AndreyPavlov please check it
14:32:53 <elmiko> thanks SergeyLukjanov and AndreyPavlov =)
14:33:11 <AndreyPavlov> SergeyLukjanov: sure, i will
14:33:17 <SergeyLukjanov> #topic API v2 progress
14:33:21 <elmiko> hey
14:33:26 <elmiko> so, i've got a few updates here
14:33:38 <elmiko> 1. please, more reviews on #link https://review.openstack.org/#/c/212172/
14:33:50 <elmiko> i need to add something about tempest tests, but this spec is quite old now...
14:34:07 <elmiko> 2. i'm adding more content into #link https://wiki.openstack.org/wiki/Sahara/api-v2
14:34:17 <elmiko> i will soon start adding the individual work items to that page
14:34:21 <elmiko> and, finally
14:34:45 <elmiko> 3. i'm getting an experimental version of the /v2 endpoint up and running with the project id removed from the uri
14:34:52 <SergeyLukjanov> cool!
14:34:55 <elmiko> any questions, comments, or concerns?
14:34:58 <elmiko> =)
14:34:59 <SergeyLukjanov> elmiko  great work
14:35:10 <SergeyLukjanov> I've added CR to my list
14:35:19 <elmiko> sweet, thanks SergeyLukjanov !
14:35:50 <elmiko> that's all i had
14:35:57 <SergeyLukjanov> anything to discuss now, elmiko ?
14:36:01 <SergeyLukjanov> :)
14:36:05 <elmiko> not unless there are questions
14:36:28 <SergeyLukjanov> #topic Open discussion
14:36:51 <elmiko> trove and hbase, i hope everyone has taken a look at this #link http://lists.openstack.org/pipermail/openstack-dev/2016-January/083422.html
14:37:09 <elmiko> i've commented as much as i can, but if we have some hbase experts it would be nice if they could read that
14:37:29 <elmiko> trove is proposing adding hbase standalone and pseudo-distrib. as datastores
14:37:43 <elmiko> by themselves, these modes are not production ready
14:37:59 <elmiko> so the next question is, where will this go with trove, and can we help with some sort of integration?
14:38:15 <elmiko> i don't think they want to do that currently, but still we should help to expose the issues they might run into
14:41:13 <huichun> SergeyLukjanov: I have seen many data pipeline solution (aws, LinkedIn'solution..)  compared with our Sahara EDP engine, I think maybe next release we should talk about this integration,IMO, data pipe line is more useful and practical
14:41:14 <elmiko> different topic, has anyone been looking at openstacksdk, and i think we should start planning to integrate =)
14:41:17 <huichun> elmiko:
14:41:53 <elmiko> interesting, i'm not that familiar with data pipeline
14:42:12 <elmiko> huichun: do you have any links that we could read about it?
14:42:50 <SergeyLukjanov> elmiko, re trove vs hbase - I don't know how to say that many of HBase users using not only Hbase, but other parts of Hadoop world
14:43:13 <SergeyLukjanov> elmiko, re standalone and pseudo == 100% useless
14:43:17 <elmiko> SergeyLukjanov: right...
14:43:24 <huichun> elmiko: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html
14:43:24 <elmiko> and that was brought up on the ML
14:43:29 <elmiko> huichun: thanks!
14:44:07 <elmiko> SergeyLukjanov: anyways, it would be nice to have more voices in that thread
14:44:10 <SergeyLukjanov> elmiko re sdk - I haven't seen yet, it'll be cool if you can make some short overview
14:44:21 <elmiko> SergeyLukjanov: ack, i'll do that for next meeting =)
14:44:29 <SergeyLukjanov> huichun do you mean to support pipelines from sahara or implement in sahara?
14:44:32 <SergeyLukjanov> elmiko, thx!
14:44:45 <elmiko> #action elmiko do short overview of openstacksdk for next meeting
14:45:27 <huichun> SergeyLukjanov:  we have two option,one is create a standalone project named data pipeline(use Sahara)
14:46:04 <huichun> Second is redesign current EDP engine to support  data pipeline
14:47:39 <weiting> Actually we did so many discussions internally abotu data pipeline idea, we thought it could be another project in OpenStack and equivalent with AWS Data Pipeline.
14:48:06 <elmiko> interesting
14:48:22 <huichun> SergeyLukjanov:  many company use data pipeline to solve their problem with EMR
14:48:27 <SergeyLukjanov> yeah, we were having a lot of discussions about it in Mirantis too
14:48:47 <huichun> LinkedIn. Netflix etc
14:48:53 <SergeyLukjanov> sounds like it should live one more layer upper
14:49:29 <SergeyLukjanov> like some project that will use trove, sahara, murano as a providers of building blocks and actions and mistral to define and execute flows
14:50:09 <huichun> Yes
14:50:20 <weiting> Yes, agree with SergeyLukjanov.
14:50:43 <huichun> But it's a huge change, actually redesign the EDP parts
14:51:16 <weiting> And if it is possible Sahara can also be used in Murano for better support.
14:51:53 <weiting> An example like AWS, Data Pipeline also call EMR to build a Hadoop cluster.
14:52:14 <elmiko> i like the idea of higher level openstack apps
14:52:24 <weiting> But Data Pipeline's capabilities are not only for Hadoop cluster, but also for the data flow control.
14:52:47 <SergeyLukjanov> we have some ideas of implementing sahara support in murano to able to use all sahara capabilities from murano
14:52:51 <weiting> That's why we thought this should be a high level openstack apps.
14:52:58 <SergeyLukjanov> but last time we checked murano was too limited for it
14:53:31 <elmiko> ah, too bad
14:53:33 <SergeyLukjanov> in theory it fits the murano goals as it related to the app catalog and it itself is a set of building blocks
14:53:58 <kzaitsev_mb> are you talking about a.o.o or murano as app-catalog? =)
14:54:05 <SergeyLukjanov> elmiko it was around half of year ago, so, I think we'll need to re-avaluate it in Mitaka for sure
14:54:13 <elmiko> SergeyLukjanov: +1
14:54:18 <weiting> Our idea is Sahara can keep the focus for provisioning a Hadoop/Spark cluster and we can have a higher level project to control the data flow.
14:54:52 <elmiko> weiting: i'm actually working on something similar, albeit smaller in scope, to propose as a talk for openstack summit in austin
14:55:03 <SergeyLukjanov> kzaitsev_mb re murano as catalog, a.o.o is just a list of apps
14:55:31 <weiting> elmiko, great to hear this info.
14:55:32 <SergeyLukjanov> some parts of data flow could be (and I think should be) supported directly by sahara
14:55:45 <SergeyLukjanov> but only the BigData related parts and not too upper layer
14:55:55 <elmiko> agreed
14:56:28 <SergeyLukjanov> in general - the data flows are super wide term and it includes every resource and capabilities that could be provided by openstack
14:56:57 <SergeyLukjanov> including integration with AWS for example for multicloud usage
14:57:44 <elmiko> yea, this is perfect for higher order applications
14:57:55 <elmiko> we should keep sahara focused on its specialties
14:58:29 <SergeyLukjanov> agree
14:58:42 <SergeyLukjanov> 1 min left
14:58:49 <SergeyLukjanov> any last minute q?
14:59:05 <elmiko> everyone having a nice new year? ;)
14:59:44 <SergeyLukjanov> so far so good
14:59:48 <SergeyLukjanov> #endmeeting