18:06:45 <SergeyLukjanov> #startmeeting savanna
18:06:46 <openstack> Meeting started Thu Jul 25 18:06:45 2013 UTC.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:06:47 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:06:49 <openstack> The meeting name has been set to 'savanna'
18:06:52 <SergeyLukjanov> hey everyone!
18:07:04 <rnirmal> hey
18:07:16 <ruhe> hi
18:07:17 <aignatov3> hallo
18:07:38 <akuznetsov> hi
18:07:47 <ivan_berezovskiy> hi
18:08:01 <NikitaKonovalov> hi
18:08:01 <SergeyLukjanov> Red Hat and Hortonworks guys, are you here?
18:08:37 <mattf> hi
18:08:52 <jmaron> hi
18:09:14 <SergeyLukjanov> ok, we have a quorum, let's start
18:09:21 <SergeyLukjanov> #topic Agenda
18:09:24 <SergeyLukjanov> #info Savann 0.2.1 release preparations
18:09:34 <SergeyLukjanov> #info General news/updates
18:09:49 <SergeyLukjanov> #info Gluster integration steps
18:09:52 <SergeyLukjanov> #info EDP discussions
18:10:06 <SergeyLukjanov> #info Scalable architecture discussions
18:10:35 <SergeyLukjanov> #topic Savann 0.2.1 release preparations
18:10:59 <SergeyLukjanov> There are several open issues that should be closed to release 0.2.1 version
18:11:24 <SergeyLukjanov> #info [core] image tags validation - https://bugs.launchpad.net/savanna/+bug/1204410
18:11:32 <SergeyLukjanov> looks like it's near to be merged
18:11:48 <SergeyLukjanov> #link https://review.openstack.org/#/c/38440/
18:11:57 <aignatov3> hmm, image tags validation already got several approvlas
18:12:06 <SergeyLukjanov> #info [hdp] convert method - https://bugs.launchpad.net/savanna/+bug/1202663
18:12:24 <SergeyLukjanov> jmaron, could you please write updates on it?
18:12:37 <jmaron> working on that one.  have a potential fix, but internally working thru dev env issues
18:12:38 <jmaron> ok
18:12:57 <SergeyLukjanov> jmaron, thank you, so, work in progress state
18:13:25 <SergeyLukjanov> #info [hdp] prepare Centos with cloud-init image
18:13:38 <SergeyLukjanov> jmaron, do you now anything about image preparations?
18:14:08 <jmaron> I know we are putting image creation automation in place.  ErikB may be able to provide more info
18:14:14 <SergeyLukjanov> ErikB, please
18:14:32 <jmaron> (he may be on the phone with Ilya)
18:14:43 <SergeyLukjanov> I'll create an issue to track this activity
18:15:02 <SergeyLukjanov> #action Sergey to create an issue to cover image creation for HDP plugin
18:15:22 <SergeyLukjanov> #info [hdp] update docs - https://bugs.launchpad.net/savanna/+bug/1204178
18:15:47 <SergeyLukjanov> any updates on it?
18:15:57 <SergeyLukjanov> jmaron?
18:16:38 <jmaron> the plan is to address this shortly.  quick question:  are you looking for something like https://savanna.readthedocs.org/en/latest/userdoc/vanilla_plugin.html?
18:17:04 <SergeyLukjanov> yep, absolutely
18:17:09 <jmaron> ok
18:17:12 <ruhe> quick-start guide will be nice too
18:17:46 <SergeyLukjanov> ok, let's go on
18:17:53 <jmaron> I'm not sure what a quick-start with respect to plugin would mean
18:17:54 <SergeyLukjanov> jmaron, thanks for the updates
18:18:14 <SergeyLukjanov> #info elements for diskimage-builder looks ready to release
18:18:29 <aignatov3> it's up to you the doc content, but some info about hdp plugin is needed
18:18:31 <SergeyLukjanov> are there any thoughts about what should be fixed in elements?
18:18:37 <SergeyLukjanov> mattf, ivan_berezovskiy?
18:18:48 <ruhe> jmaron, an example hadoop configuration which user would be able to pass to convert method
18:19:42 <mattf> SergeyLukjanov, for 0.2.1? nothing urgent comes to mind.
18:19:48 <ivan_berezovskiy> I think nothing to change
18:19:51 <SergeyLukjanov> mattf, yep
18:19:55 <aignatov3> great
18:19:55 <SergeyLukjanov> https://bugs.launchpad.net/savanna/+bug/1201794 and https://bugs.launchpad.net/savanna/+bug/1199970 has been already fixed
18:20:11 <SergeyLukjanov> thanks
18:21:11 <SergeyLukjanov> ok, i think that's all that I can remember about preparations for 0.2.1 release
18:21:23 <SergeyLukjanov> are there any thoughts on it?
18:21:57 <SergeyLukjanov> ok, let's move on
18:22:14 <SergeyLukjanov> #topic General news/updates
18:23:18 <SergeyLukjanov> #info most of the issues has been fixed and all blueprints has been implemented, the overall status of 0.2.1 release preparations is green
18:23:29 <aignatov3> Sergey can you shed more light about Savanna branching and backporting?
18:23:39 <SergeyLukjanov> yes, sure
18:23:54 <SergeyLukjanov> now the stable/0.2 branch created
18:24:03 <SergeyLukjanov> and master is open for 0.3 code
18:24:23 <SergeyLukjanov> there will be some limitations to make migration to new architecture easier
18:24:38 <SergeyLukjanov> I hope that will do most of the refactoring in about week
18:25:04 <SergeyLukjanov> and it'll be clear to understand in such style code should be writted
18:25:14 <SergeyLukjanov> in depends mostly on db interop
18:25:21 <mattf> should folks hold off on core changes until that refactoring is done?
18:25:44 <SergeyLukjanov> all patches for 0.2.1 should be merged to the master and then back ported to the stable/0.2 branch
18:26:10 <SergeyLukjanov> mattf, we want to avoid 'world lock' for the refactoring stage
18:26:32 <SergeyLukjanov> I think that we can do it in background w/o breaking the code
18:26:57 <SergeyLukjanov> and it's now very important to start working on EDP and it'll be not good to block it by refactoring
18:27:34 <mattf> in the case of conflicts, it'll be the responsibility of the team doing the refactor to handle the conflicts?
18:27:38 <SergeyLukjanov> we'll check all change requests for compatibility with new db interop and will place comments to make the refactoring easier
18:28:06 <SergeyLukjanov> mattf, yep, I think
18:28:17 <mattf> sounds good
18:28:35 <aignatov3> thanks guys, I've got your thoughts
18:28:50 <SergeyLukjanov> currently Nikita working on initial refactoring and Dmitry will join him at the next week
18:29:05 <SergeyLukjanov> (nkonovalov and dmescheryakov)
18:29:21 <SergeyLukjanov> mm, are there any other updates/news?
18:29:50 <mattf> 2 from me
18:30:05 <mattf> i filed a bp about gluster integration
18:30:11 <mattf> and a bug about fedora packaging
18:30:18 <SergeyLukjanov> great, thank you
18:30:29 <aignatov3> +2
18:30:30 <SergeyLukjanov> #link https://blueprints.launchpad.net/savanna/+spec/savanna-in-fedora
18:30:33 <SergeyLukjanov> #link https://blueprints.launchpad.net/savanna/+spec/hcfs-gluster-support
18:30:45 <aignatov3> +2 for matt
18:30:48 <mattf> it'd be nice to target them for 0.3, but that could be optimistic
18:31:08 <ruhe> mattf, maybe we should post our mail conversation about hcfs to the blueprint?
18:31:11 <SergeyLukjanov> mattf, I've already target them to 0.3 because of their improtance
18:31:48 <mattf> ruhe, i tried to include the relevant pieces from our thread. if i missed something feel free to add it
18:32:37 <SergeyLukjanov> ok, great
18:32:50 <mattf> fyi, i didn't post anything about native ubuntu packages. someone more familiar with that world should do that instead.
18:33:26 <SergeyLukjanov> we are waiting for volunteer :)
18:33:34 <aignatov3> I think we should create a separate bp for that
18:33:39 <mattf> sane approach
18:33:55 <mattf> ^^ waiting volunteer
18:33:55 <aignatov3> and think how and who will do that :))
18:34:09 <SergeyLukjanov> aignatov3, yep, do it please
18:34:28 <SergeyLukjanov> #action aignatov to create a blueprint for ubuntu packaging
18:34:42 <aignatov3> yes, will do
18:35:04 <IlyaE> are you referring to ubuntu packaging of Savanna itself or for base images?
18:35:21 <SergeyLukjanov> IlyaE, ubuntu packaging of Savanna
18:35:52 <aignatov3> yep, hust ubuntu packages with Savanna inside
18:35:59 <IlyaE> ok, I'll have a volunteer for you :)
18:35:59 <aignatov3> *just
18:36:03 <SergeyLukjanov> ok, looks like there are no other news or updates
18:36:48 <SergeyLukjanov> #info qa guys started working on UI tests using webdriver
18:37:25 <SergeyLukjanov> #topic Gluster integration steps
18:37:46 <SergeyLukjanov> mattf, could you please write what steps should be done to integrate with Gluster?
18:38:00 <mattf> can i refer you to the bp?
18:38:38 <SergeyLukjanov> there are some steps in it
18:38:40 <SergeyLukjanov> #link https://blueprints.launchpad.net/savanna/+spec/hcfs-gluster-support
18:38:57 <SergeyLukjanov> are there any questions about them?
18:39:03 <mattf> that'll be the canonical location for steps and be expanded over time
18:40:00 <SergeyLukjanov> mattf, it'll be good to add todo items to the "Work Items" section
18:40:06 <SergeyLukjanov> in future
18:40:21 <mattf> sure thing
18:40:31 <rnirmal> this is going to be an optional component right ?
18:40:49 <mattf> optional in what sense?
18:41:03 <rnirmal> a deployer decides wether or not to setup gluster
18:41:30 <mattf> i imagine the image will be built w/ some gluster element and be tagged appropriately when registered
18:42:04 <SergeyLukjanov> rnirmal, yes, sure, it'll be an option to use or not Gluster, it'll be looks like integration with Swift I think
18:42:04 <ruhe> we'll also need to have this integration in plugins. and in oozie for EDP
18:42:07 <mattf> so in that sense it would be optional
18:42:14 <rnirmal> ok think I'm missing some pieces .. suppose it's not similar to swift where you'll have to run a separate service
18:43:11 <SergeyLukjanov> there are 2 use cases that mattf described in bp afaiu - using external Gluster and installing Gluster to Hadoop cluster
18:43:59 <rnirmal> ok that's good
18:44:45 <SergeyLukjanov> ok, let's move on
18:44:56 <SergeyLukjanov> #topic EDP discussions
18:45:04 <SergeyLukjanov> akuznetsov, please
18:45:57 <akuznetsov> I created first draft for REST API implementation
18:46:06 <akuznetsov> you can see it in gerrit
18:46:27 <rnirmal> can you link to the docs or etherpad
18:46:28 <akuznetsov> #link https://review.openstack.org/#/c/38627/
18:46:50 <ruhe> #link https://etherpad.openstack.org/edp_v3_components
18:46:58 <SergeyLukjanov> #info elements for building images w/ Oozie are in WIP state
18:48:24 <aignatov3> another update on EDP: Oozie DIB elements for Vanilla plugin are in progress
18:48:35 <akuznetsov> we plan to use oozie for Savanna and Hadoop cluster interaction
18:49:26 <SergeyLukjanov> are there any other thoughts about EDP?
18:50:02 <rnirmal> so EDP is going to use oozie for job management scheduling etc
18:50:18 <aignatov3> also I'm playing with plugin code to launch Oozie process like a node process in terms of savanna
18:50:18 <SergeyLukjanov> yes, that's our plan for now
18:50:38 <SergeyLukjanov> aignatov3, great, looking for the change request from you :)
18:51:16 <rnirmal> can users run job flows on existing clusters or does it create new clusters ?
18:51:23 <ruhe> both
18:51:29 <SergeyLukjanov> we want to support both flows?
18:51:36 <ruhe> we want
18:51:43 <rnirmal> ok so user specifiable is good
18:52:08 <SergeyLukjanov> (oops, the '?' is incorrect)
18:52:19 <akuznetsov> rnirmal user will choose there to run job on exiting cluster on create a new one
18:52:41 <aignatov3> there -> where?
18:53:10 <rnirmal> cool thanks.. I'll look thru the docs
18:53:16 <ruhe> in future version we might want to implement/integrate smth like netflix genie
18:53:55 <SergeyLukjanov> rune likes pretty sexy diagrams :)
18:54:00 <SergeyLukjanov> ruhe*
18:54:10 <rnirmal> well aren't both similar
18:55:05 <ruhe> it's not about diagrams :)
18:55:19 <rnirmal> ie. EDP and genie in some aspects .. haven't looked at genie in depth
18:55:28 <SergeyLukjanov> (I mean netflix lipstick)
18:55:48 <ruhe> or twitter ambrose? :)
18:55:52 <SergeyLukjanov> yep:)
18:56:01 <SergeyLukjanov> ok, lat's move on
18:56:05 <SergeyLukjanov> #topic  Scalable architecture discussions
18:56:25 <SergeyLukjanov> I send an email with BP of new scalable arch
18:56:41 <SergeyLukjanov> mattf, thank you for taking a look on it
18:56:50 <SergeyLukjanov> I hope that I address all your comments
18:57:13 <mattf> i've not gotten to your response yet. i'll hit it later today hopefully. we can chat on irc too.
18:57:26 <rnirmal> so not sure I totally agree with all the agent comments.
18:57:35 <mattf> re edp and genie -- there's an interesting opportunity there
18:58:10 <rnirmal> I'll respond to the email.. we are almost out of time
18:58:17 <mattf> rnirmal, my comments or SergeyLukjanov's?
18:58:19 <mattf> ok
18:58:41 <rnirmal> SergeyLukjanov's and I'll update on mattf's comments
18:58:55 <SergeyLukjanov> the summary is that we want to implement the base part of arch changes - refactor code to be able to run in distributed mode and collect requirements for the right approach - using agents or not and about tasks management too
18:59:10 <rnirmal> basically I think the agent approach is more scalable
18:59:20 <rnirmal> yeah agree with splitting out the api db code first
18:59:32 <rnirmal> no arguments there :)
18:59:33 <SergeyLukjanov> because of the focus on edp and teams bandwidth
19:00:06 <SergeyLukjanov> rnirmal, I think that we should name the current activity something like "the first steps of scaling savanna"
19:00:15 <SergeyLukjanov> it'll be much more cleaner
19:00:26 <rnirmal> SergeyLukjanov: totally agree.. also I'm just getting caught up with savanna
19:00:35 <aignatov3> guys, let's end and move to the savanna channel if needed
19:00:42 <SergeyLukjanov> because we are now trying to collect info about bottlenecks and requirements
19:00:53 <SergeyLukjanov> yep, we are out of time
19:00:55 <mattf> +1 data driven
19:01:00 <mattf> thanks all!
19:01:07 <SergeyLukjanov> thanks everyone
19:01:07 <aignatov3> thx
19:01:16 <rnirmal> thanks
19:01:26 <SergeyLukjanov> #info JFYI you can always use openstack-dev@lists.openstack.org mailing lists and #savanna irc channel to find us and ask your questions
19:01:28 <jmaron> thx
19:01:33 <SergeyLukjanov> #endmeeting