18:00:33 <SergeyLukjanov> #startmeeting sahara
18:00:34 <openstack> Meeting started Thu May 22 18:00:33 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:37 <tmckay1> here here here
18:00:37 <openstack> The meeting name has been set to 'sahara'
18:01:09 <sreshetnyak> hi
18:01:11 <SergeyLukjanov> #info me, dmitryme and aignatov are on vacations now, so, not really high activity
18:01:28 <tmckay> In Vegas yet? :)
18:01:37 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:01:51 <SergeyLukjanov> tmckay, I'm returned to home tonight
18:02:05 <SergeyLukjanov> recovering from summit and additional days in ATL ;)
18:02:12 <elmiko> lol
18:02:19 <SergeyLukjanov> #topic News / updates
18:02:21 <SergeyLukjanov> folks, please
18:02:22 <tmckay> I want to an indoor waterpark for 2 days
18:02:42 <dmitryme> tmckay: the last news from aignatov is that in the end his final balance is +18$ in roulette
18:02:56 <SergeyLukjanov> #info a lot of things discussed on summit - https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#Sahara_.28ex._Savanna.29
18:03:14 <bob_nettleton> working on upgrading the HDP plugin to use the latest Ambari, minor checkins to sahara-image-elements
18:03:31 <dmitryme> I am going to speak about Sahara on Berlin Buzzwords next week
18:03:32 <dmitryme> http://berlinbuzzwords.de/session/hadoop-openstack-cloud
18:03:43 <bob_nettleton> I've also just discovered a bug in diskimage-builder, that is basically causing any yum-based images to fail :  https://bugs.launchpad.net/diskimage-builder/+bug/1322278
18:03:44 <SergeyLukjanov> bob_nettleton, please, don't forget re backward compat
18:03:51 <elmiko> patching the HDP plugin for disconnected mode and helping the Horizon folks with reviews. also reading up on Pecan.
18:03:52 <tmckay> heh.  updates for me:  I filed a security bug to get OSSG opinions on swift authentication.  Also, I'm playing with the Spark plugin
18:04:38 <SergeyLukjanov> sounds cool
18:04:40 <bob_nettleton> SergeyLukjanov, Hi Sergey, we're planning on supporting the existing HDP versions with the upgrade, is that the backwards compatibility you're concerned about?
18:05:21 <alazarev> I’m restoring dev env after month of performance mesurement, make several small fixes (on review now)
18:05:23 <SergeyLukjanov> bob_nettleton, it should work with clusters installed by prev. version of plugin, if it will, then everything is ok
18:06:20 <SergeyLukjanov> #topic Action items from the last meeting
18:06:22 <bob_nettleton> SergeyLukjanov, I'm not sure if that is the case.  I'll have to check.  Not sure if Ambari supports this.
18:06:25 <SergeyLukjanov> #info DONE: aignatov prepare data for retrospective on summit
18:06:52 <SergeyLukjanov> #topic Bug triage day after summit
18:07:07 <SergeyLukjanov> so, the current plan is to make a bg triage day May 27
18:07:17 <SergeyLukjanov> (Tue next week)
18:07:22 <SergeyLukjanov> any objections?
18:07:51 <tmckay> no objection from me
18:07:56 <SergeyLukjanov> #agreed Bug Triage day - May 27
18:07:56 <alazarev> no objections
18:08:12 <SergeyLukjanov> #topic Design summit
18:08:19 <SergeyLukjanov> any comments re design summit?
18:08:42 <mattf> bob_nettleton, that bug looks annoying. we should default to a known good dib rev and add a flag for --master that just gets the most recent (occasionally broken) tip
18:08:42 <elmiko> i thought our sessions were pretty good, i'm curious about the api v2 changes that are coming.
18:09:24 <elmiko> mattf: +1 to the --master flag
18:09:25 <mattf> bob_nettleton, defensive git-clone'ing, heh
18:09:32 <SergeyLukjanov> #info design summit sessions were very productive and we have a good plans for Juno and mostly all question has been discussed enough
18:09:37 <bob_nettleton> mattf, I agree, we should gradually integrate to the latest diskimage-builder, since it seems to break every few days lately.
18:09:56 <tellesnobrega> i have a question about how to make EDP pluggable investigation, if is there going to be a team doing this and how i can help with it
18:10:08 <bob_nettleton> mattf, :)
18:10:09 <tmckay> The stuff I'm currently doing is aligned with the priorities we set in the EDP session.  Get something going on swift auth eval, and familiarize myself with a spark cluster on Sahara as a prelim to pluggable job design and EDP on Spark
18:10:22 <mattf> i'd fire off some code for it, but i'm mired in hadoop summit atm
18:10:38 <notmyname> please let me know if I can help answer questions about swift auth integration
18:11:00 <SergeyLukjanov> notmyname, thanks
18:11:03 <tmckay> tellesnobrega, just starting to think about that.  I want to see how Spark jobs work, play with it from the command line before we start designing something
18:11:16 <tmckay> Yesterday was my first day back :)
18:11:21 <SergeyLukjanov> tmckay, yup,that's the correct approach :)
18:11:37 <tmckay> notmyname, are you an OSSG guy?
18:11:42 <tellesnobrega> tmckay: sounds good, if you need a hand with that i'm up to it
18:11:47 <elmiko> mattf: i can take a look into the --master flag
18:12:00 <notmyname> tmckay: I'm the swift PTL
18:12:34 <tmckay> tellesnobrega, certainly.  If you have ideas based on common elements between Oozie and Storm or Spark, feel free to write them up.  We can start an etherpad
18:12:58 <tmckay> notmyname, great, thanks.  I can include you on the bug.
18:13:18 <SergeyLukjanov> any comments re design summit?
18:13:25 <SergeyLukjanov> program pods
18:13:27 <SergeyLukjanov> ?
18:13:31 <tellesnobrega> tmckay: sure, i will have to take a deeper look into oozie the next days and how it works in sahara so we can start this up
18:13:32 <alazarev> SergeyLukjanov: summit was great ;)
18:13:37 <tellesnobrega> +1
18:13:44 <elmiko> SergeyLukjanov: pods were nice, good place to unwind and get some quiet coding time
18:14:30 <SergeyLukjanov> okay, if there no more questions / comments, let's move on
18:14:36 <SergeyLukjanov> I'd like to discuss the
18:14:43 <elmiko> also, great job to all folks hosting the design sessions. from my perspective they were welcoming and productive.
18:14:44 <SergeyLukjanov> #topic -specs vs. lp for blueprints
18:15:11 <SergeyLukjanov> are you all folks know what is the %program%-specs repos for?
18:15:31 <SergeyLukjanov> if not I can try to make a brief overview
18:15:40 <elmiko> i'd like an overview
18:15:44 <alazarev> SergeyLukjanov: I don’t know
18:15:54 <SergeyLukjanov> okay
18:16:08 <SergeyLukjanov> so, let's start from the issue that it solves
18:16:28 <SergeyLukjanov> it solves the incoming features review/approval issues
18:16:36 <tmckay> tellesnobrega, it may actually be possible to create an oozie extension to run spark, too.  We should consider that.  Or maybe someone already has made one.
18:16:49 <SergeyLukjanov> so, it's a separated repo for reviewing specs for blueprints as rst docs
18:17:12 <SergeyLukjanov> than when it's reviewed and approved related changes could be merged to the codebase
18:17:27 <SergeyLukjanov> it's needed to make blueprints more informative
18:17:39 <tellesnobrega> tmckay: hum, i dont know much about oozie yet, but i will pick it up this weekend so we can better discuss it and I can play along with you on spark and later with storm
18:17:45 <elmiko> SergeyLukjanov: is this in addition to launchpad or instead of?
18:17:53 <mattf> SergeyLukjanov, core issue is bps are vague and terse?
18:18:07 <SergeyLukjanov> mattf, yup
18:18:25 <SergeyLukjanov> elmiko, in this case lp will be used for release notes :)
18:18:30 <mattf> so there's going to be a repo where design docs are managed (version controlled and approved)?
18:18:46 <SergeyLukjanov> mattf, yup
18:18:54 <SergeyLukjanov> example - https://github.com/openstack/nova-specs
18:19:08 <SergeyLukjanov> it was started as experiment by nova in I
18:19:28 <SergeyLukjanov> and in J all core projects starting using it
18:19:48 <SergeyLukjanov> but non-core projects are 50/50 for using it IIRC
18:19:59 <SergeyLukjanov> for ex. troev will not use -specs in J
18:20:32 <SergeyLukjanov> it's mostly a question of efforts needed to write specs vs. blueprints vague description
18:20:41 <SergeyLukjanov> that's why I'd like to discuss it with you folks
18:20:58 <elmiko> SergeyLukjanov: if we want to add a bp we should setup a review against an addition to the -spec repo?
18:21:05 <mattf> for the most part our bps are way too vague and terse
18:21:34 <SergeyLukjanov> elmiko, yup, and after approval of spec, bp in lp will be approved and related changes could be merged
18:21:44 <elmiko> ok
18:21:45 <mattf> anything that can improve our bps gets a +1 from me, maybe pilot it for J and if it works +2 for K
18:21:55 <elmiko> i'm looking at nova-spec now, and i like how they have it structured
18:22:18 <elmiko> +1 for trying it out
18:22:38 <mattf> i suggest only piloting it because we don't have all the same problems of scale that nova has
18:22:45 <SergeyLukjanov> we can pilot it for some huge blueprints for ex.
18:22:52 <alazarev> something like “plan a lot, code slow” vs “plan a little, code fast, refactor frequently”?
18:22:58 <mattf> so if we have half out bps going through spec that'd be a good way to eval
18:22:59 <crobertsrh> +1 for better bps....the question is, Is it the system that causes us to have poor bps, or is it us that causes us to have poor bps?
18:23:22 <mattf> alazarev, that's not clear to me
18:23:30 <elmiko> crobertsrh: probably a little from column A, a little from column B....
18:23:34 <mattf> we might plan enough, code cast, refactor plan & code
18:23:39 <mattf> fast*
18:23:57 <mattf> crobertsrh, i think it's mostly just us
18:24:00 <jspeidel> crobertsrh, it is currently too easy to submit a bad blueprint
18:24:10 <alazarev> mattf: the question is what is “fast enough”
18:24:10 <crobertsrh> I'm guilty of a few terse bps.  Totally my fault.  I would have tried to "game" any system, I suspect. :(
18:24:11 <SergeyLukjanov> it's mostly about describing your plans for bp to be able to discuss architecture (for ex.) before actual implementation
18:24:17 <mattf> if we're so bad that we need the system to help, ok, but that's too bad
18:24:48 <mattf> i'm guilty of overly detailed bps that are bigger than folks want to fit into their heads to eval
18:25:00 <bob_nettleton> mattf, agreed.  if we end up approving vague and terse specs, then the only thing accomplished is that we've moved the blueprints to version control.
18:25:10 <tmckay> +1 on better specs, if we all used the "spec" link feature on the current blueprints we'd be better off
18:25:40 <elmiko> i like that the nova-spec repo has a template bp that is pretty well defined
18:25:53 <crobertsrh> Yeah, I like the template idea that they have
18:25:59 <SergeyLukjanov> so, sounds like all are +1 for trying -specs process for the huge blueprints to improve overall blueprints descriptions
18:26:03 <SergeyLukjanov> am I correct?
18:26:09 <elmiko> SergeyLukjanov: +1 on +1
18:26:24 <crobertsrh> +1.  We should have our own template for such specs.
18:26:33 <elmiko> yea, definitely need a template
18:26:35 <alazarev> +1
18:26:36 <SergeyLukjanov> crobertsrh, sure, it's just a technical q.
18:27:20 <crobertsrh> If you leave part of a spec blank, it probably needs a brief explanation for why it's blank....thus leaving it not blank anymore.  No blank sections.
18:27:44 <SergeyLukjanov> Juno-1 will be june 12, so, I think we could start using it from Juno-2
18:27:51 <tmckay> +1, I'm up for trying a process improvement.  If it's awful, we can do something else
18:28:29 <mattf> everyone is going to have to make a conscious effort, else no tool/system/process will really help
18:28:38 <SergeyLukjanov> tmckay, many projects starting working on it and using it, so, I think the process will be good improved in Juno
18:29:07 <tmckay> okey doke
18:29:18 <SergeyLukjanov> mattf, one more issue with lp - it's really difficult to discuss and track specs in lp
18:29:32 <mattf> that's for sure
18:29:37 <SergeyLukjanov> and you can just subscribe on the specs repo to track incoming features
18:29:37 <mattf> gerrit isn't much better tho
18:29:47 <elmiko> if we end up liking the -spec approach, we'll probably need to modify some of our docs
18:30:14 <mattf> SergeyLukjanov, are -specs managed by gerrit or github pull request?
18:30:37 <SergeyLukjanov> mattf, gerrit, github is just only one of the mirrors
18:30:39 * mattf sees gerrit committing to nova-specs
18:30:57 <mattf> how do you subscribe to a repo to track incoming features?
18:31:29 <SergeyLukjanov> mattf, subscribe in gerrit to receive emails about new CRs in -specs repo
18:31:43 <mattf> ahh, k
18:31:49 <SergeyLukjanov> ok, so, agreed, that we need to try -specs approach for example on several blueprints and then decide how it works for us
18:32:10 <SergeyLukjanov> probably it'll be so complex and awful that we all start writing good blueprints ;)
18:32:16 <elmiko> lol
18:32:52 <SergeyLukjanov> #agreed to make a pilot of -specs for several blueprints and then decide how it works for us
18:32:52 <mattf> SergeyLukjanov, is there still a plan to move off lp?
18:33:04 <SergeyLukjanov> mattf, yup, storyboard is in progress
18:33:10 <SergeyLukjanov> storyboard.openstack.org
18:33:17 <SergeyLukjanov> several infra projects are already on it
18:33:42 <SergeyLukjanov> storyboard plans for MVP / Juno - https://etherpad.openstack.org/p/juno-infra-storyboard
18:33:59 <mattf> gotcha
18:34:10 <SergeyLukjanov> #action SergeyLukjanov to prepare -specs infra for pilot
18:34:41 <alazarev> do we have plans to move to storyboard?
18:35:11 <SergeyLukjanov> alazarev, sure, when it'll be ready
18:35:21 <SergeyLukjanov> alazarev, you can read an etherpad for details
18:35:51 <SergeyLukjanov> any major topics to discuss on today's meeting?
18:36:02 <mattf> none here
18:36:08 <tmckay> what is "MVP" ?
18:36:18 <mattf> minimum viable product (usually)
18:36:26 <SergeyLukjanov> mattf, exactly
18:36:27 <tmckay> sounds military
18:36:36 <mattf> it's all agile and stuff
18:36:40 <mattf> lean and startup-y
18:36:46 <tmckay> lol
18:36:57 <tmckay> It has a pulse, ship it
18:37:03 <mattf> often credited to "the lean startup" iirc
18:37:14 <SergeyLukjanov> tmckay, and it's funny that it's "most valuable player" in sports
18:37:17 <mattf> sometimes "it's a web form, ship it"
18:37:31 <tmckay> SergeyLukjanov, yeah, that threw me off
18:38:01 <SergeyLukjanov> mattf, yup, the bonus is that lp doesn't provide many features :)
18:38:32 <mattf> and we seem to use fewer and fewer of them
18:38:58 <SergeyLukjanov> mattf, yup, because it works bad for us :(
18:39:20 <SergeyLukjanov> #topic Open discussion
18:40:36 <SergeyLukjanov> probably we should discuss thing that bob_nettleton is working on
18:40:40 <SergeyLukjanov> (ambari update)
18:40:48 <bob_nettleton> ok
18:41:16 <SergeyLukjanov> we agreed that we should released plugin versions working for the next release
18:41:29 <SergeyLukjanov> alazarev could describe it in details
18:41:50 <bob_nettleton> SergeyLukjanov,  not sure I understand what the issue is.
18:42:12 <SergeyLukjanov> bob_nettleton, are you adding new version of hdp plugin?
18:42:32 <SergeyLukjanov> could you please describe more detailed what are working on?
18:42:54 <bob_nettleton> SergeyLukjanov, yes, I'm upgrading the HDP plugin to use a newer version of Ambari, in order to support later versions of HDP (past 2.0.6)
18:43:26 <alazarev> once hadoop version is suported in released Sahara - we can’t just remove it
18:43:26 <bob_nettleton> SergeyLukjanov, we're planning on continuing to support the versions of HDP already supported by the plugin (1.3.2, 2.0.6)
18:43:28 <jspeidel> SergeyLukjanov, we agreed that a specific version of a plugin would continue to support any stack version previously supported
18:43:46 <alazarev> new version of plugin need to be created with updated components
18:44:01 <bob_nettleton> alazarev, we're not removing support for the older versions of HDP.
18:44:01 <jspeidel> not that future versions of a plugin would continue to support all previously supported stacks
18:44:18 <mattf> SergeyLukjanov, this compatibility feature we want to add, sounds like a good candidate for a sahara-specs
18:45:13 <jspeidel> but that being said, the work that Bob is doing wouldn't remove support for any previously supported stacks
18:46:26 <mattf> there was a lot of discussion on what our compatibility statement is as a project - afaik it's not written down in a form that a user can easily consume
18:47:05 <SergeyLukjanov> mattf, good idea
18:47:10 <tmckay> +1
18:47:22 <SergeyLukjanov> #info backward compat could be used for -specs pilot
18:48:25 <alazarev> +1
18:48:38 <tellesnobrega> +1
18:50:23 <SergeyLukjanov> anything else to chat about today?
18:51:02 <tmckay> not from me
18:53:30 <SergeyLukjanov> okay
18:53:35 <SergeyLukjanov> thank you all folks
18:53:39 <SergeyLukjanov> #endmeeting