18:01:31 <SergeyLukjanov> #startmeeting sahara
18:01:31 <tosky> pong
18:01:32 <openstack> Meeting started Thu May 29 18:01:31 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:34 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:01:36 <openstack> The meeting name has been set to 'sahara'
18:01:45 <crobertsrh> hello
18:01:49 <SergeyLukjanov> o/
18:01:54 <mattf> hi
18:02:00 <aignatov> o/
18:02:48 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:02:56 <SergeyLukjanov> #topic News / updates
18:02:59 <SergeyLukjanov> folks, please
18:03:36 <aignatov> I’m resuming implementation sahara resource in heat
18:03:49 <aignatov> just resolving comments I had a month ago :)
18:04:19 <aignatov> also found several issues in hadoop 2 vanilla plugin
18:04:30 <elmiko> finishing up the dib repo branch change, trying to get the final version of disconnected hdp plugin, starting to look into swift security issues, and helping with horizon reviews
18:04:44 <SergeyLukjanov> I'm working on auth conf patches, it's extremely near to be finished, working on setting up -specs for pilot and on improving sahara testing in gate (all images building in gate is coming soon)
18:05:12 <aignatov> the first one is that service provisioning is not ran in parallel
18:05:28 <aignatov> and that there was not ability to run HDFS service only
18:05:42 <tmckay> crobertsrh and I tracked down an intermittent bug that was causing clusters to be stuck in "Starting" sometimes ... mostly Fedora.  Problem is in DIB
18:05:52 <crobertsrh> merging into horizon is still ongoing.  Reviews have been sporadic.  I think I'm up to date with all of them though.....awaiting further abuse.
18:06:07 <tmckay> currently back working on spark plugin/edp after bug day
18:06:09 <mattf> crobertsrh, is there a unit test hurdle?
18:06:38 <aignatov> tmckay: are you going to implement edp functionality in spark?
18:06:52 <SergeyLukjanov> #info bug triage day happens
18:06:55 <crobertsrh> Yes, we still need unit tests.  They are on my radar.
18:07:08 <tmckay> aignatov, that is my mission!  We might be able to do it easily with oozie java action or oozie shell action, investigating
18:07:26 <tmckay> aignatov, that would let us separate the effort from a pluggable job model
18:07:30 <crobertsrh> I think now that I'm caught up on reviews, I'll start adding unit tests.
18:07:35 <SergeyLukjanov> #link http://status.openstack.org/bugday/
18:07:41 * mattf nods
18:07:44 <aignatov> tmckay: sahara’s datasources are capable with spark?
18:07:52 <aignatov> tmckay: right
18:07:58 <tmckay> aignatov, heh, too early to tell
18:08:07 <aignatov> ok, just wondering
18:08:14 <elmiko> SergeyLukjanov: lol, i think we win bugday
18:08:21 <tmckay> aignatov, I can launch spark clusters now, then there was bugday, now edp....
18:08:29 <SergeyLukjanov> elmiko,exactly, it's a great result
18:08:31 <crobertsrh> Yeah, we really kicked butt on bug day
18:08:55 <aignatov> also guys, as continuation, we have to file blueprints for all points we declared at summit :)
18:09:00 <aignatov> points to implement
18:09:02 <SergeyLukjanov> now we need bug fix day :)
18:09:10 <aignatov> SergeyLukjanov: what do you think?
18:09:30 <tmckay> blueprint day?
18:09:39 <aignatov> why not? :)
18:09:43 <tmckay> and bug fixing day, too
18:10:01 <SergeyLukjanov> aignatov, I'll add bps for releasing/versioning/elements after end of discussions
18:10:07 <tmckay> "days" make me happy because I don't feel bad about ignoring other stuff
18:10:13 <SergeyLukjanov> :)
18:10:27 <mattf> bugfixday =?= weekday
18:10:41 <SergeyLukjanov> we could make a bug fix day before the j2 freeze
18:10:47 <SergeyLukjanov> j1*
18:10:58 <SergeyLukjanov> June 9
18:11:05 <elmiko> +1 to before j1 freeze
18:11:32 <SergeyLukjanov> #topic juno-1
18:11:36 <SergeyLukjanov> #link https://launchpad.net/sahara/+milestone/juno-1
18:12:07 <SergeyLukjanov> looks nice presuming re-targeting some bps
18:12:48 <SergeyLukjanov> #info Note: juno-1 dev milestone release is June 12
18:12:51 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Juno_Release_Schedule
18:13:24 <SergeyLukjanov> #info j1 freeze / m-p branch cut is June 10
18:13:32 <SergeyLukjanov> any question re j1?
18:14:10 <SergeyLukjanov> #topic Action items from the last meeting
18:14:14 <aignatov> no questions :)
18:14:23 <SergeyLukjanov> [WIP] SergeyLukjanov to prepare -specs infra for pilot
18:14:39 <SergeyLukjanov> we already have a repo for specs
18:14:49 <SergeyLukjanov> I'll configure jobs and add templates soon
18:15:19 <mattf> for later - can we get http://www.openstack.org/software/roadmap/ cleaned up? it still references savanna and spells it savannah
18:15:37 <SergeyLukjanov> mattf, I'll contact o.o maintainers
18:16:00 <SergeyLukjanov> #action SergeyLukjanov contact http://www.openstack.org/software/roadmap/ maintainers to update sahara-related things
18:16:22 <SergeyLukjanov> mattf, thanks for catching this
18:17:03 <mattf> i'm at hadoop summit all next week, so my bps should be pushed
18:17:25 <SergeyLukjanov> mattf, ack
18:17:42 <SergeyLukjanov> #topic Subprojects releasing
18:18:03 <SergeyLukjanov> so, the main question is about sahara-image-elements
18:18:43 <mattf> what are the hurdles to getting sahara-ci to be intelligent enough to avoid rebuilding images on each commit?
18:18:50 <SergeyLukjanov> my idea is to make it pypi package like it's done for dib and rework our scripts on python to make easier to support and test them
18:18:50 <alazarev> I believe it should be in sahara repo
18:19:09 <elmiko> SergeyLukjanov: +1
18:19:11 <aignatov> I agree with SergeyLukjanov
18:19:29 <alazarev> +1
18:19:39 <SergeyLukjanov> mattf, we can do it, it's not a reason itself
18:20:08 <mattf> when you say rework our scripts, do you mean convert bash -> python?
18:20:17 <SergeyLukjanov> mattf, yup
18:20:24 * mattf gulps
18:20:34 <SergeyLukjanov> mattf, as elmiko proposed in ML
18:20:43 <bob_nettleton> SergeyLukjanov, I'm not sure I agree with this.  are you talking about the Sahara elements as well, or just the top-level script.
18:20:46 <mattf> i'm not a fan of that
18:20:59 <SergeyLukjanov> and we'll be able just to add dib to requirements of sahara-image-elements
18:21:12 <elmiko> if we are going to turn it into a pypi package i think we should at least make the diskimage-create a python script
18:21:15 <SergeyLukjanov> mattf, bob_nettleton, it's only about diskimage-create
18:21:28 <SergeyLukjanov> it's now a bunch of args-handling code
18:21:58 <mattf> it can be cleaned up significantly w/o being rewritten in another language
18:22:30 <bob_nettleton> SergeyLukjanov, my concern is that it has taken a while to get the image building repo to be basically stable, so a move like this may not be desirable at this time.
18:22:40 <elmiko> my main gripe about continuing to use shell script is that adding args is a pita
18:22:59 <SergeyLukjanov> elmiko, agreed
18:23:13 <elmiko> also, most of the shell stuff could be converted to subprocess anyway
18:23:13 <mattf> elmiko, you're suggesting the elements should stay bash tho?
18:23:30 <elmiko> mattf: yea, i don't see why we need to break the elements just the head script
18:23:31 <SergeyLukjanov> anyway, question of rewriting diskimage-create on python is a separated one, it's not required for making it pypi package
18:23:37 <bob_nettleton> elmiko, sure, that makes sense. still worried about moving this project over though, just for stability reasons.
18:24:05 <SergeyLukjanov> mattf, /me and elmiko aren't proposing changing elements, only diskimage-create script
18:24:05 <elmiko> bob_nettleton: agreed, i would only propose creating a diskimage-create.py in addition until we have feature parity
18:24:34 <SergeyLukjanov> elmiko, yup, and well tested
18:24:35 <bob_nettleton> elmiko, ok, that sounds ok to me, provided we leave the main sahara elements in bash for the time being.
18:24:39 <mattf> i'm for merging the elements into the main sahara repository -- i was sold w/ the argument that we have changes that impact both at the same time
18:25:30 <elmiko> mattf: if we move elements into the main sahara repo, where does diskimage-create.sh end up ?
18:25:31 <bob_nettleton> mattf, +1, as long as the CI system can handle changes in a smart way, as you mentioned earlier.
18:26:05 <mattf> elmiko, sahara/smthn
18:26:15 <SergeyLukjanov> no pypi package, no dependency on diskimage-builder, no separated jobs in gate and etc. if we merge it into sahara
18:26:26 <mattf> contrib ? dib ? elements ?
18:26:41 <SergeyLukjanov> the whole OS infra is done to make jobs per project, not per directory
18:26:41 <mattf> what's the value of a separate pypi package?
18:27:06 <elmiko> i'm ok with moving the image-elements stuff into the main sahara package, it makes some sense to me.
18:27:41 <SergeyLukjanov> heh, I'm now sad that I've initially started this discussion on summit with my option to merge elements into sahara
18:27:51 * mattf grins
18:27:59 <mattf> you were too convincing and logical
18:28:17 <SergeyLukjanov> you guys say about lots of concerns on summit
18:28:30 <mattf> ...healthy debate
18:28:50 <SergeyLukjanov> and that's why I've re-iterated on it and for me - there are less + for merging
18:28:53 <mattf> i thought we were essentially settling on a merge
18:29:18 * mattf should have worn his devils advocate hat
18:29:55 <aignatov> instead of implementing new functional to support elements in sahara repo and sahara-ci … and instead of reworking to another launguage  we could leave it as is in current repo and just release it each milestone
18:30:18 <bob_nettleton> SergeyLukjanov, besides the separate pypi package, are there other benefits to making this change, other than the obvious one of it being nice to have everything in one project?
18:30:27 <elmiko> aignatov: path of least resistance?
18:30:27 <aignatov> it works now well, why do we need change it at all?
18:30:30 <mattf> imho having it in sync w/ the plugins is more important than "releasing" w/ a separate package
18:31:01 <mattf> i'm still not sure what the value of a separate pypi package is
18:31:01 <alazarev> aignatov: to be synced with plugins
18:31:03 <SergeyLukjanov> mattf, both repos have releases j1 - j1 and etc
18:31:10 * mattf gets pull onto the phone
18:31:24 <aignatov> elmiko: yes, I think community has a lots of more important task then moving one repo to another
18:31:29 <SergeyLukjanov> mattf, and master should work on master, we'll probably never will be able to build a fresh image for each CI job
18:31:31 <tmckay> mattf, +1 on synchronizing, the bug we found with rc.local and vanilla on Fedora as an example
18:32:11 <SergeyLukjanov> tmckay, we'll not find such bugs, because it's impossible to run full test matrix
18:32:24 <SergeyLukjanov> especially building new image in each job
18:32:39 <tmckay> SergeyLukjanov, I mean having the code in the same repo to make changes
18:32:50 <elmiko> aignatov: agreed about more important tasks
18:32:57 <SergeyLukjanov> aignatov, ++
18:33:15 <alazarev> aignatov: ++
18:33:22 <SergeyLukjanov> from the images publishing PoV - it could be done w/o magic only for separated repo
18:33:31 <SergeyLukjanov> automatically*
18:34:14 <SergeyLukjanov> so, my point is to keep it as is because: we have much more important tasks, simpler CI, packaged scripts with dib, direct dependency on dib, we could even add gate tests to make dib gating on us
18:34:30 <SergeyLukjanov> oh, that's a new idea - add gate tests to make dib gating on us
18:34:42 <SergeyLukjanov> it's possible only for separated repo I think
18:34:44 <SergeyLukjanov> in good way
18:35:10 <SergeyLukjanov> tmckay, re coupling - there are no real issues right now with it IMO
18:35:35 <tmckay> I'm fine with keeping it as is
18:35:36 <bob_nettleton> SergeyLukjanov, if adding a gate like that is possible, I think it's something to consider.  Mike's recent patch will help stabilize the diskimage-create.sh script, but there have been many breakages in DIB proper recently.
18:36:03 <SergeyLukjanov> probably merging is preventive optimization atm
18:36:33 <SergeyLukjanov> I've been proposing it to make releasing easier but miss some things that were rised on summit and after it
18:36:42 <SergeyLukjanov> raised*
18:36:53 <bob_nettleton> while I do like the idea of everything being under one project, I'm ok with keeping things as they are as well.
18:37:21 <elmiko> i think the dib gating issue should be investigated more, i like the idea of merging but i agree there are a lot of details and we have higher priority issues.
18:38:22 <SergeyLukjanov> any more objections for keeping sahara-image-elements as is?
18:39:21 <aignatov> actually I’m ok with moving elements to sahara, there are a lot of advantages keeping it together but lets do it not in this release cycle :)
18:40:01 <elmiko> aignatov: +1
18:40:17 <SergeyLukjanov> currently I'm absolutely against moving them to sahara but we could re-iterate on it next cycle when we'll have more experience of working with it
18:40:18 <aignatov> just lets specify releasing strategy for elements project, it are minimal efforts :)
18:40:59 <SergeyLukjanov> so, I'm writing agreed message
18:41:25 <SergeyLukjanov> #agreed keep sahara-image-elements releasing as is
18:41:37 <SergeyLukjanov> #undo
18:41:38 <openstack> Removing item from minutes: <ircmeeting.items.Agreed object at 0x23b35d0>
18:41:48 <aignatov> for Juno cylce should be added
18:41:50 <aignatov> :)
18:42:04 <SergeyLukjanov> #agreed keep sahara-image-elements releasing as is, re-iterate next cycle
18:42:57 <SergeyLukjanov> and one more question is about -extra internals
18:43:03 <SergeyLukjanov> we have job samples @ extra
18:43:10 <SergeyLukjanov> swiftfs @ extra
18:43:17 <SergeyLukjanov> rest samples @ sahara
18:43:45 <SergeyLukjanov> I remember that there were some ideas about them
18:43:46 <tmckay> I think job samples could move to sahara, because integration tests use some of the same code
18:43:51 <elmiko> i like the idea of bringing the examples into sahara
18:44:01 <tmckay> in fact, all of the integration test jobs could be made examples
18:44:22 <tmckay> and, we store them with source code too and a README with a compile line
18:44:35 <tmckay> for users, and so that we know how to change them if necessary :)
18:45:10 <aignatov> as for rest samples I think we can move it to the docs :)
18:46:09 <SergeyLukjanov> tmckay, ++
18:46:11 <SergeyLukjanov> aignatov, ++
18:46:24 <elmiko> tmckay, aignatov, +1
18:46:42 <tmckay> also some of those things are in the cli tests, too, but I'm not sure how to fix that
18:46:59 <tmckay> maybe there is some way to point at sahara
18:47:42 <SergeyLukjanov> tmckay, you could add conf option that points to jars for example
18:47:52 <tmckay> ah, good idea
18:47:54 <aignatov> if all agree with moving rest samples to the docs I can take this action item on me
18:48:09 <SergeyLukjanov> #agreed move rest samples to docs
18:48:28 <aignatov> I’ll also rework them with new hadoop and changes in edp made in icehouse
18:48:37 <tmckay> I can move the edp examples, I am familiar with the use in both repos
18:48:39 <SergeyLukjanov> #action aignatov create bp re moving/updating rest samples docs and do it
18:48:47 <SergeyLukjanov> tmckay, awesome
18:48:56 <SergeyLukjanov> #agreed move edp samples to sahara
18:49:00 <aignatov> tmckay: cool
18:49:30 <SergeyLukjanov> #action tmckay create bp re moving edp samples to sahara and make test jobs examples
18:49:49 <SergeyLukjanov> it sounds like we should add a page to the docs re edp examples
18:49:56 <aignatov> yes
18:49:57 <SergeyLukjanov> instead of README file
18:50:01 <tmckay> +1
18:50:03 <elmiko> SergeyLukjanov: +1
18:50:15 <SergeyLukjanov> okay
18:50:34 <SergeyLukjanov> anything else to move?:)
18:50:42 <SergeyLukjanov> or keep as is
18:50:42 <tmckay> change the name?
18:50:45 * tmckay ducks
18:50:45 <SergeyLukjanov> hm
18:51:04 <tmckay> that was a bad joke
18:51:20 <elmiko> tmckay: savanna -> sahara -> tundra?
18:51:29 <tmckay> lol
18:51:36 <aignatov> -> desert
18:51:36 <tmckay> the Toyota people might get mad
18:51:37 * SergeyLukjanov nervously  nod
18:51:50 <SergeyLukjanov> -> arctica
18:51:56 <elmiko> nice
18:51:56 <SergeyLukjanov> -> Space
18:51:59 <elmiko> lol
18:52:04 <aignatov> -> nothing
18:52:08 <tmckay> the final frontier
18:52:11 <SergeyLukjanov> -> zeromq
18:52:16 <elmiko> oh man...
18:52:17 <tmckay> heh
18:52:21 <aignatov> lol
18:52:36 <aignatov> what’s about integrtion tests for UI? should we move it? ;)
18:52:40 <SergeyLukjanov> Hadoop on steroids == ZeroDoop
18:52:48 * elmiko likes
18:52:53 <SergeyLukjanov> (zeromq == sockets on steroids)
18:53:01 <tmckay> Beverly
18:53:14 <SergeyLukjanov> heh, 7 mins left, lets return back to the meeting
18:53:28 <SergeyLukjanov> #topic Pilot bps for -specs
18:53:36 <mattf> so we're punting on image-elements?
18:53:51 <tmckay> seems like
18:53:55 <elmiko> mattf: yea
18:53:58 <tmckay> til next time
18:53:58 <mattf> gotcha
18:54:08 <SergeyLukjanov> backward compat could be used for -specs pilot (it was proposed on prev. meeting)
18:54:35 <SergeyLukjanov> we should try some small bp too to taste how it's crazy to create spec for simple bp
18:54:35 <tmckay> or moving edp examples, should be a simple blueprint
18:54:53 <elmiko> SergeyLukjanov: +1 to trying some small bps
18:54:56 <SergeyLukjanov> good candidate for small bp try
18:55:24 <SergeyLukjanov> 5 mins left
18:55:32 <SergeyLukjanov> #topic Open discussion
18:55:36 <SergeyLukjanov> time for some q.
18:55:54 <elmiko> bob_nettleton: did you see my email about the packages for ambari?
18:57:07 <aignatov> should we send small wrap-up to ML about edp plans like it was done for releasing and backward compat of api?
18:57:26 <elmiko> aignatov: sounds like a good idea to me
18:57:34 <bob_nettleton> elmiko, yes, sorry I haven't replied yet.  I haven't had a chance to look into that.  we might need John Speidel to review your patch as well, since he's probably the expert on that area of the HDP plugin code.
18:57:40 <SergeyLukjanov> aignatov, tmckay, it'll be really great if you could collaborate on it and send to ML
18:58:03 <elmiko> bob_nettleton: ok, i'm open to expanding the detection code to look for the installed packages before giving up on an instance
18:58:32 <elmiko> bob_nettleton: my intention is that the package detection code would only run in situations where there is no connection to the internet
18:58:37 <aignatov> ok, we’ll do
18:58:37 <mattf> are we at a point w/ heat that we can enable it and start ignoring the username attr on images?
18:58:42 <tmckay> aignatov, you mean a wrapup from the EDP design session?
18:58:48 <bob_nettleton> elmiko, ok, makes sense
18:58:53 <aignatov> tmckay: yes
18:59:19 <tmckay> #action aignatov and tmckay will do a wrapup via the EDP design session from Summit via openstack-dev
18:59:32 <SergeyLukjanov> mattf, probably it's time to hide this field when heat enabled
18:59:54 <mattf> j1 or j2?
19:00:05 <SergeyLukjanov> mattf, but I prefer to enable it by default when we add some tests for heat to the gate
19:00:13 <SergeyLukjanov> mattf, for guarantees
19:00:14 <SergeyLukjanov> so, j2
19:00:18 <mattf> k
19:00:25 <SergeyLukjanov> I'll create a bp for it
19:00:30 <SergeyLukjanov> and try to make a spec
19:00:31 <aignatov> j2 seems good
19:00:36 <SergeyLukjanov> with work items
19:00:52 <SergeyLukjanov> #action SergeyLukjanov create bp with steps to enable heat be default
19:01:11 <aignatov> out of time
19:01:20 <SergeyLukjanov> #action SergeyLukjanov create bp about removing/hiding username@image for heat based provisioning
19:01:22 <SergeyLukjanov> ooops
19:01:25 <SergeyLukjanov> #endmeeting