18:07:36 <SergeyLukjanov> #startmeeting savanna
18:07:37 <openstack> Meeting started Thu Aug 22 18:07:36 2013 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:07:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:07:40 <openstack> The meeting name has been set to 'savanna'
18:07:44 <aignatov> hi
18:07:48 <rnirmal> hi
18:07:54 <ruhe> hello
18:07:54 <SergeyLukjanov> o/
18:08:08 <SergeyLukjanov> #topic Agenda
18:08:09 <SergeyLukjanov> #info News / updates
18:08:09 <SergeyLukjanov> #info Roadmap cleanup / update
18:08:09 <SergeyLukjanov> #info Action items from the last meeting
18:08:09 <SergeyLukjanov> #info PTL election results
18:08:09 <SergeyLukjanov> #info REST API refactoring discussion
18:08:09 <SergeyLukjanov> #info EDP discussions
18:08:10 <SergeyLukjanov> #info General discussion
18:08:23 <SergeyLukjanov> #topic News / updates
18:08:33 <sacharya> hi
18:08:45 <SergeyLukjanov> folks, let's share news
18:09:00 <SergeyLukjanov> first of all, we merged conductor code
18:09:14 <SergeyLukjanov> and migration to the new db layer has been completed
18:09:55 <SergeyLukjanov> #info conductor code merged, migration to the new db layer completed
18:10:08 <akuznetsov> On edp part we have a REST API for data source, job and job origin creation
18:10:11 <aignatov> it's me with Oozie
18:10:15 <aignatov> again
18:10:18 <SergeyLukjanov> #info first part of UI code for EDP merged
18:10:49 <nadya> in edp part we have a prototype that successfully works with pig-workflow creation and running
18:10:51 <mattf> #info savanna-image-elements (savanna-extra) and python-savannaclient packages have been accepted into Fedora 19. https://bugzilla.redhat.com/show_bug.cgi?id=998702  and https://bugzilla.redhat.com/show_bug.cgi?id=9987021 respectively.
18:10:54 <SergeyLukjanov> #info integration tests for HDP plugin added
18:11:04 <aignatov> i've updated vanilla plugin to be able to start oozie service with needed heap size
18:11:08 <akuznetsov> I hope tomorrow we have a first draft for scheduling job on demand cluster
18:12:26 <SergeyLukjanov> sounds great
18:12:32 <mattf> topic for general discussion - for 0.3, should we plan to release to Fedora & EPEL at the same time as OpenStack or should Fedora & EPEL happen after?
18:12:39 <SergeyLukjanov> any other news/updates?
18:12:58 <SergeyLukjanov> mattf, ok, let's discuss it in the general section
18:13:06 <nadya> guys from RedHat began to work with binaries
18:13:06 <aignatov> also we've backported some fixes related to hdp plugin
18:13:22 <aignatov> to stable/0.2
18:13:35 <SergeyLukjanov> #info EDP support partially added to the client
18:13:51 <SergeyLukjanov> ok, let's move on
18:14:01 <SergeyLukjanov> #topic Roadmap cleanup / update
18:14:21 <SergeyLukjanov> are there any thoughts about adding/updating points to the Roadmap?
18:14:39 <SergeyLukjanov> ok, great :)
18:14:46 <SergeyLukjanov> #topic Action items from the last meeting
18:14:59 <SergeyLukjanov> #info there are no action items from the last meeting
18:15:02 <mattf> nothing from me for this week. i did notice a blog post about savanna that identified us as planning to incubate for 1.0, source was the roadmap.
18:16:17 <ruhe> can you give us a link?
18:16:41 <mattf> i'll try to dig it up later
18:16:55 <ruhe> thanks
18:16:59 <SergeyLukjanov> ok, let's move on
18:17:04 <SergeyLukjanov> #topic PTL election results
18:17:07 <mattf> #info Congrats to SergeyLukjanov, who won the PTL election 14-0 - https://wiki.openstack.org/wiki/Savanna/PTL
18:17:07 <SergeyLukjanov> mattf, please
18:17:27 <ruhe> congrats Sergey
18:17:30 <mattf> i'll post the results to os-dev as well
18:17:32 <SergeyLukjanov> mattf, good news ;)
18:17:37 <rnirmal> congrats..
18:17:57 <SergeyLukjanov> mattf, thank you for preparing and driving election process!
18:17:59 <mattf> the electorate contained 20 voters, so SergeyLukjanov got over 50%, which is nice given he was uncontested
18:18:05 <rnirmal> nice "Sergey Lukjanov (14) won against None (0)"
18:18:14 <tmckay> wow, 14-0
18:18:17 <aignatov> great news
18:18:23 <tmckay> None got None
18:18:25 <mattf> SergeyLukjanov, you're all welcome, and congrats to you Sergey
18:18:47 <SergeyLukjanov> thanks :)
18:18:57 <SergeyLukjanov> #info REST API refactoring discussion
18:19:04 <SergeyLukjanov> rnirmal, please
18:19:05 <rnirmal> #link https://blueprints.launchpad.net/savanna/+spec/refactor-v1api
18:19:26 <rnirmal> that's the blueprint. I have some notes at #link https://etherpad.openstack.org/savanna_refactor_v1_api
18:20:09 <mattf> ruhe, http://bighadoop.wordpress.com/2013/08/18/openstack-savanna-fast-hadoop-cluster-provisioning-on-openstack/
18:20:13 <rnirmal> It may seem a whole lot to tackle at once...but I'd like to discuss each item before I or someone else can proceed to implement those changes
18:20:47 <ruhe> mattf, thanks again
18:20:54 <rnirmal> how do you'll want to proceed.. can I talk about one or 2 items today ?
18:21:36 <rnirmal> I'm trying to put some high level points here #link https://wiki.openstack.org/wiki/Savanna/ApiUpdates
18:21:59 <SergeyLukjanov> rnirmal, maybe it'll be more useful to start a thread in ML about it to involve more folks to it? btw we can discuss several items today
18:22:26 <rnirmal> SergeyLukjanov: yes I'll do that as well, that may be more helpful for a better conversation
18:22:33 <ruhe> rnirmal, so the goal is to separate the concerns for a Deployer and User. Is it a common practise in other OS components?
18:23:00 <rnirmal> ruhe: yes thru making parts of the extensions or api as admin_only
18:23:37 <rnirmal> I don't think any other project other than Trove that has a separate admin api
18:23:38 <SergeyLukjanov> rnirmal, yep, we're thinking about adding admin_only apis
18:24:08 <sacharya> And also, only mandatory features thats common between all plugins should be in core.. so users can theoretically switch between providers
18:24:09 <rnirmal> a separate admin api would be nice... but making some admin_only extensions will be a good place to start
18:24:38 <rnirmal> true that as well... I think so far both the plugins are on par
18:24:56 <SergeyLukjanov> rnirmal, we have some plans on updating REST API code from flask to more OS common solutions
18:25:22 <rnirmal> SergeyLukjanov: yeah there's been some discussions on using a common solution, but don't think one has emerged
18:25:25 <SergeyLukjanov> currently there is no ability to add extensions easily
18:25:48 <SergeyLukjanov> I don't think that it could be done in 0.3 version
18:25:51 <rnirmal> sorry I'm not too familiar with flask..
18:26:08 <tmckay> I think keystone is an example of an api split between admin and user, iirc
18:26:10 <rnirmal> which maybe fine
18:26:19 <SergeyLukjanov> but it's a good time to discuss how the 2.0 API whould looks like
18:26:26 <rnirmal> thank tmckay yeah keystone has that
18:26:41 * tmckay loves flask
18:26:56 <rnirmal> so I guess before we start off... 2.0 is how you'll want it to go rather than updating 1.0 ?
18:27:21 <SergeyLukjanov> currently the are a lot of code done that depends on API - client, dashboard, etc, so we don't want to break it
18:28:05 <SergeyLukjanov> rnirmal, currently we have now plans for the 2.0 REST API, but it looks like that it's a great candidate to be the next REST API polished and more functional than current one
18:28:47 <rnirmal> SergeyLukjanov: would you consider that as the primary API we could say go to incubation with ?
18:29:39 <SergeyLukjanov> rnirmal, I think we have no time to rework the API from scratch before the 0.3 release
18:30:02 <rnirmal> SergeyLukjanov: agree it's going to take some effort.. when is 0.3 slated for ?
18:30:11 <SergeyLukjanov> mid of Oct
18:30:22 <sacharya> and when is the plan to apply for incubation?
18:30:41 <SergeyLukjanov> we'll send an incubation application in several weeks
18:30:53 <rnirmal> hmm we could depending on the extent of the changes...
18:31:17 <rnirmal> but I think a 2.0 api will give us a little more freedom to work things out
18:31:49 <SergeyLukjanov> rnirmal, yep, it could be discussed and designed much better if we'll have much time to do it
18:33:53 <rnirmal> totally agree... do we have a consensus there
18:34:07 <SergeyLukjanov> any objections for moving API improvements to the next release?
18:34:26 <crobertsrh> Makes sense to me
18:34:31 <nadya> rnirmal, do you think that migration to new API may be done gradually?
18:35:09 <rnirmal> nadya: as in not make it too drastically different from the current 1.0?
18:35:18 <SergeyLukjanov> #info move API improvements to the next release (REST API v. 2.0 ?)
18:36:54 <SergeyLukjanov> any other thoughts?
18:36:54 <rnirmal> it's going to change some existing workflows in how the current api is getting used. so from that case there would be a migration
18:37:18 <rnirmal> but hopefully most of the backend will remain close to what it is now
18:39:02 <SergeyLukjanov> #action rnirmal to start thread in ML to discuss REST API improvements
18:39:19 <SergeyLukjanov> rnirmal, thank you
18:39:24 <SergeyLukjanov> #topic EDP discussions
18:39:33 <rnirmal> thanks SergeyLukjanov I'll create a new blueprint for 2.0 and work from that
18:39:38 <rnirmal> and delete the existing one
18:39:55 <ruhe> rnirmal, i'd suggest to announce such blueprints in the mailing list, since IRC is not a reliable channel
18:40:09 <aignatov> agreed with ruhe
18:40:12 <SergeyLukjanov> +1
18:40:18 <SergeyLukjanov> open discussion for EDP questions
18:40:23 <rnirmal> ruhe: ok will do.. yeah I figured it was so.. I've been trying to talk about it on irc for the past 2 weeks :)
18:40:53 <rnirmal> I've got an EDP question
18:41:12 <rnirmal> why 1.1 api for EDP? should be look at keeping the EDP and provisioning api's separate
18:41:18 <rnirmal> with it's own versioning?
18:42:04 <rnirmal> I wasn't involved early on so not sure what the thought process was like
18:42:05 <SergeyLukjanov> rnirmal, we have no plans for doing it now, it should be discussed too
18:42:15 <rnirmal> ok
18:42:17 <SergeyLukjanov> can't see any pros and cons now for doing it
18:42:34 <rnirmal> yeah don't want to change it now, but just wondering
18:42:38 <SergeyLukjanov> but I think that EDP depends on the main API
18:43:15 <ruhe> my opinion - let's keep things simple for now. if we'll see that separate versioning is really necessary, then make it's own versioning
18:43:16 <akuznetsov> rnirmal  EDP can add some feature to main API for example transient cluster
18:44:36 <rnirmal> ok looks like there's some reason to keep it layered than separate
18:44:48 <rnirmal> cool thanks.
18:45:30 <tmckay> I have a minor question related to EDP rest API.  I've noticed that we don't have a good handler for "duplicate" errors from the db.
18:45:45 <tmckay> "error_code": 500,
18:45:45 <tmckay> "error_message": "Internal Server Error",
18:45:45 <tmckay> "error_name": "INTERNAL_SERVER_ERROR"
18:45:48 <tmckay> is typical
18:46:17 <tmckay> This suggests to me a "check_not_exist" validation on the urls.  I chatted with nadya about this yesterday.  Thoughts?
18:46:31 <tmckay> (for create calls, that is)
18:46:54 <SergeyLukjanov> tmckay, there are some problems with handling several errors...
18:47:13 <rnirmal> hmm that should be a 400 bad request, suggesting duplicate information.
18:47:22 <rnirmal> not sure if 409 would be applicable in this case
18:47:30 <rnirmal> since the resource does not exist yet
18:47:33 <SergeyLukjanov> rnirmal, yep, 400 looks better
18:47:46 <SergeyLukjanov> tmckay, could you please create an issue for it?
18:48:02 <tmckay> yes.  Should it be blueprint, or just an issue?
18:48:12 <aignatov> just an issue
18:48:15 <rnirmal> just looks like a bug
18:48:23 <tmckay> will do
18:48:36 <ruhe> i like everything which would help users to understand the reason of error, and provide more information to us developers in the logs
18:48:49 <tmckay> absolutely
18:50:11 <SergeyLukjanov> any other questions/thoughts?
18:50:39 <SergeyLukjanov> #action tmckay to create an issue about incorrect handling of "duplicate" errors
18:51:13 <aignatov> I think we should have an validation logic to check if Oozie service presents on existing cluster running for EDP purposes
18:51:22 <aignatov> if doen't present
18:51:33 <aignatov> stop runing jobs
18:51:47 <aignatov> starting job)
18:52:25 <SergeyLukjanov> sounds reasonable
18:53:15 <akuznetsov> yes we should add this check
18:53:23 <SergeyLukjanov> #action aignatov to create an issue about lack of the validation while starting job (check for Oozie before it)
18:53:40 <SergeyLukjanov> ok, let's move on
18:53:42 <SergeyLukjanov> #info General discussion
18:53:42 <nadya> will we support several swifts: one for input data and one for output?
18:54:12 <mattf> when we do a release we produce tarballs for the openstack community. i'd like opinions on if (a) we should try to synchronize release of packages to fedora+epel or (b) we should have fedora+epel lag the openstack community release
18:54:13 <nadya> I mean one swift from other clour e.g.
18:54:19 <nadya> *cloud
18:55:18 <rnirmal> mattf: b
18:55:34 <ruhe> mattf, what shall we do when we have ubuntu packages? (b) is the only option imho
18:55:54 <SergeyLukjanov> absolutely agree with ruhe
18:55:55 <akuznetsov> also i think there is also valid use case for copy data from one tenant to other
18:56:10 <mattf> ruhe, what makes you say that?
18:56:27 <SergeyLukjanov> and additionally it's not a common process for other OS projects
18:56:47 <mattf> SergeyLukjanov, yup
18:57:29 <ruhe> well, if we choose (a) we'll sync with fedora+epel. then we'll create ubuntu packages and we'll want to sync with ubuntu part. it looks too complex to me
18:57:30 <aignatov> rnirmal, do you have plans to finish your bugfix patch https://review.openstack.org/#/c/42005/
18:57:47 <rnirmal> aignatov: yes added more more check will push that out soon
18:57:53 <mattf> ruhe, (a) is only about pushing packages to fed+epel, not synchronizing our release with theirs
18:58:02 <ruhe> oh, i see now
18:58:20 <mattf> the impact of (a) would be some time window to do release engineering work to prepare the push before official release
18:58:51 <aignatov> rnirmal, ok, great
18:59:09 <rnirmal> aignatov: added the fix for your comment.. just went back and looked at where all it was used
18:59:39 <aignatov> ok
18:59:45 <ruhe> mattf, since you're the only person who can do that, i think it might be a SPOF for release of Savanna
19:00:00 <SergeyLukjanov> mattf, you mean to try to release packages to fed+epel at the same time w/o moving savanna release date?
19:00:01 <aignatov> it seems we are out of time in this channel
19:00:05 <mattf> could be, that power can be expanded tho
19:00:10 <SergeyLukjanov> aignatov, yep
19:00:20 <mattf> SergeyLukjanov, yeah
19:00:23 <SergeyLukjanov> thank you folks, let's move to the #savanna channel
19:00:31 <mattf> oops, ok
19:00:48 <SergeyLukjanov> #info JFYI you can always use openstack-dev@lists.openstack.org mailing lists and #savanna irc channel to find us and ask your questions
19:00:54 <SergeyLukjanov> #endmeeting