18:10:14 <SergeyLukjanov> #startmeeting savanna
18:10:15 <openstack> Meeting started Thu Dec 12 18:10:14 2013 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:10:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:10:18 <openstack> The meeting name has been set to 'savanna'
18:10:24 <SergeyLukjanov> #topic Agenda
18:10:27 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SavannaAgenda#Next_meetings
18:10:36 <SergeyLukjanov> #topic Action items from the last meeting
18:10:54 <SergeyLukjanov> both action items from the last meeting are on me and not yet fully completed
18:11:06 <SergeyLukjanov> #action SergeyLukjanov to check that all blueprints created and ping guys to make them if not
18:11:13 <SergeyLukjanov> #action SergeyLukjanov add links to the blueprints to roadmap
18:11:35 <SergeyLukjanov> #topic News / updates
18:11:42 <SergeyLukjanov> folks, please
18:12:09 <SergeyLukjanov> we're completely moved our unit tests to be excecuted by testr
18:12:18 <SergeyLukjanov> and working on integration tests
18:12:19 <crobertsrh> I'm currently working on job relaunch from the UI.  I'm getting close having something working.
18:12:26 <mattf> i filed a cr for the foundation of the cli, i'd like lots of feedback before i add more to it - https://review.openstack.org/#/c/61565/
18:12:30 <tmckay> I am working on https://blueprints.launchpad.net/savanna/+spec/edp-oozie-java-action to add general jar jobs from oozie (as opposed to mapreduce jobs)
18:12:32 <SergeyLukjanov> in addition I'm working on tempest tests
18:12:39 <aignatov> I'm continue working on Heat integration patch, it is opened for review.
18:12:44 <aignatov> So you are welcome
18:12:58 <aignatov> basic functionality are implemented
18:13:09 <SergeyLukjanov> mattf, adding an item about cli before the open discussion
18:13:20 <aignatov> #link https://review.openstack.org/#/c/55978/
18:13:47 <SergeyLukjanov> dmitryme is working on the unified agents proposal
18:13:56 <NikitaKonovalov> integration tests appeared to dependent on nose and are failing now
18:14:06 <aignatov> dmitryme is here :)
18:14:16 <aignatov> I saw him joined recently
18:14:32 <SergeyLukjanov> NikitaKonovalov, that's because we're using nose.attr in tests, it should be replaced with testr analogue
18:14:40 <NikitaKonovalov> so need to make some chages to get rid of nose in tests code
18:14:44 <SergeyLukjanov> NikitaKonovalov, I'll take a look on it after the meeting
18:14:49 <NikitaKonovalov> ok
18:15:02 <SergeyLukjanov> any other updates?
18:15:41 <jmaron> still working on rack awareness for hdp, but now detoured into making EDP work in neutron over private networks
18:15:48 <ErikB> We have put together a script to install Savanna and are wondering if it makes sense to contribute?
18:16:01 <ErikB> (Savanna UI and API)
18:16:20 <mattf> ErikB, is it anything like the puppet module that is under review?
18:16:38 <jmaron> haven't seen much work/progress on puppet module...
18:16:39 <SergeyLukjanov> jmaron, great, but I have no ideas for now for the correct approach to solve your problem
18:16:39 <ErikB> I haven't looked at the puppet piece, so not sure.
18:16:50 <aignatov> ErikB, how is installation implemented?
18:16:56 <ErikB> bash script
18:17:02 <mattf> oh, i've one of those too
18:17:06 <jmaron> trying to work around the need to build a fully functional remote interface during periodic tasks
18:17:10 <mattf> i'm hoping to ditch it for the puppet modules
18:17:20 <aignatov> ok, I think we may add somehing to savanna-extra repo
18:17:27 <SergeyLukjanov> ErikB, the muppet manifests are under review atm
18:17:34 <ErikB> OK, I will check it out.
18:17:36 <SergeyLukjanov> ErikB, and I think it'll be merged soon
18:17:41 <jmaron> muppet?  I like it :)
18:17:43 <SergeyLukjanov> ErikB, stackforge/puppet-savanna
18:17:52 <mattf> ErikB, https://review.openstack.org/#/c/61156/
18:17:58 <SergeyLukjanov> jmaron, yup :)
18:18:05 <mattf> SergeyLukjanov, probably today
18:18:14 <SergeyLukjanov> mattf, my +2 in on it
18:18:27 <mattf> it's on my queue
18:18:39 <jmaron> (now we definitely need to create a puppet derivative called muppet)
18:18:56 <SergeyLukjanov> jmaron, puppet fork called muppet ;)
18:19:01 <mattf> jmaron, +1
18:19:17 <SergeyLukjanov> btw Jenkins was in a list of J release names ;)
18:19:54 <aignatov> names for openstack?
18:20:20 <dmitryme> aignatov: names for the release
18:20:23 <SergeyLukjanov> yep
18:20:26 <dmitryme> like Grizzly was for G
18:20:31 <dmitryme> or Havana for H
18:20:32 <mattf> https://wiki.openstack.org/wiki/Release_Naming
18:20:38 <SergeyLukjanov> I'll propose Savanna for S release
18:20:44 <dmitryme> :-)
18:21:09 <mattf> SergeyLukjanov, i've people stumbling over savanna v gnu savannah already!
18:21:28 <SergeyLukjanov> mattf, yep, I know
18:21:36 <dmitryme> they actually have very strict naming convention - use names of places nearby Summit locaiton
18:21:57 <SergeyLukjanov> dmitryme, maybe the S summit will be somewhere in savanna
18:22:11 <ErikB> :-)
18:22:22 <aignatov> but from Russian point of view it looks very good because Savanna is Саванна :)
18:22:25 <alazarev> or in Savannah :)
18:23:17 <SergeyLukjanov> funny report - http://stackalytics.com/report/users/slukjanov
18:23:34 <SergeyLukjanov> ok, looks like there are no more updates
18:23:36 <SergeyLukjanov> let's move on
18:23:47 <SergeyLukjanov> #topic Roadmap update / cleanup
18:23:55 <SergeyLukjanov> no updates from the last meeting
18:24:02 <SergeyLukjanov> action items are still on me
18:24:12 <SergeyLukjanov> I hope that I do it thos week
18:24:13 <mattf> SergeyLukjanov, i think that graphic suggests you need a vacation
18:24:34 <SergeyLukjanov> mattf, planning 2w vacation from end of Dec
18:25:26 <aignatov> on the next week?
18:25:29 <aignatov> from
18:25:45 <alazarev> SergeyLukjanov: this is not vacation, this is just russian holidays, aren't they?
18:26:17 <SergeyLukjanov> alazarev, yup, but I'm extending it to be full 2w
18:26:34 <SergeyLukjanov> #topic savanna client cli
18:26:43 <SergeyLukjanov> mattf, please, could you please amke a small intro
18:26:46 <SergeyLukjanov> make*
18:27:00 <mattf> sure, there's nearly a savanna cli
18:27:08 <mattf> https://blueprints.launchpad.net/python-savannaclient/+spec/python-savannaclient-cli
18:27:19 <mattf> initial commit is https://review.openstack.org/#/c/61565/
18:27:36 <mattf> the foundation is based on novaclient, with minimal changes
18:27:55 <mattf> i'd appreciate feedback both on the blueprint (incomplete) and especially on the initial commit.
18:28:10 <mattf> i don't want to go too far into the cli impl if we aren't agreed on the basics
18:28:13 <mattf> .
18:28:23 <SergeyLukjanov> I don't really like it looks like in nova client... have you tried to diff into the some other clients?
18:28:58 <aignatov> mattf, my first comment - please remove all unused commented code ;)
18:29:08 <mattf> will you quantify "don't really like it looks like"?
18:29:23 <SergeyLukjanov> to clarify - mattf, I really appreciate your work on it
18:29:33 <mattf> aignatov, so i'm purposely keeping that code so we have an idea of how far we've drifted from nova
18:29:48 <SergeyLukjanov> mattf, I've seen into the neutron client, it looks more object oriented
18:29:59 <SergeyLukjanov> mattf and testable/supportable
18:30:11 <SergeyLukjanov> mattf, but of course, a lot of lines of code
18:30:26 <jmaron> the nova comparison is with regard to command options, code structure, or both?
18:30:37 <jmaron> command option syntax
18:30:43 <mattf> SergeyLukjanov, i didn't dig into the neutron code. the keystone folks wanted me to use theirs, but it was only a partial implementation.
18:30:43 <SergeyLukjanov> and there is a common client in oslo, I don't actually know which projects are using it
18:31:05 <mattf> SergeyLukjanov, seemed like none
18:31:20 <SergeyLukjanov> SergeyLukjanov, yep
18:31:23 <SergeyLukjanov> oh
18:31:30 <SergeyLukjanov> SergeyLukjanov, how are you?
18:31:36 <SergeyLukjanov> SergeyLukjanov, fine, thx
18:31:39 <tmckay> :)
18:31:44 <mattf> jmaron, code structure. many OS CLIs are based on novaclient. so when they ahve to migrate to something shared, we'll be close to whatever migration path is created.
18:31:58 <jmaron> mattf, thx
18:32:02 <mattf> SergeyLukjanov, vacation, definitely vacation
18:32:08 <SergeyLukjanov> mattf, :)
18:32:33 <SergeyLukjanov> mattf, in two words I really like to have a cli in our client
18:32:40 <mattf> SergeyLukjanov, does neutron already have a test harness for their cli?
18:32:52 <SergeyLukjanov> mattf, I'm not sire
18:32:55 <SergeyLukjanov> sure*
18:33:09 <mattf> that'd be a nice reason to jump to another foundation
18:33:44 <SergeyLukjanov> there tons of tests here
18:33:45 <SergeyLukjanov> https://github.com/openstack/python-neutronclient/tree/master/neutronclient/tests/unit
18:33:55 <mattf> basically, i'm very happy to steal from other clients for the framing. i'm really just interested in adding savanna specific walls and furniture
18:34:08 <aignatov> mattf, did you look at heat client, it looks nice and simple
18:34:19 <aignatov> and contains shell tests :)
18:34:22 <aignatov> https://github.com/openstack/python-heatclient
18:34:38 <SergeyLukjanov> mattf, it'll be great if you take a look on 'new' clients
18:34:53 <mattf> aignatov, i did, the main reason to go w/ nova is migration path
18:35:28 <mattf> the keystone folks are trying to create a standard client, with security done right
18:35:31 <mattf> but it's not done
18:35:50 <SergeyLukjanov> mattf, :(
18:35:51 <mattf> i imagine that and others will eventually migrate to something in oslo, but it doesn't exist right now
18:36:05 <mattf> and my aim is to make a savanna cli, not a framework for creating clis
18:36:36 <aignatov> maybe we should create new initiative in OpenStack, CLI as a service ;)
18:36:37 <mattf> sounds like folks would like me to take a second look at heat and neutron?
18:36:49 <mattf> aignatov, indeed, you can be ptl
18:36:57 <mattf> when you have something that works, i'll migrate to it
18:36:59 * mattf grins
18:37:01 <SergeyLukjanov> aignatov, CLIaaS
18:37:07 <aignatov> mattf, lol
18:37:35 <jmaron> without the 'I' would be classier...
18:37:49 <SergeyLukjanov> jmaron, nice)
18:37:58 <mattf> SergeyLukjanov, oh wait
18:38:13 <SergeyLukjanov> mattf, I'm here, don't worry :)
18:38:14 <mattf> i remember neutronclient now. instead of having a module system then just do it all in a single shell.py
18:38:45 <mattf> (forgive me, i reviewed most of the clients back around summit time)
18:39:21 <mattf> i wasn't too interested in that structure, especially since we'll have to have multiple api versions
18:39:35 <mattf> we arguably already have 1.0 and 1.1, soon also 2.0
18:39:46 <mattf> though we've stuck them together in a single "api" module
18:40:03 <SergeyLukjanov> mattf, it looks like neutronclient have a file/class per each operation
18:40:16 <mattf> SergeyLukjanov, so that and the fact no one else i could see was using neutron is why i backed away from it
18:40:33 <SergeyLukjanov> https://github.com/openstack/python-neutronclient/blob/master/neutronclient/neutron/v2_0/network.py#L114
18:40:40 <SergeyLukjanov> mattf :)
18:40:47 <mattf> SergeyLukjanov, the way they pull it together isn't as flexible as the api version approach
18:41:21 <SergeyLukjanov> mattf, btw I'm just trying to cover all approaches
18:41:36 <SergeyLukjanov> mattf, nova client isn't bad approach by default
18:42:21 <SergeyLukjanov> so if we do something not too bad, it'll be enough to understand what is bad and migrate earlier
18:42:27 <mattf> aignatov, heatclient is definitely simpler than novaclient. it actually duplicates some of the functionality we provide in our Client() too
18:42:57 <mattf> SergeyLukjanov, i'm happy for the line of questioning, keeps decisions open and well reasoned
18:43:02 <mattf> (or at least partially reasoned)
18:43:57 <mattf> in the end, the impl is pretty simple. there's a shell.py in savannaclient that loads other shell.py from variable api version modules. the api version shell.py implements do_blah() methods that are the cli verbs
18:44:09 <mattf> so we have savannaclient/shell.py and savannaclient/api/shell.py
18:44:43 <mattf> the outer one also parses the auth information and creates the Client() for use w/i the do_ methods
18:45:09 <aignatov> mattf, I'd like this approach
18:45:10 <mattf> aignatov, how strongly do you feel about heat, because we can debate it some more after the meeting
18:45:34 <aignatov> mattf, i've just had a quick look during this meeting
18:45:55 <SergeyLukjanov> mattf, I'll agree with something that'll work ok :)
18:46:02 <jmaron> curious:  do we see a need for savanna-manage client, crossing tenant boundaries for management tasks across cluster instances?
18:46:33 <SergeyLukjanov> jmaron, I think it'll be eventually done by using admin role
18:46:40 <aignatov> and I've compared heatclient and novaclient code in your patch and make a opinion :)
18:47:00 <mattf> jmaron, the client allows you to change the tenant you're working against via env or arg
18:47:22 <mattf> aignatov, you'll render an opinion into the review today?
18:47:46 <jmaron> understood.  Just thought an admin may want to get a full picture rather than iterate across tenants
18:48:06 <mattf> jmaron, ahhh, that's an interesting idea
18:48:16 <mattf> i don't think the current api allows that very easily
18:48:23 <jmaron> true
18:48:44 <aignatov> mattf, I'll try
18:49:06 <dmitryme> As for the UI, there is an Admin tab here which allows browsing resources regardless of tenant
18:49:15 <dmitryme> it might be worth checking how it works
18:49:22 <mattf> ok, take the rest of this to the review / #savanna ?
18:49:25 <SergeyLukjanov> jmaron, mattf, current API doesn't allow to do it, but I think that we should add it to v2
18:49:36 <jmaron> +1
18:49:42 <SergeyLukjanov> looks like a time for open discussion
18:49:48 <SergeyLukjanov> #topic Open Discussion
18:50:37 <tmckay> edp question, we can follow up on #savanna.  With the addition of java actions to oozie support, I think we need a name change for job types.  Currently we have Hive, Pig, and JAR.
18:50:52 <tmckay> I think we need Hive, Pig, Mapreduce, and ??
18:51:03 <tmckay> Because current JAR really means "mapreduce"
18:51:38 <tmckay> it will be a different workflow generator, arguments allowed, ect
18:51:50 <akuznetsov> tmckay JavaAction?
18:51:52 <dmitryme> tmckay: and what does added java action do?
18:52:26 <akuznetsov> possible we should add a oozie workflow
18:52:30 <tmckay> Java actions allow Oozie to run a main(), instead of building a mapreduce job out of the specified mapper and reducer classes
18:52:44 <dmitryme> aha, I see
18:52:46 <tmckay> so, the hadoop example pi estimator is a java action
18:52:59 <tmckay> It can't be run as mapreduce directly without a rewrite
18:53:05 <dmitryme> indeed it is hard to set a different names
18:53:24 <tmckay> java action launches a single mapper, which runs main(), which typically launches other mappers and reducers
18:53:36 <jmaron> I see a HelloWorld example coming down the pipe….
18:53:43 <aignatov> tmckay, just Java, but JAR should be renamed to MapReduce defenitely
18:54:01 <tmckay> Yes, that was my thought.... Hive, Pig, MapReduce, and Java
18:54:02 <dmitryme> aignatov: makes sense
18:54:19 <SergeyLukjanov> are there any job types in Oozie?
18:54:27 <tmckay> that will cause changes in the UI, docs, etc...
18:54:29 <SergeyLukjanov> we can use them if yes
18:54:39 <tmckay> I think just the action names
18:54:43 <SergeyLukjanov> yep, but looks like we really need i
18:54:56 <tmckay> oh, agreed, just making a note :)
18:55:00 <SergeyLukjanov> Hive, Pig, MapReduce, and Java works for me
18:55:05 <SergeyLukjanov> tmckay :)
18:55:25 <tmckay> crobertsrh, this means you too ^^ :)
18:55:54 <aignatov> agreed with akusnetsov, Oozie workflow should be supported as well :) but didm;t know how it works :) :)
18:56:02 <tmckay> akuznetsov, yes, we should add oozie workflow from user too
18:56:17 <tmckay> first, java.  Then, oozie.
18:56:26 <tmckay> Christmas present for me, heh
18:56:31 <SergeyLukjanov> and it'll be great to be ablt to configure oozie to take jobs from swift
18:56:36 <aignatov> Btw, MapReduce action has sub actions streaming and pipe
18:56:50 <tmckay> yes, also on the roadmap I think
18:57:04 <akuznetsov> yes oozie is most complicated because it should involve all types of job pig, hive jar etc...
18:57:50 <aignatov> also, tmckay, you know that right now EDP uses oozie-workflow schema 0.2, but we have 4.0.0 ooze in images which allows to run on schema 0.4 and greater
18:58:13 <akuznetsov> streaming and pipe will be required some image preparation if for example pipes will use a some specific python
18:58:17 <aignatov> which Oozie is used in HWX plugin?
18:58:24 <tmckay> aignatov, didn't know that.  Should we switch the schema to 0.4?
18:59:08 <jmaron> aignatov, not sure about version number.  whatever is distributed with HDP 1.3.2
18:59:11 <SergeyLukjanov> we're out of time guys, 1 min left
18:59:12 <tmckay> any need to support both, with a config?
18:59:14 <aignatov> tmckay: If it have new features allowing you to create workflows in simple manner, why not? :)
18:59:27 <jmaron> quick note:  I've run into an issue with periodic tasks requiring access to a full context (specifically, the service catalog etc).  Trying to work around it, but it makes me think there may be a need for defining "privileged" periodic tasks or allowing such access to existing periodic tasks?  Will ruminate some more, may send email out to list at some point….
19:00:03 <mattf> jmaron, yikes
19:00:09 <SergeyLukjanov> jmaron, it's needed to query neutron, yes/
19:00:10 <aignatov> I only know that 0.3 or 0.4 Oozie version contains <global> tag which allow to apply job tracker and name node definitions in a single  place
19:00:10 <SergeyLukjanov> ?
19:00:17 <SergeyLukjanov> thank you all!
19:00:18 <SergeyLukjanov> #endmeeting