20:00:28 <harlowja> #startmeeting state-management
20:00:29 <openstack> Meeting started Thu Jun 13 20:00:28 2013 UTC.  The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:32 <openstack> The meeting name has been set to 'state_management'
20:00:35 <harlowja> hi all!
20:00:48 <kebray> hello
20:00:50 <harlowja> hey hey
20:00:56 <kchenweijie> hi
20:01:06 <harlowja> yo yo
20:01:14 <jlucci> hurr
20:01:22 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement#Agenda_for_next_meeting
20:01:37 <harlowja> not so much from my end to discuss, everyone busy doing *work* i thinks
20:02:00 <harlowja> i invited jgriffith so maybe we can also talk about cinder more stuffs
20:02:16 <harlowja> H2 targets and H3 and such
20:02:51 <harlowja> i guess go over action items from last time first
20:03:00 <harlowja> #link http://eavesdrop.openstack.org/meetings/state_management/2013/state_management.2013-06-06-20.00.html
20:03:13 <harlowja> no adrian, so can skip that :-p
20:03:22 <harlowja> i didn't get around to the launchpad openness issue yet :(
20:03:43 <harlowja> *seems like the launchpad is restricted access only, unsure how to open from restricted
20:04:09 <jgriffith> :)
20:04:15 <harlowja> jlucci i think u did some stuff also
20:04:17 <jgriffith> harlowja: sorry.. late as usual
20:04:21 <harlowja> np
20:04:41 <harlowja> just going over anything we said was an action item from last week
20:04:42 <jlucci> Yeah, so starting on my first action item - abstracting out celery so user only sees taskflow task class
20:04:49 <harlowja> woot
20:05:00 <harlowja> and there was a mini-meeting with heat people
20:05:01 <jlucci> Using .provides and .requires from your task annotations are coming in supah handy
20:05:06 <hemna> harlowja, did you want me to pull the patch and test again yet?
20:05:21 <harlowja> hemna u can if u want, i'm still doing local testing (slowly but surely)
20:05:27 <hemna> ok
20:06:16 <harlowja> so all good stuff, i think the takeaway from the heat meeting, was that they have a little bit of the same stuff as taskflow and we should come up with a plan to figure out what goes where and how
20:06:31 <harlowja> *so more to dicsuss there i think
20:07:26 <kebray> harlowja:  agreed.
20:07:34 <harlowja> coolness
20:07:39 <jlucci> I've been talking with Randall on how to go about some of that
20:07:45 <harlowja> so thx jlucci !
20:08:21 <harlowja> it'd be nice to document that stuff, after it becomes less 'cloudy'
20:08:53 <jlucci> agreed
20:09:00 <harlowja> so do we want to do a little round of what is everyone working on, would that be useful?
20:09:24 <kebray> I have two things for when we get to "Open Topics".. just fyi.
20:09:28 <harlowja> k
20:09:50 <harlowja> #topic round-of-things-happening
20:10:37 <harlowja> so i'm bringing up grizzly on y!s openstack vms, testing cinder stuff there, slowly making progress, getting some good code review feedback from jgriffith and others
20:10:47 <harlowja> #link https://review.openstack.org/#/c/29862/
20:11:00 <harlowja> also having to attend all there annoying internal meetings :-p
20:11:05 <harlowja> *all these
20:11:05 <jgriffith> hehe :)
20:11:55 <harlowja> thats about what i am doing, working on http://anvil.readthedocs.org/ since thats my setup tool of choice (i build that so have to use it)
20:12:04 <harlowja> *built that*
20:12:42 <harlowja> jlucci kchenweijie what have u guys been doing :-P
20:12:56 <harlowja> *or anyone else doing stuff out there in the ether?
20:12:58 <jlucci> so
20:13:04 <jlucci> https://wiki.openstack.org/wiki/DistributedTaskManagement
20:13:21 <harlowja> jlucci u look like the progressive insurance lady??
20:13:24 <harlowja> haha
20:13:34 <jlucci> Wrote up some documentation on the distributed approach to accompany the code/explain why distributed should be used
20:13:34 <jlucci> lol
20:13:41 <jlucci> Yeah. That's totally my picture up there.
20:13:48 <harlowja> lol
20:14:01 <harlowja> i knew it
20:14:11 <jlucci> Other than that. Still working on test cases/fleshing out code. And then also abstracting out celery through the taskflow task class
20:14:24 <harlowja> very awesome
20:14:29 <jlucci> Yeah…people stop me in the street all the time. so annoying. :P
20:14:34 <harlowja> ha
20:14:54 <harlowja> *stop trying to buy insurance from me gosh darn it
20:14:55 <harlowja> ha
20:15:07 <kchenweijie> i guess its my turn now :P
20:15:09 <kebray> Well, I'll cover one of my Open Topics now:    harlowja, to build a little more on the conversation we were having yesterday about distributed execution engines for data-dependent control flows.  This paper describes some of the advantages in the intro: http://research.microsoft.com/pubs/173947/thesis-corrected.pdf
20:15:15 <kchenweijie> or not xD
20:15:31 <kebray> sorry kchenweijie.. I'm just piling on to jlucci's update.
20:15:37 <kchenweijie> no problem
20:15:58 <kebray> Anyway.. that's bedtime reading for later discussion.
20:16:06 <kebray> we can move on.
20:16:09 <harlowja> i wonder what the 'thesis-uncorrected' was :-P
20:16:49 <harlowja> cool kchenweijie what have u been up to
20:16:56 <harlowja> kebray def something to read over :)
20:17:10 <kchenweijie> ive been working towards making the general api we discussed yesterday
20:17:35 <kchenweijie> ive made a basic template for what each backend api should be able to do and im working towards making the current sqlalchemy api match that
20:17:43 <harlowja> awesome
20:17:53 <kchenweijie> right now, i only have the persistence portion of the full api, but i want to make sure this works before i continue
20:18:00 <harlowja> makes sense to me
20:18:16 <kchenweijie> thats it for me
20:18:22 <harlowja> jgriffith this will help provide the persistance layer for tasks that can be pretty nice for future awesomeness in cinder
20:18:25 <harlowja> *and elsewhere*
20:18:55 <harlowja> it will be a key part of everything in the futurei  hope :)
20:19:01 <harlowja> no pressure kchenweijie  ;)
20:19:08 <kchenweijie> i got this.
20:19:08 <jgriffith> cool
20:19:10 <kchenweijie> maybe.
20:19:11 <harlowja> :)
20:19:16 <jgriffith> thanks for explanation
20:19:22 <harlowja> np
20:19:49 <harlowja> ok, well all great stuff, i think everyone is pretty heads down, any other folk from att or nttdata around, i think they might be working on some stuff also
20:20:20 <harlowja> #action harlowja  follow up with them offline
20:21:05 <harlowja> so before open discussion, maybe have our honored guest jgriffith and us can discuss about cinder stuff
20:21:12 <jgriffith> hehe
20:21:14 <harlowja> #topic cinder-steps
20:21:26 <harlowja> hear ye hear ye
20:21:27 <jgriffith> So I think things are looking good
20:21:31 <jgriffith> this is going to help a ton
20:21:50 <jgriffith> my only concern up to this point is some of the complications it introduces WRT debug
20:21:59 <jgriffith> and getting everybody up to speed on the new task-flows
20:22:03 <harlowja> agreed
20:22:21 <jgriffith> I think it's worth the investment however...
20:22:34 <jgriffith> I'm most anxious to see what take up is lookign like in other projects :)
20:22:37 <harlowja> so the debug one, i can see what u mean there, initially i think the way it will work (without the persistance part) will be pretty similar to the current way of working
20:22:42 <hemna> it's pretty complicated to use IMO
20:23:03 <jlucci> What makes it complicated for you hemna?
20:23:21 <harlowja> also jgriffith i think we can address the 'up to speed' part somehow, documentation maybe, not exactly sure
20:23:59 <harlowja> and hemna how can we make it less complicated, at your service :(
20:24:00 <jgriffith> harlowja: Yeah, that's a normal part of something new.. not a terribly big deal
20:24:07 <hemna> heh
20:24:07 <harlowja> *oops thats :) not :(
20:24:20 <hemna> well I'm still trying to wrap my head around it
20:24:21 <kebray> harlowja, jgriffith, what do you all see as the path to getting another project to take dependency on TF?  I have "some" pull with reddwarf.
20:24:33 <jlucci> haha Yeah. I second the let us know how to make it less complicated. :D
20:24:37 <hemna> maybe I'm just dumb :)
20:24:45 <harlowja> nah
20:24:47 <jgriffith> kebray: so I don't have any feelings on any project over another
20:24:49 <harlowja> we all dumb
20:24:51 <kebray> If myself or adrian can do anything to help, totally willing to do so on getting other projects on board.
20:25:03 <jgriffith> kebray: I just want to make sure it moves in to a common usage patters
20:25:06 <jgriffith> pattern
20:25:11 <hemna> I'll go through the review again and see if I can come up with some constructive ideas
20:25:11 <harlowja> agreed
20:25:15 <harlowja> thx hemna
20:25:30 <jgriffith> and I think when it gets broader scope there will be better feed-back and it'll grow
20:25:31 <hemna> I haven't looked at it much in a few weeks due to my work I've been doing
20:25:34 <harlowja> i think it likely has room for improvements hemna aas all new stuff does
20:25:40 <hemna> sure
20:25:43 <harlowja> jgriffith agreed
20:25:46 <jgriffith> harlowja: I think to hemna 's point...
20:25:51 <hemna> I really think it's a great idea
20:25:59 <kebray> jgriffith:  ok, makes sense. We're working on a plan to pitch to the Heat team for integration there… not sure how successful we will be yet.. timing wise, I'm thinking we must do so by next summit.  I'd like to see harlowja give lot's of presentations and talks on TF at next summit :-)
20:26:04 <jgriffith> harlowja: it's hard to pick out some of the things via the source at first (at least it was for me)
20:26:12 <harlowja> so kebray maybe reddwarf, maybe ironic, john garbutt from nova could there also, it depends on resourcing i think and who can devote time/energy to using it
20:26:27 <jgriffith> I need to have a look at your seq diagrams again... that may help
20:26:28 <harlowja> kebray lots of presentations, wow
20:26:29 <harlowja> haha
20:26:35 <hemna> It just was a bit hard to follow the logic in the review and understand the base concepts that require you to get first before the source makes sense.
20:26:44 <harlowja> hemna understandable
20:26:47 <jgriffith> kebray: haha!
20:26:53 <jgriffith> kebray: keep signing up harlowja :)
20:26:57 <harlowja> lol
20:27:00 <kebray> hehe
20:27:01 <hemna> :)
20:27:12 <harlowja> jgriffith hemna so i think both of those are the initial 'head-wrapping' problems
20:27:15 <harlowja> *i think*
20:27:23 <jgriffith> harlowja: I think you're correct
20:27:25 <harlowja> likely we can provide some small examples that help that
20:27:28 <hemna> yah
20:27:32 <kebray> jlucci volunteered to go to Hong Kong.  I'll have better luck signing her up for stuff no doubt.
20:27:33 <harlowja> that would make sense
20:27:34 <hemna> it's a stumbling block for me
20:27:38 <harlowja> agreed
20:27:44 <jgriffith> harlowja: kebray any feedback/interest from Nova, Glance ?
20:28:02 <harlowja> so glance, i haven't been in touch with them that much
20:28:05 <jgriffith> or are they awaiting conductor?
20:28:07 <jlucci> I got this kebray. You know you want to send me. ;p
20:28:17 <kebray> John Garbutt(sp?)  has successfully refactored some stuff in Nova in preparation for being able to pull in TF library in the future.
20:28:24 <harlowja> jgriffith i think nova is starting to get some of the taskflow concepts, but slowly getting it
20:28:26 <kchenweijie> man, wish i had these opportunities to travel :P
20:28:45 <jgriffith> harlowja: got ya
20:28:47 <kebray> I'm watching the Nova blueprint on that to see how it progresses… so far it's looking good.
20:28:51 <hemna> we aren't sure about HongKong either....still trying to twist mgmt's arm to get one of us go
20:28:59 <harlowja> #link https://review.openstack.org/#/c/29077/
20:29:01 <harlowja> for example
20:29:09 <harlowja> "nova/conductor/tasks/live_migrate.py"
20:29:35 <harlowja> are very similar to our tasks, but from what i feel, its a little bit of a fight to even get that in, so taskflow ontop of that might have to wait a little
20:29:46 <harlowja> #link https://review.openstack.org/#/c/30868/
20:29:48 <jgriffith> I see... some prep work there :)
20:29:50 <harlowja> ya
20:29:52 <harlowja> i think so
20:30:04 <harlowja> *slowly but surely i think*
20:30:18 <jgriffith> Oh... there, that helps
20:30:20 <jgriffith> thanks!
20:30:26 <jgriffith> Ok...
20:30:28 <harlowja> i've been in contact with john g (the other john g) and keeping in contact
20:30:39 <jgriffith> So my only concern was that we'd have something unique in Cinder
20:30:43 <jgriffith> Sometimes that's ok
20:30:49 <harlowja> not as far as i can tell
20:30:50 <jgriffith> especially if it's better :)
20:31:02 <jgriffith> but just wanted to check on the project wide effort
20:31:11 <harlowja> agreed
20:31:13 <harlowja> its a valid concern
20:31:30 <harlowja> #action harlowja create little taskflow examples (not in unit tests, maybe in source tree or something)
20:31:40 <hemna> +1!
20:31:55 <kebray> I think we all want this to be widely adopted.  Need to work together to keep evangelizing this all over the place.
20:31:59 <harlowja> agreed
20:32:23 <kebray> And, I of course want to spin out a new service on top of it, Convection.
20:32:50 <kebray> It's a gab in the community's portfolio IMO.
20:32:57 <kebray> s/gap/gab
20:33:17 <hemna> better than a scab!
20:33:45 <harlowja> lol
20:33:46 <harlowja> haha
20:34:05 <harlowja> would u guys think that a little wiki that lists the status/developments in each core project might help?
20:34:09 <harlowja> *nothing major*
20:34:24 <hemna> yah I think so
20:34:35 <harlowja> any volunteers ;)
20:34:35 <kebray> can't hurt.
20:35:07 <kebray> crickets.
20:35:09 <harlowja> lol
20:35:16 <harlowja> ha
20:35:32 <harlowja> #action harlowja make little cross-project-status wiki
20:35:35 <harlowja> :-p
20:35:49 <jlucci> I'm thinking we could create a general TaskFlow wiki
20:35:59 <harlowja> could could
20:36:00 <jlucci> Sub-wikis being different types of patterns
20:36:07 <jlucci> (linear, distributed, etc)
20:36:08 <harlowja> +examples and descriptions?
20:36:15 <jlucci> With use cases for each pattern
20:36:18 <jlucci> yeah
20:36:19 <harlowja> that might help hemna and jgriffith
20:36:24 <harlowja> good idea jlucci
20:36:29 <jlucci> Lots of pretty pictures too. :D
20:36:32 <harlowja> haha
20:36:40 <harlowja> u need to get omnigraffle then :-p
20:36:46 <harlowja> and then become architect
20:36:47 <harlowja> haha
20:36:49 <hemna> yah I think that would help a lot
20:36:52 <jlucci> But then we could add that cross-project-status on there too
20:37:01 <harlowja> agreed
20:37:16 <harlowja> jlucci do u want to start said wiki?? ;)
20:37:17 <jlucci> Just throwing this out there again… https://wiki.openstack.org/wiki/DistributedTaskManagement …already has some examples on there
20:37:18 <kebray> I'm still not convinced why linear / distributed need to be differentiated.  There is an ordering to execution in both task-dependent-execution and data-driven-execution.
20:37:21 <jlucci> o. O
20:37:27 <harlowja> lol
20:37:48 <harlowja> sure kebray agreed
20:37:51 <kebray> You can execute linear task flows using a distributed engine.
20:37:56 <jlucci> Er, I guess linear wasn't the best word there. The traditional workflow patterns are different from a distributed approach
20:38:11 <jlucci> So those should get a separate explanation from distributed
20:38:21 <harlowja> and how it operates slightly differently and all that
20:38:51 <harlowja> so lets do that, i can make it along with the other wikis
20:38:56 <kebray> It's an "under-the-hood" operational difference, no?   I'm not sure why the top-of-the hood would be painted any different colors.
20:39:15 <harlowja> lol, as long as u buy insurance from jlucci
20:39:20 <kebray> haha
20:39:25 <hemna> pulled the latest patch set down....
20:39:40 <harlowja> thx, i'm reworking a little bit of it due to the reviews comments
20:39:43 <harlowja> *but not that much*
20:40:11 <jlucci> haha You're right kebray. Examples would be the same on-top-of-the-hood
20:40:17 <harlowja> #action harlowja make base-wiki
20:40:22 <jlucci> Figured it would be nice to explain the under-the-hoods workings too
20:40:54 <harlowja> agreed, lets work on that, i think it will be very useful for all
20:40:57 <harlowja> especially going forward
20:41:08 <harlowja> *more people will wonder about this and such
20:41:19 <kebray> If the top-of-the-hood is the same, I'd like to see us align efforts on the under-the-hood if possible. Seems silly to put effort into two different backends that get us the same thing on the front end.. if one approach is better than the other, let's debate and resolve.
20:41:44 <harlowja> well its only silly if u can get everyone to use the under-the-hood engine at the current time
20:41:50 <hemna> harlowja, fwiw, looks like you need to rebase
20:41:55 <harlowja> hemna hmmm, durn
20:41:57 <harlowja> kk
20:42:01 <harlowja> will do that
20:42:24 <harlowja> so lets open up for anything else
20:42:36 <harlowja> #topic open-discuss
20:42:49 <kebray> harlowja I guess I'm assuming what everyone else cares about is the front end and results… not the underlying mechanics unless they have to debug the library because its' broken.. but, that should weigh in to our decision on which under-the-hood implementation to go with.
20:43:14 <harlowja> well its also the support structure the under-the-hood involves right
20:43:17 <kebray> I'm also concerned that, if we go traditional workflow pattern route, that we're reinventing spiff.
20:43:32 <kebray> Unless we use spiff of course.
20:43:34 <harlowja> i think we already surpassed what spiff is calable of
20:43:39 <harlowja> *capable of
20:43:57 <hemna> hey, volume create worked this time :)
20:43:58 <kebray> yeah?   you have documentation on that?  might be a good think to add to the wiki ;-)
20:44:02 <hemna> as did delete
20:44:04 <hemna> :)
20:44:10 <kebray> go harlowja!   rock on!
20:44:14 <harlowja> source code feature analysis by me looking at it, lol
20:44:31 <harlowja> thats my docs, haha, but sure i can document
20:44:52 <jlucci> Yeah, I've been meaning to go research differences between spiff/current implementation
20:45:07 <jlucci> If that's something you wanted to do harlowja, that'd be awesome. If not, I volunteer myself to go check it out. :P
20:45:09 <harlowja> spiff has more of a way to read in workflows from backing files
20:45:23 <harlowja> otherwise i think we are beyond what spiff can do
20:45:37 <kebray> I'm just imagining the push back we'll get from the community.. would like to proactively document our choices and rational as we go… not because I want to mitigate questions without knowing what questions we will get, but because I care about being able to articulate why we made the choices we've made.
20:45:40 <harlowja> #action harlowja jlucci spiff document comparison
20:45:50 <harlowja> kebray 100% agree
20:47:15 <harlowja> so i think this proactive documenation goes back into the wiki thing
20:47:23 <kebray> ok, last open topic from me:   just want to ask how ya'll are currently governing the code commits.  I'm hoping ya'll get more code contributors soon.  I think it's primarily just harlowja, jloucci, and kevinXYX (too long an IRC name) contributing right now.. what does it take to get code into trunk, two +1's?   who is core?
20:48:11 <kebray> And, is the process working for ya'll?  If so, awesome, we can move on.  If not, let's discuss.
20:48:13 <jlucci> Ummm. It's mostly been a "Hey, harlowja, this look good?" type of thing
20:48:21 <harlowja> ya
20:48:53 <harlowja> the core reviewers is jlucci and me and devananda and hemna  i think
20:49:10 <harlowja> https://review.openstack.org/#/q/status:open+project:stackforge/taskflow,n,z
20:49:21 <harlowja> if anyone wants to review that'd be cool
20:49:40 <harlowja> i think the governing is just pretty normal, review stuff when u get time
20:49:52 <harlowja> i started turning on the unit tests in jenkins and such
20:49:53 <kebray> k.. I'm not hearing requests for change.. so, sounds like the process if working.  that's good :-)
20:50:07 <kebray> is working.
20:50:11 <harlowja> #link https://review.openstack.org/#/c/32289/
20:50:13 <harlowja> its a WIP i think
20:50:30 <harlowja> so we'll have testing being activated soon
20:50:38 <harlowja> *after i fix that*
20:50:43 <jlucci> shweet
20:50:53 <kebray> excellent.
20:51:31 <harlowja> it does bring a good other topic up
20:51:40 <harlowja> when should we aim for something on pypi?
20:52:20 <harlowja> likely we'll need that before cinder can offically use it
20:52:21 <harlowja> :)
20:52:29 <kebray> would that be a question for J G.?
20:52:33 <harlowja> and have it appear in openstack-requirements and such
20:52:45 * harlowja not sure what the current process for new dependencies is
20:52:59 <jgriffith> harlowja: kebray I wouldn't worry about that just yet personally
20:53:12 <jgriffith> my strategy so far on stuff like this has been copy/paste
20:53:16 <jgriffith> get it all sorted then move to pypi
20:53:21 <jgriffith> I'm doing the same thing with brick
20:53:34 <jgriffith> and who knows, maybe it'll end up in OSLO :)
20:53:44 <harlowja> kk, thx jgriffith
20:53:51 <kebray> yes, thx jgriffith
20:53:54 <jgriffith> just my 2 cents
20:54:22 <jgriffith> kebray: haha... by J G you meant releses perhaps, not John G :)
20:54:24 <jgriffith> LOL
20:54:25 <jgriffith> sorry
20:54:35 <kebray> no, I meant jgriffith.
20:54:40 <jgriffith> Oh... Good :)
20:54:43 <harlowja> to many JGs ;)
20:54:44 <harlowja> ha
20:54:45 <jgriffith> then I don't feel dumb :)
20:55:13 <harlowja> kebray did u want to talk about the data/task/information depenedency stuff, or is that still brewing in the brain
20:55:25 <kebray> I do, but not enough time left I don't think.
20:55:29 <harlowja> kk
20:55:36 <harlowja> durn
20:55:55 <jlucci> We could move that discussion over to the taskflow irc?
20:56:02 <harlowja> http://en.wikipedia.org/wiki/Yahoo!_Pipes might be something u are interested in :-P
20:56:06 <kebray> My encouragement is we target one backend first if the results on the front end are the same… I have good reason for wanting to back distributed-data-dependent execution.
20:56:24 <kebray> I think jlucci and I can get our thoughts out on wiki (as she has already done)… and, circulate for further disucssion.
20:56:30 <harlowja> cool
20:56:46 <harlowja> alright sounds good folks, lots to still do :)
20:57:00 <kebray> I gotta run to another quick meeting. will circle back around on this topic soon.
20:57:07 <harlowja> kk and we are almost out of time
20:57:25 <harlowja> for those that want to talk to us more, #openstack-state-management
20:57:44 <harlowja> thanks for all helping make this happen :)
20:58:01 <harlowja> smiley face stickers for all
20:58:02 <harlowja> lol
20:58:11 <jlucci> :p
20:58:19 <harlowja> #endmeeting