20:00:28 <harlowja> #startmeeting state-management 20:00:29 <openstack> Meeting started Thu Jun 13 20:00:28 2013 UTC. The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:00:32 <openstack> The meeting name has been set to 'state_management' 20:00:35 <harlowja> hi all! 20:00:48 <kebray> hello 20:00:50 <harlowja> hey hey 20:00:56 <kchenweijie> hi 20:01:06 <harlowja> yo yo 20:01:14 <jlucci> hurr 20:01:22 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement#Agenda_for_next_meeting 20:01:37 <harlowja> not so much from my end to discuss, everyone busy doing *work* i thinks 20:02:00 <harlowja> i invited jgriffith so maybe we can also talk about cinder more stuffs 20:02:16 <harlowja> H2 targets and H3 and such 20:02:51 <harlowja> i guess go over action items from last time first 20:03:00 <harlowja> #link http://eavesdrop.openstack.org/meetings/state_management/2013/state_management.2013-06-06-20.00.html 20:03:13 <harlowja> no adrian, so can skip that :-p 20:03:22 <harlowja> i didn't get around to the launchpad openness issue yet :( 20:03:43 <harlowja> *seems like the launchpad is restricted access only, unsure how to open from restricted 20:04:09 <jgriffith> :) 20:04:15 <harlowja> jlucci i think u did some stuff also 20:04:17 <jgriffith> harlowja: sorry.. late as usual 20:04:21 <harlowja> np 20:04:41 <harlowja> just going over anything we said was an action item from last week 20:04:42 <jlucci> Yeah, so starting on my first action item - abstracting out celery so user only sees taskflow task class 20:04:49 <harlowja> woot 20:05:00 <harlowja> and there was a mini-meeting with heat people 20:05:01 <jlucci> Using .provides and .requires from your task annotations are coming in supah handy 20:05:06 <hemna> harlowja, did you want me to pull the patch and test again yet? 20:05:21 <harlowja> hemna u can if u want, i'm still doing local testing (slowly but surely) 20:05:27 <hemna> ok 20:06:16 <harlowja> so all good stuff, i think the takeaway from the heat meeting, was that they have a little bit of the same stuff as taskflow and we should come up with a plan to figure out what goes where and how 20:06:31 <harlowja> *so more to dicsuss there i think 20:07:26 <kebray> harlowja: agreed. 20:07:34 <harlowja> coolness 20:07:39 <jlucci> I've been talking with Randall on how to go about some of that 20:07:45 <harlowja> so thx jlucci ! 20:08:21 <harlowja> it'd be nice to document that stuff, after it becomes less 'cloudy' 20:08:53 <jlucci> agreed 20:09:00 <harlowja> so do we want to do a little round of what is everyone working on, would that be useful? 20:09:24 <kebray> I have two things for when we get to "Open Topics".. just fyi. 20:09:28 <harlowja> k 20:09:50 <harlowja> #topic round-of-things-happening 20:10:37 <harlowja> so i'm bringing up grizzly on y!s openstack vms, testing cinder stuff there, slowly making progress, getting some good code review feedback from jgriffith and others 20:10:47 <harlowja> #link https://review.openstack.org/#/c/29862/ 20:11:00 <harlowja> also having to attend all there annoying internal meetings :-p 20:11:05 <harlowja> *all these 20:11:05 <jgriffith> hehe :) 20:11:55 <harlowja> thats about what i am doing, working on http://anvil.readthedocs.org/ since thats my setup tool of choice (i build that so have to use it) 20:12:04 <harlowja> *built that* 20:12:42 <harlowja> jlucci kchenweijie what have u guys been doing :-P 20:12:56 <harlowja> *or anyone else doing stuff out there in the ether? 20:12:58 <jlucci> so 20:13:04 <jlucci> https://wiki.openstack.org/wiki/DistributedTaskManagement 20:13:21 <harlowja> jlucci u look like the progressive insurance lady?? 20:13:24 <harlowja> haha 20:13:34 <jlucci> Wrote up some documentation on the distributed approach to accompany the code/explain why distributed should be used 20:13:34 <jlucci> lol 20:13:41 <jlucci> Yeah. That's totally my picture up there. 20:13:48 <harlowja> lol 20:14:01 <harlowja> i knew it 20:14:11 <jlucci> Other than that. Still working on test cases/fleshing out code. And then also abstracting out celery through the taskflow task class 20:14:24 <harlowja> very awesome 20:14:29 <jlucci> Yeah…people stop me in the street all the time. so annoying. :P 20:14:34 <harlowja> ha 20:14:54 <harlowja> *stop trying to buy insurance from me gosh darn it 20:14:55 <harlowja> ha 20:15:07 <kchenweijie> i guess its my turn now :P 20:15:09 <kebray> Well, I'll cover one of my Open Topics now: harlowja, to build a little more on the conversation we were having yesterday about distributed execution engines for data-dependent control flows. This paper describes some of the advantages in the intro: http://research.microsoft.com/pubs/173947/thesis-corrected.pdf 20:15:15 <kchenweijie> or not xD 20:15:31 <kebray> sorry kchenweijie.. I'm just piling on to jlucci's update. 20:15:37 <kchenweijie> no problem 20:15:58 <kebray> Anyway.. that's bedtime reading for later discussion. 20:16:06 <kebray> we can move on. 20:16:09 <harlowja> i wonder what the 'thesis-uncorrected' was :-P 20:16:49 <harlowja> cool kchenweijie what have u been up to 20:16:56 <harlowja> kebray def something to read over :) 20:17:10 <kchenweijie> ive been working towards making the general api we discussed yesterday 20:17:35 <kchenweijie> ive made a basic template for what each backend api should be able to do and im working towards making the current sqlalchemy api match that 20:17:43 <harlowja> awesome 20:17:53 <kchenweijie> right now, i only have the persistence portion of the full api, but i want to make sure this works before i continue 20:18:00 <harlowja> makes sense to me 20:18:16 <kchenweijie> thats it for me 20:18:22 <harlowja> jgriffith this will help provide the persistance layer for tasks that can be pretty nice for future awesomeness in cinder 20:18:25 <harlowja> *and elsewhere* 20:18:55 <harlowja> it will be a key part of everything in the futurei hope :) 20:19:01 <harlowja> no pressure kchenweijie ;) 20:19:08 <kchenweijie> i got this. 20:19:08 <jgriffith> cool 20:19:10 <kchenweijie> maybe. 20:19:11 <harlowja> :) 20:19:16 <jgriffith> thanks for explanation 20:19:22 <harlowja> np 20:19:49 <harlowja> ok, well all great stuff, i think everyone is pretty heads down, any other folk from att or nttdata around, i think they might be working on some stuff also 20:20:20 <harlowja> #action harlowja follow up with them offline 20:21:05 <harlowja> so before open discussion, maybe have our honored guest jgriffith and us can discuss about cinder stuff 20:21:12 <jgriffith> hehe 20:21:14 <harlowja> #topic cinder-steps 20:21:26 <harlowja> hear ye hear ye 20:21:27 <jgriffith> So I think things are looking good 20:21:31 <jgriffith> this is going to help a ton 20:21:50 <jgriffith> my only concern up to this point is some of the complications it introduces WRT debug 20:21:59 <jgriffith> and getting everybody up to speed on the new task-flows 20:22:03 <harlowja> agreed 20:22:21 <jgriffith> I think it's worth the investment however... 20:22:34 <jgriffith> I'm most anxious to see what take up is lookign like in other projects :) 20:22:37 <harlowja> so the debug one, i can see what u mean there, initially i think the way it will work (without the persistance part) will be pretty similar to the current way of working 20:22:42 <hemna> it's pretty complicated to use IMO 20:23:03 <jlucci> What makes it complicated for you hemna? 20:23:21 <harlowja> also jgriffith i think we can address the 'up to speed' part somehow, documentation maybe, not exactly sure 20:23:59 <harlowja> and hemna how can we make it less complicated, at your service :( 20:24:00 <jgriffith> harlowja: Yeah, that's a normal part of something new.. not a terribly big deal 20:24:07 <hemna> heh 20:24:07 <harlowja> *oops thats :) not :( 20:24:20 <hemna> well I'm still trying to wrap my head around it 20:24:21 <kebray> harlowja, jgriffith, what do you all see as the path to getting another project to take dependency on TF? I have "some" pull with reddwarf. 20:24:33 <jlucci> haha Yeah. I second the let us know how to make it less complicated. :D 20:24:37 <hemna> maybe I'm just dumb :) 20:24:45 <harlowja> nah 20:24:47 <jgriffith> kebray: so I don't have any feelings on any project over another 20:24:49 <harlowja> we all dumb 20:24:51 <kebray> If myself or adrian can do anything to help, totally willing to do so on getting other projects on board. 20:25:03 <jgriffith> kebray: I just want to make sure it moves in to a common usage patters 20:25:06 <jgriffith> pattern 20:25:11 <hemna> I'll go through the review again and see if I can come up with some constructive ideas 20:25:11 <harlowja> agreed 20:25:15 <harlowja> thx hemna 20:25:30 <jgriffith> and I think when it gets broader scope there will be better feed-back and it'll grow 20:25:31 <hemna> I haven't looked at it much in a few weeks due to my work I've been doing 20:25:34 <harlowja> i think it likely has room for improvements hemna aas all new stuff does 20:25:40 <hemna> sure 20:25:43 <harlowja> jgriffith agreed 20:25:46 <jgriffith> harlowja: I think to hemna 's point... 20:25:51 <hemna> I really think it's a great idea 20:25:59 <kebray> jgriffith: ok, makes sense. We're working on a plan to pitch to the Heat team for integration there… not sure how successful we will be yet.. timing wise, I'm thinking we must do so by next summit. I'd like to see harlowja give lot's of presentations and talks on TF at next summit :-) 20:26:04 <jgriffith> harlowja: it's hard to pick out some of the things via the source at first (at least it was for me) 20:26:12 <harlowja> so kebray maybe reddwarf, maybe ironic, john garbutt from nova could there also, it depends on resourcing i think and who can devote time/energy to using it 20:26:27 <jgriffith> I need to have a look at your seq diagrams again... that may help 20:26:28 <harlowja> kebray lots of presentations, wow 20:26:29 <harlowja> haha 20:26:35 <hemna> It just was a bit hard to follow the logic in the review and understand the base concepts that require you to get first before the source makes sense. 20:26:44 <harlowja> hemna understandable 20:26:47 <jgriffith> kebray: haha! 20:26:53 <jgriffith> kebray: keep signing up harlowja :) 20:26:57 <harlowja> lol 20:27:00 <kebray> hehe 20:27:01 <hemna> :) 20:27:12 <harlowja> jgriffith hemna so i think both of those are the initial 'head-wrapping' problems 20:27:15 <harlowja> *i think* 20:27:23 <jgriffith> harlowja: I think you're correct 20:27:25 <harlowja> likely we can provide some small examples that help that 20:27:28 <hemna> yah 20:27:32 <kebray> jlucci volunteered to go to Hong Kong. I'll have better luck signing her up for stuff no doubt. 20:27:33 <harlowja> that would make sense 20:27:34 <hemna> it's a stumbling block for me 20:27:38 <harlowja> agreed 20:27:44 <jgriffith> harlowja: kebray any feedback/interest from Nova, Glance ? 20:28:02 <harlowja> so glance, i haven't been in touch with them that much 20:28:05 <jgriffith> or are they awaiting conductor? 20:28:07 <jlucci> I got this kebray. You know you want to send me. ;p 20:28:17 <kebray> John Garbutt(sp?) has successfully refactored some stuff in Nova in preparation for being able to pull in TF library in the future. 20:28:24 <harlowja> jgriffith i think nova is starting to get some of the taskflow concepts, but slowly getting it 20:28:26 <kchenweijie> man, wish i had these opportunities to travel :P 20:28:45 <jgriffith> harlowja: got ya 20:28:47 <kebray> I'm watching the Nova blueprint on that to see how it progresses… so far it's looking good. 20:28:51 <hemna> we aren't sure about HongKong either....still trying to twist mgmt's arm to get one of us go 20:28:59 <harlowja> #link https://review.openstack.org/#/c/29077/ 20:29:01 <harlowja> for example 20:29:09 <harlowja> "nova/conductor/tasks/live_migrate.py" 20:29:35 <harlowja> are very similar to our tasks, but from what i feel, its a little bit of a fight to even get that in, so taskflow ontop of that might have to wait a little 20:29:46 <harlowja> #link https://review.openstack.org/#/c/30868/ 20:29:48 <jgriffith> I see... some prep work there :) 20:29:50 <harlowja> ya 20:29:52 <harlowja> i think so 20:30:04 <harlowja> *slowly but surely i think* 20:30:18 <jgriffith> Oh... there, that helps 20:30:20 <jgriffith> thanks! 20:30:26 <jgriffith> Ok... 20:30:28 <harlowja> i've been in contact with john g (the other john g) and keeping in contact 20:30:39 <jgriffith> So my only concern was that we'd have something unique in Cinder 20:30:43 <jgriffith> Sometimes that's ok 20:30:49 <harlowja> not as far as i can tell 20:30:50 <jgriffith> especially if it's better :) 20:31:02 <jgriffith> but just wanted to check on the project wide effort 20:31:11 <harlowja> agreed 20:31:13 <harlowja> its a valid concern 20:31:30 <harlowja> #action harlowja create little taskflow examples (not in unit tests, maybe in source tree or something) 20:31:40 <hemna> +1! 20:31:55 <kebray> I think we all want this to be widely adopted. Need to work together to keep evangelizing this all over the place. 20:31:59 <harlowja> agreed 20:32:23 <kebray> And, I of course want to spin out a new service on top of it, Convection. 20:32:50 <kebray> It's a gab in the community's portfolio IMO. 20:32:57 <kebray> s/gap/gab 20:33:17 <hemna> better than a scab! 20:33:45 <harlowja> lol 20:33:46 <harlowja> haha 20:34:05 <harlowja> would u guys think that a little wiki that lists the status/developments in each core project might help? 20:34:09 <harlowja> *nothing major* 20:34:24 <hemna> yah I think so 20:34:35 <harlowja> any volunteers ;) 20:34:35 <kebray> can't hurt. 20:35:07 <kebray> crickets. 20:35:09 <harlowja> lol 20:35:16 <harlowja> ha 20:35:32 <harlowja> #action harlowja make little cross-project-status wiki 20:35:35 <harlowja> :-p 20:35:49 <jlucci> I'm thinking we could create a general TaskFlow wiki 20:35:59 <harlowja> could could 20:36:00 <jlucci> Sub-wikis being different types of patterns 20:36:07 <jlucci> (linear, distributed, etc) 20:36:08 <harlowja> +examples and descriptions? 20:36:15 <jlucci> With use cases for each pattern 20:36:18 <jlucci> yeah 20:36:19 <harlowja> that might help hemna and jgriffith 20:36:24 <harlowja> good idea jlucci 20:36:29 <jlucci> Lots of pretty pictures too. :D 20:36:32 <harlowja> haha 20:36:40 <harlowja> u need to get omnigraffle then :-p 20:36:46 <harlowja> and then become architect 20:36:47 <harlowja> haha 20:36:49 <hemna> yah I think that would help a lot 20:36:52 <jlucci> But then we could add that cross-project-status on there too 20:37:01 <harlowja> agreed 20:37:16 <harlowja> jlucci do u want to start said wiki?? ;) 20:37:17 <jlucci> Just throwing this out there again… https://wiki.openstack.org/wiki/DistributedTaskManagement …already has some examples on there 20:37:18 <kebray> I'm still not convinced why linear / distributed need to be differentiated. There is an ordering to execution in both task-dependent-execution and data-driven-execution. 20:37:21 <jlucci> o. O 20:37:27 <harlowja> lol 20:37:48 <harlowja> sure kebray agreed 20:37:51 <kebray> You can execute linear task flows using a distributed engine. 20:37:56 <jlucci> Er, I guess linear wasn't the best word there. The traditional workflow patterns are different from a distributed approach 20:38:11 <jlucci> So those should get a separate explanation from distributed 20:38:21 <harlowja> and how it operates slightly differently and all that 20:38:51 <harlowja> so lets do that, i can make it along with the other wikis 20:38:56 <kebray> It's an "under-the-hood" operational difference, no? I'm not sure why the top-of-the hood would be painted any different colors. 20:39:15 <harlowja> lol, as long as u buy insurance from jlucci 20:39:20 <kebray> haha 20:39:25 <hemna> pulled the latest patch set down.... 20:39:40 <harlowja> thx, i'm reworking a little bit of it due to the reviews comments 20:39:43 <harlowja> *but not that much* 20:40:11 <jlucci> haha You're right kebray. Examples would be the same on-top-of-the-hood 20:40:17 <harlowja> #action harlowja make base-wiki 20:40:22 <jlucci> Figured it would be nice to explain the under-the-hoods workings too 20:40:54 <harlowja> agreed, lets work on that, i think it will be very useful for all 20:40:57 <harlowja> especially going forward 20:41:08 <harlowja> *more people will wonder about this and such 20:41:19 <kebray> If the top-of-the-hood is the same, I'd like to see us align efforts on the under-the-hood if possible. Seems silly to put effort into two different backends that get us the same thing on the front end.. if one approach is better than the other, let's debate and resolve. 20:41:44 <harlowja> well its only silly if u can get everyone to use the under-the-hood engine at the current time 20:41:50 <hemna> harlowja, fwiw, looks like you need to rebase 20:41:55 <harlowja> hemna hmmm, durn 20:41:57 <harlowja> kk 20:42:01 <harlowja> will do that 20:42:24 <harlowja> so lets open up for anything else 20:42:36 <harlowja> #topic open-discuss 20:42:49 <kebray> harlowja I guess I'm assuming what everyone else cares about is the front end and results… not the underlying mechanics unless they have to debug the library because its' broken.. but, that should weigh in to our decision on which under-the-hood implementation to go with. 20:43:14 <harlowja> well its also the support structure the under-the-hood involves right 20:43:17 <kebray> I'm also concerned that, if we go traditional workflow pattern route, that we're reinventing spiff. 20:43:32 <kebray> Unless we use spiff of course. 20:43:34 <harlowja> i think we already surpassed what spiff is calable of 20:43:39 <harlowja> *capable of 20:43:57 <hemna> hey, volume create worked this time :) 20:43:58 <kebray> yeah? you have documentation on that? might be a good think to add to the wiki ;-) 20:44:02 <hemna> as did delete 20:44:04 <hemna> :) 20:44:10 <kebray> go harlowja! rock on! 20:44:14 <harlowja> source code feature analysis by me looking at it, lol 20:44:31 <harlowja> thats my docs, haha, but sure i can document 20:44:52 <jlucci> Yeah, I've been meaning to go research differences between spiff/current implementation 20:45:07 <jlucci> If that's something you wanted to do harlowja, that'd be awesome. If not, I volunteer myself to go check it out. :P 20:45:09 <harlowja> spiff has more of a way to read in workflows from backing files 20:45:23 <harlowja> otherwise i think we are beyond what spiff can do 20:45:37 <kebray> I'm just imagining the push back we'll get from the community.. would like to proactively document our choices and rational as we go… not because I want to mitigate questions without knowing what questions we will get, but because I care about being able to articulate why we made the choices we've made. 20:45:40 <harlowja> #action harlowja jlucci spiff document comparison 20:45:50 <harlowja> kebray 100% agree 20:47:15 <harlowja> so i think this proactive documenation goes back into the wiki thing 20:47:23 <kebray> ok, last open topic from me: just want to ask how ya'll are currently governing the code commits. I'm hoping ya'll get more code contributors soon. I think it's primarily just harlowja, jloucci, and kevinXYX (too long an IRC name) contributing right now.. what does it take to get code into trunk, two +1's? who is core? 20:48:11 <kebray> And, is the process working for ya'll? If so, awesome, we can move on. If not, let's discuss. 20:48:13 <jlucci> Ummm. It's mostly been a "Hey, harlowja, this look good?" type of thing 20:48:21 <harlowja> ya 20:48:53 <harlowja> the core reviewers is jlucci and me and devananda and hemna i think 20:49:10 <harlowja> https://review.openstack.org/#/q/status:open+project:stackforge/taskflow,n,z 20:49:21 <harlowja> if anyone wants to review that'd be cool 20:49:40 <harlowja> i think the governing is just pretty normal, review stuff when u get time 20:49:52 <harlowja> i started turning on the unit tests in jenkins and such 20:49:53 <kebray> k.. I'm not hearing requests for change.. so, sounds like the process if working. that's good :-) 20:50:07 <kebray> is working. 20:50:11 <harlowja> #link https://review.openstack.org/#/c/32289/ 20:50:13 <harlowja> its a WIP i think 20:50:30 <harlowja> so we'll have testing being activated soon 20:50:38 <harlowja> *after i fix that* 20:50:43 <jlucci> shweet 20:50:53 <kebray> excellent. 20:51:31 <harlowja> it does bring a good other topic up 20:51:40 <harlowja> when should we aim for something on pypi? 20:52:20 <harlowja> likely we'll need that before cinder can offically use it 20:52:21 <harlowja> :) 20:52:29 <kebray> would that be a question for J G.? 20:52:33 <harlowja> and have it appear in openstack-requirements and such 20:52:45 * harlowja not sure what the current process for new dependencies is 20:52:59 <jgriffith> harlowja: kebray I wouldn't worry about that just yet personally 20:53:12 <jgriffith> my strategy so far on stuff like this has been copy/paste 20:53:16 <jgriffith> get it all sorted then move to pypi 20:53:21 <jgriffith> I'm doing the same thing with brick 20:53:34 <jgriffith> and who knows, maybe it'll end up in OSLO :) 20:53:44 <harlowja> kk, thx jgriffith 20:53:51 <kebray> yes, thx jgriffith 20:53:54 <jgriffith> just my 2 cents 20:54:22 <jgriffith> kebray: haha... by J G you meant releses perhaps, not John G :) 20:54:24 <jgriffith> LOL 20:54:25 <jgriffith> sorry 20:54:35 <kebray> no, I meant jgriffith. 20:54:40 <jgriffith> Oh... Good :) 20:54:43 <harlowja> to many JGs ;) 20:54:44 <harlowja> ha 20:54:45 <jgriffith> then I don't feel dumb :) 20:55:13 <harlowja> kebray did u want to talk about the data/task/information depenedency stuff, or is that still brewing in the brain 20:55:25 <kebray> I do, but not enough time left I don't think. 20:55:29 <harlowja> kk 20:55:36 <harlowja> durn 20:55:55 <jlucci> We could move that discussion over to the taskflow irc? 20:56:02 <harlowja> http://en.wikipedia.org/wiki/Yahoo!_Pipes might be something u are interested in :-P 20:56:06 <kebray> My encouragement is we target one backend first if the results on the front end are the same… I have good reason for wanting to back distributed-data-dependent execution. 20:56:24 <kebray> I think jlucci and I can get our thoughts out on wiki (as she has already done)… and, circulate for further disucssion. 20:56:30 <harlowja> cool 20:56:46 <harlowja> alright sounds good folks, lots to still do :) 20:57:00 <kebray> I gotta run to another quick meeting. will circle back around on this topic soon. 20:57:07 <harlowja> kk and we are almost out of time 20:57:25 <harlowja> for those that want to talk to us more, #openstack-state-management 20:57:44 <harlowja> thanks for all helping make this happen :) 20:58:01 <harlowja> smiley face stickers for all 20:58:02 <harlowja> lol 20:58:11 <jlucci> :p 20:58:19 <harlowja> #endmeeting