20:00:46 <harlowja> #startmeeting state-management
20:00:47 <openstack> Meeting started Thu Jun  6 20:00:46 2013 UTC.  The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:48 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:50 <openstack> The meeting name has been set to 'state_management'
20:01:01 <harlowja> hi all!
20:01:05 <jlucci> hola
20:01:09 <kchenweijie> hey there
20:01:11 <adrian_otto> hi
20:01:13 <harlowja> hi hi
20:01:23 <alexheneveld> hello
20:01:29 <harlowja> howdy
20:01:46 <harlowja> so agenda for those that are interested (and/or missed the email since it was sent out late)
20:01:48 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement#Previous_meetings
20:01:53 <changbl> hey guys
20:01:57 <harlowja> hi hi
20:02:11 <harlowja> oops wrong link
20:02:16 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement#Agenda_for_next_meeting
20:02:24 <harlowja> much betta
20:02:36 <harlowja> so lots of newness this week!
20:02:56 <harlowja> #topic where-everyone-is-at
20:03:19 <harlowja> just want to see how everyone is doing, any goodness that is being produced
20:03:22 <changbl> I am reading your taskflow code
20:03:38 <harlowja> i'm still working on the cinder review (anvil took 1.5 days away from me this week so far)
20:03:56 <harlowja> i setup the launchpad site and a initial blueprints and bug (1)
20:04:36 <harlowja> jgriffith started looking over the review and seems like he's mostly happy (so far)
20:04:45 <harlowja> #link https://review.openstack.org/#/c/29862/ (patch set 8)
20:05:07 <harlowja> hemna have you been looking at anything for H2?
20:05:28 <hemna> I've been slammed a bit
20:05:33 <harlowja> np
20:05:34 <hemna> we can chat offline if you like
20:05:37 <harlowja> k
20:05:42 <harlowja> all good
20:06:07 <harlowja> i also talked more with devananda about some usage in ironic, hoepfully taskflow can help out there
20:06:10 <hemna> do you have the launchpad url ?
20:06:29 <jlucci> https://launchpad.net/~taskflow-dev and https://launchpad.net/taskflow
20:07:03 <hemna> jlucci, thanks
20:07:08 <harlowja> so i think kchenweijie has been continuing on the db work, correct?
20:07:15 <kchenweijie> thats right
20:07:20 <harlowja> and jlucci on the celery/distributed goods
20:07:31 <jlucci> https://review.openstack.org/#/c/32036/
20:07:39 <harlowja> #link https://review.openstack.org/#/c/32036/
20:07:43 <jlucci> ^^ distributed goods
20:07:45 <uvirtbot> jlucci: Error: "^" is not a valid command.
20:07:48 <harlowja> :)
20:08:05 <harlowja> all cool stuff, awesome work!
20:08:23 <harlowja> #topic launchpad
20:08:52 <harlowja> as jlucci pointed out, there is a new taskflow site and dev group!
20:08:57 <harlowja> #link https://launchpad.net/taskflow
20:08:57 <hemna> nice
20:09:10 <changbl> +1 jlucci
20:09:15 <harlowja> it seems to be a good place to register what people are working on and bugs and such
20:09:23 <harlowja> sound good to all?
20:09:34 <changbl> yes
20:09:35 <hemna> +1
20:09:43 <harlowja> feel free to join the https://launchpad.net/~taskflow-dev group also
20:09:47 <harlowja> i think its an open group
20:10:28 <harlowja> if not then let me know and i'll try to fix that setting :-p
20:10:36 <hemna> pending approval :P
20:10:38 <harlowja> kk
20:11:07 <harlowja> hmmm, wonder why it can't be switched to open, well nm, either is fine i guess
20:11:20 <harlowja> #action harlowja  investigate dev openness
20:12:34 <harlowja> so thats good stuff, nothing to exciting there :)
20:12:44 <harlowja> any questions about it, just bug me :-p
20:12:54 <harlowja> #topic stackforge
20:13:03 <harlowja> and we finally made it to stackforge!
20:13:06 <harlowja> #link https://github.com/stackforge/taskflow
20:13:09 <harlowja> yaaaa!
20:13:21 <changbl> great!
20:13:23 <harlowja> the yahoo repo i removed so that we can not diverge there btw
20:13:34 <harlowja> so u might have to adjust your git remote if u were connected to the other one
20:13:37 <harlowja> just fyi
20:13:44 <harlowja> if that doesn't work out, i can try to help there
20:14:01 <harlowja> but that allows us to use review.openstack.org
20:14:29 <harlowja> right now there are 4 core people, more are welcome if they feel like they want to help out
20:14:42 <harlowja> jlucci devananda harlowja (and rohitk who isn't online)
20:15:15 <harlowja> i'd like to get hemna and other core members from involved teams in there, so they can have inputs (since the library will be used in there project)
20:15:23 <harlowja> hemna would u want to be in there?
20:15:51 <hemna> sure
20:16:16 <harlowja> cool
20:16:29 <harlowja> hemna u have been added
20:16:35 <hemna> rockin
20:16:58 <harlowja> i guess we can follow somewhat of the normal process for core people, or keep it pretty simple for now
20:17:07 <harlowja> the whole voting thing someday maybe, lol
20:17:27 <harlowja> sound good?
20:17:31 <jlucci> +1
20:17:40 <hemna> +1
20:17:47 <harlowja> coolness
20:18:20 <harlowja> alright, any questions on that before next topic?
20:18:56 <harlowja> going once, going twice, sold
20:19:08 <harlowja> #topic fsm-pattern
20:19:34 <harlowja> sooo right now there exists a set of patterns in the taskflow library that allow users of the library to structure there tasks in different sequences
20:20:00 <harlowja> 1 being a linear sequence (where a task can have inputs that are satisfied by a previous task, and output something for tasks 'down the line' to use)
20:20:45 <harlowja> the second being a graph sequence (using topological sort) which allows for tasks to have inputs satisfied by some other task in said graph, with the running order of the set of tasks determined by the dependency ordering
20:20:56 <harlowja> and right now thats currently the 2 patterns :)
20:21:04 <hemna> ok
20:21:13 <hemna> I think most of the cinder tasks will be linear no?
20:21:25 <harlowja> yes, i think those 2 patterns likely solve many of the problems
20:21:48 <harlowja> where u have action X -> Y -> Z
20:22:03 <harlowja> or X1->Y and X2->Y with Y->Z (graph)
20:22:19 <harlowja> *no loops allowed*
20:22:46 <hemna> ok cool
20:23:00 <hemna> sorry, but I have to run to the bus stop to get my kids...bbiab
20:23:46 <harlowja> np
20:24:17 <harlowja> but maybe its to early to tell, but there likely will be a desire for a more generic mechanism where u can structure your tasks via a FSM (inputs, outputs are part of tasks) and u could imagine a FSM table structure (X triggers Y after Z inputed provided) being defined that would allow for people creating FSMs with tasks
20:24:30 <harlowja> i think jlucci might be getting close to something like this
20:24:58 <jlucci> yupyup. Want me to give a quick overview of how that's working with the distributed pattern?
20:25:03 <harlowja> sure
20:25:36 <jlucci> Shweet. So, high-level overview, when you choose a distributed flow, a client in charge of task management is created for you
20:25:51 <jlucci> That client keeps track of all tasks and those task's "listeners/callbacks"
20:26:16 <changbl> any example code?
20:26:16 <jlucci> So whenever a task completes successfully, the client is notified, and in turn notified all tasks that were listening for the now-successful task that it has completed
20:26:17 <harlowja> thats similar to the FSM 'state transition table' in a way right?
20:26:35 <harlowja> *to a FSM 'state transition table'
20:26:57 <jlucci> Rightright
20:27:05 <jlucci> Umm, as far as example code goes -
20:27:20 <jlucci> https://review.openstack.org/#/c/32036/ This contains the entirety of the distributed code right now
20:27:24 <jlucci> more specifically
20:27:35 <jlucci> the client
20:27:36 <jlucci> https://review.openstack.org/#/c/32036/1/taskflow/distributed/DTClient.py
20:27:45 <harlowja> and the pattern.py file ?
20:27:55 <jlucci> teh flow
20:27:56 <jlucci> https://review.openstack.org/#/c/32036/1/taskflow/patterns/distributed_flow.py
20:27:59 <harlowja> *distributed_flow.py
20:28:03 <jlucci> yeah, that's the pattern
20:28:03 <harlowja> ya, thx
20:28:21 <harlowja> pretty neat
20:28:31 <jlucci> And these (not-quite-functioning tests lolz) show how the flow would be used at the user-level
20:28:31 <jlucci> https://review.openstack.org/#/c/32036/1/taskflow/tests/unit/test_distributed_flow.py
20:28:53 <adrian_otto> I did follow up with the Galera guys, and they indicated that a Galera cluster (which could be deployed using heat) would be suitable for a backing store for a lock service because of the synchronous nature of the replication.
20:29:44 <harlowja> adrian_otto good to know
20:29:53 <adrian_otto> it would not be fast or scalable, but it would work for those that don't want to use ZK for whatever reason
20:30:42 <harlowja> adrian_otto cool, lets save that for a little bit after the distributed stuffs
20:31:22 <harlowja> jlucci so would there be a wrapper needed for the task class/decorator to make it work with that pattern, or would that be only internally used in the distributed_pattern class?
20:31:25 <adrian_otto> I'm just following up because I took a previous action item to ask that.
20:31:35 <harlowja> np :)
20:31:41 <harlowja> its much appreciated
20:32:21 <harlowja> jlucci the test looks pretty neat, seems to match how other tasks are formed (and connected into the FSM via the link method right?)
20:32:29 <jlucci> So, as of now, I'm using our task backend data structure, but not the actual task object
20:32:32 <jlucci> I'm using celery tasks
20:32:47 <jlucci> The choice for this is that celery has these cool things called 'signals'
20:32:55 <harlowja> that exposes the celery usage, i wonder if that can be something internal only to the distributed_flow pattern?
20:32:56 <jlucci> That act just like zookeeper notifiers
20:33:15 <harlowja> cool
20:33:23 <jlucci> Well, this might be reaching, but since you're declaring a task with a decorator right now
20:33:29 <jlucci> And that's what celery already does
20:34:01 <jlucci> We could just have the users always use @task.whatever to define a task, and then on our side, we'll know whether to use that to mean a celery or taskflow task
20:35:19 <harlowja> but then we are exposing users to celery, instead of nicely compartalmenalizing  it in the distributed_flow
20:35:37 <harlowja> eck, bad spelling
20:36:11 <jlucci> oh, okay. Sorry. Misunderstood what you meant the first time
20:36:13 <jlucci> mmm
20:36:28 <jlucci> I think I could wrap it in the taskflow task object
20:36:40 <jlucci> Going to take some thinking through/hacking though.
20:36:43 <harlowja> or translate it when u do add() link() ?
20:36:51 <adrian_otto> are you suggesting that distributed and sigular tasks have the same interface, but the distributed has more attached?
20:37:03 <harlowja> ya
20:37:03 <adrian_otto> or are you suggesting divergent usage?
20:37:20 <harlowja> no, same task interface, how the distributed_flow connects them or wraps them is up to it
20:37:34 <adrian_otto> ok, good.
20:37:47 <jlucci> Could we add that as an action item for me? Or whatever the "go do this" thing is?
20:38:23 <harlowja> #action jlucci think about how taskflow decorator/task object can be translated in the distriubted_flow pattern to something that celery can use (thus not exposing celery to everyone)
20:38:26 <harlowja> i think that will do it
20:38:48 <jlucci> haha I think so. :P
20:38:51 <harlowja> #action bug josh if it makes no sense
20:39:17 <adrian_otto> in other words: "abstract celery".
20:39:32 <devananda> \o
20:39:36 <jlucci> haha That's too simple adrian_otto We like to keep it overly difficult here. :P
20:39:48 <harlowja> until someday in the future when the only thing uses is the distributed_flow, then just say use celery at that point, lol
20:40:39 <harlowja> don't want to overly restrict what or how people can use taskflow this early in the game (if ever)
20:41:05 <devananda> sorry i'm late... <reading scrollback>
20:41:08 <harlowja> np
20:41:12 <harlowja> devananda do all the work, thx
20:41:40 <harlowja> anything else jlucci for the distributed stuffs? we'll see if that might solve the FSM case by itself
20:42:01 <harlowja> if not we can see what else we can do
20:42:12 <jlucci> Nope. That's pretty much it. Just need to do some more hacking on it
20:42:22 <harlowja> coolness
20:42:31 <harlowja> #topic cinder/nova integration
20:42:41 <harlowja> so cinder is ongoing by me, slowly but surely i think
20:42:48 <harlowja> #link https://review.openstack.org/#/c/29862/
20:42:59 <harlowja> i think hemna will get involved there when he gets some time and such
20:43:24 <harlowja> johnthetubaguy i think is doing some of the stuff in nova (but not using taskflow just yet, but doing something that should easily map to taskflow)
20:43:49 <harlowja> any ntt folks here?? i think they might be doing some nova stuff also
20:44:06 <harlowja> any heat folks?
20:44:25 <kebray> I'm lurking.
20:44:34 <kebray> But, I'm not Heat core.
20:44:49 <harlowja> kk, just don't want to lose heat focus also, i haven't been super involved in there meetings
20:45:07 <harlowja> make sure convection materializes using taskflow if thats being worked on
20:45:11 <adrian_otto> randallburt:  welcome
20:45:20 <randallburt> hello everybody
20:45:25 <kebray> There's enough overlap from the Rackspace side that we are keeping an eye towards Heat.
20:45:28 * kebray thinks
20:45:32 <harlowja> kebray thats much appreciated
20:45:33 <randallburt> how can I be of service?
20:45:57 <kebray> jlucci sits amongst the rackspace Heat devs.
20:45:57 <adrian_otto> anything Heat can use from taskflow?
20:46:03 <adrian_otto> emerging recent needs?
20:46:19 <randallburt> nothing I can think of beyond its general stated purpose
20:46:51 <harlowja> adrian_otto kebray do we want to set any kind of mini-milestone where a piece of heat tries out taskflow?
20:46:53 <adrian_otto> I have not seen anything on the Heat ML on the taskflow subject
20:46:57 <randallburt> tbh, integration of it is low on the havana timeframe from what I can see, but progress on tf has looked pretty good so far, so you never know
20:47:10 <harlowja> a mini-milestone/piece of heat might be cool
20:47:36 <alexheneveld> heat is using a lot of python coroutines which are nifty but don't really lend themselves to this :(
20:47:37 <kebray> the Heat core team feels we have a full plate for Havana, so they are unlikely to want a milestone...
20:47:46 <randallburt> its not likely but not impossible at this stage. there's scaling/multi engine work that would make it difficult at this point
20:48:02 <randallburt> what alexheneveld said ;)
20:48:28 <harlowja> interesting, how do coroutines usage make tasks harder to use?
20:48:33 <kebray> but, that's not to say at some point we can't leverage jlucci and the Heat devs she sits next to and go after our own milestone.
20:48:38 <alexheneveld> at some point i see a major refactoring in heat to replace that nifty work with tasks but not much appetite for havana
20:49:07 <kebray> alexheneveld  agreed
20:49:10 <harlowja> alexheneveld ok, be interesting to get a braindump someday, is there anything written up that might be useful?
20:49:14 <adrian_otto> +1
20:49:27 <alexheneveld> the coroutines are about thread-sharing with a lot of back and forth
20:49:52 <harlowja> sounds like u guys made your own message passing?
20:50:08 <kebray> jlucci may not know it yet, but I'm going to ask her help document a plan of integration for TF in Heat.. if we need to do any selling, discussion, I'd like to get that started early.
20:50:09 <adrian_otto> I really think that coroutine stuff will get yanked out in the future
20:50:13 <randallburt> nope, that part is more to do with higher concurrency
20:50:28 <jlucci> haha Awesome. I'll get right on that kebray. :p
20:50:34 <harlowja> kebray i guess jlucci knows now ;)
20:50:37 <randallburt> bumbumbuuuuuum
20:50:44 <alexheneveld> harlowja it's like that, but it is really a core python feature (just rarely used) - but short of it is you don't have access to task objects (on the plus side you don't have to write task objects, you can use core python syntax)
20:50:54 <adrian_otto> kebray: I'm happy to help out as needed
20:51:18 <harlowja> alexheneveld interesting
20:51:19 <kebray> thanks adrian_otto!
20:51:37 <randallburt> IMO, its not a question of selling it as much as it is a question of timing.
20:51:41 <hemna> back
20:51:42 <randallburt> and now's a bad time ;)
20:51:43 <alexheneveld> adrian_otto: +1 will have to go at some point - this work will be much better and i think they will agree, once it is clear
20:51:45 <harlowja> np
20:52:02 <adrian_otto> yep
20:52:09 <adrian_otto> alexheneveld: I agree
20:52:59 <adrian_otto> randallburt: Yes, there is plenty on the plate now, but it can't hurt to see about filling it in after H release
20:53:09 <harlowja> i wonder about how corountines are pretty much just individual FSM's (in a way), FSM's that connect to one another, but i guess u guys have thought about this more than me, haha
20:53:12 <adrian_otto> planting the seends for it
20:53:29 <harlowja> ok cool, not much time left!
20:53:34 <adrian_otto> they don't use them for communication
20:53:40 <adrian_otto> they use it as a concurrency intrument
20:53:46 <randallburt> sure, no harm in that. Would be nice to start the bp for it now and get the ball rolling before I summit anyway
20:53:55 <adrian_otto> s/intrument/instrument/
20:54:16 <adrian_otto> yes, but I would wait before proposing that
20:54:21 <adrian_otto> as you said it's a matter of timing.
20:54:21 <jlucci> Yeah, and I'll talk to you offline randallburt, but I'd like to get an overview of the co-routine system if you find time soon
20:54:42 <harlowja> adrian_otto sure, if u have concurrent workflows that have ordering dependencies (start X after Y is produced by that other workflow Z) then it seems similar, but maybe not, haha
20:54:53 <harlowja> jlucci me too, so fill me in :)
20:54:55 <randallburt> jlucci:  sure, I haven't kept up with Zane's fork, but we can look at it soon-ish
20:55:18 <jlucci> Yeah. That would be a good action item as well. Research heat co-routines or whatevers
20:55:35 <randallburt> and adrian_otto is right, its not a task issue as much as changes being made to the existing system that would make doing both a bit…  messy
20:55:37 <harlowja> #action jlucci get overview of heat co-routine and report back?
20:55:45 <alexheneveld> harlowja: yes it is just fsm so would map to tasks but it is native python so not a small job (plus state is persisted).  also i can't take any credit, it is all zaneb's work!
20:56:08 <jlucci> Works for me. :D
20:56:11 <harlowja> alexheneveld cool, looking forward to more info
20:56:29 <alexheneveld> #action alexheneveld get sam corbett from our side to give you chapter and verse :)
20:56:51 <harlowja> +- singing a choir song
20:57:02 <harlowja> lol
20:57:19 <harlowja> ok cool, so lets finish up with any action items from last time (probably should have started with this)
20:57:29 <harlowja> #topic action-items-that-should-of-been-done-at-the-start
20:57:41 <harlowja> adrian_otto so galleria stuff quick :)
20:57:55 <adrian_otto> provided update earlier.
20:57:55 <harlowja> speed IRC, lol
20:57:59 <harlowja> perfect :)
20:58:06 <adrian_otto> Galera should work.
20:58:19 <adrian_otto> of the options for a SQL backend, it's my preferred one.
20:58:24 * adrian_otto done
20:58:26 <harlowja> agreed
20:58:28 <dansmith> (two minute warning :)
20:59:04 <harlowja> adrian_otto would u want to update https://wiki.openstack.org/wiki/StructuredWorkflowLocks#Databases with anything u learned
20:59:11 <harlowja> 1 minute, ahhhh
20:59:24 <adrian_otto> #adction adrian_otto to update wiki with Galera as a SQL option
20:59:37 <harlowja> #link https://wiki.openstack.org/wiki/StructuredWorkflowLocks#Databases
20:59:38 <jlucci> nopressurenopressure
20:59:42 <harlowja> ah
20:59:46 <harlowja> ok, anything else?
20:59:53 <hemna> looks good guys
21:00:01 <harlowja> thx :) mailing list for anything else!
21:00:05 <adrian_otto> #action drian_otto to update wiki with Galera as a SQL option
21:00:12 <jlucci> Thanks guys!
21:00:14 <harlowja> #action adrian_otto to update wiki with Galera as a SQL option
21:00:21 <alexheneveld> bye folks
21:00:23 <harlowja> #endmeeting