20:00:25 <harlowja> #startmeeting state-management
20:00:26 <openstack> Meeting started Thu Sep  5 20:00:25 2013 UTC and is due to finish in 60 minutes.  The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:29 <openstack> The meeting name has been set to 'state_management'
20:00:34 <harlowja> howday!
20:00:40 <melnikov> hi there
20:01:28 <harlowja> hi hi
20:02:57 <harlowja> so might be pretty short if its just us :-P
20:03:14 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement#Agenda_for_next_meeting
20:04:10 <harlowja> oks then, ha
20:04:25 <harlowja> i guess the big thing this week is https://review.openstack.org/#/c/45139/
20:04:29 <harlowja> #link https://review.openstack.org/#/c/45139/
20:04:50 <harlowja> and working on adjusting taskflow to the engine concept
20:05:09 <harlowja> aka unification of engines into taskflow (and vice-versa)
20:05:28 <changbl> Hi folks
20:05:31 <harlowja> howdy
20:05:37 <melnikov> hi
20:05:38 <changbl> so many commits from you
20:05:46 <changbl> good progress:)
20:05:54 <adrian_otto> hi
20:05:56 <changbl> i just do not have much time to review them... quite a hectic week
20:06:05 <harlowja> its ok :)
20:06:16 <harlowja> hi adalbas
20:06:17 <harlowja> *oops
20:06:23 <harlowja> hi adrian_otto
20:07:03 <harlowja> so 45139 is the other big one this week, it does yes break a little stuff, but i think its just a pain we have to get over, one thats worth just doing
20:07:35 <melnikov> +1 for 45139
20:08:17 <adalbas> :)
20:09:08 <harlowja> melnikov after that goes in i think u mentioned that anastasia is working on fixing up the graph 'flow'', i assume that doesn't mean she would be doing the engine 'threaded graph action equivalent'
20:09:18 <harlowja> so maybe i can do that up
20:09:45 <harlowja> and melnikov after 45139 we can see about resumption, i think u were getting started on that right?
20:10:27 <melnikov> yeah that's right
20:10:46 <adrian_otto> harlowja: there is a grammar error in the description of 45139. Can that be edited, or are you stuck with it?
20:11:00 <harlowja> never stuck
20:11:03 <harlowja> :)
20:11:20 <adrian_otto> "flows with there".. I think you meant "flows with their"
20:11:57 <harlowja> hmmm, sounds good to me, i always forget the diff between there/their :-P
20:12:01 <harlowja> hey i just write code, hahaha
20:12:20 <adrian_otto> yeah, "there" is a place you can point at
20:12:57 <harlowja> :) thx
20:12:57 <adrian_otto> Can you elaborate on "Threaded flow and graph flow are broken by this commit, so we
20:12:57 <adrian_otto> temporarily disable their tests."
20:13:22 <harlowja> sure
20:13:27 <harlowja> in the review or here?
20:13:31 <harlowja> or both?
20:13:35 <adrian_otto> does that mean that you are intentionally introducing a code regression?
20:13:53 <adrian_otto> I suppose start here, and if needed we can place it in the review
20:15:42 <hodos> so...I was implementing flow for volume backups...should I wait now until action_engine stuff is done ?
20:17:27 <harlowja> so engines are a major restructuring is what i would call it, one that inverts how flows work (for the better imho) so since gerrit doesn't have the concept of a feature branch and since taskflow is still pretty new i think its worth the effort to break taskflow for a short period until we have engines fully integrated
20:18:20 <harlowja> yes i know thats controversial, but i think its a needed change to get this goign in the right direction
20:19:18 <harlowja> hodos if u are using the cinder code, cinder already has a copy of taskflow that should work for your need, when cinder pulls in the newer taskflow with action_engine via running taskflow update.py then it will need to alter the flows there, but i don't think the alteration will be to painful
20:19:20 <adrian_otto> ok, sorry I missed where this came from. Can we back up a bit, and restate the original problem?
20:19:32 <harlowja> sure
20:20:00 <harlowja> so flows now have a lot of concept of how they run inside there module, so for example linear_flow knows how to run itself
20:20:17 <harlowja> this means its not really easily possible to seperate the running from the structure
20:20:38 <harlowja> which creates a problem for nesting of flows and running the different structures via different mechanisms
20:21:01 <harlowja> so engines/blocks were created to be this thing that 'runs' and the existing flows would just define structure
20:21:25 <adrian_otto> is the problem that the concept of a flow is not abstracted enough from the patterns? Meaning a flow has different usage or features depending on which pattern is used?
20:22:04 <harlowja> i think the problem was that flows were running themselves instead of asking something else to run it (an engine)
20:22:26 <adrian_otto> ok, why is that bad?
20:23:32 <adrian_otto> please understand I'm not arguing the point, I'm just seeking clarity
20:23:35 <harlowja> np
20:24:14 <harlowja> so there was alot of repeated work that flows were having, that is duplicated when flows know how to run themselves, state management for example
20:24:29 <harlowja> and that means each flow, would need to have the logic to do that
20:25:11 <harlowja> instead if flows just define structure, they can let the thing running the flows (engine) centralize all that duplication
20:25:25 <adrian_otto> ok, I can see how that's inefficient compared to the engine approach, and causes more maintenance, and probably complexity.
20:25:45 <harlowja> ya, so melnikov has been helping lead the changes to do that
20:26:04 <adrian_otto> so, in other words engines are a place to centralize the state management, and simplify flows.
20:26:11 <harlowja> correct
20:26:19 <adrian_otto> ok, I think I got it now, thanks.
20:26:45 <adrian_otto> maybe we could add that in to the commit description as well, to help others grok it a bit more easily.
20:27:25 <harlowja> np so that review is bringing all the picture together, except it was getting bigger due to the changes, and since there isn't a shared feature branch concept i thought it would be good to get that out there and in, so that melnikov and anastasia can continue on the path of fixing the other 2 broken flows
20:27:27 <harlowja> sure adrian_otto
20:27:58 <harlowja> so it might be broken for a few more days, but the break i think will make things simpler due the above :)
20:28:23 <harlowja> the basic though will work, so hodos u should be able to use the linear_flow and engines right now
20:28:29 <adrian_otto> it might also be worth documenting (somewhere) the intent around what's broken by this approach, and the plan for how it gets fixed.
20:28:33 <harlowja> sure
20:28:59 <harlowja> sounds good
20:29:08 <hodos> yes, i figured that's fine, thanks
20:29:33 <adrian_otto> you already mentioned the two patterns that need test bypass, but we are missing a statement of intent about how that will be solved.
20:29:49 <harlowja> yup yup, will add that adrian_otto
20:29:56 <adrian_otto> tx
20:31:11 <harlowja> ok, was going to see about jessica doing a distributed engine instead of 'distributed flow' (reworking that code she is already working on) but will talk with her off-meeting
20:31:54 <harlowja> hodos do u want to chat about your work, be interesting to hear
20:32:04 <harlowja> #topic new-use-cases
20:32:41 <harlowja> is it for cinder? or is this the rashka (sp?) project
20:33:09 <harlowja> oh, before i forget
20:33:20 <harlowja> #action harlowja update comment per adrian_otto comments
20:33:43 <hodos> i'm just in a state where I'm trying to work off the create_volume flow and make something similar with volume_backup
20:33:47 <harlowja> #action harlowja make bugs/blueprints to ensure intent/fixes are  noted
20:33:49 <hodos> i'm just in a state where I'm trying to work off the create_volume flow and make something similar with create_backup
20:33:53 <hodos> cinder, yup
20:34:19 <harlowja> cool
20:35:21 <harlowja> is that one pretty straightfoward, if u get stuck with taskflow understanding def let us help :)
20:35:50 <hodos> what I'm also trying to achieve is to be able to track activity performed by the flow
20:36:30 <hodos> I'm wondering if a backup is performed appliance -- appliance without coping data through cinder
20:36:59 <hodos> is flow a good point where we can update DB to track the progress
20:37:13 <harlowja> :)
20:38:03 <harlowja> so there is a mechanism built-in to be notified of the activity a flow (or with the newer code a engine) is going through
20:38:25 <harlowja> those notification 'recievers' can be arbitary callbacks
20:38:36 <harlowja> so u could have one callback be something that updates a db
20:38:58 <harlowja> similar tracking is needed itself by taskflow for advanced features like resuming
20:39:00 <hodos> ok
20:39:33 <harlowja> as far as the appliance -- appliance i'm not so sure there of the context
20:40:27 <harlowja> if its things that happen outside cinder (not via cinder) i'm not sure how u would track the activity there
20:40:47 <hodos> well it should be addressed in the driver
20:41:03 <hodos> it's vendor specific
20:41:41 <vito-ordaz> do you have any plans for monitoring flow progress?
20:41:54 <hodos> read up ^
20:42:39 <harlowja> vito-ordaz so i think taskflow should expose the mechansism to do notifications of flow progress, taskflow itself for resuming tasks will need to know the progress also
20:43:31 <harlowja> but the internal taskflow resuming mechanism doesn't have to be the same as the notifications being emitted, notifications i think are the best way to let people plug-in to what the engine/flow is doing
20:44:10 <harlowja> the neat thing about having that mechanism is that we can easily add-in something like
20:44:12 <harlowja> #link https://blueprints.launchpad.net/taskflow/+spec/builtin-metric
20:45:35 <harlowja> which will make nice graphs/pictures/metrics of what tasks/flows are doing
20:46:37 <hodos> that looks good
20:46:47 <hodos> https://blueprints.launchpad.net/cinder/+spec/backend-activity
20:47:03 <kebray> sorry I'm late.
20:47:11 <harlowja> intersting hodos
20:47:17 <hodos> we can address this with taskflows I think
20:47:56 <harlowja> hodos potentially, thats a big change :)
20:48:09 <harlowja> *big change for cinder i think
20:48:16 <harlowja> kebray howdy!
20:49:03 <harlowja> hodos very interesting
20:49:46 <harlowja> i sorta imagine something like the following, cinder asks volume driver to give back 'flow' for given 'action', cinder maybe adds its own flows to that returned one, submits them all the taskflow engine
20:49:58 <harlowja> *them all to taskflow engine
20:50:07 <vito-ordaz> what you think about this patch https://review.openstack.org/#/c/44362/
20:50:39 <harlowja> taskflow engine then can figure out how to suspend, resume (or other advanced thingy)
20:51:02 * kebray just caught up on the discussion on engines.
20:51:07 <vito-ordaz> all flow stored in db, and cinder provide API to get current state of each flow and also progress based on progress of each flow task
20:51:23 <harlowja> vito-ordaz u might want to talk with melnikov about engines :)
20:51:34 <harlowja> its looking very similar :)
20:51:44 * kebray still thinks that if you implement engines, the what's the point of flows?  Let's just have tasks nest tasks, no need to have another abstraction with flows so that flows embed in flows.
20:52:56 <harlowja> flows define how your thing is desired to run
20:53:02 <harlowja> not all of them will be flows within flows
20:53:16 <harlowja> some parts of your flow u want to run in parallel, others not
20:53:29 <harlowja> others u want there to be a strong dependency between tasks in the flow (graph)
20:53:48 <hodos> harlowja: yes that's how I thought about backend-activity as a nested flows
20:54:35 <hodos> can't you always have a strict dependency via requires/updates
20:54:36 <hodos> ?
20:56:04 <harlowja> hodos yes, that is possible
20:56:37 <harlowja> but say we didn't have a linear flow, then we would require people to create artificial dependencies in requires/provides to make it run linearly
20:56:41 <harlowja> which seems annoying/painful
20:57:04 <harlowja> in my view there would be the following 3 main flows
20:57:28 <harlowja> linear, unordered (can be ran in any order, even in parallel), graph (can be ran in parallel as long as dependents run before)
20:57:53 <harlowja> so a engine would know how to interpret those 3 root 'patterns'
20:58:58 <harlowja> but u might not want your engine to run via threads, so u would pick the single threaded one that would know how to run those 3 root patterns, or u may run the single threaded engine in tests, then switch to a celery/distributed engine in production
20:59:41 <harlowja> thats my thoughts on why engines are good, they disconnect the root patterns from how they are ran, instead of previously having the root patterns be very aware of how to run themselves
21:00:37 <harlowja> anyways, lets discuss more in #openstack-state-management if needed :)
21:00:40 <harlowja> times up! :)
21:00:55 <harlowja> #endmeeting