20:00:11 <harlowja> #startmeeting state-management
20:00:12 <openstack> Meeting started Thu Sep 12 20:00:11 2013 UTC and is due to finish in 60 minutes.  The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:15 <openstack> The meeting name has been set to 'state_management'
20:00:17 <harlowja> ya, got it right, ha
20:01:23 <harlowja> howdy all
20:01:29 <iv_m> hi there
20:01:34 <harlowja> hi there
20:01:40 <changbl> hello
20:01:45 <harlowja> hi hi
20:01:53 <harlowja> so lets see
20:01:56 <harlowja> #link http://eavesdrop.openstack.org/meetings/state_management/2013/state_management.2013-09-05-20.00.html
20:02:01 <harlowja> #topic last-week-actions
20:02:26 <harlowja> so i did the actions items there, for making sure the review that went in for mostly unification of engines is documented and bugs assigned, ya for that :)
20:02:57 <harlowja> anastisia is working on https://bugs.launchpad.net/taskflow/+bug/1221448
20:02:59 <uvirtbot> Launchpad bug 1221448 in taskflow "Re-integrate graph flow" [Critical,In progress]
20:03:16 <harlowja> and https://bugs.launchpad.net/taskflow/+bug/1221505 will follow that
20:03:18 <uvirtbot> Launchpad bug 1221505 in taskflow "Re-integrate threaded flow" [Critical,New]
20:04:20 <harlowja> so i think that was all the stuff there :)
20:04:34 <harlowja> oh ya, nearly forgot
20:04:35 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement#Agenda_for_next_meeting
20:04:49 <harlowja> #topic status
20:05:04 <harlowja> any one want to report anything interesting in status, i think its mostly heads-down work at the moment :)
20:05:14 <harlowja> fixing up those bugs, reviewing alot (me)
20:05:33 <harlowja> and communication with others on new people wanting to use taskflow :)
20:05:51 <harlowja> iv_m changbl  anything u want to report?
20:05:59 <harlowja> *if not thats fine also
20:06:21 <changbl> we have people here want to use taskflow:)
20:06:30 <harlowja> sweet
20:06:33 <harlowja> +2
20:06:33 <harlowja> ha
20:07:02 <changbl> they are very interested, and by end of this year, we have to deliver a demo, which will use transational orchestration
20:07:13 <harlowja> nice, hopefully taskflow can help alot there
20:07:15 <changbl> I gave a presentation, and we will use taskflow
20:07:19 <harlowja> awesome
20:07:27 <harlowja> is it for a public project (or something else?)
20:07:43 <changbl> at&t internal
20:07:50 <harlowja> cool
20:07:51 <changbl> at least for this moment
20:07:55 <harlowja> np
20:08:13 <harlowja> any usage that helps people do there stuff better is good usage :)
20:08:21 <changbl> btw, i do want to say something about engine/storage/flow
20:08:33 <harlowja> sure
20:08:36 <harlowja> go for it :)
20:08:44 <changbl> first, thanks you guys for the code, great work
20:09:05 <changbl> my concern is that I find it harder to use the library :)
20:09:05 <harlowja> thx iv_m  :)
20:09:20 <harlowja> define harder?
20:09:27 <kebray> hello
20:09:32 <harlowja> hi kebray
20:09:40 <changbl> first, you need to define tasks, then define flows, then define engines, then to inject some storage
20:09:52 <harlowja> ok
20:10:16 <changbl> i am not sure this give people impression that they need to learn a lot before using this library
20:10:58 <iv_m> tasks was there anyway; everything else i think should be done by job
20:11:05 <kebray> Can we make it so a user just defines tasks and their relationships, and that's it, done!    There should be a default engine.
20:11:21 <harlowja> well something has to run them :-p
20:11:46 <harlowja> but ya, i think i see where changbl is coming from, and i think we can make it a little more streamlined
20:12:07 <iv_m> agreed
20:12:40 <harlowja> like i think define tasks, define flows will be a prerequsite
20:12:50 <changbl> thanks harlowja. I think defining tasks is necessary, and probably so is flow. This actually reflect the name of this library, TaskFlow. Then storage and engines are by default, no need be explicit
20:13:36 <harlowja> agreed, would it help if we documented the different components better also?
20:13:43 <kebray> agreed.   Keep-it-simple.
20:13:48 <harlowja> explaining the 4 major pieces that currently exist (flow, task, engine, storage)
20:13:53 <changbl> sure, harlowja
20:14:13 <kebray> Storage and Engines should be configured by the service operator (for defaults and whether or not they allow you to switch from those defaults in code or not).
20:14:26 <harlowja> sure
20:14:42 <harlowja> except service operator in taskflow is the library operator :)
20:14:46 <kebray> But, Tasks and Relations (what ya'll are calling the flow) should be enough to run on any TaskFlow compatible service.
20:14:57 <kebray> sure, library operator... but, it's in a config file :-)
20:15:09 <harlowja> well thats up to the library operator to make that config file :)
20:15:16 <kebray> hence the service operator is the one with permissions to edit the config file.. just so happens to be the developer :-)
20:15:48 <harlowja> k, as long as taskflow doesn't force this config file on people
20:16:02 <kebray> what's wrong with a config file?
20:16:20 <harlowja> a library to me doesn't say u must use config files to use this library
20:16:28 <kebray> Sure... the library can have defaults.
20:16:28 <harlowja> a library lets the surronding application decide that
20:16:57 <harlowja> defaults sure, but defaults != config file
20:17:30 <kebray> agreed
20:17:33 <harlowja> for example, the library takes a configuration object, the surronding application using the library can translate the config file they use into that configuration object
20:17:52 <kebray> ok.  I concede.
20:17:59 <harlowja> :)
20:18:00 <harlowja> haha
20:18:03 <kebray> pew pew
20:18:14 <harlowja> ya, i'm trying to get rid of oslo.cfg dependency in taskflow, its releated to this :-P
20:18:29 <harlowja> oslo.cfg forces a config file on people
20:18:55 <harlowja> #action harlowja document (flow, task, engine, storage) and those concepts
20:19:27 <harlowja> iv_m do u want to document some ideas on how we might make it more simple to just use flow, task?
20:19:36 <harlowja> or make it really easy to just say make an engine
20:19:58 <kebray> ok... in the case of just using the library, it's ok then if the library defaults to one engine, but all of our docs and examples should show how to set the desired engine.   e.g.   set engine(default = blah); define task 1; define task 2;, define flow; flow.execute()
20:20:06 <iv_m> we've got jobs in taskflow
20:20:31 <iv_m> i think that's the job that has to put everything together
20:20:37 * kebray wants a reminder why we have jobs?   we don't need so many primitives!
20:20:50 <harlowja> kebray those are pseudo-code so i wouldn't exactly quote them ;)
20:21:15 <kebray> my pseudo-code won't execute?   doh... my degrees is useless.
20:21:17 <harlowja> so the job is the top level 'job' for a 'action'
20:21:20 <harlowja> haha
20:21:32 <harlowja> *sorry, top level connector u could call it
20:21:33 <kebray> doh..  not actions again.
20:21:33 <changbl> I have same convern with kebray , too many primitives... :)
20:21:36 <harlowja> nope
20:21:47 <harlowja> to me there is a valid reason for these primitives :)
20:22:00 <harlowja> jobs has been there from the start folks :-P
20:22:07 <harlowja> ;)
20:22:08 <harlowja> #link https://wiki.openstack.org/wiki/StructuredWorkflowPrimitives#Jobs
20:22:12 <kebray> +1 for fewer primitives.
20:22:13 <changbl> at least we need very clear definition of them
20:22:24 <harlowja> ya, thats why i made that wiki a long time ago, ha
20:22:48 <kebray> why do we need jobs?   that link doesn't say what a job is good for.
20:22:57 <changbl> i read it, but memory fades quickly ...
20:23:01 <harlowja> lol
20:23:14 <kebray> harlowja I encouraged getting rid of jobs back then ;-)
20:23:18 <harlowja> hmmmm, idk
20:23:22 <harlowja> nice try kebray
20:23:22 <harlowja> ha
20:23:40 <kebray> so, why are jobs needed?
20:23:53 <changbl> but I clearly remember task and flow, because of the name of this library :)
20:23:58 <harlowja> lol
20:24:05 <harlowja> :)
20:24:07 <harlowja> kebray so
20:24:09 <harlowja> sooo
20:24:36 <harlowja> so lets say we have an api, 'nova-boot'
20:24:50 <harlowja> it is itself composed of many different flows
20:24:58 <harlowja> across various components
20:25:06 <harlowja> how do u connect them all together?
20:25:13 <kebray> when you say flows here, you don't mean TaskFlow flows, correct?
20:25:23 <kebray> or do you?
20:25:38 <harlowja> i am thinking in taskflow terms here
20:25:41 <kebray> ok.
20:25:59 <harlowja> *but we can change the term if u want*
20:26:00 <harlowja> :)
20:26:08 <kebray> so, implementation of nova-boot... there may be more than one TaskFlow flow that make up the full nova-boot implementation.  got it.
20:26:32 <harlowja> right, and at any time there is likely one 'thing' doing that set of 'flows'
20:26:40 <kebray> ?
20:26:48 <changbl> harlowja, why is there more than one flow?
20:26:55 <changbl> i mean for nova-boot
20:27:05 <kebray> yeah, one could argue that you could composite those individual flows into one flow.
20:27:40 <harlowja> sure, ok, so ya u could use composites i think, but lets say not everyone can change there whole architecture to use composites
20:27:44 <kebray> and, the one flow IS the job.
20:27:46 <harlowja> which i don't think nova and others can
20:27:50 <kebray> so, job = flow.
20:28:10 <kebray> change architecture?   Not sure what you mean.. TaskFlow would be new.
20:28:45 <harlowja> ok, change architecture meaning nova-boot is really split across 3-4 nova components right now
20:29:02 <harlowja> so it would be tough for them to initially unify it under a composite (i think)
20:29:12 <harlowja> *conductor here can help
20:29:36 <harlowja> so thats part uno of jobs), being that top level connecting thing
20:29:37 <kebray> hmm.. ok.
20:29:45 <harlowja> wait till the end for questions, ha
20:29:47 <harlowja> ha
20:29:48 <harlowja> lol
20:29:58 <harlowja> haha
20:30:19 <kebray> so, you now are talking about the multiple flows running on multiple distributed systems... sounds like... eh, you know what I'm gonna say ;-)   ok, let's wait and finish discussing at the end.
20:30:21 <harlowja> so the other part of the job that i think is relevant, and iv_m is referring to is that a job also connects the flows to there storage (state)
20:30:58 <harlowja> job == storage + other metadata + flows to be done
20:31:05 <harlowja> ha
20:31:32 * kebray thinks the simple case needs to work without a job, including storage state :-/
20:31:48 <changbl> +1 kebray
20:31:52 <changbl> :)
20:31:53 <harlowja> and this job thing, i think if done correctly (maybe) is this root level thing (or reference to thing) that we can put in zookeeper, so that the job can be transfeered to another entity if the entity running that job craps the bucket
20:32:16 <harlowja> so thats my definition of a job
20:32:20 <changbl> folks, I envision when applying taskflow to OpenStack, defining tasks will be the main heavy-lifting work. Running them could be as simple as one-liner, like: results = Flow(task1, task2, task3, multi_threaded=True, context={}).run()
20:32:41 <kebray> distributed solves that problem at the engine, without needing to plug zookeeper in for jobs.
20:32:46 <kebray> fwiw :-)
20:32:54 <harlowja> sure sure
20:33:29 <changbl> btw, I will help on zookeeper, stay tuned
20:33:33 <harlowja> changbl so the engine rework was to get away from flows knowing about how they run, thats beneficial in that u can iterate on the thing that does the running independently of the flow itself
20:33:45 <kebray> alright.. I've caused enough trouble.  You can continue the meeting harlowja.  I'll keep up my opinion during open questions or on separate IRC channel.
20:34:01 <harlowja> kebray but its question time
20:34:02 <harlowja> :)
20:34:05 <kebray> perfect!
20:34:09 <harlowja> lol
20:34:37 <harlowja> changbl so yes, it does make Flow(task1, task2, task3, multi_threaded=True,) harder to do as 1 object, but i think 2 objects not so hard :)
20:34:58 <changbl> harlowja, hmmm, about the "benefit of iterating on the thing", what is that?
20:35:07 <harlowja> the engine basically
20:35:11 <kebray> So, can I define two tasks, their relation, define the storage state and engine I want, then execute?  Or, do I have to create a job... or actions(?) or blocks, or whatever other primitives?
20:35:22 * kebray really needs to download and play with latest code.
20:35:28 <changbl> +1 kebray !
20:35:29 <harlowja> kebray ya, download the code
20:35:29 <harlowja> ha
20:35:32 <changbl> :)
20:36:09 <harlowja> kebray u create 2 tasks, their releation, then hand it off to engine
20:36:24 <harlowja> u don't need to create a job at the moment
20:36:32 <kebray> #action kebray use TaskFlow and provide feedback next week
20:36:36 <harlowja> +2
20:36:38 <harlowja> lol
20:36:47 <iv_m> i think we can provide a callable that creates engine and runs flow on it, like taskflow.run(flow, cfg=None)
20:36:59 <harlowja> iv_m agreed, that would probably be pretty useful
20:37:05 <kebray> +1 iv_m
20:37:24 <iv_m> #action iv_m implement taskflow.run
20:37:28 <harlowja> sweet
20:37:30 <harlowja> :)
20:37:32 <changbl> +1 iv_m anything that makes the library simpler to use
20:37:37 <harlowja> +1 changbl iv_m
20:38:20 <harlowja> i think as a library u guys are spot on, we need to make it possible to use the basics simply, but not limit those who want to use more than the basics
20:38:33 <kebray> agreed.
20:39:03 <changbl> +1 harlowja definitely, advanced users deserve advanced features
20:39:03 <harlowja> anyway, lets switch topic a little, ha
20:39:07 <harlowja> :)
20:39:23 <harlowja> #topic resumption
20:39:44 <harlowja> so might not be much to talk about here, but just wanting to ack to folks that we are continuing workin on this
20:40:03 <harlowja> iv_m was helping clear up what states are valid and such for this
20:40:04 <harlowja> #link https://wiki.openstack.org/wiki/TaskFlow/States_of_Task_and_Flow
20:40:21 <harlowja> the vendor specific states are an intreesting idea
20:41:04 <kebray> I gotta run folks.. sorry for the storm of questions.  I'll try out the library as soon as I can and try to be more constructive with ya'lls time.
20:41:11 <harlowja> kebray its ok :)
20:41:18 <harlowja> its ok boss
20:41:20 <harlowja> lol
20:41:40 <harlowja> anyways, if people want to read that over, check it out
20:41:57 <harlowja> thx to iv_m for doing that
20:42:01 <harlowja> *starting that
20:42:11 <changbl> +1 iv_m harlowja . Just wonder, who will be vendors? :)
20:42:21 <harlowja> ya, the naming is a good question
20:42:21 <harlowja> ha
20:42:29 <harlowja> so imagine a driver in nova
20:42:53 <harlowja> the task would be like call driver() create
20:43:03 <harlowja> but the driver may have its own mini-set of states its going through
20:43:11 <harlowja> *now composite flows here can also solve this
20:43:26 <harlowja> but lets say that not everything can adapt to composites at the moment
20:43:44 <harlowja> so the driver itself may go through something like [libvirt_turned_on, libvirt_didXYZ, libvirt_done]
20:43:59 <harlowja> while the surronding task would go through [pending, started, running, stopped]
20:44:04 <changbl> fair enough, I got it:) thanks
20:44:11 <changbl> good idea!
20:44:31 <harlowja> ya, it will be useful for cinder/nova (others i think)
20:44:43 <harlowja> also the other one thats likely useful to talk about
20:44:44 <changbl> one question: do we handle station transition from say RUNNING => LIBVIRT_ON?
20:44:53 <iv_m> to me that looks like progress notification, only with named states instead of percents
20:44:54 <changbl> s/station/state/
20:44:55 <harlowja> negative
20:45:00 <harlowja> so i don't think so changbl
20:45:21 <harlowja> not at the moment, those will be more of like substates that only the driver is aware of
20:45:55 <changbl> harlowja, iv_m, ok, i see, just notification/reporting purpose
20:45:59 <harlowja> i think so, to start
20:46:19 <harlowja> if the driver itself can resume from say ' libvirt_didXYZ' then it can try, but taskflow i don't think will help much there
20:46:30 <harlowja> since its sort of outside taskflow 'control'
20:46:31 <changbl> ok, fair enough
20:46:43 <harlowja> cool
20:46:48 <harlowja> other one that is sorta releated
20:46:50 <harlowja> #link https://blueprints.launchpad.net/taskflow/+spec/task-progress
20:47:03 <harlowja> similar purpose, but as iv_m said this one is more the percentage number :)
20:47:11 <harlowja> and not the state name [libvirt_turned_on, libvirt_didXYZ, libvirt_done]
20:47:43 <harlowja> the above would be like [33% , 66%, 100%]
20:47:48 <harlowja> (at the end of each state)
20:47:58 <harlowja> useful for doing progress like stuff
20:48:10 <harlowja> so then u could say `nova boot --poll` might actually do something nice :)
20:48:33 <harlowja> thats the general idea there
20:48:43 <harlowja> *more details in that blueprint
20:48:57 <changbl> so the numbers are, say your have n tasks, then 1/n * 100%, 2/n * 100%,, .... n/n * 100%,? Or you have other way to compute the percentage?
20:49:49 <harlowja> we should have the percentage, i think part of this also is about letting the 'driver' report subtask progress also (in that example)
20:49:59 <harlowja> so taskflow can emit 1/n * 100
20:50:21 <changbl> ok
20:50:28 <harlowja> and the individual task (the thing that calls the driver can let the driver report 0-100%) also
20:50:52 <changbl> cool
20:51:15 <harlowja> so that seems like a nice to have
20:51:27 <harlowja> lets see
20:52:02 <harlowja> otherwise, https://blueprints.launchpad.net/taskflow/+spec/flow-verification is also another one i'm (and iv_m) working on
20:52:11 <harlowja> feel free to check those out
20:52:28 <harlowja> #topic new use-cases
20:52:41 <harlowja> so just want to do a couple of shout-outs
20:52:42 <harlowja> :)
20:52:52 <changbl> go ahead~
20:53:03 <harlowja> savannah people recently started trying to figure out how to use taskflow
20:53:05 <harlowja> so +1 for that
20:53:16 <changbl> +1 !
20:53:19 <harlowja> after meeting with nexenta, they are very interested (for more cinder usage)
20:53:40 <changbl> softare-defined storage?
20:53:43 <harlowja> ya
20:53:45 <harlowja> #link https://review.openstack.org/#/c/45376/
20:54:03 <harlowja> so they want to help make sure it can be used in cinder (and elsewhere), so thats great
20:54:34 <harlowja> john garbutt (nova core) is also hoping in Icehouse we can get soem more traction in nova
20:54:37 <harlowja> so thats great to
20:54:48 <changbl> nice!
20:54:54 <harlowja> and billingstack is of course still working on using taskflow
20:55:06 <harlowja> so thats all good stuff, more usage the better :)
20:55:15 <harlowja> ok, end shoutouts, ha
20:55:20 <harlowja> and at&t
20:55:22 <harlowja> :-p
20:55:26 <changbl> :)
20:55:44 <harlowja> alright, not much time left, but lets open up for anything left over
20:55:47 <harlowja> #topic open-discuss
20:55:56 <harlowja> oh, other news
20:56:10 <harlowja> iv_m has been helping alot (thx ivan) with taskflow, so he's in taskflow-core (yaaa)
20:56:13 <harlowja> +1 iv_m
20:56:15 <harlowja> :)
20:56:19 <changbl> yaaa
20:56:22 <changbl> +1 iv_m
20:56:22 <iv_m> thx)
20:56:53 <harlowja> and lets see, that about all i got, ha
20:56:57 <changbl> https://review.openstack.org/#/c/45376/ i see this one got -2?
20:57:10 <harlowja> feature freeze for cinder
20:57:15 <harlowja> so i think thats the main reason
20:57:27 <changbl> all right... but create_volume is taskflow-based right?
20:57:30 <harlowja> yup
20:57:43 <changbl> k
20:58:03 <changbl> more questions I will ping you guys in the project channel
20:58:06 <harlowja> cool
20:58:13 <harlowja> other idea, thinking about summit sessions
20:58:18 <harlowja> keep it in mind
20:58:24 <harlowja> likely can have quite a few of them if we want :-P
20:58:43 <changbl> not going :(
20:58:47 <harlowja> :(
20:59:07 <harlowja> durn
20:59:14 <changbl> i voted for your talk, you guys have fun there:)
20:59:15 <harlowja> anyways, time pretty much up
20:59:19 <harlowja> thx! :)
20:59:31 <harlowja> channel #openstack-state-management for any more stuff
20:59:36 <harlowja> thx all for coming
20:59:38 <harlowja> till next week
20:59:43 <harlowja> *or anytime in the above channel
20:59:48 <harlowja> #endmeeting