20:00:05 <harlowja> #startmeeting openstack-state-management
20:00:06 <openstack> Meeting started Thu Jan 16 20:00:05 2014 UTC and is due to finish in 60 minutes.  The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:07 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:10 <openstack> The meeting name has been set to 'openstack_state_management'
20:00:14 <harlowja> howdy folks!
20:01:14 <harlowja> anyone around, if not short meeting :)
20:01:26 <harlowja> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/024477.html
20:01:51 <harlowja> maybe everyone just busy working :)
20:01:53 <harlowja> which is fine
20:02:58 <harlowja> will wait a few
20:04:49 <harlowja> zzzz
20:04:50 <harlowja> lol
20:05:18 <harlowja> anyways, i'll update my own status!
20:05:22 <harlowja> #topic status
20:05:33 * harlowja doing reviews, working on scoping
20:05:46 <harlowja> flushed out some of the previous action items i had to do
20:06:15 <harlowja> #link http://eavesdrop.openstack.org/meetings/state_management/2014/state_management.2014-01-09-20.00.html
20:06:28 <harlowja> wrote up https://wiki.openstack.org/wiki/TaskFlow#Workflow_ownership_transfer
20:06:30 <harlowja> #link https://wiki.openstack.org/wiki/TaskFlow#Workflow_ownership_transfer
20:06:39 <harlowja> hopefully explains the general idea
20:07:18 <harlowja> wrote up some of the cinder persistence ideas
20:07:22 <harlowja> #link https://etherpad.openstack.org/p/cinder-taskflow-persistence
20:07:32 <harlowja> and started the pros/cons of oslo joining
20:07:34 <harlowja> https://etherpad.openstack.org/p/oslo-taskflow
20:07:53 <harlowja> working with doug on some of this, seeing if i can flush those out
20:08:48 <changbl> hi,
20:09:12 <harlowja> hi changbl
20:09:21 <harlowja> one sec, need to move rooms, in a meeting room with lots of people :-P
20:09:35 <changbl> ok...
20:10:35 <harlowja> kk
20:10:38 <harlowja> moved to couch, haha
20:11:02 <changbl> reading your status update
20:11:17 <harlowja> kk
20:11:19 <harlowja> :)
20:11:37 <harlowja> didn't get around to documenting *real* zookeeper usage yet
20:11:43 <harlowja> will connect with tooz people on some of this
20:11:52 <changbl> tooz?
20:12:03 <harlowja> #link https://github.com/stackforge/tooz
20:12:45 <harlowja> they are also doing zookeeper stuffs changbl
20:12:46 <changbl> "The Tooz project aims at centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications."
20:12:52 <harlowja> https://github.com/stackforge/tooz/tree/master/tooz/drivers
20:12:57 <harlowja> yup
20:13:04 <harlowja> i wanna use it for locking eventually in taskflow
20:13:09 <changbl> did not zookeeper/kazoo already provide coordination apis?
20:13:15 <harlowja> :)
20:13:31 <harlowja> i think they are more of the api ontop of those services/apis
20:13:42 <harlowja> not reinventing zookeeper
20:13:43 <changbl> man...
20:14:01 <changbl> i think zookeeper is good enough... :)
20:14:10 <changbl> anyway
20:14:11 <changbl> move on
20:14:14 <harlowja> u won't get much disagreement from me (i'm biased, ha)
20:14:29 <harlowja> kk, where was i
20:15:00 <harlowja> so ya, just doing that writeup, trying to figure out how scopes should work, and what the restrictions are
20:15:19 <harlowja> and working in reviews when i can :)
20:15:58 <harlowja> changbl any status u have, just working on that review, looks like its pretty close :)
20:16:09 <harlowja> might just be u and me here, not sure where others are, haha
20:16:18 <changbl> yes, saw you comments of wanting more :)
20:16:27 <changbl> will work on sometime this week
20:16:31 <changbl> got diverted to other things
20:16:38 <harlowja> kk, np
20:16:46 <harlowja> hopefully not to bad, the exception translating should be easy
20:16:53 <changbl> ok...
20:17:01 <harlowja> thx changbl :)
20:17:05 <changbl> there are other things in my queue for zk
20:17:14 <harlowja> hmmmm
20:17:15 <changbl> like locking, an example, and ACLs
20:17:25 <harlowja> for taskflow or just in general?
20:17:32 <changbl> for taskflow, sir
20:17:43 <harlowja> ah, neat, we should connect with jd__ and the tooz folks
20:18:00 <harlowja> see where things should land
20:18:15 <changbl> tooz, hmm...
20:18:26 <harlowja> and least discuss i think :)
20:18:32 <harlowja> *at least
20:18:33 <changbl> sure...
20:18:45 <changbl> kazoo, zookeeper, tooz, zake...
20:18:48 <harlowja> :-P
20:18:50 <changbl> look at these funny names...
20:18:53 <harlowja> lol
20:19:53 <changbl> you will give me the write up on how to tesk zookeper with jenkins right?
20:19:56 <harlowja> sure
20:19:59 <changbl> great
20:20:15 <harlowja> the basic idea i see, at least how tooz is doing it
20:20:26 <harlowja> https://github.com/stackforge/tooz/blob/master/tox.ini#L11
20:20:34 <harlowja> in that run_tests.sh they setup zookeeper
20:20:49 <harlowja> so we could do something similar
20:20:53 <changbl> oh, nice
20:20:58 <changbl> you have a run_test.sh
20:21:17 <harlowja> but will bug infra about how we can just have zookeeper running automatically (this is how there mysql/postgres works)
20:21:30 <changbl> ok
20:21:49 <harlowja> seems odd that we should have to start much, but maybe we have to, idk
20:22:06 <harlowja> #action harlowja bug infra about zookeeper usage (perhaps not having to start it ourselves...)
20:22:57 <harlowja> https://github.com/stackforge/tooz/blob/master/run_tests.sh#L22 and all that changbl , but gotta be an easier way i hope
20:23:37 <harlowja> will do that soon, i hope :)
20:24:19 <changbl> only one zookeeper instance?
20:24:50 <harlowja> seems like it, its not a full quorum and all that
20:25:19 <changbl> ok, i guess it is fine for testing
20:25:20 <harlowja> be nice to have infra have a zookeeper server quorum that can just be used
20:25:34 <changbl> though i use 3 for my testing
20:25:39 <harlowja> ya, thats more typical
20:26:45 <harlowja> #topic help-needed-reviews-integration-other
20:26:59 <harlowja> not sure if anyone around doing much integration, i havent done as much as i want recently
20:27:29 <harlowja> some cinder reviews are getting farther along, i know a nova-snapshot one is also, slowly but surely
20:28:23 <harlowja> so i guess we skip this week, since those people aren't around (or are ok/doing fine)
20:28:40 <changbl> yes
20:28:52 <harlowja> #topic checkpointing
20:29:24 <harlowja> so those people aren't online either, so i guess we can skip that, reviews are up, so feel free to jump on looking at those
20:29:33 <changbl> what is checkpointing and why is it needed?
20:30:04 <harlowja> :)
20:30:31 <harlowja> so thats a good question, part of its usage is to revert to a single 'point' instead of full reversion
20:30:39 <harlowja> *revert to last checkpoint
20:31:09 <changbl> when reaching a checkpoint, what does taskflow do?
20:31:48 <harlowja> right, need anatatasia / ivan for that question :)
20:32:00 <harlowja> but my understanding is that acts as a save point
20:32:08 <changbl> :)
20:32:12 <changbl> save data into backend?
20:32:29 <harlowja> ya, right now we save alot of indivudal data
20:32:38 <harlowja> which is fine, but a checkpoint allows us to save less (if its desired)
20:32:49 <harlowja> and to also allow for discarding data before the checkpoint
20:33:14 <changbl> still confused why checkpoint can save less, but i can save my questions for next meeting
20:33:42 <harlowja> k, maybe save less is a bad word, but it can collapse the previous tasks that have executed, and discard some of there data
20:34:17 <harlowja> say X task produces Y output used by Z task, Z task produces A output, then checkpoint *here*, at this point u can discard Y output
20:34:52 <changbl> wonder how much data can be saved .... it is just one output...
20:34:59 <harlowja> *that was a made up example ;)
20:35:07 <changbl> yes...
20:35:20 <harlowja> https://wiki.openstack.org/wiki/TaskFlow/Checkpointing
20:35:27 <harlowja> other ideas i think are on there
20:35:42 <harlowja> but lets grab those 2 folks and we can get them to dive deeper
20:36:04 <changbl> sure
20:36:09 <changbl> will read it
20:36:12 <harlowja> kk
20:36:36 <harlowja> but good questions changbl
20:36:57 <changbl> :)
20:37:22 <harlowja> #topic scoping
20:37:43 <harlowja> harlowja what is scoping and why is it needed?
20:37:46 <harlowja> :)
20:37:54 <changbl> yes :)
20:37:58 <harlowja> ha
20:38:12 <changbl> save me some typing:)
20:38:15 <harlowja> ;)
20:39:19 <harlowja> so a flaw that i think currently exists in taskflow is that when u construct flows and organize them in a hierachy, and then u submit it to an engine, u don't actually retain that structure when saving (only the root saves all the information)
20:39:42 <harlowja> so u have a hierachy of X flow (which contains Y flow which contains Z flow)
20:39:56 <harlowja> all the details are stored under X (not into there corresponding flow)
20:40:14 <harlowja> so this seems a little confusing to users that expect to examine a logbook and find the details for the given flow Y or Z
20:40:52 <harlowja> so the basic concept is to save it under the corresponding flow instead (which then does retain that connection)
20:41:19 <harlowja> it also affects the lookup of requirements for a task (to retain the correct ordering and all that)
20:41:24 <harlowja> make more sense?
20:41:33 <changbl> like ACLs?
20:41:43 <harlowja> ummm, maybe
20:41:56 <harlowja> also like scoping rules in a programming language (in a way)
20:42:26 <harlowja> *so thats why i called it scoping
20:43:08 <changbl> oh, you mean the relation (Z is Y and  Y is in X) is also retained when they are stored in logbook?
20:43:46 <harlowja> so a task exists in one of X or Y right
20:44:09 <harlowja> so it would lookup it its local X or Y first (as well as save data there directly)
20:44:25 <harlowja> and it would continue up the chain of its visible scopes to find requirements
20:44:49 <harlowja> this then retains the correct logbook format, u now find the task results in the flow that u created it in
20:44:56 <harlowja> instead of just the root flow
20:45:06 <harlowja> *which in logbook terms is a flowdetail
20:45:13 <changbl> ok, i kind of get what you mean
20:45:26 <changbl> will this involve changing a lot of code?
20:46:11 <harlowja> it mainly alters the engine lookup (where these lookups recide, and tracking that scope)
20:46:20 <harlowja> https://review.openstack.org/#/c/64895/19/taskflow/tests/unit/test_scoping.py (some examples there)
20:47:16 <changbl> k. seems no change to api, which is good
20:47:17 <AmriUnix> hello guys !!!
20:48:41 <harlowja> changbl only major change is the engine helpers api, which previously just took in the root flow_detail, which now doesn't make sense to just take that in (but it should take in the full logbook instead)
20:48:55 <harlowja> AmriUnix hi
20:49:20 <changbl> i think AmrUnix just went in to say hi.... :)
20:49:22 <harlowja> lol
20:49:24 <changbl> and then left
20:49:37 <harlowja> :)
20:51:02 <harlowja> changbl so does that sorta make sense?
20:51:08 <harlowja> its a little complicated, i know
20:51:19 * harlowja working on test cases to fully make sure that its ok
20:51:22 <changbl> i think i got what it means
20:51:26 <harlowja> kk
20:53:00 <harlowja> :)
20:53:10 <harlowja> so ya, i'll keep on adding more tests and stuff
20:53:16 <changbl> thx
20:53:58 <harlowja> #topic other-work-going-on
20:55:07 <changbl> just wonder, where are other folks today? we used to have many people in this meeting
20:55:23 <harlowja> not quite sure :-/
20:55:31 <harlowja> i pinged some of them
20:55:55 <harlowja> any other work that people want to bring up / questions?
20:55:58 <changbl> where are rackspace folks? did not see them for a while
20:56:21 <harlowja> busy with whatever they are doing i guess, never quite know :-P
20:56:29 <changbl> :-P
20:57:30 <harlowja> kk, open discuss for leftover time
20:57:34 <harlowja> #topic open-discuss
20:57:49 <harlowja> 3 minutes!
20:57:51 <changbl> no more questions from me, I guess we just need to make openstack use taskflow as much as possible
20:58:02 <harlowja> *make* is a hard one, ha
20:58:20 <changbl> a better word?
20:58:28 <harlowja> work with, haha
20:58:34 <changbl> :)
20:59:17 <harlowja> alright, guess it was just me and changbl
20:59:21 <harlowja> thx changbl for showing up :)
20:59:25 <changbl> :)
20:59:26 <harlowja> maybe more people next time, haha
20:59:31 <changbl> :)
20:59:44 <changbl> this concludes all my meeting this week
20:59:48 <changbl> gonna focus on real work now
20:59:48 <harlowja> sweet!
20:59:56 <harlowja> for others #openstack-state-management if u need more :)
20:59:58 <harlowja> #end-meeting
21:00:03 <harlowja> #endmeeting