20:00:10 <harlowja> #startmeeting state-management
20:00:11 <openstack> Meeting started Thu Jan  9 20:00:10 2014 UTC and is due to finish in 60 minutes.  The chair is harlowja. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:00:14 <openstack> The meeting name has been set to 'state_management'
20:00:46 <harlowja> hi folks
20:00:51 <harlowja> welcome back from vacation/s
20:01:00 <harlowja> #link https://wiki.openstack.org/wiki/Meetings/StateManagement
20:01:09 <harlowja> agenda ^
20:03:18 <harlowja> anyone here, lol
20:03:20 <harlowja> me so lonely
20:03:21 <harlowja> lol
20:03:22 <iv_m_> hi hi
20:03:28 <haruka> hi
20:03:44 <harlowja> hi hi
20:03:59 <changbl> hi, harlowja
20:04:07 <harlowja> sweet
20:04:08 <harlowja> more people!
20:04:14 <changbl> I am on another meeting meanwhile:) listen-only mody
20:04:19 <changbl> s/mody/mode/
20:04:25 <harlowja> more meetings ftw, lol
20:04:30 <harlowja> np
20:04:43 <harlowja> #topic last-action-items
20:04:46 <harlowja> #link http://eavesdrop.openstack.org/meetings/state_management/2013/state_management.2013-12-19-20.00.html
20:04:50 <harlowja> long time ago we had a meeting
20:05:07 <harlowja> and i've still gotta do my action items, crap
20:05:22 <harlowja> well i did one of them, 'minilib for zk like stuff on pypi'
20:05:29 <harlowja> #link https://pypi.python.org/pypi/zake/
20:05:38 <changbl> yes, reading your code
20:05:40 <harlowja> useful for doing testing like stuff with zookeeper
20:05:49 <changbl> i have a working zk cluster
20:05:52 <harlowja> nice
20:05:56 <changbl> will also use your zake for fake testing
20:05:57 <changbl> I hope i can get zookeeper backend done this week
20:06:11 <iv_m_> great
20:06:11 <harlowja> changbl apparently there is also a way we can plugin to the jenkins testing servers to also use zookeeper there
20:06:24 <harlowja> #link https://github.com/openstack-infra/config/blob/master/modules/jenkins/manifests/slave.pp#L125
20:06:25 <changbl> how? can you give me some guideline?
20:06:33 <harlowja> changbl sure, let me fwd u an email
20:06:43 <harlowja> that i was having with the tooz folks
20:06:52 <harlowja> or write it up somewhere
20:07:01 <changbl> Great. I was just wondering how to write fake tests and real tests
20:07:22 <changbl> don't have much experience in writing openstack-style tests/env, will need to consult you
20:07:26 <harlowja> np
20:07:42 <harlowja> real tests i'll see about figuring out after the base zookeeper jobboard stuff goes in
20:07:50 * harlowja i haven't quite figured it out either, ha
20:07:59 <harlowja> but it seems like zookeeper is running on the test slaves
20:08:02 <harlowja> *or should be*
20:08:12 <changbl> k, we can surely do this together at some point
20:08:16 <harlowja> ya
20:08:37 <harlowja> #action harlowja writeup small wiki describing how to connect tests -> zookeepr (which apparently is on test slves)
20:08:44 <changbl> thanks~
20:08:50 <harlowja> #action harlowja add jobboard stuff to mind-shift page
20:08:57 <harlowja> #action  start brainstorm about using taskflow persistence model in cinder
20:09:02 <harlowja> ok, i'll do it this time, i swear!
20:09:09 <harlowja> *the ones i forgot to do*
20:09:57 <harlowja> ok, lets jump into 0.1.2 stuff
20:10:01 <harlowja> #topic 0.1.2 stuff
20:10:13 <harlowja> so just overview of what happened and why 0.1.2
20:10:34 <harlowja> we bound to sqlalchenmy that was in the openstack/requirements repo, it has very tight restrictions around the versions that are allowed
20:10:51 <harlowja> #link https://github.com/openstack/requirements/blob/master/global-requirements.txt#L92
20:11:10 <harlowja> so this meant that 0.1.1 had a very strict bound also
20:12:01 <harlowja> so sdague was (and is) working on making that requirement in there be 0.8.x, but taskflow having a very tight bound on the requirement (using the one from the globalrequirements) it causes the upgrade to fail when sdague  was trying to bump the requirement version
20:12:41 <harlowja> so lesson being, taskflow probably shouldn't have tight bounds if possible (especially for entrypoints), and we should try to work with as many versions as possible (within reason)
20:12:51 <iv_m_> #link https://review.openstack.org/65123 the main stuff
20:12:51 <iv_m_> #link https://review.openstack.org/65199 also
20:12:51 <iv_m_> #link https://review.openstack.org/65135 for infra config
20:13:38 <harlowja> right, the above will allow us to remove the requirement bound, and also the infra one will allow us to test against various sqlalchemy versions (i tested it locally on a box, using postgres, mysql, all 3 versions of sqlalchemy, no problems showed up, so thats good)
20:14:01 <iv_m_> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/023387.html ML discussion
20:14:04 <harlowja> once those 2 reviews go in, then the jenkins slaves will run against those 3 variations (+ a few others) for us
20:14:39 <harlowja> and that means, we shouldn't hit this problem again (but review times will likely go up, as this adds like 10 new envs to build/test in)
20:15:11 <harlowja> right now it seems the testing of taskflow takes about 1.2 minutes on avg
20:15:20 <harlowja> it will probably take more time than that with all these new envs
20:15:26 <iv_m_> 20 envs if i'm counting right
20:15:29 <harlowja> :)
20:15:50 <harlowja> ya, watching zuul i haven't noticed any project with that many envs, so we'll see how this goes
20:16:01 <harlowja> max i've seen is like 11
20:16:16 <harlowja> anyways, so thats the reason for 0.1.2
20:16:43 <harlowja> https://etherpad.openstack.org/taskflow012 has a few reviews that i'd like to get in before that, nothing majorly a problem that i see
20:17:22 <harlowja> jenkins though is acting up today, so hopefully tommorow 0.1.2
20:17:55 <harlowja> *or later today, we'll see
20:18:32 <harlowja> k
20:18:36 <harlowja> onward to 0.2.0
20:18:42 <harlowja> #topic 0.2.0 status
20:18:52 <harlowja> so this one i think is progressing nicely
20:19:04 <harlowja> and mostly even ontime (i think we said around jan)
20:20:08 <harlowja> to me its composed of [checkpoints, jobboard reference impl, base remote-worker-model, smart-reviersion, other cleanups..]
20:20:13 <harlowja> iv_m_ does that sound about right
20:20:58 <harlowja> i'd rather not add https://review.openstack.org/#/c/64895/ in to 0.2.0 (even though i think its neat)
20:21:45 <harlowja> i think anastasia is making good progress on checkpointing
20:22:20 <iv_m_> #link https://review.openstack.org/#/q/status:open+project:stackforge/taskflow+branch:master+topic:checkpoints,n,z
20:22:26 * iv_m_ plays for link bot today
20:22:37 <harlowja> although i need to bug her about https://review.openstack.org/#/c/64111/ (since i'm still not quite sure if checkpoints belong so tightly connected to flows)
20:23:03 <harlowja> to me they almost feel like a new thing in the execution graph, not so tightly connected to flows
20:23:16 <iv_m_> her ultimate goal is reversion-strategies, with checkpoints and smart-revert as intermediate points
20:23:33 <harlowja> sure
20:24:09 <harlowja> i almost feel like checkpoints 'wrap' subgraphs (in a way) and that means they don't need to be coupled to flows
20:24:35 <harlowja> but i'll try to bug her about this
20:25:14 <harlowja> and the zookeeper stuff, anyone feel free to review, no more commits from me i think :-P
20:25:22 <harlowja> #link https://review.openstack.org/#/c/54220/
20:25:23 <harlowja> ;)
20:25:27 <iv_m_> will do
20:25:49 <harlowja> thx iv_m_
20:26:09 <harlowja> what else did i miss, hmmm
20:26:27 <changbl> harlowja, will review your 54220
20:26:33 <harlowja> changbl sweet, thx
20:26:49 <harlowja> the changeset is at the waaay bottom of the page, lol
20:26:57 <changbl> way too long...
20:27:00 <harlowja> :-P
20:27:33 <iv_m_> it's not even patchset 100
20:27:33 <harlowja> ah changbl do u want to try to get your zookeeper logbook in 0.2
20:27:46 <changbl> definitely
20:28:06 <harlowja> iv_m_ we need to at least get it to 2^6
20:28:22 <harlowja> changbl cool, i guess lets see where we are by next week
20:28:34 <harlowja> and we can see if we can decide on a 0.2 'cut-date'
20:28:42 <harlowja> seems doable
20:29:18 <changbl> sure
20:29:20 <harlowja> cool
20:29:46 <harlowja> alright, next also intereting topic, oslo
20:29:51 <harlowja> #topic oslo
20:30:03 <iv_m_> oh
20:30:06 <harlowja> so this was an interseting question that also came out of the sqlalchemy question
20:30:56 <harlowja> will summarize, taskflow could become a part of oslo, if we want, i think certain folks would like that since it then means we can be more tightly integrted/gated on in openstack (part of the reason for this was the requirements, which i think we have solved in the above reviews anyway)
20:31:19 <harlowja> from what i can tell it wouldn't mean we lose independence, or momentum
20:31:23 <harlowja> dhellmann tyt
20:31:27 <harlowja> ^yt
20:31:58 <harlowja> it would i guess gain us more exposure/publicity (maybe)
20:32:22 <iv_m_> what it means 'tight integration'?
20:32:30 <iv_m_> except cool repo url
20:34:13 <harlowja> ya, so i think it means that if we change taskflow, that if its in oslo, then the infra team would feel comfortable automatically activating jobs to make sure cinder,nova... work with the newer/updated taskflow
20:34:23 <harlowja> thats what i think it means by tighter integration
20:35:14 <harlowja> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/023541.html
20:35:51 <harlowja> how about i investigate that a little more, and maybe can report back on next week, its still a little cloudy what it would buy
20:36:03 <harlowja> repo name change, possibly above testing feature
20:36:32 <iv_m_> well, i don't see any reason for testing/gating openstack with our master
20:36:55 <harlowja> sure, understandable
20:37:25 <harlowja> anyways, let me write up a pros/cons thing, and see if i can filter it out to what it really means
20:37:52 <harlowja> i think also part of it is 'unknown' as afaik no other library really has this tight of integration with openstack (and isn't already in oslo)
20:38:17 <harlowja> the oslo.* modules are already tightly integrated into openstack (imho to tightly)
20:38:23 <harlowja> ^ some of them, not all
20:39:55 <harlowja> #action harlowja work on etherpad with oslo pros/cons so that we can understand it a little better
20:40:58 <harlowja> #topic integration-status
20:41:13 <harlowja> so i'm seeing some good reviews still going on
20:42:13 <harlowja> https://etherpad.openstack.org/p/TaskflowIcehouseWhoWhatWhereWhy has most of them (hopefully didn't miss any)
20:42:50 <harlowja> feel free to update that
20:43:27 <harlowja> seems like most are still working on those, so thx for those that are :)
20:43:37 <harlowja> and let us know if we can help (or review, or all the above)
20:46:09 <harlowja> cool, onwards
20:46:21 <harlowja> ah, nexst one was checkpointing, lets skip that an i'll ping anastasia
20:46:37 <harlowja> #topic scoping
20:47:03 <iv_m_> on checkpointing you can ping me, too
20:47:07 <harlowja> iv_m_ kk
20:47:08 <harlowja> thx
20:47:28 <harlowja> so the scoping idea came out of a brainstorm from iv_m_ and me a few days ago
20:47:41 <harlowja> the idea is similar to http://en.wikipedia.org/wiki/Scope_%28computer_science%29
20:47:48 <harlowja> *but much simpler
20:49:03 <harlowja> currently a flow structure will be flattened into a single execution graph, this will be done inside the engine code, currently this flattening discards the locality information of where a task exists and what flow it exists in
20:49:13 <harlowja> *done inside the engine compile() phase
20:49:37 <iv_m_> which can be considered a good thing in fact
20:49:49 <harlowja> this means that a engine can execute with a single flow_detail (since we just flattened it down to a single graph)
20:49:54 <harlowja> iv_m_ sure, its good and bad :-P
20:50:35 <harlowja> so scoping changes that, and doesn't lose the locality information, but also makes it so that its no longer a single flow_detail (and associated storage object) which is referenced for task requirements and failures and ...
20:50:44 <harlowja> ^ complexity...
20:51:41 <harlowja> so i worked on a change @ https://review.openstack.org/#/c/64895/ that keeps the old flat way of working (no scopes, 1 flow detail used) and can also add scopes (via config, which will result in many flow _details)
20:52:12 <harlowja> each way may be useful (or not, depending on the user)
20:52:22 <harlowja> so just something to think about
20:53:07 <harlowja> *in a way what exists before was that u created flows with nested flows, and then they all got flattened and referred to a single global scope
20:53:43 <harlowja> with the above review, scope is tracked, so that nested flows refer to there local scopes (and ... to global scope)
20:54:22 <harlowja> anyways feel free to check it out, think over it, seems like a nice to have (the ability to work either way)
20:55:16 <harlowja> k, thats all i got, lets open up for 5 more minutes of fun
20:55:20 <harlowja> #topic open-discuss
20:56:02 <harlowja> anything anyone wants to bring up, new ideas, new problems, new solutions
20:56:15 <harlowja> *or other newness
20:57:22 <harlowja> going once
20:57:37 <harlowja> goiinnng twice
20:57:50 <harlowja> going thrice
20:57:59 <harlowja> soolld
20:58:18 <harlowja> we'll be in #openstack-state-management as usual folks
20:58:22 <harlowja> thanks for coming!
20:58:29 <harlowja> #endmeeting