19:01:21 <amitgandhinz> #startmeeting Poppy Weekly Meeting 19:01:22 <openstack> Meeting started Thu Jan 22 19:01:21 2015 UTC and is due to finish in 60 minutes. The chair is amitgandhinz. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:23 <tonytan4ever> o/ 19:01:26 <openstack> The meeting name has been set to 'poppy_weekly_meeting' 19:01:26 <miqui> 0/ 19:01:32 <amitgandhinz> #topic RollCall 19:01:41 <amitgandhinz> hi tonytan4ever, miqui ! 19:01:43 <cpowell_> o/ 19:01:44 <miqui> o/ 19:01:47 <tonytan4ever> hi 19:01:53 <miqui> hi folks.... 19:02:04 <obulpathi> o/ 19:02:13 <catherineR> o/ 19:02:22 <megan_w_> o/ 19:02:37 <amitgandhinz> #link Agenda - https://wiki.openstack.org/wiki/Meetings/Poppy 19:02:50 <amitgandhinz> #topic Last Weeks Items 19:03:07 <amitgandhinz> #link http://eavesdrop.openstack.org/meetings/poppy_weekly_meeting/2015/poppy_weekly_meeting.2015-01-15-19.00.html 19:03:35 <amitgandhinz> tonytan4ever to reach out to the TaskFlow IRC channel to learn the differences between celery and taskflow 19:03:36 <wbrothers> o/ 19:03:47 <amitgandhinz> we have an agenda later to talk about this in detail 19:03:54 <tonytan4ever> I did some research about TaskFlow 19:04:05 <tonytan4ever> and compared it with Celery. 19:04:09 <miqui> yeah..i did some pking at it.. 19:04:18 <amitgandhinz> cool. we can dig in a bit 19:04:30 <amitgandhinz> wbrothers to add doc to cdn vendor wiki page on how to add a mimic plugin for their api 19:04:46 <tonytan4ever> This is the advantage with TF: a ) Tasks are grouped into workflows that can define ordering and dependency 19:05:01 <amitgandhinz> tonytan4ever: save it for the agenda items in a little bit ;-) 19:05:02 <tonytan4ever> . which are compiled to a directed graph and can be executed in parallel. These workflows can have portions that retry, and can revert their work on failure,. 19:05:15 <tonytan4ever> OK 19:05:53 <amitgandhinz> wbrothers: did you document how to build mimic plugin's for cdn providers? 19:06:53 <amitgandhinz> ok moving on then 19:07:09 <amitgandhinz> ALL - investigate worker management 19:07:17 <amitgandhinz> we'll discuss that shortly with the TF stuff 19:07:19 <wbrothers> No I haven't 19:07:30 <amitgandhinz> ok thanks wbrothers ,i'll push it forwrad 19:07:42 <amitgandhinz> #action wbrothers to add doc to cdn vendor wiki page on how to add a mimic plugin for their api 19:08:10 <amitgandhinz> #topic Updates on Blueprints and Bugs 19:08:13 <amitgandhinz> #link https://launchpad.net/poppy/+milestone/kilo-3 19:08:29 <amitgandhinz> yes you read that right! we are now in Kilo-3 cycle 19:08:52 <amitgandhinz> i released Kilo-2 yesterday which contained mostly bug fixes (https://launchpad.net/poppy/+milestone/kilo-2) 19:09:14 <amitgandhinz> i moved any remaining bugs forward into k3, and groomed the blueprints targeted 19:09:27 <miqui> thanks amitgandhinz 19:09:33 <amitgandhinz> the focus is now back on to features 19:09:59 <amitgandhinz> lets quickly go over the bugs that are being worked on.... 19:10:12 <amitgandhinz> https://bugs.launchpad.net/bugs/1408296 19:10:39 <amitgandhinz> obulpathi: are you currrently working on this or can i unassign you 19:10:40 <obulpathi> I still have to start this 19:10:52 <amitgandhinz> same for #1408301 ? 19:11:11 <obulpathi> yes 19:11:27 <amitgandhinz> ok, removing your name 19:11:30 <obulpathi> ok 19:12:25 <amitgandhinz> miqui: https://bugs.launchpad.net/bugs/1381091 19:12:54 <amitgandhinz> are you working on this or should i remove your name also 19:13:10 <miqui> keep my name on it... 19:13:23 <amitgandhinz> ok 19:13:50 <amitgandhinz> then there are 3 bugs in progress that need reviews or are being worked on 19:14:06 <amitgandhinz> obulpathi: https://bugs.launchpad.net/bugs/1380679 19:14:13 <amitgandhinz> what is the status of this one, its been around for a while 19:14:38 <obulpathi> I will finish it up by this week 19:14:44 <amitgandhinz> i dont see a PR for it 19:14:49 <obulpathi> lets leave the status the same for now 19:14:54 <amitgandhinz> ok 19:15:11 <amitgandhinz> ok, moving on to blueprints 19:15:25 <amitgandhinz> https://blueprints.launchpad.net/poppy/+spec/queue-driver 19:15:34 <amitgandhinz> we are going to discuss this in a bit, so no update needed right now 19:15:54 <amitgandhinz> malini is out so cant get update on https://blueprints.launchpad.net/poppy/+spec/conformance-tests 19:16:10 <amitgandhinz> obulpathi: https://blueprints.launchpad.net/poppy/+spec/mimic-racksapce-dns 19:16:20 <obulpathi> I did some progreass on thsi 19:16:33 <obulpathi> good chunks of API are in working state: CREATE and search 19:16:48 <obulpathi> about 50% is done 19:16:53 <amitgandhinz> so change it to Good Progress then 19:16:56 <obulpathi> yep 19:17:18 <amitgandhinz> miqui: any update on https://blueprints.launchpad.net/poppy/+spec/home-doc 19:17:47 <miqui> started to look at this... did some one else in the team make a change? 19:17:55 <miqui> i notice some differences... 19:18:03 <amitgandhinz> i dont think so 19:18:21 <amitgandhinz> let me look... 19:18:51 <amitgandhinz> https://github.com/stackforge/poppy/commits/master/poppy/transport/pecan/controllers/v1/home.py 19:18:56 <amitgandhinz> no changes since dev 2 19:18:58 <amitgandhinz> dec 2 19:19:06 <miqui> hmm, ok... well.. i will dedicate more time to it.. sometimes i cannot correlate the actual output with the verb/action 19:19:15 <miqui> ah k... 19:19:36 <miqui> well...i'll keep at it... 19:19:40 <amitgandhinz> ok 19:19:45 <miqui> probably get it wrong first couple of passes... 19:20:01 <miqui> so miost likely i'll be asking a review 19:20:26 <amitgandhinz> thats fine, all part of the process =) 19:21:00 <amitgandhinz> ok any other updates for any of the bp here https://launchpad.net/poppy/+milestone/kilo-3 19:21:11 <amitgandhinz> kilo 3 ends on March 5 19:21:35 <obulpathi> nop 19:21:39 <obulpathi> no updates 19:21:50 <amitgandhinz> most of these stories are related to adding some more basic features customers need, and cleaning up code to make it better or more maintainable 19:22:29 <amitgandhinz> ok moving on then... 19:22:39 <amitgandhinz> #topic New Items - Taskflow vs Celery Discussion 19:22:46 <tonytan4ever> OK 19:23:00 <amitgandhinz> from miqui: http://paste.forj.io/show/289/ 19:23:14 <miqui> yeah that was my chat with tf folks... 19:23:22 <amitgandhinz> go for it tonytan4ever, tell us what you learned over the past week 19:23:29 <miqui> then started looking at their architecture... 19:24:12 <tonytan4ever> In short TaskFlow provides mechanism for workflow patterns, gives you ability to build task based on dependencies and retries, and things of that nature. 19:25:12 <amitgandhinz> so our tasks and workflow is basically 1. talk to cdn provider. 2. update storage. 3. update dns 19:25:13 <tonytan4ever> So the dependencies and retry mechanism are buzzword here 19:25:40 <tonytan4ever> Yes, execept in create we need to update storage first. 19:25:45 <amitgandhinz> ok 19:25:52 <cpowell_> from miqui's discussion, "celery is highly (imho) intrusive" stuck out to me alot 19:25:58 <amitgandhinz> and dependency is 2 only happens after 1 succeeds? 19:26:15 <tonytan4ever> Yeah 19:26:28 <tonytan4ever> But I argue since we only have three tasks at most 19:26:34 <amitgandhinz> and retry is retry on failure (eg rate limiting, server unavailable, etc) 19:26:47 <tonytan4ever> yeh 19:27:15 <miqui> true, workflow might be simple now...but could grow tomore use cases... 19:27:17 <tonytan4ever> Following what I want to argue, since we only have 3 tasks at most, their dependencies/retries are not that important. 19:27:43 <tonytan4ever> And TF does have a draw back that monitoring worker process is not as easy as in Celery. 19:27:47 <amitgandhinz> yeh, that could be added later if needed/wanted 19:27:55 <obulpathi> +1 19:28:04 <amitgandhinz> how do you monitor workers in TF? 19:28:11 <tonytan4ever> You can only read logs. 19:28:12 <miqui> good qurstion... 19:28:39 <tonytan4ever> Celery has flower which is favored by me and will be liked by Ops as well. 19:29:03 <amitgandhinz> harlowja: ^^ 19:29:24 <harlowja> hallo 19:29:34 <obulpathi> hi 19:29:38 <miqui> ...hi 19:29:42 <amitgandhinz> harlowja: if you have a sec, the Poppy team is discussing between choosing TaskFlow or Celery for our tasks/workers 19:29:42 <tonytan4ever> H harlowja 19:29:47 <harlowja> hi there 19:30:02 <harlowja> well i'm still of the believe that celery is a full-app solution, u adopt it all or none of it :) 19:30:11 <harlowja> while taskflow i think isn't 19:30:16 <harlowja> *or wasn't intended to be 19:30:28 <amitgandhinz> how does monitoring workers work with TF? 19:30:52 <harlowja> so it doesn't have a flower like thing (yet) but u are free to make one 19:31:07 <harlowja> but depends on what u mean by monitoring workers 19:31:20 <miqui> up/down? 19:31:34 <amitgandhinz> up/down, do we need to scale them out more 19:31:38 <tonytan4ever> For example, we need to know if a worker has been successfully completed 19:31:39 <obulpathi> progress and failures 19:31:40 <miqui> reached a threshold (like max pool size) 19:31:43 <harlowja> don't there exist a ton of existing solutions to monitor this stuff? :) 19:32:15 <harlowja> well completed succesfully is different than up/down; up/down i'd hope can be monitored by the existing tools that monitor process no? 19:32:28 <miqui> yes 19:32:36 <amitgandhinz> cpowell_: thoughts on monitoring? 19:32:37 <harlowja> progress and failures, that would need to be added, but i don't suspect its hard 19:33:13 <cpowell_> normally just looking for process name 19:33:24 <cpowell_> count number of PIDs 19:34:17 <cpowell_> even with Celery, your celery worker would be managed by Supervisord or something 19:34:25 <miqui> sensuapp easily do this... 19:34:38 <cpowell_> <3 Sensu 19:34:44 <amitgandhinz> lol 19:34:44 <harlowja> ya, i think there is like 10 trillion things that do this no? 19:34:51 <miqui> yup 19:35:00 <miqui> nagios 4, etc...etc.... 19:35:21 <amitgandhinz> tonytan4ever: so what does celery's flower give us that we need? 19:35:35 <tonytan4ever> It gives us all what we need. 19:35:55 <harlowja> pretty pictures :-P 19:36:00 <tonytan4ever> Up/down, scale situation, task success/failure. 19:36:07 <cpowell_> have we considered documentation? 19:36:15 <tonytan4ever> Yeah, and it's pretty. 19:36:31 <harlowja> if celery docs are much better than taskflow, i failed, ha 19:36:37 <harlowja> i hope its nearly the same :-P 19:36:46 <harlowja> *and/or better 19:37:06 <amitgandhinz> how widely used is taskflow currently (in production systems)? 19:37:08 <tonytan4ever> I would say I can read both celery and tf. Tf's docs are fine for me. 19:37:24 <harlowja> amitgandhinz so cinder has been using it (not in the same manner u have in mind) for a while 19:37:31 <harlowja> http://www.rackspace.com/cloud/big-data/ uses the workers 19:37:35 <harlowja> and i guess thats been going for a year 19:37:49 <harlowja> mirantis uses it in https://github.com/MirantisLabs/pumphouse 19:38:08 <harlowja> it though becomes a question of what funcitonality do u want to us 19:38:09 <harlowja> *use 19:38:22 <harlowja> since u don't have to use the workers (if u just want to run workflows in local threads/processes) 19:38:42 <harlowja> celery i think has bigger names, but thats imho partially because its older 19:38:55 <amitgandhinz> yeh we want it to perform async tasks that are initiated from the rest api, so we will be using the workers 19:39:34 <tonytan4ever> Yeah. To use worker/queue is our primary reason to have this discussion here in the first. 19:39:35 <harlowja> imho; i'd do an evaluation, decide on what u guys feel is appropriate and go with it (if its not taskflow, thats fine) 19:39:47 <harlowja> use the right tool that u guys feel is ok :) 19:40:06 <harlowja> celery hasn't been accepted to well into openstack (its been tried like 3+ times) 19:40:34 <harlowja> but maybe that situation is better now 19:40:52 <amitgandhinz> i saw barbican's issues with getting it accepted, but wasnt sure how wide spread its lack of acceptance was 19:41:17 <tonytan4ever> Yeah, plus celery is FreeBSD license. 19:41:46 <amitgandhinz> tonytan4ever: from your view, besides monitoring, what other benefits do you see with using celery? 19:41:52 <harlowja> ah the good ole http://lists.openstack.org/pipermail/openstack-tc/2013-December/000436.html :-/ 19:42:02 <amitgandhinz> harlowja: lol ya 19:42:11 <harlowja> amitgandhinz i don't think its been attempted since 19:42:21 <harlowja> but its been a year, maybe time to try again :) 19:42:27 <harlowja> *seems to happen once per year, lol 19:42:37 <harlowja> *year+- 19:42:43 <tonytan4ever> Celery also has simple exception/failure handling mechanism, for us only with three tasks it should be good enough. 19:43:15 <harlowja> tonytan4ever what does freebsd license help? 19:43:17 <harlowja> apache not good? 19:43:31 * harlowja curious 19:43:44 <miqui> i thought apache was preferred... 19:43:53 <tonytan4ever> That's a drawback of celery comparing to TF, harlowja 19:44:00 <amitgandhinz> i think thats what tonytan4ever was implying =P 19:44:12 <harlowja> tonytan4ever oh, i'm not such a license expert, not really sure the diff, ha 19:44:33 <tonytan4ever> And we can adopt our current async workers for celery task functions pretty easily. 19:44:44 <tonytan4ever> Not too much code their. 19:44:56 <tonytan4ever> This is the benefit of celery ^^ 19:45:40 <harlowja> meh, i'm meh meh on that, stuff is never as easy as that ;) 19:46:01 <tonytan4ever> With tf we'd have to break those function down into tasks and combine them into flows. 19:46:20 <obulpathi> The Apache 2.0 licenses contain a patent grant, which means that at least the authors of the code are giving you any rights that you need for the authors' patents that happen to be in the code that you are using. 19:46:34 <harlowja> tonytan4ever ya, the amount of refactoring either way is imho marginal 19:46:37 <obulpathi> FreeBSD license does not grant the use the rights on patents used by the code 19:46:40 <amitgandhinz> im ok with a bit more work if the code becomes more granular into units 19:46:46 <obulpathi> s othat the major upside of Apache license 19:46:53 <harlowja> obulpathi good to know 19:47:01 <obulpathi> :) 19:47:39 <amitgandhinz> ok do we have more questions on celery vs taskflow or are we ready to vote on it? 19:47:39 <miqui> amitgandhinz: +1, granular so it does one thing and one thing well.. 19:47:56 <harlowja> it might be useful to prototype both ways, idk 19:47:57 <harlowja> just a thought 19:48:10 <harlowja> *up to u guys though 19:48:21 <miqui> /and let tf handle the life cycle of it... 19:49:26 <amitgandhinz> #startvote Should we implement Taskflow or Celery? Taskflow, Celery, Both 19:49:27 <openstack> Begin voting on: Should we implement Taskflow or Celery? Valid vote options are Taskflow, Celery, Both. 19:49:28 <openstack> Vote using '#vote OPTION'. Only your last vote counts. 19:49:54 <cpowell_> well, judging from the tone of the email harlowja linked, I think that celery is a non-starter... 19:49:57 <tonytan4ever> #vote Celery 19:50:15 <harlowja> cpowell_ the tone might of calmed down over the past year (i'm must not sure), lol 19:50:20 <harlowja> *i'm just not sure 19:50:35 <amitgandhinz> we could always seed the thought again on the dev list 19:51:01 <harlowja> *bring your flak jackets (just incase) 19:51:05 <amitgandhinz> lol 19:51:24 <amitgandhinz> cpowell_: miqui: obulpathi: ready to vote yet, or do you have more questions? 19:51:34 <cpowell_> are there docs for prodution installation/configuration 19:51:51 <cpowell_> I see how to use it in the code... 19:51:59 <miqui> #vote TaskFlow 19:52:09 <harlowja> probably could be made better cpowell_ in that area; but i don't think its to much work there 19:52:26 <harlowja> be nice to have some examples/recommendations for that i guess 19:52:57 <harlowja> the rackspace guys maybe can make that (they've been using this model/workers... for a whie) 19:52:58 <harlowja> *while 19:53:12 * harlowja will poke them on this, ha 19:53:21 <harlowja> and poke and poke :-P 19:53:21 <amitgandhinz> cpowell_: we should sync up with the big data folks 19:53:35 <cpowell_> ok 19:53:39 <harlowja> def amitgandhinz cpowell_ #openstack-state-management channel (dkrause or jimbohickville) 19:53:52 <harlowja> dkrause not on right now, but jimbo (greg) is 19:54:00 <amitgandhinz> we can do that offline 19:54:01 <harlowja> nm, he left to, ha 19:54:09 * amitgandhinz heard us and ran =P 19:54:11 <harlowja> :) 19:54:21 <amitgandhinz> ok only 6 minutes left 19:54:24 <amitgandhinz> can we vote? 19:55:21 <cpowell_> I reserve the right to complain about either choice 19:55:32 <harlowja> lol 19:55:35 <cpowell_> #vote TaskFlow 19:55:36 <tonytan4ever> You can vote both right, lol 19:55:37 <amitgandhinz> lol, no we hold you to it for ever and ever 19:55:40 <obulpathi> #vote None (Start with only queues and implement worker mechanism later, depending on the needs and our ability to handle TaskFlow. I prefer TaskFlow tough, if we can manage it) 19:55:41 <openstack> obulpathi: None (Start with only queues and implement worker mechanism later, depending on the needs and our ability to handle TaskFlow. I prefer TaskFlow tough, if we can manage it) is not a valid option. Valid options are Taskflow, Celery, Both. 19:55:54 <obulpathi> cpowell_: Same thoughts here 19:56:06 <amitgandhinz> #vote TaskFlow 19:56:22 <tonytan4ever> OK. I guess it's TaskFlow then... 19:56:23 <obulpathi> #vote TaskFlow 19:56:26 <cpowell_> I am insterested in TaskFlow, but from an Ops perspective I need to see more 19:56:47 <amitgandhinz> miqui: ??? 19:57:42 <amitgandhinz> its going to have to start with a prototype either way 19:57:48 <miqui> voting.. 19:57:52 <miqui> #vote TaskFlow 19:58:16 <amitgandhinz> ok i think thats everyone 19:58:21 <amitgandhinz> #endvote 19:58:22 <openstack> Voted on "Should we implement Taskflow or Celery?" Results are 19:58:23 <openstack> Celery (1): tonytan4ever 19:58:24 <openstack> Taskflow (4): miqui, obulpathi, cpowell_, amitgandhinz 19:58:44 <amitgandhinz> ok, so lets do a prototype of taskflow 19:59:03 <tonytan4ever> I am alway be minority... Lol. 19:59:03 <amitgandhinz> and we should build it in a way such that if in the future we want to build celery workers we arent locked into a corner 19:59:16 <amitgandhinz> tonytan4ever: even minority's have a voice =) 19:59:26 <tonytan4ever> OK. 19:59:31 <tonytan4ever> agreed. 19:59:47 <amitgandhinz> the kicker for me i feel is openstack acceptance of taskflow, and the monitoring is still possible with taskflow workers (with a little mrore work) 20:00:06 <amitgandhinz> and the nice to have of smaller units of tasks makes for better code 20:00:12 <amitgandhinz> at the expense of more time to build it 20:00:16 <amitgandhinz> anyway, time is up 20:00:23 <amitgandhinz> we should continue discussing in the channel 20:00:24 <amitgandhinz> thanks harlowja 20:00:25 <harlowja> cools, #openstack-state-management guys 20:00:28 <tonytan4ever> See you all. 20:00:36 <amitgandhinz> #endmeeting