16:00:16 <rakhmerov> #startmeeting Mistral 16:00:16 <openstack> Meeting started Mon Dec 1 16:00:16 2014 UTC and is due to finish in 60 minutes. The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:21 <openstack> The meeting name has been set to 'mistral' 16:00:31 <rakhmerov> hi 16:01:04 <NikolayM> hi ! 16:02:30 <rakhmerov> NikolayM, is Jenkins now working fine? 16:02:44 <NikolayM> yes 16:02:55 <NikolayM> two patches is already merged 16:03:04 <rakhmerov> ok 16:03:55 <dzimine> hi all 16:04:03 <rakhmerov> I wonder why devstack gate failed for https://review.openstack.org/#/c/137763/ 16:04:09 <rakhmerov> Nikolay, please take a look 16:04:14 <rakhmerov> hi dzimine 16:04:21 <rakhmerov> ok, let's start 16:04:38 <NikolayM> rakhmerov, it is just ok 16:04:45 <rakhmerov> why? 16:05:10 <rakhmerov> was it before Nastya's patch merged in or after that? 16:05:23 <NikolayM> before 16:05:35 <rakhmerov> ok, then it should be fine 16:05:37 <rakhmerov> agenda: https://wiki.openstack.org/wiki/Meetings/MistralAgenda 16:05:39 <NikolayM> after Nastya's patch devstack works fine 16:05:44 <rakhmerov> ok 16:05:55 <rakhmerov> #topic Review Action Items 16:06:07 <rakhmerov> we had no previous action items 16:06:28 <rakhmerov> #topic Current status (progress, issues, roadblocks, further plans) 16:08:25 <rakhmerov> my status: I took PTO last Friday. Almost finished fixing that race condition, today I changed the design a little bit and just want to do more testing. There's still something that I need to think about it though but I may want to postpone it for a while and switch to "join" 16:09:15 <dzimine> nothing on my side or winson's : long break last week in the US 16:09:53 <rakhmerov> I'm a little confused. Was it really a whole week break? 16:09:55 <NikolayM> added advanced tests on workflow-resume, fixed resume algorithm, fixed "Application context not found" error in tests, fixed bug when we didn't see error at action class init 16:10:09 <rakhmerov> I thought thanksgiving is just one day 16:10:50 <dzimine> yes, but as it's Thu, everyone is taking Fri off. 16:11:11 <rakhmerov> NikolayM, very good ) I'm so glad we fixed that "Application context not found" error 16:11:17 <rakhmerov> it was a pain in the butgt 16:11:27 <rakhmerov> dzimine, ok 16:11:48 <rakhmerov> NikolayM, I think that's why nobody answered us in ML about WSME 16:12:12 <rakhmerov> #topic Release "Kilo-1" progress 16:12:23 <NikolayM> I also think so :) 16:13:10 <NikolayM> rakhmerov, what is left for Kilo-1? 16:13:26 <rakhmerov> yep, just a sec 16:13:47 <rakhmerov> ok, as far as "kilo-1" release we now have effectively 2 BPs that didn't start yet 16:13:54 <rakhmerov> "pause-before" and HA 16:14:21 <rakhmerov> Bryan was going to start "pause-before" this week as far as I remember 16:14:29 <rakhmerov> and it's pretty simple to do I think 16:14:42 <rakhmerov> if he has some problems we'll help him 16:14:52 <dzimine> is "join" in Kilo-1? 16:15:08 <rakhmerov> #action rakhmerov, get in touch with Bryan to see how his work's going 16:15:13 <rakhmerov> dzimine, yes 16:15:29 <rakhmerov> I'll continue to work on it tomorrow 16:16:06 <rakhmerov> by the end of this week most likely I'll get it done 16:17:08 <rakhmerov> Nastya is also about to start doing HA testing, she's now figuring out what's needed: where and how we can do it 16:17:39 <rakhmerov> I don't think this is something that could be done just till the end of this cycle 16:17:41 <rakhmerov> though 16:17:49 <NikolayM> we have mistral 0.2 planning doc at https://docs.google.com/a/mirantis.com/document/d/1ycCOBS4gj5-3vbHjBEThTJ9gSNOOWGpX0kgdcfVw9IY/edit# 16:17:52 <dzimine> before or after rakhmerov's fix? 16:17:57 <rakhmerov> IMO, this is an ongoing piece of work 16:18:03 <NikolayM> don't dorget about it :) 16:18:20 <rakhmerov> we won't 16:18:25 <rakhmerov> dzimine, what do you mean? 16:19:14 <rakhmerov> regarding HA testing I mean that we'll do some basic testing for now to make sure our main functionality works at scale 16:19:28 <rakhmerov> but we'll keep working on that moving on 16:19:45 <dzimine> I mean any load may be problematic without the fix that you have on a review. 16:19:56 <dzimine> Anyways it'll take care of itself :) 16:20:02 <dzimine> by exposing the bugs. 16:20:10 <rakhmerov> ooh, I hope it will be in in 1 day 16:20:16 <rakhmerov> in the worst case in 2 days 16:20:30 <rakhmerov> yes :)) 16:20:46 <dzimine> Nastya do you have a plan on what/how to test? 16:21:26 <dzimine> we had a weird bug when workflow locked on 2 cpu VM. I'll ask Winson to report it. 16:21:35 <rakhmerov> btw, dzimine, if you have your own requirements on what needs to be tested first (I mean your interest) please share 16:21:56 <rakhmerov> yes, you told me but it's really weird 16:22:12 <rakhmerov> would like to see some details (may be logs?) 16:22:25 <rakhmerov> I think Nastya is not here today 16:23:04 <dzimine> yes, I think he managed to repro it on his dev env, I have just asked him to file it. 16:23:24 <rakhmerov> we already discussed the general plan and she definitely has a high-level understanding 16:23:34 <rakhmerov> ok 16:23:42 <rakhmerov> would be helpful 16:24:24 <rakhmerov> btw, dzimine, did you try Mistral in multiprocess configuration on some of your tasks? 16:24:52 <rakhmerov> it's when you have separate engines and executors (may be even api's) 16:25:11 <dzimine> no. 16:25:17 <rakhmerov> ok 16:28:04 <rakhmerov> ok, I think we're mostly on target. Productivity after summits is always not the best but I feel it's not increasing :) 16:28:14 <rakhmerov> #topic Open Discussion 16:28:25 <rakhmerov> any questions to discuss from your side? 16:28:30 <dzimine> ok, little thing: 16:28:38 <rakhmerov> some concerns, roadblocks? 16:28:38 <rakhmerov> yes 16:28:49 <dzimine> task defaults with reverse workflows, obviously don't work, right? 16:29:02 <rakhmerov> ....it's now increasing.... 16:29:10 <dzimine> the question is how to set up a default failure policy with reverse workflow. 16:29:32 <rakhmerov> dzimine, we need to check it 16:29:35 <dzimine> with "join" implemented it may not be pressing, but at least need t odocument. 16:29:38 <rakhmerov> I'm not sure it doesn't work 16:30:04 <rakhmerov> I guess policies should work with task defaults in case of reverse workflows 16:30:07 <dzimine> honestly I didn't check if it can work, was just curious 16:30:12 <rakhmerov> let us check that pls 16:30:14 <dzimine> let's check. 16:30:18 <rakhmerov> yep 16:30:56 <dzimine> Next: I am thinking we (I?) should do a bit more in writing specs. 16:30:58 <rakhmerov> #action akuznetsova: check if task-defaults work for reverse workflows (policies) 16:31:03 <dzimine> We do write blueprints, 16:31:16 <dzimine> but not in hi level of details. 16:31:22 <rakhmerov> agree 16:31:36 <rakhmerov> it's all about priorities, you know 16:31:39 <dzimine> and than, when feature is implemented 1) not all corner cases are there, 16:31:50 <dzimine> and 2) we dont know if we are done or not. 16:31:59 <rakhmerov> yes 16:32:01 <dzimine> Example: for-each. 16:32:13 <dzimine> It's so good to have it in, 16:32:20 <rakhmerov> strictly speaking, "for-each" is not 100% ready 16:32:27 <dzimine> now I have a bag of feedback on this. 16:32:40 <rakhmerov> result ordering, sequential processing are not done 16:32:46 <NikolayM> for-each for now has a set of limitations 16:33:26 <rakhmerov> dzimine, can you pls share that feedback in some form? BPs, specs? 16:33:50 <rakhmerov> NikolayM, please create new BPs for what's left on "for-each" 16:33:55 <dzimine> well that's my point, "when"? is it ready? what is left? where do we track this? all this project management routines that keep it safe 16:34:15 <rakhmerov> #action: NikolayM, create new BPs for what's left on "for-each" 16:34:30 <NikolayM> ok 16:34:43 <dzimine> So the smallest step I think is to put more details into the blueprint, that describes in more details on what needs to be done, how it is expected to work, etc. 16:35:06 <rakhmerov> https://blueprints.launchpad.net/mistral/+spec/mistral-dataflow-collections 16:35:26 <dzimine> rakhmerov and I have discussed pretty deep corner cases of for-each, I can help capture them in a blueprint. 16:35:49 <rakhmerov> let's just have a discussion in the BP itself (I know it's not very convenient but there isn't a better way now I think) 16:36:01 <rakhmerov> ok 16:36:35 <dzimine> And a new one that we thought of and people just asked to do is to introduce a "parallel" count to "throttle" the execution. parallel=1 makes it sequential. 16:36:41 <rakhmerov> I am ok with creating new BPs if you find it convenient 16:37:11 <rakhmerov> you still mean "for-each" thing? 16:37:16 <dzimine> yes 16:37:40 <rakhmerov> ooh, you're saying it's not just a binary choice "parallel" or "sequential" 16:37:43 <rakhmerov> ok 16:37:57 <rakhmerov> I see 16:38:20 <rakhmerov> NikolayM, it's going to be even much more interesting than we thought ) 16:38:49 <rakhmerov> I think it's a nice feature 16:39:48 <dzimine> I'll type it up in the etherpad referenced from the blueprint. 16:39:51 <rakhmerov> because if we, for example, have an array of a thousand elements and need to create a thousand parallel iterations then it may be an overage for just one task 16:39:58 <rakhmerov> ok 16:40:11 <rakhmerov> this is very good 16:40:29 <dzimine> But back to a general comment, let's do a write-up on what we implment, and if there are any cut corners call them out after implementation. 16:40:51 <rakhmerov> ok 16:41:31 <dzimine> the use case for parallel throttling is also concern on the target system. In our example it's VMWare vCenter and the user wants to keep it at no more than 5 new VM requests at a time. 16:41:55 <rakhmerov> yes, I'm actually thrilled to get "join" and "for-each" asap 16:42:19 <rakhmerov> yeah, you told me 16:43:03 <rakhmerov> but when we were talking about it I was thinking about a system-wide throttling rather than a "for-each" parameter 16:43:11 <rakhmerov> it should be both eventually though 16:44:17 <rakhmerov> #action dzimine, put information about additional features/requirements for "for-each" in the etherpad 16:44:26 <dzimine> ^^ done. 16:44:32 <rakhmerov> ok 16:44:44 <rakhmerov> one question that I have 16:45:01 <rakhmerov> do we need to support Keystone API v2 in Mistral? 16:45:06 <rakhmerov> any input on that? 16:45:09 <dzimine> Nikolay, how do you track what's not done in for-each? E.g., ordering of results, handling multiple arrays, etc? 16:45:31 <dzimine> Keystone v2 - anyone asked for it? 16:45:39 <rakhmerov> yes, that's the point 16:45:49 <dzimine> what's the context& 16:45:50 <dzimine> ? 16:45:55 <rakhmerov> 2-3 people last week asked about it 16:46:45 <rakhmerov> the point is: they have Icehouse OpenStack where they don't have Keystone API v3 enabled because they think it's still experimental 16:47:08 <rakhmerov> my understanding 16:47:15 <NikolayM> dzimine, it is in my notepad 16:47:30 <rakhmerov> unfortunately, I don't have a comprehensive info on that topic at the moment 16:48:06 <rakhmerov> so they said: can we have Mistral work with v2 or create some workaround for that? 16:48:19 <rakhmerov> the problem is that we use trusts 16:48:36 <rakhmerov> which actually don't work well too for us 16:49:06 <rakhmerov> but at least the concept of trusts seems to be helpful for us 16:49:46 <rakhmerov> so, I'm not expecting us to answer that right now but we need to figure that out within days 16:50:00 <rakhmerov> already create a BP (in discussion status) 16:50:03 <rakhmerov> created 16:50:43 <rakhmerov> you can leave your thoughts in it: https://blueprints.launchpad.net/mistral/+spec/mistral-keystone-api-v2-support 16:50:43 <dzimine> what will it take to support v2? did someone looked at it? NikolayM you know keystone pretty well? 16:51:42 <rakhmerov> the whole problem is in auth token expiration 16:52:12 <rakhmerov> for example, when a trigger runs a workflow we don't have user credentials 16:53:16 <rakhmerov> there are only two ways: authenticate using Mistral's system credentials (from config) or use a previously trust (basically impersonated credentials) 16:53:32 <rakhmerov> the first way is ugly because it may be a security breach 16:54:36 <rakhmerov> honestly, I don't see a good solution at this point 16:55:06 <rakhmerov> technically, if we come up with a solution it should be easy to implement 16:55:58 <rakhmerov> Dmitri, maybe you could discuss this thing with your guys 16:56:11 <dzimine> we don't use much of keystone. 16:56:34 <rakhmerov> yeah, but some of you may have seen a similar problem before 16:56:41 <rakhmerov> ok 16:56:43 <dzimine> and the problem with credentials is not limited to keystone. Talk to Fuel guys, they'll need sets of credentials to all sorts of targets 16:56:54 <rakhmerov> I know 16:56:55 <rakhmerov> yes 16:57:45 <rakhmerov> I heard that keystone folks were planning to drop trusts completely, btw, and introduce a different thing 16:57:55 <rakhmerov> but didn't see any detailed description so far 16:58:14 <rakhmerov> ok, let's finish for today 16:58:28 <rakhmerov> about a minute and a half left 16:58:43 <rakhmerov> thanks for coming 16:59:03 <rakhmerov> Dmitri, let's be in touch with you. We need your input 16:59:12 <rakhmerov> bye 16:59:29 <rakhmerov> #endmeeting