16:00:40 #startmeeting Mistral 16:00:41 Meeting started Mon Dec 16 16:00:40 2013 UTC and is due to finish in 60 minutes. The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:42 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:44 The meeting name has been set to 'mistral' 16:00:57 Hi, guys! 16:01:09 hi! 16:01:23 let's start 16:02:08 hi 16:02:11 #topic Review AIs 16:03:00 btw, here's the link to agenda: https://wiki.openstack.org/wiki/Meetings/MistralAgenda 16:03:31 the first one: 16:03:34 rakhmerov, submit blueprints for TaskFlow about lacking functionality (i.e. asynchronous tasks) 16:04:23 for now I submitted 1 BP for TaskFlow related to asynchronous tasks processing: https://blueprints.launchpad.net/taskflow/+spec/async-tasks 16:04:50 this is the most important thing for us that TaskFlow folks are working on now 16:04:53 hi! 16:05:01 hi ) 16:05:59 like we discussed before we decided to implement Mistral engine without TaskFlow for now and work with TaskFlow guys on additional requirements 16:06:05 Did you have a chance to discuss the BP with TaskFlow team? Any feedback? 16:06:51 well, we discussed in details the whole problem with Joshua and Ivan Melnikov and based on that I created a BP 16:07:04 and we're on the same page with them here 16:07:53 so yes, Ivan was going to implement some prototype soon and we need to check the status of that 16:07:58 Ok, cool 16:08:15 #action rakhmerov, check the status of async tasks in TaskFlow with Ivan 16:09:03 we also hav an AI: Evaluate an interface of salt minion 16:09:21 we didn't get to it last week 16:10:01 we need to pas it on to this week, I think it will be possible by the end of the week 16:10:23 #action rakhmerov, Evaluate an interface of salt minion 16:10:56 the same thing about the next one: Work on joel_c requirement about communication with VM agent 16:12:03 we need to get in touch with him to discuss the details 16:13:05 gokrokve, can we set up a meeting with joel_c about that? 16:13:10 There is a meeting with Joel at 9 16:13:20 Yes. I will ask him about it 16:13:20 ooh 16:13:28 You can also join today 16:13:46 I would prefer to have a separate one 16:13:59 and know about it in advance :) 16:14:35 like we discussed before it can be our next step in development 16:14:56 so we need to find out what he needs exactly to take it into account 16:16:55 #action gokrokve, set up a meeting with joel_c to find out about his requirements 16:17:09 let's move to discussing PoC status 16:17:11 ok 16:17:20 #topic PoC status 16:18:11 at this point we implemented approximately 90% of what we planned 16:19:26 including Rest API, python client for Rest API, Scheduler (to run workflows according to cron patterns), authentication and lots of other infrastructural things 16:19:46 the main thing that is left is the engine itself 16:21:22 last Wed we started working on the 3rd version of it since, as we think, we finally came up with really simple idea of how it can be built with regard to further HA and Scalability requirements 16:22:49 so by now we've written all the required code for it and it's just a matter of waiting for a couple of commits (dependencies) to be merged to start testing it 16:23:08 I'm planning to start testing it all together tomorrow 16:23:43 so by the end of this week we're planning to demonstrate the results of what we did 16:23:58 and we agreed to release it with tag "poc" 16:24:56 so generally, the work on engine turned to be the most tricky thing but hopefully we found a good solution 16:25:21 It expected to be the most tricky thing, isn't it? 16:25:31 yeah, right 16:25:48 from the very beginning we didn't have a clear vision of how it should be implemented 16:26:07 and I think it's a good achivement that we now have it (at least we think so) 16:26:36 even though we're a little bit behind the schedule 16:26:44 ok 16:27:01 one special thing that I was going to discuss again is Demo App for PoC 16:27:17 #topic Discuss Demo App for PoC 16:29:22 NikolayM started working on it and looks like we have to implement it in a different way if we want to get it done quickly, not like we planned initially (create VMs, volumes etc.) 16:29:58 BTW, I didn't manage to find any description on wiki or in ether pads. Do we have any? 16:30:09 agree with Renat 16:30:25 igormarnat, let me doublecheck that 16:30:39 the problem is that in order to implement a simple scenario like "Create VMs", "Attach volumes" etc. we need something like a Data Flow already 16:31:25 because say if we create a VM as the first step we need a way to transfer a VM id on to the next task 16:31:47 can we simplify the scenario e.g. only backup of VMs 16:31:57 this is not yet clear from DSL perspective 16:31:57 yes 16:32:19 Or we can create the very simple data flow, something really primitive, can't we? 16:32:30 akuznetsov, yes I think we can but let's ask NikolayM416 16:33:03 igormarnat, I think given our current activities it'll take time, I think ~1 week 16:33:21 if we can afford this then yes, we can follow that way 16:33:47 so let me list our options first and then we can decide which way to proceed 16:34:09 Yep, I just tried to add something to the list of options, not to select one:) 16:34:13 1. Create a simple workflow with OpenStack services that do not require Data Flow 16:34:45 2. Create a simple implementation of Data Flow (~1 week estimation) 16:35:47 3. Create a simple Demo App that exposes 1-2 endpoints, uploads a workflow with webhooks only leading back to those endpoints of the app 16:36:14 so option 3 does not assume having OpenStack installed 16:37:12 and it generally it can just work locally, when an endpoint gets a request to execute an action from Mistral we could just log it in a beautiful way and demonstrate how tasks are executed 16:37:24 in what order and with what parameters 16:37:44 so what are your thoughts guys? 16:37:55 BTW, what does Stan say about this? 16:38:13 I think NikolayM has already started working on option 3 16:38:18 today 16:38:27 but we can change that direction right now 16:39:22 NikolayM416, what do you think about option with just making backups? 16:39:36 do you think it has some pitfalls? 16:41:15 for me it seems to be fine 16:41:47 I just want to make sure we will be able to implement it quickly 16:41:49 rakhmerov: I think it is good idea but it requires auth 16:41:53 people are waiting :) 16:42:08 authentication should be done 16:42:20 I think .3 is fine, providing that we'll start working on implementation of PoC of data flow in parallel with discussion of results of current PoC 16:42:38 igormarnat, agree with you 16:43:18 ok, let me make a suggestion 16:43:32 given on the information that you guys provided 16:44:54 in my understanding, https://review.openstack.org/#/c/62037/ is the only thing we're waiting to be able to create simple (w/o Data Flow) workflows working with OpenStack, right? 16:45:21 it's now failing on the tests 16:45:32 I think we need to fix it somewhere 16:45:45 akuznetsov, can you please take a look at it tomorrow? 16:46:28 Yes 16:46:33 so my suggestion is to proceed with option 3 (simple app w/o OpenStack at all) for now, it should be quickly enough (~1-2 days) 16:47:09 and once we merge all required things for authentication and keystone trust mechanism we can create a simple workflow for backing VMs up 16:47:30 it should be pretty easy too 16:47:49 so, in other words, my suggestion is: 16:48:16 1. High priority: simple demo app with endpoints to demonstrate Mistral w/o OpenStack 16:48:46 2. Medium priority: simple workflow for backing up VMs on schedule 16:49:04 once we get auth done 16:49:09 sounds good for you? 16:49:53 Looks good because it's going to work both with and w/o OS, and will not require changes in engine and app, only an addition of data flow, right? 16:50:03 right 16:50:09 my point here is to start testing engine as soon as possible 16:50:10 I think we should demonstrate Mistral with OpenStack because Mistral is OpenStack project, not an some abstract workflow as service 16:50:34 we will, now worries 16:50:38 no worries 16:50:46 as we get these things done with auth 16:51:33 rakhmerov, ok, when we will have the first prototype which will deploy something? 16:51:35 no worries is much better than now worry:) 16:51:52 yes indeed :) 16:51:59 it just bothers me that we didn't start testing all main mechanisms together: REST API, client, Scheduler, engine 16:52:25 and that's why I think simple app w/o OpenStack would allow us to finally start doing it 16:52:46 I'm pretty sure we'll find a lot of problems once we start doing that 16:53:00 yes ) 16:53:28 but again, don't get me wrong: I want to demonstrate it with OpenStack too 16:54:09 Agree, this is the best way to go 16:54:54 Anything else in agenda? 16:54:58 #action NikolayM, get a simple demo app done by Wed w/o OpenStack 16:55:34 #action akuznetsov, finalize work related with authentication and keystone trusts 16:55:58 well, this is the main thing I wanted to bring up actually 16:56:11 and I'm glad we agreed on that 16:56:32 The audience is small for now, it helps sometimes:) 16:56:40 yep :) 16:56:51 the other thing I was going to talk about is the engine architecture itself but looks like we don't have time today 16:57:16 we can do it next time along with results we'll get done 16:57:19 Yep, let's postponed it for the future. For now you can drop the link here, if any 16:57:45 my bad, I think it's still not published on the wiki 16:58:06 shame on me (I'm too deep into development now) 16:58:37 #action rakhmerov, publish POC Mistral Engine architecture on wiki 16:58:54 ok, let's wrap up for today 16:59:00 thanks to everyone! 16:59:09 #endmeeting