16:01:23 <rakhmerov> #startmeeting Mistral
16:01:24 <openstack> Meeting started Mon Dec  9 16:01:23 2013 UTC and is due to finish in 60 minutes.  The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:25 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:27 <openstack> The meeting name has been set to 'mistral'
16:01:30 <rakhmerov> hello everyone
16:01:37 <gokrokve> hi
16:01:39 <NikolayM> hi
16:01:44 <stanlagun> hi
16:02:25 <joel_c> hello
16:03:07 <rakhmerov> hey
16:03:42 <rakhmerov> here's the proposed agenda for today's meeting: https://wiki.openstack.org/wiki/Meetings/MistralAgenda
16:03:50 <akuznetsov> Hi
16:05:19 <rakhmerov> additionally, I would add a topic "Detailed plan for PoC demo app" to that agenda. I was going to do that but looks like I forgot, sorry
16:05:38 <rakhmerov> so let's start with previous action items
16:05:43 <gokrokve> That is fine. Lets add this to the end
16:06:02 <rakhmerov> yes, 1 sec
16:06:07 <gokrokve> I meant PoC demo app.
16:06:46 <gokrokve> What is the list of AI?
16:07:04 <rakhmerov> done
16:07:05 <rakhmerov> ok
16:07:11 <gokrokve> joel_c: Hi Joel!
16:07:18 <rakhmerov> #topic previous AIs
16:07:27 <gokrokve> joel_c: Glad to see you here.
16:07:33 <rakhmerov> 1. nmakhotkin, rakhmerov Check once more what TaskFlow lacks to be able to use it in Mistral
16:07:49 <rakhmerov> So we have some new info on that
16:07:58 <gokrokve> #1 is it a doxument?
16:08:02 <gokrokve> #1 is it a document?
16:08:09 <rakhmerov> and that's pretty important
16:08:28 <gokrokve> We need to have this publically available to start converstaion with Joshua
16:09:24 <rakhmerov> we finally realized the most important thing that TaskFlow lacks now. It doens't support asynchronous tasks. Here's the link to the etherpad where we started discussing that: https://etherpad.openstack.org/p/async-taskflow-tasks
16:09:33 <rakhmerov> it's published
16:10:11 <gokrokve> Cool. Keep Joshua in the loop. We need a close collaboration here.
16:10:22 <rakhmerov> yes, sure
16:10:30 <gokrokve> Discuss with him what they\we can do.
16:10:40 <rakhmerov> we keep talking to them
16:10:44 <gokrokve> Submit  BPs for taskflow.
16:10:48 <gokrokve> Great!
16:11:34 <rakhmerov> #action rakhmerov, submit blueprints for TaskFlow about lacking functionality (i.e. asynchronous tasks)
16:12:31 <rakhmerov> so for now we've made a decision to create our own engine without TaskFlow, it will allow us to understand what we need from TaskFlow better
16:13:01 <joel_c> rakhmerov: a fork of taskflow?
16:13:09 <rakhmerov> it doesn't mean we're not going to use TaskFlow but at this point it just makes more sense
16:13:25 <rakhmerov> joel_c: not really a fork
16:13:44 <rakhmerov> well, actually not sure what you mean?
16:13:56 <gokrokve> joel_c: It will be a PoC for Mistral which will help to figure out requirements for taskflow
16:14:06 <rakhmerov> yes
16:14:11 <joel_c> gokrokve: rakhmerov: understood.
16:14:26 <gokrokve> Once we have a list of requirements we will with taskflow team to implement them
16:14:42 <gokrokve> We are working with taskflow team pretty closely
16:15:14 <gokrokve> We had a lot of meetings with them to define Mistral architecture and how taskflow fits to it
16:16:04 <rakhmerov> the main point here is that we first need to think about the architecture of Mistral as if we didn't know about TaskFlow existance. Then with knowledge of what we need we could provide more valueable requirements for TaskFlow so that we can apply it within Mistral
16:16:13 <rakhmerov> hope it makes sense
16:16:13 <gokrokve> There are some gaps in taskflow library which should be filled before we can use it. So our idea is to create a simple engine for Mistral and based on it we will prepare a list of features we need from taskflow.
16:16:25 <rakhmerov> yes
16:16:40 <rakhmerov> it's jut more clear experiment if you will
16:17:17 <joel_c> thanks for the clarification
16:17:22 <rakhmerov> otherwise we will end up creating workarounds for using TaskFlow rather than concentrating on Mistral architecture
16:17:47 <rakhmerov> but again, we'll try to contribute our ideas into TaskFlow and eventually use it at some point
16:18:34 <rakhmerov> so now we're prototyping the engine implementation without TaskFlow and we've almost done the initial version
16:18:58 <rakhmerov> NikolayM, is it going to be committed today?
16:20:30 <rakhmerov> ok, not sure if he's here but he was planning to do that at least
16:20:34 <rakhmerov> let's move on
16:20:36 <NikolayM> yes, I just finish general ideas
16:20:43 <rakhmerov> ok, cool
16:21:28 <rakhmerov> after that we're planning to look at it, play with it and think how we can further address HA and scalability requirements
16:21:47 <rakhmerov> we also had the second action item
16:21:52 <gokrokve> Renat, do we have a scope of PoC defined? What use cases will be covered?
16:22:03 <gokrokve> Ok. Go with 2nd AI
16:22:19 <rakhmerov> akuznetsov, nmakhotkin, rakhmerov Create a detailed plan for the demo application for PoC
16:22:33 <rakhmerov> which is what I just included into the agenda
16:22:47 <rakhmerov> last time we decided to devote this meeting to exactly that topic
16:23:08 <gokrokve> What is demo application?
16:23:28 <rakhmerov> so as you may guess, it's not done yet even though we have some thoughts
16:23:38 <gokrokve> I wonder if we can have scheduled event use case covered.
16:23:59 <gokrokve> I believe, that use case was interesting for Joel.
16:24:12 <gokrokve> For scheduled backups for example.
16:24:13 <rakhmerov> gokrove: yes, we have a bluprint mistral-poc, it pretty much describes what PoC is going to address
16:24:21 <gokrokve> Link?
16:24:26 <rakhmerov> yes, it will be covered
16:24:30 <rakhmerov> ok, sec
16:24:47 <rakhmerov> https://blueprints.launchpad.net/mistral/+spec/mistral-poc
16:24:59 <rakhmerov> it's not really detailed, sorry
16:25:30 <rakhmerov> additionally you may want to look at https://etherpad.openstack.org/p/MistralAPISpecification and https://etherpad.openstack.org/p/MistralDSLSpecification
16:25:49 <rakhmerov> to better understand what's going to be implemented in PoC
16:26:06 <rakhmerov> but yes, this use case will be covered
16:26:19 <rakhmerov> but just one note here
16:27:02 <rakhmerov> for PoC we are planning to implement only simple actions like "sending HTTP requests"
16:27:38 <rakhmerov> so scheduling backups will be possible only if we represent it with a set of REST or AMQP calls
16:28:15 <rakhmerov> joel_c: if you have any questions, please ask, we'll be happy to clarify them
16:29:01 <joel_c> rakhmerov: will do.  so far so good. ;)
16:29:24 <rakhmerov> ok :)
16:29:30 <rakhmerov> later on we're going implement more capabilities in Mistral like convenient callback to the applications so that we could implement some actions within python code itself
16:30:19 <rakhmerov> ok, let's task what Demo App is going to do with PoC
16:30:42 <rakhmerov> #topic Detailed plan for Demo App to demonstrate PoC capabilities
16:31:03 <rakhmerov> not sure if I answered your the question what Demo App is
16:32:05 <gokrokve> By the way, we can evaluate an interface of salt minion
16:32:15 <gokrokve> This is an agent of salt stack.
16:32:22 <rakhmerov> so Demo App is an application (may be just a python script) that will be uploading a workflow into Mistral and tell Mistral run a workflow using either the corresponding REST API call or just defining periodic events within a workflow definition itself (using DSL)
16:32:27 <gokrokve> Probably we can send some hooks to it.
16:32:35 <rakhmerov> ok
16:32:44 <gokrokve> It is zeroMQ based.
16:33:01 <rakhmerov> #action Evaluate an interface of salt minion
16:33:18 <gokrokve> There is an e-mail thread about unified agent.
16:33:33 <gokrokve> Salt minion was mentioned as potential basis.
16:33:47 <rakhmerov> Demo App in our case may just build a workflow definition taking some input parameters (like how many VMs this workflow should create)
16:34:07 <rakhmerov> gokrokve: got it, thanks, we'll look at it
16:35:44 <rakhmerov> Initially we were planning create a Demo App that would build a workflow like "Create several VMs" -> "Create Cinder Volumes" -> "Attach Cinder Volumes" -> "Format Volumes"
16:36:20 <rakhmerov> so the app itself seems to be easy to implement but I personally have a couple of concerns here
16:36:54 <rakhmerov> are we going to be able to perform all these actions like "Create VMs" using just REST calls?
16:37:04 <rakhmerov> any comments on this guys?
16:37:33 <rakhmerov> or you can also suggest a different plan for Demo App
16:38:41 <rakhmerov> it is something we need to decide quickly :)
16:38:50 <joel_c> gokrokve: were you suggesting that the demo would also integrate through to an agent within the VM representing application deployment/configuration?
16:39:08 <joel_c> gokrokve: and the agent would be based on salt?
16:39:24 <rakhmerov> no, honestly we were not
16:39:40 <gokrokve> rakhmerov: Actually yes.
16:40:22 <rakhmerov> it can be made later but I believe we've not decided to implement this in PoC
16:40:23 <joel_c> it would be great if that was achieved
16:40:23 <gokrokve> I think that might be useful as PoC can be used not only for demo but for some real use cases if it works with agent on VM side
16:40:49 <joel_c> agreed.  this USE case is valuable.
16:40:55 <gokrokve> joel_c: Yes. Do you want to try PoC code in your lab for your use cases?
16:41:13 <rakhmerov> joel_c: Yes
16:41:28 <joel_c> Yes, this will play well into custom collection(stack)/application deployments
16:42:09 <gokrokve> joel_c: Cool. Do you have any specific requirements to use Mistral?
16:42:38 <gokrokve> It will not be a production code, so you can play only in a lab with it.
16:43:09 <joel_c> nothing specific at this time however if the PoC included what was just discussed it would be enough to demonstrate clearly how it can be evolved into something for production
16:43:44 <gokrokve> joel_c: Great. We will definetely include this to PoC.
16:44:32 <rakhmerov> as far as VM agents, I totally agree about communicating with VM agents but I'm pretty sure we won't be able to achive that this week :) However, given this new info we'll try to implement it quickly
16:44:56 <gokrokve> rakhmerov: Not this week.
16:45:19 <gokrokve> rakhmerov: for this week we just need a base engine for Mistral.
16:45:23 <rakhmerov> well, if you're talking about PoC it's supposed to be finished on Friday
16:45:32 <gokrokve> PoC for engine.
16:45:35 <gokrokve> Yes.
16:45:45 <rakhmerov> not sure I understand you
16:46:00 <gokrokve> Integration with agent can be done later. Let say nex week.
16:46:04 <gokrokve> Integration with agent can be done later. Let say next week.
16:46:25 <rakhmerov> this Friday is a deadline for Mistral PoC (REST API, Scheduler, Engine)
16:46:35 <gokrokve> So, do not change a scope of work for this week, but plan next week activities with agent.
16:46:40 <rakhmerov> yep, next week I think we can do it
16:46:47 <gokrokve> rakhmerov: Good.
16:46:57 <rakhmerov> definitely, ok
16:48:13 <rakhmerov> #action Work on joel_c requirement about communication with VM agent
16:50:54 <rakhmerov> So I guess we talked on Demo App for PoC and understand what we're planning to do
16:51:05 <rakhmerov> I was going to discuss one more thing
16:51:11 <rakhmerov> #topic Discuss Mistral Engine & requirements to TaskFlow
16:51:16 <igormarnat> rakhmerov: you might want also to involve Dmitry into the discussions of VM agent - he seems to be working on unified agent for several projects, getting the reqs and elaborating blueprint
16:51:32 <rakhmerov> ok
16:51:41 <rakhmerov> Dmitry Mesheryakov?
16:51:51 <akuznetsov> yes
16:51:58 <rakhmerov> ok thanks
16:53:24 <rakhmerov> as far as requirements to TaskFlow - we've already talked about it and and it is in progress
16:54:42 <rakhmerov> #action Arrange a meeting with Dmitry Mesheryakov about Unified Agent (include Stan and others)
16:55:40 <rakhmerov> looks like we don't have much time to discuss Mistral Engine now
16:57:08 <rakhmerov> basically now it's going to schedule workflow executions in separate threads without distributing anything
16:57:38 <rakhmerov> of course, it's not going to be neither fault tolerant nor scalable
16:58:27 <rakhmerov> so my main concern here is to propose an architecture to address these requirements
16:59:13 <rakhmerov> so that Mistral would be stateless and we could scale it easily
16:59:26 <rakhmerov> I suggest we discuss it on the next meeting
16:59:40 <rakhmerov> so, let's finish now
16:59:47 <rakhmerov> thanks to everyone!
16:59:49 <rakhmerov> bye
16:59:58 <rakhmerov> #endmeeting