16:00:08 <rakhmerov> #startmeeting Mistral
16:00:09 <openstack> Meeting started Mon Aug 25 16:00:08 2014 UTC and is due to finish in 60 minutes.  The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:10 <rakhmerov> hi
16:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:13 <openstack> The meeting name has been set to 'mistral'
16:00:28 <NikolayM> hi!
16:00:29 <enykeev> hey!
16:01:16 <rakhmerov> ok, let's start
16:01:24 <m4dcoder> hi
16:01:28 <akuznetsova> hi
16:01:31 <rakhmerov> hi guys
16:01:44 <rakhmerov> #topic Review Action Items
16:01:53 <rakhmerov> rakhmerov, skype on Tue with Nastya about integration testing scenarious
16:02:05 <rakhmerov> done (more detailed about this later
16:02:19 <rakhmerov> rakhmerov, rename the current "linear" workflow type to "direct" and discuss with the team what other workflow types may be needed
16:02:23 <rakhmerov> partially done
16:02:41 <rakhmerov> I've done renaming but didn't discuss about other workflow types yet
16:02:43 <dzimine> Hi
16:02:48 <rakhmerov> hi Dmitri
16:03:15 <rakhmerov> btw, folks from Fuel want a new type of workflow where tasks' order is based on priorities
16:03:52 <rakhmerov> #action rakhmerov and others: discuss workflow other types
16:04:02 <rakhmerov> rakhmerov, nmakhotkin: when doing renamings make sure to rename "finish" to "complete
16:04:12 <rakhmerov> not done yet but we keep it in mind
16:04:27 <rakhmerov> #topic Current Status
16:04:39 <rakhmerov> let's report everyone's status quickly
16:05:53 <NikolayM> I've almost done with delayed calls (it is for further needed for running delayed tasks),
16:06:18 <rakhmerov> my status: 1. Made a mixed workflow work (including action execution and calling one workflow from another as a subflow), added workflow as an individual entity, done aforementioned renamind and a bunch of small fixes, also discussed API v2 with lots of people
16:06:19 <bhavenst> Hi
16:06:33 <rakhmerov> hi Bryan!
16:06:39 <bhavenst> Sorry a few min late..
16:06:41 <enykeev> Just started today. Skimmed through the code and current reviews to figure out what we might need for API v2.
16:06:44 <NikolayM> and also almost done actions in DB, only issue with tests
16:06:44 <rakhmerov> np
16:07:08 <rakhmerov> ok, cool
16:07:09 <akuznetsova> Most part of time I worked with other project, so I am still working on new test scenarios, want to finish it on this week and close appropriate blueprint
16:07:28 <rakhmerov> ok
16:07:55 <rakhmerov> akuznetsova, pretty soon we'll need to start working on integration tests for v2
16:08:20 <rakhmerov> I assume we'll be able to copy and adjust a lot of existing tests from v1
16:08:32 <akuznetsova> rakhmerov, ok, I will be ready)
16:08:49 <rakhmerov> I know you will )
16:08:53 <rakhmerov> thanks
16:09:12 <rakhmerov> bhavenst, can you please report your status briefly?
16:09:51 <bhavenst> Yeah sure, I'll try to update the bug and BP I'm working today on after the latest comments
16:10:02 <rakhmerov> yeah
16:10:20 <rakhmerov> it's kind of a weird thing with that FlushError
16:10:46 <rakhmerov> we fetched this commit today with Nikolay and rollback test fails for us locally
16:11:11 <bhavenst> Yes, very weird.
16:11:11 <rakhmerov> throwing DBError which is actually logical if we look at the code
16:11:24 <rakhmerov> but for some reason jenkins say "ok" :)
16:11:29 <bhavenst> Right, that's what I had for some time, but then it started failing for me, throwing the FlushError
16:11:47 <rakhmerov> other than that the patch looks good (just small cosmetic suggestions)
16:11:58 <rakhmerov> hm
16:12:03 <rakhmerov> very weird
16:12:26 <rakhmerov> I would make sure first of all that you have all the dependencies as they're listed in requirements
16:12:32 <bhavenst> I'll play with it some more.  I could probably wrap whatever exception comes up at the mistral api layer anyway
16:12:37 <rakhmerov> may be you have different versions or something..
16:12:46 <bhavenst> I had to change just a bit to get it to work
16:12:54 <bhavenst> (requirements)
16:12:54 <rakhmerov> yeah, look at my comments, I created a bug for this
16:13:18 <rakhmerov> ooh, that could be the reason actually )
16:13:26 <rakhmerov> ok, please take a look at this
16:13:51 <bhavenst> OK, I'll revert those files and see if I can get tox to work properly
16:13:59 <rakhmerov> yeah
16:13:59 <bhavenst> Lots of pain due to corp proxy
16:14:13 <rakhmerov> for now, I'm ok to have FlushError there in except clause
16:14:21 <rakhmerov> and get this patch merged in
16:14:27 <bhavenst> OK sounds good
16:14:28 <rakhmerov> but then we need to fix that bug
16:14:40 <rakhmerov> the way how we handle exceptions in DB layer now is wrong
16:14:52 <rakhmerov> because we only catch some of the exceptions
16:15:24 <rakhmerov> so we need to clearly understand what exception hierarchy is and make a right decision how to handle them
16:15:25 <rakhmerov> ok
16:15:31 <rakhmerov> let's move on
16:15:33 <bhavenst> Yeah that would be good
16:16:14 <rakhmerov> as far as the other patch, I'm really sorry, wasn't able to look at it today
16:16:17 <rakhmerov> will do tomorrow
16:16:38 <bhavenst> no problem, I got the comment that on-success and requires shouldn't be combined, so trying to make a test workflow covering what I need without mixing
16:16:53 <rakhmerov> ok
16:16:57 <rakhmerov> #topic API V2 Discussion (integration with Glance, versioned client etc.)
16:17:16 <rakhmerov> ok, I have two questions to discuss here
16:17:53 <rakhmerov> Angus left a comment related to Glance integration in https://docs.google.com/a/mirantis.com/document/d/12j66DZiyJahdnV8zXM_fCRJb0qxpsVS_eyhZvdHlIVU/edit
16:18:29 <rakhmerov> basically it's about how we store workbooks
16:19:27 <rakhmerov> one of the ideas was to use Glance as a repository for Mistral artifacts (previously workbooks only, now it's gonna be separate entities: workflows, actions, triggers)
16:20:05 <rakhmerov> so the question is: how do we design API correctly to leave a possibility to get integrated with Glance?
16:21:02 <rakhmerov> for now, I really don't want to do it (in 0.1) but at least we need to understand what we need to do later on and have our design compatible with it
16:21:43 <rakhmerov> Angus mentioned using glance_id to identify workbooks in glance
16:22:20 <enykeev> I would paraphrase this question differently, does anyone in Mistral team has any knowledge of Glance?
16:22:38 <rakhmerov> would it be enough just to have field 'glance_id' in all our high-level entities like workflows to be able to fetch the source from Glance if we want to?
16:22:48 <rakhmerov> :))
16:22:57 <rakhmerov> Nikolay should
16:22:59 <rakhmerov> I guess
16:23:29 <dzimine> Will it couple our API with Glancr
16:24:11 <akuznetsova> what about  possibility to use mistral without openstack?  will it be possible in this case?
16:24:13 <bhavenst> Maybe a more generic reference type field that can contain glance (or whatever) references
16:24:45 <rakhmerov> dzimine, I don't want Mistral to be able to use Glance only
16:25:05 <rakhmerov> akuznetsova, yes, it should be possible
16:25:34 <rakhmerov> bhavenst, can you elaborate on this please?
16:26:01 <rakhmerov> uri kind of thing?
16:26:10 <bhavenst> Well, just trying to reduce the coupling.  By putting glance_id it kind of binds to glance
16:26:26 <dzimine> Our currency is the DSL of actions workflows and triggers, where a user stores it is not Mistral business I think
16:26:27 <rakhmerov> say, "glance: 1-2-3-4" or "mistral: my_workflow"
16:26:28 <bhavenst> storage locator reference or something
16:26:37 <bhavenst> yeah exactly
16:26:45 <dzimine> Glance, S3 shouldn't matter
16:27:34 <rakhmerov> bhavenst, looks interesting
16:28:10 <rakhmerov> dzimine, yes, but eventually Mistral server should be able to fetch workflow definition from somewhere
16:28:21 <rakhmerov> in order to start a workflow
16:28:46 <dzimine> Where is the glance integration requirement is coming? May be the use case will help reason about it
16:29:01 <zaneb> the advantage of Glance is that you can provide an ID instead of a URL, so you know it's safe to fetch (because you choose the endpoint yourself)
16:29:21 <rakhmerov> hi zaneb )
16:29:34 <zaneb> o/ :)
16:30:06 <rakhmerov> I see
16:30:23 <zaneb> so in Heat we give you the option of passing a whole template or a URL, which is a mistake
16:30:44 <zaneb> in the future, we want it to be a whole template or a glance artifact catalog id
16:31:04 <rakhmerov> ok
16:31:09 <dzimine> Ok
16:31:52 <rakhmerov> zaneb, how do you see to specify/configure this?
16:32:00 <rakhmerov> I'm not sure I clearly see at this point
16:32:11 <rakhmerov> say, you decided to use Glance only
16:32:22 <rakhmerov> then you decided that you don't want to use Glance
16:32:24 <zaneb> also, you can use the user's token to fetch it, so they don't have to publish the template publicly
16:32:31 <rakhmerov> how would the API change?
16:33:12 <zaneb> right now (speaking for Heat) the API accepts either an inline template file or a URL with which to fetch one
16:33:20 <zaneb> for any operations that require a template
16:33:43 <zaneb> the URL fetching part of that is bad
16:33:45 <rakhmerov> so we're supposed to have two mutually exclusive properties, right?
16:33:54 <zaneb> in future we want to fetch only from Glance
16:33:58 <zaneb> rakhmerov: exactly
16:34:04 <rakhmerov> and raise an error if they're specified both
16:34:05 <rakhmerov> ok
16:34:07 <rakhmerov> makes sense
16:35:04 <zaneb> although what Angus is suggesting may be something different
16:35:28 <zaneb> like, not upload workbooks at all to Mistral, and have triggers refer to workbooks in Glance
16:35:43 * zaneb can't actually read the Google doc
16:35:45 <rakhmerov> yep, that's clear
16:36:33 <rakhmerov> zane what's your email?
16:37:02 <rakhmerov> what Angus is suggesting looks ok to me
16:37:59 <rakhmerov> so, I would suggest for now that we have this property 'glance_id' so that we could Glance integration moving forward
16:38:24 <rakhmerov> and we'll be able to identify our artefacts either by id/name or by glance id
16:38:52 <rakhmerov> and let's throw it in the ML
16:39:07 <rakhmerov> #action rakhmerov, start a ML discussion about Glance integration
16:39:08 <dzimine> do we have a suggestion described somewhere I can’t see details in the document, only basic idea.
16:39:21 <rakhmerov> no, that's all we have
16:39:31 <dzimine> If Angus can sum it up…
16:39:43 <rakhmerov> I can talk to Angus tomorrow (it's a late night for him now)
16:39:48 <rakhmerov> sure
16:39:59 <rakhmerov> I'll try to catch him tomorrow the first thing
16:40:12 <rakhmerov> #rakhmerov catch Angus to talk about Glance integration
16:41:05 <rakhmerov> the next question regarding new API is about how we're planning to alter the client
16:41:12 <rakhmerov> Python API and CLI
16:41:25 <rakhmerov> alternatives that I heard from people:
16:41:51 <rakhmerov> 1. Honestly support two versions and switch between them based on the environment var
16:42:00 <rakhmerov> both python API and CLI
16:42:19 <rakhmerov> 2. Don't support older versions in CLI for simplicity
16:42:38 <rakhmerov> so that CLI always works with the latest API version
16:42:40 <dzimine> I wonder: do we have the API clear and final? once we do we can put off the CLI and Python client till next week, when we have API ready.
16:42:41 <rakhmerov> opinions?
16:42:56 <dzimine> Just want to be sure we close on all API today, and get the work going.
16:43:43 <rakhmerov> dzimine, no, we don't have a final version on paper but I'm sure we understand 90% of what should be done
16:44:08 <dzimine> 90% of shared understanding is good (_shared_ :))
16:44:15 <rakhmerov> :)
16:44:29 <rakhmerov> enykeev, btw it may be a good thing to describe the API in a more or less formal way first
16:44:32 <dzimine> re CLI - this is easy, support only new is reasonable.
16:44:38 <rakhmerov> and throw it in the ML
16:44:39 <dzimine> Python client is a diff story, may be.
16:44:47 <dzimine> because Solum is using Python client.
16:44:53 <rakhmerov> yes
16:44:54 <rakhmerov> but
16:45:02 <rakhmerov> Solum also uses CLI :)
16:45:02 <dzimine> Let’s think throuhg our transition plan for Solum.
16:45:04 <rakhmerov> actively
16:45:19 <rakhmerov> I wonder how it will be if we support only latest API in CLI
16:45:53 <rakhmerov> the solution here may be just to have back support (not sure about the right term for this)
16:45:54 <dzimine> e.g., they can continue using older Client version which works against an older API
16:46:20 <rakhmerov> so that if they have a problem with the older version we can make a minor release of the old client
16:46:33 <dzimine> My off-the top of the head thinking:
16:46:40 <rakhmerov> ok
16:47:08 <dzimine> API compatibility is for clients, to not break clients using older version of API. And give new clienst new functionality.
16:47:12 <akuznetsova> rakhmerov, our current version of CLI has a lot of bugs that will be fixed in new one, so i think it is good idea to support new version
16:47:14 <dzimine> Python Client and CLI are clients
16:47:22 <dzimine> => API can treat them just the saem.
16:47:38 <dzimine> but are there any established practicies within Openstack in this area?
16:48:43 <rakhmerov> some projects support all, some just the latest
16:48:57 <enykeev> dzimine, nova, heat and glance all support at least two versions in their python clients, but as far as I can tell, glance only supports latest API in CLI
16:49:01 <rakhmerov> Kirill, you looked at this today, can you give an info?
16:49:02 <rakhmerov> ok
16:49:28 <rakhmerov> and they're all core projects, hm...
16:49:35 <enykeev> I haven't dig too deep though...
16:49:36 <rakhmerov> with different approach to this
16:49:39 <rakhmerov> ok
16:49:58 <rakhmerov> akuznetsova, yes, good concern
16:50:14 <rakhmerov> ok, so I think we need to talk to Angus again
16:50:27 <rakhmerov> but generally I'd be ok to support the latest version only
16:50:32 <rakhmerov> looks reasonable
16:50:34 <enykeev> +1
16:50:45 <rakhmerov> #action rakhmerov, talk to Angus about versioning CLI
16:51:47 <rakhmerov> #topic New action design discussion (storing actions in DB etc.)
16:51:48 <dzimine> renat are you clear on how we spin a minor version of an old client, if the new client goes forward?
16:52:25 <rakhmerov> dzimine, I've never done this before. Need to talk to our folks to find out
16:52:31 <rakhmerov> I believe we can do it
16:53:15 <rakhmerov> so Nikolay, not sure we have enough time to discuss your today changes in actions
16:53:21 <dzimine> let’s validate we can, than we decide finally.
16:53:23 <rakhmerov> but at least let's start
16:53:32 <rakhmerov> dzimine, ok, sure
16:54:02 <rakhmerov> #action rakhmerov, validate that we can release a minor version for the old client of the new one goes forward
16:54:32 <rakhmerov> Nikolay, do you have any concerns that you'd like to discuss here?
16:55:29 <rakhmerov> one of the things that I'd like to let you know (especially Dmitri) that we've come to a conclusion that we'll have to run a script manually for now to sync actions in db
16:55:55 <dzimine> ok,
16:56:04 <rakhmerov> and the simplest and the most reliable way to do that is just to wipe out all system actions and then recreate them
16:56:07 <rakhmerov> yep
16:56:08 <dzimine> what sync_db() method in action factory is doing than?
16:56:18 <rakhmerov> yes
16:56:27 <rakhmerov> but imagine we have 10 engine instance )
16:56:47 <rakhmerov> then there's no need to call this method on each node start
16:57:44 <rakhmerov> either we need that nodes communicate about this somehow (say some of them become a master node) or just simplify our life for now and run it manually
16:57:49 <rakhmerov> with a script
16:57:57 <dzimine> I think it’s an OK first step, it may be ok for a long time actually.
16:58:01 <rakhmerov> which could be a part of installation process for Mistral actually
16:58:07 <rakhmerov> ok
16:58:41 <rakhmerov> we also talked to Nikolay about what component (API, engine or executor) should be responsible for this
16:58:45 <rakhmerov> I think engine only
16:58:52 <rakhmerov> execution may not even have access to be
16:59:13 <rakhmerov> and this is not an API's responsibility apparently
16:59:24 <rakhmerov> ok, time to shut the meeting down
16:59:38 <rakhmerov> thanks guys to all for coming!
16:59:41 <enykeev> thanks!
16:59:46 <bhavenst> thanks
16:59:47 <rakhmerov> it was a good meeting
16:59:49 <dzimine> chees.
16:59:51 <rakhmerov> see ya
16:59:55 <akuznetsova> thanks
16:59:58 <rakhmerov> #endmeeting