16:00:08 #startmeeting Mistral 16:00:09 Meeting started Mon Aug 25 16:00:08 2014 UTC and is due to finish in 60 minutes. The chair is rakhmerov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:10 hi 16:00:10 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:13 The meeting name has been set to 'mistral' 16:00:28 hi! 16:00:29 hey! 16:01:16 ok, let's start 16:01:24 hi 16:01:28 hi 16:01:31 hi guys 16:01:44 #topic Review Action Items 16:01:53 rakhmerov, skype on Tue with Nastya about integration testing scenarious 16:02:05 done (more detailed about this later 16:02:19 rakhmerov, rename the current "linear" workflow type to "direct" and discuss with the team what other workflow types may be needed 16:02:23 partially done 16:02:41 I've done renaming but didn't discuss about other workflow types yet 16:02:43 Hi 16:02:48 hi Dmitri 16:03:15 btw, folks from Fuel want a new type of workflow where tasks' order is based on priorities 16:03:52 #action rakhmerov and others: discuss workflow other types 16:04:02 rakhmerov, nmakhotkin: when doing renamings make sure to rename "finish" to "complete 16:04:12 not done yet but we keep it in mind 16:04:27 #topic Current Status 16:04:39 let's report everyone's status quickly 16:05:53 I've almost done with delayed calls (it is for further needed for running delayed tasks), 16:06:18 my status: 1. Made a mixed workflow work (including action execution and calling one workflow from another as a subflow), added workflow as an individual entity, done aforementioned renamind and a bunch of small fixes, also discussed API v2 with lots of people 16:06:19 Hi 16:06:33 hi Bryan! 16:06:39 Sorry a few min late.. 16:06:41 Just started today. Skimmed through the code and current reviews to figure out what we might need for API v2. 16:06:44 and also almost done actions in DB, only issue with tests 16:06:44 np 16:07:08 ok, cool 16:07:09 Most part of time I worked with other project, so I am still working on new test scenarios, want to finish it on this week and close appropriate blueprint 16:07:28 ok 16:07:55 akuznetsova, pretty soon we'll need to start working on integration tests for v2 16:08:20 I assume we'll be able to copy and adjust a lot of existing tests from v1 16:08:32 rakhmerov, ok, I will be ready) 16:08:49 I know you will ) 16:08:53 thanks 16:09:12 bhavenst, can you please report your status briefly? 16:09:51 Yeah sure, I'll try to update the bug and BP I'm working today on after the latest comments 16:10:02 yeah 16:10:20 it's kind of a weird thing with that FlushError 16:10:46 we fetched this commit today with Nikolay and rollback test fails for us locally 16:11:11 Yes, very weird. 16:11:11 throwing DBError which is actually logical if we look at the code 16:11:24 but for some reason jenkins say "ok" :) 16:11:29 Right, that's what I had for some time, but then it started failing for me, throwing the FlushError 16:11:47 other than that the patch looks good (just small cosmetic suggestions) 16:11:58 hm 16:12:03 very weird 16:12:26 I would make sure first of all that you have all the dependencies as they're listed in requirements 16:12:32 I'll play with it some more. I could probably wrap whatever exception comes up at the mistral api layer anyway 16:12:37 may be you have different versions or something.. 16:12:46 I had to change just a bit to get it to work 16:12:54 (requirements) 16:12:54 yeah, look at my comments, I created a bug for this 16:13:18 ooh, that could be the reason actually ) 16:13:26 ok, please take a look at this 16:13:51 OK, I'll revert those files and see if I can get tox to work properly 16:13:59 yeah 16:13:59 Lots of pain due to corp proxy 16:14:13 for now, I'm ok to have FlushError there in except clause 16:14:21 and get this patch merged in 16:14:27 OK sounds good 16:14:28 but then we need to fix that bug 16:14:40 the way how we handle exceptions in DB layer now is wrong 16:14:52 because we only catch some of the exceptions 16:15:24 so we need to clearly understand what exception hierarchy is and make a right decision how to handle them 16:15:25 ok 16:15:31 let's move on 16:15:33 Yeah that would be good 16:16:14 as far as the other patch, I'm really sorry, wasn't able to look at it today 16:16:17 will do tomorrow 16:16:38 no problem, I got the comment that on-success and requires shouldn't be combined, so trying to make a test workflow covering what I need without mixing 16:16:53 ok 16:16:57 #topic API V2 Discussion (integration with Glance, versioned client etc.) 16:17:16 ok, I have two questions to discuss here 16:17:53 Angus left a comment related to Glance integration in https://docs.google.com/a/mirantis.com/document/d/12j66DZiyJahdnV8zXM_fCRJb0qxpsVS_eyhZvdHlIVU/edit 16:18:29 basically it's about how we store workbooks 16:19:27 one of the ideas was to use Glance as a repository for Mistral artifacts (previously workbooks only, now it's gonna be separate entities: workflows, actions, triggers) 16:20:05 so the question is: how do we design API correctly to leave a possibility to get integrated with Glance? 16:21:02 for now, I really don't want to do it (in 0.1) but at least we need to understand what we need to do later on and have our design compatible with it 16:21:43 Angus mentioned using glance_id to identify workbooks in glance 16:22:20 I would paraphrase this question differently, does anyone in Mistral team has any knowledge of Glance? 16:22:38 would it be enough just to have field 'glance_id' in all our high-level entities like workflows to be able to fetch the source from Glance if we want to? 16:22:48 :)) 16:22:57 Nikolay should 16:22:59 I guess 16:23:29 Will it couple our API with Glancr 16:24:11 what about possibility to use mistral without openstack? will it be possible in this case? 16:24:13 Maybe a more generic reference type field that can contain glance (or whatever) references 16:24:45 dzimine, I don't want Mistral to be able to use Glance only 16:25:05 akuznetsova, yes, it should be possible 16:25:34 bhavenst, can you elaborate on this please? 16:26:01 uri kind of thing? 16:26:10 Well, just trying to reduce the coupling. By putting glance_id it kind of binds to glance 16:26:26 Our currency is the DSL of actions workflows and triggers, where a user stores it is not Mistral business I think 16:26:27 say, "glance: 1-2-3-4" or "mistral: my_workflow" 16:26:28 storage locator reference or something 16:26:37 yeah exactly 16:26:45 Glance, S3 shouldn't matter 16:27:34 bhavenst, looks interesting 16:28:10 dzimine, yes, but eventually Mistral server should be able to fetch workflow definition from somewhere 16:28:21 in order to start a workflow 16:28:46 Where is the glance integration requirement is coming? May be the use case will help reason about it 16:29:01 the advantage of Glance is that you can provide an ID instead of a URL, so you know it's safe to fetch (because you choose the endpoint yourself) 16:29:21 hi zaneb ) 16:29:34 o/ :) 16:30:06 I see 16:30:23 so in Heat we give you the option of passing a whole template or a URL, which is a mistake 16:30:44 in the future, we want it to be a whole template or a glance artifact catalog id 16:31:04 ok 16:31:09 Ok 16:31:52 zaneb, how do you see to specify/configure this? 16:32:00 I'm not sure I clearly see at this point 16:32:11 say, you decided to use Glance only 16:32:22 then you decided that you don't want to use Glance 16:32:24 also, you can use the user's token to fetch it, so they don't have to publish the template publicly 16:32:31 how would the API change? 16:33:12 right now (speaking for Heat) the API accepts either an inline template file or a URL with which to fetch one 16:33:20 for any operations that require a template 16:33:43 the URL fetching part of that is bad 16:33:45 so we're supposed to have two mutually exclusive properties, right? 16:33:54 in future we want to fetch only from Glance 16:33:58 rakhmerov: exactly 16:34:04 and raise an error if they're specified both 16:34:05 ok 16:34:07 makes sense 16:35:04 although what Angus is suggesting may be something different 16:35:28 like, not upload workbooks at all to Mistral, and have triggers refer to workbooks in Glance 16:35:43 * zaneb can't actually read the Google doc 16:35:45 yep, that's clear 16:36:33 zane what's your email? 16:37:02 what Angus is suggesting looks ok to me 16:37:59 so, I would suggest for now that we have this property 'glance_id' so that we could Glance integration moving forward 16:38:24 and we'll be able to identify our artefacts either by id/name or by glance id 16:38:52 and let's throw it in the ML 16:39:07 #action rakhmerov, start a ML discussion about Glance integration 16:39:08 do we have a suggestion described somewhere I can’t see details in the document, only basic idea. 16:39:21 no, that's all we have 16:39:31 If Angus can sum it up… 16:39:43 I can talk to Angus tomorrow (it's a late night for him now) 16:39:48 sure 16:39:59 I'll try to catch him tomorrow the first thing 16:40:12 #rakhmerov catch Angus to talk about Glance integration 16:41:05 the next question regarding new API is about how we're planning to alter the client 16:41:12 Python API and CLI 16:41:25 alternatives that I heard from people: 16:41:51 1. Honestly support two versions and switch between them based on the environment var 16:42:00 both python API and CLI 16:42:19 2. Don't support older versions in CLI for simplicity 16:42:38 so that CLI always works with the latest API version 16:42:40 I wonder: do we have the API clear and final? once we do we can put off the CLI and Python client till next week, when we have API ready. 16:42:41 opinions? 16:42:56 Just want to be sure we close on all API today, and get the work going. 16:43:43 dzimine, no, we don't have a final version on paper but I'm sure we understand 90% of what should be done 16:44:08 90% of shared understanding is good (_shared_ :)) 16:44:15 :) 16:44:29 enykeev, btw it may be a good thing to describe the API in a more or less formal way first 16:44:32 re CLI - this is easy, support only new is reasonable. 16:44:38 and throw it in the ML 16:44:39 Python client is a diff story, may be. 16:44:47 because Solum is using Python client. 16:44:53 yes 16:44:54 but 16:45:02 Solum also uses CLI :) 16:45:02 Let’s think throuhg our transition plan for Solum. 16:45:04 actively 16:45:19 I wonder how it will be if we support only latest API in CLI 16:45:53 the solution here may be just to have back support (not sure about the right term for this) 16:45:54 e.g., they can continue using older Client version which works against an older API 16:46:20 so that if they have a problem with the older version we can make a minor release of the old client 16:46:33 My off-the top of the head thinking: 16:46:40 ok 16:47:08 API compatibility is for clients, to not break clients using older version of API. And give new clienst new functionality. 16:47:12 rakhmerov, our current version of CLI has a lot of bugs that will be fixed in new one, so i think it is good idea to support new version 16:47:14 Python Client and CLI are clients 16:47:22 => API can treat them just the saem. 16:47:38 but are there any established practicies within Openstack in this area? 16:48:43 some projects support all, some just the latest 16:48:57 dzimine, nova, heat and glance all support at least two versions in their python clients, but as far as I can tell, glance only supports latest API in CLI 16:49:01 Kirill, you looked at this today, can you give an info? 16:49:02 ok 16:49:28 and they're all core projects, hm... 16:49:35 I haven't dig too deep though... 16:49:36 with different approach to this 16:49:39 ok 16:49:58 akuznetsova, yes, good concern 16:50:14 ok, so I think we need to talk to Angus again 16:50:27 but generally I'd be ok to support the latest version only 16:50:32 looks reasonable 16:50:34 +1 16:50:45 #action rakhmerov, talk to Angus about versioning CLI 16:51:47 #topic New action design discussion (storing actions in DB etc.) 16:51:48 renat are you clear on how we spin a minor version of an old client, if the new client goes forward? 16:52:25 dzimine, I've never done this before. Need to talk to our folks to find out 16:52:31 I believe we can do it 16:53:15 so Nikolay, not sure we have enough time to discuss your today changes in actions 16:53:21 let’s validate we can, than we decide finally. 16:53:23 but at least let's start 16:53:32 dzimine, ok, sure 16:54:02 #action rakhmerov, validate that we can release a minor version for the old client of the new one goes forward 16:54:32 Nikolay, do you have any concerns that you'd like to discuss here? 16:55:29 one of the things that I'd like to let you know (especially Dmitri) that we've come to a conclusion that we'll have to run a script manually for now to sync actions in db 16:55:55 ok, 16:56:04 and the simplest and the most reliable way to do that is just to wipe out all system actions and then recreate them 16:56:07 yep 16:56:08 what sync_db() method in action factory is doing than? 16:56:18 yes 16:56:27 but imagine we have 10 engine instance ) 16:56:47 then there's no need to call this method on each node start 16:57:44 either we need that nodes communicate about this somehow (say some of them become a master node) or just simplify our life for now and run it manually 16:57:49 with a script 16:57:57 I think it’s an OK first step, it may be ok for a long time actually. 16:58:01 which could be a part of installation process for Mistral actually 16:58:07 ok 16:58:41 we also talked to Nikolay about what component (API, engine or executor) should be responsible for this 16:58:45 I think engine only 16:58:52 execution may not even have access to be 16:59:13 and this is not an API's responsibility apparently 16:59:24 ok, time to shut the meeting down 16:59:38 thanks guys to all for coming! 16:59:41 thanks! 16:59:46 thanks 16:59:47 it was a good meeting 16:59:49 chees. 16:59:51 see ya 16:59:55 thanks 16:59:58 #endmeeting