04:00:23 #startmeeting masakari 04:00:24 Meeting started Tue Jun 27 04:00:23 2017 UTC and is due to finish in 60 minutes. The chair is samP. Information about MeetBot at http://wiki.debian.org/MeetBot. 04:00:25 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 04:00:27 The meeting name has been set to 'masakari' 04:00:40 Hi 04:00:43 hi 04:00:44 hi 04:01:11 abhishekk: First of all, congratulations for become a glance core 04:01:22 samP: thank you 04:01:39 abhishekk: congratulations! 04:01:58 sagara: thank you 04:02:27 OK, lets start 04:02:45 #topic Critical bugs 04:02:56 Any bugs to discuss? 04:03:46 #link: https://bugs.launchpad.net/masakari/+bug/1690995 04:03:48 Launchpad bug 1690995 in masakari "If masakari recover "resized" instance that's state was "stopped" before resizing, it will be "active"." [Undecided,In progress] - Assigned to Dinesh Bhor (dinesh-bhor) 04:04:50 According to the bug report if the instance is in stopped state before resize then after recovery it should be stopped. 04:05:32 Its looks like a nova bug to me.. 04:06:37 In this particular patch we have decided to stop the instance after recovery if it is in vm_state other than active and stopped : https://review.openstack.org/#/c/469029/ 04:07:04 samP: After resized operation and before confirm_resize, if the compute host goes down, in that case after evacuation the vm_state should be stopped instead of active as per bug reporter 04:07:10 Dinesh_Bhor: that is one possible solution.. 04:08:04 tpatil: yes..it should be on "stopped" 04:09:07 it should be stopped only in case the previous action on the instance is stopped before user calling resize operation 04:09:14 user stopped it for a reason, and make it active after evacuate could cause troubles. 04:09:45 tpatil: correct 04:09:53 samP:In that case, we will need to get instance_actions to take this decision 04:10:03 samP: thank you for explaining. Your explanation is right. 04:10:41 In future, if we decide to move the workflow to Mistral, we cannot call db_apis to get instance actions. Will need to add restFul apis to get instance actions 04:10:56 tpatil: I thought in previous discussion we agreed to use instance_actions to refer its previous status 04:11:20 tpatil: Ah...got it 04:12:33 tpatil: There is List Actions For Server API in nova. 04:12:34 samP: we have confirmed there is an existing nova api to get instance actions (instance_action_list) 04:12:36 #link https://developer.openstack.org/api-ref/compute/#servers-actions-servers-os-instance-actions 04:13:36 rkmrHonjo: I am looking at same 04:13:58 rkmrHonjo: Yes, the api is present 04:14:26 tpatil: can we use this API to get previous actions? 04:14:35 We may need to add this action in mistral to get the instance_actions for an instance 04:14:38 samP:Yes 04:15:31 This can be fixed in the same patch: https://review.openstack.org/#/c/469029/ I will fix it 04:15:31 tpatil: so, we do not need to call db_apis, right? 04:15:46 samP: correct 04:15:50 Dinesh_Bhor: thank you.. 04:16:01 tpatil: OK, thanks 04:17:56 However, regarding these cases, my initial concern stays same.. 04:18:44 which is, VMs are turn on instantly before we turn them off.. 04:19:35 IMO, we have to fix these in nova 04:20:44 samP: I agree. We need some internal state change (or something locking) API in Nova. 04:20:45 such as, in this case, evacuate does not turn the instance to active, if states before resize it stopped. 04:20:45 Ideally, user will expect the vm_state to be resized instead of stopped or active. 04:20:51 samP: First we need to find out the reason why nova is not allowing evacuatiuon if instance is in reset state 04:21:32 samP: we need to convince them, otherwise nova will never accept this fix if there is any strong reason 04:21:45 abhishekk: agree.. 04:21:46 s/reset/resize 04:24:02 can some one please check why nova does not allow this? 04:24:34 Then we could start to talk with nova, if we have strong reason. 04:25:28 I will do that, but if I cannot, I will ask someone to help me 04:25:45 sagara: sure..thank you.. 04:25:54 sagara: lets use ML with [masakari] 04:26:12 samP: so what we want to confirm with nova community is after evacuation the instance should be set to active or stopped state based on the previous action before resize 04:26:16 sagara: thank you. 04:26:51 #action: sagara find out the reason why nova is not allowing evacuatiuon if instance is in reset state 04:26:56 tpatil: yes 04:27:08 sagara: thank you. 04:27:43 OK then, any other bugs 04:28:23 if not, lets move to pike work items 04:28:32 #topic pike work items 04:29:00 samP: the nova is not allowing resize vm state to be evacuated before the instance files resding on the source compute node will not be accessible during evacuation 04:29:16 s/before/because 04:30:04 tpatil: but that does not true in shared storage case, right? 04:30:12 but in case of shared_storage, it should be allowed 04:31:26 tpatil: in non shared storage cases, it does make sense 04:31:26 samP: Can nova community allow evacuation in resize vm_state based on the shared_storage/non-shared storage, is the main question 04:32:47 samP: Let's discuss this point on the nova ML 04:32:54 tpatil: might hard to convince them, but lets give it a try 04:33:12 tpatil: sure 04:33:37 sagara: Do you want to send this mail or should I request Abhishek to send it 04:34:59 tpatil: Please send a mail to Nova ML. thanks. 04:35:06 sagara: OK 04:35:42 I will look up the reason from source or commit log. 04:35:55 Thanks.. 04:36:29 sagara: could you please share those in ML with [masakari], then abhishekk could use those info 04:36:58 samP: Yes, of course. 04:37:08 sagara: thanks 04:38:19 OK, about pike work items... need to do lot of review work.. 04:39:05 For recovery method customization, to add mistral driver in masakari we have analyzed some issues 04:39:18 1. need to add some actions in mistral, like filter vms on the basis of metadata, custom evacuation in which we need to find out aggregate host and add reserved_host to that aggregate, confirmation of evacuation, reset instances state to stop if instance is in state other than active or stop etc. 04:39:34 #link Pike work items https://etherpad.openstack.org/p/masakari-pike-workitems 04:39:51 2. Mistral workflow executes in asynchronous way and to get the output we have to continuously hit the "mistral execution-get " API 04:40:16 problem: If any workflow is executing and mistral-engine service goes down suddenly during execution and workflow remains in "RUNNING" state until the mistral-engine comes up again and processes the pending workflow (there must be some periodic task which is picking up the pending executions). 04:41:01 3. Assuming we have a mistral driver then in case of recovery_method as a "reserved_host" we are going to loop through the reserved_hosts and will pass the reserved_host one by one to workflow for execution. In this case if operator has configured "send e-mail" action after completion of workflow execution. 04:41:01 For partial workflow execution operator will get multiple emails that the workflow is failed and it has evacuated some instances out of total. 04:41:57 abhishekk: That is lot of Mistral work there... 04:42:22 For point 3 consider we have set reserved_host recovery action, 04:42:31 samP: yes 04:42:45 Is it possible to ask Mistral team to help up with this.. 04:43:05 samP: Right now, mistral has only one action servers_evacuate which takes instance id for evacuation 04:43:52 samP: but we will need to add a major action in mistral to evacuate instances from a failed compute host 04:43:52 tpatil: is that the evacuate work flow? 04:44:54 samP: for reference https://github.com/gryf/mistral-evacuate 04:45:19 abhishekk: ah.. that is not a part of Mistral 04:45:31 samP: yes 04:45:57 samP: yes, but we will need to work with Mistral community to add the required actions in mistral 04:46:20 we could define our own workflows for evacuation or start, stop etc.. 04:46:26 samP: we need to make major changes to this workflow to address our need, for example filter on the basis of metadata, add host to aggreagte, reset instance state to error etc 04:46:41 abhishekk: yes 04:47:09 Apart from making all these changes, there are few challenges 04:48:01 the problem is, does Mistral engine support those requirements? <- this is the part we should discuss with Mistral Team 04:48:27 Problem 1: As pointed above by Abhishek, if mistral_engine is down, the workflow will remain in running state forever 04:48:35 Problem 2 04:49:21 For reserved_host recovery_method, there is a possibility all instance might not be evacuated on a reserved_host 04:49:58 so we will need to call workflow again from masakari passing the next reserved host 04:50:54 in this particular case, if operator has configured email action, these email will be sent multiple times 04:51:59 tpatil: IMHO, problem1 is very important and can not be done without Mistral support 04:53:29 for Problem2, Mistral work flow could return value with "not done yet, need another reserved host" and then masakari could call it again with another reserved host. 04:54:08 samP: Problem 1: Yes, mistral_api should check mistral_engine is up or down and according return status to the caller 04:54:30 tpatil: agree 04:55:48 samP: problem 2: that's what is in my mind too, but then we may need to add some conditions to execute partial actions from the workflow on the mistral side for the subsequent evacuation of the remaining instances 04:56:12 samP: that's anyway ,we will need in masakari specs for discussion 04:56:44 samP: that's anyway ,we will need to add in masakari specs for discussion 04:57:00 tpatil: agree 04:57:50 I think we should use ML for this. 04:57:58 + masakari spec 04:58:23 we could get Mistral community feed back for this in ML 04:58:37 samP: First, we will upload a new PS in masakari-specs and then start this discussion on the ML if required 04:58:49 tpatil: sure.. 04:59:20 samP: Thanks 04:59:27 sounds good! 04:59:35 we only have 1min left 04:59:57 please continue the discussion on #openstack-masakari or ML with [masakari] 05:00:08 thank you all.... 05:00:15 #endmeeting