15:00:51 #startmeeting manila 15:00:55 Meeting started Thu Mar 30 15:00:51 2017 UTC and is due to finish in 60 minutes. The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:57 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:00 The meeting name has been set to 'manila' 15:01:01 hello all 15:01:03 hi 15:01:04 \o 15:01:05 hello 15:01:06 hi 15:01:07 hello 15:01:08 hello 15:01:08 hi 15:01:15 hi 15:01:20 o/ 15:01:41 #agenda https://wiki.openstack.org/wiki/Manila/Meetings 15:02:13 I don't have any announcements today but I wanted to remind you all that Pike-1 is 2 weeks away 15:02:36 I'm reviewing specs today 15:02:43 I know several of them are ready and just waiting 15:02:52 good news 15:03:05 #topic ensure share 15:03:11 #link https://review.openstack.org/#/c/446494 15:03:14 hi 15:03:21 zhongjun_: you're up 15:03:32 bswartz: thanks, According to your advice, I add two methods to slove ensure share problem in spec. 15:03:35 hi 15:03:40 o/ 15:04:05 bswartz: I am not sure which one is better for manila? 15:04:18 o/ 15:04:57 zhongjun_: based on our discussion earlier, I don't think we should be prioritizing things such as avoiding restarting m-shr with this feature 15:05:17 avoiding m-shr restarts _is_ a valuable goal, but it's unrelated to ensure_share IMO 15:05:38 to add some context: one method is automatic detection/flag of shares that need to run ensure, and the other is using CLI to do so when admin wants 15:06:00 ensure_share is about being as smart as we can about handling issues that might occur when we do restart the service 15:06:36 in particular dealing with the effects of software upgrades 15:06:56 either the manila software itself or the backend software/configuration 15:07:49 ok, We could open another patch to sovle the things that we don't need to restart 15:08:06 if there are enhancements we can make that allow is to "fix" some things without restarting which currently do require a restart, then I'd like to see that proposed separately 15:08:34 I don't know if others have different opinions -- that's how I think about it 15:10:09 zhongjun_: based on my reading of the current version of the spec, I think we might need to split some content out of this spec into a separate one 15:10:47 the biggest problem with the current ensure_share is that it was misunderstood and new driver developers were unclear about what they should do with it 15:10:57 yeah 15:11:22 bswartz: but forget about without restarting, we still have two methods. I still need to concider all problems. 15:11:25 I want to stick to the model where it's called once on driver startup for some subset of the shares on that backend (possibly all, possibly none) 15:11:30 the on_restart() concept seems clear, but we'll need to make sure it looks clear to a newbie in the end 15:12:00 the problem today is that it's always all shares, and drivers do whatever they want there 15:12:02 what subset? 15:12:10 markstur: that's what we need to define 15:12:18 bswartz: for that the share needs to be flagged as needing to be "ensured" 15:12:31 ahh 15:12:34 ganso: I think we can do better than that 15:12:42 who does the flagging? 15:13:03 bswartz: zhongjun_ has two methods to perform that flagging: on demand CLI and automatic 15:13:43 I'd like to focus on the automatic 15:14:03 on-demand maintenance is fine, but it's objectively worse than automatic 15:14:29 in a cloud everything should be automatic 15:14:44 ganso: I think maybe ben don't want a new CLI, right? 15:14:48 if manila already knows which shares "need attention" to be sent to a driver, then why wait for restart 15:14:58 zhongjun_: it is not preferable 15:15:20 markstur is right 15:15:29 I think a CLI makes sense for when it is good for an admin to decide when to take the action 15:15:36 get it know automatically is the problem 15:15:37 markstur: the point is that determining which ones need attention might itself be an expensive operation 15:15:48 but I like the push to try to sort out other stuff first and leave the CLI as a separate tool 15:15:54 so we only do that work on startup 15:16:38 or more likely, it's something that's not too expensive, but very unlikely to change except at times when there are restarts 15:16:47 again I'm thinking in particular about software upgrades 15:16:53 bswartz: makes sense. I was thinking from a driver startup perspective and needed to think more of a manila restart process too 15:16:59 you can't upgrade your manila software without restarting m-shr 15:17:08 maybe we could introduce the CLI first, and mark it deprecated when we already have the tested automatically features 15:17:10 looks like we need both: sometimes, we need to set this "needs-attention" flag during a database migration, and we'd need to let the administrator set it 15:17:35 and in most cases, upgrading the software on your backend will involve downtime and/or a restart of m-shr so the driver can recognize the software upgrade 15:17:35 bswartz: If we tag share, the admin still seems so busy? 15:18:02 I've seen where just a driver change (and the version in stats) can easily do some clever things on a driver upgrade with the current ensure 15:18:26 The same thing on a core manila restart also has the possibility of dealing with upgrade stuff 15:18:32 gouthamr: we need to be careful to not run in a case where the problem that needs attention needs admin intervention (like direct update to DB entry) to be fixed, on such cases tagging the share may not be useful at all 15:19:36 ganso: true... 15:19:57 let me spell out the kind of use case I worry about 15:20:45 suppose a vendor improves the software on their storage controller to address a limitation with manila -- they add some feature which was missing 15:21:02 next they update their manila driver to take advantage of the new feature 15:21:28 however the driver changes require storing some additional metadata in manila, or changing existing fields in some way to remain compatible 15:22:01 now when an actual user who has the old version of manila and that driver and that vendor's storage controller wants to upgrade, something needs to happen 15:22:52 they need to decide whether to upgrade manila first or the storage controller first, and once they do, they need to perform those upgrades, and plan one of more restarts of the manila services 15:23:19 at some point something in the driver needs to flip the switch to start allowing the new feature which is now enabled by the new software 15:23:50 ensure_share seems like the ideal way to update any per-share information that needs updating in this case 15:24:31 the catch is, we don't want to be ensuring all the shares every time we restart, because most of the time the software isn't being upgraded 15:24:43 so how do we know when to do it? 15:25:08 bswartz: store driver version with share DB model? 15:25:17 manila needs some way to ask the driver, and the driver needs some way to persist enough state to tell the difference between a no-upgrade restart and an upgrade-restart 15:25:23 bswartz: and when version changed we run this method? 15:25:39 vponomaryov: yes that's the kind of thing I'm thinking about 15:25:58 however a plain "driver version" might not be flexible enough because versioning itself is a complex topic 15:26:21 bswartz: then we define versioning rules first 15:26:43 I was thinking something as simple as a unique hash which doesn't change unless the software is modified 15:26:44 We could store driver infos(inculde version) in method 1, It is up to driver 15:26:53 bswartz, vponomaryov: that looks similar to what zhongjun_ proposes in her spec 15:26:56 bswartz: where we change major part only when we need to run this ensure share method 15:27:51 driver could tell manila whether the driver need to update, It is also up to drvier 15:27:57 ganso: as long as it's automatic and there's no API/CLI invoking required 15:28:29 zhongjun_: it is already possible 15:28:34 so that would address the question of: should we ensure all or none of the shares 15:28:42 zhongjun_: driver can either skip "ensure_share" execution or not 15:28:55 the other question is, is there a case to ensure some subset of the shares? (like cinder does with volumes) 15:30:11 bswartz: we are most likely to run ensure_share for all shares of a certain backend stanza, even if only a few are needed, that would be okay I guess 15:30:57 vponomaryov: yes 15:31:56 ganso: I can't think of a case for a subset, but we have to consider the scalability implications then 15:32:28 if it's all-or-nothing for ensure share, then the cases where we do all of them might cause *very* slow startups on backends with large numbers of shares 15:33:34 bswartz: a subset could be only the shares belonging to an array that was upgraded... a subset within that subset would be only the ones affected by the upgrade, but it seems too complex to try to detect this smaller subset effectively, it may be too intensive to do so 15:34:35 bswartz: s/intensive/expensive 15:34:38 now you guys are making it sound like these upgrades should be controlled by and admin CLI to do them in reasonable chunks 15:34:40 yeah 15:34:47 bswartz: We could do it in another patch, if we really need to do it. 15:35:03 markstur: admins already do control upgrades 15:35:17 I'd rather avoid making upgrades any harder than they already are 15:35:45 I mean if this feature-enhancing on-startup will tweek 1000 shares and take a day, then maybe the admin would prefer it doesn't 15:36:03 and control the share upgrade in smaller chunks 15:36:47 just sayin' the concerns mentioned above made me think this ^ 15:37:53 markstur: updating whole backend I would expect some time soncumption 15:38:06 but hopefully most upgrades are a fast loop thru a list with little tweak of some setting 15:38:24 unfortunately some "upgrades" are mandatory and whether you do them in small chunks or all at once, you can't bring up your services until they're all done 15:38:31 vponomaryov: so... 15:38:44 markstur: so, small chunk does not really solve anything 15:39:02 if there's an "upgrade" which isn't really mandatory, then yes it could be handled as a background task *after* the driver fully comes online 15:39:05 So if manila is upgraded and there are a bunch of backends and some come up right away but one is off-line for a significant period chunking thru upgrade... 15:39:29 that'd be kind of bad, but OK. I suppose it'd be hard for us to know and warn an admin how long it'd actually take anyway 15:40:15 maybe we need a real scenario before we worry too much about background upgrades and chunking and CLI 15:40:42 yeah that's what I was hoping we would arrive at with this spec 15:41:41 I really like the vision and creativity of this group, but speculative features w/o customer need can be a real problem 15:42:41 who has experience from production deployment and manila restart? 15:42:41 fwiw, the cinder team faced some of the same issues and there was discussion about their ensure_volume and similar problems back in Ft Collins (summer 2016) 15:43:40 markstur: our real customer just want to update the shares without restart 15:44:49 that would be rolling upgrades 15:45:13 zhongjun_: in that case maybe ensure_share is the wrong vehicle and we need a polling-based approach or a new API (or even the UNIX SIGHUP approach) 15:45:14 yeah but just rolling shares not like rolliing all of manila or OS 15:45:42 rolling upgrades solves a different problem IMO 15:46:00 rolling upgrades is about minimizing cloud downtime while upgrading openstack 15:46:23 well... taking that concept to manila only 15:46:35 what zhongjun_ refers to is upgrading individual shares without manila downtime 15:46:39 we would like to perform an upgrade without restarting services 15:46:39 rolling upgrades, and update the shares that is up to driver. 15:47:12 also, we wouldn't be interested in keeping the old behavior in this case, it is the case where the share wouldn't work without the upgrade fix 15:47:44 vkmc: !!! 15:48:00 how do you upgrade without a restart? 15:48:19 that's a neat trick I'd like to learn 15:48:45 bswartz: I don't see a problem with that... if we can reload the config and update the db, an array update becomes very simple from manila point of view 15:49:04 bswartz: s/update/upgrade 15:49:05 oh you mean upgrade the backend 15:49:12 bswartz: yea, that's one use case 15:49:23 yes in theory m-shr can stay up while the backend gets upgraded 15:49:34 bswartz, I'm not saying it's possible to do it for realz... you can simulate it though 15:49:41 in the sense that customers won't see the downtime 15:49:46 bswartz: if driver version is changed, then it has to be restarted to reload python binaries 15:49:47 although you might get some errors on inflight operations 15:49:52 and yeah, it requires some juggling 15:49:57 ganso: that's what I was thinking 15:50:22 although if the Linux kernel can support so-called "live patching" then maybe someday python will too 15:50:28 bswartz: that's why we shouldn't worry that much about avoiding restarts, we can't solve every use case avoiding restarts 15:50:38 ganso: +10 15:50:42 still, my understanding was that ensure shares wanted to address a different problem 15:50:55 markstur: yeah, we could rolling part of manila shares, but tag shares maybe it will make admin a little busy 15:51:13 that's why I'm more focused on making ensure_share do the right thing when the driver restarts, and no on avoiding restarts 15:51:23 yeah. manual stuff doesn't scale well 15:51:55 okay anything else on the topic of ensure_share? 15:51:58 bswartz: the thing is, we can't have restart as a trigger 15:52:03 bswartz: as it is today 15:52:13 ganso: why not? 15:52:18 it seems fine to me 15:52:18 bswartz: as part of the use cases aren't solved by driver restarts 15:52:25 ganso: I don't suppose you want to right a m-upg to do background upgrades 15:52:28 bswartz: like, upgraded the array, driver version remains the same, restart is not needed 15:52:32 s/right/write/ 15:52:36 markstur: lol no 15:52:40 markstur: lol 15:52:55 bswartz: we could decouple them 15:53:03 bswartz: just need a new trigger 15:53:04 ganso++ 15:53:05 * markstur thinks ganso could be convinced if it was a good idea 15:53:11 ganso: I never said ensure_share solves all the use cases -- just that it solves some and it has the benefit of being mostly idiot-proof 15:54:05 other solutions like new APIs require the admin to do something and I guarantee some fraction of admins will forget or not do it 15:54:21 markstur: lol I am not very fond of creating new services :P 15:54:29 anyways since we're low on time 15:54:31 #topic open discussion 15:54:37 anything else to discuss today? 15:55:36 alright 15:55:49 thanks all and please keep reviewing specs 15:56:01 thanks 15:56:02 #endmeeting