15:00:18 #startmeeting Manila 15:00:19 Meeting started Thu Dec 17 15:00:18 2015 UTC and is due to finish in 60 minutes. The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:21 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:23 The meeting name has been set to 'manila' 15:00:27 hello all 15:00:28 Hi 15:00:33 hello 15:00:34 \o 15:00:35 hi 15:00:37 hi 15:00:57 #agenda https://wiki.openstack.org/wiki/Manila/Meetings 15:00:58 hi 15:01:18 hi 15:01:26 o/ 15:01:27 hi 15:01:34 #topic announcements 15:01:54 So I decided to cancel the next 2 meetings 15:02:08 due to the winter holidays 15:02:30 +1 15:02:31 I know many people won't be able to attend, and although some people will be working during that time, we don't really need to hold the meeting 15:02:53 so the next IRC meeting will be Jan 7th 15:03:30 #topic Proposal for share groups 15:03:45 If you didn't see the ML post yesterday, please review the wiki. 15:03:51 #link https://wiki.openstack.org/wiki/Manila/design/manila-generic-groups 15:04:02 After building CGs, and then stepping back and looking at it, we think Manila will be much better served with a generic grouping construct. 15:04:02 #link http://lists.openstack.org/pipermail/openstack-dev/2015-December/082490.html 15:04:14 That will enable all Manila actions for share groups on *any* driver, while still having the opportunity for drivers to provide a backend-specific advantaged implementation if possible. 15:04:30 It would simplify and normalize the user experience. 15:04:39 It would simplify development of future features. 15:04:46 And it would make testing everything easier. 15:04:52 To do this, we would morph the CG implementation into a generic group construct, add group types, and re-add CGs using the new framework. 15:05:02 We would also build group support into the Manila UI (think tree table, perhaps?), so it would be easily usable by everyone. 15:05:09 I tried to capture lots of implementation details in the wiki, but I'm sure y'all have questions. Thoughts? 15:05:15 So there was a preview of this last week 15:05:28 who has read the writeup? 15:05:33 I have 15:05:38 I do 15:05:38 I have 15:05:55 s/do/have 15:05:55 The wiki page? 15:05:58 any concerns? 15:06:02 dustins: yes 15:06:02 dustins: yes 15:06:08 Yup, I have 15:06:26 I like this idea 15:06:34 This seems like a fantastic idea from a lot of different areas 15:06:44 just statement about two person-weeks efforts looks tooooo optimistic 15:06:46 i haven't had a chance to read it yet but got the early version in discussions 15:06:55 Incidentally, we don't see this as a huge effort, since we already know where to touch the code to add groups. 15:07:11 This should fit in M.2 if we choose to proceed. 15:07:22 Better organization in the CLI and GUI, operations that spread across the groups, backend neutrality 15:07:37 cknight: you talk only about server side? 15:07:53 cknight: to fit M-2? 15:08:01 Or rather spread across the members of a group 15:08:09 vponomaryov: I think we'd do everything in parallel. 15:08:31 vponomaryov: Only the UI might lag a bit, but doing groups in the UI becomes much more compelling than just for CGs. 15:09:00 vponomaryov: So I hope we'd do server, client & UI in Mitaka. 15:09:00 it looks like this will affect a lot existing APIs, should we consider this change a v3 candidate? 15:09:20 ganso: No, it shouldn't affect many existing APIs. 15:09:23 ganso: microversions are used for such cases 15:09:26 ganso: the affected APIs should all be experimental ones 15:09:33 ganso: CGs are experimental, so we can change those at will. 15:09:42 what if I want to allow-access to a group of shares 15:10:06 so that will be a new API command, existing allow-access to single share will remain the same then, right? 15:10:09 One concern I have is if the group snapshot or group replication can be sometimes consistent and sometimes not, it could cause confusion to customers 15:10:11 ganso: I think the APIs wouldn't be overloaded for shares & groups, so group APIs would be incremental. 15:10:54 xyang: that's true, but I don't see how this proposal makes things any worse than they already are 15:10:56 xyang: That's fair. We could talk about public group extra specs for things that users need to know. 15:11:17 I think this proposal makes it better in some (but possibly not all) cases 15:11:18 bswartz: that is inconsistent behaviour 15:11:29 bswartz: +1 15:11:33 if it is always consistent, then no confusion 15:12:17 xyang: I'm not sure how it's different than shares that are sometimes deduped and sometimes not. Same for other extra specs. 15:12:21 I'm very against inconsistent behavior, but in cases where it can't be avoided, we have ways of dealing with it 15:12:37 bswartz: +1 15:12:53 we've seen this with snapshot support and the solution is to make sure the user knows what he's going to get before he creates his first share 15:12:58 cknight: why add group type and not extend share type? 15:13:21 cknight: dedupe is different. It could be where I'm coming from, I only know consistent group snapshot and replication 15:13:32 vponomaryov: I had that thought as well, but Ben pointed out that the public extra specs on share types wouldn't map well to groups. 15:14:10 cknight: do you have example? 15:14:19 cknight: that would not fit? 15:14:37 cknight: shouldn't all shares in a group be of the same type? 15:14:40 I know that administrators will find all the different capabilities challenging, but when I look at the various alternatives, cknight's proposal seems like the least bad option from the administrator's point of view (thinking about providing a consistent experience to his end users) 15:15:07 vponomaryov: DHSS perhaps? Ben also wanted to be explicit about share types vs. share group types. I agree it would be possible and faster to reuse share types for this. 15:15:41 I think that group types are fundamentally different from share types 15:15:41 ganso: Not necessarily. That wasn't true for CGs. 15:15:53 I would like them as public extra specs, then i think it's easy enough as a user to know the abilities of a group 15:15:56 Something where you could apply the share type to a share group and it would propagate to all members of the group? 15:16:16 bswartz: +1 15:16:24 ameade: +1 It should be simple to define a set of public group extra specs. I outlined several on the wiki. That should allay Xing's concern. 15:16:43 cknight: that could complicate things a lot, thinking about migration... if I want to migrate a group, but a certain share in the group is not the same type, I may not be able to migrate all shares to the same destination backend anymore 15:17:28 ganso: I'm worried about complexity too by adding a group type 15:17:37 ganso: you already have that problem if you don't have other backends that match the share type 15:18:02 When CG was first introduced in Cinder in Juno, it was supposed to be dependent on a Type Group too. It was in the first few revisions of the spec. 15:18:31 It got dropped when someone raised concerns on how to support multiple Type Groups 15:18:32 ganso: It does mean you would have to check each group member first to make sure each would be a valid migration target, but that doesn't sound terrible. And the problem already exists now, where you can create a CG with multiple share types. 15:18:48 whether it's a good idea to allow multiple share types in a group comes down to whether you agree with the use case for CGs -- that you want to split your dataset across 2 or more shares of different types and maintain consistency between them 15:18:51 so it got dropped due to the increased complexity 15:19:16 ameade: yes but would that migration succeed or fail? if fail because a single member does not fit, then it is the same as using a group type... being restrictive here reduces granularity and complexity 15:19:21 cknight: ^ 15:19:46 xyang: I can understand the reluctance at the time, but looking back now at all the complexity both projects have built, I'm looking for simplifying and unifying constructs. 15:19:48 cknight: I did not participate in CG development, but would it be bad to have the restriction of same share types within a CG? 15:19:56 it wouldn't fail due to a single member, it would fail because there is no other backend that could support the group 15:20:34 ameade: exactly, the way I understand that is the same as the restriction 15:20:57 ganso: initially the proposal was to support one volume type in one CG in Cinder, but quickly there are concerns that it should support multiple volume types 15:21:17 ganso: so it is changed to support multiple volume types and the scheduler will pick one backend 15:21:31 i think having mixed cgs is an important use case 15:21:42 ganso: I'm not sure how restrictive that would be. But the problem isn't limited to CGs or migration. There will simply have to be logic that can evaluate the suitability of a backend to contain a group based on its type and its members' types. It's not that complicated. 15:22:25 cknight: can we leverage share types rather than introducing another group type construct? 15:22:25 xyang: I know what you're talking about with the increased complexity, but after taking a look at what it will require to unify the new features we have (cgs, migration, replication) I worry that the complexity there is even worse. I think we've managed to fool ourselves into thinking we can graft features onto the side of Manila with little complexity because we're just delaying the costs until later 15:22:28 xyang: a backend that supports all type variations within a CG then... that will restrict the number of destination backends. 15:22:40 ganso: The goal would be to have that logic live in one place, so the migration, replication, etc. engines can use it as needed. 15:23:11 ganso: we have to choose one backend though because there one manager that handles one backend 15:23:23 ganso: this is a architecture limitation 15:23:36 xyang, cknight: I understand now, thanks :) 15:24:39 xyang: regarding another group type construct, that's what vponomaryov asked. It's possible, but Ben was keen to keep it separate for clarity. 15:25:10 cknight: is it going to be required ? 15:25:31 vponomaryov: is what required? using groups? or group types? 15:25:44 cknight: usage of group types for each share 15:25:51 vponomaryov: no 15:26:10 vponomaryov: it seems the groupe type is related to group model, not share model 15:26:30 group types would be for groups -- shares don't have to go in a group -- and group could even have a default group type so you don't have to specify one if you don't care 15:26:32 vponomaryov: and the share is part of a group, but is also still an individual share 15:26:41 cknight: ok, do we plan to allow existing share inclusion to some group later? 15:26:43 vponomaryov: If there were no code in Manila today, we could choose to implement every share in its own group, but that is an implementation detail, and I think we're too far along to do that. 15:27:17 vponomaryov: +1 15:27:30 vponomaryov 15:27:38 vponomaryov: I think we could do that. We didn't allow that in the first release of CGs, so you could only assign group membership at either share or group creation time. 15:27:38 in that case the existing api should change 15:27:41 cknight: what about add a share to a group or remove a share from a group? A non-pool group can support it but a pool group cannot 15:28:01 vponomaryov: But I added 'group modification' to the list of group capabilities so we could consider that. 15:28:32 cknight: just to confirm one thing, we are enforcing grouping, right? there will be no more shares without groups, a single share has to be a single group, right? 15:28:37 xyang: I think cknight said he was explicitly avoiding any pool-related groupings 15:28:50 pools are not groups in the current proposal 15:29:02 bswartz: oh? 15:29:12 therefore pool-based replication is still out of scope 15:29:13 ganso: shares and groups are first class citizens as per the wiki.. as far as i understand, so shares can exist without groups 15:29:16 bswartz: isn't the current CG implemented for a pool? 15:29:19 cknight: is that accurate? 15:29:42 a different set of API for pool? 15:29:43 xyang: CGs can span pools, or not -- it's up to the backend 15:29:47 bswartz: Yes. Pool-based replication or anything similar is fundamentally different, because a pool contains data from multiple tenants. 15:30:11 pools are never visible to the end user 15:30:21 gouthamr_, cknight: having these two possibilities make for two possible branches of code, it may be very complicated to test every feature like this 15:30:26 pools are just for admins and for the scheduler 15:30:29 cknight: so what about the existing CG snapshot? will that be supported by this new group? 15:30:36 I'm confused 15:31:20 xyang: I think the proposal include ripping out the existing CG experimental APIs and replacing them with group-based consistent snapshot APIs 15:31:27 xyang: the way I understand, the generic group would have the "consistent snapshot" group type, the backend would read that and do the consistent snapshot when snapshots are triggered 15:31:29 xyang: This proposal replaces the Manila CG implementation. The admin creates a group with the CG extra spec, and the user creates a group of that type and invokes snapshot on it. 15:32:06 xyang: When the group snapshot call arrives in the manager, the manager sees that the type of that group is CG, so it hands it off to the driver for handling. 15:32:08 cknight: so my question is still relevant then. the existing CG implementation cannot add/remove share 15:32:46 xyang: Yes, your question is definitely relevant. It's up to all of us to decide whether group membership can change anytime, or only at share / group creation time. 15:33:06 xyang: the solution there is to make add/remove from group an optional feature, some drivers may support it and others may not 15:33:45 bswartz: ok, that's big difference between pool-based and non-pool-based grouping 15:33:54 bswartz: +1 I think that's unavoidable, since some backends can't change CG membership anytime. That's why I suggested the 'group_modification' extra spec. 15:33:58 bswartz: +1 Manila could allow it, and fail if the driver doesn't support it. 15:34:24 xyang: But since type creation is always admin-only, it's up to the admins to create the types correctly. 15:34:31 We should try to avoid "adds" that make group actions turn into "no host found" 15:34:36 if we enforce this restriction, it may be simple for backends, because AFAIK, for CG to work, the shares have to be created in a previously defined CG... so having both behaviors (generic that allows add/remove and generic with CG extra-spec) would increase complexity 15:35:21 backends would need to know to behave both ways if they support CG 15:35:30 the ones that don't, need to implement add/remove 15:35:30 If this is a loose grouping, it implies that there should be a possibility to add/remove 15:35:53 ganso: one issue that I've learned is that CGs work very differently on different platforms, and we don't always agree on what aspect of a CG matters most 15:36:14 markstur_: +1 15:36:30 ganso: this proposal allows us to capture the important features of a CG -- such as "can take a consistent snapshot" and "can replicate with mutually ordered writes" and advertise them individually 15:36:57 ganso, markstur_, when you add share to a group, you provide a share type, which is already supported by the group 15:37:22 I like the idea of "group_modification" extra-spec because it can be added later, we could start simple by enforcing "create share in a group" 15:37:28 ganso, markstur_: actually that is create and add. If it is add, it doesn't go thru scheduler 15:37:33 ganso: +1 That was my thought. 15:37:39 ganso: +1 15:37:41 ganso: +1 15:37:49 xyang, I'm thinking that with multiple share types, the group might function until a share of the 2nd type shows up and then smoe feature won't find a target anymore 15:38:00 ganso: A lot of this can be done incrementally. We do the groups & group types first, then vertical slices adding one grouped action at a time. 15:38:03 That's probably an admin mistake, but still would be annoying 15:38:33 markstur_: i dont think that could happen because you schedule the group based on all the share types 15:38:36 markstur_: if admin changes things afterwards, it affects not only CG but everything else 15:38:46 so it would end up on a backend that can do all of them 15:39:11 ameade: yes 15:39:25 what if migrate can't find a place that suits the whole group 15:39:46 The group coud be valid and claim "migration_support", but no host found 15:39:50 then you're sol 15:39:53 I don't know how to solve that one 15:39:54 markstur_: you create a share which does not support dedupe as defined in the type; now admin adds dedupe support to the same type. now what happens to the existing share? 15:40:01 markstur_: the same issue could happen with migration of single shares 15:40:44 markstur_: the destination backend would need to support all the share types within that group that is being migrated... else it can't be migrated 15:41:08 xyang: changes to share types have never affected existing shares of that type 15:41:16 Again it probably comes down to the admin needs to know how to do it right 15:41:25 generally we advise against modifying a share type after it's been used 15:41:33 bswartz: but how does that currently play with migration? 15:41:46 bswartz: yeah the best way is to have immutable share types, but is that practical? 15:41:48 bswartz: understood. I'm saying now if you look at the type and the share, how can you tell whether it support dedupe or not by the type 15:41:58 ameade: it works if you're clever, or lucky 15:42:31 xyang: actually that's not true for the reason you just mentioned 15:43:12 bswartz: the share type says dedupe enabled, but the share does not have it 15:43:26 xyang: if the admin changes the share type, then that can happen 15:43:32 xyang: I don't understand, why wouldn't it? 15:43:46 bswartz: +1 15:43:52 bswartz: the share was created before that 15:43:56 we considered this issue when thinking about snapshot support -- you can create a share type without snapshot support, and create some shares, then later modify the type to have snapshot support 15:44:33 you can't rely on the share type to know if the share has snapshot support or not, therefore we store in the share itself whether it had snapshot support at the time of creation 15:44:39 bswartz: unless if you are saying the admin will go to the array and manually change things 15:45:01 bswartz : or replication_type in share replication. 15:45:35 if there are other aspects that manila needs to know about -- we have to store those with the shares -- we can never rely on the share type to know the actual capabilities of any given share 15:45:42 xyang, bswartz: those are probably going to be the case where shares may have to be migrated (or re-created), even in the same backend, in order to support those features... most of them cannot be simply enabled/disabled on backends after share has been created 15:45:50 this is a downside of mutable share types 15:46:33 ganso: if that is involved, yes. I have got bug report because someone just changed type without doing anything for existing shares on the array 15:46:43 xyang, bswartz: this is something retype API has to handle... I do not see this as a problem for the generic groups approach at this moment 15:46:49 xyang: using manage/unmanage? 15:46:56 okay I know this discussion could keep going, but we need to move on to other topics 15:47:21 ganso: if you do those steps, yes. 15:47:30 please keep providing feedback on the proposal 15:47:55 xyang: that is a bad practice I would say... that's why manage/unmanage is admin-only, the admin needs to know carefully what he is doing 15:47:58 Yes, please keep the feedback coming! 15:48:07 cknight: some concrete examples of how your proposal might play out would probably help 15:48:11 Don't hesitate to update the wiki. 15:48:16 bswartz: OK, will do. 15:48:37 give an example of a multi-vendor deployment with some real complexity, and show how group types makes that tractable 15:48:53 bswartz: Sure. 15:49:21 #topic Liberty.1 stable release 15:49:36 so it's come to my attention that we're late with our liberty.1 release 15:50:04 there's not a fixed schedule, but generally the other project lined up their liberty.1 releases with the mitaka-1 milestone 15:50:24 so I'm going to tag a liberty.1 as soon as I can manage it 15:50:36 I'm merging stuff that's been backported today 15:50:51 did anyone else have something that urgently needs backporting? 15:51:12 some distros are looking at liberty.1 as their base for their downstream branches 15:51:29 bswartz: there are manila-ui bug that should be done and backported 15:51:52 bswartz: current latest mailaclient release 1.5.0 not compatible with mainla-ui 15:51:52 vponomaryov: fixed in master already? 15:52:03 vponomaryov: yes we need releases of manilaclient too 15:52:04 bswartz: current fix broke it actually 15:52:10 :-( 15:52:21 bswartz: but added support of not released latest client 15:52:44 bswartz: you can find chat logs about it today in manila channel 15:52:56 okay so ignoring the client and UI -- is there anything in the manila project that needs backporting? 15:52:57 bswartz: toabctl made heads up 15:53:24 I need to spend more time catching up on the client issues, but I think the server is almost there 15:53:57 okay I'll read about the client problems 15:54:08 that's all I had to say about liberty.1 15:54:36 #topic open discussion 15:54:40 A little question: Did we need to separation of user to user and group in Access_allow API. 15:54:40 eg: access-allow %share% group %group_name% 15:54:40 Because we can not distinguish between user and user groups from only user name. 15:54:51 bswartz: there is https://review.openstack.org/#/c/257358/ which I would like to have fixed in liberty 15:55:23 toabctl: this isn't merged in master yet 15:55:40 let's prioritize reviews of that one so it can get a workflow today and the backport can start 15:55:50 I'll hold off on a liberty.1 tag until that backport happens 15:55:54 bswartz: Ci feels bad today 15:56:01 bswartz: ok 15:56:05 vponomaryov: indeed 15:56:16 the new gerrit system is ..... weird 15:56:29 bswartz: +1 Very weird. 15:56:33 bswartz: +1 15:56:34 bswartz: +1 . it was weird but now it's even more weird 15:56:34 +1 15:56:42 as with every GUI upgrade :D 15:56:50 bswartz: zhongjun had a question 15:57:05 zhongjun: to answer your question -- we don't currently 15:57:10 that is weird 15:57:37 But how to deal with this problem? 15:57:45 zhongjun: I'm not opposed to changing that if there's a good reason to, but so far nobody has had a problem with using the same API for doing CIFS group access 15:58:09 the driver/backend is supposed to figure out if the user is actually a group and do the right thing 15:58:32 bswartz: that seems like a good alternative 15:59:04 typically user names and group names don't overlap 15:59:31 and when they do, the current manila behaviour is undefined 16:00:07 so if someone has a problem with that undefined behaviour, propose an API change to make groups explicit 16:00:09 time check 16:00:15 xarses: I'm aware 16:00:24 typically user names and group names don't overlap. All of this like this? 16:00:45 thanks everything I think we're done 16:00:54 #endmeeting