15:01:25 <bswartz> #startmeeting manila 15:01:25 <openstack> Meeting started Thu Nov 14 15:01:25 2013 UTC and is due to finish in 60 minutes. The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:28 <openstack> The meeting name has been set to 'manila' 15:01:43 <bswartz> good morning folks 15:01:53 <xyang1> good morning 15:01:56 <jcorbin> Good Morning 15:02:09 <bswartz> I don't have an agenda today 15:02:24 <bswartz> today is my first day back since I extended my stay in Hong Kong 15:02:44 <bswartz> we can talk about the conference however 15:03:07 <glenng> *conference was good" 15:03:27 <bswartz> yeah I enjoyed meeting many of you at the conference 15:03:30 <vbellur> glenng: +1 15:03:55 <xyang1> nice meeting everyone 15:03:56 <bswartz> I think that the unconference session was successful 15:04:23 <bswartz> for those of you who weren't able to be there, the unconference session was well attended, and the questions asked were good ones 15:05:40 <bswartz> the main thing that i realized after seeing how much interest there is in the project is that we need to get the project into a state where vendors can write backends very soon 15:06:07 <caitlin56> +1 15:06:08 <bswartz> I suspect the best way to do that is to get the mutitenant support working ASAP, and to add support for that to the NetApp driver 15:06:12 <vbellur> bswartz: agree 15:06:20 <glenng> bswartz: Agree. 15:06:26 <xyang1> +1 15:06:48 <bswartz> that means that the so-called Generic driver (or "LVM" driver) is less important in the short term 15:07:19 <bswartz> the generic driver has been sucking up a lot of time and attention and ultimately it's not critical to getting other backends implemented 15:07:32 <vponomaryov> bswartz: I can make update about multitenancy 15:07:34 <caitlin56> LVM driver is of most interest to the TC - i suspect most of the people here want to build their own drivers. 15:07:40 <bswartz> it IS critical for an actual release, so we can't simply ignore it 15:08:10 <vponomaryov> 1) Generic LXC share driver - https://review.openstack.org/#/c/55265/ 15:08:13 <bswartz> oh thanks for reminding me caitlin56 15:08:41 <vponomaryov> 2) Neutron API module - https://review.openstack.org/#/c/55555/ 15:08:41 <vponomaryov> 3) Network allocation in ShareManager - https://review.openstack.org/#/c/55558/ 15:08:54 <bswartz> evidently the TC considered our incubation request last tuesday, while I was on an airplane 15:08:56 <xyang1> I thought we are going with full blown VM? 15:09:33 <bswartz> I was under the impression that it would be considered next tuesday but they're considering it all this week 15:09:42 <bswartz> so the LVM driver is an issue for them 15:09:59 <vbellur> bswartz: even I was under the impression that it would be on 19th 15:10:03 <bswartz> there are a few important points about that 15:10:11 <bswartz> vbellur: I bet it's because I told you that date 15:10:23 <bswartz> vbellur: I was the one who was mistaken however 15:10:48 <vbellur> bswartz: possibly so, I never bothered to check either. 15:10:52 <bswartz> 1) We must stop calling it the "LVM driver" -- it shall becalled the "Generic Driver" henceforth 15:11:54 <bswartz> 2) I propose that the generic driver uses virtualization (based on nova) and a block device backend (based on cinder) 15:12:24 <bswartz> vponomaryov: I have yet to look at the LXC code that's been written, but if it works I don't want to ignore it 15:12:32 <caitlin56> bswartz: and implements generic NFS/CIFS from stock linux? 15:12:55 <vponomaryov> bswartz: yes, it works, tested 15:13:02 <bswartz> vponomaryov: As I mentioned above, everything related to the generic driver is low priority in the short term though 15:13:20 <vponomaryov> for each implementation of protocols can be written class-helper 15:13:43 <caitlin56> yponmaryov: but isn't setting up things like tenant specific DNS take extra work (even though I have seen scripts that demonstrate how). 15:13:44 <vponomaryov> but not tied yet with neutron code 15:13:46 <bswartz> vponomaryov: I'll look at it soon, the other 2 changes are more interesting however 15:14:10 <jcorbin> vponomaryov: First code review link works, other 2 give permission errors. Is this expected? 15:14:37 <bswartz> yes it seems that the submissions are marked private currently 15:15:25 <vponomaryov> jcorbin: yes, it hat restricted access, at least, for now 15:16:10 <bswartz> vponomaryov: please add access to jcorbin as he's familiar with neutron and I'd like his feedback as well 15:16:38 <bswartz> anyone else who's curious please ping vponomaryov too 15:17:05 <navneet> mee too 15:17:09 <xyang1> vponomaryov: please add me too 15:17:16 <vbellur> vponomaryov: me too 15:17:19 <vponomaryov> jcorbin: provide your username in launchpad 15:17:52 <bswartz> so the other thing that's clear from the conference is that there is strong interest in the "tunneled" style of access, expecially for backends like ceph and gluster 15:17:57 <glenng> vponomaryov: My Launchpad name is ggobeli 15:18:04 <bswartz> Any IBM/GPFS folks here? 15:18:19 <jcorbin> vponomaryov: My Launchpad name is osjcorbin. Thanks! 15:18:26 <bill_az> vponomaryov: also me - billowen@us.ibm.com 15:18:28 <bswartz> vponomaryov: it looks like *Everyone* is interested may as well mark it public 15:18:31 <xyang1> vponomaryov: my launchpad name is: xing-yang 15:18:34 <bill_az> And I'm IBM/GPFS 15:18:56 <bswartz> bill_az: have you given any thought to the multitenant access issues? 15:18:59 <vbellur> vponomaryov: my launchpad id is vbellur 15:19:08 <bswartz> bill_az: is GPFS going to be in the same boat as ceph and gluster? 15:19:18 <vponomaryov> bswartz: I can inform our devs, I am not the owner of gerrit commit 15:19:28 <bswartz> vponomaryov: ty 15:19:29 <bill_az> bswartz: I have been discussing it with others on the team 15:19:51 <bill_az> bswartz: we have some things coming in gpfs 4.1 in the spring that will help w/ multitenancy 15:19:52 <caitlin56> bswartz: by "tunneled" do you mean some form of "virtFS"? 15:19:53 <navneet> mine is singn 15:20:28 <bswartz> So just to recap, many clustered filesystems such as ceph and gluster doesn't have facilities to provide "virtual servers" on different tenant networks 15:20:56 <bswartz> These clustered filesystems assume full network connectivity between the client and every server node 15:21:29 <caitlin56> bswartz: I believe the fundamental issue is who deals with tenant users. The "virtFS" approach puts this on the Hypervisor, the virtual server approach puts it on the file server. 15:21:54 <bswartz> caitlin56: yes 15:22:06 <gregsfortytwo1> that's part 1, part 2 is the networking constraints imposed by the full connectivity bswartz mentioned 15:22:26 <caitlin56> My evaluation is that relying on the hypervisor a) will add a minimum of one year to the project and b) doesn't truly secure the files even then. 15:22:43 <gregsfortytwo1> bill_az, I assume gpfs also requires connectivity to every node, right? 15:22:50 <bswartz> so for clustered filesystems that aren't able to create a virtual-server-per-tenant, due to the number of network interfaces involved, we need a hypervisor-mediated approach to secure multitenancy 15:23:24 <bswartz> that could mean virtfs, or NFS-over-PPP, or NFS-over-vsock 15:23:26 <caitlin56> Nobody is *unable* to do a virtual server per tenant, they just haven't done it yet. 15:23:41 <bill_az> gregsfortytwo1: yes -thats correct 15:24:06 <gregsfortytwo1> caitlin56: if "server" — implying one — is nonsensical in your architecture you're unable 15:24:10 <bswartz> caitlin56: I think the issue is the number of network interfaces involved -- it's simply impractical 15:24:21 <caitlin56> Tunneling NFS with less-than-a-tenant-network sounds like a simpler approach than virtFS. 15:24:26 <gregsfortytwo1> we are all able to implement multienancy even if we haven't, but that's not the biggest issue from Manila's perspective 15:24:28 <bill_az> gregsfortytwo1: but we plan to limit access through isolated filesystems and encryption keys 15:24:37 <bswartz> if you have a 200 node ceph cluster, you can't create 200 additional virtual network interfaces for every tenant 15:25:26 <caitlin56> But even with tunneled NFS, the server is going to have to do per-tenant authentication, which I presume they are also incapable of doing today. 15:25:28 <gregsfortytwo1> bill_az: yeah, that's essentially the path Ceph is walking down, though not very quickly 15:26:09 <bswartz> caitlin56: the theory would be to create a NFS-to-ceph bridge, or an NFS-to-glusterFS bridge 15:26:25 <bill_az> gregsfortytwo1: it will be tough for us for icehouse - but should have that for j 15:26:25 <bswartz> the NFS itself would have no authentication 15:26:37 <caitlin56> That's fine. That's a vendor provided proxy. Effectively a virtual server. 15:26:44 <gregsfortytwo1> caitlin56: no, we can handle that already in most of the stack; not sure about gluster; sounds like gpfs can/will 15:26:44 <bswartz> fortunately bad security is something NFS is very good at 15:27:27 <caitlin56> gregfortytwo: my main concern is avoiding an openstack managed common UID/GID pool that everyone maps to. 15:27:27 <vbellur> gregsfortytwo1: gluster can handle that too 15:29:03 <gregsfortytwo1> caitlin56: ah, hadn't even gotten to that issue yet in my head :) 15:29:10 <bswartz> okay so I'm glad we're all thinking about the issue here -- tunneled FS support will need to come soon after the full network connectivity style of multitenant support 15:29:18 <bswartz> issues* 15:29:49 <caitlin56> question: does it need to be a lib, or can it just be a NAS proxy? 15:29:51 <bswartz> those are the 2 main roadblocks to allowing various storage vendors to write functional backends 15:30:38 <bswartz> the actual implementation of the bridge/proxy will need to run on the compute node -- whether it's a manila-agent, or changes to nova, or some library -- I'm not sure about taht 15:30:56 <bswartz> for a prototype we should keep the code in manila 15:31:25 <bswartz> long term it might make sense to host it elsewhere -- especially if it's generically useful outside of manila 15:31:30 <caitlin56> So this would be a pluggable library that ran in the same context as the manilla agent? 15:31:58 <bswartz> I want to be careful with the term "manila agent" because it could mean a lot of different things 15:32:18 <caitlin56> You would want it to be on a path that made sense for bulk data transfer - preferably "close" to the guest. 15:32:43 <bswartz> the idea of automatic mounts inside the nova instances is still interesting to many -- and those may necessitate a different kind of agent 15:33:13 <bswartz> but absolutely it can/should run on the compute node 15:33:43 <bswartz> possibly "manila-bridge" or "manila-gateway" would be a more descriptive term 15:33:51 <caitlin56> bswartz: proxying TCP connections is something compatible with the current infrastructure. Passing full file-systems through hypervisors will be a lot more work. 15:34:13 <bswartz> caitlin56: we're talking about the later not the former 15:34:37 <bswartz> I'm aware that it will be real work 15:34:39 <caitlin56> bswartz: distinguishing between manila-bridge (control plane) vs. manila-gateway (user-plane) sounds like a good idea to me. 15:35:32 <bswartz> anyways today I just wanted to lay out the priorities 15:36:02 <bswartz> 1) get the neutron code working so we can do full network multitenancy 15:36:25 <caitlin56> I first looked at NAS proxies for VMs two years ago, and concluded that intercepting at the network layer was more feasible than at the FS-operation layer. 15:36:30 <bswartz> 2) implement some tunneling/virtfs code to support mediated multitenancy 15:36:40 <bswartz> 3) implement the generic driver 15:36:58 <caitlin56> I feared that even if Xen and KVM adopted "virtFS" it would take two years to deploy. In those 2 years there has been no progress on adopting a virtFS API. 15:37:29 <bswartz> caitlin56: I'd be curious to know why that is -- let's take that discussion offline 15:37:51 <bswartz> regarding virtfs -- if that turns out to be a blocker than NFSv3-over-vsock may be a faster path to get something working 15:38:37 <gregsfortytwo1> s/offline/to #openstack-manila? :) 15:38:46 <caitlin56> Let each proxy decide whether to support NFSv3 and/or NFSv4. 15:38:56 <bswartz> there are lots of technologies that can be applied here -- I'm confident we'll find something that works 15:38:57 <caitlin56> or for that matter CIFS. 15:40:14 <bswartz> caitlin56: we're going to have to write these proxies ourselves though -- I'm not eager to take on tons of work 15:40:39 <caitlin56> Correct, these are pluggable proxies. Someone has to provide their own proxy. 15:40:54 <bswartz> If NFSv3 works as a bridge protocol, I think it will be hard to argue that there's a significant advantage to using something like cifs or v4 15:41:11 <caitlin56> Am I correct that neither CEPH or Gluster would want soeone else writing the proxies to access their networks? 15:41:34 <bswartz> caitlin56: I think you're incorrect 15:41:52 <bswartz> I should be the job of manila to provide the glue 15:41:56 <gregsfortytwo1> no reason to independently implement setting up an nfs server instance ;) 15:42:19 <bswartz> my vision is to implement one thing that works for everyone 15:42:44 <caitlin56> So you want a manila coded proxy that maps UID/GIDs for NFSv3 to NFSv3? 15:42:52 <bswartz> once we have a gateway of sorts that runs on the compute node, it shouldn't matter what's on the actual backend -- as long as the compute node can mount it, it can be tunneled through to the guest 15:43:30 <bswartz> caitlin56: I don't want any UID/GID mapping to occur 15:43:47 <bswartz> if we do this correctly, it should be safe to do no mapping 15:43:48 <caitlin56> Another form of proxy would translate Any-NAS-Protocol to My-backends-proprietary-protocol. 15:44:04 <aostapenko> in lxc implementation we can mount directly to vms 15:44:37 <aostapenko> we will can :) 15:44:58 <bswartz> okay so we have some conflicting ideas here 15:45:19 <bswartz> maybe my vision will not work out 15:45:26 <bswartz> can we at least agree on the priorities I mentioned? 15:46:00 <bswartz> those who are interested in the tunneling approach should probably form a working group and hash out the issues 15:46:09 <caitlin56> If you aren't doing UID/GID mapping then you would have to trust the gateway to do mount authentication. Then all you need to map to a universal pool are the mounts. 15:47:06 <bswartz> I'm interested in the tunneled approach working correctly, so I'll organize those discussions 15:47:38 * caitlin56 is interested in both models - could see implementing either on our end. 15:47:51 <bswartz> I suspect that we might be making this harder than it needs to be though 15:48:29 <bswartz> My instinct is that we should write a prototype first, and then discover all the ways that it's wrong 15:48:31 <vbellur> bswartz: priorities seem fine to me 15:48:35 <caitlin56> You can make it simpler if you just state the caveat that a gateway solution requires trusting the gateway. 15:48:58 <bswartz> caitlin56: oh yes, absolutely 15:49:23 <vbellur> bswartz: +1 to the prototype 15:49:34 <bswartz> caitlin56: the gateway approach is akin to how you must trust the hypervisor to faithfully present block devices to you 15:49:37 <bill_az> bswartz: +1 - get the simlest --realistic-- prototype working asap as a model and to help discussion 15:49:50 <jcorbin> bswartz: +1 to the prototype 15:50:28 <bswartz> this "manila-gateway" thing would be running on the compute node, and it would be part of the cloud infrastructure -- hidden from the tenant's view 15:50:30 <caitlin56> +1, but we should not wait onthe multi-tenant network solution. 15:50:33 <gregsfortytwo1> the gateway would be essentially the generic implementation, plus some holes that different drivers can plug in order to set up their own backends, right? 15:50:39 <bswartz> the tenant would have no choice but to trust it 15:51:20 <vbellur> gregsfortytwo1: sounds like a good model 15:51:26 <caitlin56> bswartz: the multi-tenant network approach with virtual servers avoids needing to trust the compute host. 15:51:40 <bswartz> gregsfortytwo1: the gateway is really a get-out-of-jail-free card for backends that can't connect themselves directly to tenant networks on account of the limitted-number-of-network-interfaces problem 15:52:06 <bswartz> caitlin56: yeah I believe that's the best method for backends than can support it -- it's just that not all can 15:53:06 <bswartz> in fact it's come to my attention that even some more traditional NFS-based storage devices (I'm thinking of Isolon) can't directly support multitenancy, and may need something like this to work in public cloud environments 15:53:17 <caitlin56> bswartz: if the server trusts the gateway to mount only exports approved by the correct authentication server then you only need a single LAN netowrk between the compute hosts and the file servers. 15:53:56 <bswartz> caitlin56: yes that's the idea 15:54:03 <bswartz> okay since we're nearly out of time 15:54:09 <bswartz> #topic open discussion 15:54:23 <bswartz> does anyone else have another topic that I didn't bring up? 15:55:31 <bswartz> wow that quieted things down 15:55:31 <bill_az> bswartz: is there a summary of manila activities at summit? 15:55:49 <bswartz> bill_az: yes 15:56:03 <bswartz> as I mentioned at the beginning, we have a unconference session which went well 15:56:13 <bswartz> we had* a unconference session 15:56:47 <bswartz> and we also got together for drinks and I managed to get my boss to pay the bar tab 15:56:48 <bill_az> bswartz: sorry - i missed the first few minutes - is there minutes or summary published? 15:57:16 <bswartz> thanks to everyone who showed up -- it's always great to meet face to face before we go off and spend 6 months coding together 15:57:22 <vbellur> bswartz: thanks for the drinks! :) 15:57:22 <bill_az> bswartz: sorry I missed both! (especially drinks...) 15:58:14 <bswartz> and I can't tell you how many people grabbed me personally in the hallways to talk about manila 15:58:23 <bswartz> the interest level is high 15:58:34 <bswartz> we need to move fast while we have momentum 15:58:37 <vbellur> bswartz: +1 15:59:14 <bswartz> I'll be talking to TC members about the incubation request -- we should have an answer on tuesday 15:59:31 <bill_az> bswartz: that all sounds good - Thanks 15:59:33 <bswartz> and with that we're out of time 15:59:35 <glenng> Let's hope, 15:59:36 <bswartz> thanks everyone 15:59:42 <vbellur> thanks all 15:59:51 <bswartz> #endmeeting