15:01:25 <bswartz> #startmeeting manila
15:01:25 <openstack> Meeting started Thu Nov 14 15:01:25 2013 UTC and is due to finish in 60 minutes.  The chair is bswartz. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:28 <openstack> The meeting name has been set to 'manila'
15:01:43 <bswartz> good morning folks
15:01:53 <xyang1> good morning
15:01:56 <jcorbin> Good Morning
15:02:09 <bswartz> I don't have an agenda today
15:02:24 <bswartz> today is my first day back since I extended my stay in Hong Kong
15:02:44 <bswartz> we can talk about the conference however
15:03:07 <glenng> *conference was good"
15:03:27 <bswartz> yeah I enjoyed meeting many of you at the conference
15:03:30 <vbellur> glenng: +1
15:03:55 <xyang1> nice meeting everyone
15:03:56 <bswartz> I think that the unconference session was successful
15:04:23 <bswartz> for those of you who weren't able to be there, the unconference session was well attended, and the questions asked were good ones
15:05:40 <bswartz> the main thing that i realized after seeing how much interest there is in the project is that we need to get the project into a state where vendors can write backends very soon
15:06:07 <caitlin56> +1
15:06:08 <bswartz> I suspect the best way to do that is to get the mutitenant support working ASAP, and to add support for that to the NetApp driver
15:06:12 <vbellur> bswartz: agree
15:06:20 <glenng> bswartz: Agree.
15:06:26 <xyang1> +1
15:06:48 <bswartz> that means that the so-called Generic driver (or "LVM" driver) is less important in the short term
15:07:19 <bswartz> the generic driver has been sucking up a lot of time and attention and ultimately it's not critical to getting other backends implemented
15:07:32 <vponomaryov> bswartz: I can make update about multitenancy
15:07:34 <caitlin56> LVM driver is of most interest to the TC - i suspect most of the people here want to build their own drivers.
15:07:40 <bswartz> it IS critical for an actual release, so we can't simply ignore it
15:08:10 <vponomaryov> 1) Generic LXC share driver - https://review.openstack.org/#/c/55265/
15:08:13 <bswartz> oh thanks for reminding me caitlin56
15:08:41 <vponomaryov> 2) Neutron API module - https://review.openstack.org/#/c/55555/
15:08:41 <vponomaryov> 3) Network allocation in ShareManager - https://review.openstack.org/#/c/55558/
15:08:54 <bswartz> evidently the TC considered our incubation request last tuesday, while I was on an airplane
15:08:56 <xyang1> I thought we are going with full blown VM?
15:09:33 <bswartz> I was under the impression that it would be considered next tuesday but they're considering it all this week
15:09:42 <bswartz> so the LVM driver is an issue for them
15:09:59 <vbellur> bswartz: even I was under the impression that it would be on 19th
15:10:03 <bswartz> there are a few important points about that
15:10:11 <bswartz> vbellur: I bet it's because I told you that date
15:10:23 <bswartz> vbellur: I was the one who was mistaken however
15:10:48 <vbellur> bswartz: possibly so, I never bothered to check either.
15:10:52 <bswartz> 1) We must stop calling it the "LVM driver" -- it shall becalled the "Generic Driver" henceforth
15:11:54 <bswartz> 2) I propose that the generic driver uses virtualization (based on nova) and a block device backend (based on cinder)
15:12:24 <bswartz> vponomaryov: I have yet to look at the LXC code that's been written, but if it works I don't want to ignore it
15:12:32 <caitlin56> bswartz: and implements generic NFS/CIFS from stock linux?
15:12:55 <vponomaryov> bswartz: yes, it works, tested
15:13:02 <bswartz> vponomaryov: As I mentioned above, everything related to the generic driver is low priority in the short term though
15:13:20 <vponomaryov> for each implementation of protocols can be written class-helper
15:13:43 <caitlin56> yponmaryov: but isn't setting up things like tenant specific DNS take extra work (even though I have seen scripts that demonstrate how).
15:13:44 <vponomaryov> but not tied yet with neutron code
15:13:46 <bswartz> vponomaryov: I'll look at it soon, the other 2 changes are more interesting however
15:14:10 <jcorbin> vponomaryov: First code review link works, other 2 give permission errors. Is this expected?
15:14:37 <bswartz> yes it seems that the submissions are marked private currently
15:15:25 <vponomaryov> jcorbin: yes, it hat restricted access, at least, for now
15:16:10 <bswartz> vponomaryov: please add access to jcorbin as he's familiar with neutron and I'd like his feedback as well
15:16:38 <bswartz> anyone else who's curious please ping vponomaryov too
15:17:05 <navneet> mee too
15:17:09 <xyang1> vponomaryov: please add me too
15:17:16 <vbellur> vponomaryov: me too
15:17:19 <vponomaryov> jcorbin: provide your username in launchpad
15:17:52 <bswartz> so the other thing that's clear from the conference is that there is strong interest in the "tunneled" style of access, expecially for backends like ceph and gluster
15:17:57 <glenng> vponomaryov: My Launchpad name is ggobeli
15:18:04 <bswartz> Any IBM/GPFS folks here?
15:18:19 <jcorbin> vponomaryov: My Launchpad name is osjcorbin. Thanks!
15:18:26 <bill_az> vponomaryov:  also me - billowen@us.ibm.com
15:18:28 <bswartz> vponomaryov: it looks like *Everyone* is interested may as well mark it public
15:18:31 <xyang1> vponomaryov: my launchpad name is: xing-yang
15:18:34 <bill_az> And I'm IBM/GPFS
15:18:56 <bswartz> bill_az: have you given any thought to the multitenant access issues?
15:18:59 <vbellur> vponomaryov: my launchpad id is vbellur
15:19:08 <bswartz> bill_az: is GPFS going to be in the same boat as ceph and gluster?
15:19:18 <vponomaryov> bswartz: I can inform our devs, I am not the owner of gerrit commit
15:19:28 <bswartz> vponomaryov: ty
15:19:29 <bill_az> bswartz:  I have been discussing it with others on the team
15:19:51 <bill_az> bswartz:  we have some things coming in gpfs 4.1 in the spring that will help w/ multitenancy
15:19:52 <caitlin56> bswartz: by "tunneled" do you mean some form of "virtFS"?
15:19:53 <navneet> mine is singn
15:20:28 <bswartz> So just to recap, many clustered filesystems such as ceph and gluster doesn't have facilities to provide "virtual servers" on different tenant networks
15:20:56 <bswartz> These clustered filesystems assume full network connectivity between the client and every server node
15:21:29 <caitlin56> bswartz: I believe the fundamental issue is who deals with tenant users. The "virtFS" approach puts this on the Hypervisor, the virtual server approach puts it on the file server.
15:21:54 <bswartz> caitlin56: yes
15:22:06 <gregsfortytwo1> that's part 1, part 2 is the networking constraints imposed by the full connectivity bswartz mentioned
15:22:26 <caitlin56> My evaluation is that relying on the hypervisor a) will add a minimum of one year to the project and b) doesn't truly secure the files even then.
15:22:43 <gregsfortytwo1> bill_az, I assume gpfs also requires connectivity to every node, right?
15:22:50 <bswartz> so for clustered filesystems that aren't able to create a virtual-server-per-tenant, due to the number of network interfaces involved, we need a hypervisor-mediated approach to secure multitenancy
15:23:24 <bswartz> that could mean virtfs, or NFS-over-PPP, or NFS-over-vsock
15:23:26 <caitlin56> Nobody is *unable* to do a virtual server per tenant, they just haven't done it yet.
15:23:41 <bill_az> gregsfortytwo1:  yes -thats correct
15:24:06 <gregsfortytwo1> caitlin56: if "server" — implying one — is nonsensical in your architecture you're unable
15:24:10 <bswartz> caitlin56: I think the issue is the number of network interfaces involved -- it's simply impractical
15:24:21 <caitlin56> Tunneling NFS with less-than-a-tenant-network sounds like a simpler approach than virtFS.
15:24:26 <gregsfortytwo1> we are all able to implement multienancy even if we haven't, but that's not the biggest issue from Manila's perspective
15:24:28 <bill_az> gregsfortytwo1:  but we plan to limit access through isolated filesystems and encryption keys
15:24:37 <bswartz> if you have a 200 node ceph cluster, you can't create 200 additional virtual network interfaces for every tenant
15:25:26 <caitlin56> But even with tunneled NFS, the server is going to have to do per-tenant authentication, which I presume they are also incapable of doing today.
15:25:28 <gregsfortytwo1> bill_az: yeah, that's essentially the path Ceph is walking down, though not very quickly
15:26:09 <bswartz> caitlin56: the theory would be to create a NFS-to-ceph bridge, or an NFS-to-glusterFS bridge
15:26:25 <bill_az> gregsfortytwo1:  it will be tough for us for icehouse - but should have that for j
15:26:25 <bswartz> the NFS itself would have no authentication
15:26:37 <caitlin56> That's fine. That's a vendor provided proxy. Effectively a virtual server.
15:26:44 <gregsfortytwo1> caitlin56: no, we can handle that already in most of the stack; not sure about gluster; sounds like gpfs can/will
15:26:44 <bswartz> fortunately bad security is something NFS is very good at
15:27:27 <caitlin56> gregfortytwo: my main concern is avoiding an openstack managed common UID/GID pool that everyone maps to.
15:27:27 <vbellur> gregsfortytwo1: gluster can handle that too
15:29:03 <gregsfortytwo1> caitlin56: ah, hadn't even gotten to that issue yet in my head :)
15:29:10 <bswartz> okay so I'm glad we're all thinking about the issue here -- tunneled FS support will need to come soon after the full network connectivity style of multitenant support
15:29:18 <bswartz> issues*
15:29:49 <caitlin56> question: does it need to be a lib, or can it just be a NAS proxy?
15:29:51 <bswartz> those are the 2 main roadblocks to allowing various storage vendors to write functional backends
15:30:38 <bswartz> the actual implementation of the bridge/proxy will need to run on the compute node -- whether it's a manila-agent, or changes to nova, or some library -- I'm not sure about taht
15:30:56 <bswartz> for a prototype we should keep the code in manila
15:31:25 <bswartz> long term it might make sense to host it elsewhere -- especially if it's generically useful outside of manila
15:31:30 <caitlin56> So this would be a pluggable library that ran in the same context as the manilla agent?
15:31:58 <bswartz> I want to be careful with the term "manila agent" because it could mean a lot of different things
15:32:18 <caitlin56> You would want it to be on a path that made sense for bulk data transfer - preferably "close" to the guest.
15:32:43 <bswartz> the idea of automatic mounts inside the nova instances is still interesting to many -- and those may necessitate a different kind of agent
15:33:13 <bswartz> but absolutely it can/should run on the compute node
15:33:43 <bswartz> possibly "manila-bridge" or "manila-gateway" would be a more descriptive term
15:33:51 <caitlin56> bswartz: proxying TCP connections is something compatible with the current infrastructure. Passing full file-systems through hypervisors will be a lot more work.
15:34:13 <bswartz> caitlin56: we're talking about the later not the former
15:34:37 <bswartz> I'm aware that it will be real work
15:34:39 <caitlin56> bswartz: distinguishing between manila-bridge (control plane) vs. manila-gateway (user-plane) sounds like a good idea to me.
15:35:32 <bswartz> anyways today I just wanted to lay out the priorities
15:36:02 <bswartz> 1) get the neutron code working so we can do full network multitenancy
15:36:25 <caitlin56> I first looked at NAS proxies for VMs two years ago, and concluded that intercepting at the network layer was more feasible than at the FS-operation layer.
15:36:30 <bswartz> 2) implement some tunneling/virtfs code to support mediated multitenancy
15:36:40 <bswartz> 3) implement the generic driver
15:36:58 <caitlin56> I feared that even if Xen and KVM adopted "virtFS" it would take two years to deploy. In those 2 years there has been no progress on adopting a virtFS API.
15:37:29 <bswartz> caitlin56: I'd be curious to know why that is -- let's take that discussion offline
15:37:51 <bswartz> regarding virtfs -- if that turns out to be a blocker than NFSv3-over-vsock may be a faster path to get something working
15:38:37 <gregsfortytwo1> s/offline/to #openstack-manila? :)
15:38:46 <caitlin56> Let each proxy decide whether to support NFSv3 and/or NFSv4.
15:38:56 <bswartz> there are lots of technologies that can be applied here -- I'm confident we'll find something that works
15:38:57 <caitlin56> or for that matter CIFS.
15:40:14 <bswartz> caitlin56: we're going to have to write these proxies ourselves though -- I'm not eager to take on tons of work
15:40:39 <caitlin56> Correct, these are pluggable proxies. Someone has to provide their own proxy.
15:40:54 <bswartz> If NFSv3 works as a bridge protocol, I think it will be hard to argue that there's a significant advantage to using something like cifs or v4
15:41:11 <caitlin56> Am I correct that neither CEPH or Gluster would want soeone else writing the proxies to access their networks?
15:41:34 <bswartz> caitlin56: I think you're incorrect
15:41:52 <bswartz> I should be the job of manila to provide the glue
15:41:56 <gregsfortytwo1> no reason to independently implement setting up an nfs server instance ;)
15:42:19 <bswartz> my vision is to implement one thing that works for everyone
15:42:44 <caitlin56> So you want a manila coded proxy that maps UID/GIDs for NFSv3 to NFSv3?
15:42:52 <bswartz> once we have a gateway of sorts that runs on the compute node, it shouldn't matter what's on the actual backend -- as long as the compute node can mount it, it can be tunneled through to the guest
15:43:30 <bswartz> caitlin56: I don't want any UID/GID mapping to occur
15:43:47 <bswartz> if we do this correctly, it should be safe to do no mapping
15:43:48 <caitlin56> Another form of proxy would translate Any-NAS-Protocol to My-backends-proprietary-protocol.
15:44:04 <aostapenko> in lxc implementation we can mount directly to vms
15:44:37 <aostapenko> we will can :)
15:44:58 <bswartz> okay so we have some conflicting ideas here
15:45:19 <bswartz> maybe my vision will not work out
15:45:26 <bswartz> can we at least agree on the priorities I mentioned?
15:46:00 <bswartz> those who are interested in the tunneling approach should probably form a working group and hash out the issues
15:46:09 <caitlin56> If you aren't doing UID/GID mapping then you would have to trust the gateway to do mount authentication. Then all you need to map to a universal pool are the mounts.
15:47:06 <bswartz> I'm interested in the tunneled approach working correctly, so I'll organize those discussions
15:47:38 * caitlin56 is interested in both models - could see implementing either on our end.
15:47:51 <bswartz> I suspect that we might be making this harder than it needs to be though
15:48:29 <bswartz> My instinct is that we should write a prototype first, and then discover all the ways that it's wrong
15:48:31 <vbellur> bswartz: priorities seem fine to me
15:48:35 <caitlin56> You can make it simpler if you just state the caveat that a gateway solution requires trusting the gateway.
15:48:58 <bswartz> caitlin56: oh yes, absolutely
15:49:23 <vbellur> bswartz: +1 to the prototype
15:49:34 <bswartz> caitlin56: the gateway approach is akin to how you must trust the hypervisor to faithfully present block devices to you
15:49:37 <bill_az> bswartz:  +1 - get the simlest --realistic-- prototype working asap as a model and to help discussion
15:49:50 <jcorbin> bswartz: +1 to the prototype
15:50:28 <bswartz> this "manila-gateway" thing would be running on the compute node, and it would be part of the cloud infrastructure -- hidden from the tenant's view
15:50:30 <caitlin56> +1, but we should not wait onthe multi-tenant network solution.
15:50:33 <gregsfortytwo1> the gateway would be essentially the generic implementation, plus some holes that different drivers can plug in order to set up their own backends, right?
15:50:39 <bswartz> the tenant would have no choice but to trust it
15:51:20 <vbellur> gregsfortytwo1: sounds like a good model
15:51:26 <caitlin56> bswartz: the multi-tenant network approach with virtual servers avoids needing to trust the compute host.
15:51:40 <bswartz> gregsfortytwo1: the gateway is really a get-out-of-jail-free card for backends that can't connect themselves directly to tenant networks on account of the limitted-number-of-network-interfaces problem
15:52:06 <bswartz> caitlin56: yeah I believe that's the best method for backends than can support it -- it's just that not all can
15:53:06 <bswartz> in fact it's come to my attention that even some more traditional NFS-based storage devices (I'm thinking of Isolon) can't directly support multitenancy, and may need something like this to work in public cloud environments
15:53:17 <caitlin56> bswartz: if the server trusts the gateway to mount only exports approved by the correct authentication server then you only need a single LAN netowrk between the compute hosts and the file servers.
15:53:56 <bswartz> caitlin56: yes that's the idea
15:54:03 <bswartz> okay since we're nearly out of time
15:54:09 <bswartz> #topic open discussion
15:54:23 <bswartz> does anyone else have another topic that I didn't bring up?
15:55:31 <bswartz> wow that quieted things down
15:55:31 <bill_az> bswartz:  is there a summary of manila activities at summit?
15:55:49 <bswartz> bill_az: yes
15:56:03 <bswartz> as I mentioned at the beginning, we have a unconference session which went well
15:56:13 <bswartz> we had* a unconference session
15:56:47 <bswartz> and we also got together for drinks and I managed to get my boss to pay the bar tab
15:56:48 <bill_az> bswartz: sorry - i missed the first few minutes - is there minutes or summary published?
15:57:16 <bswartz> thanks to everyone who showed up -- it's always great to meet face to face before we go off and spend 6 months coding together
15:57:22 <vbellur> bswartz: thanks for the drinks! :)
15:57:22 <bill_az> bswartz:  sorry I missed both!  (especially drinks...)
15:58:14 <bswartz> and I can't tell you how many people grabbed me personally in the hallways to talk about manila
15:58:23 <bswartz> the interest level is high
15:58:34 <bswartz> we need to move fast while we have momentum
15:58:37 <vbellur> bswartz:  +1
15:59:14 <bswartz> I'll be talking to TC members about the incubation request -- we should have an answer on tuesday
15:59:31 <bill_az> bswartz:  that all sounds good - Thanks
15:59:33 <bswartz> and with that we're out of time
15:59:35 <glenng> Let's hope,
15:59:36 <bswartz> thanks everyone
15:59:42 <vbellur> thanks all
15:59:51 <bswartz> #endmeeting