16:03:10 <pc_m> #startmeeting vpnaas
16:03:11 <openstack> Meeting started Tue Aug 18 16:03:10 2015 UTC and is due to finish in 60 minutes.  The chair is pc_m. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:03:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:03:15 <openstack> The meeting name has been set to 'vpnaas'
16:03:20 <pc_m> #topic Announcements
16:03:58 <pc_m> It looks like stable/kilo jobs are working now, as the q-vpn service has been enabled for older releases.
16:04:37 <amotoki> cool. thanks for debugging the problem.
16:04:57 <pc_m> There are still more fixes being done, getting neutron to work, when adv svc plugins missing.
16:05:21 <pc_m> I still need to touch base with dougwig to find out what is left, but I think we are out of panic-mode :)
16:05:40 <pc_m> The local tunnel IP change has upstreamed.
16:05:50 <pc_m> Anyone have other announces for the team?
16:06:06 <pc_m> announcements
16:06:38 <madhu_ak> yeah. waiting for dougwig to look my comments for gate jobs via infra
16:07:08 <madhu_ak> #link: https://review.openstack.org/#/c/211767/
16:07:53 <pc_m> Thanks. I'll peek at it too.
16:07:56 <madhu_ak> likewise, this one can be tested once the above goes in: #link https://review.openstack.org/#/c/211768/
16:08:39 <madhu_ak> once it is properly tested, #link, this one can test: #link https://review.openstack.org/#/c/211381/
16:09:05 <madhu_ak> pc_m addressed your comments for migrating tempest dependancies
16:09:23 <pc_m> OK. Will look back at them.
16:09:23 <madhu_ak> #link: https://review.openstack.org/#/c/211778/
16:09:40 <pc_m> Everyone, this is to move API tests from neutron to VPN.
16:10:00 <pc_m> #topic Devstack Plugin
16:10:49 <pc_m> I have a pair of commits, the first is to modify devstack to allow q-vpn or neutron-vpnaas as service names, and the second is to rename the devstack plugin to neutron-vpnaas
16:11:28 <pc_m> First is upstream, second is out for reivew - #link https://review.openstack.org/#/c/213253
16:11:54 <pc_m> Currently, it works for tests, but we end up with two q-vpn agent processes running. This fixes that.
16:12:19 <pc_m> #topic VPN Functional Tests for Neutron Commits
16:12:48 <pc_m> No work on this. Been fighting fires on the devstack plugin. Will look at, later, once things calm down more.
16:13:05 <pc_m> #topic Multiple Local Subnets on VPN connection
16:13:28 <pc_m> Please look at the review of dev reference doc #link https://review.openstack.org/#/c/191944
16:14:14 <pc_m> I've been working on implementation of an endpoint-group API, and have the create done. Will push out for early review soon, and will continue with rest of CRUD for that.
16:14:45 <pc_m> Just implementing subnet, CIDR, and VLAN as group types for now. Others can be added later as needed.
16:15:44 <pc_m> #topic Reviews and Bugs
16:16:07 <pc_m> Take a look at the open bugs and reviews and please help out. Reviews at #link https://review.openstack.org/#/q/status:open+project:openstack/neutron-vpnaas,n,z
16:16:50 <skolekonov> Hi guys, I have a question
16:17:01 <pc_m> For many of these commits that involve infra changes, we need to get team/neutron core review, before having infra look at things, so any help in reviews is going to move things along smoother.
16:17:07 <pc_m> skolekonov: sure
16:17:13 <skolekonov> Could you please take a look at this bug #link https://bugs.launchpad.net/neutron/+bug/1481329 ?
16:17:13 <openstack> Launchpad bug 1481329 in neutron "VPN agent works incorrectly with DVR on multinode setup" [Undecided,New] - Assigned to Elena Ezhova (eezhova)
16:17:19 <skolekonov> I want to clarify the situation around VPNaaS compatibility with DVR.
16:17:54 <pc_m> skolekonov: Have you talked with DVR folks at all on this by any chance?
16:18:18 <pc_m> skolekonov: May want to talk to carl_baldwin to see who could answer questions best.
16:18:38 <skolekonov> No, I haven't, but I've asked my question in openstack-dev - http://lists.openstack.org/pipermail/openstack-dev/2015-August/071364.html
16:18:54 <pc_m> skolekonov: My understanding was that VPN could only be used on the edge router.
16:19:06 <pc_m> North/South router.
16:19:29 <pc_m> skolekonov: Not sure if there is anything to prevent it from be used elsewhere.
16:20:01 <amotoki> Is FIP associated when the problem occurs?
16:20:13 <pc_m> skolekonov: Check with Carl or Maybe Brian Haley, as they can give you good direction on a DVR point of view.
16:20:24 <skolekonov> It also seems like VPNaaS doesn't work when instances have floating IPs in DVR mode
16:20:54 <amotoki> anyway, it would be helpful you describe the detail on how to reproduce the problem.
16:21:09 <pc_m> amotoki: +1
16:21:35 <skolekonov> I did it in email thread generally
16:22:26 <skolekonov> so full compatibility with DVR is not on vpnass team roadmap, right?
16:22:59 <skolekonov> *vpnaas, sorry
16:23:04 <amotoki> #link http://lists.openstack.org/pipermail/openstack-dev/2015-August/071364.html
16:23:52 <Aish> https://review.openstack.org/#/c/213781/  is missed in the list pc_m posted before.  Both that and  https://review.openstack.org/#/c/213819/1 are to include rally scenario tests for vpn. And are WIP. But any suggestions at this stage will be helpful. pc_m: I looked into your comments. Will be addressing them in my next patch.
16:24:26 <pc_m> skolekonov: So, if with DVR you have a router with a GW IP you cannot establish a VPN connection with another cloud?
16:25:04 <skolekonov> pc_m, yep
16:25:06 <pc_m> Aish: You probably want to add a vpnaas tag to the reviews so that VPN team sees them.
16:25:37 <Aish> sure will do. pc_m
16:25:51 <pc_m> skolekonov: Don't know. Really haven't tried VPN with DVR at all. So, no, it is not on the roadmap.
16:26:15 <skolekonov> pc_m, I see, thank you
16:26:17 <pc_m> skolekonov: Would you like to champion that effort?
16:26:39 <amotoki> any help is really welcome. no so many folks including me are running DVR :-(
16:28:03 <skolekonov> pc_m, Actually, I don't have much experience with OpenStack code, more with deployment part. So I'm not sure
16:28:27 <pc_m> skolekonov: So I'd advise several things. One is to contact the DVR folks and ask them about the issue. You can contact them at the L3 meeting worst case.
16:29:20 <bharath> skolekonov pc_m: Currently vpn works with DVR however getting it to work with FIP may be an issue
16:29:38 <pc_m> Two is to see if there is community interest in providing that type of capability. We need someone to step up to want to drive that effort.
16:30:39 <pc_m> bharath: thanks. SO may be an issue with the implementation.
16:30:51 <pc_m> using FIPs.
16:31:46 <pc_m> skolekonov: Getting back to what amotoki mentioned. It may help to write up the exact scenario/topology that you are trying so that people can look at the issue.
16:31:56 <pc_m> helpful
16:32:02 <bharath> Indeed pc_m. In the case of DVR, as the snat is made a separate namespace from qrouter, the router gateway IFace sits in the former namespace and thus VPN is configured in SNAT
16:33:08 <skolekonov> pc_m, I'll add details to the thread
16:33:20 <pc_m> skolekonov: thanks
16:33:34 <pc_m> bharath: Good info!
16:34:23 <pc_m> Are there any other bugs/reviews that people want to highlight in the meeting?
16:35:37 <pc_m> #topic Open Discussion
16:35:40 <ajmiller> I have an open discussion item.
16:35:48 <pc_m> ajmiller: good timing :)
16:36:14 <pc_m> ajmiller: go ahead
16:36:16 <ajmiller> We have been doing a security review for our deployment of Openstack, and have found a couple of possible issues with VPNaaS
16:36:32 <pc_m> :(
16:36:55 <ajmiller> the 'ipsec' processes are running as root. and thus we are concerned that this introduces the first case of a single vulnerability enabling a remote root exploit of the network services nodes.
16:37:43 <ajmiller> We have been trying to ensure that two vulnerabilities are necessary to enable a remote root exploit. Why does pluto run as root? Is there a way to run pluto as a non-root user?
16:38:47 <ajmiller> The concern is that this is an outward-facing service, so if someone could exploit a vulnerability in pluto, they are running free in the control plane node.
16:39:23 <amotoki> as far as i understand, there is no specific reason to run pluto as root. it is just because we need root to run 'ip netns exec'.
16:39:42 <amotoki> we can improve the command line.
16:40:04 <bryan_> so a better design is to oinly run as root the small bit that needs root access, and call that from the bulk of the code running as non-root.
16:40:07 <pc_m> ajmiller: Don't know for sure.
16:40:09 <bryan_> I work with ajmiller at HP.
16:40:14 <ajmiller> amotoki, yes, and we should be able to use "rootwrap" or somesuch to do that?
16:40:25 <bryan_> yes, there are many solutions
16:40:27 <amotoki> though I haven't investigated...
16:41:24 <bryan_> since neutron, nove, glance, swift, and everything else I can find run as their own users, I feel strongly that ipsec pluto should also
16:41:28 <pc_m> it is currently using rootwrap.
16:41:40 <ajmiller> ok
16:41:50 <bryan_> why, if it is running as root, does it need rootwrap?
16:41:54 <pc_m> bryan_: Neutron doesn't use rootwrap at all?
16:42:08 <bryan_> I don't know, but the processes run as non-root users
16:42:38 <bryan_> that is, the processes I can find with ps
16:42:41 <amotoki> I think metadata-proxy is run as non-root now (previously as root)
16:42:57 <bryan_> There may be ephemeral processes running as root I don't see in the ps output.
16:42:58 <amotoki> so we can take a similar approach for pluto.
16:43:10 <bryan_> I think so.
16:43:29 <ajmiller> amotoki, so if we look at metadata-proxy we might be able to find some history and/or guidance?
16:43:30 <pc_m> Only speculating, but I think it runs rootwrap daemon to setup namespace and then invokes the oswan process.
16:43:54 <bryan_> so oswan process must run as root for VPNaaS to work?
16:44:02 <bryan_> It is the "small bit" that needs root access?
16:44:21 <amotoki> ajmiller: i think so.
16:44:44 <bryan_> ok, so has any special review been done on that code?  This is the first time we're running as root an listening on an untrusted network
16:44:52 <bryan_> I think
16:45:08 <bryan_> so we've created a prime attack target, we need to ensure it is adequately defended
16:45:19 <pc_m> bryan_: I don't know if it is needed to work. I'm just saying that is what it does, from what I recall.
16:45:40 <bryan_> OK, so we should detemrine if it is needed.  If not lose it; if so harden it
16:47:35 <amotoki> from the point of security vulnerabilty process, if you have potential concern, you can file a bug in a private mode.
16:47:40 <pc_m> bryan_: yes. I wasn't involved in the original effort, so I don't know why all the root access is done.
16:48:14 <bryan_> who was?  I'd like to talk with them, maybe there is a good reason
16:48:23 <bryan_> maybe it's just an oversight
16:48:25 <pc_m> Nachi Ueno.
16:48:31 <bryan_> I want to at least ask the question.
16:48:35 <bryan_> thank you
16:48:42 <pc_m> Not sure if he is involved much in OS any more.
16:48:58 <ajmiller> I can probably test running the ipsec process as non-root in a test env.
16:48:59 <bryan_> hey, things follw me around forever, so I'll ask him.
16:49:18 <pc_m> ajmiller: That would be nice.
16:49:24 <amotoki> I don't have a exact link, but I believe you can find an option "security bug" or private when reporting a bug.
16:49:28 <bryan_> yeah, maybe it just works.
16:49:35 <ajmiller> amotoki thanks
16:49:40 <bryan_> In any case, we'll find out what breaks and know how the design might be changed
16:49:48 <pc_m> I think we need to create a namespace for the config files, and then run the process.
16:50:10 <amotoki> but note that openstack security ml can be subscribed by anyonne interested in, so it is not perfect.
16:50:10 <pc_m> I'd imagine the namespace create is a root action.
16:50:27 <ajmiller> pc_m it is
16:50:31 <pc_m> Also there is some subtle difference between OSwan and SSwan.
16:50:55 <ajmiller> pc_m that brings up a question I have.  Is the future with SSwan?
16:51:04 <pc_m> They do some extra steps in SSwan (which I don't fully understand).
16:51:07 <pc_m> ajmiller: yes
16:51:20 <ajmiller> ok, thanks
16:51:30 <pc_m> OSwan will not be supported in Ubuntu (15.10+ or something like that).
16:51:59 <pc_m> Now Fedora and others support SSwan (or Libraswan), so it'll be the new reference impl in the future.
16:52:08 <ajmiller> got it
16:52:35 <pc_m> You can look in the device drivers for the rootwrap stuff going on.
16:52:54 <pc_m> That's one area I never dug into (or wanted to :)
16:53:22 <ajmiller> I'm only vaguely familiar with that.
16:53:52 <pc_m> My understand is that the VPN is handled in the kernel, so I don't know if these processes need to run as root to configure/control.
16:54:11 <bryan_> al, I can help look into this if you like
16:54:38 <pc_m> IOW I think pluto just controls the VPN implementation that is running in the kernel.
16:54:59 <pc_m> Would be interested in hearing what you guys find out.
16:55:16 <bryan_> is pluto getting traffic from the network?
16:55:17 <ajmiller> pc_m, bryan_ and I will look into thi
16:55:19 <amotoki> /fyi/ re: security vulnerabilty namanagement process, this page describes the process and so on: https://wiki.openstack.org/wiki/Vulnerability_Management    I think "Process" link is the entry point.
16:55:20 <ajmiller> this
16:55:32 <pc_m> #action ajmiller to try using VPN w/o root mode - bryan_ to help
16:55:45 <pc_m> amotoki: thanks
16:56:43 <amotoki> anyway I agree we should run any process or command with non-root user as possible as we can.
16:56:52 <pc_m> Thanks for looking into this ajmiller and bryan_!
16:57:02 <amotoki> thanks too
16:57:07 <bryan_> you are welcome.  We'll let you know what we find.
16:57:14 <pc_m> Looks like we're just about out of time. Anything else?
16:57:23 <ajmiller> thats it for me.
16:57:51 <pc_m> Thanks for joining in everyone!
16:57:54 <pc_m> #endmeeting