22:00:18 <adrian_otto> #startmeeting containers
22:00:19 <openstack> Meeting started Tue Jun 17 22:00:18 2014 UTC and is due to finish in 60 minutes.  The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:00:22 <openstack> The meeting name has been set to 'containers'
22:00:24 <adrian_otto> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2014-06-17_2200_UTC Our Agenda
22:00:26 <erw> o/
22:00:27 <Slower> Ian Main o/
22:00:29 <adrian_otto> #topic Roll Call
22:00:30 <funzo> Chris Alfonso
22:00:32 <adrian_otto> Adrian Otto
22:00:43 <meghal> Meghal Gosalia
22:01:06 <dguryanov> Dmitry Guryanov
22:01:16 <erw> Eric Windisch
22:01:20 <jtaleric> Joe Talerico
22:01:36 <songole> Subra Ongole
22:02:25 <adrian_otto> welcome everyone
22:02:54 <adrian_otto> if you have not yet recorded your attendance, you may chime in at any time
22:03:04 <adrian_otto> #topic Announcements
22:03:05 <apmelton> Andrew Melton 0/
22:03:16 <adrian_otto> would the team members like to make any announcements?
22:03:38 <adrian_otto> #topic Review Action Items
22:03:40 <adrian_otto> (none)
22:03:49 <adrian_otto> #topic Mid-Cycle Meetup for Containers
22:04:16 <adrian_otto> Yahoo! has offered to host a mid-cycle meetup for us in the SF Bay area.
22:04:38 <adrian_otto> I'd like you all to have input on what dates we should plan this for
22:04:47 <adrian_otto> and gather consensus about what number of days is optimal
22:05:33 <adrian_otto> thoughts on this?
22:05:57 <erw> number of days first, I think.
22:05:58 <adrian_otto> something in early Aug might make sense
22:06:03 <adrian_otto> ok, number of days
22:06:13 <adrian_otto> ideas ranging from 1-3 have been offered in prior discussions
22:07:13 <Slower> I vote winter.. because I'm from canada
22:07:38 <adrian_otto> having a couple of days together would give us a chance to enter detailed design efforts
22:07:42 <Slower> that's a joke
22:07:48 <adrian_otto> Slower: ;-)
22:08:00 <erw> I’m thinking we only need two, but I don’t mind going longer if others feel it necessary
22:08:40 <adrian_otto> who would need to travel a long distance to attend an SF area meetup? I know erw would come from PA.
22:08:47 <adrian_otto> I would come from LA
22:08:54 <apmelton> I'll be coming from VA if I attend
22:08:56 <Slower> I can fly out of spokane WA
22:09:35 <adrian_otto> ok, so at least a few of us would travel. Is a shorter trip easier to clear with your travel budgets? If so, we should consider that too
22:09:59 <adrian_otto> if we can have a healthier group for a shorter time, I'd prefer that to a smaller group for longer
22:10:33 <adrian_otto> ok, so how about dates to avoid?
22:10:40 <apmelton> first week in AUG for me
22:11:01 <funzo> sept 12-21 i'll be away
22:11:07 <Slower> august 16-31 I'm away
22:11:21 <adrian_otto> Jul 11-24 I am on vacation
22:11:29 <Slower> probably with a few days on each end
22:11:55 <apmelton> Also Nova meet up is July 28th to 30th I believe
22:11:57 <marcoemorais> adrian_otto: we should prolly bias towards meeting sooner rather than later, once you get into mid-july lots of ppl leaving for vaca and such
22:12:11 <erw> Philadelphia… but if it’s at Yahoo, then I can spend the other days at the office
22:12:41 <erw> I don’t yet know of any conflicts — other than the Nova meetup
22:13:01 <marcoemorais> apmelton: where is the nova meetup being held?
22:13:10 <adrian_otto> So if we did July 31 and Aug 1
22:13:11 <apmelton> Beaverton, OR (portland)
22:13:20 <adrian_otto> oh, well.
22:13:33 <adrian_otto> who is hosting that one?
22:13:43 <apmelton> intel if I remember correctly
22:13:45 <erw> Intel
22:13:56 <marcoemorais> https://www.eventbrite.com.au/e/openstack-nova-juno-mid-cycle-developer-meetup-tickets-11878128803
22:14:12 <adrian_otto> what about doing one weekday, and one Saturday
22:14:17 <funzo> adrian_otto: i don't know yet if I can for sure get the budget, but if I do - the end of july also doesn't work for me
22:14:28 <funzo> so many vacations, so little time
22:14:49 <adrian_otto> yeah, summer meetups are really tricky
22:14:53 <Slower> I vote weekdays
22:15:19 <apmelton> Yea, if I'm flying to/from the West Coast, I prefer weekdays as well
22:15:20 <adrian_otto> ok, so how about in the range of Aug 11-15?
22:15:36 <apmelton> I can do Aug 11-15
22:15:42 <adrian_otto> maybe Tue Aug 12, and Wed Aug 13
22:15:50 <erw> adrian_otto: weekdays are better for me, but I can be flexible.
22:15:51 <funzo> that would work for me
22:16:42 <adrian_otto> I have a trip planned for those dates, but I's likely to be cancelled, so that might be ok
22:16:44 <funzo> I haven't paid any attention to a mid-cycle meetup until just now, so I haven't made any budget requests
22:16:57 <marcoemorais> I am on vacay week of 8/11
22:17:09 <funzo> if I can't make it out, i'd be interested in g+ hangout
22:17:19 <Slower> I need to be in vermont on aug 16th..
22:17:28 <Slower> well if all goes well
22:17:32 <adrian_otto> ok, how about Aug 18-22
22:18:28 <adrian_otto> The trove Mid-Cycle meetup is Aug 20-22 in Cambridge, MA
22:18:28 * erw pulls up the roadmap
22:18:29 <Slower> I'm on vac August 16 - 31
22:18:40 <adrian_otto> but I'm pretty sure that won't be a problem for us
22:19:21 <erw> we’re already into feature proposal freeze by then
22:19:53 <adrian_otto> 4th of july week is usually impossible
22:20:12 <adrian_otto> but… let's check that week
22:20:16 <apmelton> I'm on vacation week of july 4th
22:20:18 <adrian_otto> the 4th is on a friday
22:20:52 <adrian_otto> any others?
22:21:21 <erw> no strong objection, although I’m sure my family will kill me.
22:21:37 <adrian_otto> I would miss my wife's birthday
22:21:45 <adrian_otto> I love you all, but not that much
22:21:46 <funzo> I'm drunk on that day
22:21:48 <adrian_otto> heh
22:22:10 <Slower> funzo: well you may as well be in a meeting then..
22:22:24 <adrian_otto> ok, so let's table this, I'll run a doodle poll on it, and let's find a time when we can at least get some of us together
22:22:40 <adrian_otto> watch the ML for that link
22:22:48 <Slower> I liked the 14th 15th idea
22:22:58 <Slower> aug
22:23:05 <adrian_otto> k
22:23:18 <erw> doodle ++
22:23:25 <adrian_otto> #topic Cinder Support Discussion
22:23:28 <erw> also gives us time to sync with SOs
22:23:56 <adrian_otto> se had a good collaboration this week exploring the options in this wiki:
22:24:04 <adrian_otto> #link https://etherpad.openstack.org/p/container-block-storage Options for Cinder Support and unprivileged Container-in-Container
22:24:17 <adrian_otto> the idea is to find options that might kill two birds with one stone
22:24:26 <adrian_otto> let's take a moment to scan through that
22:25:21 <funzo> scanned
22:25:45 <adrian_otto> ok, are there any additional options that should be considered in addition to these?
22:26:02 <erw> adrian_otto: a variant of the API solution...
22:26:16 <adrian_otto> Option 1?
22:26:16 <erw> where the container uses an OpenStack API to initiate the mount
22:26:31 <erw> yes, variant of option #1
22:26:41 <adrian_otto> is that different from Option 5?
22:26:57 <erw> it’s the hybrid between #1 and #5
22:27:12 <adrian_otto> ok, explain further, please
22:27:53 <erw> it would be a nova extension for mounting filesystems, not a docker one.
22:27:54 <adrian_otto> maybe list that as option 11
22:28:03 <erw> yeah, I can add it in.
22:28:06 <adrian_otto> ok
22:28:23 <adrian_otto> so I'd like to have consensus around ruling some of these out
22:28:35 <adrian_otto> to narrow the list to ones we really like
22:28:42 <funzo> so should we just move forward with #8 in the short term? - then work on one of the other options...?
22:28:57 <funzo> i'm concerned that most of these options has quite a runway on them
22:28:57 <adrian_otto> great minds think alike
22:29:13 <adrian_otto> so let's look at #8 for a moment
22:29:28 <funzo> essentially providing the block device and that's it, right?
22:29:32 <adrian_otto> #8 will allow you to expose a cinder volume to a container
22:30:01 <funzo> which we have done in docker and as a POC in nova-docker (need to do a lot of refactoring before it's suitable for upstream)
22:30:07 <adrian_otto> but the container will not be able to mount it unless it is running in privileged mode, which they won't unless that is intentionally hacked by the cloud operator
22:30:14 <funzo> well... it could range from a little to a lot of refactoring
22:30:49 <funzo> is anyone against moving forward with that idea?
22:30:54 <Slower> container mount service?
22:31:00 <adrian_otto> one thing I am uncertain about is if we took this approach (assuming a first step in a sequence of these approaches), will the openstack CI need to be refactored, or not?
22:31:03 <Slower> just thinking out loud
22:31:13 <erw> I don’t think that #8 is really an “option” as much as it is, “don’t do this at all"
22:31:19 <erw> I mean, it’s technically the status quo.
22:31:42 <adrian_otto> erw, I don't think the whole Nova team fathoms that option
22:31:59 <adrian_otto> I think the initial impression is "cinder does not work, period"
22:32:02 <erw> adrian_otto: tempest might need changes to support it, yes.
22:32:26 <adrian_otto> ok, so assuming changes to tempest are needed, are members of this team willing to volunteer to make those changes?
22:32:34 <funzo> yes
22:32:39 <erw> yes
22:32:48 <adrian_otto> ok, so in that case, it's hard to argue against this as a first step
22:32:59 <adrian_otto> any alternate points of view to consider?
22:33:04 <funzo> option #8 doesn't really paint us into a corner
22:33:14 <funzo> we'll still have to decide on the next option
22:33:14 <erw> agreed - I wouldn’t block merging the code for #8 on filesystem mounting
22:33:36 <erw> since it’s a dependency for filesystem mounting, really. ;-)
22:34:22 <adrian_otto> ok, so any objections to recording this with a #agreed as a first step?
22:34:35 <adrian_otto> then we can explore what comes next
22:35:00 <erw> #agreed
22:35:11 <Slower> well #8 is status quo
22:35:13 <Slower> easy enough
22:35:36 <adrian_otto> well, I want something that would allow a merge of nova-docker back into nova
22:35:55 <erw> adrian_otto: I don’t think they’d block merging into nova based on partial support.
22:36:05 <erw> they might block merging based on not having full support
22:36:06 <adrian_otto> and I think at least a wiki that talks about cinder support and Containers with Nova
22:36:16 <adrian_otto> and break it down into sections LXC, Docker, etc.
22:36:33 <adrian_otto> right, this would assume an argument about what "full" support means
22:36:41 <erw> precisely
22:36:46 <adrian_otto> today if I use cinder to make a volume, it does not mount it for me
22:37:11 <erw> it doesn’t even make filesystems for you
22:37:15 <adrian_otto> right
22:37:20 <adrian_otto> it just gives me raw blocks
22:37:22 <erw> the contract has nothing to do iwth filesystems or mountability
22:37:30 <adrian_otto> right.
22:37:38 <erw> but some folks get hung up on implied contracts and “user experience"
22:37:40 <erw> ;-)
22:38:09 <adrian_otto> #agreed our first step for cinder support with Containers shall be addressed by option 8 in https://etherpad.openstack.org/p/container-block-storage
22:38:37 <adrian_otto> ok, so let's assume for a moment that we have some sensible documentation, and have moved passed arguments about this
22:38:49 <adrian_otto> which approaches can we rule out as a second step
22:38:51 <erw> adrian_otto: fwiw, I’ve been waving the banner to gather internal discussion at Docker on the other options, for step #2
22:39:07 <adrian_otto> erw, awesome, thanks!
22:39:11 <adrian_otto> anything new to share?
22:39:35 <adrian_otto> I propose we rule out option 6.
22:39:54 <erw> not yet.  It has a card in the big-bucket-of-things-we-know-about ;-)
22:39:58 <apmelton> adrian_otto: with user namespaces though, isn't allowing mount secure?
22:40:14 <adrian_otto> good question apmelton.
22:40:21 <funzo> erw: is there also still a card re snapshots and the registry communication with glance?
22:40:25 <erw> apmelton: there were comments on the mailing list that it isn’t.
22:40:28 <adrian_otto> I don't feel that I know user namespaces well enough to make that judgement
22:41:15 <erw> and that came from Bottomley, who knows better than most anyone else
22:41:15 <adrian_otto> I see that primarily as a UID mapping facility
22:41:19 <apmelton> so, I know from a high level how to use it, but at the kernel level (where the attack is going to happen) I'm still fairly new to it
22:41:32 <adrian_otto> I am not aware of containment of the CAP_XXX rights
22:43:00 <adrian_otto> ok, so any other thoughts about ruling out Option 6?
22:43:25 <apmelton> adrian_otto: from my understanding, it doesn't affect CAP_XXX, but if you're able to escalate yourself to root by exploiting mount some how, you will only be root from inside the containers context
22:43:49 <erw> adrian_otto: so my only positive support for #6 is that it is no worse than nova-baremetal/ironic and you can still run non-privileged containers *inside* of it.
22:44:07 <erw> adrian_otto: so the security impact, overall, is actually fairly small - from a certain perspective
22:44:19 <adrian_otto> I think the fear is that there are if (0 == uid) conditions in the kernel today
22:44:30 <adrian_otto> so being root in the container is still bad
22:45:33 <erw> adrian_otto: which, for that matter, we allow being root in the container with nova-docker already.
22:45:40 <adrian_otto> so unless there is a complete threat analysis on that…. maybe we can find one… then we should remain concerned
22:46:00 <adrian_otto> erw: please tell me more about that?
22:47:29 <erw> adrian_otto: docker lets you run as root inside the containers. Nova-docker just lets you run containers with Docker. It doesn’t force the user in those containers to be non-root
22:47:46 <adrian_otto> ok
22:47:50 <erw> anyway, there are people that WANT to use nova-docker as a baremetal hypervisor. Defaulting to priviledged containers would match their requirements. It would allow nested containers and mounting filesystems. Those are advantages of disregarding security
22:48:32 <erw> with, again, the idea that you could just run a docker-in-docker image that drops all priviledges and runs with limited SET_CAP and a constricted set of cgroups
22:48:32 <adrian_otto> so we are one step closer on this
22:49:16 <erw> the biggest negative is that people won’t know it’s insecure by default, and it doesn’t match how people typically use Docker.
22:49:42 <adrian_otto> erw: fair enough. That's something that could be addressed with documentation, and embedded warnings
22:50:12 <erw> long term, I think #6 is a bad idea, as we aim to do all these things without being priviledged
22:50:36 <Slower> #1 is the way to go forward right now imo
22:50:38 <adrian_otto> can we all agree that #6 is not our preferred long term outcome?
22:51:02 <erw> #agreed
22:51:02 <adrian_otto> I'd like to record that today as well, if that's how we all feel about it.
22:51:11 <Slower> #agreed
22:51:11 <adrian_otto> any objections?
22:51:20 <funzo> nope
22:51:34 <Slower> nop
22:51:44 <erw> adrian_otto: I might like to add that we might want to consider adding an admin-only extension to enable it per-container or a flag in the driver… based on user feedback
22:51:44 <apmelton> until there is a complete analysis of user namespaces, I'll agree
22:51:49 <adrian_otto> #agreed Option #6 from https://etherpad.openstack.org/p/container-block-storage is not our preferred outcome. Secure by default is preferred.
22:52:15 <adrian_otto> ok, last item on the agenda before open discussion is Host Agent
22:52:21 <adrian_otto> #topic Host Agent Discussion
22:52:31 <adrian_otto> f a host agent is added for use by Nova to interact with guest operating systems (including container instances), what technical considerations should we be aware of?
22:52:34 <adrian_otto> +i
22:53:05 <adrian_otto> this could be used to help with things like creating/mounting filesystems
22:53:18 <funzo> i'm afraid I don't have much background on this one
22:53:19 <adrian_otto> creation of nested containers
22:53:33 <erw> guest agents for containers seems very, very wrong.
22:53:35 <adrian_otto> funzo: think an in-guest agent like trove has
22:53:54 <adrian_otto> erw: that's my default position on this as well
22:54:03 <funzo> would this be totally different than cloud-init?
22:54:24 <adrian_otto> funzo: well, it would be persistent, so it would work past system boot
22:54:36 <apmelton> by host-agent, is that an agent running at the same level as say the docker daemon, or libvirt?
22:54:41 <adrian_otto> possibly to help with mounting shared filesystems, that sort of thing
22:54:49 <apmelton> or an agent inside the container the host can talk to?
22:54:57 <adrian_otto> apmelton: host agent is on the level of libvirt
22:55:05 <adrian_otto> guest agent is within every guest
22:55:10 <erw> adrian_otto: perhaps it’s a side-issue, but I think it would be neat to replace the metadata service in general with a pub-sub based approach. Apps or agents could subscribe to the updates from the host
22:55:21 <erw> my default consideration for that would be MQTT, btw
22:55:27 <apmelton> in nova libvirt world, nova pretty much has to run on the same host as libvirt, I suppose that's not the case for docker?
22:55:58 <erw> apmelton: not strictly, no.
22:56:21 <erw> although practically, nova does things on the local host — so the driver does have to run on the host
22:56:46 <erw> apmelton: what you describe better matches  vmware and xen.
22:56:57 <apmelton> ya
22:56:57 <adrian_otto> with the introduction of libswarmd, Docker does not need to be on the remote
22:57:02 <Slower> I'm noticing that with volume support, it'd have to be on the same machine
22:57:11 <adrian_otto> the API deamon I mean
22:57:31 <adrian_otto> actually, it is there, but listening locally instead of to the network
22:57:34 <apmelton> so, I've thought about a host agent for libvirt as well, that way we could just run host-agent+libvirt on a pool of machines and let nova compute manage them
22:57:53 <Slower> or at least it is likely that is how it will be implemented
22:57:59 <apmelton> but, running a host-agent isn't very different from running nova compute
22:58:00 <erw> adrian_otto: you don’t even need libswarm for that, but the point is that we manipulate namespaces and such in nova-docker at present
22:58:16 <erw> adrian_otto: but we could move that code into libcontainer and leverage libswarm….
22:58:17 <adrian_otto> erw: nod.
22:58:17 <Slower> I don't see the advantage of having the daemon not on the host with nova anyway..
22:58:39 <adrian_otto> Slower: neither do I
22:58:41 <adrian_otto> time check
22:58:51 <adrian_otto> #topic Open Discussion
22:59:05 <adrian_otto> we can revisit both of those topics again to continue moving them forward
22:59:08 <Slower> snapshot support for nova docker
22:59:14 <erw> Slower: nova api for multi-API cloud? *shrug*
22:59:17 <adrian_otto> do you all feel that these meetings are moving us in the right direction?
22:59:26 <funzo> maybe we can add snapshots to next weeks agenda if we run out of time?
22:59:40 <adrian_otto> funzo: yes, we can
22:59:41 <Slower> snapshot is the next big hurdle after volume support
22:59:46 <erw> agreed
22:59:47 <funzo> we definitely need to talk about stuff and agree to it, we have to do it here or out of band
23:00:01 <erw> adrian_otto: yes.
23:00:17 <adrian_otto> ok, then we will continue as long as you feel it's helpful
23:00:17 * Slower agrees to take on all action items from todays meeting
23:00:24 <Slower> ;-)
23:00:28 <adrian_otto> Slower: <3
23:00:37 <adrian_otto> thanks everyone. Catch you next time.
23:00:41 <adrian_otto> #endmeeting