22:00:18 #startmeeting containers 22:00:19 Meeting started Tue Jun 17 22:00:18 2014 UTC and is due to finish in 60 minutes. The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot. 22:00:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 22:00:22 The meeting name has been set to 'containers' 22:00:24 #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2014-06-17_2200_UTC Our Agenda 22:00:26 o/ 22:00:27 Ian Main o/ 22:00:29 #topic Roll Call 22:00:30 Chris Alfonso 22:00:32 Adrian Otto 22:00:43 Meghal Gosalia 22:01:06 Dmitry Guryanov 22:01:16 Eric Windisch 22:01:20 Joe Talerico 22:01:36 Subra Ongole 22:02:25 welcome everyone 22:02:54 if you have not yet recorded your attendance, you may chime in at any time 22:03:04 #topic Announcements 22:03:05 Andrew Melton 0/ 22:03:16 would the team members like to make any announcements? 22:03:38 #topic Review Action Items 22:03:40 (none) 22:03:49 #topic Mid-Cycle Meetup for Containers 22:04:16 Yahoo! has offered to host a mid-cycle meetup for us in the SF Bay area. 22:04:38 I'd like you all to have input on what dates we should plan this for 22:04:47 and gather consensus about what number of days is optimal 22:05:33 thoughts on this? 22:05:57 number of days first, I think. 22:05:58 something in early Aug might make sense 22:06:03 ok, number of days 22:06:13 ideas ranging from 1-3 have been offered in prior discussions 22:07:13 I vote winter.. because I'm from canada 22:07:38 having a couple of days together would give us a chance to enter detailed design efforts 22:07:42 that's a joke 22:07:48 Slower: ;-) 22:08:00 I’m thinking we only need two, but I don’t mind going longer if others feel it necessary 22:08:40 who would need to travel a long distance to attend an SF area meetup? I know erw would come from PA. 22:08:47 I would come from LA 22:08:54 I'll be coming from VA if I attend 22:08:56 I can fly out of spokane WA 22:09:35 ok, so at least a few of us would travel. Is a shorter trip easier to clear with your travel budgets? If so, we should consider that too 22:09:59 if we can have a healthier group for a shorter time, I'd prefer that to a smaller group for longer 22:10:33 ok, so how about dates to avoid? 22:10:40 first week in AUG for me 22:11:01 sept 12-21 i'll be away 22:11:07 august 16-31 I'm away 22:11:21 Jul 11-24 I am on vacation 22:11:29 probably with a few days on each end 22:11:55 Also Nova meet up is July 28th to 30th I believe 22:11:57 adrian_otto: we should prolly bias towards meeting sooner rather than later, once you get into mid-july lots of ppl leaving for vaca and such 22:12:11 Philadelphia… but if it’s at Yahoo, then I can spend the other days at the office 22:12:41 I don’t yet know of any conflicts — other than the Nova meetup 22:13:01 apmelton: where is the nova meetup being held? 22:13:10 So if we did July 31 and Aug 1 22:13:11 Beaverton, OR (portland) 22:13:20 oh, well. 22:13:33 who is hosting that one? 22:13:43 intel if I remember correctly 22:13:45 Intel 22:13:56 https://www.eventbrite.com.au/e/openstack-nova-juno-mid-cycle-developer-meetup-tickets-11878128803 22:14:12 what about doing one weekday, and one Saturday 22:14:17 adrian_otto: i don't know yet if I can for sure get the budget, but if I do - the end of july also doesn't work for me 22:14:28 so many vacations, so little time 22:14:49 yeah, summer meetups are really tricky 22:14:53 I vote weekdays 22:15:19 Yea, if I'm flying to/from the West Coast, I prefer weekdays as well 22:15:20 ok, so how about in the range of Aug 11-15? 22:15:36 I can do Aug 11-15 22:15:42 maybe Tue Aug 12, and Wed Aug 13 22:15:50 adrian_otto: weekdays are better for me, but I can be flexible. 22:15:51 that would work for me 22:16:42 I have a trip planned for those dates, but I's likely to be cancelled, so that might be ok 22:16:44 I haven't paid any attention to a mid-cycle meetup until just now, so I haven't made any budget requests 22:16:57 I am on vacay week of 8/11 22:17:09 if I can't make it out, i'd be interested in g+ hangout 22:17:19 I need to be in vermont on aug 16th.. 22:17:28 well if all goes well 22:17:32 ok, how about Aug 18-22 22:18:28 The trove Mid-Cycle meetup is Aug 20-22 in Cambridge, MA 22:18:28 * erw pulls up the roadmap 22:18:29 I'm on vac August 16 - 31 22:18:40 but I'm pretty sure that won't be a problem for us 22:19:21 we’re already into feature proposal freeze by then 22:19:53 4th of july week is usually impossible 22:20:12 but… let's check that week 22:20:16 I'm on vacation week of july 4th 22:20:18 the 4th is on a friday 22:20:52 any others? 22:21:21 no strong objection, although I’m sure my family will kill me. 22:21:37 I would miss my wife's birthday 22:21:45 I love you all, but not that much 22:21:46 I'm drunk on that day 22:21:48 heh 22:22:10 funzo: well you may as well be in a meeting then.. 22:22:24 ok, so let's table this, I'll run a doodle poll on it, and let's find a time when we can at least get some of us together 22:22:40 watch the ML for that link 22:22:48 I liked the 14th 15th idea 22:22:58 aug 22:23:05 k 22:23:18 doodle ++ 22:23:25 #topic Cinder Support Discussion 22:23:28 also gives us time to sync with SOs 22:23:56 se had a good collaboration this week exploring the options in this wiki: 22:24:04 #link https://etherpad.openstack.org/p/container-block-storage Options for Cinder Support and unprivileged Container-in-Container 22:24:17 the idea is to find options that might kill two birds with one stone 22:24:26 let's take a moment to scan through that 22:25:21 scanned 22:25:45 ok, are there any additional options that should be considered in addition to these? 22:26:02 adrian_otto: a variant of the API solution... 22:26:16 Option 1? 22:26:16 where the container uses an OpenStack API to initiate the mount 22:26:31 yes, variant of option #1 22:26:41 is that different from Option 5? 22:26:57 it’s the hybrid between #1 and #5 22:27:12 ok, explain further, please 22:27:53 it would be a nova extension for mounting filesystems, not a docker one. 22:27:54 maybe list that as option 11 22:28:03 yeah, I can add it in. 22:28:06 ok 22:28:23 so I'd like to have consensus around ruling some of these out 22:28:35 to narrow the list to ones we really like 22:28:42 so should we just move forward with #8 in the short term? - then work on one of the other options...? 22:28:57 i'm concerned that most of these options has quite a runway on them 22:28:57 great minds think alike 22:29:13 so let's look at #8 for a moment 22:29:28 essentially providing the block device and that's it, right? 22:29:32 #8 will allow you to expose a cinder volume to a container 22:30:01 which we have done in docker and as a POC in nova-docker (need to do a lot of refactoring before it's suitable for upstream) 22:30:07 but the container will not be able to mount it unless it is running in privileged mode, which they won't unless that is intentionally hacked by the cloud operator 22:30:14 well... it could range from a little to a lot of refactoring 22:30:49 is anyone against moving forward with that idea? 22:30:54 container mount service? 22:31:00 one thing I am uncertain about is if we took this approach (assuming a first step in a sequence of these approaches), will the openstack CI need to be refactored, or not? 22:31:03 just thinking out loud 22:31:13 I don’t think that #8 is really an “option” as much as it is, “don’t do this at all" 22:31:19 I mean, it’s technically the status quo. 22:31:42 erw, I don't think the whole Nova team fathoms that option 22:31:59 I think the initial impression is "cinder does not work, period" 22:32:02 adrian_otto: tempest might need changes to support it, yes. 22:32:26 ok, so assuming changes to tempest are needed, are members of this team willing to volunteer to make those changes? 22:32:34 yes 22:32:39 yes 22:32:48 ok, so in that case, it's hard to argue against this as a first step 22:32:59 any alternate points of view to consider? 22:33:04 option #8 doesn't really paint us into a corner 22:33:14 we'll still have to decide on the next option 22:33:14 agreed - I wouldn’t block merging the code for #8 on filesystem mounting 22:33:36 since it’s a dependency for filesystem mounting, really. ;-) 22:34:22 ok, so any objections to recording this with a #agreed as a first step? 22:34:35 then we can explore what comes next 22:35:00 #agreed 22:35:11 well #8 is status quo 22:35:13 easy enough 22:35:36 well, I want something that would allow a merge of nova-docker back into nova 22:35:55 adrian_otto: I don’t think they’d block merging into nova based on partial support. 22:36:05 they might block merging based on not having full support 22:36:06 and I think at least a wiki that talks about cinder support and Containers with Nova 22:36:16 and break it down into sections LXC, Docker, etc. 22:36:33 right, this would assume an argument about what "full" support means 22:36:41 precisely 22:36:46 today if I use cinder to make a volume, it does not mount it for me 22:37:11 it doesn’t even make filesystems for you 22:37:15 right 22:37:20 it just gives me raw blocks 22:37:22 the contract has nothing to do iwth filesystems or mountability 22:37:30 right. 22:37:38 but some folks get hung up on implied contracts and “user experience" 22:37:40 ;-) 22:38:09 #agreed our first step for cinder support with Containers shall be addressed by option 8 in https://etherpad.openstack.org/p/container-block-storage 22:38:37 ok, so let's assume for a moment that we have some sensible documentation, and have moved passed arguments about this 22:38:49 which approaches can we rule out as a second step 22:38:51 adrian_otto: fwiw, I’ve been waving the banner to gather internal discussion at Docker on the other options, for step #2 22:39:07 erw, awesome, thanks! 22:39:11 anything new to share? 22:39:35 I propose we rule out option 6. 22:39:54 not yet. It has a card in the big-bucket-of-things-we-know-about ;-) 22:39:58 adrian_otto: with user namespaces though, isn't allowing mount secure? 22:40:14 good question apmelton. 22:40:21 erw: is there also still a card re snapshots and the registry communication with glance? 22:40:25 apmelton: there were comments on the mailing list that it isn’t. 22:40:28 I don't feel that I know user namespaces well enough to make that judgement 22:41:15 and that came from Bottomley, who knows better than most anyone else 22:41:15 I see that primarily as a UID mapping facility 22:41:19 so, I know from a high level how to use it, but at the kernel level (where the attack is going to happen) I'm still fairly new to it 22:41:32 I am not aware of containment of the CAP_XXX rights 22:43:00 ok, so any other thoughts about ruling out Option 6? 22:43:25 adrian_otto: from my understanding, it doesn't affect CAP_XXX, but if you're able to escalate yourself to root by exploiting mount some how, you will only be root from inside the containers context 22:43:49 adrian_otto: so my only positive support for #6 is that it is no worse than nova-baremetal/ironic and you can still run non-privileged containers *inside* of it. 22:44:07 adrian_otto: so the security impact, overall, is actually fairly small - from a certain perspective 22:44:19 I think the fear is that there are if (0 == uid) conditions in the kernel today 22:44:30 so being root in the container is still bad 22:45:33 adrian_otto: which, for that matter, we allow being root in the container with nova-docker already. 22:45:40 so unless there is a complete threat analysis on that…. maybe we can find one… then we should remain concerned 22:46:00 erw: please tell me more about that? 22:47:29 adrian_otto: docker lets you run as root inside the containers. Nova-docker just lets you run containers with Docker. It doesn’t force the user in those containers to be non-root 22:47:46 ok 22:47:50 anyway, there are people that WANT to use nova-docker as a baremetal hypervisor. Defaulting to priviledged containers would match their requirements. It would allow nested containers and mounting filesystems. Those are advantages of disregarding security 22:48:32 with, again, the idea that you could just run a docker-in-docker image that drops all priviledges and runs with limited SET_CAP and a constricted set of cgroups 22:48:32 so we are one step closer on this 22:49:16 the biggest negative is that people won’t know it’s insecure by default, and it doesn’t match how people typically use Docker. 22:49:42 erw: fair enough. That's something that could be addressed with documentation, and embedded warnings 22:50:12 long term, I think #6 is a bad idea, as we aim to do all these things without being priviledged 22:50:36 #1 is the way to go forward right now imo 22:50:38 can we all agree that #6 is not our preferred long term outcome? 22:51:02 #agreed 22:51:02 I'd like to record that today as well, if that's how we all feel about it. 22:51:11 #agreed 22:51:11 any objections? 22:51:20 nope 22:51:34 nop 22:51:44 adrian_otto: I might like to add that we might want to consider adding an admin-only extension to enable it per-container or a flag in the driver… based on user feedback 22:51:44 until there is a complete analysis of user namespaces, I'll agree 22:51:49 #agreed Option #6 from https://etherpad.openstack.org/p/container-block-storage is not our preferred outcome. Secure by default is preferred. 22:52:15 ok, last item on the agenda before open discussion is Host Agent 22:52:21 #topic Host Agent Discussion 22:52:31 f a host agent is added for use by Nova to interact with guest operating systems (including container instances), what technical considerations should we be aware of? 22:52:34 +i 22:53:05 this could be used to help with things like creating/mounting filesystems 22:53:18 i'm afraid I don't have much background on this one 22:53:19 creation of nested containers 22:53:33 guest agents for containers seems very, very wrong. 22:53:35 funzo: think an in-guest agent like trove has 22:53:54 erw: that's my default position on this as well 22:54:03 would this be totally different than cloud-init? 22:54:24 funzo: well, it would be persistent, so it would work past system boot 22:54:36 by host-agent, is that an agent running at the same level as say the docker daemon, or libvirt? 22:54:41 possibly to help with mounting shared filesystems, that sort of thing 22:54:49 or an agent inside the container the host can talk to? 22:54:57 apmelton: host agent is on the level of libvirt 22:55:05 guest agent is within every guest 22:55:10 adrian_otto: perhaps it’s a side-issue, but I think it would be neat to replace the metadata service in general with a pub-sub based approach. Apps or agents could subscribe to the updates from the host 22:55:21 my default consideration for that would be MQTT, btw 22:55:27 in nova libvirt world, nova pretty much has to run on the same host as libvirt, I suppose that's not the case for docker? 22:55:58 apmelton: not strictly, no. 22:56:21 although practically, nova does things on the local host — so the driver does have to run on the host 22:56:46 apmelton: what you describe better matches vmware and xen. 22:56:57 ya 22:56:57 with the introduction of libswarmd, Docker does not need to be on the remote 22:57:02 I'm noticing that with volume support, it'd have to be on the same machine 22:57:11 the API deamon I mean 22:57:31 actually, it is there, but listening locally instead of to the network 22:57:34 so, I've thought about a host agent for libvirt as well, that way we could just run host-agent+libvirt on a pool of machines and let nova compute manage them 22:57:53 or at least it is likely that is how it will be implemented 22:57:59 but, running a host-agent isn't very different from running nova compute 22:58:00 adrian_otto: you don’t even need libswarm for that, but the point is that we manipulate namespaces and such in nova-docker at present 22:58:16 adrian_otto: but we could move that code into libcontainer and leverage libswarm…. 22:58:17 erw: nod. 22:58:17 I don't see the advantage of having the daemon not on the host with nova anyway.. 22:58:39 Slower: neither do I 22:58:41 time check 22:58:51 #topic Open Discussion 22:59:05 we can revisit both of those topics again to continue moving them forward 22:59:08 snapshot support for nova docker 22:59:14 Slower: nova api for multi-API cloud? *shrug* 22:59:17 do you all feel that these meetings are moving us in the right direction? 22:59:26 maybe we can add snapshots to next weeks agenda if we run out of time? 22:59:40 funzo: yes, we can 22:59:41 snapshot is the next big hurdle after volume support 22:59:46 agreed 22:59:47 we definitely need to talk about stuff and agree to it, we have to do it here or out of band 23:00:01 adrian_otto: yes. 23:00:17 ok, then we will continue as long as you feel it's helpful 23:00:17 * Slower agrees to take on all action items from todays meeting 23:00:24 ;-) 23:00:28 Slower: <3 23:00:37 thanks everyone. Catch you next time. 23:00:41 #endmeeting