16:01:22 <adrian_otto> #startmeeting containers
16:01:23 <openstack> Meeting started Tue Mar 22 16:01:22 2016 UTC and is due to finish in 60 minutes.  The chair is adrian_otto. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:27 <openstack> The meeting name has been set to 'containers'
16:01:43 <adrian_otto> #link https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2016-03-22_1600_UTC Our Agenda
16:01:52 <adrian_otto> #topic Roll Call
16:01:54 <adrian_otto> Adrian Otto
16:01:54 <coreyob> Corey O'Brien
16:01:58 <muralia> murali allada
16:01:59 <jward> John Ward
16:01:59 <Tango> Ton Ngo
16:01:59 <dane_leblanc> o/
16:02:00 <hongbin> o/
16:02:04 <rpothier> o/
16:02:07 <tcammann> hello
16:02:07 <levi_b> Levi Blackstone
16:02:07 <juggler> Perry Rivera
16:02:11 <eghobo> o/
16:02:53 <adrian_otto> hello coreyob muralia jward Tango dane_leblanc hongbin rpothier tcammann levi_b juggler and eghobo
16:03:01 <muralia> hi everyone
16:03:09 <juggler> hello
16:03:25 <adrian_otto> #topic Announcements
16:03:32 <adrian_otto> 1) stable/mitaka Branches have been cut for magnum and python-magnumclient last week
16:03:51 <madhuri> o/
16:03:53 <adrian_otto> bugs are still allowed, but new features require an FFE
16:04:13 <adrian_otto> note that we are also in string freeze, so if you are changing strings, we should discuss your patches
16:04:39 <adrian_otto> I have not yet cut a branch for magnum-ui as someone asked me to hold off to get some things merged
16:04:54 <adrian_otto> I'm planning on branching it this week
16:05:03 <adrian_otto> let me know if there is any good reason to delay
16:05:23 <adrian_otto> 2) New networking features are available in experimental docker branch
16:05:35 <adrian_otto> #link https://github.com/docker/docker/blob/master/experimental/vlan-networks.md documentation
16:05:43 <adrian_otto> #link https://dl.dropboxusercontent.com/content_link/o3RUvDMDJfOnKQ2MEosboPkqluNMvJZnhAp6bd2UKoxH7lDvVgi0vQ37DTBkS6Ry/file?dl=1 blog post preview
16:05:55 <adrian_otto> #link http://www.slideshare.net/bbsali0/docker-networking-with-new-ipvlan-and-macvlan-drivers slides
16:06:07 <adrian_otto> these are interesting features because they do not involve tunneled overlays
16:06:15 <adrian_otto> We should look into this as a team and see if it makes sense to explore exposing any of these in the Docker Swarm bay.
16:06:32 <adrian_otto> at least as a non-default alternate
16:06:46 <adrian_otto> has anyone looked at this yet?
16:07:36 <adrian_otto> ok, we can revisit this once you've had a chance to look at it
16:07:47 <adrian_otto> 3) PTL Elections are open, and Magnum is having an election.
16:07:49 <adrian_otto> If you made a commit to any of the magnum related projects during the Mitaka release cycle, you may cast a vote for PTL. Check your inbox for a message with subject "Poll: The OpenStack Magnum PTL Election" to vote.
16:08:00 <adrian_otto> #topic Review Action Items
16:08:07 <adrian_otto> 1) adrian_otto to start an ML thread about Magnum functional gate tests taking forever to execute in OpenStack CI.
16:08:15 <adrian_otto> Status: Abandoned due to lack of recent examples to cite.
16:08:23 <adrian_otto> Let me know if you notice this symptom come back again, and I would be happy to revisit this.
16:08:41 <adrian_otto> has anyone notices this as an issue in the past week?
16:09:32 <adrian_otto> #topic Blueprint/Bug Review
16:10:00 <adrian_otto> any work items (blueprints, bugs, reviews) requiring team discussion today (not the certificate storage one, that is next on the agenda)?
16:10:58 <adrian_otto> did I lose you all in announcements?
16:11:40 <Tango> :)
16:11:41 <juggler> here :)
16:11:44 <coreyob> :)
16:12:01 <adrian_otto> ok, you are just a quiet bunch today
16:12:01 <hongbin> :)
16:12:17 <Tango> coffee hasn't kicked in yet
16:12:23 <juggler> ^
16:12:29 <muralia> :)
16:12:40 <adrian_otto> #topic Discussion of certificate storage
16:12:51 <adrian_otto> #link https://etherpad.openstack.org/p/magnum-barbican-alternative Etherpad summarizing the concern and a variety of options to address it
16:12:52 <adrian_otto> Please record your comments in the etherpad, and we can discuss it here a bit too.
16:15:07 <tcammann> Not sure how Anchor solves any of these problems
16:15:31 <adrian_otto> Anchor would make sense in the context of a Barbican based solution
16:15:32 <Tango> We should list use cases
16:15:35 <tcammann> It would be another component that a typical cloud doesn't contain
16:15:42 <adrian_otto> Tango ++
16:16:04 <tcammann> The simple solution to the problem is just store the private certs in the db
16:16:15 <tcammann> problem solved with the caveat it is not secure
16:16:18 <madhuri> And we have that already in Magnum
16:16:24 <madhuri> #link https://bugs.launchpad.net/magnum/+bug/1551838
16:16:25 <openstack> Launchpad bug 1551838 in Magnum "magnum local_cert_manager doesn't use x509keypair model" [Low,Confirmed] - Assigned to Madhuri Kumari (madhuri-rai07)
16:16:29 <adrian_otto> what I do have is three examples of teams that have picked up Magnum, want to use it, find that Barbican is a requirement, and then put Magnum back down again.
16:16:35 <madhuri> We have x59keypair for it
16:17:11 <madhuri> It was long discussed that we are not making Barbican any hard dependency in Magnum
16:17:11 <tcammann> madhuri: yes we should use that. I'm not sure why we used local cert storage over the db
16:17:36 <madhuri> So we have another option i.e store certs in Magnum DB
16:17:36 <adrian_otto> I have one example of a team who has used the local cert storage option, accepting the security drawbacks of that appraoach
16:17:56 <tcammann> local cert storage doesn't scale > 1 conductor
16:18:22 <adrian_otto> madhuri: I have been allergic to that in the past, and I"m willing to list it as an option, but I will continue to argue against that, as it's just a weak model.
16:18:34 <redrobot> adrian_otto I'm curious to learn the specifics behind the team's opposition to deployin Barbican...
16:18:48 <redrobot> adrian_otto but I guess that's a topic for another meeting...
16:18:55 * redrobot does not want to derail conversation.
16:18:59 <tcammann> It isn't our objection it is the objection of operators
16:19:08 <rm_work> Probably that is my fault ^^ as the Magnum team had Octavia's cert manager as an example, which had a demo "local storage" solution (though we did not use it and ended up deleting it)
16:19:18 <adrian_otto> I am interested in that as well, redrobot, but we must address the reality that this is an adoption inhibitor
16:19:34 <redrobot> tcammann understood... but I would like to understand the why.
16:19:42 <rm_work> It looks like the current cert_manager code is based off the Octavia/LBaaS example
16:19:59 <madhuri> rm_work: Correct
16:20:25 <adrian_otto> one thing we could see about is maybe deploying a bay that runs a barbican that is set up with SoftHSM or something, so that it's there by default, and maybe you can switch to using your own if you have one?
16:20:50 <tcammann> adrian_otto: no
16:21:01 <adrian_otto> tcammann: ok, why?
16:21:08 <madhuri> redrobot: It is not that we don't want to use Barbican. It is just that we don't want to make any hard dependency over it.
16:21:11 <tcammann> that's not our job.
16:21:32 <hongbin> IMO, we need an in-tree alternative for Barbican
16:21:33 <adrian_otto> tcammann: please elaborate
16:21:39 <rm_work> Barbican actually deploys easily already with an insecure storage backend, which is not any worse than doing your own storage in DB/filesystem I think >_>
16:21:48 <tcammann> We are containers. We don't want to take on the burden of deploying another service
16:22:01 <tcammann> wouldn't we need keystone also
16:22:43 <rm_work> LBaaS made the choice to require a Barbican deployment for TLS support, but I guess it is not a *hard requirement* for our basic service, so that is a little different
16:22:51 <hongbin> tcammann: +1
16:23:01 <adrian_otto> Each of the clouds who want to adopt Magnum all have keystone v3 available, in spite of comments made in the ML thread.
16:23:03 <tcammann> it's going to be a huge amount of work to setup a seperate "service" bay which doesn't run in user space and connects to Magnum and keystone
16:23:37 <adrian_otto> so that's why I approved an implementation based on KEystone credential store as a barbican alternative (non default, with warnings)
16:23:40 <adrian_otto> #link https://blueprints.launchpad.net/magnum/+spec/barbican-alternative-store
16:23:55 <adrian_otto> I think this will be a relatively small amount of code in Magnum
16:24:24 <adrian_otto> and should address the adoption inhibitor concern
16:24:38 <tcammann> aslong as we don't have to deploy keystone adrian_otto :P
16:24:40 <adrian_otto> and for cloud operators who get serious about hosting containers will use Barbican
16:24:50 <adrian_otto> tcammann: yes, that's the idea
16:26:03 <hongbin> Folks, how about the idea of implementing an additional authentication backend (in parallel with TLS)
16:26:06 <tcammann> barbican > keystone > db > local storage
16:26:31 <adrian_otto> tcammann:  agreed
16:26:59 <redrobot> it should be noted that the Keystone team does not recommend using the credential store in that way for this blueprint.
16:27:17 <hongbin> tcammann: I think the direction of storing secrets in places other than Barbican will get a lot of disagreement from the OpenStack community
16:27:27 <adrian_otto> redrobot: it's still better than an in-magnum-db solution
16:27:35 <adrian_otto> because we can use an existing service API
16:27:47 <hongbin> tcammann: We don't want to proceed with the direction the community is not happy with
16:28:13 <hongbin> tcammann: Also, there is also an problem to getting the security tag
16:28:14 <adrian_otto> it would be cool if the keystone credential store had an option to back-end on barbican (hint, hint)
16:28:39 <tcammann> hongbin: we are specifically not recommending operators do this, but we allow it for operators to preview our service with less of a burden
16:29:13 <hongbin> tcammann: Yes, I proposed that in the ML before. There are a lot of disagreements ....
16:29:25 <madhuri> tcammann: IMO it is not good to use keystone now when we reluctant to use Barbican which is specifically designed for key storage
16:29:38 <madhuri> hongbin: +1
16:29:39 <tcammann> If we have a secure option we can still get a security tag
16:29:56 <hongbin> true
16:30:20 <tcammann> Anything controversial will get disagreements on the ML
16:30:24 <adrian_otto> hongbin: I am willing to argue this with the TC, that having an option that uses keystone using reasonable methods for data encryption, and accompanying that with warnings that the security tag relates to recommended use of Magnum, and that using the keystone option is not recommended, but available for evaluation purposes.
16:30:36 <rm_work> Yes, I do not quite understand the opposition to using Barbican, it is a very simple deployment for preview
16:30:39 <adrian_otto> we will label the option in the config file accordingly
16:30:44 <rm_work> we do it in our gate (!) with no issues
16:30:54 <adrian_otto> just like you can use SSL with a clear cipher, stupid
16:31:04 <dstanek> adrian_otto: using barbican as a backend was mentioned at some point, but i don't think it's on anybody's roadmap
16:31:07 <tcammann> or use --disable-tls!!
16:31:55 <tcammann> This isn't a discussion on baribcan this is a discussion on giving Operators an alternative
16:32:35 <redrobot> tcammann I've heard plenty of arguments for an alternative... but they're usually using some existing alternative, not trying to develop a new one.
16:32:48 <adrian_otto> right, and I agree it's worth having an alternative as long as we provide guidance for appropriate use that is clear and present at the time/place you select an alternative.
16:33:24 <Tango> "Battery included but replaceable"
16:33:41 <hongbin> Tango: +1
16:33:43 <tcammann> We already have an alternative but it only scales to 1 conductor node.
16:34:35 <tcammann> If we just fix that to use db rather than local storage, then we have an easy fix and an alternative that scales
16:34:44 <adrian_otto> the BP I mentioned above would allow for scale out beyond a single conductor
16:34:44 <tcammann> with exactly the same caveats as local storage
16:35:18 <adrian_otto> right
16:35:22 <adrian_otto> minus one
16:35:36 <adrian_otto> it allows for encryption at rest
16:36:06 <adrian_otto> which we could also do for local file storage once we have the code for the credential store solution
16:36:22 <adrian_otto> which we already mentioned we could forklift from heat, and propose for oslo
16:37:49 <hongbin> WFM, as long as the proposal goes through TC/security/....
16:38:00 <adrian_otto> in the Elimination of certificates section, Hongbin offered a set of alternatives that would allow a totally different approach. If there are no cert files, then our certificate storage problem vanishes
16:38:16 <redrobot> dumb question: you guys are not actually wanting to provision the certs from barbican?  just store/retrieve of the certs?
16:38:37 <tcammann> redrobot: just store
16:38:43 <tcammann> and retrieve
16:38:54 <adrian_otto> redrobot: right, this is just about the cert storage
16:38:57 <redrobot> have you considered Castellan?  it's basically oslo.key_manager
16:39:27 <hongbin> redrobot: The problem of Castellan is that it has only one backend: Barbican
16:39:33 <adrian_otto> #link https://github.com/openstack/castellan Castellan
16:39:53 <redrobot> hongbin yeah, but since you're planning on developing new code, why not develop it as a Castellan implementation?
16:39:59 <redrobot> hongbin other projects would benefit from it.
16:40:10 <redrobot> hongbin and a KMIP implementation is in the works.
16:40:14 <rm_work> Also, Castellan does not include Certificate storage I think? Unless you are just saying to store them as keys <_<
16:40:36 <redrobot> rm_work Castellan does provide a facility to store certificates.  The only thing it does not provide is a way to provision certs.
16:41:19 <adrian_otto> redrobot: so Castellan can work without Barbican?
16:41:26 <rm_work> redrobot: when did that happen? that is what I was pushing for but got consistently rejected
16:41:38 <hongbin> If we have to develop a cert storage solution, I prefer to start it from Magnum tree and move it to Castellan later
16:41:50 <hongbin> That allows faster development
16:41:59 <redrobot> rm_work cert store/retrieve has always been in scope for Castellan.  our understanding was that you were pushing for Cert provisioning.
16:42:36 <redrobot> adrian_otto yes.  Castellan was created to consolidate key_manager interfaces from Cinder and Nova
16:42:54 <rm_work> redrobot: lol no, we discussed at length, I had two separate things, one for storage/retrieval and one for provisioning, BOTH got rejected, and it was especially noted that the storage/retrieval did not grant useful function without inclusion of provisioning, and you did not want to include provisioning >_<
16:43:05 <rm_work> but we can talk about this any time out-of-band >_> not a topic for here
16:43:41 <redrobot> adrian_otto we've been hearing the concern about wanting a secure key store, but also not wanting to deploy Barbican basically since the project started.
16:44:21 <adrian_otto> "the project" == Castellan ?
16:44:41 <redrobot> adrian_otto since barbican started
16:44:58 <adrian_otto> oh, from projects like Nova, etc?
16:45:25 <redrobot> adrian_otto yes... also Swift has Castellan integration on their roadmap
16:46:10 <redrobot> adrian_otto we tried to make it an oslo lib, but the oslo team was not interested in maintaining it
16:46:36 <hongbin> redrobot: do you know why?
16:46:38 <adrian_otto> redrobot: ok, so if I understand your response correctly, Castellan can be used without Barbican, and we could use that instead of Keysone. If we did that, where would our certs live? In the Magnum database? They would be encrypted at rest if we did that?
16:46:52 <rm_work> Castellan is a lib
16:46:59 <rm_work> which provides a layer of abstraction for storage
16:47:15 <rm_work> the backend would be .... whatever you want, so, it could be keystone or DB or whatever
16:47:22 <rm_work> but it allows switching backend easily as a driver
16:47:40 <rm_work> THOUGH this is basically already done in-tree with https://github.com/openstack/magnum/blob/master/magnum/common/cert_manager/cert_manager.py following the octavia model
16:47:59 <rm_work> (I tried to get this code added to Castellan previously but it was rejected)
16:49:30 <redrobot> in theory it could be anything... in practice we have three implementations with varying degrees of maturity.  The most mature is the Barbican implementation (YOUR PROJECT->Castellan->Barbican), but also planned are an insecure-probably-not-for-production software only solution, as well as a KMIP impl (YOUR PROJECT->Castellan->KMIP device)
16:50:25 <adrian_otto> ok, I think we have a good set of options to think over
16:50:42 <adrian_otto> let's wrap up on this topic, and advance to open discussion
16:50:47 <adrian_otto> any parting thoughts on this one?
16:51:11 <adrian_otto> #topic Open Discussion
16:52:24 <madhuri> #link https://review.openstack.org/#/c/269250/
16:52:50 <madhuri> Murano now have Magnum app to deploy different COE clusters
16:53:58 <adrian_otto> sweet!
16:54:09 <juggler> +1 madhuri
16:54:18 <madhuri> Thanks!
16:55:34 <adrian_otto> This is really awesome, Madhuri. Thanks so much for leading this.
16:55:53 <madhuri> Thanks adrian_otto :)
16:55:59 <adrian_otto> Now you ahve a really interesting story to tell in your Summit talk coming up in Austin.
16:56:25 <madhuri> Yes I do :)
16:58:15 <yolanda> so from my side, i did some fixes in magnum tests to allow to run with different images, i uploaded one image to fedorapeople
16:58:37 <yolanda> however i'm hitting some bugs in the kubernetes api tests, and i haven't discovered the root cause
16:59:04 <Tango> yolanda: I tried out that image and Kubernetes seems to work fine
16:59:08 <adrian_otto> thanks for your work on that, yolanda. I marked one of your patches for merge.
16:59:28 <yolanda> Tango i also tried mine on devstack and looked good. But is not passing functional testing
16:59:33 <adrian_otto> We are approaching our scheduled time slot. Our next meeting will be 2016-03-29 at 1600 UTC.
16:59:37 <yolanda> #link http://logs.openstack.org/68/294468/1/check/gate-functional-dsvm-magnum-k8s-dib-nv/81ddbbe/console.html
16:59:41 <Tango> yolanda: I can follow up with you on the problem with the API problem
16:59:47 <yolanda> that's error i'm getting
16:59:48 <adrian_otto> thanks everyone for attending today.
17:00:03 <adrian_otto> #endmeeting