21:02:26 <strigazi> #startmeeting containers
21:02:27 <openstack> Meeting started Tue Jan 29 21:02:26 2019 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:30 <openstack> The meeting name has been set to 'containers'
21:02:34 <strigazi> #topic Roll Call
21:02:45 <cbrumm> o/
21:02:48 <strigazi> o/
21:02:55 <jakeyip> o/
21:04:00 <imdigitaljim> o/
21:04:38 <strigazi> #topic Stories/Tasks
21:05:24 <strigazi> From my side:
21:05:34 <strigazi> k8s v1.11.6 images rebuilt https://review.openstack.org/#/c/633478/
21:06:00 <strigazi> (v1.11.7 was out just yesterday)
21:06:37 <strigazi> tiller deployment can be taken in, please review https://review.openstack.org/#/c/612336/
21:07:40 <imdigitaljim> from us:
21:07:52 <strigazi> two patches for upgrades: add openssh clients to the heat-agent https://review.openstack.org/#/c/633504/ AND add the agent to all nodes: https://review.openstack.org/#/c/561858/
21:07:53 <imdigitaljim> we just got kubernetes org approval to submit
21:08:06 <imdigitaljim> so we might have some openstack-ccm updates to provide in the near future
21:08:34 <imdigitaljim> cluster autoscaler work and inplace upgrades for the centos driver
21:08:57 <imdigitaljim> once this is in place and stable ill be making effort to upstreaming the driver
21:09:09 <imdigitaljim> we run v1.13.2 atm as well
21:09:31 <imdigitaljim> we also want to finalize any core magnum changes we'd probably need to do
21:10:06 <strigazi> imdigitaljim: please build also the ci for the special centos image.
21:10:13 <imdigitaljim> cluster-autoscaler work is k8s specific (mostly), trivial driver changes if any i forget
21:10:19 <colin-> o/
21:10:19 <imdigitaljim> yes i definitely will
21:10:23 <imdigitaljim> make an mvp centos CI
21:11:47 <imdigitaljim> although strigazi: i might need some help getting that setup im unfamiliar
21:12:11 <imdigitaljim> to connect with existing stuff to be easily consumed
21:12:22 <strigazi> for the cluster-autoscaler I'm testing our own https://github.com/cernops/autoscaler/pull/3
21:12:34 <imdigitaljim> ill forward to our cas man
21:12:34 <strigazi> I'm giving priority to that one.
21:12:44 <flwang> sorry, i'm late
21:13:12 <colin-> nice strigazi looks like a lot of progress
21:15:39 <flwang> strigazi: what are we discussing?
21:16:21 <flwang> i'd like to discuss the k8s image versions we support, related to patch https://review.openstack.org/633650
21:16:58 <strigazi> flwang: looks ok, maybe we can use the vars file
21:17:25 <strigazi> other than that it is ok
21:17:58 <strigazi> flwang: what do you think?
21:18:33 <flwang> strigazi: use vars is also ok for me and it's more clean i think, i will propose another patchset
21:19:11 <strigazi> flwang: thanks, building all of them is fine
21:19:27 <strigazi> I have another thing I want to discuss
21:19:46 <strigazi> We have an issue in k8s_fedora
21:19:53 <strigazi> maybe in centos that imdigitaljim is solved
21:20:07 <strigazi> k8s uostrean beeds three CAs
21:20:21 <strigazi> CA= certificate authority not cluster autoscaler
21:20:28 <strigazi> one  for etcd
21:20:36 <strigazi> one for front-proxy
21:20:43 <strigazi> and one for the API
21:20:58 <strigazi> In magnum we create only one.
21:21:41 <strigazi> In the same manner that we create the service account keys, we need two more CAs.
21:22:00 <flwang> what's the affection now?
21:22:23 <flwang> affection/impact i mean
21:22:36 <strigazi> For example, a new feature like the metrics-server doesn't work correctly
21:22:37 <flwang> is there any security hole now?
21:22:39 <strigazi> see here:
21:22:49 <imdigitaljim> oh?
21:23:03 <imdigitaljim> helm chart metrics server?
21:23:04 <strigazi> https://review.openstack.org/#/c/632392/
21:23:21 <strigazi> https://review.openstack.org/#/c/632392/4/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-master.sh@75
21:23:31 <strigazi> This option kind of works
21:23:43 <strigazi> but the proper thing to do is to have tree CAs
21:23:52 <strigazi> see docs here:
21:24:35 <strigazi> https://github.com/kubernetes/website/blob/master/content/en/docs/setup/certificates.md
21:25:00 <flwang> so you mean though we can use the current, only one, CA, but we'd better to create 3, right?
21:25:18 <strigazi> installing the metrics-server with helm or not, it doesn'r matter
21:25:23 <strigazi> flwang: yes
21:25:38 <strigazi> quoting docs: You can create a single root CA, controlled by an administrator. This root CA can then create multiple intermediate CAs, and delegate all further creation to Kubernetes itself.
21:26:17 <imdigitaljim> yeah
21:26:24 <imdigitaljim> 1 CA is okay
21:26:24 <strigazi> having these three CAs would make the transition to kubeadm ~trivial
21:26:27 <imdigitaljim> 3 CA is better
21:26:31 <flwang> strigazi: i see
21:26:47 <strigazi> kubeadm will try to create these three CAs.
21:26:53 <flwang> strigazi: so i think it's important but not urgent issue
21:27:01 <strigazi> if you provide them it will use them
21:27:08 <flwang> cool
21:27:22 <imdigitaljim> also note: we dont use kubeadm at this time
21:27:27 <imdigitaljim> in case you were wondering
21:27:52 <imdigitaljim> theres a few issues in using kubeadm that we need to be better before/if we transition to them
21:28:02 <imdigitaljim> i think they are publicly listed in their design docs
21:28:31 <strigazi> kubeadm or not, to use API aggregators we need to deploy front-proxy certs properly
21:29:09 <openstackgerrit> Feilong Wang proposed openstack/magnum stable/pike: support http/https proxy for discovery url  https://review.openstack.org/633755
21:31:16 <strigazi> docs for front proxy: https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/
21:32:19 <imdigitaljim> Warning: Do not reuse a CA that is used in a different context unless you understand the risks and the mechanisms to protect the CA’s usage.
21:32:20 <imdigitaljim> :D
21:32:24 <imdigitaljim> i see your concern
21:32:25 <openstackgerrit> Keith Berger proposed openstack/magnum stable/pike: support http/https proxy for discovery url  https://review.openstack.org/633755
21:33:47 <strigazi> I can mention a horrible side-effect of using one CA. using the same CA to sign certs for kubelets and etcd, means compromise of kubelet certs gives access to etcd.
21:34:20 <imdigitaljim> sure
21:34:22 <strigazi> provided that there is a route from from the kubelet node to etcd
21:34:40 <imdigitaljim> preventing lateral movement is always good
21:34:41 <strigazi> same for the etcd that calico uses
21:35:22 <imdigitaljim> ill probably start working on solving some of these in the upcoming weeks
21:35:30 <imdigitaljim> good point out strigazi
21:35:44 <colin-> yeah. been working on Octavia on the side and have been going through this due to all the CAs it uses to secure communications between components
21:36:23 <colin-> and wanting them to be signed uniquely by service made deployment more complex but ultimately better for the security, imo
21:37:54 <strigazi> it is a ~chicken-egg problem. Who sings whose certs :)
21:38:19 <imdigitaljim> we were looking at other solutions like vault as a CA
21:38:22 <imdigitaljim> and such
21:38:24 <strigazi> The problem to cert management in kubernetes is more CAs
21:38:32 <imdigitaljim> https://github.com/hashicorp/vault
21:38:43 <strigazi> s/problem/solution/
21:38:44 <imdigitaljim> not sure their CA capabilities
21:41:49 <strigazi> Anything else anyone? (the day is ending for me)
21:42:10 <imdigitaljim> not here
21:42:11 <imdigitaljim> o/
21:42:42 <strigazi> flwang colin- jakeyip anything else?
21:43:00 <strigazi> jakeyip: for tempest python3, all good?
21:43:44 <jakeyip> Nothing to report :)
21:44:22 <strigazi> cool
21:44:49 <colin-> nope
21:45:00 <strigazi> let's end the meeting then
21:45:09 <colin-> have a good night!
21:45:17 <strigazi> see you next week
21:45:21 <strigazi> colin-: cheers
21:45:22 <cbrumm> bye all
21:45:49 <strigazi> #endmeeting