21:01:48 <strigazi> #startmeeting containers
21:01:49 <openstack> Meeting started Tue Aug 21 21:01:48 2018 UTC and is due to finish in 60 minutes.  The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:52 <openstack> The meeting name has been set to 'containers'
21:01:57 <strigazi> #topic Roll Call
21:02:02 <imdigitaljim> o/
21:02:05 <colin-> hi
21:02:05 <strigazi> o/
21:02:51 <strigazi> #topic Announcements
21:03:03 <strigazi> kubernetes v1.11.2 is up: https://hub.docker.com/r/openstackmagnum/kubernetes-kubelet/tags/
21:03:04 <canori02> o/
21:03:32 <strigazi> #topic Blueprints/Bugs/Ideas
21:03:42 <strigazi> hello canori02
21:04:18 <imdigitaljim> i saw your proxy merge. I'll rebase those other ones and we'll be good to go =]
21:04:26 <imdigitaljim> on a sidenote we recently switched over to DS proxy as well
21:04:37 <imdigitaljim> so if you want i can throw a PR for that at some point?
21:04:57 <strigazi> imdigitaljim: you changed to DS downstream?
21:05:02 <imdigitaljim> yeah
21:05:25 <imdigitaljim> making motion towards self-hosted
21:05:36 <imdigitaljim> to make in-place upgrades smoother
21:05:40 <strigazi> I think it is better to move to DS if we have a plan for most of the components
21:06:06 <imdigitaljim> well ill get a PR up for it soon
21:06:09 <strigazi> you have only proxt as DS
21:06:10 <strigazi> ?
21:06:15 <imdigitaljim> yeah currently
21:06:21 <imdigitaljim> everything else is static still for now
21:06:34 <strigazi> and calico is a DS right?
21:06:42 <imdigitaljim> yes
21:06:49 <imdigitaljim> and keystone-auth plugin is DS
21:06:52 <imdigitaljim> for masters
21:07:37 <imdigitaljim> nodeSelector: node-role.kubernetes.io/master: ""
21:07:39 <imdigitaljim> in other words
21:07:55 <imdigitaljim> ccm is the same as well
21:08:24 <strigazi> node selector can be used for api and sch and cm I guess
21:08:39 <imdigitaljim> in most deployments that is exactly whats its for
21:08:53 <imdigitaljim> but yeah that'd be appropriate too
21:09:02 <imdigitaljim> and a few tolerations
21:09:49 <imdigitaljim> i have some concerns about the reliability on self-hosted
21:09:53 <imdigitaljim> in terms of node reboot
21:09:59 <imdigitaljim> so i think ill have to plan those out
21:10:06 <imdigitaljim> and see what our 'competition' does
21:10:42 <imdigitaljim> but in general i think our near term goals will be in-place upgrade
21:10:44 <strigazi> static pods shouldn't be a problem if kubelet starts
21:11:03 <imdigitaljim> yeah thats what i was thinking
21:11:10 <imdigitaljim> except we'd probably need to keep etcd static as well
21:11:13 <imdigitaljim> without etcd it cant join
21:11:29 <imdigitaljim> what kubeadm does is it throws up a static set for rejoining
21:11:39 <imdigitaljim> and then tears it down when reconnected to the clusterr
21:11:56 <imdigitaljim> which i think would be preferable
21:12:02 <strigazi> What do you mean with a static set?
21:12:38 <strigazi> for single master that I have tried, it is just a static pod, isn't it?
21:12:54 <imdigitaljim> it ends up being one yes
21:13:10 <imdigitaljim> i think i'll have to investigate the workflow a little more
21:13:27 <imdigitaljim> make sure im also understanding it correctly
21:13:40 <strigazi> if the data of etcd are in place, rebooting the node isn't a problem
21:13:59 <imdigitaljim> but if etcd is self-hosted as well
21:14:05 <strigazi> using a static pod or not
21:14:07 <imdigitaljim> theres no apiserver online to interact with it
21:14:09 <imdigitaljim> no?
21:14:34 <strigazi> kubelet can't start pods without an api server
21:14:36 <imdigitaljim> (multimaster scenario)
21:14:41 <imdigitaljim> exactly
21:15:07 <strigazi> In multimaster I don't know what kubeadm does
21:15:23 <strigazi> if it converts the static pods to a deployment
21:15:26 <strigazi> or ds
21:15:27 <imdigitaljim> i think this problem is significantly easier in single master
21:16:04 <strigazi> well if, you use static pods, it is the "same" for multi and single master
21:16:26 <imdigitaljim> yeah
21:16:32 <strigazi> if reboot one by one
21:17:15 <imdigitaljim> are you assuming etcd is also self-hosted in this?
21:17:17 <strigazi> if you reboot all of them in one go maybe the result is different
21:17:19 <imdigitaljim> or not
21:17:42 <strigazi> I don't see a diference
21:18:22 <strigazi> kubelet for static pods is like systemd for processes :)
21:18:30 <flwang> sorry i'm late
21:18:45 <colin-> welcome
21:18:58 <imdigitaljim> https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.10.md#optional-and-alpha-in-v19-self-hosting
21:19:33 <imdigitaljim> i believe there are some unaccounted for situations
21:20:44 <strigazi> oh, you mean make even etcd a ds
21:20:55 <strigazi> I wouldn't do that :)
21:21:10 <strigazi> it sounds terrifying
21:21:13 <imdigitaljim> haha
21:21:24 <imdigitaljim> yeah just some concerns to look at
21:21:35 <imdigitaljim> definitely possible to overcome though
21:22:07 <strigazi> ok, since flwang is also in,
21:22:37 <flwang> ds for etcd?
21:22:51 <colin-> yes, mad science lab :)
21:23:00 <strigazi> I was rebasing the upgrades api and I spent a good three hourds debugging. (ds for etcd, yes)
21:23:54 <strigazi> The new change for the svc account keys was "breaking" the functionality
21:24:21 <flwang> strigazi: breaking?
21:24:37 <strigazi> but I finally, figured it out, when magnum creates the dict for params to pass to heat
21:25:00 <strigazi> it adds and generate the keys for the service account
21:25:38 <strigazi> and if since it generates every time magnum creates the params it means that the keys will be different
21:25:52 <strigazi> "breaking" not really breaking
21:26:01 <flwang> ah
21:26:11 <flwang> so do we need any change for the key generating?
21:26:40 <strigazi> maybe, but for now I just ignore those two params
21:26:57 <strigazi> I'll push after the meeting
21:27:00 <imdigitaljim> thats a crazy issue
21:27:00 <imdigitaljim> yeah
21:27:07 <imdigitaljim> maybe we should save in barbican
21:27:09 <imdigitaljim> or
21:27:14 <flwang> strigazi: ok, i will keep an eye on that part
21:27:14 <imdigitaljim> whatever secret backend
21:27:28 <imdigitaljim> and extract or create depending on update/create?
21:27:35 <strigazi> we should do the same thing we do for the ca
21:27:48 <imdigitaljim> so yeah^
21:27:52 <strigazi> imdigitaljim we can do that too
21:28:39 <strigazi> the only thing I don't like is having the secret in barbican and then passing it as a heat paramter
21:28:52 <colin-> i think that would be useful without too much cost
21:28:59 <colin-> oh
21:29:02 <strigazi> It will stay in heat db for ever
21:29:10 <colin-> that's lame
21:29:11 <imdigitaljim> thats true
21:29:22 <strigazi> encrypted, but still
21:29:27 <imdigitaljim> can trustee be extended to allow cluster to interact with barbican?
21:29:34 <strigazi> yes
21:29:47 <imdigitaljim> (or secret backend)
21:30:11 <strigazi> we don't have to do anything actually, the trustee with the trust can talk to barbican
21:30:18 <imdigitaljim> yeah
21:30:28 <imdigitaljim> thats what i was hoping :]
21:30:32 <strigazi> vault or other is different
21:30:32 <imdigitaljim> i havent actually tried it
21:30:51 <imdigitaljim> people can add support for other backend as they need imho
21:30:52 <strigazi> Let's see in stein
21:31:10 <flwang> imdigitaljim: i'm interested in your keystone implementation btw
21:31:20 <imdigitaljim> oh sure
21:31:38 <flwang> imdigitaljim: especially if it needs api restart
21:31:46 <imdigitaljim> ive been working through some changes on new features here and we just accepted some alpha customers so we've been tied up
21:31:51 <imdigitaljim> but ill revisit those PR's and then some
21:32:10 <imdigitaljim> which api restart?
21:32:29 <flwang> k8s api server
21:33:02 <imdigitaljim> oh im not doing a restart, im confused?
21:33:06 <strigazi> why it needs an api restart?
21:33:09 <flwang> because based on testing, you have to restart k8s api server after you got the service URL of keystone auth service
21:33:21 <flwang> if it's deployed as DS
21:33:32 <imdigitaljim> oh yeah i havent needed to at all
21:33:33 <strigazi> flwang: when I tried I didn't restarted the k8s-api
21:33:51 <strigazi> flwang you can use host-network
21:34:07 <imdigitaljim> ^
21:34:11 <imdigitaljim> i dont know if you're using gthat
21:34:12 <flwang> then I really want to know how  did you do that if you don't deployed it on master
21:34:18 <flwang> master's kubelet
21:34:20 <imdigitaljim> but we are doing hostNetwork: true
21:34:28 <imdigitaljim> i do
21:34:35 <imdigitaljim> tolerations:
21:34:35 <imdigitaljim> - key: dedicated
21:34:35 <imdigitaljim> value: master
21:34:35 <imdigitaljim> effect: NoSchedule
21:34:35 <imdigitaljim> - key: CriticalAddonsOnly
21:34:37 <imdigitaljim> value: "True"
21:34:39 <imdigitaljim> effect: NoSchedule
21:34:41 <imdigitaljim> nodeSelector:
21:34:43 <imdigitaljim> node-role.kubernetes.io/master: ""
21:35:26 <imdigitaljim> we're putting most kube-system resources on masters
21:35:37 <flwang> so your k8s-keystone-auth service is running as DS and not running on master, right?
21:35:41 <imdigitaljim> not like dashboards and whatnot
21:35:46 <imdigitaljim> its running a DS on master
21:35:50 <imdigitaljim> and not on minion
21:35:53 <flwang> ah, right
21:36:24 <flwang> then yes, for that case, it's much easier and could avoid the api restart
21:36:33 <imdigitaljim> oh okay yeah
21:36:38 <flwang> so, should we just get the kubelet back on master?
21:36:54 <strigazi> it seems too reasonable :)
21:37:09 <strigazi> to not get it
21:37:20 <imdigitaljim> o/ ill be glad to add it back
21:37:37 <imdigitaljim> also strigazi: could we make the flannel parts also a software deployment
21:37:46 <imdigitaljim> are those necesssary to be apart of cloud-init?
21:37:58 <strigazi> we could, no they don't
21:38:06 <imdigitaljim> i've noticed we're nearing the capacity on cloud-init data
21:38:17 <flwang> if we all agree 'kubelet' back on master, then it's easy
21:38:20 <imdigitaljim> yeah
21:38:30 <flwang> we just need to drop some 'if' check for calico
21:39:04 <strigazi> that should be enough
21:39:05 <imdigitaljim> how do you all feel about leveraging a helm for prometheus/dashboard/etc
21:39:10 <imdigitaljim> instead of using our scripts going forwad?
21:39:13 <imdigitaljim> forward?
21:39:32 <imdigitaljim> helm charts and such are much cleaner/easier to maintain
21:39:33 <strigazi> we were discussing this today
21:39:53 <strigazi> yes, we could
21:40:02 <flwang> to be more clear, should kubelet on master in Rocky?
21:40:17 <strigazi> the question is
21:40:40 <strigazi> if we don't put it Rocky, how many user will cherry-pick downstream
21:40:50 <strigazi> if we don't put it Rocky, how many users will cherry-pick downstream
21:40:56 <imdigitaljim> i think it would be appropriate for rocky
21:41:12 <imdigitaljim> stein is a bit far out for such a critical change
21:41:47 <flwang> Stein will be a very long release
21:41:57 <flwang> the longest so far IIRC
21:42:17 <strigazi> yes it will
21:42:32 <flwang> I think the risk is low and the benefit is big
21:42:39 <strigazi> +1
21:43:15 <flwang> ok, then I will propose a patch this week
21:43:29 <flwang> I'm glad to see Magnum team is so productive
21:43:39 <imdigitaljim> \o/
21:43:46 <strigazi> :)
21:44:04 <colin-> yeah i think that's worthwhile, it will provide a lot of benefit for consumers
21:44:53 <strigazi> The only area we haven't push a lot, is the -1 stable branch
21:45:21 <strigazi> usually stable and master are in very good shape and are up to date
21:45:51 <strigazi> but current-stable -1 is a little behind
21:46:46 <strigazi> I don't know if we can put a lot of effort into old branches
21:47:25 <strigazi> the ones that we are present here run stable + patches
21:49:12 <strigazi> since we will push some more patches in rocky should we give it a timeline of two or three weeks?
21:50:19 <strigazi> the bracnh is cut, packagers will have everything in place, we can do as many releases as we want with non-breaking changes like these
21:50:22 <strigazi> makes sense?
21:51:04 <strigazi> imdigitaljim: flwang colin- canori02 ^^
21:51:18 <canori02> Makes sense
21:52:08 <colin-> yeah that seems reasonable
21:52:33 <imdigitaljim> yeah
21:52:37 <imdigitaljim> that sounds great
21:52:59 <strigazi> imdigitaljim: colin- you use rpms? containers? ansible?
21:53:41 <imdigitaljim> for magnum?
21:53:44 <strigazi> flwang: canori02 you?
21:53:46 <strigazi> imdigitaljim: yes
21:53:47 <imdigitaljim> containers
21:53:59 <strigazi> kolla?
21:54:13 <imdigitaljim> puppet + containers
21:54:26 <canori02> ansible here
21:54:50 <strigazi> interesting we have a spectrum
21:54:59 <strigazi> we use puppet + rpms
21:55:09 <flwang> sorry, was in standup meeting
21:55:15 <flwang> i'm reading the log
21:55:32 <strigazi> but we have a large koji infra for rpms
21:56:06 <flwang> we're using puppet+debian pkg
21:56:42 <strigazi> you deploy on debian sid?
21:57:50 <strigazi> it is  strech now
21:59:20 <strigazi> anything else folks?
21:59:58 <strigazi> see you next week or just around
22:00:04 <strigazi> #endmeeing
22:00:06 <strigazi> #endmeeting