21:01:48 <strigazi> #startmeeting containers 21:01:49 <openstack> Meeting started Tue Aug 21 21:01:48 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:52 <openstack> The meeting name has been set to 'containers' 21:01:57 <strigazi> #topic Roll Call 21:02:02 <imdigitaljim> o/ 21:02:05 <colin-> hi 21:02:05 <strigazi> o/ 21:02:51 <strigazi> #topic Announcements 21:03:03 <strigazi> kubernetes v1.11.2 is up: https://hub.docker.com/r/openstackmagnum/kubernetes-kubelet/tags/ 21:03:04 <canori02> o/ 21:03:32 <strigazi> #topic Blueprints/Bugs/Ideas 21:03:42 <strigazi> hello canori02 21:04:18 <imdigitaljim> i saw your proxy merge. I'll rebase those other ones and we'll be good to go =] 21:04:26 <imdigitaljim> on a sidenote we recently switched over to DS proxy as well 21:04:37 <imdigitaljim> so if you want i can throw a PR for that at some point? 21:04:57 <strigazi> imdigitaljim: you changed to DS downstream? 21:05:02 <imdigitaljim> yeah 21:05:25 <imdigitaljim> making motion towards self-hosted 21:05:36 <imdigitaljim> to make in-place upgrades smoother 21:05:40 <strigazi> I think it is better to move to DS if we have a plan for most of the components 21:06:06 <imdigitaljim> well ill get a PR up for it soon 21:06:09 <strigazi> you have only proxt as DS 21:06:10 <strigazi> ? 21:06:15 <imdigitaljim> yeah currently 21:06:21 <imdigitaljim> everything else is static still for now 21:06:34 <strigazi> and calico is a DS right? 21:06:42 <imdigitaljim> yes 21:06:49 <imdigitaljim> and keystone-auth plugin is DS 21:06:52 <imdigitaljim> for masters 21:07:37 <imdigitaljim> nodeSelector: node-role.kubernetes.io/master: "" 21:07:39 <imdigitaljim> in other words 21:07:55 <imdigitaljim> ccm is the same as well 21:08:24 <strigazi> node selector can be used for api and sch and cm I guess 21:08:39 <imdigitaljim> in most deployments that is exactly whats its for 21:08:53 <imdigitaljim> but yeah that'd be appropriate too 21:09:02 <imdigitaljim> and a few tolerations 21:09:49 <imdigitaljim> i have some concerns about the reliability on self-hosted 21:09:53 <imdigitaljim> in terms of node reboot 21:09:59 <imdigitaljim> so i think ill have to plan those out 21:10:06 <imdigitaljim> and see what our 'competition' does 21:10:42 <imdigitaljim> but in general i think our near term goals will be in-place upgrade 21:10:44 <strigazi> static pods shouldn't be a problem if kubelet starts 21:11:03 <imdigitaljim> yeah thats what i was thinking 21:11:10 <imdigitaljim> except we'd probably need to keep etcd static as well 21:11:13 <imdigitaljim> without etcd it cant join 21:11:29 <imdigitaljim> what kubeadm does is it throws up a static set for rejoining 21:11:39 <imdigitaljim> and then tears it down when reconnected to the clusterr 21:11:56 <imdigitaljim> which i think would be preferable 21:12:02 <strigazi> What do you mean with a static set? 21:12:38 <strigazi> for single master that I have tried, it is just a static pod, isn't it? 21:12:54 <imdigitaljim> it ends up being one yes 21:13:10 <imdigitaljim> i think i'll have to investigate the workflow a little more 21:13:27 <imdigitaljim> make sure im also understanding it correctly 21:13:40 <strigazi> if the data of etcd are in place, rebooting the node isn't a problem 21:13:59 <imdigitaljim> but if etcd is self-hosted as well 21:14:05 <strigazi> using a static pod or not 21:14:07 <imdigitaljim> theres no apiserver online to interact with it 21:14:09 <imdigitaljim> no? 21:14:34 <strigazi> kubelet can't start pods without an api server 21:14:36 <imdigitaljim> (multimaster scenario) 21:14:41 <imdigitaljim> exactly 21:15:07 <strigazi> In multimaster I don't know what kubeadm does 21:15:23 <strigazi> if it converts the static pods to a deployment 21:15:26 <strigazi> or ds 21:15:27 <imdigitaljim> i think this problem is significantly easier in single master 21:16:04 <strigazi> well if, you use static pods, it is the "same" for multi and single master 21:16:26 <imdigitaljim> yeah 21:16:32 <strigazi> if reboot one by one 21:17:15 <imdigitaljim> are you assuming etcd is also self-hosted in this? 21:17:17 <strigazi> if you reboot all of them in one go maybe the result is different 21:17:19 <imdigitaljim> or not 21:17:42 <strigazi> I don't see a diference 21:18:22 <strigazi> kubelet for static pods is like systemd for processes :) 21:18:30 <flwang> sorry i'm late 21:18:45 <colin-> welcome 21:18:58 <imdigitaljim> https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.10.md#optional-and-alpha-in-v19-self-hosting 21:19:33 <imdigitaljim> i believe there are some unaccounted for situations 21:20:44 <strigazi> oh, you mean make even etcd a ds 21:20:55 <strigazi> I wouldn't do that :) 21:21:10 <strigazi> it sounds terrifying 21:21:13 <imdigitaljim> haha 21:21:24 <imdigitaljim> yeah just some concerns to look at 21:21:35 <imdigitaljim> definitely possible to overcome though 21:22:07 <strigazi> ok, since flwang is also in, 21:22:37 <flwang> ds for etcd? 21:22:51 <colin-> yes, mad science lab :) 21:23:00 <strigazi> I was rebasing the upgrades api and I spent a good three hourds debugging. (ds for etcd, yes) 21:23:54 <strigazi> The new change for the svc account keys was "breaking" the functionality 21:24:21 <flwang> strigazi: breaking? 21:24:37 <strigazi> but I finally, figured it out, when magnum creates the dict for params to pass to heat 21:25:00 <strigazi> it adds and generate the keys for the service account 21:25:38 <strigazi> and if since it generates every time magnum creates the params it means that the keys will be different 21:25:52 <strigazi> "breaking" not really breaking 21:26:01 <flwang> ah 21:26:11 <flwang> so do we need any change for the key generating? 21:26:40 <strigazi> maybe, but for now I just ignore those two params 21:26:57 <strigazi> I'll push after the meeting 21:27:00 <imdigitaljim> thats a crazy issue 21:27:00 <imdigitaljim> yeah 21:27:07 <imdigitaljim> maybe we should save in barbican 21:27:09 <imdigitaljim> or 21:27:14 <flwang> strigazi: ok, i will keep an eye on that part 21:27:14 <imdigitaljim> whatever secret backend 21:27:28 <imdigitaljim> and extract or create depending on update/create? 21:27:35 <strigazi> we should do the same thing we do for the ca 21:27:48 <imdigitaljim> so yeah^ 21:27:52 <strigazi> imdigitaljim we can do that too 21:28:39 <strigazi> the only thing I don't like is having the secret in barbican and then passing it as a heat paramter 21:28:52 <colin-> i think that would be useful without too much cost 21:28:59 <colin-> oh 21:29:02 <strigazi> It will stay in heat db for ever 21:29:10 <colin-> that's lame 21:29:11 <imdigitaljim> thats true 21:29:22 <strigazi> encrypted, but still 21:29:27 <imdigitaljim> can trustee be extended to allow cluster to interact with barbican? 21:29:34 <strigazi> yes 21:29:47 <imdigitaljim> (or secret backend) 21:30:11 <strigazi> we don't have to do anything actually, the trustee with the trust can talk to barbican 21:30:18 <imdigitaljim> yeah 21:30:28 <imdigitaljim> thats what i was hoping :] 21:30:32 <strigazi> vault or other is different 21:30:32 <imdigitaljim> i havent actually tried it 21:30:51 <imdigitaljim> people can add support for other backend as they need imho 21:30:52 <strigazi> Let's see in stein 21:31:10 <flwang> imdigitaljim: i'm interested in your keystone implementation btw 21:31:20 <imdigitaljim> oh sure 21:31:38 <flwang> imdigitaljim: especially if it needs api restart 21:31:46 <imdigitaljim> ive been working through some changes on new features here and we just accepted some alpha customers so we've been tied up 21:31:51 <imdigitaljim> but ill revisit those PR's and then some 21:32:10 <imdigitaljim> which api restart? 21:32:29 <flwang> k8s api server 21:33:02 <imdigitaljim> oh im not doing a restart, im confused? 21:33:06 <strigazi> why it needs an api restart? 21:33:09 <flwang> because based on testing, you have to restart k8s api server after you got the service URL of keystone auth service 21:33:21 <flwang> if it's deployed as DS 21:33:32 <imdigitaljim> oh yeah i havent needed to at all 21:33:33 <strigazi> flwang: when I tried I didn't restarted the k8s-api 21:33:51 <strigazi> flwang you can use host-network 21:34:07 <imdigitaljim> ^ 21:34:11 <imdigitaljim> i dont know if you're using gthat 21:34:12 <flwang> then I really want to know how did you do that if you don't deployed it on master 21:34:18 <flwang> master's kubelet 21:34:20 <imdigitaljim> but we are doing hostNetwork: true 21:34:28 <imdigitaljim> i do 21:34:35 <imdigitaljim> tolerations: 21:34:35 <imdigitaljim> - key: dedicated 21:34:35 <imdigitaljim> value: master 21:34:35 <imdigitaljim> effect: NoSchedule 21:34:35 <imdigitaljim> - key: CriticalAddonsOnly 21:34:37 <imdigitaljim> value: "True" 21:34:39 <imdigitaljim> effect: NoSchedule 21:34:41 <imdigitaljim> nodeSelector: 21:34:43 <imdigitaljim> node-role.kubernetes.io/master: "" 21:35:26 <imdigitaljim> we're putting most kube-system resources on masters 21:35:37 <flwang> so your k8s-keystone-auth service is running as DS and not running on master, right? 21:35:41 <imdigitaljim> not like dashboards and whatnot 21:35:46 <imdigitaljim> its running a DS on master 21:35:50 <imdigitaljim> and not on minion 21:35:53 <flwang> ah, right 21:36:24 <flwang> then yes, for that case, it's much easier and could avoid the api restart 21:36:33 <imdigitaljim> oh okay yeah 21:36:38 <flwang> so, should we just get the kubelet back on master? 21:36:54 <strigazi> it seems too reasonable :) 21:37:09 <strigazi> to not get it 21:37:20 <imdigitaljim> o/ ill be glad to add it back 21:37:37 <imdigitaljim> also strigazi: could we make the flannel parts also a software deployment 21:37:46 <imdigitaljim> are those necesssary to be apart of cloud-init? 21:37:58 <strigazi> we could, no they don't 21:38:06 <imdigitaljim> i've noticed we're nearing the capacity on cloud-init data 21:38:17 <flwang> if we all agree 'kubelet' back on master, then it's easy 21:38:20 <imdigitaljim> yeah 21:38:30 <flwang> we just need to drop some 'if' check for calico 21:39:04 <strigazi> that should be enough 21:39:05 <imdigitaljim> how do you all feel about leveraging a helm for prometheus/dashboard/etc 21:39:10 <imdigitaljim> instead of using our scripts going forwad? 21:39:13 <imdigitaljim> forward? 21:39:32 <imdigitaljim> helm charts and such are much cleaner/easier to maintain 21:39:33 <strigazi> we were discussing this today 21:39:53 <strigazi> yes, we could 21:40:02 <flwang> to be more clear, should kubelet on master in Rocky? 21:40:17 <strigazi> the question is 21:40:40 <strigazi> if we don't put it Rocky, how many user will cherry-pick downstream 21:40:50 <strigazi> if we don't put it Rocky, how many users will cherry-pick downstream 21:40:56 <imdigitaljim> i think it would be appropriate for rocky 21:41:12 <imdigitaljim> stein is a bit far out for such a critical change 21:41:47 <flwang> Stein will be a very long release 21:41:57 <flwang> the longest so far IIRC 21:42:17 <strigazi> yes it will 21:42:32 <flwang> I think the risk is low and the benefit is big 21:42:39 <strigazi> +1 21:43:15 <flwang> ok, then I will propose a patch this week 21:43:29 <flwang> I'm glad to see Magnum team is so productive 21:43:39 <imdigitaljim> \o/ 21:43:46 <strigazi> :) 21:44:04 <colin-> yeah i think that's worthwhile, it will provide a lot of benefit for consumers 21:44:53 <strigazi> The only area we haven't push a lot, is the -1 stable branch 21:45:21 <strigazi> usually stable and master are in very good shape and are up to date 21:45:51 <strigazi> but current-stable -1 is a little behind 21:46:46 <strigazi> I don't know if we can put a lot of effort into old branches 21:47:25 <strigazi> the ones that we are present here run stable + patches 21:49:12 <strigazi> since we will push some more patches in rocky should we give it a timeline of two or three weeks? 21:50:19 <strigazi> the bracnh is cut, packagers will have everything in place, we can do as many releases as we want with non-breaking changes like these 21:50:22 <strigazi> makes sense? 21:51:04 <strigazi> imdigitaljim: flwang colin- canori02 ^^ 21:51:18 <canori02> Makes sense 21:52:08 <colin-> yeah that seems reasonable 21:52:33 <imdigitaljim> yeah 21:52:37 <imdigitaljim> that sounds great 21:52:59 <strigazi> imdigitaljim: colin- you use rpms? containers? ansible? 21:53:41 <imdigitaljim> for magnum? 21:53:44 <strigazi> flwang: canori02 you? 21:53:46 <strigazi> imdigitaljim: yes 21:53:47 <imdigitaljim> containers 21:53:59 <strigazi> kolla? 21:54:13 <imdigitaljim> puppet + containers 21:54:26 <canori02> ansible here 21:54:50 <strigazi> interesting we have a spectrum 21:54:59 <strigazi> we use puppet + rpms 21:55:09 <flwang> sorry, was in standup meeting 21:55:15 <flwang> i'm reading the log 21:55:32 <strigazi> but we have a large koji infra for rpms 21:56:06 <flwang> we're using puppet+debian pkg 21:56:42 <strigazi> you deploy on debian sid? 21:57:50 <strigazi> it is strech now 21:59:20 <strigazi> anything else folks? 21:59:58 <strigazi> see you next week or just around 22:00:04 <strigazi> #endmeeing 22:00:06 <strigazi> #endmeeting