21:01:48 #startmeeting containers 21:01:49 Meeting started Tue Aug 21 21:01:48 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:50 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:52 The meeting name has been set to 'containers' 21:01:57 #topic Roll Call 21:02:02 o/ 21:02:05 hi 21:02:05 o/ 21:02:51 #topic Announcements 21:03:03 kubernetes v1.11.2 is up: https://hub.docker.com/r/openstackmagnum/kubernetes-kubelet/tags/ 21:03:04 o/ 21:03:32 #topic Blueprints/Bugs/Ideas 21:03:42 hello canori02 21:04:18 i saw your proxy merge. I'll rebase those other ones and we'll be good to go =] 21:04:26 on a sidenote we recently switched over to DS proxy as well 21:04:37 so if you want i can throw a PR for that at some point? 21:04:57 imdigitaljim: you changed to DS downstream? 21:05:02 yeah 21:05:25 making motion towards self-hosted 21:05:36 to make in-place upgrades smoother 21:05:40 I think it is better to move to DS if we have a plan for most of the components 21:06:06 well ill get a PR up for it soon 21:06:09 you have only proxt as DS 21:06:10 ? 21:06:15 yeah currently 21:06:21 everything else is static still for now 21:06:34 and calico is a DS right? 21:06:42 yes 21:06:49 and keystone-auth plugin is DS 21:06:52 for masters 21:07:37 nodeSelector: node-role.kubernetes.io/master: "" 21:07:39 in other words 21:07:55 ccm is the same as well 21:08:24 node selector can be used for api and sch and cm I guess 21:08:39 in most deployments that is exactly whats its for 21:08:53 but yeah that'd be appropriate too 21:09:02 and a few tolerations 21:09:49 i have some concerns about the reliability on self-hosted 21:09:53 in terms of node reboot 21:09:59 so i think ill have to plan those out 21:10:06 and see what our 'competition' does 21:10:42 but in general i think our near term goals will be in-place upgrade 21:10:44 static pods shouldn't be a problem if kubelet starts 21:11:03 yeah thats what i was thinking 21:11:10 except we'd probably need to keep etcd static as well 21:11:13 without etcd it cant join 21:11:29 what kubeadm does is it throws up a static set for rejoining 21:11:39 and then tears it down when reconnected to the clusterr 21:11:56 which i think would be preferable 21:12:02 What do you mean with a static set? 21:12:38 for single master that I have tried, it is just a static pod, isn't it? 21:12:54 it ends up being one yes 21:13:10 i think i'll have to investigate the workflow a little more 21:13:27 make sure im also understanding it correctly 21:13:40 if the data of etcd are in place, rebooting the node isn't a problem 21:13:59 but if etcd is self-hosted as well 21:14:05 using a static pod or not 21:14:07 theres no apiserver online to interact with it 21:14:09 no? 21:14:34 kubelet can't start pods without an api server 21:14:36 (multimaster scenario) 21:14:41 exactly 21:15:07 In multimaster I don't know what kubeadm does 21:15:23 if it converts the static pods to a deployment 21:15:26 or ds 21:15:27 i think this problem is significantly easier in single master 21:16:04 well if, you use static pods, it is the "same" for multi and single master 21:16:26 yeah 21:16:32 if reboot one by one 21:17:15 are you assuming etcd is also self-hosted in this? 21:17:17 if you reboot all of them in one go maybe the result is different 21:17:19 or not 21:17:42 I don't see a diference 21:18:22 kubelet for static pods is like systemd for processes :) 21:18:30 sorry i'm late 21:18:45 welcome 21:18:58 https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.10.md#optional-and-alpha-in-v19-self-hosting 21:19:33 i believe there are some unaccounted for situations 21:20:44 oh, you mean make even etcd a ds 21:20:55 I wouldn't do that :) 21:21:10 it sounds terrifying 21:21:13 haha 21:21:24 yeah just some concerns to look at 21:21:35 definitely possible to overcome though 21:22:07 ok, since flwang is also in, 21:22:37 ds for etcd? 21:22:51 yes, mad science lab :) 21:23:00 I was rebasing the upgrades api and I spent a good three hourds debugging. (ds for etcd, yes) 21:23:54 The new change for the svc account keys was "breaking" the functionality 21:24:21 strigazi: breaking? 21:24:37 but I finally, figured it out, when magnum creates the dict for params to pass to heat 21:25:00 it adds and generate the keys for the service account 21:25:38 and if since it generates every time magnum creates the params it means that the keys will be different 21:25:52 "breaking" not really breaking 21:26:01 ah 21:26:11 so do we need any change for the key generating? 21:26:40 maybe, but for now I just ignore those two params 21:26:57 I'll push after the meeting 21:27:00 thats a crazy issue 21:27:00 yeah 21:27:07 maybe we should save in barbican 21:27:09 or 21:27:14 strigazi: ok, i will keep an eye on that part 21:27:14 whatever secret backend 21:27:28 and extract or create depending on update/create? 21:27:35 we should do the same thing we do for the ca 21:27:48 so yeah^ 21:27:52 imdigitaljim we can do that too 21:28:39 the only thing I don't like is having the secret in barbican and then passing it as a heat paramter 21:28:52 i think that would be useful without too much cost 21:28:59 oh 21:29:02 It will stay in heat db for ever 21:29:10 that's lame 21:29:11 thats true 21:29:22 encrypted, but still 21:29:27 can trustee be extended to allow cluster to interact with barbican? 21:29:34 yes 21:29:47 (or secret backend) 21:30:11 we don't have to do anything actually, the trustee with the trust can talk to barbican 21:30:18 yeah 21:30:28 thats what i was hoping :] 21:30:32 vault or other is different 21:30:32 i havent actually tried it 21:30:51 people can add support for other backend as they need imho 21:30:52 Let's see in stein 21:31:10 imdigitaljim: i'm interested in your keystone implementation btw 21:31:20 oh sure 21:31:38 imdigitaljim: especially if it needs api restart 21:31:46 ive been working through some changes on new features here and we just accepted some alpha customers so we've been tied up 21:31:51 but ill revisit those PR's and then some 21:32:10 which api restart? 21:32:29 k8s api server 21:33:02 oh im not doing a restart, im confused? 21:33:06 why it needs an api restart? 21:33:09 because based on testing, you have to restart k8s api server after you got the service URL of keystone auth service 21:33:21 if it's deployed as DS 21:33:32 oh yeah i havent needed to at all 21:33:33 flwang: when I tried I didn't restarted the k8s-api 21:33:51 flwang you can use host-network 21:34:07 ^ 21:34:11 i dont know if you're using gthat 21:34:12 then I really want to know how did you do that if you don't deployed it on master 21:34:18 master's kubelet 21:34:20 but we are doing hostNetwork: true 21:34:28 i do 21:34:35 tolerations: 21:34:35 - key: dedicated 21:34:35 value: master 21:34:35 effect: NoSchedule 21:34:35 - key: CriticalAddonsOnly 21:34:37 value: "True" 21:34:39 effect: NoSchedule 21:34:41 nodeSelector: 21:34:43 node-role.kubernetes.io/master: "" 21:35:26 we're putting most kube-system resources on masters 21:35:37 so your k8s-keystone-auth service is running as DS and not running on master, right? 21:35:41 not like dashboards and whatnot 21:35:46 its running a DS on master 21:35:50 and not on minion 21:35:53 ah, right 21:36:24 then yes, for that case, it's much easier and could avoid the api restart 21:36:33 oh okay yeah 21:36:38 so, should we just get the kubelet back on master? 21:36:54 it seems too reasonable :) 21:37:09 to not get it 21:37:20 o/ ill be glad to add it back 21:37:37 also strigazi: could we make the flannel parts also a software deployment 21:37:46 are those necesssary to be apart of cloud-init? 21:37:58 we could, no they don't 21:38:06 i've noticed we're nearing the capacity on cloud-init data 21:38:17 if we all agree 'kubelet' back on master, then it's easy 21:38:20 yeah 21:38:30 we just need to drop some 'if' check for calico 21:39:04 that should be enough 21:39:05 how do you all feel about leveraging a helm for prometheus/dashboard/etc 21:39:10 instead of using our scripts going forwad? 21:39:13 forward? 21:39:32 helm charts and such are much cleaner/easier to maintain 21:39:33 we were discussing this today 21:39:53 yes, we could 21:40:02 to be more clear, should kubelet on master in Rocky? 21:40:17 the question is 21:40:40 if we don't put it Rocky, how many user will cherry-pick downstream 21:40:50 if we don't put it Rocky, how many users will cherry-pick downstream 21:40:56 i think it would be appropriate for rocky 21:41:12 stein is a bit far out for such a critical change 21:41:47 Stein will be a very long release 21:41:57 the longest so far IIRC 21:42:17 yes it will 21:42:32 I think the risk is low and the benefit is big 21:42:39 +1 21:43:15 ok, then I will propose a patch this week 21:43:29 I'm glad to see Magnum team is so productive 21:43:39 \o/ 21:43:46 :) 21:44:04 yeah i think that's worthwhile, it will provide a lot of benefit for consumers 21:44:53 The only area we haven't push a lot, is the -1 stable branch 21:45:21 usually stable and master are in very good shape and are up to date 21:45:51 but current-stable -1 is a little behind 21:46:46 I don't know if we can put a lot of effort into old branches 21:47:25 the ones that we are present here run stable + patches 21:49:12 since we will push some more patches in rocky should we give it a timeline of two or three weeks? 21:50:19 the bracnh is cut, packagers will have everything in place, we can do as many releases as we want with non-breaking changes like these 21:50:22 makes sense? 21:51:04 imdigitaljim: flwang colin- canori02 ^^ 21:51:18 Makes sense 21:52:08 yeah that seems reasonable 21:52:33 yeah 21:52:37 that sounds great 21:52:59 imdigitaljim: colin- you use rpms? containers? ansible? 21:53:41 for magnum? 21:53:44 flwang: canori02 you? 21:53:46 imdigitaljim: yes 21:53:47 containers 21:53:59 kolla? 21:54:13 puppet + containers 21:54:26 ansible here 21:54:50 interesting we have a spectrum 21:54:59 we use puppet + rpms 21:55:09 sorry, was in standup meeting 21:55:15 i'm reading the log 21:55:32 but we have a large koji infra for rpms 21:56:06 we're using puppet+debian pkg 21:56:42 you deploy on debian sid? 21:57:50 it is strech now 21:59:20 anything else folks? 21:59:58 see you next week or just around 22:00:04 #endmeeing 22:00:06 #endmeeting