| *** sapd1_x has joined #openstack-containers | 03:17 | |
| *** rcernin has quit IRC | 03:25 | |
| *** rcernin_ has joined #openstack-containers | 03:26 | |
| *** dave-mccowan has quit IRC | 04:07 | |
| *** jmlowe has quit IRC | 04:18 | |
| *** ykarel has joined #openstack-containers | 04:29 | |
| *** jmlowe has joined #openstack-containers | 04:29 | |
| *** ykarel has quit IRC | 04:37 | |
| *** ykarel has joined #openstack-containers | 04:38 | |
| *** ykarel has quit IRC | 04:41 | |
| *** ykarel has joined #openstack-containers | 04:44 | |
| *** ykarel has quit IRC | 04:46 | |
| *** ykarel has joined #openstack-containers | 04:48 | |
| *** ykarel has quit IRC | 04:54 | |
| *** ykarel has joined #openstack-containers | 04:56 | |
| *** ykarel has quit IRC | 05:08 | |
| *** ykarel has joined #openstack-containers | 05:10 | |
| *** ykarel has quit IRC | 05:17 | |
| *** ykarel has joined #openstack-containers | 05:18 | |
| *** ykarel has quit IRC | 05:28 | |
| *** ykarel has joined #openstack-containers | 05:31 | |
| *** vishalmanchanda has joined #openstack-containers | 06:49 | |
| *** mgoddard has quit IRC | 08:10 | |
| *** mgoddard has joined #openstack-containers | 08:26 | |
| *** flwang1 has joined #openstack-containers | 08:27 | |
| *** nikparasyr has joined #openstack-containers | 08:41 | |
| flwang1 | brtknr: spyros: around? | 08:47 |
|---|---|---|
| brtknr | flwang1: morning! | 08:48 |
| *** iokiwi has quit IRC | 08:48 | |
| flwang1 | brtknr: got a good sleep? | 08:48 |
| *** dioguerra has joined #openstack-containers | 08:48 | |
| brtknr | flwang1: ~3 hours | 08:48 |
| *** iokiwi has joined #openstack-containers | 08:49 | |
| brtknr | i had the day off yesterday | 08:49 |
| flwang1 | brtknr: oh, no | 08:49 |
| flwang1 | let's see if Spyros will join us, otherwise, we can cancel and talk later | 08:49 |
| brtknr | but they are both getting better I think, we went to get tested for Covid... waiting for result later today | 08:50 |
| brtknr | I am pretty sure its just common cold as me and my partner have no symptoms and the babies have never been ill before | 08:51 |
| *** strigazi has joined #openstack-containers | 08:51 | |
| brtknr | apart from when they had their vaccines | 08:52 |
| flwang1 | oh, god bless you | 08:54 |
| flwang1 | i believe you will ok | 08:54 |
| flwang1 | strigazi: hello there | 08:54 |
| strigazi | flwang1: hello o/ | 08:55 |
| flwang1 | strigazi: good to see you | 08:56 |
| flwang1 | team, add your topics on https://etherpad.opendev.org/p/magnum-weekly-meeting | 08:56 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Add CT observations field to the database and API https://review.opendev.org/737840 | 09:00 |
| flwang1 | #startmeeting magnum | 09:01 |
| openstack | Meeting started Wed Aug 26 09:01:01 2020 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:01 |
| *** openstack changes topic to " (Meeting topic: magnum)" | 09:01 | |
| openstack | The meeting name has been set to 'magnum' | 09:01 |
| flwang1 | #topic roll call | 09:01 |
| *** openstack changes topic to "roll call (Meeting topic: magnum)" | 09:01 | |
| flwang1 | o/ | 09:01 |
| brtknr | o/ | 09:01 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Very-WIP: Drop hyperkube https://review.opendev.org/748141 | 09:01 |
| dioguerra | o/ | 09:01 |
| strigazi | o/ | 09:01 |
| flwang1 | thank you for join, guys | 09:02 |
| flwang1 | shall we go through the topic list? | 09:03 |
| strigazi | +1 | 09:03 |
| flwang1 | ok, recently we (catalyst cloud) just done a security review by 3rd party and we got some good comments to improve | 09:04 |
| flwang1 | hence you probably see there is a patch i proposed to separate the ca for k8s, etcd and front-proxy, though we discussed this long time ago | 09:05 |
| *** sapd1_x has quit IRC | 09:05 | |
| flwang1 | the patch https://review.opendev.org/746864 has been rebased on the ca rotate patch and tested locally | 09:06 |
| strigazi | We knew that before, right? That each node could contact etcd. | 09:06 |
| flwang1 | strigazi: yep | 09:06 |
| flwang1 | each node can use the kubelet cert to access etcd | 09:06 |
| flwang1 | to be more clear ^ | 09:06 |
| strigazi | or kube-proxy | 09:07 |
| flwang1 | yes | 09:07 |
| strigazi | or any cert signed by the CA | 09:07 |
| flwang1 | so, please review that one asap, hopefully we can get it in this V cycle | 09:07 |
| strigazi | I have a question though | 09:08 |
| flwang1 | sure | 09:08 |
| flwang1 | i'm listening | 09:08 |
| strigazi | We need this patch. I'm not against it by any means it helps a lot. | 09:08 |
| strigazi | The trust and trustee are in all nodes anyway. So one can get whatever certs they need, this is known, right? | 09:09 |
| strigazi | And | 09:09 |
| strigazi | What about serving service account keypair via the API? | 09:09 |
| strigazi | That's it | 09:10 |
| flwang1 | re the trust and trustee, yep. that's a good point, we can try to limit the request in the future to only allow it came from master node | 09:10 |
| flwang1 | though i don't know how yet | 09:11 |
| strigazi | So RBAC on magnum API | 09:11 |
| strigazi | I see different trustees or application creds as a solution. | 09:11 |
| flwang1 | some silly(home-made) RBAC probably | 09:11 |
| flwang1 | strigazi: that also works | 09:11 |
| strigazi | Why silly, it can't get better than this | 09:12 |
| strigazi | different policy per role | 09:12 |
| flwang1 | sorry, i mean, openstack don't reayll have a good rbac design yet | 09:12 |
| strigazi | This point needs discussion. Maybe a spec. I just wanted to mention it. | 09:13 |
| strigazi | For the 2nd point? | 09:13 |
| strigazi | Since we pass type in the API we could serve the service account RSA keypair | 09:14 |
| flwang1 | yep, it's a very good point. thanks for the reminder | 09:14 |
| flwang1 | strigazi: can you explain more about the benefit of serving the service account rsa keypair by api? | 09:15 |
| strigazi | Add a new master NG | 09:15 |
| strigazi | ATM we can't access the RSA keypair. It is a hidden (as it should) parameter in the heat stack | 09:16 |
| flwang1 | master NG for what? resizing? | 09:16 |
| strigazi | resizing is not blocked by this | 09:17 |
| strigazi | adding a new one | 09:17 |
| strigazi | master-ng-B | 09:17 |
| flwang1 | i can't see the point of have another master NG | 09:17 |
| flwang1 | for worker nodes, NG makes good sense | 09:17 |
| strigazi | I want to use bigger master nodes, how do I do this? | 09:18 |
| flwang1 | but what's the value of multi master NG | 09:18 |
| flwang1 | nova resizing? | 09:18 |
| strigazi | That's an option, but you see my point. | 09:19 |
| flwang1 | sure | 09:19 |
| flwang1 | very good comments, i appreciate it | 09:19 |
| flwang1 | next one? | 09:20 |
| strigazi | +1 | 09:20 |
| flwang1 | #topic PodSecurityPolicy and Calico | 09:20 |
| *** openstack changes topic to "PodSecurityPolicy and Calico (Meeting topic: magnum)" | 09:20 | |
| strigazi | What did they break again? | 09:20 |
| flwang1 | i'm still working on this, maybe you guys can help confirm | 09:20 |
| flwang1 | after using --admission-controller-list label with PodSecurityPolicy, the calico pod can't start, the cluster can't be started | 09:21 |
| flwang1 | with this post http://blog.tundeoladipupo.com/2019/06/01/Kubernetes,-PodSecurityPolicy-and-Kubeadm/ i think we need a dedicated psp for calico if we want to enable PodSecurityPolicy | 09:22 |
| strigazi | we have a very privileged PSP for calico, it doesn't work? | 09:22 |
| strigazi | calico is using a PSP already | 09:23 |
| flwang1 | strigazi: good to know. i haven't reviewed the code yet | 09:23 |
| flwang1 | i will do another test, and it would be nice if you guys can help test it as well. | 09:23 |
| flwang1 | the PSP is getting a common requirement for enterprise user | 09:23 |
| flwang1 | EKS is enabling it by default | 09:24 |
| flwang1 | btw, i just found our default admission list in the code is very old, should we update it? | 09:24 |
| strigazi | sure | 09:24 |
| strigazi | At CERN we have it in out CT | 09:24 |
| strigazi | At CERN we have it in our CT | 09:24 |
| flwang1 | right | 09:25 |
| strigazi | https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/calico-service-v3-3-x.sh#L18 | 09:25 |
| flwang1 | i will propose a patch based on the default list from v1.16.x, sounds ok? | 09:25 |
| *** k_mouza has joined #openstack-containers | 09:26 | |
| *** ykarel_ has joined #openstack-containers | 09:27 | |
| strigazi | sure | 09:27 |
| strigazi | maybe for V we do the list for 1.19? | 09:27 |
| flwang1 | strigazi: can do | 09:28 |
| strigazi | cool | 09:29 |
| flwang1 | re the calico psp, maybe i missed something, but at line https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/calico-service.sh#L30 | 09:29 |
| *** ykarel has quit IRC | 09:29 | |
| strigazi | yeap, that's it | 09:29 |
| flwang1 | i can see this role name, but seems we didn't create the role? | 09:29 |
| strigazi | we do | 09:29 |
| strigazi | https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/kube-apiserver-to-kubelet-role.sh#L125 | 09:30 |
| *** ykarel_ has quit IRC | 09:30 | |
| flwang1 | ok, got it, so we are using magnum.privileged as the psp | 09:31 |
| strigazi | yes | 09:31 |
| strigazi | same as GKE was doing (at least when I checked) | 09:31 |
| flwang1 | ok, i will test this again | 09:32 |
| flwang1 | let's move on? | 09:32 |
| strigazi | +1 | 09:32 |
| flwang1 | #topic Add observations field to CT | 09:32 |
| *** openstack changes topic to "Add observations field to CT (Meeting topic: magnum)" | 09:32 | |
| flwang1 | dioguerra: ^ | 09:33 |
| *** ykarel has joined #openstack-containers | 09:33 | |
| dioguerra | Hey, so the idea of this is to add a new field visible to the user where we can add observations | 09:34 |
| dioguerra | The idea came so that we could label the CT with DEPRECATED/PRODUCTION/BETA or something similar | 09:34 |
| flwang1 | does the field have to be a enum? | 09:35 |
| flwang1 | or the list can be defined by the admin via config option? | 09:35 |
| flwang1 | sorry, i haven't read the code yet | 09:35 |
| dioguerra | no, i made it so it is free text (other admins might want to add other observations like HA, Multimaster | 09:35 |
| dioguerra | you agree? | 09:36 |
| flwang1 | i'm not sure, if it's kind like a label or tag, then if i want to have several tags for the CT, i have to do something like AAA-BBB-CCC, is it? | 09:37 |
| dioguerra | its not a label, its a new field with free text | 09:39 |
| dioguerra | so --obersvations <something> | 09:39 |
| *** rcernin_ has quit IRC | 09:39 | |
| flwang1 | i understand | 09:39 |
| flwang1 | i'm just saying a free text is making it like a free tag | 09:40 |
| brtknr | Its basically a tag to filter cluster templates right? "observations" is a bit of a mouthful. Can we just call it "tags" | 09:40 |
| *** k_mouza has quit IRC | 09:40 | |
| brtknr | does the current implementation allow user to filter using this field? | 09:41 |
| dioguerra | it is not a filter (although you might use it for that) its just a ref http://paste.openstack.org/show/797160/ | 09:41 |
| dioguerra | brtknr: no | 09:42 |
| flwang1 | hmm...i understand this is neat than putting the HA or Prod into the cluster template name, but I expect to make it more useful to deserve having a dedicated db field | 09:43 |
| flwang1 | dioguerra: i'm not saying i don't like the idea. actually it will be useful, i probably need a bit time to think about it. | 09:44 |
| dioguerra | well, the idea of doing filtering with the field cross my mind. We can add it now if you would like or later... | 09:45 |
| brtknr | dioguerra: i think that would be the thing which would add value to this proposal | 09:46 |
| flwang1 | shall we move to next topic? | 09:46 |
| jakeyip | will it be easier to filter using tags instead of free text? | 09:47 |
| flwang1 | we have 15 mins left | 09:47 |
| jakeyip | database don't match well on TEXT | 09:47 |
| jakeyip | sorry, cont' please | 09:47 |
| dioguerra | in our usecase we usually only have some visible templates (3 to 4) so filtering is not really required. everything else is hidden | 09:49 |
| dioguerra | jakeyip: tag would be better for the DB yes. but that restricts what you want to put. | 09:49 |
| jakeyip | do we have a description field? | 09:50 |
| flwang1 | i would suggest putting the discussion into the patch | 09:50 |
| flwang1 | not here | 09:50 |
| flwang1 | move on? | 09:50 |
| jakeyip | +1 | 09:50 |
| flwang1 | #topic Drop hyperkube https://review.opendev.org/748141 | 09:51 |
| *** openstack changes topic to "Drop hyperkube https://review.opendev.org/748141 (Meeting topic: magnum)" | 09:51 | |
| flwang1 | strigazi: ^ | 09:51 |
| flwang1 | tell us more? | 09:51 |
| strigazi | k8s community and some ex-openstack members though we should not have too much fun with hyperkube and dropped it. | 09:51 |
| strigazi | I have a solution there that gets a tarball with kubelet, kubectl, kubadm and kube-proxy (90mb) | 09:52 |
| strigazi | it works, running the kubelet from a bin | 09:52 |
| strigazi | and the rest components from their respective images. | 09:52 |
| strigazi | All good so far, now the problems | 09:53 |
| flwang1 | kubeadm? | 09:53 |
| strigazi | flwang1: well, it is there | 09:53 |
| strigazi | flwang1: can't skip it | 09:53 |
| strigazi | even the kube-proxy binary, we don't need it | 09:53 |
| flwang1 | sounds like another breaking change | 09:53 |
| strigazi | we need only kubelet and kubectl | 09:53 |
| strigazi | flwang1: which one? kubadm? | 09:54 |
| brtknr | hmm why does k8s.gcr.io make other binaries available in containers but not kubelet I wonder | 09:54 |
| strigazi | flwang1: I just mention what is in the tarball. Is it clear? | 09:55 |
| flwang1 | strigazi: i wonder if the new change will allow old cluster to be upgraded to | 09:55 |
| brtknr | i suppose that is why the PS is "Very WIP" | 09:55 |
| strigazi | flwang1: there is literally nothing we can do for clusters that reference k8s.gcr.io/hyperkube. | 09:55 |
| strigazi | flwang1: if your clusters use <my-registry>/hyperkube, we can build a hyperkube | 09:56 |
| *** k_mouza has joined #openstack-containers | 09:57 | |
| dioguerra | i need to go o/ | 09:57 |
| strigazi | brtknr: does it matter? They decided to stop building it. (the reason is CVEs in debian base image) | 09:57 |
| strigazi | brtknr: flwang1: hello? | 09:58 |
| flwang1 | strigazi: i appreciate the work, my only concern is we need to work out a design that make sure old cluster can be ugpraded | 09:58 |
| strigazi | flwang1: what is your situation? (regarding there is literally nothing we can do for clusters that reference k8s.gcr.io/hyperkube. && if your clusters use <my-registry>/hyperkube, we can build a hyperkube) | 09:59 |
| flwang1 | we're getting hyperkube from dockerhub/catalystcloud | 09:59 |
| strigazi | pro account i guess | 10:00 |
| flwang1 | now i'm trying to build hyperkube for v1.19.x | 10:00 |
| strigazi | the free account won't cut it any more | 10:00 |
| strigazi | it relatively easy. But let me rephrase | 10:00 |
| strigazi | Shall we move to the binary for V, so that we don't have to maintain a new image? | 10:01 |
| strigazi | brtknr: ^^ flwang1 ^^ | 10:01 |
| flwang1 | strigazi: moving to binary is our goal i think, and we don't have choice | 10:02 |
| strigazi | flwang1: easy to build, hard to maintain. | 10:02 |
| flwang1 | you mean maintain the binanry? | 10:03 |
| strigazi | flwang1: no, the image | 10:03 |
| flwang1 | i see | 10:03 |
| brtknr | I'm okay with that, I echo flwang's concern that existing clusters can also be upgraded, which should be possible. | 10:03 |
| flwang1 | yep, but again, as public cloud, and a GAed service, we can't break the upgrade path | 10:04 |
| flwang1 | though we should be able to do magic in the upgrade-k8s.sh | 10:04 |
| flwang1 | at least, the good thing is we don't have to replace the operating system | 10:05 |
| strigazi | The upstream project broke it. So for V we do the binary hoping they won't break it again. | 10:05 |
| flwang1 | that's a good excuse for us, but we can't use it for our public cloud customer unfortunately :( | 10:05 |
| flwang1 | strigazi: i will review your patch and see how can we resolve the upgrade issue | 10:06 |
| flwang1 | strigazi: brtknr: anything else? | 10:09 |
| strigazi | I'm good | 10:09 |
| brtknr | strigazi: any reason not to use binaries everything? | 10:10 |
| brtknr | there is also a server binaries tarball i assume is for master node | 10:11 |
| strigazi | brtknr: they are 300mb and I think it is more secure and elegant running in containers. | 10:11 |
| flwang1 | ok, let me end the meeting first | 10:12 |
| flwang1 | #endmeeting | 10:12 |
| *** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting" | 10:12 | |
| openstack | Meeting ended Wed Aug 26 10:12:27 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:12 |
| flwang1 | thank you guys | 10:12 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-08-26-09.01.html | 10:12 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-08-26-09.01.txt | 10:12 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-08-26-09.01.log.html | 10:12 |
| strigazi | brtknr: also, kube-proxy is easier to run in containers since they do some iptables magic in the image., | 10:14 |
| strigazi | brtknr: what are the arguments on using binaries? | 10:14 |
| brtknr | strigazi: i assumed it might involve less image download | 10:17 |
| brtknr | 300 mb on master, 100 mb on workers | 10:17 |
| flwang1 | strigazi: btw, is CERN enabling PodSecurityPolicy in your template？ | 10:19 |
| strigazi | brtknr: it won't, the server tar contains all bins and images. | 10:19 |
| brtknr | strigazi: yeah, 300md is less than what is currently downloaded isnt it? | 10:20 |
| strigazi | brtknr: no, hyperkube image is less than 300mb compressed. | 10:21 |
| brtknr | strigazi: i see | 10:23 |
| brtknr | it would be good to avoid downloading kube-proxy twice | 10:26 |
| strigazi | I couldn't make kube-proxy run outside the image. As a matter of fact, kube-proxy should be a ds. | 10:27 |
| strigazi | brtknr: because of the iptables magic I mentioned | 10:28 |
| brtknr | ds=daemonset? | 10:29 |
| brtknr | i was wondering if we should run all the kube-services as static pods? | 10:29 |
| strigazi | it is a good-ish move | 10:30 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: Add CT observations field to the database and API https://review.opendev.org/737840 | 11:48 |
| *** dave-mccowan has joined #openstack-containers | 13:17 | |
| *** sapd1_x has joined #openstack-containers | 13:22 | |
| *** kevko has quit IRC | 13:54 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: ci: Log in to DockerHub using docker_login https://review.opendev.org/747867 | 15:05 |
| *** sapd1_x has quit IRC | 15:25 | |
| *** ykarel is now known as ykarel|away | 15:55 | |
| *** ykarel|away has quit IRC | 16:00 | |
| *** nikparasyr has left #openstack-containers | 16:30 | |
| *** k_mouza has quit IRC | 16:41 | |
| *** k_mouza has joined #openstack-containers | 16:44 | |
| *** k_mouza has quit IRC | 16:48 | |
| *** sapd1_x has joined #openstack-containers | 17:04 | |
| *** k_mouza has joined #openstack-containers | 17:22 | |
| *** k_mouza has quit IRC | 17:26 | |
| *** sapd1_x has quit IRC | 17:47 | |
| openstackgerrit | Vishal Manchanda proposed openstack/magnum-ui master: [DNM] Test magnum-ui with horizon jobs on focal https://review.opendev.org/747169 | 17:51 |
| *** k_mouza has joined #openstack-containers | 17:56 | |
| *** k_mouza has quit IRC | 18:01 | |
| openstackgerrit | Vishal Manchanda proposed openstack/magnum-ui master: [DNM] Test magnum-ui with horizon jobs on focal https://review.opendev.org/747169 | 18:15 |
| *** k_mouza has joined #openstack-containers | 19:57 | |
| *** k_mouza has quit IRC | 20:01 | |
| *** vishalmanchanda has quit IRC | 20:37 | |
| *** k_mouza has joined #openstack-containers | 21:33 | |
| *** k_mouza has quit IRC | 22:06 | |
| *** rcernin has joined #openstack-containers | 22:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!