| *** ivve has quit IRC | 00:33 | |
| *** trungnv has joined #openstack-containers | 01:42 | |
| *** trungnv has quit IRC | 01:49 | |
| *** goldyfruit_ has quit IRC | 01:53 | |
| *** ramishra has joined #openstack-containers | 02:39 | |
| *** ykarel has joined #openstack-containers | 03:15 | |
| *** udesale has joined #openstack-containers | 04:02 | |
| *** yolanda has quit IRC | 06:45 | |
| *** ykarel is now known as ykarel|afk | 06:46 | |
| *** ykarel|afk is now known as ykarel|lunch | 06:55 | |
| *** rcernin has quit IRC | 07:04 | |
| *** ivve has joined #openstack-containers | 08:46 | |
| *** ykarel|lunch is now known as ykarel | 09:06 | |
| *** yolanda has joined #openstack-containers | 09:22 | |
| *** pcaruana has joined #openstack-containers | 10:10 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Fix cert_manager_api with x509keypair https://review.opendev.org/694739 | 10:14 |
|---|---|---|
| *** ykarel_ has joined #openstack-containers | 10:28 | |
| *** ykarel has quit IRC | 10:30 | |
| *** rcernin has joined #openstack-containers | 10:53 | |
| *** ykarel_ is now known as ykarel | 10:56 | |
| dioguerra | Can someone add some love on this MR? https://review.opendev.org/#/c/691646/ | 11:01 |
| *** ramishra has quit IRC | 11:14 | |
| *** udesale has quit IRC | 11:17 | |
| *** ykarel is now known as ykarel|afk | 12:14 | |
| dioguerra | brtknr: flwang1: ^^ | 12:23 |
| brtknr | dioguerra: Looks fine :) what does the adapter do? | 12:25 |
| *** ramishra has joined #openstack-containers | 12:26 | |
| dioguerra | brtknr: the adapter makes use of the custom.metrics.k8s.io api to expose user custom metrics. | 12:30 |
| *** rcernin has quit IRC | 12:31 | |
| dioguerra | By leveraging Prometheus, the user can use or add metrics based on custom definitions to scale pods using the kubernetes HPA | 12:31 |
| dioguerra | This way i can use (number of http connections per pod) for example, to scale pods accordingly | 12:31 |
| brtknr | I'm wondering if these should be enabled by default if monitoring_enabled is true | 12:33 |
| brtknr | dioguerra: ^ | 12:34 |
| *** namrata has joined #openstack-containers | 12:34 | |
| dioguerra | i also though on setting it to false... mainly because we 'kinda' decided that components would be opt-in. | 12:35 |
| dioguerra | I can change it since the tests broke in python27, don know why | 12:35 |
| *** lpetrut has joined #openstack-containers | 12:42 | |
| namrata | Hi folks,we have deployed a kubernetes cluster with magnum on Rocky but we see kube-system kube-dns-autoscaler-57bd7f54d5-4cg9r 0/1 ImagePullBackOff 0 4d20h Warning Failed 15s (x2 over 35s) kubelet, cluster-test-xdu3aeoxp2ov-minion-0 Failed to pull image | 12:42 |
| namrata | "docker.io/googlecontainer/cluster-proportional-autoscaler-amd64:1.7.1": rpc error: code = Unknown desc = repository docker.io/googlecontainer/cluster-proportional-autoscaler-amd64 not found: does not exist or no pull access Warning Failed 15s (x2 over 35s) kubeletIn our opinion the issue is the image location. and the related | 12:42 |
| namrata | patches https://review.opendev.org/#/c/652057/https://review.opendev.org/#/c/690053/ which are not for rocky | 12:42 |
| namrata | can anybody help me with the workaround for rocky | 12:42 |
| namrata | Hi folks,we have deployed a kubernetes cluster with magnum on Rocky but we see http://paste.openstack.org/show/786266/In our opinion the issue is the image location. and the related patches https://review.opendev.org/#/c/652057/ https://review.opendev.org/#/c/690053/ which are not for rocky | 12:44 |
| namrata | can anybody help me with the workaround for rocky | 12:44 |
| dioguerra | brtknr^^ change to false then? | 12:47 |
| brtknr | i prefer true | 12:48 |
| *** namrata has quit IRC | 12:49 | |
| dioguerra | namrata: edit your deployment image with the gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.7.1 and check if it works. | 12:52 |
| dioguerra | If not you have a problem elsewhere. (maybe flannel). do not use podman. | 12:53 |
| dioguerra | If you try and it works, patch locally or submit a mr to fix for rocky | 12:53 |
| *** namrata has joined #openstack-containers | 13:09 | |
| *** sapd1 has quit IRC | 13:16 | |
| *** KeithMnemonic has joined #openstack-containers | 13:40 | |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/magnum master: nodegroup list with --limit gives wrong next URL https://review.opendev.org/694230 | 13:50 |
| *** ramishra has quit IRC | 13:59 | |
| *** ramishra has joined #openstack-containers | 14:13 | |
| *** namrata has quit IRC | 14:19 | |
| brtknr | dioguerra: do you know if masters can be resized? | 14:23 |
| brtknr | strigazi: ^ | 14:32 |
| *** goldyfruit has joined #openstack-containers | 14:40 | |
| *** goldyfruit_ has joined #openstack-containers | 14:51 | |
| *** goldyfruit has quit IRC | 14:53 | |
| *** udesale has joined #openstack-containers | 15:22 | |
| *** udesale has quit IRC | 16:06 | |
| *** jmlowe has joined #openstack-containers | 16:07 | |
| *** ykarel|afk is now known as ykarel | 16:08 | |
| *** namrata has joined #openstack-containers | 16:16 | |
| namrata | dioguerra now I dont see ImagePullBackOff issue now but i have all the pods in CrashLoopBackOff http://paste.openstack.org/show/786287/ | 16:33 |
| namrata | can anybody help me with this I am running on openstack rocky | 16:34 |
| *** namrata has quit IRC | 16:48 | |
| *** jmlowe has quit IRC | 17:04 | |
| brtknr | namrata | 17:08 |
| *** namrata has joined #openstack-containers | 17:35 | |
| namrata | brtknr hi | 17:36 |
| brtknr | use stein | 17:37 |
| brtknr | namrata: | 17:37 |
| namrata | any workaround for rocky our regions are on release rocky | 17:38 |
| namrata | or we cant get it work on rocky | 17:38 |
| namrata | is this known in rocky any changes which can be backported for rocky | 17:39 |
| namrata | brtknr | 17:39 |
| brtknr | not sure, havent used rocky in a while | 17:39 |
| brtknr | magnum stein will work with other openstack services running on rocky | 17:40 |
| brtknr | what version of fedora atomic are you using? | 17:40 |
| brtknr | you need stein to use fa29 | 17:40 |
| brtknr | and fa27 is EOL, its no longer maintained | 17:41 |
| namrata | currently using fedora atomic 27 | 17:41 |
| brtknr | hmm then your setup should work, so not sure | 17:41 |
| brtknr | what output do you get when you describe the failing pods? | 17:42 |
| namrata | I see http://paste.openstack.org/show/786288/ | 17:43 |
| namrata | brtknr it was earlier working in rocky in our regions started seeing the issue from last 3-4 days | 17:47 |
| brtknr | I though you said your dns autoscaleer was running? | 17:48 |
| brtknr | which version of k8s? | 17:49 |
| namrata | no its not running it was only able to pull of the image successfully then it says same error like other pods Back-off restarting failed container | 17:51 |
| namrata | brtknr I have tried all the kube_tag versions v1.14.8, v1.15.4 v1.15.5 and v1.16.3 and http://paste.openstack.org/show/786290/ | 17:53 |
| namrata | sorry v1.16.2 | 17:58 |
| *** ykarel is now known as ykarel|away | 18:15 | |
| brtknr | Use v1.14.6 | 18:19 |
| brtknr | 1.14.6+ is not compatible with rocky | 18:19 |
| *** ykarel|away has quit IRC | 18:20 | |
| brtknr | namrata | 18:32 |
| namrata | brtknr I am trying with lower versions but then it will be exposed to billion laughs vulerability | 18:33 |
| namrata | so cant use in production I guess | 18:33 |
| *** gouthamr_ is now known as gouthamr | 18:51 | |
| *** namrata has quit IRC | 19:13 | |
| *** namrata has joined #openstack-containers | 19:18 | |
| namrata | brtknr I tried versions below v1.14.6 and it runs sucessfully but after billion laughs vulnerability I ran v1.15.5 and v1.14.8 and they were running fine but what happened in the interim that now they are not supported | 19:33 |
| brtknr | namrata: i dont know how you were running 1.14.8, none of us have been able to | 19:34 |
| brtknr | you need train to support 1.14.8 | 19:34 |
| namrata | I am sure as I was running my gitlab CI/CD pipeline in kubernetes cluster with versions v1.14.8 and v1.15.5 but it stopped working some days before | 19:36 |
| namrata | brtknr so what do you say about if rocky dosent support v1.14.6+ then its not ideal for production | 19:38 |
| brtknr | if you care about CVEs, no, its not idea for prod | 19:40 |
| brtknr | but again, you can run magnum train in prod | 19:40 |
| brtknr | independent of other openstack services... | 19:40 |
| brtknr | the only issue is if you want to run fedora coreos, you need heat from train too | 19:41 |
| namrata | brtknr thanks for your help and time. | 19:47 |
| *** jmlowe has joined #openstack-containers | 20:16 | |
| *** namrata has quit IRC | 20:22 | |
| *** dim79 has joined #openstack-containers | 20:56 | |
| *** mgariepy has quit IRC | 20:57 | |
| dim79 | hi all | 20:58 |
| dim79 | I'm trying to get CSI and found, that 1.14 is not compatible with latest magnum etcd (at least k8s matrix says this) | 20:58 |
| dim79 | any plans to update the etd image? | 20:59 |
| dim79 | *etcd | 20:59 |
| dim79 | or any guide how to build it - I'll try to get 3.3.10 than | 21:00 |
| *** goldyfruit_ has quit IRC | 21:02 | |
| flwang | dim79: where's the magnum version you're using? | 21:09 |
| dim79 | stein | 21:10 |
| flwang | it's tricky | 21:10 |
| flwang | for stein, you have to use the etcd atomic system container | 21:10 |
| flwang | but so far, i think we only have 3.2.7 | 21:11 |
| dim79 | yep | 21:11 |
| dim79 | any guide how to prepare a new one? | 21:11 |
| flwang | wait a sec, let me check | 21:11 |
| flwang | dim79: try this one | 21:15 |
| flwang | https://github.com/projectatomic/atomic-system-containers/tree/master/etcd | 21:15 |
| *** goldyfruit has joined #openstack-containers | 21:17 | |
| dim79 | flwang - thank you very much, it looks like the same guide. do you know what version of etcd is used in fedora rawhide now? | 21:18 |
| dim79 | and - if I would be on 'master' branch of magnum - what do you recommend to use - podman? what etcd image is used there? | 21:19 |
| flwang | with podman, at least the 3.2.26 works, and i think you can use any etcd image from gcr, i didn't try higher version | 21:21 |
| flwang | dim79: is it a prod env? | 21:21 |
| dim79 | not yet | 21:21 |
| flwang | ok, i know it's tricky, but please blame k8s :) it's releasing so fast | 21:21 |
| flwang | we can't make v1.15.4 and above works for fedora atomic with the atomic system container | 21:22 |
| flwang | that's why we switch to podman | 21:22 |
| flwang | and another reason is the fedora coreos driver can be benefited from the new podman support | 21:22 |
| flwang | since there is no atomic system container on fedora coreos | 21:23 |
| flwang | that's the background, just FYI | 21:23 |
| flwang | do you mean CSI need at least etcd 3.3.10? | 21:23 |
| flwang | my team is going to work on CSI soon, so i'm happy to support you to get your work done | 21:24 |
| dim79 | I've build CSI out of cloud-provider-openstack, fixed manifests in a way they are defined in magnum - and stucked with attach/detach issue | 21:25 |
| dim79 | so, volume is created but is not attached automatically | 21:25 |
| dim79 | and that, investigating further, I've seen that I have k8s 1.13 running from my k8s v1.14.x images :( | 21:26 |
| dim79 | this means some of CSI-related features are not enabled or working differently | 21:27 |
| flwang | dim79: for stein, i think you should be able to run v1.15.2 | 21:27 |
| flwang | i tested and it works | 21:27 |
| dim79 | good to know | 21:27 |
| flwang | i can't see a strong depedency between etcd and csi | 21:27 |
| flwang | so i would suggest using v1.15.2 and test again | 21:28 |
| dim79 | ok, than my deduction today is wrong - any other reason why kubectl reports 1.13, whilst k8s images are v1.14 ? | 21:29 |
| dim79 | > v1.15.2 | 21:29 |
| dim79 | I'll try | 21:29 |
| dim79 | I've deducted that kubectl reports 1.13 due to k8s compatibility matrix - where etcd 3.3.10 is required for k8s 1.14 | 21:31 |
| dim79 | from k8s sources, cmd/kubeadm/app/constants/constants.go: | 21:32 |
| dim79 | // SupportedEtcdVersion lists officially supported etcd versions with corresponding Kubernetes releases | 21:32 |
| dim79 | SupportedEtcdVersion = map[uint8]string{ | 21:32 |
| dim79 | 12: "3.2.24", | 21:32 |
| dim79 | 13: "3.2.24", | 21:32 |
| dim79 | 14: "3.3.10", | 21:32 |
| dim79 | } | 21:32 |
| dim79 | likely I'm mistaken, but I have no other idea | 21:34 |
| flwang | i don't really understand your current issue, sorry. what do you mean "kubectl reports 1.13 due to k8s compatibility matrix - where etcd 3.3.10 is required for k8s 1.14"? | 21:36 |
| flwang | you have to use correct kubectl version with your k8s api version | 21:37 |
| dim79 | flwang - the fact that nobody else except me have/had this issue - votes that the reason is in my images build procedure | 21:38 |
| dim79 | so, now I have aanother idea, thank you :) | 21:38 |
| brtknr | flwang: the coredns patch also requires fa29 | 21:39 |
| flwang | dim79: np | 21:40 |
| flwang | brtknr: sorry, what do you mean? | 21:40 |
| brtknr | this one: https://review.opendev.org/#/c/692834/5 | 21:40 |
| brtknr | btw hello :) | 21:40 |
| flwang | https://review.opendev.org/#/c/693140/3 ? | 21:40 |
| brtknr | yyep | 21:41 |
| brtknr | thanks! | 21:42 |
| brtknr | earlier you said stein supports 1.15.x | 21:42 |
| brtknr | to dim79 | 21:42 |
| brtknr | but for 1.15.x suppport, stein needs this backport https://review.opendev.org/#/c/694032/3 | 21:42 |
| brtknr | im happy to leave the other OCCM and multi-NIC patches off stein 8.2.0 | 21:43 |
| brtknr | im happy to drop the other OCCM and multi-NIC patches off stein 8.2.0 as only stackhpc requires it, and not an urgent issue for most users | 21:44 |
| brtknr | but we may need to consider supporting podman in stein | 21:45 |
| brtknr | in order to allow users to remain up to date | 21:45 |
| flwang | yep, you're correct, we need https://review.opendev.org/#/c/694032/ to support at least v1.15.3 for stein | 21:47 |
| flwang | i just +2 ed | 21:47 |
| brtknr | thanks | 21:48 |
| brtknr | flwang: iirc master cannot be resized right? | 21:53 |
| flwang | brtknr: no, master can't be resized at this moment | 21:54 |
| *** dim79 has quit IRC | 22:05 | |
| *** pcaruana has quit IRC | 22:26 | |
| *** rcernin has joined #openstack-containers | 22:46 | |
| *** ivve has quit IRC | 23:26 | |
| *** goldyfruit has quit IRC | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!