| openstackgerrit | Merged openstack/python-magnumclient master: Add py38 package metadata https://review.opendev.org/723771 | 00:36 |
|---|---|---|
| *** k_mouza has joined #openstack-containers | 03:57 | |
| *** dasp_ has joined #openstack-containers | 03:58 | |
| *** dasp has quit IRC | 04:00 | |
| *** k_mouza has quit IRC | 04:01 | |
| *** ondrejburian has quit IRC | 04:15 | |
| *** ykarel|away is now known as ykarel | 04:15 | |
| *** ondrejburian has joined #openstack-containers | 04:17 | |
| *** vishalmanchanda has joined #openstack-containers | 05:07 | |
| *** jmlowe has quit IRC | 05:15 | |
| *** jmlowe has joined #openstack-containers | 05:26 | |
| *** sapd1_x has joined #openstack-containers | 05:30 | |
| *** udesale has joined #openstack-containers | 05:41 | |
| *** pcaruana has joined #openstack-containers | 06:20 | |
| *** sapd1_x has quit IRC | 06:22 | |
| *** xinliang has joined #openstack-containers | 06:33 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: [k8s] Support CA certs rotate https://review.opendev.org/724203 | 06:51 |
| *** belmoreira has joined #openstack-containers | 06:58 | |
| *** born2bake has joined #openstack-containers | 07:07 | |
| *** PrinzElvis has joined #openstack-containers | 07:13 | |
| born2bake | brtknr hello | 07:15 |
| *** xinliang has quit IRC | 07:19 | |
| *** born2bake has quit IRC | 07:36 | |
| *** born2bake has joined #openstack-containers | 07:40 | |
| *** xinliang has joined #openstack-containers | 07:45 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/ussuri: Update .gitreview for stable/ussuri https://review.opendev.org/722526 | 08:02 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/ussuri: Update TOX_CONSTRAINTS_FILE for stable/ussuri https://review.opendev.org/722527 | 08:03 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Update master for stable/ussuri https://review.opendev.org/722528 | 08:03 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Add Python3 victoria unit tests https://review.opendev.org/722529 | 08:04 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Deprecation note for devicemapper and overlay https://review.opendev.org/722163 | 08:05 |
| *** PrinzElvis has quit IRC | 08:07 | |
| *** PrinzElvis has joined #openstack-containers | 08:07 | |
| *** PrinzElvis has quit IRC | 08:11 | |
| *** PrinzElvis has joined #openstack-containers | 08:12 | |
| brtknr | born2bake: hi | 08:17 |
| born2bake | yeah, sorry I left irc chat could not see if you responded or not. | 08:17 |
| born2bake | I am wondering, https://github.com/stackhpc/magnum-terraform/blob/master/vars.tf - if something better to be changed in vars so I can create cluster | 08:18 |
| born2bake | Currently, its failing due to heat_container_agent_tag = "ussuri-dev" | 08:18 |
| born2bake | it cant find that image | 08:18 |
| brtknr | born2bake: still there from what i can tell https://hub.docker.com/r/openstackmagnum/heat-container-agent/tags | 08:19 |
| *** xinliang has quit IRC | 08:19 | |
| brtknr | born2bake: can your VM reach the internet? | 08:20 |
| born2bake | hm might be the case yeah... | 08:20 |
| born2bake | Also, I tested my octavia setup with kubespray and its working perfectly well. Thus, will try to troubleshoot why its not working with masters created by magnum. | 08:27 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: [WIP] List all nodes of a cluster https://review.opendev.org/724609 | 08:29 |
| openstackgerrit | Merged openstack/magnum master: Update master for stable/ussuri https://review.opendev.org/722528 | 08:29 |
| *** ykarel is now known as ykarel|lunch | 08:39 | |
| *** yolanda has joined #openstack-containers | 08:41 | |
| *** k_mouza has joined #openstack-containers | 09:08 | |
| *** k_mouza has quit IRC | 09:20 | |
| *** k_mouza has joined #openstack-containers | 09:31 | |
| openstackgerrit | Merged openstack/magnum stable/ussuri: Update .gitreview for stable/ussuri https://review.opendev.org/722526 | 09:35 |
| *** brtknr has quit IRC | 09:37 | |
| openstackgerrit | Merged openstack/magnum stable/ussuri: Update TOX_CONSTRAINTS_FILE for stable/ussuri https://review.opendev.org/722527 | 09:39 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: [k8s] Fix docker storage of Fedora CoreOS https://review.opendev.org/718296 | 09:40 |
| *** brtknr has joined #openstack-containers | 09:42 | |
| *** flwang1 has quit IRC | 09:48 | |
| openstackgerrit | Merged openstack/magnum master: Add Python3 victoria unit tests https://review.opendev.org/722529 | 09:49 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum stable/train: Use cluster name for fixed_network instead of private https://review.opendev.org/722280 | 10:06 |
| *** xinliang has joined #openstack-containers | 10:20 | |
| born2bake | brtknr when magnum automatically creates fixed networks, can I somehow specify what subnet I want to have in them? | 10:21 |
| *** ykarel|lunch is now known as ykarel | 10:25 | |
| brtknr | born2bake: umm | 10:41 |
| brtknr | as in specify the cidr? | 10:41 |
| born2bake | yes | 10:41 |
| born2bake | cause by default its 10.0.0.0/24 | 10:42 |
| brtknr | yes https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml#L173 | 10:43 |
| *** k_mouza has quit IRC | 10:43 | |
| brtknr | sadly this is not document, would love to have this in there if you are happy to contribute: https://docs.openstack.org/magnum/latest/user/#flannel-network-cidr | 10:46 |
| *** k_mouza has joined #openstack-containers | 10:55 | |
| *** rcernin has quit IRC | 11:06 | |
| *** xinliang has quit IRC | 11:07 | |
| *** udesale_ has joined #openstack-containers | 11:12 | |
| *** udesale has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack/magnum stable/train: Use cluster name for fixed_network instead of private https://review.opendev.org/722280 | 12:03 |
| *** k_mouza has quit IRC | 12:03 | |
| born2bake | for some reason i have keypair in my cluster template, in my cluster, and still isntances are created without key O_O | 12:03 |
| *** rcernin has joined #openstack-containers | 12:31 | |
| *** ramishra has quit IRC | 12:41 | |
| *** k_mouza has joined #openstack-containers | 12:49 | |
| openstackgerrit | Merged openstack/python-magnumclient master: Fix raw_request of SessionClient https://review.opendev.org/724243 | 13:03 |
| *** ykarel is now known as ykarel|afk | 13:14 | |
| *** rcernin has quit IRC | 13:29 | |
| *** ramishra has joined #openstack-containers | 13:36 | |
| *** ramishra has quit IRC | 13:44 | |
| *** ramishra has joined #openstack-containers | 13:57 | |
| *** ykarel|afk is now known as ykarel | 14:09 | |
| *** colin- has joined #openstack-containers | 14:26 | |
| *** ramishra has quit IRC | 14:31 | |
| born2bake | brtknr how could I fix Error from server (Forbidden): nodes is forbidden: User "Magnum User" cannot list resource "nodes" in API group "" at the cluster scope ? | 14:34 |
| brtknr | i think you need to configure k8s_keystone_auth_policy.json file | 14:35 |
| *** ricolin has quit IRC | 14:37 | |
| born2bake | in keystone? | 14:42 |
| brtknr | no /etc/magnum/ | 14:56 |
| brtknr | did you use --use-keystone flag? | 14:56 |
| brtknr | I dont know where you are seeing this error | 14:56 |
| *** ricolin has joined #openstack-containers | 15:01 | |
| *** k_mouza has quit IRC | 15:04 | |
| born2bake | using terraform scripts of yours | 15:05 |
| born2bake | it export config files | 15:05 |
| born2bake | in the end | 15:05 |
| born2bake | brtknr http://paste.openstack.org/show/792953/ | 15:08 |
| brtknr | born2bake: ok | 15:09 |
| brtknr | and thats not working? | 15:09 |
| *** k_mouza has joined #openstack-containers | 15:10 | |
| born2bake | then i go to cd ~/.kube/k8s-calico-coreos | 15:10 |
| born2bake | Error from server (Forbidden): nodes is forbidden: User "Magnum User" cannot list resource "nodes" in API group "" at the cluster scope | 15:10 |
| born2bake | where can I specify that flag ? --use-keystone flag | 15:12 |
| brtknr | born2bake: i really dont understand your issue? | 15:13 |
| brtknr | why do you cd ~/.kube/k8s-calico-coreos? | 15:13 |
| brtknr | did you do export KUBECONFIG=~/.kube/k8s-calico-coreos/config? | 15:13 |
| born2bake | KUBECONFIG=~/.kube/k8s-calico-coreos/config | 15:16 |
| born2bake | bn@WINDOWS-MMBRA93:~/.kube/k8s-calico-coreos$ kubectl get node | 15:16 |
| born2bake | Error from server (Forbidden): nodes is forbidden: User "Magnum User" cannot list resource "nodes" in API group "" at the cluster scope | 15:16 |
| brtknr | born2bake: is this devstack? | 15:16 |
| born2bake | kolla | 15:17 |
| *** ricolin has quit IRC | 15:17 | |
| brtknr | what do you have in your /etc/kolla/magnum/ | 15:17 |
| born2bake | you mean conf file? | 15:30 |
| born2bake | brtknr in magnum I have keystone_auth_default_policy.sample file | 15:36 |
| born2bake | I assume for kolla it needs to be reconfigured yeah? | 15:36 |
| *** sapd1_x has joined #openstack-containers | 15:37 | |
| brtknr | yes i think so | 15:38 |
| born2bake | also wondering, if autoscale is working fine for you? cause I am getting E0430 15:39:10.561649 1 leaderelection.go:320] error retrieving resource lock kube-system/cluster-autoscaler: leases.coordination.k8s.io "cluster-autoscaler" is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system" | 15:39 |
| born2bake | deploy is up and running but not functioning | 15:39 |
| *** ioni has quit IRC | 15:43 | |
| *** sapd1_x has quit IRC | 15:44 | |
| *** belmoreira has quit IRC | 15:49 | |
| born2bake | with 1.18.1 auto-scaler, rbac needs to be updated. | 15:54 |
| brtknr | born2bake: are you trying to use magnum as a non-admin user? | 15:54 |
| born2bake | nope, i am authorized...using openstack cli without any issues | 15:55 |
| born2bake | I think cosmicsound had the same problem... he is using k-a as well | 15:55 |
| born2bake | we cant use kubectl externally...only on master node | 15:56 |
| brtknr | born2bake: this is basically what you need to do: https://github.com/openstack/magnum/blob/master/devstack/lib/magnum#L245 | 16:06 |
| born2bake | but its in devstack only right? | 16:06 |
| brtknr | born2bake: well, yes but also generally applicable. | 16:06 |
| brtknr | its quite a permissive setting, you can make it more restrictive if you like | 16:07 |
| brtknr | gives the user who created the cluster admin rights on the cluster | 16:07 |
| born2bake | I am not really familiar with repo yet :) not sure where to add it if not in devstack | 16:07 |
| brtknr | /etc/kolla/magnum/ | 16:13 |
| brtknr | to /etc/kolla/magnum/ | 16:14 |
| born2bake | which file it should be? there is no magnum file like you have in devstack | 16:17 |
| born2bake | lxkong https://github.com/openstack/magnum/blob/df3d5a31502304a61ec53ac7dc65ee3ff3d45001/magnum/drivers/common/templates/kubernetes/fragments/enable-auto-scaling.sh I think needs to be updated if using 1.18 version. Its failing due to https://github.com/kubernetes/autoscaler/issues/2628 ; Once I ve added coordination.k8s.io - "leases.coordination.k8s.io "cluster-autoscaler" is forbidden: User "system:serviceaccount:kube-system:cluster-a | 16:26 |
| born2bake | utoscaler-account" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system" error was gone. However, I am still getting: E0430 16:25:53.824519 1 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler-account" cannot list resource "csinodes" in API group | 16:26 |
| born2bake | "storage.k8s.io" at the cluster scope | 16:26 |
| *** ykarel is now known as ykarel|afk | 16:31 | |
| brtknr | please propose a patch if you think that will fix the issue :) | 16:33 |
| brtknr | born2bake: i can also see the issue | 16:33 |
| brtknr | its an easy first patch :) | 16:33 |
| brtknr | born2bake: ^ | 16:34 |
| *** ykarel|afk is now known as ykarel|away | 16:36 | |
| *** ioni has joined #openstack-containers | 16:36 | |
| *** udesale_ has quit IRC | 16:39 | |
| born2bake | I wd add it but it still didnt fix the whole issue | 16:50 |
| born2bake | just tested also auto-healer...and got error http://paste.openstack.org/show/792962/ | 16:51 |
| born2bake | so node has been deleted... but new node didnt come up. while autoscaler sends - W0430 16:50:11.579712 1 scale_up.go:383] Node group default-worker is not ready for scaleup - backoff | 16:52 |
| brtknr | you should look at why the new node cannot be added | 17:08 |
| brtknr | do you have enough capacity? | 17:08 |
| *** vishalmanchanda has quit IRC | 17:29 | |
| *** k_mouza has quit IRC | 18:12 | |
| *** jmlowe has quit IRC | 18:24 | |
| *** jmlowe has joined #openstack-containers | 18:27 | |
| born2bake | brtknr is there any way I can delay autoscaler yaml file deployment? | 18:55 |
| born2bake | I noticed that autoscaler is failing if its applied instantly when cluster is created | 18:55 |
| brtknr | born2bake: do you have nested virt enabled? | 19:01 |
| born2bake | I assume, if my env was deployed via kolla-ansible, it should have added it... | 19:06 |
| born2bake | cat /sys/module/kvm_intel/parameters/nested | 19:08 |
| born2bake | Y | 19:08 |
| *** markguz_ has joined #openstack-containers | 19:43 | |
| markguz_ | Hi container folks. Quick question. can i run the latest version of magnum (10rc) with a downlevel keystone (rocky) ? | 19:44 |
| *** k_mouza has joined #openstack-containers | 19:57 | |
| *** k_mouza has quit IRC | 20:02 | |
| *** k_mouza has joined #openstack-containers | 20:13 | |
| born2bake | brtknr checked twice... autoscaler is failing for me if its instantly applied once cluster is created but not all worker nodes are ready yet | 20:15 |
| born2bake | so need to put some delay on it i guess somehow :) | 20:15 |
| born2bake | so manifest will be applied after some time | 20:16 |
| *** k_mouza has quit IRC | 20:17 | |
| *** mgariepy has quit IRC | 20:22 | |
| *** mgariepy has joined #openstack-containers | 20:32 | |
| brtknr | born2bake: ok i will test again tomorrow but i have never seen this issue before | 20:50 |
| brtknr | please file a bug on storyboard describing how to reproduce this | 20:51 |
| *** Jeffrey4l has quit IRC | 20:53 | |
| *** johanssone has quit IRC | 20:53 | |
| *** openstackgerrit has quit IRC | 20:53 | |
| *** Jeffrey4l has joined #openstack-containers | 20:53 | |
| *** johanssone has joined #openstack-containers | 20:54 | |
| born2bake | brtknr actually, no I just started autoscaler after cluster was up-to-date setup....and it still failing with http://paste.openstack.org/show/792970/ | 21:12 |
| born2bake | so the only way I can make it working....I start cluster with autoscaler enabled but with lease and csi nodes apigroups errors....once i fix them, then scaling is working properly | 21:13 |
| *** rcernin has joined #openstack-containers | 22:04 | |
| *** openstackgerrit has joined #openstack-containers | 22:27 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Deprecation note for devicemapper and overlay https://review.opendev.org/722163 | 22:27 |
| *** rcernin has quit IRC | 22:33 | |
| *** rcernin has joined #openstack-containers | 22:34 | |
| born2bake | pretty sure its something related to cloud provider...not autoscaler | 22:48 |
| born2bake | sometimes when I deploy autoscaler....cloud provider fails with errors...crash and then autoscaler crash afterwards | 22:48 |
| *** born2bake has quit IRC | 23:26 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!