| *** flwang has quit IRC | 00:12 | |
| *** ttsiouts has joined #openstack-containers | 01:32 | |
| *** ttsiouts has quit IRC | 02:05 | |
| *** itlinux has joined #openstack-containers | 02:52 | |
| *** hongbin has joined #openstack-containers | 03:02 | |
| *** anyrude10 has joined #openstack-containers | 03:39 | |
| anyrude10 | Hi Team, I am installing k8s cluster on Magnum Rocky with Fedora 25 atomic image, my stack remains Create in progress and eventually fails. On debugging I found in the master node file /var/log/cloud-init-output.log the following error : error: unable to decode "STDIN": no kind "ClusterRole" is registered for version "rbac.authorization.k8s.io/v1beta1". Can you please help me in resolving the issue | 03:41 |
|---|---|---|
| *** udesale has joined #openstack-containers | 03:42 | |
| *** ramishra has joined #openstack-containers | 03:43 | |
| ykarel | anyrude10, any reason to use Fedora25 atomic? u should use fedora 27 atomic which is used in magnum CI, that should be good to use | 03:46 |
| ykarel | https://github.com/openstack/magnum/blob/stable/rocky/magnum/tests/contrib/gate_hook.sh#L88 | 03:46 |
| ykarel | https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 | 03:47 |
| anyrude10 | yes, previously I was using Fedora 27 only, but it failed in curl http://127.0.0.1:8080/healthz, so i just thought to degrade the image. Any pointers that could help | 03:50 |
| anyrude10 | ykarel, I have manual openstack rocky setup in which I am install Magnum service and creating k8s cluster, but the kube_master remains create in progress and after 1 hr, stack fails. Can you please help a little further so that I ca resolve the issue | 03:53 |
| *** itlinux has quit IRC | 03:54 | |
| ykarel | anyrude10, let me find a link, steps there will help u in narrow down the issue | 03:56 |
| anyrude10 | ykarel, sure that would be a great help | 03:56 |
| ykarel | anyrude10, follow comments for the bug: https://bugs.launchpad.net/magnum/+bug/1655007 | 03:57 |
| openstack | Launchpad bug 1655007 in Magnum "Magnum cluster takes forever to create with status 'kube_masters create in progress'" [Undecided,New] | 03:57 |
| ykarel | anyrude10, main reason is kube master is not able to notify to heat about it completion | 03:57 |
| ykarel | and ^^ can happen due to multiple reason | 03:58 |
| ykarel | you need to find that reason and comments in bug will help you | 03:58 |
| anyrude10 | i'll go through it and get back to you | 03:58 |
| ykarel | ok | 03:59 |
| ykarel | anyrude10, first of all check cloud-init-output log there u will get some hint | 04:01 |
| ykarel | and use f27 image | 04:01 |
| anyrude10 | in Fedora 25, the error in cloud-init-output.log is Failed to start heat-container-agent.service: Unit heat-container-agent.service not found. I'll try using Fedora 27 | 04:02 |
| *** ttsiouts has joined #openstack-containers | 04:03 | |
| anyrude10 | ykarel, need 1 input, how can I resolve my controller hostname in kube_master node while creating the cluster? What I do is I login into the node and manually update the /etc/hosts file | 04:06 |
| ykarel | anyrude10, so your endpoints contains hostname instead of ip? | 04:16 |
| *** hongbin has quit IRC | 04:18 | |
| anyrude10 | ykarel, yes all the endpoints contains hostname | 04:18 |
| ykarel | anyrude10, okk i am not sure about best method, but for now you can update magnum scripts to do that automatically | 04:20 |
| ykarel | you can do that in ca_script which runs first: https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml#L618 | 04:21 |
| ykarel | mainly https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/fragments/atomic-install-openstack-ca.sh | 04:22 |
| anyrude10 | ykarel, i'll give it a try | 04:26 |
| ykarel | okk | 04:28 |
| *** ykarel is now known as ykarel|afk | 04:28 | |
| *** ykarel|afk has quit IRC | 04:32 | |
| *** spsurya has joined #openstack-containers | 04:32 | |
| *** ttsiouts has quit IRC | 04:36 | |
| *** ramishra has quit IRC | 04:43 | |
| *** ttsiouts has joined #openstack-containers | 04:46 | |
| *** ramishra has joined #openstack-containers | 04:47 | |
| *** ykarel|afk has joined #openstack-containers | 04:48 | |
| *** ykarel|afk is now known as ykarel | 04:49 | |
| anyrude10 | ykarel, I have used Fedora 27, and facing the error curl http://127.0.0.1:8080/healthz, curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused | 05:08 |
| *** pcaruana has joined #openstack-containers | 05:08 | |
| ykarel | anyrude10, you checked cloud-init-output log as described in bug comments? | 05:14 |
| anyrude10 | ykarel, yes I have checked. The issue is on controller, endpoints are created on Management IP 10.0.39.11 and external network ip is 10.0.61.5 , and my kube-master has a floating IP on external network 10.0.61.23 which reaches the controller. I hae also resolved the controller hostname in Kube master as 10.0.61.5 controller, which is reachable. But curl http://controller:9511/v1/ gets connection refused | 05:22 |
| ykarel | anyrude10, so http://controller:9511/v1/ will not work until and unless magnum is listening on external network and public endpoint is created with external interface | 05:25 |
| *** sidx64 has joined #openstack-containers | 05:52 | |
| *** ttsiouts has quit IRC | 06:13 | |
| anyrude10 | ykarel, i have figured out that issue and now my cloud-init-output.log completed without any error, just healthz check failed' ']' ping ok occurred. Now as per the bug, when I run the kubectl get nodes command, it shows no resources found | 06:27 |
| ykarel | anyrude10, what is ' '] and ping ok? | 06:31 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: WIP: k8s_fedora: Update to kubernetes v1.14.0 https://review.openstack.org/649609 | 06:50 |
| *** henriqueof has joined #openstack-containers | 06:59 | |
| *** ttsiouts has joined #openstack-containers | 07:04 | |
| *** sidx64 has quit IRC | 07:12 | |
| *** sidx64_ has joined #openstack-containers | 07:12 | |
| *** sidx64 has joined #openstack-containers | 07:15 | |
| *** gsimondon has joined #openstack-containers | 07:16 | |
| *** sidx64_ has quit IRC | 07:17 | |
| *** gsimondo1 has joined #openstack-containers | 07:23 | |
| *** ramishra_ has joined #openstack-containers | 07:24 | |
| *** ramishra has quit IRC | 07:25 | |
| *** gsimondon has quit IRC | 07:26 | |
| *** sidx64 has quit IRC | 07:31 | |
| *** ramishra_ is now known as ramishra | 07:36 | |
| *** ykarel is now known as ykarel|lunch | 07:39 | |
| anyrude10 | ykarel, My kube_master has been deployed successfully, but in Minions i am facing issue in Flannel service. curl -sf --cacert /etc/flanneld/certs/ca.crt --cert /etc/flanneld/certs/proxy.crt --key /etc/flanneld/certs/proxy.key 'https://10.0.0.8:2379/v2/keys/atomic.io/network/config?quorum=false&recursive=false&sorted=false' + echo 'Waiting for flannel configuration in etcd... | 07:39 |
| *** sidx64 has joined #openstack-containers | 07:42 | |
| *** rcernin has quit IRC | 07:48 | |
| *** ramishra_ has joined #openstack-containers | 08:29 | |
| *** ramishra has quit IRC | 08:32 | |
| *** sidx64 has quit IRC | 08:34 | |
| *** sidx64 has joined #openstack-containers | 08:43 | |
| *** anyrude10 has quit IRC | 08:58 | |
| *** anyrude10 has joined #openstack-containers | 09:01 | |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| anyrude10 | Hi, I am installing k8s cluster using magnum rocky and fedora 27. In minion, cloud-init-output.log is showing the following log: curl -sf --cacert /etc/flanneld/certs/ca.crt --cert /etc/flanneld/certs/proxy.crt --key /etc/flanneld/certs/proxy.key 'https://10.0.0.4:2379/v2/keys/atomic.io/network/config?quorum=false&recursive=false&sorted=false' + echo 'Waiting for flannel configuration in etcd...' Waiting for flannel co | 09:12 |
| *** sidx64 has quit IRC | 09:22 | |
| *** flwang1 has joined #openstack-containers | 09:26 | |
| flwang1 | strigazi: around for a catch up? | 09:27 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Revert "Specify internal network to improve stability in a multi-NIC scenario." https://review.openstack.org/650249 | 09:29 |
| flwang1 | strigazi: as for above patch, after reverted it, does the v1.14.0 work for you? | 09:31 |
| *** sidx64 has joined #openstack-containers | 09:56 | |
| *** yolanda has joined #openstack-containers | 10:07 | |
| *** sidx64 has quit IRC | 10:11 | |
| *** sidx64_ has joined #openstack-containers | 10:11 | |
| *** mkuf has quit IRC | 10:14 | |
| *** mkuf has joined #openstack-containers | 10:19 | |
| brtknr | flwang1: strigazi: hmm yes it doesn't play well with 1.14 | 10:35 |
| brtknr | I reverted it it seems okay now | 10:35 |
| flwang1 | brtknr: thanks for the feedback | 10:35 |
| *** ttsiouts has quit IRC | 10:54 | |
| *** ttsiouts has joined #openstack-containers | 10:55 | |
| *** ttsiouts has quit IRC | 10:59 | |
| *** ttsiouts has joined #openstack-containers | 11:01 | |
| *** ttsiouts has quit IRC | 11:11 | |
| *** ttsiouts has joined #openstack-containers | 11:12 | |
| *** ttsiouts has quit IRC | 11:16 | |
| *** ttsiouts has joined #openstack-containers | 11:21 | |
| *** ykarel is now known as ykarel|afk | 11:23 | |
| *** ioni has joined #openstack-containers | 11:24 | |
| ioni | hello guys | 11:25 |
| ioni | i'm trying for the first time magnum and i was able to spin a kubernets cluster | 11:25 |
| ioni | i was wondering if there is a way to set up the domain for template to use in order to have a FQDN hostname for all the vms that are part from the cluster | 11:26 |
| ioni | master and minions | 11:26 |
| ioni | currently the name is: k8s-cluster-somerandomtext-minion-0 | 11:26 |
| ioni | but i want something like k8s-cluster-somerandomtext-minion-0.mydomain.net | 11:27 |
| ioni | i'm trying to resolve this: | 11:27 |
| ioni | helm install stable/wordpress | 11:27 |
| ioni | Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp: lookup k8s-cluster-coreos-7pctbrnuykc6-minion-0 on 8.8.8.8:53: no such host | 11:27 |
| *** anyrude10 has quit IRC | 11:27 | |
| *** ykarel|afk is now known as ykarel | 11:30 | |
| *** udesale has quit IRC | 11:32 | |
| *** udesale has joined #openstack-containers | 11:32 | |
| *** yolanda has quit IRC | 11:39 | |
| brtknr | flwang1: Why do certs only get generated for ipv4 address and not ipv6? | 11:45 |
| brtknr | strigazi: ^ | 11:45 |
| *** mariusleu has joined #openstack-containers | 12:09 | |
| mariusleu | Hello, I am running Magnum on a Rocky release. Using Fedora-Atomic-27 | 12:10 |
| mariusleu | after creating a Kubernetes cluster, the kube_masters fail to install etcd certificates, | 12:11 |
| mariusleu | getting this in journalctl -xn: open /etc/etcd/certs/server.crt: no such file or directory | 12:11 |
| mariusleu | etcd fails to start because of this, thus kube-apiserver fails to start as well | 12:11 |
| mariusleu | I am using the Magnum's internal x509 configuration for certificates, not using Barbican | 12:12 |
| mariusleu | @channel did anyone run into this issue? | 12:14 |
| guimaluf | I'm running on queen with fedora atomic 27 and gettings this: Failed to update Node Allocatable Limits "": failed to set supported cgroup subsy | 12:15 |
| guimaluf | stems for cgroup : Failed to set config for supported subsystems : failed to write 4135825408 to memory.limit_in_bytes: write /rootfs/var/lib/containers/atomic/kubelet.0/rootfs/sys/fs/cgroup/ | 12:15 |
| guimaluf | memory/memory.limit_in_bytes: invalid argument | 12:15 |
| *** ivve has quit IRC | 12:20 | |
| brtknr | mariusleu: check your /var/log/cloud-init-output.log for errors | 12:23 |
| brtknr | guimaluf: can you create a pod? | 12:23 |
| guimaluf | brtknr, everything I try to create get 0/1 on READY | 12:24 |
| brtknr | guimaluf: how did you deploy magnum? | 12:24 |
| guimaluf | brtknr, using puppet-magnum | 12:25 |
| guimaluf | actually, I had a Pike openstack and magnum was not creating any cluster. so I upgraded to queens and at least now I have a k8s cluster. but I can't creating nothing... | 12:26 |
| guimaluf | I tought was lack of resourcers, since my instances were m1.small, but I've create another cluster with m1.medium and the result is the same | 12:27 |
| brtknr | guimaluf: how much RAM does m1.medium have? | 12:29 |
| guimaluf | brtknr, master 2GB, minions 4GB | 12:30 |
| guimaluf | I'm trying to run hello-world-like pods/deployments | 12:30 |
| brtknr | show me your cluster template | 12:31 |
| brtknr | use paste.openstack.org | 12:31 |
| dioguerra | flwangl: how did you test for NPD? shoudn't the minion status change to NotReady with https://github.com/kubernetes/node-problem-detector/blob/master/test/kernel_log_generator/problems/docker_hung | 12:31 |
| guimaluf | brtknr, http://paste.openstack.org/show/748940/ | 12:33 |
| brtknr | guimaluf: have you tried with docker_storage_driver=overlay | 12:34 |
| brtknr | guimaluf: and docker_volume_size=0 | 12:34 |
| brtknr | guimaluf: can you also tell me which version of fedora-atomic you're using? | 12:35 |
| guimaluf | brtknr, IRC I'm using fedora atomic 27 | 12:38 |
| guimaluf | brtknr, I will recreate my cluster and try with those options. should I create a new template, or can I pass those options to coe cluster create? | 12:39 |
| *** mariusleu has quit IRC | 12:39 | |
| brtknr | guimaluf: i suggest creating a new template in case it doesnt work | 12:40 |
| guimaluf | I'll give a try ;) ty | 12:40 |
| guimaluf | brtknr, Unknown attribute for argument cluster_template: hidden, maybe I'm using magnum 2.13 in a queens openstack? | 12:50 |
| guimaluf | brtknr, downgrading magnumclient I can create the new template. but I couldn't set docker-volume-size 0, I let 1 :) | 12:57 |
| *** mkuf_ has joined #openstack-containers | 13:07 | |
| *** mkuf has quit IRC | 13:11 | |
| *** mkuf_ has quit IRC | 13:11 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_fedora: Update to kubernetes v1.14.0 https://review.openstack.org/649609 | 13:42 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Revert "ci: Disable functional tests" https://review.openstack.org/642873 | 13:51 |
| *** johanssone has joined #openstack-containers | 13:52 | |
| openstackgerrit | Merged openstack/magnum master: Revert "Specify internal network to improve stability in a multi-NIC scenario." https://review.openstack.org/650249 | 13:59 |
| *** sidx64_ has quit IRC | 13:59 | |
| *** goldyfruit has joined #openstack-containers | 14:00 | |
| guimaluf | brtknr, did as you said and I got the same error Failed to update Node Allocatable Limits "": failed to set supported cgroup subsystems for cgroup : Failed to set config for supported subsystems : failed to write 4135833600 to memory.limit_in_bytes: write /rootfs/var/lib/containers/atomic/kubelet.0/rootfs/sys/fs/cgroup/memory/memory.limit_in_bytes: invalid argument | 14:07 |
| *** gsimondo1 has quit IRC | 14:16 | |
| *** ttsiouts has quit IRC | 14:22 | |
| *** ttsiouts has joined #openstack-containers | 14:23 | |
| *** ttsiouts has quit IRC | 14:24 | |
| *** ttsiouts has joined #openstack-containers | 14:24 | |
| brtknr | guimaluf: where do you see that error? | 14:47 |
| *** itlinux has joined #openstack-containers | 14:54 | |
| NobodyCam | ioni: I solved that issue but creating a systemd service that builds a /etc/hosts file on the master node at startup | 14:59 |
| *** ttsiouts has quit IRC | 15:00 | |
| *** ttsiouts has joined #openstack-containers | 15:00 | |
| *** ttsiouts has quit IRC | 15:01 | |
| *** ttsiouts has joined #openstack-containers | 15:02 | |
| NobodyCam | trick is to create the hosts inside the kube-* services containers | 15:06 |
| *** yolanda has joined #openstack-containers | 15:13 | |
| brtknr | flwang1: how do I resolve this when using keystone auth: Error from server (Forbidden): nodes is forbidden: User "demo" cannot list resource "nodes" in API group "" at the cluster scope | 15:15 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: [k8s] Set traefik to stable version v1.7.9 https://review.openstack.org/650331 | 15:15 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: [k8s] Set traefik to stable version v1.7.9 https://review.openstack.org/650347 | 15:21 |
| *** ttsiouts has quit IRC | 15:25 | |
| *** ttsiouts has joined #openstack-containers | 15:26 | |
| *** ttsiouts has quit IRC | 15:30 | |
| *** henriqueof has quit IRC | 15:40 | |
| *** ykarel is now known as ykarel|away | 15:41 | |
| *** yolanda has quit IRC | 15:52 | |
| *** udesale has quit IRC | 16:03 | |
| *** ykarel|away has quit IRC | 16:22 | |
| *** ivve has joined #openstack-containers | 16:34 | |
| ioni | NobodyCam, i believe is mostly problem with coreos, with fedora-atomic, works fine | 16:43 |
| *** ramishra_ has quit IRC | 16:46 | |
| *** ivve has quit IRC | 16:48 | |
| *** ykarel|away has joined #openstack-containers | 17:22 | |
| *** itlinux has quit IRC | 17:25 | |
| guimaluf | brtknr, I see that error when I run `kubectl get events` or `kubectl describe nodes` | 17:28 |
| *** hongbin has joined #openstack-containers | 17:53 | |
| *** itlinux has joined #openstack-containers | 18:08 | |
| *** mkuf_ has joined #openstack-containers | 18:12 | |
| *** sidx64 has joined #openstack-containers | 18:22 | |
| *** sidx64_ has joined #openstack-containers | 18:26 | |
| *** sidx64 has quit IRC | 18:27 | |
| *** ivve has joined #openstack-containers | 18:32 | |
| *** goldyfruit has quit IRC | 18:36 | |
| *** itlinux has quit IRC | 18:42 | |
| flwang1 | dioguerra: ping | 18:50 |
| flwang1 | brtknr: still around? | 18:50 |
| *** mkuf_ has quit IRC | 18:54 | |
| *** itlinux has joined #openstack-containers | 19:13 | |
| flwang1 | brtknr: by default, user don't have permission to list nodes | 19:22 |
| flwang1 | brtknr: here is the default policy https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/enable-keystone-auth.sh#L72 | 19:23 |
| *** goldyfruit has joined #openstack-containers | 20:44 | |
| *** sidx64_ has quit IRC | 21:06 | |
| *** goldyfruit has left #openstack-containers | 21:42 | |
| *** mgariepy has quit IRC | 22:01 | |
| *** tbarron has quit IRC | 22:01 | |
| *** brtknr has quit IRC | 22:01 | |
| *** aspiers has quit IRC | 22:01 | |
| *** guimaluf has quit IRC | 22:01 | |
| *** nwonknu has quit IRC | 22:01 | |
| *** FracKen has quit IRC | 22:01 | |
| *** FracKen has joined #openstack-containers | 22:02 | |
| *** nwonknu has joined #openstack-containers | 22:07 | |
| *** aspiers has joined #openstack-containers | 22:23 | |
| *** openstackgerrit has quit IRC | 22:28 | |
| *** openstackgerrit has joined #openstack-containers | 22:39 | |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: [k8s] Add nginx based ingress controller https://review.openstack.org/648655 | 22:39 |
| *** pcaruana has quit IRC | 22:57 | |
| *** ykarel|away has quit IRC | 23:01 | |
| *** hongbin has quit IRC | 23:09 | |
| *** sapd1_x has joined #openstack-containers | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!