Wednesday, 2018-05-09

*** fragatina has quit IRC00:12
*** Nel1x has joined #openstack-containers00:40
*** harlowja has quit IRC00:51
*** armaan has quit IRC01:15
*** armaan has joined #openstack-containers01:16
*** ianychoi has joined #openstack-containers01:21
*** username_ has joined #openstack-containers01:21
*** username_ is now known as username__01:23
*** hongbin has joined #openstack-containers01:37
*** hongbin_ has joined #openstack-containers01:44
*** hongbin has quit IRC01:45
*** gsimondon has joined #openstack-containers01:51
*** gsimondon has quit IRC01:55
*** yamamoto has joined #openstack-containers02:20
*** yamamoto has quit IRC02:24
*** username__ has quit IRC02:35
*** yamamoto has joined #openstack-containers02:37
*** fragatina has joined #openstack-containers02:55
*** fragatina has quit IRC02:55
*** fragatina has joined #openstack-containers02:56
*** fragatina has quit IRC02:57
*** fragatina has joined #openstack-containers02:57
*** rcernin has quit IRC03:14
*** ricolin_ has joined #openstack-containers03:14
*** yamamoto has quit IRC03:31
*** ramishra has joined #openstack-containers03:36
*** udesale has joined #openstack-containers03:36
*** threestrands has joined #openstack-containers03:38
*** threestrands has quit IRC03:38
*** threestrands has joined #openstack-containers03:38
*** yamamoto has joined #openstack-containers03:44
*** gyan_ has joined #openstack-containers03:44
*** hongbin_ has quit IRC03:46
*** dave-mccowan has quit IRC03:48
*** udesale has quit IRC04:00
*** udesale has joined #openstack-containers04:01
*** Nel1x has quit IRC04:08
*** janki has joined #openstack-containers04:10
*** ykarel|away has joined #openstack-containers04:18
*** udesale_ has joined #openstack-containers04:36
*** udesale has quit IRC04:39
*** zhubingbing__ has joined #openstack-containers04:42
*** zhubingbing_ has quit IRC04:45
*** ykarel|away is now known as ykarel05:07
*** yamamoto has quit IRC05:15
*** yamamoto has joined #openstack-containers05:20
*** udesale_ has quit IRC05:22
*** akhil_jain has joined #openstack-containers05:26
*** iranzo has joined #openstack-containers05:47
*** iranzo has joined #openstack-containers05:47
*** udesale_ has joined #openstack-containers05:51
*** udesale_ has quit IRC05:53
*** udesale has joined #openstack-containers05:53
*** threestrands has quit IRC05:54
*** pcaruana has joined #openstack-containers05:57
*** armaan has quit IRC06:05
*** armaan has joined #openstack-containers06:05
*** gsimondon has joined #openstack-containers06:11
*** armaan_ has joined #openstack-containers06:13
*** armaan has quit IRC06:16
*** armaan_ has quit IRC06:33
*** armaan has joined #openstack-containers06:34
*** inovas_ has quit IRC06:44
*** inovas_ has joined #openstack-containers06:45
*** mago_ has joined #openstack-containers07:06
*** olivenwk has joined #openstack-containers07:07
*** yamamoto_ has joined #openstack-containers07:10
*** yamamoto has quit IRC07:10
*** yamamoto_ has quit IRC07:29
*** yamamoto has joined #openstack-containers07:30
*** masber has quit IRC07:36
*** mgoddard has joined #openstack-containers07:36
*** mdnadeem has joined #openstack-containers07:41
*** mdnadeem has quit IRC07:41
*** armaan has quit IRC07:50
*** armaan has joined #openstack-containers07:51
*** ykarel is now known as ykarel|lunch07:58
*** armaan has quit IRC08:05
*** armaan has joined #openstack-containers08:06
mago_Hi team, following our discussions here with strigazi, I'm in the process of installing Queens/Magnum alongside a standard Pike installation (Keystone, Nova, ... Heat). The services are running in Kolla containers behind HAproxy on three controller nodes, with MySQL DB replication performed by Galera.08:10
mago_I'm hitting several issues in my deployment, I'm not sure if they are related to each other, and also not sure if they are related to the Queens vs Pike compatibility08:11
*** mdnadeem has joined #openstack-containers08:12
strigazimago_: no issue between magnum queens and other openestack projetcs in pike08:12
mago_strigazi, thanks for confirming. No special configuration to perform then ? I eliminate this potential issue of my list :)08:14
mago_1st issue : After the installation, when performing basic "openstack coe ..." commands, I have ERROR logs concerning MySQL (by oslo) and RabbitMQ. Please see : http://paste.openstack.org/show/720642/08:15
*** yamamoto has quit IRC08:15
mago_However, the creation of a K8s cluster completes successfully from an Openstack point of view (CLUSTER CREATED state)08:16
*** yamamoto has joined #openstack-containers08:17
strigazimago_: it looks like connectivity issues?08:18
mago_2nd issue : On my K8s master, cloud init shows several errors  and system pods don't start : http://paste.openstack.org/show/720643/08:21
mago_strigazi, concerning Mysql, yes it looks like. I hope my network isn't so bad ... What about the RabbitMQ erros and the "durable" parameter ? I tried changing amqp_durable_queues to True in magnum.conf, but got then other issues08:23
strigazimago_: the deafault config should work08:23
strigazifor rabbit ^^08:23
strigaziYou are using very small flavors?08:24
mago_m1.small, that is 2OGB of disk with Fedora Atomic 2708:24
strigaziYou need to use f27 atomic with magnum queens08:24
strigaziERROR: There is not enough free space in volume group atomicos to create data volume of size MIN_DATA_SIZE=2G08:24
strigazim1.small should work08:25
mago_strigazi, Yes, I'm using f2708:26
mago_and yes, m1.small worked for me in a small test deployment08:26
strigaziatomic containers list08:27
strigaziatomic images  list08:27
strigazilsblk08:28
strigazidf -u08:28
*** udesale_ has joined #openstack-containers08:28
*** udesale has quit IRC08:28
mago_df -u ?08:29
strigazion the master node08:29
mago_this option isn't recognized08:29
strigazidf -h , sorry08:30
mago_http://paste.openstack.org/show/720644/08:30
strigaziyou habe 10gb not 2008:31
strigazijournalctl -u kube-controller-manager --no-pager08:32
mago_ah yes true. I first tried with a 10GB flavor, saw this issue you mentioned, retried with a 20GB flavor and hit the same08:32
mago_forgot that sorry08:32
mago_I deleted the second cluster (the one with 20GB when I saw that the same ERROR log was in the cloud init output08:33
*** lpetrut has joined #openstack-containers08:33
mago_I could boot a cluster with a 20GB of disk or more08:33
strigazithis one works in our env https://ftp-stud.hs-esslingen.de/pub/Mirrors/alt.fedoraproject.org/atomic/stable/Fedora-Atomic-27-20180314.0/CloudImages/x86_64/images/08:35
mago_OK, I have Fedora-Atomic-27-20180212.2.x86_64.qcow208:35
mago_I will try with this image instead08:35
strigazithis shouldn't be a problem08:35
mago_ah ok08:35
mago_strigazi, I'm currently booting a new cluster using your provided image and m1.medium flavor (40GB of disk) to eliminate those issues. FYI, the template I use : http://paste.openstack.org/show/720646/08:48
strigazimago_ docker_storage_driver use overlay208:48
*** salmankhan has joined #openstack-containers08:49
strigazimago_: be sure that 8.8.8.8 is usable in your cloud08:49
mago_ok, will try with this docker_storage_driver (alhough I don't know what it means yet :) ). The 8.8.8.8 is usable in my environment yes.08:50
mago_strigay, shall I tag with 1.9.3 or not necessary ?08:51
mago_strigazy ^08:51
mago_strigazi ^ *08:52
mago_(sorry for mispelling !! :) )08:52
*** yamamoto has quit IRC08:55
*** gyankum has joined #openstack-containers08:55
*** qwebirc25298 has joined #openstack-containers08:56
*** qwebirc25298 has quit IRC08:57
*** udesale_ has quit IRC08:59
*** udesale_ has joined #openstack-containers08:59
*** udesale_ has quit IRC09:00
*** udesale has joined #openstack-containers09:00
*** salmankhan has quit IRC09:00
mago_strigazi, ok cluster creation in progress, this should take a while. thanks for the help so far, I will see if the K8s errors are resolved with this settings / flavor / images ..09:01
mago_Concerning the RabbitMQ errors and the "durable" parameter, do you have any idea ?09:01
*** salmankhan has joined #openstack-containers09:03
*** flwang has quit IRC09:03
*** yamamoto has joined #openstack-containers09:11
*** flwang has joined #openstack-containers09:17
*** salmankhan has quit IRC09:20
*** serlex has joined #openstack-containers09:28
*** ykarel|lunch is now known as ykarel09:28
*** openstackgerrit has joined #openstack-containers09:32
openstackgerritLingxian Kong proposed openstack/magnum master: Use Octavia for LoadBalancer type service  https://review.openstack.org/56717309:32
*** salmankhan has joined #openstack-containers09:37
*** yamamoto has quit IRC10:49
*** yamamoto has joined #openstack-containers10:51
*** mgoddard_ has joined #openstack-containers10:54
*** mgoddard has quit IRC10:56
*** mgoddard_ is now known as mgoddard10:56
*** ykarel_ has joined #openstack-containers10:59
*** ykarel has quit IRC11:00
*** yamamoto has quit IRC11:25
*** yamamoto has joined #openstack-containers11:31
*** yamamoto has quit IRC11:32
*** Nel1x has joined #openstack-containers11:37
*** armaan has quit IRC11:41
*** armaan has joined #openstack-containers11:41
mago_Hi again strigazi. I solved my RabbitMQ issue (needed to delete and recreate some exchange and queues with the correct parameters). I assume the MySQL issues are connectivity issues (after all, the cluster creation is successful). I now only have the k8s issues to solve.11:50
mago_I'm using the F27 image you provided, and a medium (40GB of disk) flavor. The creation goes to the end and appear shows as successful from an Openstack point of view11:51
mago_The error concerning size disk disappeared now. I still have those issues : http://paste.openstack.org/show/720667/11:53
mago_I was wondering if this could be related to an API version .. ?11:53
mago_I'm not using any "kube_tag" and was wondering if it is necessary ? K8s is installed in 1.9.311:54
*** yamamoto has joined #openstack-containers11:58
*** astellwag has quit IRC12:15
*** armaan has quit IRC12:16
*** jmlowe has quit IRC12:20
*** pcaruana has quit IRC12:21
*** Nel1x has quit IRC12:28
*** armaan has joined #openstack-containers12:30
*** janki has quit IRC12:30
*** janki has joined #openstack-containers12:31
strigazimago_: check the content of /var/lib/cloud/instance/scripts/part-017 and try to execure the commands manually from the master12:33
strigazimago_: can you do kubectl --version ?12:33
mago_Kubernetes v1.9.312:34
strigazimago_: can you show me the full output of cloudinit output? without passwords and endpoints :)12:35
strigazikubectl version12:35
strigazito see the api server version as weel12:35
strigazito see the api server version as well12:35
strigazikubectl get cs12:35
mago_Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"archive", BuildDate:"2018-02-13T11:42:06Z", GoVersion:"go1.10rc2", Compiler:"gc", Platform:"linux/amd64"}12:35
mago_Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"archive", BuildDate:"2018-02-13T11:42:06Z", GoVersion:"go1.10rc2", Compiler:"gc", Platform:"linux/amd64"}12:35
strigazijournalctl -u kube-controller-manager --no-pager12:35
strigazijournalctl -u kube-apiserver --no-pager12:35
strigazicat /etc/kubernetes/apiserver12:36
*** yamamoto has quit IRC12:39
mago_cloud-init-output : http://paste.openstack.org/show/720673/12:41
mago_journalctl -u kube-controller-manager --no-pager : http://paste.openstack.org/show/720674/12:43
*** dave-mccowan has joined #openstack-containers12:43
mago_journalctl -u kube-apiserver --no-pager : http://paste.openstack.org/show/720675/12:44
*** yamamoto has joined #openstack-containers12:44
mago_cat /etc/kubernetes/apiserver : http://paste.openstack.org/show/720676/12:45
*** yamamoto has quit IRC12:45
brtknrAnyone know whether does openstacksdk support Magnum yet? there is something called clustering but I am not sure whether that has anything to do with Magnum12:46
mago_kubectl get cs : http://paste.openstack.org/show/720677/12:46
strigazibrtknr https://github.com/openstack/python-magnumclient/tree/master/magnumclient/osc https://github.com/openstack/python-magnumclient/blob/master/setup.cfg#L3212:47
brtknrAh cool thanks12:47
strigazimago_ the controller clrearly has an issue12:49
brtknrAre there plans to integrate it into the main OpenstackSDK?12:50
*** gyan_ has quit IRC12:51
*** gyankum has quit IRC12:51
strigazimago_: cat /etc/kubernetes/controller-manager12:52
brtknrOr use os-client-config?12:52
strigazibrtknr: all projects have openstacksdk plugins, they are not in the same repo12:52
mago_cat /etc/kubernetes/controller-manager : http://paste.openstack.org/show/720678/12:53
*** lpetrut has quit IRC12:53
strigazimago_: error starting controllers: failed to start certificate controller: error reading CA key file "/etc/kubernetes/certs/ca.key": open /etc/kubernetes/certs/ca.key: no such file or directory12:54
*** armaan has quit IRC12:54
strigazimago_: can you check again? I think the controller manager should ok now12:55
strigazidoes this file exist? /etc/kubernetes/certs/ca.key ?12:55
mago_strigazi, the ca.key file does not exist12:55
mago_the controller manager is currently down. I can try to start it again ?12:56
strigazimago_: try to remove these two: --cluster-signing-cert-file=/etc/kubernetes/certs/ca.crt --cluster-signing-key-file=/etc/kubernetes/certs/ca.key and restart the controller manager12:56
strigaziyou can try a cluster again with:12:56
brtknrhow do you install the plugin before installing the packages?12:57
strigazi--label cert_manager_api=true12:57
brtknr*before=beyond12:57
strigaziyum install python-openstackclient python-magnumclient12:58
mago_Removing those two options, the controller manager starts12:58
strigazior pip install12:58
strigazimago_: I think you don't have this patch https://github.com/openstack/magnum/commit/5a34d7d830ad4b6a714f079d4575e1705df434f3#diff-9f34669d308c2375d1f18763f49d587b12:59
brtknri already did that but i still cant reference to openstack.osc13:00
*** inovas_ has quit IRC13:00
*** inovas_ has joined #openstack-containers13:01
strigaziyum? pip?13:01
mago_strigazi, thank you. I will try with --label cert_manager_api=true13:01
brtknri installed using pip inside a venv13:01
*** jmlowe has joined #openstack-containers13:02
brtknri open ipython then run import openstack.cluster_infra13:02
brtknri want to create an ansible python library to create containers13:03
brtknri think what you showed me is the syntax for using openstack cli to interact with magnum13:04
brtknri want to interact with magnum from python script13:05
mago_strigazi : One more question : Is the label a workaround of the patch ? Does applying the patch makes the use of the label not necessary ?13:05
*** lpetrut has joined #openstack-containers13:09
*** flwang1 has quit IRC13:12
*** ykarel_ is now known as ykarel13:13
*** yolanda has joined #openstack-containers13:16
*** yolanda_ has quit IRC13:18
*** flwang1 has joined #openstack-containers13:21
*** ricolin_ has quit IRC13:32
*** ricolin has joined #openstack-containers13:33
*** armaan has joined #openstack-containers13:34
*** olivenwk has quit IRC13:35
*** olivenwk has joined #openstack-containers13:35
*** livelace has joined #openstack-containers13:36
*** pcaruana has joined #openstack-containers13:40
*** yamamoto has joined #openstack-containers13:46
brtknrnevermind, i found a way to do it  but it wasnt well documented13:51
*** hongbin has joined #openstack-containers13:52
*** ykarel is now known as ykarel|away13:53
*** yamamoto has quit IRC13:56
*** lpetrut has quit IRC14:02
*** pcaruana has quit IRC14:05
*** pcaruana has joined #openstack-containers14:08
*** lpetrut has joined #openstack-containers14:08
*** lpetrut has quit IRC14:18
*** pcaruana has quit IRC14:25
*** pcaruana has joined #openstack-containers14:26
*** udesale has quit IRC14:46
*** janki has quit IRC14:46
*** srihas has quit IRC14:52
*** lpetrut has joined #openstack-containers15:05
*** serlex has quit IRC15:06
*** gyankum has joined #openstack-containers15:08
*** gyan_ has joined #openstack-containers15:08
*** gsimondo1 has joined #openstack-containers15:10
*** gsimondon has quit IRC15:12
*** gsimondo1 has quit IRC15:15
*** fragatina has quit IRC15:20
*** armaan has quit IRC15:25
*** ykarel|away has quit IRC15:26
*** armaan has joined #openstack-containers15:26
*** serlex has joined #openstack-containers15:30
*** olivenwk has quit IRC15:43
*** mago_ has quit IRC15:46
*** serlex has quit IRC15:47
*** ykarel|away has joined #openstack-containers15:55
*** ramishra has quit IRC16:09
*** ykarel|away has quit IRC16:16
*** gsimondon has joined #openstack-containers16:24
*** lpetrut has quit IRC16:25
*** gsimondon has quit IRC16:29
*** ramishra has joined #openstack-containers16:31
*** udesale has joined #openstack-containers16:32
*** gyankum has quit IRC16:33
*** gyan_ has quit IRC16:34
*** lpetrut has joined #openstack-containers16:34
*** iranzo has quit IRC16:38
*** masber has joined #openstack-containers16:40
*** gsimondon has joined #openstack-containers16:45
*** ramishra has quit IRC16:45
*** ricolin has quit IRC16:48
*** gsimondon has quit IRC16:50
*** armaan has quit IRC16:55
*** armaan has joined #openstack-containers16:55
*** lpetrut has quit IRC17:00
*** lpetrut has joined #openstack-containers17:01
*** mgoddard has quit IRC17:04
*** gsimondon has joined #openstack-containers17:05
*** fragatina has joined #openstack-containers17:09
*** gsimondon has quit IRC17:10
*** gsimondon has joined #openstack-containers17:13
*** mdnadeem has quit IRC17:16
*** salmankhan has quit IRC17:17
*** gsimondo1 has joined #openstack-containers17:18
*** gsimondon has quit IRC17:18
*** mgoddard has joined #openstack-containers17:23
*** udesale has quit IRC17:50
*** gsimondo1 has quit IRC17:51
*** janki has joined #openstack-containers17:52
*** armaan has quit IRC18:00
*** armaan has joined #openstack-containers18:00
*** pcichy has joined #openstack-containers18:11
*** janki has quit IRC18:13
*** gsimondon has joined #openstack-containers18:16
*** armaan has quit IRC18:17
*** armaan has joined #openstack-containers18:17
*** armaan has quit IRC18:18
*** armaan has joined #openstack-containers18:18
*** mdnadeem has joined #openstack-containers18:20
*** oikiki has joined #openstack-containers18:24
*** mgoddard has quit IRC18:32
*** mdnadeem has quit IRC18:39
*** gsimondon has quit IRC18:43
*** fragatina has quit IRC19:19
*** gsimondon has joined #openstack-containers19:49
*** jmlowe has quit IRC19:58
*** lpetrut has quit IRC20:14
*** jmlowe has joined #openstack-containers20:53
*** pcaruana has quit IRC20:59
*** kbyrne has quit IRC21:09
*** kbyrne has joined #openstack-containers21:12
*** dave-mccowan has quit IRC21:17
*** dave-mccowan has joined #openstack-containers21:21
*** fragatina has joined #openstack-containers21:26
*** fragatina has quit IRC21:26
*** fragatina has joined #openstack-containers21:27
*** jmlowe has quit IRC21:39
*** threestrands has joined #openstack-containers22:33
*** threestrands has quit IRC22:33
*** threestrands has joined #openstack-containers22:33
*** gsimondon has quit IRC22:33
*** threestrands_ has joined #openstack-containers22:36
*** threestrands has quit IRC22:38
*** hongbin has quit IRC22:43
*** gsimondon has joined #openstack-containers22:50
*** gsimondon has quit IRC22:54
*** oikiki has quit IRC23:01
*** gsimondon has joined #openstack-containers23:10
*** gsimondon has quit IRC23:15
*** oikiki has joined #openstack-containers23:16
*** oikiki has quit IRC23:17
*** livelace has quit IRC23:23
*** gsimondon has joined #openstack-containers23:31
*** gsimondon has quit IRC23:35
*** lpetrut has joined #openstack-containers23:47
*** gsimondon has joined #openstack-containers23:51
*** lpetrut has quit IRC23:52
*** gsimondon has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!