| *** rtjure has quit IRC | 00:00 | |
| *** rtjure has joined #openstack-containers | 00:03 | |
| *** hongbin has quit IRC | 00:10 | |
| *** yamamoto has joined #openstack-containers | 00:14 | |
| *** yamamoto has quit IRC | 00:20 | |
| *** itlinux has joined #openstack-containers | 00:24 | |
| *** itlinux has quit IRC | 00:50 | |
| *** Chealion has quit IRC | 00:57 | |
| *** Chealion has joined #openstack-containers | 00:58 | |
| *** oikiki has quit IRC | 01:07 | |
| *** oikiki has joined #openstack-containers | 01:07 | |
| *** oikiki has quit IRC | 01:10 | |
| *** daidv has joined #openstack-containers | 01:15 | |
| *** yamamoto has joined #openstack-containers | 01:16 | |
| *** yamamoto has quit IRC | 01:21 | |
| *** rcernin has quit IRC | 01:29 | |
| *** rcernin has joined #openstack-containers | 01:30 | |
| *** itlinux has joined #openstack-containers | 01:45 | |
| *** kiennt26 has joined #openstack-containers | 01:52 | |
| *** itlinux has quit IRC | 01:53 | |
| *** ramishra has joined #openstack-containers | 02:01 | |
| *** wangbo has joined #openstack-containers | 02:04 | |
| *** yamamoto has joined #openstack-containers | 02:17 | |
| *** yamamoto has quit IRC | 02:22 | |
| *** itlinux has joined #openstack-containers | 02:47 | |
| *** dpawar has joined #openstack-containers | 02:48 | |
| *** dpawar has quit IRC | 02:49 | |
| *** fragatina has quit IRC | 02:55 | |
| *** manheim has joined #openstack-containers | 02:56 | |
| *** fragatina has joined #openstack-containers | 02:59 | |
| *** fragatina has quit IRC | 02:59 | |
| *** fragatina has joined #openstack-containers | 03:00 | |
| *** fragatina has quit IRC | 03:01 | |
| *** manheim has quit IRC | 03:01 | |
| *** dpawar has joined #openstack-containers | 03:02 | |
| *** fragatina has joined #openstack-containers | 03:06 | |
| *** eliqiao_ has joined #openstack-containers | 03:09 | |
| *** yamamoto has joined #openstack-containers | 03:09 | |
| *** ricolin has joined #openstack-containers | 03:10 | |
| *** fragatina has quit IRC | 03:10 | |
| *** dpawar has quit IRC | 03:26 | |
| *** dpawar has joined #openstack-containers | 03:26 | |
| *** itlinux has quit IRC | 03:27 | |
| *** hongbin has joined #openstack-containers | 03:34 | |
| *** ramishra has quit IRC | 03:39 | |
| *** absubram has quit IRC | 03:59 | |
| *** itlinux has joined #openstack-containers | 04:01 | |
| *** mdnadeem has joined #openstack-containers | 04:12 | |
| *** ykarel|away has joined #openstack-containers | 04:12 | |
| *** ykarel|away is now known as ykarel | 04:31 | |
| *** janki has joined #openstack-containers | 04:32 | |
| *** ramishra has joined #openstack-containers | 04:35 | |
| *** dpawar has quit IRC | 04:46 | |
| *** wangbo has quit IRC | 04:49 | |
| *** kiennt26 has quit IRC | 04:58 | |
| *** dpawar has joined #openstack-containers | 05:01 | |
| *** hongbin has quit IRC | 05:05 | |
| *** wangbo has joined #openstack-containers | 05:06 | |
| *** fragatina has joined #openstack-containers | 05:08 | |
| *** fragatina has quit IRC | 05:09 | |
| *** fragatina has joined #openstack-containers | 05:10 | |
| *** manheim has joined #openstack-containers | 05:12 | |
| *** janonymous has joined #openstack-containers | 05:12 | |
| *** manheim has quit IRC | 05:17 | |
| *** janki has quit IRC | 05:17 | |
| *** yamamoto_ has joined #openstack-containers | 05:18 | |
| *** yamamoto has quit IRC | 05:21 | |
| *** fragatin_ has joined #openstack-containers | 05:25 | |
| *** fragatina has quit IRC | 05:27 | |
| *** janki has joined #openstack-containers | 05:35 | |
| *** ramishra has quit IRC | 06:11 | |
| *** ramishra has joined #openstack-containers | 06:18 | |
| *** lpetrut has joined #openstack-containers | 06:24 | |
| yasemin | ykarel, hi, i updated magnum service files about SSL problem, I followed https://review.openstack.org/#/c/447687/ , but i have a problem about swarm cluster | 06:27 |
|---|---|---|
| yasemin | magum-conductor logs 23486 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: The Parameter (verify_ca) was not defined in template. | 06:27 |
| yasemin | 06:27 | |
| yasemin | any idea ? | 06:27 |
| *** rtjure has quit IRC | 06:29 | |
| ykarel | yasemin, looks like you missed some params | 06:30 |
| *** rtjure has joined #openstack-containers | 06:32 | |
| ykarel | yasemin, can you recheck your swarm templates are same as in https://review.openstack.org/#/c/447687/ | 06:32 |
| *** openstackgerrit has quit IRC | 06:33 | |
| yasemin | ykarel, okey | 06:34 |
| *** ykarel_ has joined #openstack-containers | 06:36 | |
| *** ykarel has quit IRC | 06:38 | |
| *** ramishra has quit IRC | 06:38 | |
| *** dpawar has quit IRC | 06:57 | |
| *** ramishra has joined #openstack-containers | 06:59 | |
| *** absubram has joined #openstack-containers | 07:01 | |
| *** dpawar has joined #openstack-containers | 07:05 | |
| *** absubram has quit IRC | 07:05 | |
| *** hishh has joined #openstack-containers | 07:07 | |
| *** dsariel has joined #openstack-containers | 07:07 | |
| *** itlinux has quit IRC | 07:07 | |
| *** absubram has joined #openstack-containers | 07:09 | |
| *** mjura has joined #openstack-containers | 07:13 | |
| *** rcernin has quit IRC | 07:17 | |
| *** ykarel__ has joined #openstack-containers | 07:23 | |
| *** ykarel_ has quit IRC | 07:25 | |
| *** yamamoto_ has quit IRC | 07:27 | |
| *** manheim has joined #openstack-containers | 07:29 | |
| *** jberkus has quit IRC | 07:38 | |
| yasemin | ykarel, | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server raise e | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server InvalidParameterValue: ERROR: Failed to validate: Error parsing template: while parsing a block mapping | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server in "<unicode string>", line 1, column 1: | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server heat_template_version: 2014-10-16 | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server ^ | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server expected <block end>, but found '<block mapping start>' | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server in "<unicode string>", line 83, column 2: | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server verify_ca: | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server ^ | 07:44 |
| yasemin | 2017-11-22 10:40:53.732 24086 ERROR oslo_messaging.rpc.server | 07:44 |
| yasemin | ^C | 07:44 |
| yasemin | i didnt change this magnum-15.1.9/lib/python2.7/site-packages/magnum/conductor/handlers/cluster_conductor.py", line 81, in cluster_create | 07:44 |
| yasemin | 07:45 | |
| yasemin | but it gives the error | 07:45 |
| *** ykarel__ is now known as ykarel|lunch | 07:46 | |
| *** dsariel has quit IRC | 07:46 | |
| *** threestrands has quit IRC | 07:47 | |
| *** pcaruana has joined #openstack-containers | 07:49 | |
| *** lpetrut has quit IRC | 07:50 | |
| yasemin | ykarel, http://paste.openstack.org/show/627036/ | 07:51 |
| yasemin | this is my kubernetes cluster cloud-init-output :/ | 07:52 |
| *** mgoddard has joined #openstack-containers | 07:53 | |
| *** lpetrut has joined #openstack-containers | 07:55 | |
| *** manheim has quit IRC | 08:04 | |
| *** manheim has joined #openstack-containers | 08:04 | |
| *** manheim has quit IRC | 08:09 | |
| *** manheim has joined #openstack-containers | 08:17 | |
| *** manheim_ has joined #openstack-containers | 08:19 | |
| *** manheim has quit IRC | 08:19 | |
| *** manheim_ has quit IRC | 08:20 | |
| *** manheim has joined #openstack-containers | 08:21 | |
| *** lpetrut has quit IRC | 08:27 | |
| *** yamamoto has joined #openstack-containers | 08:27 | |
| *** dpawar has quit IRC | 08:27 | |
| *** ykarel|lunch is now known as ykarel | 08:35 | |
| *** yamamoto has quit IRC | 08:35 | |
| *** dpawar has joined #openstack-containers | 08:38 | |
| *** magicboiz has joined #openstack-containers | 08:40 | |
| *** oikiki has joined #openstack-containers | 08:40 | |
| *** oikiki has quit IRC | 08:51 | |
| yasemin | ykarel, hi | 08:56 |
| *** oikiki has joined #openstack-containers | 09:00 | |
| *** srihas619 has joined #openstack-containers | 09:04 | |
| *** wangbo has quit IRC | 09:09 | |
| *** wangbo has joined #openstack-containers | 09:11 | |
| *** yamamoto has joined #openstack-containers | 09:11 | |
| *** oikiki has quit IRC | 09:14 | |
| ykarel | yasemin, hi | 09:18 |
| *** yamamoto has quit IRC | 09:19 | |
| *** dsariel has joined #openstack-containers | 09:19 | |
| *** yamamoto has joined #openstack-containers | 09:23 | |
| *** AlexeyAbashkin has joined #openstack-containers | 09:38 | |
| *** KwozyMan has joined #openstack-containers | 09:43 | |
| slunkad | strigazi: hello | 09:59 |
| strigazi | slunkad: hi | 09:59 |
| slunkad | strigazi: I just wanted to let you know about the opensuse image we decided to build it with dib rather than spending time with kiwi first | 10:00 |
| slunkad | strigazi: https://review.openstack.org/#/c/520063/ | 10:00 |
| strigazi | slunkad: I saw the patch, is it passing? | 10:00 |
| slunkad | no not yet | 10:00 |
| strigazi | and locally on your dev environment? | 10:01 |
| *** lpetrut has joined #openstack-containers | 10:01 | |
| srihas | I have a doubt, why do we build images, cant we install / configure any image as per requirements via cloud-init ? | 10:02 |
| strigazi | srihas for fedora we don't build | 10:02 |
| slunkad | strigazi: I'm on it now | 10:03 |
| srihas | strigazi: I mean ubuntu, fedora, centOS etc can e chosen, right? | 10:03 |
| strigazi | slunkad: cool | 10:03 |
| strigazi | srihas: each driver in magnum is tied to an operating system | 10:03 |
| srihas | strigazi: yeah, thats what my doubt is, cant we have a generic driver, is there any downside? | 10:04 |
| strigazi | srihas: There is no generic option. If we have only one, we will have a bunch of conditions. eg each OS has different docker versions. | 10:07 |
| strigazi | srihas: cloud init works differently in coreos vs fedora | 10:07 |
| srihas | ok | 10:07 |
| srihas | seems clear for me :) | 10:08 |
| *** salmankhan has joined #openstack-containers | 10:08 | |
| strigazi | srihas: we could have a generic option that could leverage kubespray for example. But if there is no interest for that and someone willing to drive it we can't procceed. | 10:09 |
| srihas | strigazi: ok | 10:09 |
| *** ramishra has quit IRC | 10:13 | |
| yasemin | ykarel, i have a problem about kubernetes | 10:16 |
| strigazi | yasemin: what is the problem? | 10:17 |
| yasemin | ykarel, cloud-init-output.log -> http://paste.openstack.org/show/627036/ | 10:17 |
| yasemin | i followed this patch https://review.openstack.org/#/c/447687/ but it is not working | 10:17 |
| yasemin | <strigazi> | 10:17 |
| yasemin | will it merge ? | 10:18 |
| strigazi | can you do sh /var/lib/cloud/instance/scripts/part-005 ? | 10:18 |
| strigazi | it is working, I have tested it locally, you miss something, we can find out what | 10:18 |
| ykarel | yasemin, how you applied https://review.openstack.org/#/c/447687/ | 10:19 |
| ykarel | on your environment | 10:19 |
| ykarel | yasemin, from the logs you shared earlier it looks you have wrongly updated templates | 10:19 |
| strigazi | btw, kubernetes was already doing insecure requests | 10:19 |
| yasemin | <strigazi> i run sh command, but it is any output | 10:20 |
| yasemin | ykarel i updated magnum templates files | 10:21 |
| yasemin | ykarel, my enviroment is openstack-ansible ocata | 10:21 |
| yasemin | and https://review.openstack.org/#/c/447687/38/magnum/drivers/common/templates/swarm/fragments/write-cluster-failure-service.yaml | 10:22 |
| yasemin | is different -> ExecStart=/usr/bin/$WAIT_CUR part | 10:22 |
| yasemin | ExecStart=/usr/bin/curl -k -i -X POST -H 'Content-Type: application/json' -H 'X-Auth-Token: $WAIT_HANDLE_TOKEN' \ | 10:23 |
| yasemin | --data-binary '{"status": "FAILURE", "reason": "$SERVICE service failed to start.", "data": "Failure"}' \ | 10:23 |
| yasemin | "$WAIT_HANDLE_ENDPOINT" | 10:23 |
| yasemin | how can i change it ? | 10:23 |
| ykarel | the patch is for master, may be we need to consider something for ocata | 10:23 |
| strigazi | ykarel make everytihng insecure or backport 447687 ? | 10:24 |
| ykarel | yasemin, coming back to your question, issue is with kubernetes or swarm cluster | 10:24 |
| *** wangbo has quit IRC | 10:24 | |
| *** openstackgerrit has joined #openstack-containers | 10:24 | |
| openstackgerrit | Merged openstack/python-magnumclient master: OSC: Add --master-flavor to coe cluster create https://review.openstack.org/489623 | 10:24 |
| ykarel | strigazi, we can backport as it's a bug fix | 10:24 |
| strigazi | sounds good for me | 10:25 |
| yasemin | ykarel it is swarm files | 10:25 |
| yasemin | but i have problems both of them :D | 10:26 |
| ykarel | Ok if you see -k is there already so no change is required | 10:27 |
| ykarel | in ocata change is not required | 10:27 |
| ykarel | https://github.com/openstack/magnum/blob/stable/ocata/magnum/drivers/common/templates/swarm/fragments/write-cluster-failure-service.yaml#L14 | 10:27 |
| yasemin | oooo i changed -k other files :/ | 10:28 |
| ykarel | strigazi, did you checked there is some issue in devstack setup in jobs | 10:29 |
| srihas | yasemin: can you reach heat endpoint manually from inside the VM ? | 10:29 |
| ykarel | http://logs.openstack.org/87/447687/38/check/magnum-functional-api/9ce7e2a/logs/devstacklog.txt.gz#_2017-11-22_09_57_06_568 | 10:29 |
| yasemin | <srihas> how ? | 10:32 |
| srihas | yasemin: take the public endpoint from controller, ´openstack endpoint list´ | 10:32 |
| srihas | or ´openstack catalog list´ | 10:34 |
| srihas | login to your master node in the cluster; telnet <endpoint IP> <endpoint port> | 10:34 |
| srihas | generally port is 8004 | 10:35 |
| yasemin | ykarel strigazi how can i solve my problems :/ | 10:35 |
| yasemin | <srihas> my openstack enviroment installed with ansible | 10:35 |
| srihas | yasemin: its doesnot matter, just execute ´openstack catalog list´ | 10:36 |
| srihas | and check for public endpoint of heat | 10:36 |
| ykarel | yasemin, first of all revert the changes you applied from review: https://review.openstack.org/#/c/447687/, | 10:39 |
| ykarel | as you have made some wrong changes | 10:39 |
| ykarel | then we can fix drivers one by one | 10:39 |
| ykarel | if i remember correctly few days ago you shared issue with swarm and i shared two review: one large and one small with you, correct? | 10:40 |
| *** magicboiz has quit IRC | 10:45 | |
| *** salmankhan has quit IRC | 10:46 | |
| *** salmankhan has joined #openstack-containers | 10:48 | |
| yasemin | ykarel, we reverted all changes | 10:52 |
| yasemin | ykarel, we are ready :D | 10:53 |
| ykarel | yasemin, now we can start with swarm driver | 10:59 |
| ykarel | yasemin, could you apply following changes: https://review.openstack.org/#/c/518700/1/magnum/drivers/common/templates/swarm/fragments/make-cert.py | 11:00 |
| yasemin | ykarel i did it and now i am trying | 11:01 |
| ykarel | yasemin, ok | 11:01 |
| ykarel | yasemin, changed only the affected lines or complete file | 11:01 |
| yasemin | i added verify lines | 11:02 |
| yasemin | ykarel, it is working :D | 11:03 |
| ykarel | yasemin, Great | 11:05 |
| ykarel | yasemin now try k8s fedora | 11:07 |
| yasemin | ykarel magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh ? | 11:09 |
| ykarel | yasemin, it looks correct | 11:10 |
| ykarel | it has -k | 11:10 |
| *** wangbo has joined #openstack-containers | 11:10 | |
| yasemin | ykarel, USER_TOKEN=`curl -k -s -i -X POST -H "$content_type" -d "$auth_json" $url \ | 11:10 |
| yasemin | | grep X-Subject-Token | awk '{print $2}' | tr -d '[[:space:]]'` | 11:10 |
| yasemin | ? | 11:10 |
| ykarel | yasemin, yes it;s correct | 11:11 |
| *** magicboiz has joined #openstack-containers | 11:12 | |
| yasemin | ykarel, where i add verify=false ? or what can i do ? | 11:12 |
| *** eliqiao_ has quit IRC | 11:13 | |
| ykarel | yasemin, no change required | 11:13 |
| ykarel | it's already covered in fedora atomic | 11:14 |
| ykarel | just deploy feora-atomic and see how it goes | 11:14 |
| yasemin | ykarel, i am trying fedora atomic 25 | 11:18 |
| yasemin | ykarel, cloud-init-output.log -> http://paste.openstack.org/show/627059/ | 11:19 |
| ykarel | yasemin, from where you took the fedora-atomic image | 11:22 |
| yasemin | ykarel, i didnt remember | 11:22 |
| ykarel | can you try the latest one, i think atomic 25 has some issues | 11:23 |
| ykarel | yasemin, https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2 | 11:24 |
| *** aluria has joined #openstack-containers | 11:26 | |
| *** AlexeyAbashkin has quit IRC | 11:31 | |
| *** adisky_ has quit IRC | 11:32 | |
| *** AlexeyAbashkin has joined #openstack-containers | 11:34 | |
| srihas | cannot create cluster-config for a cluster launched with --tls-disabled, can someone help me with it? | 11:37 |
| strigazi | srihas: magnum --version | 11:38 |
| srihas | 2.3.0 | 11:39 |
| srihas | newton on ubuntu | 11:39 |
| *** dsariel has quit IRC | 11:40 | |
| strigazi | srihas: for a swarm cluster? | 11:40 |
| strigazi | srihas: I guess you need this patch https://review.openstack.org/#/c/425259/ | 11:42 |
| srihas | strigazi: for swarm, yes | 11:46 |
| srihas | let me see the patch | 11:46 |
| srihas | strigazi: I have the environment variables set | 11:47 |
| srihas | but the .pem files are not created in --dir | 11:47 |
| yasemin | ykarel, i cant create a new magnum cluster templates on horizon. | 11:48 |
| yasemin | ykarel, horizon error.log - | 11:49 |
| yasemin | http://paste.openstack.org/show/627065/ | 11:49 |
| strigazi | srihas with tls-disabled there are no pem files :) | 11:50 |
| srihas | strigazi: ofcourse, but if i try to launch a container its complaining | 11:51 |
| srihas | that there are no certs to connect to docker daemon | 11:52 |
| strigazi | can you login to the master and check it docker is started with cers? | 11:53 |
| strigazi | can you login to the master and check it docker is started with certs? | 11:53 |
| strigazi | systemctl cat docker | 11:54 |
| srihas | strigazi: http://paste.openstack.org/show/627067/ | 11:57 |
| srihas | I have only that info, nothing mentioned about TLS | 11:58 |
| ykarel | yasemin, while creating image you need to set os_distro | 12:00 |
| *** ianychoi has quit IRC | 12:00 | |
| yasemin | ykarel, we updated image metadata | 12:00 |
| *** ianychoi has joined #openstack-containers | 12:00 | |
| ykarel | yasemin, now can you create template? | 12:01 |
| *** dsariel has joined #openstack-containers | 12:01 | |
| strigazi | ykarel: devstack fix https://review.openstack.org/#/c/522077/3 | 12:01 |
| yasemin | ykarel, yes | 12:02 |
| yasemin | ykarel, creating kubernetes cluster | 12:02 |
| ykarel | yasemin, Ok | 12:02 |
| strigazi | srihas check /etc/sysconfig/docker | 12:02 |
| ykarel | strigazi, ack | 12:02 |
| * strigazi is going to a meeting but he will try to be online | 12:03 | |
| srihas | strigazi: there is docker cert path in /etc/sysconfig/docker and there are redhat certs in /etc/docker/certs.d/ | 12:06 |
| *** janki has quit IRC | 12:08 | |
| yasemin | ykarel, i cant connect swarm nodes with ssh | 12:12 |
| ykarel | srihas, what's the content in /etc/sysconfig/docker | 12:13 |
| strigazi | and ls /etc/docker | 12:13 |
| ykarel | yasemin, what's the error | 12:13 |
| *** ramishra has joined #openstack-containers | 12:14 | |
| srihas | ykarel / strigazi I need some time please. have to fix connectivity | 12:15 |
| ykarel | srihas, ok | 12:16 |
| yasemin | ykarel, i tried swarm cluster with fedora 26 image | 12:18 |
| yasemin | WaitConditionFailure: resources.swarm_masters.resources[0].resources.master_wait_condition: swarm-manager service failed to sta | 12:18 |
| yasemin | it gave error 2 minutes | 12:19 |
| *** yamamoto has quit IRC | 12:20 | |
| yasemin | ykarel but created successfully with fedora 25 about 4 minutes | 12:20 |
| ykarel | yasemin, login to master node and check status | 12:21 |
| yasemin | i cant login master node with ssh | 12:21 |
| ykarel | yasemin, what's the Error? | 12:21 |
| yasemin | ykarel, i dont know error about ssh connection | 12:24 |
| *** yamamoto has joined #openstack-containers | 12:24 | |
| srihas | yasemin: try ssh -vv | 12:25 |
| *** AlexeyAbashkin has quit IRC | 12:29 | |
| *** KwozyMan has quit IRC | 12:30 | |
| yasemin | ykarel, OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 | 12:31 |
| yasemin | debug1: Reading configuration data /etc/ssh/ssh_config | 12:31 |
| yasemin | debug1: /etc/ssh/ssh_config line 19: Applying options for * | 12:31 |
| yasemin | debug2: ssh_connect: needpriv 0 | 12:31 |
| yasemin | debug1: Connecting to 10.1.65.28 [10.1.65.28] port 22. | 12:31 |
| yasemin | debug1: connect to address 10.1.65.28 port 22: Connection timed out | 12:31 |
| yasemin | ssh: connect to host 10.1.65.28 port 22: Connection timed out | 12:31 |
| *** KwozyMan has joined #openstack-containers | 12:32 | |
| ykarel | yasemin, is swarm-worker node created? | 12:33 |
| *** AlexeyAbashkin has joined #openstack-containers | 12:44 | |
| *** mkuiper has joined #openstack-containers | 12:45 | |
| *** mdnadeem has quit IRC | 12:46 | |
| mkuiper | Hi I have cretaed a 1 master , 2 nodes kubernetes cluster. The template was created with: magnum cluster-template-create kubernetes-cluster-tpl3 --volume-driver cinder --keypair mkuix3 --fixed-network xxxxx --fixed-subnet yyyyyy --registry-enabled --image fedora-atomic-ocata --external-network floating-ip-pool-1 --dns-nameserver 169.254.169.7 --master-flavor 2C-4G-20G-V1-S --flavor 2C-4G-20G-V1-S --coe kubernetes --tls-d | 12:49 |
| srihas | ykarel: content in /etc/sysconfig/docker is http://paste.openstack.org/show/627072/ | 12:51 |
| srihas | strigazi: ls /etc/docker/ | 12:52 |
| srihas | certs.d key.json | 12:52 |
| mkuiper | the client apparently does not accept --floating-ip-enabled False so I set that field to 0 in the magnum cluster_template table | 12:52 |
| mkuiper | I then created the cluster with magnum cluster-create kubernetes-cluster --cluster-template kubernetes-cluster-tpl3 --master-count 1 --node-count 2 --keypair mkuix3 | 12:53 |
| mkuiper | The cluster comes up and I can run for example kubernetes-bootcamp. However the coredns and dashboard pod fail to start. If I look at the kubernetes logs for coredns I get : [root@ku-asmp4rp5id-0-iaxh4sk4saur-kube-master-ag3diqa2vmgw ~]# kubectl logs coredns-3034292617-pt0kh -n kube-system 2017/11/22 12:53:22 middleware/kubernetes: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory | 12:55 |
| yasemin | ykarel, yes just only created swarm master node | 12:55 |
| mkuiper | If I look at the docker logs I get: /go/src/github.com/docker/distribution/cmd/registry/main.go:24 +0x2d panic: No username parameter provided | 12:55 |
| mkuiper | magnum version is pike. ANyone got any pointers to get these pods in the running state | 12:56 |
| yasemin | ykarel, i tried fedora atomic latest , swarm cluster create successfully but not connect with ssh | 12:56 |
| ykarel | srihas, can you try restarting docker by comment DOCKER_CERT_PATH Line | 12:58 |
| yasemin | ykarel, i restarted swarm nodes (fedora 25), i can connect but docker engine not running | 12:58 |
| yasemin | ykarel, is it related SSL or image ? | 13:00 |
| ykarel | yasemin, what's the error for docker, systemctl status docker | 13:00 |
| yasemin | ykarel, http://paste.openstack.org/show/627077/ | 13:03 |
| ykarel | yasemin, ps -eaf|grep docker | 13:07 |
| *** dsariel has quit IRC | 13:07 | |
| yasemin | ykarel, root 812 1 0 12:56 ? 00:00:00 /usr/libexec/docker/docker-containerd-current --listen unix:///run/containerd.sock --shim /usr/libexec/docker/docker-containerd-shim-current --start-timeout 2m | 13:09 |
| yasemin | fedora 1287 1255 0 13:08 pts/0 00:00:00 grep --color=auto docker | 13:09 |
| ykarel | yasemin, strange error, i need to reproduce it to check | 13:11 |
| ykarel | yasemin, can you create a new cluster to see if you face SSH issue again | 13:11 |
| yasemin | ykarel, which image ? | 13:12 |
| ykarel | yasemin, you faced SSH issue on both images? | 13:12 |
| yasemin | ykarel, in fedora 26 cluster not created just only create master node | 13:13 |
| ykarel | yasemin, Ok first try 25 | 13:14 |
| yasemin | in fedora 25 cluster create but not connect ssh, after rebooting nodes i can connect | 13:14 |
| *** wangbo has quit IRC | 13:14 | |
| *** ramishra has quit IRC | 13:16 | |
| *** ramishra has joined #openstack-containers | 13:16 | |
| yasemin | ykarel, okey i try it | 13:16 |
| *** ykarel is now known as ykarel|afk | 13:18 | |
| *** dsariel has joined #openstack-containers | 13:21 | |
| *** yamamoto has quit IRC | 13:21 | |
| *** yamamoto has joined #openstack-containers | 13:22 | |
| *** janki has joined #openstack-containers | 13:26 | |
| *** vijaykc4 has joined #openstack-containers | 13:27 | |
| *** ykarel|afk is now known as ykarel | 13:37 | |
| srihas | ykarel: no success still! its looking for keys | 13:37 |
| srihas | is the problem with docker client? :/ | 13:38 |
| ykarel | srihas, from where are you running comands | 13:38 |
| *** dsariel has quit IRC | 13:38 | |
| srihas | controller node | 13:38 |
| ykarel | srihas, try from swarm master node | 13:39 |
| ykarel | it may be a client issue | 13:39 |
| *** magicboiz has quit IRC | 13:40 | |
| yasemin | ykarel, how can i reconfigure docker network cidr ? | 13:40 |
| srihas | ykarel: worked for "busybox container" but didnt work for "helloworld" | 13:40 |
| *** dave-mccowan has joined #openstack-containers | 13:41 | |
| *** mgoddard has quit IRC | 13:42 | |
| ykarel | yasemin, no idea | 13:42 |
| ykarel | srihas, what's the error | 13:42 |
| *** AlexeyAbashkin has quit IRC | 13:43 | |
| srihas | ykarel: Trying to pull repository docker.io/library/helloworld ... | 13:44 |
| *** dave-mcc_ has joined #openstack-containers | 13:44 | |
| srihas | /usr/bin/docker-current: unauthorized: authentication required. | 13:44 |
| ykarel | srihas, check netstat -tnlp | 13:45 |
| *** dave-mccowan has quit IRC | 13:45 | |
| srihas | I wonder how its working for other containers, let me check | 13:46 |
| srihas | ykarel: http://paste.openstack.org/show/627084/ | 13:47 |
| ykarel | srihas, sudo docker ps | 13:48 |
| srihas | ykarel: only the ubuntu container I have created. because busybox exits after executing echo | 13:49 |
| ykarel | srihas, magnum cluster-show <swarm cluster name> | 13:49 |
| *** AlexeyAbashkin has joined #openstack-containers | 13:50 | |
| ykarel | srihas, from netstat output it seems swarm-manager is not running | 13:51 |
| ykarel | can you also check systemctl status swarm-manager | 13:51 |
| srihas | ykarel: cluster-show http://paste.openstack.org/show/627086/ | 13:52 |
| srihas | ykarel: you are true, its in failed statie | 13:53 |
| ykarel | srihas, tcp://MASTER-IP:2376, port 2376 should be open on master node | 13:53 |
| ykarel | yes | 13:53 |
| srihas | yeah | 13:53 |
| srihas | noticed it | 13:53 |
| ykarel | so need to fix failure of swarm-manager for things to work | 13:55 |
| yasemin | ykarel, we solved ssh problem, docker is running but now swarm manager not running | 13:55 |
| ykarel | yasemin, what't the error | 13:55 |
| yasemin | ykarel, Dependency failed for Swarm Manager. | 13:56 |
| srihas | same error for me as well ^ | 13:56 |
| strigazi | mkuiper you are using a cluster with tls enabled right? | 13:57 |
| ykarel | yasemin, check if etcd is running in your env | 13:57 |
| ykarel | srihas, in your it is running, etcdctl ls | 13:58 |
| yasemin | ykarel, yes it is running | 13:58 |
| mkuiper | no I sepcifically added --tld-disabled | 13:58 |
| ykarel | yasemin, and docker? | 13:59 |
| strigazi | mkuiper service accounts do not work without tls. When tls is disabled we don't generate any CAs | 13:59 |
| mkuiper | sorry --tls-disabled as you can see from my cluster-template-create command | 13:59 |
| mkuiper | so what is dat parameter for then? | 14:00 |
| mkuiper | anyway I will setup a cluster with tls enabled then. thx | 14:01 |
| ykarel | yasemin, systemctl list-unit-files, check for failed services | 14:01 |
| *** salmankhan has quit IRC | 14:16 | |
| *** mgoddard has joined #openstack-containers | 14:22 | |
| mkuiper | strigazi, not getting any better. I changed template creation by removein --tls-disabled and added --docker-volume-size to match admin guide on secure cluster building. The commandline now looks like: magnum cluster-template-create kubernetes-cluster-tpl4 --network-driver flannel --volume-driver cinder --keypair mkuix3 --fixed-network xxxxxx --fixed-subnet yyyyyy --registry-enabled --image fedora-atomic-oca | 14:23 |
| mkuiper | WHen I now create a cluster with: magnum cluster-create kubernetes-cluster --cluster-template kubernetes-cluster-tpl4 --master-count 1 --node-count 2 --keypair mkuix3 | 14:24 |
| *** salmankhan has joined #openstack-containers | 14:24 | |
| mkuiper | it fails pretty fast. heat stack-show says: stack_status_reason | Resource CREATE failed: StackValidationFailed: resource | | | s.kube_masters.resources[0].resources.docker_volume: | 14:24 |
| mkuiper | mmm that does not paste very well ;) | 14:25 |
| strigazi | remove docker-volume-size | 14:25 |
| mkuiper | will do | 14:25 |
| strigazi | for paste use paste.openstack.org | 14:25 |
| mkuiper | ok | 14:25 |
| *** KwozyMan has quit IRC | 14:29 | |
| *** ramishra has quit IRC | 14:46 | |
| *** ramishra has joined #openstack-containers | 14:46 | |
| *** ykarel is now known as ykarel|away | 14:48 | |
| *** rcernin has joined #openstack-containers | 14:51 | |
| *** ramishra has quit IRC | 15:05 | |
| *** ramishra has joined #openstack-containers | 15:07 | |
| mkuiper | strigazi it is still building which tells me it will fail on a waitcondition. The master vm has booted but kube-apiserver is not running. | 15:08 |
| mkuiper | journal log says: kube-apiserver[5462]: F1122 15:03:57.997560 5462 universal_validation.go:104] Validate server run options failed: unable to load client CA file: error reading /etc/kubernetes/certs/ca.crt: could not read any certificates | 15:08 |
| mkuiper | /etc/kubernetes/certs/ca.crt is there but size is 0 | 15:09 |
| *** ykarel|away has quit IRC | 15:09 | |
| strigazi | mkuiper this is with or without tls | 15:09 |
| strigazi | ? | 15:09 |
| mkuiper | this is with tls. I now recall I ran into this the last time I tried, that's why I started using --tls-disabled | 15:10 |
| strigazi | mkuiper: can the nodes reach the api? | 15:10 |
| strigazi | mkuiper: can the nodes reach the openstack api? | 15:11 |
| mkuiper | the nodes will not be build | 15:11 |
| mkuiper | ah yes they can | 15:11 |
| mkuiper | with --tld-disabled the nodes get build and I can even schedule a pod. Only then dashboard and coredns fail to (re)start | 15:11 |
| strigazi | mkuiper: what is in /var/log/cloud-init-output? | 15:11 |
| strigazi | in the master node | 15:12 |
| mkuiper | hope I got his right?! http://paste.openstack.org/show/627098/ | 15:14 |
| strigazi | The script that sets the certs failed /var/lib/cloud/instance/scripts/part-005 | 15:15 |
| strigazi | I know part-005 is that script by heart in case you are wondering | 15:15 |
| mkuiper | hahaha I was | 15:16 |
| strigazi | You can see the line : 2017-11-22 14:34:14,555 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-005 [1] | 15:16 |
| *** rcernin has quit IRC | 15:18 | |
| mkuiper | the curl on https://magnum.ams1.cloud.ecg.so/v1/certificates/uuid fails with: | 15:22 |
| mkuiper | {"errors": [{"status": 500, "code": "server", "links": [], "title": "Remote error: BadRequest Invalid input for field 'identity/password/user/password': None is not of type 'string' (HTTP 400) (Request-ID: req-695c5c7b-8b08-4c48-94e6-fb455bc560e1)\n[u'", "detail": "Remote error: BadRequest Invalid input for field 'identity/password/user/password': None is not of type 'string' (HTTP 400) (Request-ID: req-695c5c7b-8b08-4c4 | 15:22 |
| *** salmankhan has quit IRC | 15:22 | |
| mkuiper | will have a look at the magnum logs | 15:22 |
| strigazi | mkuiper content of /var/lib/cloud/instance/scripts/part-005 and /etc/sysconfig/heat-params ? | 15:23 |
| *** salmankhan has joined #openstack-containers | 15:23 | |
| *** hongbin has joined #openstack-containers | 15:24 | |
| mkuiper | http://paste.openstack.org/show/627101/ and http://paste.openstack.org/show/627102/ | 15:29 |
| *** marst has joined #openstack-containers | 15:36 | |
| mkuiper | strigazi i pasted the requested info | 15:36 |
| strigazi | I don't think it is related but try to add this in your magnum.conf | 15:37 |
| strigazi | cluster_user_trust = false | 15:39 |
| strigazi | https://docs.openstack.org/magnum/latest/configuration/sample-config.html | 15:39 |
| strigazi | i the trust section | 15:39 |
| *** ramishra has quit IRC | 15:39 | |
| strigazi | in the trust section | 15:39 |
| mkuiper | will give that a try. | 15:41 |
| *** ykarel|away has joined #openstack-containers | 15:42 | |
| mkuiper | started a build again. What is the option supposed to do? | 15:47 |
| *** AlexeyAbashkin has quit IRC | 15:49 | |
| mkuiper | strigazi, same result. ca.crt is 0 bytes. I also see that some package upgrades fail due to yum not available on the system | 15:55 |
| mkuiper | don't know whether that is related | 15:55 |
| mkuiper | is there a specific fedoara-atomic image that you would advice to use? | 15:56 |
| *** magicboiz has joined #openstack-containers | 15:58 | |
| *** hishh has quit IRC | 15:59 | |
| strigazi | with pike? | 16:01 |
| *** vijaykc4 has quit IRC | 16:02 | |
| strigazi | mkuiper https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-26-20170723.0/CloudImages/x86_64/images/Fedora-Atomic-26-20170723.0.x86_64.qcow2 | 16:02 |
| *** vijaykc4 has joined #openstack-containers | 16:03 | |
| *** vijaykc4 has quit IRC | 16:03 | |
| *** mkuiper has quit IRC | 16:05 | |
| *** haint has quit IRC | 16:07 | |
| *** manheim has quit IRC | 16:07 | |
| *** magicboiz has quit IRC | 16:08 | |
| *** manheim has joined #openstack-containers | 16:11 | |
| strigazi | mkuiper check again cloud-init-output | 16:12 |
| strigazi | if the error is again on part005 try to reproduce all the step of that file | 16:12 |
| *** manheim has quit IRC | 16:13 | |
| *** mjura has quit IRC | 16:13 | |
| *** manheim has joined #openstack-containers | 16:13 | |
| *** janki has quit IRC | 16:13 | |
| *** manheim has quit IRC | 16:17 | |
| *** nguyentrihai has joined #openstack-containers | 16:20 | |
| *** magicboiz has joined #openstack-containers | 16:21 | |
| *** magicboiz has quit IRC | 16:26 | |
| *** magicboiz has joined #openstack-containers | 16:26 | |
| *** itlinux has joined #openstack-containers | 16:26 | |
| *** dpawar has quit IRC | 16:30 | |
| *** vijaykc4 has joined #openstack-containers | 16:31 | |
| *** manheim has joined #openstack-containers | 16:32 | |
| *** manheim has quit IRC | 16:37 | |
| *** ykarel|away has quit IRC | 16:42 | |
| *** manheim has joined #openstack-containers | 16:42 | |
| *** fragatin_ has quit IRC | 16:45 | |
| *** itlinux has quit IRC | 16:47 | |
| *** nguyentrihai has quit IRC | 16:52 | |
| *** dsariel has joined #openstack-containers | 17:00 | |
| *** itlinux has joined #openstack-containers | 17:02 | |
| *** salmankhan has quit IRC | 17:08 | |
| *** jberkus has joined #openstack-containers | 17:08 | |
| *** salmankhan has joined #openstack-containers | 17:10 | |
| *** srihas619 has quit IRC | 17:12 | |
| *** salmankhan has quit IRC | 17:14 | |
| *** fragatina has joined #openstack-containers | 17:15 | |
| *** dsariel has quit IRC | 17:15 | |
| *** itlinux has quit IRC | 17:15 | |
| *** jberkus has quit IRC | 17:17 | |
| *** fragatin_ has joined #openstack-containers | 17:17 | |
| *** pcaruana has quit IRC | 17:18 | |
| *** vijaykc4 has quit IRC | 17:20 | |
| *** fragatina has quit IRC | 17:22 | |
| *** salmankhan has joined #openstack-containers | 17:25 | |
| *** AlexeyAbashkin has joined #openstack-containers | 17:33 | |
| *** manheim has quit IRC | 17:34 | |
| *** dpawar has joined #openstack-containers | 17:35 | |
| *** dpawar has quit IRC | 17:36 | |
| *** dpawar has joined #openstack-containers | 17:36 | |
| *** AlexeyAbashkin has quit IRC | 17:38 | |
| *** absubram has quit IRC | 17:39 | |
| *** jberkus has joined #openstack-containers | 17:40 | |
| *** manheim has joined #openstack-containers | 17:45 | |
| *** itlinux has joined #openstack-containers | 17:48 | |
| *** yamamoto_ has joined #openstack-containers | 17:51 | |
| *** yamamoto has quit IRC | 17:55 | |
| *** vijaykc4 has joined #openstack-containers | 17:55 | |
| *** yamamoto has joined #openstack-containers | 17:57 | |
| *** lpetrut has quit IRC | 17:59 | |
| *** yamamoto_ has quit IRC | 17:59 | |
| *** mgoddard has quit IRC | 18:04 | |
| *** janonymous has quit IRC | 18:12 | |
| *** ricolin has quit IRC | 18:12 | |
| *** dpawar has quit IRC | 18:21 | |
| *** jberkus has quit IRC | 18:30 | |
| *** fragatin_ has quit IRC | 18:30 | |
| *** fragatina has joined #openstack-containers | 18:30 | |
| *** vijaykc4 has quit IRC | 18:32 | |
| *** vijaykc4 has joined #openstack-containers | 18:32 | |
| *** vijaykc4 has quit IRC | 18:33 | |
| *** vijaykc4 has joined #openstack-containers | 18:36 | |
| *** jberkus has joined #openstack-containers | 18:42 | |
| *** jberkus has quit IRC | 18:47 | |
| *** vijaykc4 has quit IRC | 18:51 | |
| *** vijaykc4 has joined #openstack-containers | 18:52 | |
| *** vijaykc4 has quit IRC | 18:54 | |
| *** vijaykc4 has joined #openstack-containers | 18:59 | |
| *** vijaykc4 has quit IRC | 19:00 | |
| openstackgerrit | Merged openstack/magnum master: Add verify_ca configuration parameter https://review.openstack.org/447687 | 19:15 |
| *** oikiki has joined #openstack-containers | 19:18 | |
| *** lpetrut has joined #openstack-containers | 19:19 | |
| *** mgoddard has joined #openstack-containers | 19:21 | |
| oikiki | strigazi: just saw my first patch was merged!!!!! | 19:23 |
| *** dsariel has joined #openstack-containers | 19:25 | |
| *** salmankhan has quit IRC | 19:33 | |
| *** jberkus has joined #openstack-containers | 19:34 | |
| *** flwang has quit IRC | 19:38 | |
| *** mgoddard has quit IRC | 19:49 | |
| *** vijaykc4 has joined #openstack-containers | 19:50 | |
| *** flwang has joined #openstack-containers | 19:51 | |
| *** salmankhan has joined #openstack-containers | 20:07 | |
| *** salmankhan has quit IRC | 20:11 | |
| *** jberkus has quit IRC | 20:19 | |
| *** itlinux has quit IRC | 20:25 | |
| *** oikiki has quit IRC | 20:31 | |
| *** vijaykc4 has quit IRC | 20:55 | |
| *** mgoddard has joined #openstack-containers | 21:06 | |
| *** linkmark has joined #openstack-containers | 21:13 | |
| *** jmlowe has joined #openstack-containers | 21:32 | |
| *** dave-mcc_ has quit IRC | 21:39 | |
| *** mgoddard has quit IRC | 21:44 | |
| *** threestrands has joined #openstack-containers | 21:45 | |
| *** threestrands has quit IRC | 21:45 | |
| *** threestrands has joined #openstack-containers | 21:45 | |
| *** fragatina has quit IRC | 21:45 | |
| *** jmlowe has quit IRC | 21:45 | |
| *** fragatina has joined #openstack-containers | 21:45 | |
| *** fragatina has quit IRC | 21:47 | |
| *** fragatina has joined #openstack-containers | 21:48 | |
| *** lpetrut has quit IRC | 21:57 | |
| *** AlexeyAbashkin has joined #openstack-containers | 22:09 | |
| *** Dinesh_Bhor has quit IRC | 22:11 | |
| *** AlexeyAbashkin has quit IRC | 22:13 | |
| *** rcernin has joined #openstack-containers | 22:20 | |
| *** fragatina has quit IRC | 22:27 | |
| *** fragatina has joined #openstack-containers | 22:29 | |
| *** jmlowe has joined #openstack-containers | 22:32 | |
| *** fragatin_ has joined #openstack-containers | 22:36 | |
| *** fragatin_ has quit IRC | 22:37 | |
| *** fragatina has quit IRC | 22:37 | |
| *** fragatina has joined #openstack-containers | 22:38 | |
| *** marst has quit IRC | 22:46 | |
| *** dsariel has quit IRC | 22:54 | |
| *** AlexeyAbashkin has joined #openstack-containers | 23:08 | |
| *** AlexeyAbashkin has quit IRC | 23:12 | |
| *** manheim has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!