| *** hongbin has joined #openstack-zun | 02:13 | |
| openstackgerrit | Hongbin Lu proposed openstack/python-zunclient master: Handle the case that a generic api version https://review.openstack.org/616064 | 02:33 |
|---|---|---|
| openstackgerrit | Hongbin Lu proposed openstack/python-zunclient master: Add documentation on the python API https://review.openstack.org/616069 | 02:33 |
| openstackgerrit | Hongbin Lu proposed openstack/zun master: Fix race condition on multinode https://review.openstack.org/615444 | 02:47 |
| openstackgerrit | Hongbin Lu proposed openstack/zun master: [WIP] Implement delete network https://review.openstack.org/615402 | 02:47 |
| openstackgerrit | Merged openstack/python-zunclient master: Fix help message for action-list https://review.openstack.org/616356 | 02:59 |
| *** janki has joined #openstack-zun | 04:36 | |
| *** hongbin has quit IRC | 06:02 | |
| openstackgerrit | Merged openstack/python-zunclient master: Remove unused config for devstack job https://review.openstack.org/615390 | 06:44 |
| openstackgerrit | Merged openstack/python-zunclient stable/rocky: Handle different endpoint format https://review.openstack.org/614690 | 06:49 |
| openstackgerrit | Merged openstack/zun master: Set capsule.host once it is running https://review.openstack.org/615401 | 07:16 |
| openstackgerrit | Merged openstack/zun master: Use template for lower constraints jobs https://review.openstack.org/611480 | 07:16 |
| openstackgerrit | Merged openstack/zun master: Add a job to publish release notes https://review.openstack.org/611481 | 07:16 |
| openstackgerrit | Merged openstack/zun master: Add job for checking the test coverage https://review.openstack.org/611482 | 07:16 |
| *** janki has quit IRC | 12:20 | |
| *** janki has joined #openstack-zun | 13:29 | |
| *** irclogbot_3 has quit IRC | 14:02 | |
| *** janki has quit IRC | 14:30 | |
| *** hongbin has joined #openstack-zun | 14:55 | |
| Gaasmann | Is there any restriction or configuration for attaching volumes in Zun? I'm trying to create a container with a volume attached, the volume got created in cinder but the container creation is stuck with status "Creating" forever. | 16:39 |
| Gaasmann | We're using cinder backed by Ceph | 16:39 |
| *** nicolasbock_ has joined #openstack-zun | 17:09 | |
| *** irclogbot_3 has joined #openstack-zun | 18:00 | |
| hongbin | Gaasmann: as far as i know, there should not be any restriction | 18:04 |
| Gaasmann | I see a lot of commit about volumes between rocky and master, I'll try with master to see if it resolves the issue | 18:05 |
| hongbin | Gaasmann: if the container creation stuck at "Creating", there is possibly something printed on the log? | 18:05 |
| Gaasmann | Nothing useful http://paste.openstack.org/show/734438/ | 18:07 |
| hongbin | Gaasmann: i see, you are using zun with kolla | 18:08 |
| hongbin | Gaasmann: i am not sure if the cinder feature will work with kolla properly, i will give it a try later | 18:09 |
| Gaasmann | thanks | 18:10 |
| hongbin | let me create a bug to track this issue and get back to it after the summit | 18:11 |
| Gaasmann | sure | 18:12 |
| *** yevi has joined #openstack-zun | 18:22 | |
| egonzalez | hongbin iirc there was some volume missing in zun_compute containers to allow attach volumes (in kolla side) | 18:32 |
| hongbin | egonzalez: i see | 19:05 |
| hongbin | egonzalez: so enhancement needed to be done in kolla side? | 19:06 |
| egonzalez | yep, i think you have a patch in progress for that | 19:07 |
| hongbin | i don't remember i have any patch about volume, but let me double check | 19:08 |
| hongbin | couldn't search any patch about volume | 19:09 |
| hongbin | but i am on it | 19:09 |
| yevi | I am Using version 1.0.1 I can spin up containers in the ui and cli. I can attach to an interactive session with cli, but I can not connect via zun-wsproxy through zun-ui. Debug for zun-api: http://paste.openstack.org/show/734442/ | 21:22 |
| yevi | I assume I missed a config option somewhere, but not sure what. wsproxy shows - proxying from 10.10.5.80:6784 to None:None; which is why I assume it is refusing the connection | 21:24 |
| yevi | Thoughts? | 21:25 |
| *** nicolasbock_ has quit IRC | 21:32 | |
| hongbin | yevi: pong | 22:08 |
| hongbin | yevi: which version you were using? (i.e. master, rocky, queens) | 22:10 |
| hongbin | sorry, let me check 1.0.1 | 22:11 |
| yevi | queens | 22:12 |
| hongbin | yevi: i see, first, i would suggest to check this option: [websocket_proxy] base_url | 22:14 |
| hongbin | yevi: here is the document: https://docs.openstack.org/zun/queens/install/compute-install-ubuntu.html | 22:14 |
| hongbin | yevi: so, make sure the base_url is accessible from end-users (zun-ui users), which hopefully could fix the problem | 22:15 |
| yevi | it is set with wss:// to hit the controller node over port 6784 | 22:15 |
| yevi | compute has that | 22:16 |
| hongbin | yevi: the url should allow end-users to hit the compute node (docker daemon) directly | 22:16 |
| yevi | ok, so the wsproxy and docker should be on the same node? | 22:17 |
| yevi | or is there a way to tell wsproxy where the docker daemon resides? | 22:18 |
| hongbin | yevi: no necessary, wsproxy could be deployed on the controller node | 22:18 |
| hongbin | yevi: sorry, let me clarify (hope i can explain everything clearly) | 22:19 |
| yevi | I have the docker_remote_api_url pointing back to the docker daemon over port 2375 | 22:19 |
| hongbin | yevi: there are two things to check, 1. controller node's wsproxy config, 2. compute node's config | 22:20 |
| hongbin | yevi: for controller, check [websocket_proxy] wsproxy_host , make sure the host is accessible directly from end-users to controller node (wsproxy) | 22:21 |
| yevi | ok | 22:22 |
| hongbin | yevi: for compute, check [websocket_proxy] base_url , make sure the base url is accessible directly from end-users to compute node (docker daemon) | 22:22 |
| hongbin | the compute node config will be removed in the latest version for security reasons, but in queens, it is still there | 22:22 |
| hongbin | yevi: if both configs are ok but the zun-ui still doesn't work, please feel to ping me again | 22:25 |
| yevi | ok, I can Ok, I am verifying real quick, but I can connect to either seperatly, having troubles connecting to compute through wsproxy | 22:25 |
| yevi | will give you a sanitized paste of the two configs real quick | 22:25 |
| hongbin | ok | 22:26 |
| hongbin | yevi: the wsproxy is using this url to connect to compute: https://github.com/openstack/zun/blob/stable/queens/zun/container/docker/driver.py#L657 | 22:36 |
| hongbin | yevi: so, one more configs to check, in compute node, check [docker] docker_remote_api_host | 22:37 |
| yevi | compute zun.conf: http://paste.openstack.org/show/734448/ | 22:37 |
| yevi | controller zun.conf: http://paste.openstack.org/show/734449/ | 22:37 |
| * hongbin is looking | 22:39 | |
| yevi | so if I read that right on my controller I need to set remote_api_host and remote_api_port to the docker daemon sitting on my compute node. then the url will match. except that I am requesting 'wss://' not 'ws://' | 22:42 |
| yevi | I have set that before, but will test it real quick | 22:42 |
| yevi | Still have the same problem setting those specifically on the controller | 22:47 |
| yevi | I can confirm that it works on the compute node as I can attach interactive to a container with openstack cli | 22:48 |
| hongbin | i see | 22:49 |
| hongbin | but it cannot work on teh controller node if you attach interactive to the contiainer with openstack cli? | 22:50 |
| yevi | it wont work in the zun-ui | 22:52 |
| yevi | wich goes through zun-wsproxy to connect | 22:52 |
| yevi | I guess I could state that cloud shell behaves the same way | 22:55 |
| hongbin | ok | 22:55 |
| hongbin | yevi: if you check the log of zun-wsproxy, does you see anything (for example, any error)? | 22:56 |
| yevi | no errors, even with debug on only this | 22:57 |
| yevi | Nov 08 17:44:43 zun-0 zun-wsproxy[13898]: 2018-11-08 17:44:43.603 13898 INFO zun.websocket.websocketproxy [-] - Listen on 10.10.5.80:6784 | 22:57 |
| yevi | Nov 08 17:44:43 zun-0 zun-wsproxy[13898]: 2018-11-08 17:44:43.604 13898 INFO zun.websocket.websocketproxy [-] - SSL/TLS support | 22:57 |
| yevi | Nov 08 17:44:43 zun-0 zun-wsproxy[13898]: 2018-11-08 17:44:43.604 13898 INFO zun.websocket.websocketproxy [-] - proxying from 10.10.5.80:6784 to None:None | 22:57 |
| hongbin | yevi: then, if you follow the log of docker daemon, does you see the request is hitting there? | 22:58 |
| yevi | no, I did a tcpdump on both nodes on the controller node I see a syn, then reset response. On the compute node it never hits | 23:00 |
| yevi | wsproxy is refusing the connection from what I can see | 23:01 |
| yevi | verified I have no iptables rules blocking the request | 23:01 |
| hongbin | it have the same problem if you switch from 'wss' to 'ws'? | 23:02 |
| yevi | I get ssl errors if I switch to ws because the way the rest of the environment is set up. | 23:04 |
| hongbin | ok | 23:04 |
| yevi | I can see if I can bypass that, but that may take me a while | 23:05 |
| hongbin | no worry about it for now | 23:05 |
| yevi | ok | 23:05 |
| hongbin | but you can attach the container with openstack cli in compute node, right? | 23:06 |
| yevi | yes | 23:06 |
| hongbin | what are the exact command you ran? | 23:06 |
| yevi | openstack appcontainer run --name mycontainer --net network=netuuid cirros ping 8.8.8.8 | 23:09 |
| yevi | openstack appcontainer exec --interactive mycontainer /bin/sh | 23:09 |
| hongbin | ok, let try this | 23:09 |
| hongbin | openstack --debug appcontainer run --name mycontainer --net network=netuuid cirros ping 8.8.8.8 | 23:10 |
| hongbin | then, paste me the output please | 23:10 |
| hongbin | wait | 23:10 |
| hongbin | you are trying to exec in the container | 23:11 |
| hongbin | but the zun-ui won't exec | 23:11 |
| yevi | yes | 23:11 |
| hongbin | so if you do this: openstack appcontainer run -i --name mycontainer --net network=netuuid cirros /bin/sh | 23:11 |
| hongbin | it will work? | 23:12 |
| hongbin | or openstack appcontainer run --interactive --name mycontainer --net network=netuuid cirros /bin/sh | 23:12 |
| yevi | that actually does not work | 23:13 |
| hongbin | ok, that explains everything | 23:14 |
| hongbin | zun-ui is doing something that is equivalent so it didn't work as well | 23:14 |
| hongbin | then, could you run that command with the --debug option | 23:15 |
| hongbin | openstack --debug appcontainer run --interactive --name mycontainer --net network=netuuid cirros /bin/sh | 23:15 |
| yevi | ok | 23:16 |
| hongbin | brb | 23:24 |
| yevi | http://paste.openstack.org/show/734450/ | 23:29 |
| * hongbin is looking | 23:30 | |
| hongbin | it looks there is a bug on handling the 'wss' cases | 23:34 |
| yevi | ok | 23:35 |
| hongbin | so, the openstack cli doesn't hit the wsproxy at all, since it is using 'wss' but openstack cli thought it is a wrong protocol | 23:35 |
| hongbin | if you can do a hack on /usr/local/lib/python2.7/dist-packages/zunclient/common/websocketclient/websocketclient.py | 23:36 |
| yevi | I could | 23:36 |
| yevi | looking at it now | 23:38 |
| hongbin | this is the original code: http://paste.openstack.org/show/734451/ | 23:39 |
| hongbin | and you can change it to: http://paste.openstack.org/show/734452/ | 23:39 |
| hongbin | then, give it another try | 23:39 |
| yevi | it ran successfully that time, but didn't drop to a shell, give ma a moment to check some logs | 23:44 |
| hongbin | sure | 23:45 |
| yevi | http://paste.openstack.org/show/734453/ | 23:49 |
| hongbin | in the controller node, does the log of zun-wsproxy capture something? | 23:50 |
| yevi | no, nothing has hit that log | 23:50 |
| yevi | last entry was about 5 hours ago | 23:51 |
| hongbin | ok | 23:52 |
| hongbin | let me look at your config again | 23:52 |
| hongbin | i guess you can switch the url from 'wss' to 'ws' without touching anything else (just change that config in controller node) | 23:54 |
| hongbin | if you can do that, then maybe we can give another try | 23:54 |
| yevi | ok | 23:55 |
| hongbin | fwiw, here is the change: http://paste.openstack.org/compare/734455/734448/ | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!