| masber | strigazi, https://pastebin.com/raw/qzygU89X | 00:04 |
|---|---|---|
| masber | strigazi, https://pastebin.com/raw/qzygU89X | 00:14 |
| *** dave-mccowan has joined #openstack-containers | 00:17 | |
| *** jmlowe has joined #openstack-containers | 00:19 | |
| *** catintheroof has quit IRC | 00:55 | |
| *** dave-mccowan has quit IRC | 01:03 | |
| *** shu-mutou has joined #openstack-containers | 01:36 | |
| *** hongbin has joined #openstack-containers | 02:04 | |
| *** zhugaoxiao has quit IRC | 02:52 | |
| *** zhugaoxiao has joined #openstack-containers | 02:52 | |
| *** EricGonczer_ has quit IRC | 03:01 | |
| *** ramishra has joined #openstack-containers | 03:31 | |
| *** yamamoto has joined #openstack-containers | 03:33 | |
| *** cNilesh has joined #openstack-containers | 03:36 | |
| *** mdnadeem has joined #openstack-containers | 03:44 | |
| *** mdnadeem has quit IRC | 03:46 | |
| *** mdnadeem has joined #openstack-containers | 03:46 | |
| *** yee37925 has joined #openstack-containers | 03:47 | |
| *** yee379 has quit IRC | 03:47 | |
| *** mdnadeem has quit IRC | 03:48 | |
| *** mdnadeem has joined #openstack-containers | 03:48 | |
| *** catintheroof has joined #openstack-containers | 03:48 | |
| *** hongbin has quit IRC | 03:54 | |
| *** chhavi has joined #openstack-containers | 03:55 | |
| *** catintheroof has quit IRC | 03:55 | |
| *** chhavi_ has joined #openstack-containers | 04:24 | |
| *** chhavi has quit IRC | 04:26 | |
| *** cNilesh has quit IRC | 04:33 | |
| *** cNilesh has joined #openstack-containers | 04:33 | |
| *** trinaths has joined #openstack-containers | 04:42 | |
| *** cNilesh has quit IRC | 04:42 | |
| *** armaan has joined #openstack-containers | 04:43 | |
| *** cNilesh has joined #openstack-containers | 04:43 | |
| *** trinaths has quit IRC | 04:46 | |
| *** ykarel has quit IRC | 04:47 | |
| *** trinaths has joined #openstack-containers | 04:49 | |
| *** cNilesh has quit IRC | 05:05 | |
| *** cNilesh has joined #openstack-containers | 05:06 | |
| *** ykarel has joined #openstack-containers | 05:14 | |
| *** mdnadeem has quit IRC | 05:26 | |
| *** mdnadeem has joined #openstack-containers | 05:26 | |
| *** mjura has joined #openstack-containers | 05:50 | |
| *** tonanhngo has joined #openstack-containers | 05:51 | |
| *** rcernin has joined #openstack-containers | 06:04 | |
| *** liangbo has joined #openstack-containers | 06:10 | |
| *** Oku_OS-away is now known as Oku_OS | 06:11 | |
| *** rcernin has quit IRC | 06:31 | |
| *** rcernin has joined #openstack-containers | 06:34 | |
| *** rcernin has quit IRC | 06:34 | |
| *** rcernin has joined #openstack-containers | 06:34 | |
| *** armaan has quit IRC | 06:37 | |
| *** armaan has joined #openstack-containers | 06:38 | |
| *** tonanhngo has quit IRC | 06:47 | |
| strigazi | masber I'm pretty sure that the master is not deployed correctly. | 06:48 |
| strigazi | masber In the master template we have a wait condition handle that is satisfied when kubernetes' API is reachable | 06:48 |
| *** armaan has quit IRC | 06:49 | |
| *** armaan has joined #openstack-containers | 06:49 | |
| strigazi | When you create a cluster you need to go inside the master (check with nova list or heat resource-list -n 2 <your stack> | grep COMPLETE yo see when the master is Complete) | 06:49 |
| strigazi | From inside the master do curl "http://127.0.0.1:8080/healthz" | 06:51 |
| masber | strigazi, one thing is that the ip is 10.0.0.X while my internal network is 10.1.0.X | 06:51 |
| masber | ok | 06:51 |
| strigazi | masber If the master is fine and you notice that the minion node is running in nova | 06:52 |
| strigazi | You need to check if the minion can reach the master | 06:52 |
| masber | so then I changed the cluster template to use my private subnet using the --fixed-subnet parameter | 06:52 |
| masber | minions are not deployed | 06:52 |
| masber | only the master | 06:52 |
| masber | ok, let me login to the master and see if I can reach internet | 06:53 |
| strigazi | private subnet and private network | 06:53 |
| masber | sorry, reach the http://127.0.0.1:8080/healthz I mean | 06:53 |
| strigazi | sure | 06:53 |
| masber | [fedora@k8-n2xwljdyco-0-anynm5yixaog-kube-master-jepnlcex6mcy ~]$ curl "http://127.0.0.1:8080/healthz" | 06:55 |
| masber | curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused | 06:55 |
| masber | sorry I should have paste it | 06:55 |
| *** pcaruana has joined #openstack-containers | 06:55 | |
| masber | somewhere else | 06:55 |
| strigazi | journalctl -u kube-apiserver --no-pager | 06:56 |
| openstackgerrit | fengbeihong proposed openstack/magnum master: Push the DC/OS driver in magnum/drivers and add unit test https://review.openstack.org/475682 | 06:56 |
| masber | https://pastebin.com/raw/eEC0c7fc | 06:58 |
| strigazi | masber this is very clear | 06:58 |
| strigazi | ls -l /srv/kubernetes | 06:58 |
| strigazi | and cat /var/log/cloud-init-output.log | 06:59 |
| strigazi | I should all this in the troubleshooting guide... | 06:59 |
| masber | https://pastebin.com/raw/TcM1jsjY | 07:00 |
| strigazi | cat /var/lib/cloud/instance/scripts/part-005 | 07:01 |
| masber | permission denied | 07:01 |
| masber | cat: /var/lib/cloud/instance/scripts/part-005: Permission denied | 07:01 |
| strigazi | sudo !! | 07:02 |
| masber | https://pastebin.com/raw/vD6UdjMF | 07:03 |
| *** cNilesh has quit IRC | 07:03 | |
| strigazi | You can't reach magnum's url or keystone's | 07:04 |
| *** cNilesh has joined #openstack-containers | 07:04 | |
| strigazi | sudo cat /etc/syscofig/heat-params | 07:04 |
| *** dsariel has joined #openstack-containers | 07:06 | |
| masber | 1 sec, I am looking for that file | 07:07 |
| strigazi | sudo cat /etc/sysconfig/heat-params | 07:07 |
| strigazi | I had a typo | 07:07 |
| masber | https://pastebin.com/raw/WwuR2yxj | 07:08 |
| strigazi | masber We found the problem, I don't think that you will be able to curl http://10.1.0.5:9511/v1 | 07:10 |
| strigazi | or curl http://10.1.0.5:5000/v3 | 07:10 |
| masber | you are correct, I can't curl http://10.1.0.5:9511/v1, it just hangs | 07:10 |
| strigazi | or curl http://10.1.0.5:8004 | 07:10 |
| masber | curl http://10.1.0.5:5000/v3 hangs too | 07:11 |
| strigazi | Try a cluster with tls-disabled | 07:11 |
| strigazi | and | 07:11 |
| strigazi | In the node that you run magnum do: magnum-api --version I don't remember if you told me already | 07:12 |
| strigazi | Please tell me the version first I need to go to a meeting in 3' | 07:12 |
| masber | [root@openstack-controller ~]# docker exec magnum_api magnum-api --version | 07:13 |
| masber | 4.1.0 | 07:13 |
| strigazi | I thought so, We have done a point release to pass the public endpoint in the cluster | 07:14 |
| strigazi | masber in 4.1.2 Should be solved | 07:15 |
| strigazi | Try upgrading | 07:16 |
| * strigazi is going to a meeting | 07:16 | |
| *** cNilesh has quit IRC | 07:23 | |
| *** cNilesh has joined #openstack-containers | 07:23 | |
| masber | ok, I will try thanks! | 07:24 |
| *** dsariel has quit IRC | 07:41 | |
| *** mgoddard_ has joined #openstack-containers | 07:47 | |
| *** armaan has quit IRC | 07:48 | |
| *** armaan has joined #openstack-containers | 07:49 | |
| *** armaan has quit IRC | 07:53 | |
| *** armaan has joined #openstack-containers | 07:58 | |
| *** ricolin_ has quit IRC | 08:04 | |
| *** ricolin has joined #openstack-containers | 08:05 | |
| *** banshee__ has joined #openstack-containers | 08:07 | |
| *** armaan has quit IRC | 08:25 | |
| *** armaan has joined #openstack-containers | 08:26 | |
| *** sidx64 has joined #openstack-containers | 08:41 | |
| *** mdnadeem has quit IRC | 08:42 | |
| *** mdnadeem has joined #openstack-containers | 08:43 | |
| *** yamamoto has quit IRC | 08:44 | |
| *** dsariel has joined #openstack-containers | 08:45 | |
| *** armaan has quit IRC | 08:58 | |
| * strigazi is back | 09:08 | |
| *** deu_ has quit IRC | 09:09 | |
| *** banshee__ has quit IRC | 09:10 | |
| *** armaan has joined #openstack-containers | 09:14 | |
| *** deu_ has joined #openstack-containers | 09:16 | |
| *** ykarel_ has joined #openstack-containers | 09:28 | |
| *** ykarel has quit IRC | 09:31 | |
| *** yamamoto has joined #openstack-containers | 09:36 | |
| openstackgerrit | Rico Lin proposed openstack/magnum master: Add heat container agent for update hosts file https://review.openstack.org/480101 | 09:48 |
| *** ykarel__ has joined #openstack-containers | 09:52 | |
| *** liangbo has quit IRC | 09:54 | |
| *** ykarel_ has quit IRC | 09:55 | |
| *** clenimar has joined #openstack-containers | 10:01 | |
| *** livelace has joined #openstack-containers | 10:03 | |
| *** salmankhan has joined #openstack-containers | 10:03 | |
| openstackgerrit | kavitha h r proposed openstack/magnum master: Drop oslosphinx from test-requirements.txt https://review.openstack.org/480114 | 10:05 |
| *** salmankhan1 has joined #openstack-containers | 10:06 | |
| *** salmankhan has quit IRC | 10:07 | |
| *** salmankhan1 has quit IRC | 10:10 | |
| *** salmankhan has joined #openstack-containers | 10:10 | |
| *** yamamoto has quit IRC | 10:12 | |
| *** yamamoto has joined #openstack-containers | 10:14 | |
| *** yamamoto has quit IRC | 10:14 | |
| *** yamamoto has joined #openstack-containers | 10:17 | |
| openstackgerrit | Kiran Totad proposed openstack/python-magnumclient master: switch to openstackdocstheme https://review.openstack.org/480123 | 10:19 |
| *** yamamoto has quit IRC | 10:21 | |
| *** trinaths has left #openstack-containers | 10:56 | |
| *** salmankhan has quit IRC | 11:02 | |
| *** salmankhan has joined #openstack-containers | 11:02 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 11:03 | |
| *** yamamoto has joined #openstack-containers | 11:19 | |
| *** yamamoto has quit IRC | 11:25 | |
| *** yamamoto has joined #openstack-containers | 11:25 | |
| *** salmankhan has quit IRC | 11:34 | |
| *** salmankhan has joined #openstack-containers | 11:39 | |
| *** hieulq_ has joined #openstack-containers | 11:53 | |
| *** hieulq_ has quit IRC | 12:09 | |
| *** hieulq_ has joined #openstack-containers | 12:10 | |
| *** ykarel_ has joined #openstack-containers | 12:11 | |
| *** ykarel__ has quit IRC | 12:13 | |
| *** ykarel__ has joined #openstack-containers | 12:14 | |
| *** ykarel_ has quit IRC | 12:17 | |
| *** EricGonczer_ has joined #openstack-containers | 12:20 | |
| *** hieulq_ has quit IRC | 12:23 | |
| strigazi | mkrai ykarel__ Can you have a look to the swarm-mode driver? https://review.openstack.org/#/c/427761/ | 12:30 |
| *** mdnadeem has quit IRC | 12:35 | |
| *** yamamoto has quit IRC | 12:36 | |
| *** EricGonc_ has joined #openstack-containers | 12:50 | |
| *** sidx64 has quit IRC | 12:50 | |
| *** sidx64 has joined #openstack-containers | 12:52 | |
| *** yamamoto has joined #openstack-containers | 12:52 | |
| *** EricGonczer_ has quit IRC | 12:53 | |
| *** salmankhan1 has joined #openstack-containers | 12:57 | |
| *** pcaruana has quit IRC | 12:59 | |
| *** salmankhan has quit IRC | 12:59 | |
| *** salmankhan1 has quit IRC | 13:00 | |
| armaan | hello folks, does magnum supports Kubernetes 1.6 ? | 13:07 |
| *** mkrai has quit IRC | 13:20 | |
| *** ykarel__ has quit IRC | 13:25 | |
| *** EricGonc_ has quit IRC | 13:25 | |
| mgoddard_ | hi rochaport | 13:29 |
| *** askb has quit IRC | 13:29 | |
| mgoddard_ | sorry, rochaporto | 13:29 |
| *** sidx64 has quit IRC | 13:32 | |
| *** salmankhan has joined #openstack-containers | 13:33 | |
| *** pcaruana has joined #openstack-containers | 13:36 | |
| *** kevinz has joined #openstack-containers | 13:39 | |
| *** kbyrne has quit IRC | 13:39 | |
| kevinz | Hi all | 13:39 |
| *** catintheroof has joined #openstack-containers | 13:39 | |
| kevinz | Does magnum plan to host a room in PTG Denver? | 13:40 |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: Allow labels on cluster create https://review.openstack.org/473346 | 13:41 |
| *** pcaruana has quit IRC | 13:45 | |
| *** pcaruana has joined #openstack-containers | 13:50 | |
| *** mkrai has joined #openstack-containers | 13:52 | |
| *** Oku_OS is now known as Oku_OS-away | 13:52 | |
| strigazi | kevinz no | 13:56 |
| *** ykarel has joined #openstack-containers | 13:56 | |
| strigazi | armaan with coreos but on the master branch | 13:56 |
| *** kbyrne has joined #openstack-containers | 13:58 | |
| *** mkrai has quit IRC | 14:00 | |
| ykarel | strigazi, Ok will look at swarm mode tomorrow | 14:05 |
| strigazi | ykarel Thanks | 14:05 |
| *** rochapor1o has joined #openstack-containers | 14:05 | |
| *** cNilesh has quit IRC | 14:06 | |
| rochaporto | hi mgoddard_ | 14:06 |
| *** mkrai has joined #openstack-containers | 14:07 | |
| *** cNilesh has joined #openstack-containers | 14:07 | |
| *** cNilesh has quit IRC | 14:08 | |
| mgoddard_ | hi rochaporto. I'm using k8s/fedora/ironic driver, seeing some issues with docker_volume_size - the cluster doesn't seem to be picking up the cluster template value | 14:09 |
| rochaporto | i don't know much about the ironic driver... maybe strigazi? | 14:10 |
| mgoddard_ | well I think it's more to do with this https://github.com/openstack/magnum/commit/79039bb4198b08500f5d5738b464d187f494a73a | 14:11 |
| rochaporto | ah ok. you're passing the value on cluster-create? | 14:11 |
| *** hongbin has joined #openstack-containers | 14:12 | |
| rochaporto | might be the same issue i just had with labels. testing against None instead of wtypes.Unset | 14:12 |
| armaan | strigazi: ahh, okk. thanks for letting me know | 14:13 |
| strigazi | mgoddard_ if you do heat stack-show you don't see docker_voume_size ? | 14:13 |
| strigazi | For ironic also have a look to: https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/fragments/configure-docker-storage.sh#L6 | 14:14 |
| mgoddard_ | I have a cluster template with docker_volume_size=3, but when I create a cluster from it the cluster's docker_volume_size=NULL | 14:14 |
| mgoddard_ | If I understand your change correctly, my cluster should default to a docker_volume_size of 3 | 14:15 |
| rochaporto | yes it should | 14:15 |
| rochaporto | i'll double check here | 14:15 |
| strigazi | fyi if it works for VM it must work for BM they share the same class | 14:16 |
| *** yamamoto has quit IRC | 14:17 | |
| mgoddard_ | ah, I think I've got it | 14:17 |
| *** kbyrne has quit IRC | 14:18 | |
| mgoddard_ | yep, got it. The comparison with None in api/controllers/v1/cluster.py should be against wsme.Unset | 14:18 |
| mgoddard_ | I'll file a short bug and push a patch for it | 14:19 |
| strigazi | mgoddard_ on my testing env it does pick it up | 14:23 |
| mgoddard_ | strigazi: that's odd. I just raised https://bugs.launchpad.net/magnum/+bug/1702314 if that helps to clarify the issue | 14:24 |
| openstack | Launchpad bug 1702314 in Magnum "Cluster does not inherit template's docker_volume_size" [Undecided,New] | 14:24 |
| *** yamamoto has joined #openstack-containers | 14:26 | |
| strigazi | mgoddard_ http://paste.openstack.org/raw/614395/ | 14:27 |
| mgoddard_ | strigazi: how old is your setup? the related patch was only merged a few days ago | 14:28 |
| strigazi | uptime 14:28:26 up 15 days, 4:14, 5 users, load average: 2.16, 4.03, 2.41 | 14:28 |
| strigazi | mgoddard_ pip freeze | grep WSME | 14:29 |
| mgoddard_ | ah ok. I'll push my patch for review, it's just a one liner and probably easier to see in code | 14:29 |
| strigazi | mgoddard_ if cluster.docker_volume_size == == wtypes.Unset ? | 14:31 |
| strigazi | mgoddard_ if cluster.docker_volume_size == wtypes.Unset ? | 14:31 |
| *** yamamoto has quit IRC | 14:31 | |
| rochaporto | yes that's what worked for labels too | 14:31 |
| mgoddard_ | strigazi: correct | 14:32 |
| rochaporto | mgoddard_ you'll push the patch or you want me to do it? | 14:32 |
| mgoddard_ | rochaporto: I can do it, need to write it for my system anyway | 14:32 |
| rochaporto | ok cool | 14:33 |
| rochaporto | also https://github.com/openstack/magnum/blob/master/magnum/tests/unit/api/controllers/v1/test_cluster.py#L793 | 14:34 |
| rochaporto | might need a fix. we need to drop the docker_volume_size key from the bdict to make sure it's put as Unset | 14:34 |
| mgoddard_ | right | 14:36 |
| strigazi | mgoddard_ How to you talk to magnum-api? magnumclient? horizon? curl? | 14:43 |
| mgoddard_ | strigazis: magnumclient==2.5.0 | 14:43 |
| mgoddard_ | perhaps your client adds a default? | 14:44 |
| *** kbyrne has joined #openstack-containers | 14:44 | |
| strigazi | mgoddard_ Can you try with magnumclient from master? | 14:44 |
| strigazi | I'll try with 2.5.0 | 14:44 |
| strigazi | mgoddard_ we find it | 14:46 |
| strigazi | it is the client | 14:46 |
| strigazi | rochaporto when you added the parameter it seems that client somewhere passes None, even if you didn't set it somewhere | 14:47 |
| strigazi | mgoddard_ We need to fix the client first | 14:48 |
| mgoddard_ | strigazi: yes, same. I have added logs which show the new client passes None | 14:48 |
| mgoddard_ | so there are two issues | 14:48 |
| strigazi | mgoddard_ We have this in a bug already | 14:48 |
| strigazi | fetching | 14:48 |
| rochaporto | ok that makes sense | 14:49 |
| rochaporto | mgoddard_ patch will fix the Unset check on the api, and we need another to make sure the client doesn't pass None | 14:50 |
| rochaporto | this should make it consistent with horizon, etc. is this it? | 14:50 |
| mgoddard_ | sounds right | 14:50 |
| strigazi | mgoddard_ rochaporto if check with unset in the api but don't fix the client the problem will remain but in the opposite way | 14:51 |
| strigazi | mgoddard_ rochaporto if we check with unset in the api but don't fix the client the problem will remain but in the opposite way | 14:51 |
| rochaporto | yes we need both | 14:51 |
| strigazi | This is related https://bugs.launchpad.net/magnum/+bug/1635244 | 14:52 |
| openstack | Launchpad bug 1635244 in Magnum "newton client incompatible with mitaka magnum server" [Undecided,New] | 14:52 |
| rochaporto | ahh right | 14:53 |
| strigazi | When we add an optional param in the client, the client still passes a value. So new client don't work with old APIs | 14:53 |
| mgoddard_ | this is why most projects have microversions :) | 14:53 |
| mgoddard_ | although they come with their own pain | 14:54 |
| rochaporto | if we don't pass the param at all we don't need them do we? | 14:54 |
| strigazi | mgoddard_ we DO have microversions | 14:54 |
| mgoddard_ | strigazi: oh! | 14:54 |
| mgoddard_ | so you do :) | 14:55 |
| strigazi | mgoddard_ new clients but without specifying the api version, so latest | 14:55 |
| *** yamamoto has joined #openstack-containers | 14:56 | |
| openstackgerrit | Mark Goddard proposed openstack/magnum master: Fix cluster inheritence of docker_volume_size https://review.openstack.org/480220 | 14:56 |
| strigazi | mgoddard_ https://github.com/openstack/python-magnumclient/blob/7f573470b50479e21c6d43bfc9ced75be83b4d7f/magnumclient/shell.py#L61 | 14:57 |
| mgoddard_ | right, thanks | 14:57 |
| strigazi | mgoddard_ Your patch is incompatible with the latest client though | 14:58 |
| mgoddard_ | strigazi: correct. the client needs fixing | 14:59 |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: Allow labels on cluster create https://review.openstack.org/473346 | 14:59 |
| mgoddard_ | rochaporto: are you looking at the client issue or shall I? | 15:00 |
| *** salmankhan has quit IRC | 15:00 | |
| mgoddard_ | strigazi: has the latest client been released? | 15:01 |
| *** yamamoto has quit IRC | 15:01 | |
| rochaporto | if you can go for it, i have a couple other patches to push | 15:05 |
| strigazi | mgoddard_ It's not by three patches, but the client from master is broken. | 15:05 |
| strigazi | *will be broken | 15:06 |
| mgoddard_ | rochaporto: will do | 15:07 |
| *** mjura has quit IRC | 15:07 | |
| *** salmankhan has joined #openstack-containers | 15:11 | |
| *** armaan has quit IRC | 15:13 | |
| strigazi | mgoddard_ rochaporto param which are not should not be passed here https://github.com/openstack/python-magnumclient/blob/master/magnumclient/v1/clusters_shell.py#L147 | 15:21 |
| strigazi | I'm concerned what will be the behavior of existing params. I haven't gone through our cluent unit tests | 15:22 |
| mgoddard_ | it looks like they all have defaults in the client | 15:25 |
| strigazi | in the shell though, right? | 15:26 |
| *** armaan has joined #openstack-containers | 15:26 | |
| strigazi | even when I remove the default=None from https://github.com/openstack/python-magnumclient/blob/master/magnumclient/v1/clusters_shell.py#L115 I still see None on the server | 15:27 |
| mgoddard_ | right | 15:28 |
| mgoddard_ | I think if it's using argparse then the default will be None anyway | 15:29 |
| mgoddard_ | https://github.com/openstack/python-magnumclient/blob/master/magnumclient/v1/clusters_shell.py#L147 | 15:29 |
| mgoddard_ | it needs an if not None | 15:29 |
| mgoddard_ | but even that seems not to have worked :) | 15:29 |
| strigazi | mgoddard_ Worked for me... | 15:30 |
| *** rochapor1o has quit IRC | 15:32 | |
| *** ykarel has quit IRC | 15:32 | |
| mgoddard_ | my bad - git fail. Yes, that does seem to work | 15:34 |
| openstackgerrit | Mark Goddard proposed openstack/python-magnumclient master: Don't set a default for docker_volume_size https://review.openstack.org/480239 | 15:36 |
| mgoddard_ | strigazi: still supporting bays?! | 15:38 |
| strigazi | mgoddard_ yes... In queens are gone | 15:39 |
| mgoddard_ | good! | 15:39 |
| mgoddard_ | looks like docker_volume_size was not added for bays though | 15:39 |
| strigazi | mgoddard_ Can you add Related-bug: #1635244 ? | 15:39 |
| openstack | bug 1635244 in Magnum "newton client incompatible with mitaka magnum server" [Undecided,New] https://launchpad.net/bugs/1635244 | 15:39 |
| mgoddard_ | sure | 15:40 |
| strigazi | mgoddard_ indeed, but in server it is added... | 15:40 |
| *** EricGonczer_ has joined #openstack-containers | 15:41 | |
| mgoddard_ | hmm | 15:41 |
| mgoddard_ | that sounds like a different bug | 15:41 |
| strigazi | mgoddard_ I can live with this kind of bugs and they are easy to fix :) | 15:43 |
| openstackgerrit | Mark Goddard proposed openstack/python-magnumclient master: Don't set a default for docker_volume_size https://review.openstack.org/480239 | 15:44 |
| *** kevinz has quit IRC | 15:48 | |
| *** rcernin has quit IRC | 15:58 | |
| *** armaan has quit IRC | 15:59 | |
| *** salmankhan1 has joined #openstack-containers | 16:01 | |
| openstackgerrit | Mark Goddard proposed openstack/magnum master: Fix cluster inheritence of docker_volume_size https://review.openstack.org/480220 | 16:02 |
| *** yamamoto has joined #openstack-containers | 16:04 | |
| *** salmankhan has quit IRC | 16:05 | |
| *** salmankhan1 is now known as salmankhan | 16:05 | |
| strigazi | Thanks mgoddard_ | 16:08 |
| mgoddard_ | strigazi: no problem, thanks for the fast review. Probably should have used Depends-On but never mind | 16:14 |
| strigazi | mgoddard_ it's fine, I'm watching | 16:14 |
| mgoddard_ | strigazi: while I have you, I've spotted something interesting in magnum | 16:15 |
| strigazi | mgoddard_ tell me | 16:15 |
| mgoddard_ | after I create a cluster, I get a series of backtraces in the conductor log, similar to those in https://bugs.launchpad.net/kolla-ansible/+bug/1701713 | 16:16 |
| openstack | Launchpad bug 1701713 in kolla-ansible "Ironic fails to communicate with neutron, glance and inspector" [Undecided,Fix released] | 16:16 |
| mgoddard_ | argh, not that! this: https://bugs.launchpad.net/magnum/+bug/1682058 | 16:16 |
| openstack | Launchpad bug 1682058 in Magnum "Keystone API connection failed" [Undecided,New] | 16:16 |
| *** yamamoto has quit IRC | 16:17 | |
| mgoddard_ | so the failure itself is obviously an issue, seems to be a race between creation and periodic update | 16:17 |
| mgoddard_ | but the interesting part is that I see N identical backtraces in my log, all within one second | 16:18 |
| strigazi | mgoddard_ ocata or master-ish | 16:18 |
| mgoddard_ | at first I thought it was some tight loop, but it's actually every conductor worker. Each one polls at the same time and tries to do the same thing, and all fail in the same way! As I have a fairly beefy box, that's ~50 workers | 16:19 |
| mgoddard_ | this is on magnum | 16:19 |
| mgoddard_ | *master | 16:20 |
| strigazi | mgoddard_ 50 xD | 16:20 |
| mgoddard_ | I also see a fair few DB deadlocks, which is probably a similar issue - 50 workers all trying to update the DB for the same cluster! | 16:20 |
| mgoddard_ | it also hammers keystone and heat | 16:21 |
| mgoddard_ | seems like there needs to be some sort of task distribution between the worked | 16:21 |
| mgoddard_ | *workers | 16:21 |
| strigazi | mgoddard_ interesting indeed | 16:21 |
| mgoddard_ | I thought so :) | 16:21 |
| strigazi | mgoddard_ What do you use here: https://github.com/openstack/magnum/blob/master/magnum/common/rpc_service.py#L55 and here: https://github.com/openstack/magnum/blob/master/magnum/common/rpc.py#L165 | 16:23 |
| strigazi | mgoddard_ You said once per cluster or only for the first cluster ? | 16:23 |
| mgoddard_ | I think it's once per cluster | 16:24 |
| mgoddard_ | we use defaults for most things | 16:27 |
| mgoddard_ | just checked, it's once per cluster | 16:28 |
| *** EricGonczer_ has quit IRC | 16:28 | |
| strigazi | mgoddard_ I don't see it on my devstack but I see that all conductors try to do the same thing, that is not good | 16:28 |
| strigazi | mgoddard_ I have to go. I'll investigate tmr and let you know | 16:29 |
| strigazi | mgoddard_ For the traceback I think there is a bug already | 16:30 |
| *** mgoddard__ has joined #openstack-containers | 16:30 | |
| strigazi | mgoddard_ For the distribution in workers can you open a bug? | 16:30 |
| strigazi | mgoddard_ bye | 16:30 |
| * strigazi is leaving | 16:30 | |
| *** mgoddard_ has quit IRC | 16:32 | |
| *** mgoddard_ has joined #openstack-containers | 16:33 | |
| *** mgoddard__ has quit IRC | 16:36 | |
| *** salmankhan has quit IRC | 16:39 | |
| *** salmankhan1 has joined #openstack-containers | 16:39 | |
| *** mgoddard__ has joined #openstack-containers | 16:40 | |
| *** salmankhan1 is now known as salmankhan | 16:42 | |
| *** mgoddard_ has quit IRC | 16:42 | |
| *** mgoddard_ has joined #openstack-containers | 16:44 | |
| openstackgerrit | Mark Goddard proposed openstack/magnum master: Fix cluster inheritence of docker_volume_size https://review.openstack.org/480220 | 16:45 |
| *** mgoddard__ has quit IRC | 16:46 | |
| *** EricGonczer_ has joined #openstack-containers | 16:55 | |
| *** salmankhan has quit IRC | 16:57 | |
| *** armaan has joined #openstack-containers | 17:13 | |
| *** armaan_ has joined #openstack-containers | 17:15 | |
| *** armaan_ has quit IRC | 17:17 | |
| *** armaan_ has joined #openstack-containers | 17:17 | |
| *** zhugaoxiao has quit IRC | 17:18 | |
| *** armaan has quit IRC | 17:19 | |
| *** zhugaoxiao has joined #openstack-containers | 17:19 | |
| *** armaan_ has quit IRC | 17:19 | |
| *** ykarel has joined #openstack-containers | 17:30 | |
| *** chhavi_ has quit IRC | 17:35 | |
| *** ykarel has quit IRC | 17:36 | |
| *** dsariel has quit IRC | 18:02 | |
| *** yolanda has quit IRC | 18:05 | |
| *** armaan has joined #openstack-containers | 18:11 | |
| *** armaan has quit IRC | 18:15 | |
| *** hieulq has quit IRC | 18:25 | |
| *** rpi has quit IRC | 18:34 | |
| openstackgerrit | Mark Goddard proposed openstack/magnum master: Don't poll heat if no stack exists https://review.openstack.org/480286 | 18:37 |
| *** hieulq has joined #openstack-containers | 18:40 | |
| *** armaan has joined #openstack-containers | 18:47 | |
| *** mgoddard_ has quit IRC | 18:48 | |
| *** livelace has quit IRC | 18:57 | |
| *** mgoddard_ has joined #openstack-containers | 19:38 | |
| *** daidv has quit IRC | 19:48 | |
| *** hieulq has quit IRC | 19:48 | |
| *** zhugaoxiao has quit IRC | 19:59 | |
| *** daidv has joined #openstack-containers | 20:01 | |
| *** hieulq has joined #openstack-containers | 20:02 | |
| *** dsariel has joined #openstack-containers | 20:08 | |
| *** openstackgerrit has quit IRC | 20:18 | |
| *** askb has joined #openstack-containers | 20:29 | |
| *** catintheroof has quit IRC | 20:53 | |
| *** armaan has quit IRC | 21:16 | |
| *** armaan has joined #openstack-containers | 21:16 | |
| *** daidv has quit IRC | 21:17 | |
| *** hieulq has quit IRC | 21:17 | |
| *** daidv has joined #openstack-containers | 21:30 | |
| *** hieulq has joined #openstack-containers | 21:30 | |
| *** mgoddard_ has quit IRC | 21:56 | |
| *** spiette has quit IRC | 22:18 | |
| *** adisky__ has quit IRC | 22:20 | |
| *** spiette has joined #openstack-containers | 22:21 | |
| *** hongbin has quit IRC | 23:13 | |
| *** EricGonczer_ has joined #openstack-containers | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!