| *** Marga_ has joined #openstack-containers | 00:02 | |
| *** adrian_otto has quit IRC | 00:03 | |
| openstackgerrit | Kai Qiang Wu(Kennan) proposed openstack/magnum: Delete swarm bay also delete related containers https://review.openstack.org/180799 | 00:07 |
|---|---|---|
| Kennan | :lan just cherry-pick your changes, you can check that | 00:09 |
| *** dboik has joined #openstack-containers | 00:14 | |
| *** Marga_ has quit IRC | 00:17 | |
| *** Marga_ has joined #openstack-containers | 00:17 | |
| *** dboik has quit IRC | 00:18 | |
| *** oro has quit IRC | 00:20 | |
| *** jay-lau-513 has quit IRC | 00:25 | |
| *** jruano has joined #openstack-containers | 00:25 | |
| *** daneyon_ has quit IRC | 00:53 | |
| *** shakamunyi has quit IRC | 01:02 | |
| *** dboik has joined #openstack-containers | 01:05 | |
| *** dboik_ has joined #openstack-containers | 01:06 | |
| *** dboik has quit IRC | 01:09 | |
| *** kebray has joined #openstack-containers | 01:10 | |
| *** kebray has quit IRC | 01:10 | |
| *** kebray has joined #openstack-containers | 01:11 | |
| *** kebray has quit IRC | 01:16 | |
| *** kebray has joined #openstack-containers | 01:16 | |
| *** fangfenghua has joined #openstack-containers | 01:22 | |
| *** achanda has quit IRC | 01:29 | |
| *** MilesDenver has quit IRC | 01:32 | |
| lan | Kennan, cherry-pick looks good to me, but madhuri leave a comment, please check that : ) | 01:35 |
| *** erkules_ has joined #openstack-containers | 01:38 | |
| *** erkules has quit IRC | 01:41 | |
| *** Marga_ has quit IRC | 01:42 | |
| *** Marga_ has joined #openstack-containers | 01:43 | |
| *** vilobhmm1 has quit IRC | 01:56 | |
| *** dboik_ has quit IRC | 02:01 | |
| *** dboik has joined #openstack-containers | 02:01 | |
| *** hongbin_ has joined #openstack-containers | 02:06 | |
| *** unicell has quit IRC | 02:07 | |
| *** shakamunyi has joined #openstack-containers | 02:10 | |
| *** suro-patz has quit IRC | 02:16 | |
| *** achanda has joined #openstack-containers | 02:29 | |
| *** rpothier has quit IRC | 02:38 | |
| *** madhuri has joined #openstack-containers | 02:43 | |
| *** Marga_ has quit IRC | 02:45 | |
| *** achanda has quit IRC | 02:46 | |
| *** kebray has quit IRC | 02:47 | |
| *** achanda has joined #openstack-containers | 02:48 | |
| *** dims_ has quit IRC | 02:51 | |
| *** dims has joined #openstack-containers | 02:52 | |
| *** sdake_ has joined #openstack-containers | 03:09 | |
| openstackgerrit | Merged openstack/magnum: Update service support a manifest change https://review.openstack.org/174208 | 03:10 |
| openstackgerrit | Merged openstack/magnum: Update rc support a manifest change https://review.openstack.org/174209 | 03:11 |
| *** dims has quit IRC | 03:11 | |
| *** sdake has quit IRC | 03:13 | |
| *** achanda has quit IRC | 03:28 | |
| *** sdake has joined #openstack-containers | 03:36 | |
| *** sdake_ has quit IRC | 03:39 | |
| *** daneyon has joined #openstack-containers | 03:45 | |
| *** chandankumar has joined #openstack-containers | 03:46 | |
| *** Tango has joined #openstack-containers | 04:00 | |
| *** achanda has joined #openstack-containers | 04:05 | |
| *** chandankumar has quit IRC | 04:33 | |
| *** coolsvap|afk is now known as coolsvap | 05:03 | |
| *** unicell has joined #openstack-containers | 05:06 | |
| *** hongbin_ has quit IRC | 05:09 | |
| *** vilobhmm1 has joined #openstack-containers | 05:12 | |
| *** madhuri has quit IRC | 05:14 | |
| *** jruano has quit IRC | 05:15 | |
| *** madhuri has joined #openstack-containers | 05:16 | |
| *** sdake has quit IRC | 05:20 | |
| *** sdake has joined #openstack-containers | 05:24 | |
| *** madhuri has quit IRC | 05:24 | |
| *** madhuri has joined #openstack-containers | 05:25 | |
| *** sdake_ has joined #openstack-containers | 05:26 | |
| *** chandankumar has joined #openstack-containers | 05:28 | |
| *** suro-patz has joined #openstack-containers | 05:29 | |
| *** sdake has quit IRC | 05:29 | |
| *** madhuri has quit IRC | 05:31 | |
| *** madhuri has joined #openstack-containers | 05:32 | |
| *** adrian_otto has joined #openstack-containers | 05:36 | |
| *** adrian_otto has quit IRC | 05:39 | |
| *** madhuri has quit IRC | 05:43 | |
| *** suro-patz1 has joined #openstack-containers | 05:54 | |
| *** madhuri has joined #openstack-containers | 05:54 | |
| *** suro-patz has quit IRC | 05:56 | |
| *** vilobhmm1 has quit IRC | 05:59 | |
| *** madhuri_ has joined #openstack-containers | 05:59 | |
| *** madhuri has quit IRC | 06:01 | |
| *** Kennan has quit IRC | 06:02 | |
| *** Kennan has joined #openstack-containers | 06:03 | |
| *** suro-patz1 has quit IRC | 06:05 | |
| sdake_ | any ibm cats around | 06:09 |
| *** sdake_ is now known as sdake | 06:10 | |
| *** suro-patz has joined #openstack-containers | 06:13 | |
| *** Tango has quit IRC | 06:13 | |
| *** erkules_ is now known as erkules | 06:24 | |
| *** erkules has joined #openstack-containers | 06:24 | |
| *** oro has joined #openstack-containers | 06:31 | |
| *** suro-patz has quit IRC | 06:40 | |
| *** adrian_otto has joined #openstack-containers | 06:42 | |
| openstackgerrit | Madhuri Kumari proposed openstack/magnum: Update rc manifest parsing according to v1beta3 https://review.openstack.org/178755 | 07:02 |
| *** fangfenghua has quit IRC | 07:23 | |
| adrian_otto | madhuri_: you around? | 07:23 |
| madhuri_ | Hi adrian_otto | 07:23 |
| adrian_otto | hey, do you have a moment to work with me on something? | 07:24 |
| madhuri_ | Yes sure | 07:24 |
| adrian_otto | I have a devstack setup with magnum in it | 07:24 |
| madhuri_ | Ok | 07:24 |
| adrian_otto | and I'm able to create baymodels and bays | 07:24 |
| adrian_otto | but when I try to use the instructions in our quickstart to make a pod using the kubernetes example for redis-master.yaml, that fails to create a pod | 07:25 |
| adrian_otto | I get "ERROR: Field metadata['name'] can't be empty in maifest. (HTTP 400)" | 07:25 |
| madhuri_ | What is the error? | 07:25 |
| adrian_otto | do you think that's coming from k8s or from magnum | 07:25 |
| adrian_otto | I tried adding name: nodes to the yaml in the manifest at various points, but that does not seem to help | 07:26 |
| madhuri_ | I got it | 07:26 |
| madhuri_ | Just give me a second | 07:26 |
| adrian_otto | you reproduced it? | 07:26 |
| *** jay-lau-513 has joined #openstack-containers | 07:27 | |
| jay-lau-513 | join #openstack-chinese | 07:27 |
| madhuri_ | The yaml file you are using is of v1beta1 | 07:28 |
| adrian_otto | yes. | 07:28 |
| madhuri_ | You need to use v1beta3 yaml file | 07:28 |
| madhuri_ | https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/redis/v1beta3/redis-master.yaml | 07:28 |
| adrian_otto | oh | 07:28 |
| madhuri_ | Use this one | 07:28 |
| *** nshaikh has joined #openstack-containers | 07:28 | |
| madhuri_ | It will work | 07:28 |
| adrian_otto | ok, thanks!! | 07:28 |
| madhuri_ | My pleasure. | 07:29 |
| adrian_otto | that definitely worked. | 07:31 |
| madhuri_ | :) | 07:31 |
| adrian_otto | madhuri | 07:40 |
| adrian_otto | it did create, but in a failed status | 07:40 |
| adrian_otto | $ magnum pod-show redis-master | grep status | 07:40 |
| adrian_otto | | status | failed | | 07:40 |
| adrian_otto | how can I figure out why? | 07:41 |
| *** oro has quit IRC | 07:42 | |
| adrian_otto | ~$ magnum service-create --manifest ./redis-sentinel-service.yaml --bay testbay | 07:46 |
| adrian_otto | ERROR: Invalid resource state. (HTTP 409) | 07:46 |
| madhuri_ | For the second issue | 07:50 |
| madhuri_ | You can try creating the service directly from kubectl | 07:51 |
| madhuri_ | 409 here means the service alerady exists which you are trying to create | 07:51 |
| adrian_otto | ok | 07:51 |
| adrian_otto | we should improve that error message | 07:52 |
| madhuri_ | There could be many response from kubernetes | 07:53 |
| madhuri_ | and updating the message for each response could be a good or bad idea. | 07:54 |
| madhuri_ | I am not sure. | 07:54 |
| madhuri_ | But I agree the message doesn't give any idea to user whats happening. | 07:54 |
| adrian_otto | ok, so back the the previous concern | 07:55 |
| adrian_otto | I created a fresh bay | 07:55 |
| adrian_otto | and did this: magnum pod-create --manifest redis-master.yaml --bay testbay | 07:56 |
| *** achanda has quit IRC | 07:56 | |
| adrian_otto | using the correct yaml | 07:56 |
| madhuri_ | Ok | 07:56 |
| adrian_otto | $ magnum pod-show redis-master |grep status | 07:56 |
| adrian_otto | | status | failed | 07:56 |
| adrian_otto | and the status immediately shows as failed | 07:56 |
| madhuri_ | Try kubectl get events | 07:57 |
| madhuri_ | To check the failure | 07:57 |
| adrian_otto | on the kube master? | 07:58 |
| madhuri_ | Yes | 07:58 |
| adrian_otto | there are no events | 07:58 |
| adrian_otto | I just get the ehader line | 07:58 |
| adrian_otto | *header | 07:58 |
| adrian_otto | $ kubectl get events | 07:59 |
| adrian_otto | FIRSTSEEN LASTSEEN COUNT NAME KIND SUBOBJECT REASON SOURCE MESSAGE | 07:59 |
| madhuri_ | It means the pod is not created on Kubernetes node | 07:59 |
| madhuri_ | Can you run kubectl get pod | 07:59 |
| adrian_otto | empty list, only headings. | 07:59 |
| madhuri_ | This seems to be an issue | 08:00 |
| madhuri_ | It means the pod is not created on Kube node but magnum create an entry in db | 08:00 |
| openstackgerrit | Accela Zhao proposed openstack/magnum: Remove trailing spaces in container-service.rst https://review.openstack.org/180902 | 08:00 |
| madhuri_ | magnum-conductor logs might help us | 08:00 |
| adrian_otto | the kube-apiserver process on the k8s master is only listening on IPV6 | 08:00 |
| adrian_otto | is that normal? | 08:00 |
| madhuri_ | let me check | 08:01 |
| openstackgerrit | Madhuri Kumari proposed openstack/magnum: Update service manifest parsing according to v1beta3 https://review.openstack.org/178519 | 08:03 |
| openstackgerrit | Accela Zhao proposed openstack/magnum: Remove trailing spaces in container-service.rst https://review.openstack.org/180902 | 08:13 |
| *** nshaikh has quit IRC | 08:28 | |
| *** fangfenghua has joined #openstack-containers | 09:07 | |
| *** dims has joined #openstack-containers | 09:36 | |
| *** dims has quit IRC | 09:41 | |
| adrian_otto | https://bugs.launchpad.net/magnum/+bug/1452650 | 09:54 |
| openstack | Launchpad bug 1452650 in Magnum "Unable to show a container by name" [Low,Triaged] | 09:54 |
| adrian_otto | https://bugs.launchpad.net/magnum/+bug/1452648 | 09:55 |
| openstack | Launchpad bug 1452648 in Magnum "container-list fails with unexpected bay-id" [Undecided,New] - Assigned to Andrew Melton (andrew-melton) | 09:55 |
| adrian_otto | if anyone could help with those, that would be great. | 09:55 |
| *** dims_ has joined #openstack-containers | 09:59 | |
| *** jay-lau-513 has quit IRC | 10:13 | |
| openstackgerrit | Tom Cammann proposed openstack/magnum: Remove oslo config warnings https://review.openstack.org/180926 | 10:23 |
| *** adrian_otto has quit IRC | 10:31 | |
| tcammann | How can I trigger the gate job on my change. Its refusing to merge even though it is +2/A https://review.openstack.org/#/c/180158/ | 10:32 |
| *** nshaikh has joined #openstack-containers | 10:34 | |
| *** coolsvap is now known as coolsvap|afk | 10:43 | |
| *** achanda has joined #openstack-containers | 11:02 | |
| *** achanda has quit IRC | 11:07 | |
| *** jruano has joined #openstack-containers | 11:15 | |
| *** oro has joined #openstack-containers | 11:26 | |
| *** EricGonczer_ has joined #openstack-containers | 11:41 | |
| *** openstackgerrit has quit IRC | 11:52 | |
| *** openstackgerrit has joined #openstack-containers | 11:52 | |
| *** EricGonczer_ has quit IRC | 11:55 | |
| openstackgerrit | Grzegorz Grasza (xek) proposed openstack/magnum: Move our ObjectSerializer to subclass from the Oslo one https://review.openstack.org/178718 | 12:01 |
| openstackgerrit | Grzegorz Grasza (xek) proposed openstack/magnum: Make MagnumObject a subclass of Oslo VersionedObject https://review.openstack.org/180133 | 12:01 |
| openstackgerrit | Grzegorz Grasza (xek) proposed openstack/magnum: Move VersionedObject registration to the new scheme https://review.openstack.org/180577 | 12:01 |
| *** EricGonczer_ has joined #openstack-containers | 12:34 | |
| *** rpothier has joined #openstack-containers | 12:41 | |
| *** dims_ has quit IRC | 12:51 | |
| *** dims has joined #openstack-containers | 12:51 | |
| *** jruano has quit IRC | 12:53 | |
| *** hongbin_ has joined #openstack-containers | 12:57 | |
| *** rlrevell has joined #openstack-containers | 13:12 | |
| *** david-lyle has quit IRC | 13:19 | |
| *** dboik has quit IRC | 13:24 | |
| *** openstackstatus has quit IRC | 13:25 | |
| *** fangfeng_ has joined #openstack-containers | 13:25 | |
| *** openstackstatus has joined #openstack-containers | 13:25 | |
| *** ChanServ sets mode: +v openstackstatus | 13:25 | |
| *** fangfeng_ has quit IRC | 13:30 | |
| *** fangfenghua_ has joined #openstack-containers | 13:33 | |
| *** hongbin_ has quit IRC | 13:36 | |
| *** EricGonczer_ has quit IRC | 13:43 | |
| *** dboik has joined #openstack-containers | 13:44 | |
| *** dboik has quit IRC | 13:44 | |
| *** dboik has joined #openstack-containers | 13:45 | |
| *** jfarschman has joined #openstack-containers | 13:50 | |
| *** jfarschman is now known as MilesDenver | 13:50 | |
| *** oro has quit IRC | 13:55 | |
| *** zul has quit IRC | 13:57 | |
| *** hongbin_ has joined #openstack-containers | 13:59 | |
| *** jruano has joined #openstack-containers | 13:59 | |
| *** zul has joined #openstack-containers | 14:02 | |
| *** achanda has joined #openstack-containers | 14:06 | |
| *** adrian_otto has joined #openstack-containers | 14:06 | |
| *** oro has joined #openstack-containers | 14:10 | |
| *** achanda has quit IRC | 14:11 | |
| *** adrian_otto has quit IRC | 14:17 | |
| *** chandankumar has quit IRC | 14:17 | |
| *** achanda has joined #openstack-containers | 14:29 | |
| *** achanda has quit IRC | 14:31 | |
| *** jruano has quit IRC | 14:32 | |
| *** fangfenghua has quit IRC | 14:33 | |
| *** Tango|2 has joined #openstack-containers | 14:34 | |
| openstackgerrit | Merged openstack/magnum: Add unique column constraints to db https://review.openstack.org/180095 | 14:44 |
| *** adrian_otto has joined #openstack-containers | 14:45 | |
| *** adrian_otto has quit IRC | 14:45 | |
| *** adrian_otto has joined #openstack-containers | 14:48 | |
| *** adrian_otto has quit IRC | 14:56 | |
| *** achanda has joined #openstack-containers | 14:56 | |
| *** dboik_ has joined #openstack-containers | 15:01 | |
| *** achanda has quit IRC | 15:03 | |
| *** dboik has quit IRC | 15:05 | |
| *** apmelton_away is now known as apmelton | 15:10 | |
| *** kebray has joined #openstack-containers | 15:12 | |
| sdake | morning | 15:14 |
| sdake | reading scrollback looks like magnum is busted | 15:19 |
| sdake | wtb dims functonal testing gate :) | 15:19 |
| sdake | hongbin around? | 15:20 |
| hongbin_ | yes | 15:21 |
| *** chandankumar has joined #openstack-containers | 15:23 | |
| *** oro has quit IRC | 15:31 | |
| *** fangfenghua has joined #openstack-containers | 15:34 | |
| *** Marga_ has joined #openstack-containers | 15:35 | |
| *** sdake_ has joined #openstack-containers | 15:36 | |
| *** sdake_ has joined #openstack-containers | 15:36 | |
| *** Marga_ has quit IRC | 15:36 | |
| *** Marga_ has joined #openstack-containers | 15:36 | |
| dims | sdake: what's up? | 15:39 |
| *** EricGonczer_ has joined #openstack-containers | 15:39 | |
| *** daneyon_ has joined #openstack-containers | 15:41 | |
| *** nshaikh has quit IRC | 15:42 | |
| *** sdake_ has quit IRC | 15:42 | |
| *** sdake has quit IRC | 15:44 | |
| *** sdake has joined #openstack-containers | 15:44 | |
| *** daneyon has quit IRC | 15:44 | |
| *** chandankumar has quit IRC | 15:48 | |
| tcammann | sdake: I replied to your comment on https://review.openstack.org/#/c/180926 | 15:49 |
| sdake | yes I saw, I was just verifying it :) | 15:50 |
| sdake | what about the db thing? | 15:50 |
| tcammann | Its not used in the magnum repo at all, and I couldn't reproduce the warning | 15:51 |
| sdake | is that set in devstack plugin? | 15:51 |
| sdake | (keep in mind I am using kolla not devstack) | 15:51 |
| sdake | its in the dev quickstart | 15:51 |
| sdake | is there still the auth_uri deprecation warning for the api service? | 15:52 |
| tcammann | whats in the quickstart? | 15:54 |
| tcammann | I don't get any more deprecated warns using the devstack conf | 15:56 |
| *** daneyon_ has quit IRC | 15:58 | |
| *** chandankumar has joined #openstack-containers | 15:59 | |
| *** daneyon has joined #openstack-containers | 15:59 | |
| sdake | in the docs directory | 16:00 |
| sdake | there is a quickstart for non-devstack | 16:00 |
| *** dboik_ has quit IRC | 16:04 | |
| *** dboik has joined #openstack-containers | 16:04 | |
| tcammann | sdake: thanks for the +2. Could you also re-approve this change https://review.openstack.org/#/c/180158/ | 16:05 |
| tcammann | Its stuck at the moment and won't start the gate job. The infra guys said it needs to be reapproved :? | 16:06 |
| sdake | tcammann that may or may not work - jaylay may have to remoe his +a | 16:07 |
| sdake | we will see ;) | 16:07 |
| sdake | ya the gate was busted yesterday | 16:07 |
| tcammann | thanks. looks like its finally started the gate jobs. | 16:09 |
| openstackgerrit | Merged openstack/magnum: Improve validation on baymodel api calls https://review.openstack.org/180158 | 16:10 |
| apmelton | does anyone have a working devstack running a kubernetes bay at the moment? | 16:11 |
| *** unicell has quit IRC | 16:17 | |
| *** suro-patz has joined #openstack-containers | 16:17 | |
| *** suro-patz1 has joined #openstack-containers | 16:19 | |
| *** suro-patz has quit IRC | 16:22 | |
| *** Marga_ has quit IRC | 16:24 | |
| *** adrian_otto has joined #openstack-containers | 16:24 | |
| sdake | I can do that with kolla apmelton, but unfortunately resources don't launch | 16:28 |
| apmelton | :( | 16:29 |
| *** oro has joined #openstack-containers | 16:29 | |
| *** shakamunyi has quit IRC | 16:29 | |
| *** Marga_ has joined #openstack-containers | 16:31 | |
| apmelton | I've got a really old node sitting around that I'm trying it on to see if it's not broke there as well | 16:31 |
| apmelton | but it looks like the xml nova is spitting out for the instance isn't including the cinder volume | 16:32 |
| *** fangfenghua_ has quit IRC | 16:34 | |
| apmelton | yea... volumes seem to be broke as far back as april 24th | 16:34 |
| sdake | can you launch resources with master of magnum? | 16:36 |
| apmelton | sdake: nope, cinder volume issues | 16:38 |
| sdake | i took the cinder volumes out of my build ;-) | 16:39 |
| sdake | (rather template) | 16:39 |
| apmelton | sdake: doing the same right now :P | 16:40 |
| *** adrian_otto has quit IRC | 16:45 | |
| *** unicell has joined #openstack-containers | 16:47 | |
| apmelton | sdake: so, the reason the volume doesn't show up in the xml is that it's hot plugged into the instance | 16:48 |
| apmelton | I wonder if there's something about atomic that it can't handle hot plugging | 16:48 |
| sdake | not sure - there is an atomic mailing list | 16:48 |
| *** adrian_otto has joined #openstack-containers | 16:49 | |
| sdake | did it ever work? | 16:49 |
| sdake | or did you never try with atomic | 16:49 |
| sdake | adrian_otto so the api server is binding to ipv6 then? | 16:49 |
| apmelton | it had to have, otherwise the pods would never have actually gone active when we were testing the fedora-atomic-3 image | 16:49 |
| sdake | and which api server kube or magnum | 16:50 |
| sdake | apmelton can you paste your patch wherey ou removed cinder from the code | 16:50 |
| sdake | all I did was remove the resources | 16:50 |
| sdake | I thought that would be enough | 16:50 |
| *** chandankumar has quit IRC | 16:50 | |
| apmelton | sdake: https://gist.github.com/ramielrowe/ada727132b2342f1b7a5 | 16:50 |
| sdake | configure_docker_storage is what does the config job? | 16:51 |
| apmelton | yea, it's the script that was failing out cloud-init | 16:51 |
| *** sdake_ has joined #openstack-containers | 16:52 | |
| *** hongbin_ has quit IRC | 16:52 | |
| *** suro-patz1 has quit IRC | 16:53 | |
| *** vilobhmm1 has joined #openstack-containers | 16:54 | |
| *** sdake has quit IRC | 16:56 | |
| sdake_ | apmelton thanks for the diff | 16:58 |
| sdake_ | that might explain why my resources don't start | 16:58 |
| sdake_ | the docker vg is afu | 16:58 |
| sdake_ | a hotplug probably wont work of that cinder volume unless there is a dependency on the resource configure_docker_volume | 16:59 |
| sdake_ | but I'm not certain | 16:59 |
| sdake_ | definately not a nova expert | 16:59 |
| apmelton | I wonder if we should modify the template so that the volumes are created before the nodes and are just attached on boot | 16:59 |
| apmelton | on my local box I do see the volume attach calls going through | 17:00 |
| apmelton | in the nova logs | 17:00 |
| apmelton | but, I don't see them from the instance | 17:00 |
| *** Marga_ has quit IRC | 17:00 | |
| apmelton | or they're at least not at where ever our configure_docker_volume script thinks they should be | 17:01 |
| *** chandankumar has joined #openstack-containers | 17:03 | |
| apmelton | I gotta go afk for a bit, bbiaf | 17:04 |
| *** hongbin_ has joined #openstack-containers | 17:13 | |
| *** sdake has joined #openstack-containers | 17:13 | |
| *** sdake_ has quit IRC | 17:17 | |
| apmelton | back | 17:19 |
| *** Marga_ has joined #openstack-containers | 17:25 | |
| apmelton | so... | 17:27 |
| apmelton | it looks like the volume eventualy shows up | 17:27 |
| *** oro has quit IRC | 17:27 | |
| apmelton | I guess we could add a wait in that config script, that keeps checking for the device until it's there | 17:28 |
| sdake | 187 tasks in the copr buid lqueue | 17:28 |
| * sdake groans | 17:28 | |
| *** oro has joined #openstack-containers | 17:28 | |
| apmelton | whats that? | 17:29 |
| sdake | https://copr.fedoraproject.org/status/waiting/ | 17:29 |
| sdake | look down at the bottom, you will see I am building rpms for magnum | 17:29 |
| sdake | with your patch applied | 17:29 |
| apmelton | ah gotcha | 17:29 |
| *** Marga_ has quit IRC | 17:30 | |
| apmelton | looks like it's gonna be at least 9 hours ;P | 17:30 |
| apmelton | might not be bad, I might have a better fix in a bit | 17:30 |
| *** Marga_ has joined #openstack-containers | 17:30 | |
| sdake | i can't use cinder, kolla doen't support it | 17:30 |
| apmelton | ah | 17:30 |
| *** Marga_ has quit IRC | 17:31 | |
| *** fangfenghua_ has joined #openstack-containers | 17:34 | |
| *** Marga_ has joined #openstack-containers | 17:35 | |
| *** gose has joined #openstack-containers | 17:36 | |
| *** fangfenghua_ has quit IRC | 17:39 | |
| *** achanda has joined #openstack-containers | 17:43 | |
| *** suro-patz has joined #openstack-containers | 17:57 | |
| *** rlrevell1 has joined #openstack-containers | 17:59 | |
| *** rlrevell1 has quit IRC | 17:59 | |
| *** rlrevell1 has joined #openstack-containers | 17:59 | |
| *** suro-patz1 has joined #openstack-containers | 18:00 | |
| *** rlrevell has quit IRC | 18:01 | |
| *** jsotoca has joined #openstack-containers | 18:01 | |
| *** suro-patz has quit IRC | 18:03 | |
| *** vilobhmm1 has quit IRC | 18:03 | |
| *** vilobhmm1 has joined #openstack-containers | 18:03 | |
| *** vilobhmm1 has quit IRC | 18:04 | |
| *** vilobhmm1 has joined #openstack-containers | 18:04 | |
| *** sdake has quit IRC | 18:08 | |
| *** sdake has joined #openstack-containers | 18:09 | |
| *** vilobhmm1 has quit IRC | 18:13 | |
| *** vilobhmm1 has joined #openstack-containers | 18:18 | |
| *** jsotoca has quit IRC | 18:19 | |
| openstackgerrit | Merged openstack/magnum: Remove trailing spaces in container-service.rst https://review.openstack.org/180902 | 18:20 |
| openstackgerrit | Merged openstack/magnum: Remove oslo config warnings https://review.openstack.org/180926 | 18:21 |
| *** vilobhmm1 has quit IRC | 18:21 | |
| *** vilobhmm1 has joined #openstack-containers | 18:22 | |
| *** vilobhmm1 has quit IRC | 18:24 | |
| *** vilobhmm1 has joined #openstack-containers | 18:24 | |
| *** Marga_ has quit IRC | 18:32 | |
| *** Marga_ has joined #openstack-containers | 18:36 | |
| *** fangfenghua has quit IRC | 18:43 | |
| *** sdake_ has joined #openstack-containers | 18:43 | |
| *** sdake has quit IRC | 18:47 | |
| openstackgerrit | Andrew Melton proposed openstack/magnum: Make docker volume config more resilient https://review.openstack.org/181115 | 18:52 |
| *** suro-patz1 has quit IRC | 18:54 | |
| *** rlrevell has joined #openstack-containers | 18:59 | |
| *** rlrevell1 has quit IRC | 18:59 | |
| *** suro-patz has joined #openstack-containers | 19:13 | |
| *** suro-patz has quit IRC | 19:16 | |
| *** suro-patz has joined #openstack-containers | 19:16 | |
| *** zul has quit IRC | 19:16 | |
| *** fangfenghua_ has joined #openstack-containers | 19:23 | |
| *** fangfenghua_ has quit IRC | 19:27 | |
| *** Marga_ has quit IRC | 19:28 | |
| *** zul has joined #openstack-containers | 19:31 | |
| *** suro-patz1 has joined #openstack-containers | 19:44 | |
| *** suro-patz has quit IRC | 19:47 | |
| *** achanda has quit IRC | 20:04 | |
| *** achanda has joined #openstack-containers | 20:04 | |
| *** achanda has quit IRC | 20:08 | |
| *** jsotoca has joined #openstack-containers | 20:15 | |
| *** jjlehr has joined #openstack-containers | 20:21 | |
| *** Marga_ has joined #openstack-containers | 20:23 | |
| *** Marga_ has quit IRC | 20:23 | |
| *** Marga_ has joined #openstack-containers | 20:24 | |
| *** oro has quit IRC | 20:27 | |
| *** sdake_ has quit IRC | 20:29 | |
| *** jsotoca has quit IRC | 20:39 | |
| *** Marga_ has quit IRC | 20:41 | |
| *** Marga_ has joined #openstack-containers | 20:42 | |
| *** achanda has joined #openstack-containers | 20:44 | |
| *** adrian_otto has quit IRC | 20:57 | |
| *** oro has joined #openstack-containers | 20:57 | |
| *** Marga_ has quit IRC | 20:58 | |
| *** Marga_ has joined #openstack-containers | 20:58 | |
| *** adrian_otto has joined #openstack-containers | 21:07 | |
| *** sdake has joined #openstack-containers | 21:08 | |
| *** adrian_otto has quit IRC | 21:08 | |
| *** adrian_otto has joined #openstack-containers | 21:09 | |
| *** rpothier has quit IRC | 21:10 | |
| *** sdake_ has joined #openstack-containers | 21:11 | |
| *** sdake has quit IRC | 21:15 | |
| *** rlrevell has quit IRC | 21:21 | |
| *** chandankumar has quit IRC | 21:27 | |
| *** sdake_ is now known as sdake | 21:30 | |
| jjlehr | Has anyone seen this error when trying to create or start a container? | 21:31 |
| jjlehr | magnum container-create < container.json | 21:31 |
| jjlehr | ERROR: Timed out waiting for a reply to message ID 034a3b1ce9f04022ade2124c415d354a (HTTP 500) | 21:31 |
| jjlehr | This is the closest bug I found to it: https://bugs.launchpad.net/magnum/+bug/1443305 | 21:32 |
| openstack | Launchpad bug 1443305 in Magnum "container-api stuck" [Undecided,New] - Assigned to FenghuaFang (449171342-q) | 21:32 |
| *** kebray has quit IRC | 21:46 | |
| *** Marga_ has quit IRC | 21:50 | |
| *** oro has quit IRC | 21:53 | |
| *** sdake_ has joined #openstack-containers | 21:54 | |
| *** david-lyle has joined #openstack-containers | 21:56 | |
| *** sdake has quit IRC | 21:58 | |
| *** hongbin_ has quit IRC | 22:01 | |
| apmelton | jjlehr: that probably means docker was taking too long to create the container | 22:01 |
| apmelton | jjlehr: check the logs from the conductor and you'll likely see an error about a timeout from docker client | 22:02 |
| apmelton | I'm working around a short term work around: https://review.openstack.org/#/c/179257/ | 22:02 |
| apmelton | but to really fix it properly, we're gonna need to make some bigger changes | 22:03 |
| apmelton | I've started a bp for that here: https://blueprints.launchpad.net/magnum/+spec/async-container-operations | 22:03 |
| jjlehr | apmelton: thanks, I will check the logs | 22:05 |
| jjlehr | So I'm running devstak on a VM, meaning I'm doing nested virtualization. Do you think that's why docker is taking too long? | 22:06 |
| apmelton | jjlehr: could be, whats your hypervisor? | 22:06 |
| *** Marga_ has joined #openstack-containers | 22:08 | |
| jjlehr | apmelton: QEMU 2.0.0 | 22:09 |
| jjlehr | Oh, it could just be because of the first docker image pull | 22:09 |
| apmelton | jjlehr: that too | 22:09 |
| apmelton | qemu is the hypervisor your devstack vm is running on? | 22:10 |
| jjlehr | apmeltong: Ah, no. The hypervisor my devstack vm is running on is KVM | 22:11 |
| *** kebray has joined #openstack-containers | 22:12 | |
| *** dboik_ has joined #openstack-containers | 22:12 | |
| *** fangfenghua_ has joined #openstack-containers | 22:13 | |
| apmelton | I haven't tried it with KVM, but supposedly you can accelerate nested virt on KVM: https://wiki.archlinux.org/index.php/KVM#Nested_virtualization | 22:13 |
| apmelton | but likely, if you using a large image for your container, it's probably just taking a long time to download | 22:13 |
| jjlehr | I used the cirros one from the example container in dev-quickstart.rst | 22:14 |
| apmelton | hmmmm | 22:14 |
| apmelton | what kind of bandwidth does your vm have access to? | 22:15 |
| *** dboik has quit IRC | 22:16 | |
| apmelton | thinking about it... with accelerated nested virt, the extraction of the image will probably be pretty slow | 22:17 |
| *** dboik_ has quit IRC | 22:17 | |
| apmelton | without* | 22:18 |
| *** fangfenghua_ has quit IRC | 22:18 | |
| jjlehr | I'm not sure of the bandwidth but I'm checking now | 22:19 |
| jjlehr | It's pretty slow | 22:20 |
| apmelton | less than 50mbps? | 22:20 |
| jjlehr | yes | 22:21 |
| apmelton | gotcha | 22:21 |
| jjlehr | I'll try creating the container again with the same image | 22:21 |
| apmelton | jjlehr: what you might also want to do is hot patch magnum to a higher timeout for docker client | 22:22 |
| apmelton | just bump https://github.com/openstack/magnum/blob/master/magnum/conductor/handlers/common/docker_client.py#L46 to like 600 | 22:22 |
| apmelton | reinstall and restart the conductor | 22:22 |
| *** sdake has joined #openstack-containers | 22:22 | |
| jjlehr | Ok, I will try that since container-delete timed out as well | 22:23 |
| apmelton | ah hmmmm | 22:23 |
| jjlehr | I ssh-ed into the swarm node and saw that there was no container created | 22:23 |
| apmelton | jjlehr: what error if any do you see in the conductor logs? | 22:23 |
| apmelton | jjlehr: did you do a docker ps -a? | 22:24 |
| jjlehr | yes | 22:24 |
| apmelton | actually, you'll need to do from the master node "docker -H tcp://127.0.0.1:2376 ps -a" | 22:24 |
| jjlehr | I thought they're also configured to listen on the unix socket | 22:25 |
| jjlehr | I'll try that though | 22:25 |
| apmelton | the docker daemon is, but to talk to swarm, you'll need to use the tcp socket on the master | 22:25 |
| *** Marga_ has quit IRC | 22:25 | |
| *** sdake_ has quit IRC | 22:25 | |
| *** Marga_ has joined #openstack-containers | 22:26 | |
| jjlehr | shouldn't containers created by swarm also be visible by the host nodes? | 22:26 |
| apmelton | yes, but you'll have to check each node, including the master | 22:26 |
| apmelton | if you use swarm, you get all nodes with one call | 22:26 |
| jjlehr | Ah, I forgot the master is also a host | 22:27 |
| apmelton | jjlehr: while you're poking around, do you see any exceptions in the conductor's logs? | 22:27 |
| *** Marga_ has quit IRC | 22:29 | |
| jjlehr | apmelton: yes I do. It ends up complaining about "ValueError: Circular reference detected" | 22:29 |
| *** Marga_ has joined #openstack-containers | 22:29 | |
| apmelton | there should be another exception just before that | 22:29 |
| apmelton | the circular reference is rpc blowing up trying to pass up the exception | 22:29 |
| apmelton | at least I think that's whats going on with that part | 22:30 |
| jjlehr | "2015-05-07 16:17:08.149 ERROR oslo_messaging._drivers.common [-] Returning exception ('Connection aborted.', error(111, 'ECONNREFUSED')) to caller" | 22:30 |
| jjlehr | that one? | 22:30 |
| jjlehr | I don't see another one before that | 22:31 |
| apmelton | ah hmmmm | 22:31 |
| apmelton | that sounds like the conductor is having trouble connecting to the swarm api | 22:31 |
| jjlehr | [fedora@swarmbay-grzni55sid3y-swarm-manager-nhiw2k5czv2x ~]$ sudo docker -H tcp://127.0.0.1:2376 ps -a | 22:31 |
| jjlehr | FATA[0000] Cannot connect to the Docker daemon. Is 'docker -d' running on this host? | 22:31 |
| jjlehr | But I ran docker ps -a on both the master and node - no containers | 22:32 |
| apmelton | on the manager, can you do a systemctl | grep swarm-manager | 22:32 |
| apmelton | did it error out? | 22:32 |
| jjlehr | Trying to access the daemon over the tcp socket errored but not locally | 22:33 |
| apmelton | jjlehr: when you did a ps -a against the docker daemons, did you see any swarm containers running? | 22:33 |
| jjlehr | apmelton: I ran "systemctl | grep swarm-manager" on the master. Nothing came up | 22:33 |
| jjlehr | apmelton: No containers whatsoever | 22:34 |
| apmelton | hmmmmm | 22:34 |
| apmelton | can you gist/paste up /var/log/cloud-init-output.log | 22:34 |
| apmelton | I think that's what it's called | 22:34 |
| apmelton | there will be two /var/log/cloud-init* files, it's the non-empty one | 22:35 |
| apmelton | if they're both empty, something is /really/ wrong | 22:36 |
| jjlehr | https://gist.github.com/JanekLehr/82bd99c89cef0302bc67 | 22:36 |
| apmelton | hmmmmm | 22:36 |
| apmelton | do a "journalctl -u swarm-manager.service" | 22:37 |
| apmelton | is there anything there | 22:37 |
| apmelton | jjlehr: this needs to be on the master node btw | 22:37 |
| jjlehr | [fedora@swarmbay-grzni55sid3y-swarm-manager-nhiw2k5czv2x ~]$ journalctl -u swarm-manager.service | 22:37 |
| jjlehr | No journal files were found. | 22:37 |
| apmelton | very odd | 22:38 |
| jjlehr | It's on the swarm-manager | 22:38 |
| apmelton | do a run systemctl, do you see any errors? | 22:38 |
| apmelton | do a 'systemctl' | 22:38 |
| *** david-lyle has quit IRC | 22:38 | |
| jjlehr | var-lib-nfs-rpc_pipefs.mount loaded failed failed RPC Pipe File System | 22:38 |
| apmelton | is that the only failed? | 22:39 |
| jjlehr | That's the only one that shows failure, yes | 22:39 |
| apmelton | and there's no swarm-agent or swarm-manager service? | 22:39 |
| apmelton | jjlehr: were you running those journalctl commands as room/sudo or as the minion user? | 22:40 |
| apmelton | as root/sudo* | 22:40 |
| jjlehr | as fedora user, so not sudo | 22:40 |
| apmelton | ah yea, try "sudo journalctl -u swarm-manager.service" | 22:41 |
| *** kebray has quit IRC | 22:41 | |
| jjlehr | Ah, here is something | 22:42 |
| jjlehr | swarmbay-grzni55sid3y-swarm-manager-nhiw2k5czv2x.novalocal systemd[1]: Dependency failed for Swarm Manager. | 22:42 |
| *** EricGonczer_ has quit IRC | 22:42 | |
| apmelton | jjlehr: do a "sudo docker images" | 22:42 |
| apmelton | do you see the swarm image? | 22:43 |
| jjlehr | nope | 22:43 |
| jjlehr | no images | 22:43 |
| apmelton | what happens if you do a "sudo docker pull swarm" | 22:43 |
| jjlehr | It fails | 22:44 |
| jjlehr | Pulling repository swarm | 22:44 |
| jjlehr | FATA[0046] Get https://index.docker.io/v1/repositories/library/swarm/images: dial tcp: lookup index.docker.io: no such host | 22:44 |
| jjlehr | Looks like this is a network issue then | 22:45 |
| apmelton | jjlehr: from your devstack host, run sudo "/sbin/iptables -t nat -A POSTROUTING -o br-ex -j MASQUERADE" | 22:45 |
| apmelton | then try the pull again | 22:45 |
| jjlehr | That did the trick | 22:47 |
| apmelton | jjlehr: alright, what I'd do is delete the bay, and then start with a fresh one | 22:48 |
| jjlehr | apmelton: Ok, thank you for your help | 22:48 |
| apmelton | no problem jjlehr | 22:49 |
| jjlehr | Should bay-create return an error if it can't create the swarm manager properly? | 22:49 |
| jjlehr | As it couldn't in this case, because it failed to pull the image? | 22:49 |
| apmelton | jjlehr: ideally | 22:49 |
| jjlehr | I can create a bug for that | 22:50 |
| apmelton | jjlehr: thanks | 22:50 |
| apmelton | I was talking about this issue earlier today actually | 22:50 |
| apmelton | not only should we not call back to heat that it's complete until the swarm-manager (and agent) is running, but we might also want to wait until all of the nodes show up to the manager | 22:51 |
| *** sdake has quit IRC | 22:51 | |
| jjlehr | There seem to be a lot of missing communications between magnum and heat | 22:51 |
| jjlehr | but that's expected at this point :) | 22:51 |
| apmelton | heh yep | 22:51 |
| apmelton | we've got lots of improvements to look forward to :) | 22:52 |
| *** sdake has joined #openstack-containers | 22:52 | |
| jjlehr | I'm looking forward to working on them. | 22:54 |
| jjlehr | https://bugs.launchpad.net/magnum/+bug/1452936 | 22:57 |
| openstack | Launchpad bug 1452936 in Magnum "bay-create should fail when docker fails to pull swarm image" [Undecided,New] | 22:57 |
| apmelton | thanks jjlehr | 22:57 |
| jjlehr | apmelton: thank you! | 22:57 |
| *** jjlehr has quit IRC | 23:02 | |
| *** dboik has joined #openstack-containers | 23:23 | |
| *** dboik_ has joined #openstack-containers | 23:25 | |
| *** dboik has quit IRC | 23:28 | |
| *** Marga_ has quit IRC | 23:28 | |
| *** jjlehr has joined #openstack-containers | 23:34 | |
| *** jjlehr has quit IRC | 23:34 | |
| *** Marga_ has joined #openstack-containers | 23:42 | |
| openstackgerrit | Andrew Melton proposed openstack/magnum: Add local.sh to dev guides https://review.openstack.org/181255 | 23:48 |
| *** madhuri_ has quit IRC | 23:50 | |
| *** dims_ has joined #openstack-containers | 23:50 | |
| *** dims has quit IRC | 23:52 | |
| *** hongbin_ has joined #openstack-containers | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!