| *** egonzalez has joined #openstack-kayobe | 06:15 | |
| *** egonzalez has quit IRC | 06:22 | |
| *** egonzalez has joined #openstack-kayobe | 07:37 | |
| *** gkadam has joined #openstack-kayobe | 07:39 | |
| *** mgoddard has joined #openstack-kayobe | 08:02 | |
| *** ktibi has joined #openstack-kayobe | 08:03 | |
| *** dougsz has joined #openstack-kayobe | 08:12 | |
| ktibi | hi mgoddard I work on always on issue => http://paste.openstack.org/show/720261/ | 08:14 |
|---|---|---|
| mgoddard | do you know what the problem is? | 08:17 |
| ktibi | yes I found | 08:19 |
| ktibi | mgoddard, I don't why but I think it's a docker bug | 08:20 |
| ktibi | http://paste.openstack.org/show/720262/ | 08:20 |
| ktibi | docker restart container no works. I can't exec in the container after that. I need to docker stop, docker start | 08:20 |
| mgoddard | that is strange | 08:21 |
| mgoddard | does the kolla_toolbox image set up the ansible user account? | 08:21 |
| ktibi | yes | 08:21 |
| ktibi | maybe it's since I enabled live restore | 08:22 |
| ktibi | https://forums.docker.com/t/unable-to-find-user-root-no-matching-entries-in-passwd-file/26545/2 | 08:22 |
| mgoddard | this does feel familiar - I'm sure I've seen problems with docker cp before | 08:24 |
| mgoddard | are you using docker cp? | 08:24 |
| ktibi | no but I think cp it's not the issue. I don't know why but docker restart and docker stop & docker start have not same action | 08:25 |
| ktibi | ok | 08:25 |
| ktibi | If I make docker restart, the container is stuck, I can't exec bash in :/ | 08:26 |
| ktibi | I need to stop & start :/ | 08:26 |
| dougsz | ktibi, maybe you'll get some clues if you run `strace docker exec ...`? | 08:29 |
| ktibi | dougsz, a lot of information. futex(0x1563b50, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable) maybe an error :/ | 08:32 |
| dougsz | Hmm. Just need to figure out what the resource is :) | 08:33 |
| dougsz | ktibi, might also be worth checking dmesg for errors | 08:35 |
| mgoddard | futex is an implementation of a mutex - used for locking. Looks like it's doing a non-blocking acquisition of the lock | 08:35 |
| dougsz | I've seem similar weirdness on a machine which turned out to have memory errors. | 08:36 |
| ktibi | dougsz, mgoddard I think it's the live restore option | 08:54 |
| ktibi | https://github.com/moby/moby/issues/29342 | 08:54 |
| mgoddard | have you tried disabling live restore? | 08:54 |
| ktibi | I'll try on my lab | 08:54 |
| mgoddard | that issue was closed in december 16 | 08:55 |
| mgoddard | but this PR references it: https://github.com/moby/moby/issues/35873 | 08:55 |
| mgoddard | sorry, that's an issue not a PR - that is likely the new issue | 08:56 |
| ktibi | yes I think it's the issue :/ | 08:58 |
| mgoddard | ok, better set default of live-restore to false in kayobe until it's fixed :( | 08:59 |
| mgoddard | although by default kolla-ansible installs an older docker, hopefully it is fixed in that | 08:59 |
| ktibi | I don't know If you have redhat account ==> https://access.redhat.com/solutions/2991041 | 09:00 |
| mgoddard | maybe this is why we don't see the problem in CI | 09:00 |
| mgoddard | not one I can access easily | 09:00 |
| ktibi | ok cause : | 09:01 |
| ktibi | docker on RHEL does not support daemon restart and live-restore. | 09:01 |
| ktibi | docker systemd unit file has 'MountFlag=true enabled. This is not compatible with live-restore. | 09:01 |
| ktibi | bug open here : https://github.com/haxorof/ansible-role-docker-ce/issues/33 | 09:02 |
| mgoddard | :( | 09:03 |
| mgoddard | does this apply to centos too? | 09:03 |
| ktibi | yes | 09:03 |
| mgoddard | ok. Could you submit a PR to change the default of live-restore? | 09:03 |
| ktibi | yes I'll make | 09:05 |
| ktibi | I disable live-restore and some container can't start :/ | 09:05 |
| ktibi | Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /dev/console: input/output error\"": unknown | 09:06 |
| mgoddard | thanks | 09:06 |
| mgoddard | maybe you need to destroy and recreate? | 09:06 |
| ktibi | yes maybe | 09:09 |
| ktibi | you can destroy one container with kayobe ? | 09:09 |
| ktibi | mgoddard, environnement for live-restore is Red Hat Enterprise Linux Server 7.3 & docker-1.12.5-14.el7.x86_64 | 09:16 |
| ktibi | in the solution on redhat | 09:16 |
| ktibi | maybe live restore is support on 7.5 and docker-ce 17 :/ | 09:17 |
| mgoddard | perhaps | 09:18 |
| ktibi | mgoddard, ok same error when kolla try to recreate, maybe issue with the volume :/ | 09:19 |
| ktibi | http://paste.openstack.org/show/720272/ | 09:19 |
| ktibi | mgoddard, I'll change centos coreOS T_T | 09:49 |
| ktibi | to coreos* | 09:49 |
| mgoddard | ktibi: kayobe won't work with coreos... | 09:51 |
| ktibi | yes I know ^^ | 09:51 |
| mgoddard | are you sure there is no way to fix docker? | 09:52 |
| ktibi | but since I works on kayobe, I can see redhat and docker are not really compatible .... | 09:52 |
| ktibi | I prune all docker, deleted all volumes, disable live-restore. | 09:53 |
| mgoddard | no? what other issues have you seen? | 09:53 |
| ktibi | and now I 'll try to redeploy | 09:53 |
| ktibi | mgoddard, I have issue with XFS mouting, | 09:53 |
| mgoddard | is there a bug report for it? | 09:54 |
| mgoddard | redhat use docker for openshift, I'd be surprised if it is broken beyond repair | 09:57 |
| *** dougsz has quit IRC | 10:29 | |
| *** dougsz has joined #openstack-kayobe | 11:02 | |
| ktibi | mgoddard, reboot the ctrl fix the creation of container :/ | 11:45 |
| ktibi | so I can confirm ! without live-restore, I can exec after a restart. | 11:46 |
| mgoddard | ok, that's good to know | 11:48 |
| ktibi | the issue 35873 on moby is exactly my issue | 11:49 |
| mgoddard | yeah, looks like it | 11:49 |
| *** ktibi has quit IRC | 12:55 | |
| *** ktibi_ has joined #openstack-kayobe | 12:55 | |
| *** ktibi_ is now known as ktibi | 12:56 | |
| ktibi | mgoddard, I try to checkout my kolla-ansible repo with kayo | 13:50 |
| ktibi | so I make a kayobe control host upgrade | 13:51 |
| ktibi | but very strange ==> [ERROR]: failed to download the file: Failed to validate the SSL certificate for github.com:443. | 13:51 |
| ktibi | so I want to add an insecure tag on utils.galaxy_install("requirements.yml", "ansible/roles", | 13:52 |
| mgoddard | we have seen that before too | 13:53 |
| mgoddard | I think it was fixed in ansible 2.4+ | 13:53 |
| mgoddard | although it doesn't always seem to be a problem | 13:53 |
| ktibi | mgoddard, yes need to use : ansible-galaxy install -r requirements.yml -p ansible/roles/ --ignore-errors --ignore-certs | 13:59 |
| ktibi | mgoddard, what is the best command for you to checkout kolla-ansible ? | 14:00 |
| mgoddard | ktibi: control host upgrade is correct | 14:00 |
| ktibi | because I checkout src/kolla-ansible but no update in the venv | 14:00 |
| mgoddard | you can also do kayobe playbook run ansible/kolla-ansible.yml -t install | 14:00 |
| ktibi | ok great, because I searched how disable the galaxy install in cli ^^ | 14:02 |
| openstackgerrit | Doug Szumski proposed openstack/kayobe master: Add preliminary support for Monasca APIs https://review.openstack.org/566081 | 14:10 |
| ktibi | yankcrime, hi, news about backup of maria cluster ? | 14:20 |
| yankcrime | ktibi: no, i've not had time to give it any further attention unfortunately | 14:22 |
| yankcrime | to be clear this is functionality in kolla-ansible, not indirectly via kayobe | 14:22 |
| ktibi | yankcrime, ok ;) | 14:26 |
| *** egonzalez has quit IRC | 15:10 | |
| openstackgerrit | Kevin Tibi proposed openstack/kayobe master: Disable live-restore by default https://review.openstack.org/566101 | 15:50 |
| openstackgerrit | Merged openstack/kayobe master: Add preliminary support for Monasca APIs https://review.openstack.org/566081 | 16:01 |
| *** dougsz has quit IRC | 16:58 | |
| *** gkadam has quit IRC | 17:02 | |
| *** mgoddard has quit IRC | 17:03 | |
| *** mgoddard has joined #openstack-kayobe | 17:44 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.openstack.org/559706 | 17:50 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.openstack.org/559706 | 18:30 |
| *** mgoddard has quit IRC | 18:53 | |
| *** openstackgerrit has quit IRC | 19:05 | |
| *** mgoddard has joined #openstack-kayobe | 19:16 | |
| *** openstackgerrit has joined #openstack-kayobe | 19:18 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job https://review.openstack.org/559706 | 19:18 |
| *** mgoddard has quit IRC | 19:23 | |
| *** mgoddard has joined #openstack-kayobe | 19:38 | |
| *** mgoddard has quit IRC | 19:52 | |
| *** ktibi has quit IRC | 20:20 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!