Thursday, 2018-05-03

*** egonzalez has joined #openstack-kayobe06:15
*** egonzalez has quit IRC06:22
*** egonzalez has joined #openstack-kayobe07:37
*** gkadam has joined #openstack-kayobe07:39
*** mgoddard has joined #openstack-kayobe08:02
*** ktibi has joined #openstack-kayobe08:03
*** dougsz has joined #openstack-kayobe08:12
ktibihi mgoddard I work on always on issue =>
mgoddarddo you know what the problem is?08:17
ktibiyes I found08:19
ktibimgoddard, I don't why but I think it's a docker bug08:20
ktibidocker restart container no works. I can't exec in the container after that. I need to docker stop, docker start08:20
mgoddardthat is strange08:21
mgoddarddoes the kolla_toolbox image set up the ansible user account?08:21
ktibimaybe it's since I enabled live restore08:22
mgoddardthis does feel familiar - I'm sure I've seen problems with docker cp before08:24
mgoddardare you using docker cp?08:24
ktibino but I think cp it's not the issue. I don't know why but docker restart and docker stop & docker start have not same action08:25
ktibiIf I make docker restart, the container is stuck, I can't exec bash in :/08:26
ktibiI need to stop & start :/08:26
dougszktibi, maybe you'll get some clues if you run `strace docker exec ...`?08:29
ktibidougsz, a lot of information. futex(0x1563b50, FUTEX_WAIT, 0, NULL)   = -1 EAGAIN (Resource temporarily unavailable) maybe an error :/08:32
dougszHmm. Just need to figure out what the resource is :)08:33
dougszktibi, might also be worth checking dmesg for errors08:35
mgoddardfutex is an implementation of a mutex - used for locking. Looks like it's doing a non-blocking acquisition of the lock08:35
dougszI've seem similar weirdness on a machine which turned out to have memory errors.08:36
ktibidougsz, mgoddard I think it's the live restore option08:54
mgoddardhave you tried disabling live restore?08:54
ktibiI'll try on my lab08:54
mgoddardthat issue was closed in december 1608:55
mgoddardbut this PR references it:
mgoddardsorry, that's an issue not a PR - that is likely the new issue08:56
ktibiyes I think it's the issue :/08:58
mgoddardok, better set default of live-restore to false in kayobe until it's fixed :(08:59
mgoddardalthough by default kolla-ansible installs an older docker, hopefully it is fixed in that08:59
ktibiI don't know If you have redhat account ==>
mgoddardmaybe this is why we don't see the problem in CI09:00
mgoddardnot one I can access easily09:00
ktibiok cause :09:01
ktibidocker on RHEL does not support daemon restart and live-restore.09:01
ktibidocker systemd unit file has 'MountFlag=true enabled. This is not compatible with live-restore.09:01
ktibibug open here :
mgoddarddoes this apply to centos too?09:03
mgoddardok. Could you submit a PR to change the default of live-restore?09:03
ktibiyes I'll make09:05
ktibiI disable live-restore and some container can't start :/09:05
ktibiError response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /dev/console: input/output error\"": unknown09:06
mgoddardmaybe you need to destroy and recreate?09:06
ktibiyes maybe09:09
ktibiyou can destroy one container with kayobe ?09:09
ktibimgoddard, environnement for live-restore is Red Hat Enterprise Linux Server 7.3 & docker-1.12.5-14.el7.x86_6409:16
ktibiin the solution on redhat09:16
ktibimaybe live restore is support on 7.5 and docker-ce 17 :/09:17
ktibimgoddard, ok same error when kolla try to recreate, maybe issue with the volume :/09:19
ktibimgoddard, I'll change centos coreOS T_T09:49
ktibito coreos*09:49
mgoddardktibi: kayobe won't work with coreos...09:51
ktibiyes I know ^^09:51
mgoddardare you sure there is no way to fix docker?09:52
ktibibut since I works on kayobe, I can see redhat and docker are not really compatible ....09:52
ktibiI prune all docker, deleted all volumes, disable live-restore.09:53
mgoddardno? what other issues have you seen?09:53
ktibiand now I 'll try to redeploy09:53
ktibimgoddard, I have issue with XFS mouting,09:53
mgoddardis there a bug report for it?09:54
mgoddardredhat use docker for openshift, I'd be surprised if it is broken beyond repair09:57
*** dougsz has quit IRC10:29
*** dougsz has joined #openstack-kayobe11:02
ktibimgoddard, reboot the ctrl fix the creation of container :/11:45
ktibiso I can confirm ! without live-restore, I can exec after a restart.11:46
mgoddardok, that's good to know11:48
ktibithe issue 35873 on moby is exactly my issue11:49
mgoddardyeah, looks like it11:49
*** ktibi has quit IRC12:55
*** ktibi_ has joined #openstack-kayobe12:55
*** ktibi_ is now known as ktibi12:56
ktibimgoddard, I try to checkout my kolla-ansible repo with kayo13:50
ktibiso I make a kayobe control host upgrade13:51
ktibibut very strange ==> [ERROR]: failed to download the file: Failed to validate the SSL certificate for
ktibiso I want to add an insecure tag on utils.galaxy_install("requirements.yml", "ansible/roles",13:52
mgoddardwe have seen that before too13:53
mgoddardI think it was fixed in ansible 2.4+13:53
mgoddardalthough it doesn't always seem to be a problem13:53
ktibimgoddard, yes need to use : ansible-galaxy install -r requirements.yml -p ansible/roles/ --ignore-errors --ignore-certs13:59
ktibimgoddard, what is the best command for you to checkout kolla-ansible ?14:00
mgoddardktibi: control host upgrade is correct14:00
ktibibecause I checkout src/kolla-ansible but no update in the venv14:00
mgoddardyou can also do kayobe playbook run ansible/kolla-ansible.yml -t install14:00
ktibiok great, because I searched how disable the galaxy install in cli ^^14:02
openstackgerritDoug Szumski proposed openstack/kayobe master: Add preliminary support for Monasca APIs
ktibiyankcrime, hi, news about backup of maria cluster ?14:20
yankcrimektibi: no, i've not had time to give it any further attention unfortunately14:22
yankcrimeto be clear this is functionality in kolla-ansible, not indirectly via kayobe14:22
ktibiyankcrime, ok ;)14:26
*** egonzalez has quit IRC15:10
openstackgerritKevin Tibi proposed openstack/kayobe master: Disable live-restore by default
openstackgerritMerged openstack/kayobe master: Add preliminary support for Monasca APIs
*** dougsz has quit IRC16:58
*** gkadam has quit IRC17:02
*** mgoddard has quit IRC17:03
*** mgoddard has joined #openstack-kayobe17:44
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job
*** mgoddard has quit IRC18:53
*** openstackgerrit has quit IRC19:05
*** mgoddard has joined #openstack-kayobe19:16
*** openstackgerrit has joined #openstack-kayobe19:18
openstackgerritMark Goddard proposed openstack/kayobe master: WIP: Add seed VM provisioning CI job
*** mgoddard has quit IRC19:23
*** mgoddard has joined #openstack-kayobe19:38
*** mgoddard has quit IRC19:52
*** ktibi has quit IRC20:20

Generated by 2.15.3 by Marius Gedminas - find it at!