SamYaple | RA_: if you are forking it anyway, it would be very easy to modify it to not require docker 1.9+ (one line change i think) | 00:01 |
---|---|---|
*** weiyu has joined #kolla | 00:06 | |
*** salv-orl_ has quit IRC | 00:15 | |
*** diogogmt has quit IRC | 00:17 | |
*** sdake has joined #kolla | 00:24 | |
*** weiyu has quit IRC | 00:27 | |
*** blahRus has quit IRC | 00:27 | |
*** sdake_ has joined #kolla | 00:28 | |
*** sdake has quit IRC | 00:29 | |
*** sdake has joined #kolla | 00:30 | |
*** sdake_ has quit IRC | 00:33 | |
*** mbound has quit IRC | 00:34 | |
*** sdake_ has joined #kolla | 00:35 | |
*** sdake has quit IRC | 00:38 | |
*** mbound has joined #kolla | 00:42 | |
*** RA_ has quit IRC | 00:42 | |
asalkeld | sdake_: if you about, could you knock this in please (fixes unit tests): https://review.openstack.org/#/c/272836/1 | 00:43 |
*** frzn_ has quit IRC | 00:44 | |
*** mbound has quit IRC | 00:44 | |
mandre | asalkeld, done | 00:49 |
openstackgerrit | Merged openstack/kolla-mesos: Make sure any config overrides are reset https://review.openstack.org/272836 | 00:50 |
asalkeld | thanks mandre | 00:50 |
*** dave-mccowan has quit IRC | 00:53 | |
mandre | hmm, github has been down for more than 20 min now | 00:54 |
*** dims_ has quit IRC | 00:55 | |
openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Fix cinder endpoint registration https://review.openstack.org/269959 | 00:58 |
openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add api_interface and service_name to hostvars https://review.openstack.org/272776 | 00:58 |
*** dave-mccowan has joined #kolla | 00:58 | |
asalkeld | mandre: yikes | 00:59 |
SamYaple | jpeeler: hear that! github is down :P | 01:08 |
*** Liuqing has joined #kolla | 01:09 | |
*** diogogmt has joined #kolla | 01:14 | |
britthouser | Nice Unicorn though! | 01:22 |
Allen_Gao | Oops! | 01:23 |
*** neilus has quit IRC | 01:26 | |
openstackgerrit | Merged openstack/kolla-mesos: Fix cinder endpoint registration https://review.openstack.org/269959 | 01:29 |
dmsimard | git.openstack.org for the win | 01:29 |
*** jasonsb has joined #kolla | 01:31 | |
*** ssurana has quit IRC | 01:32 | |
*** ssurana has joined #kolla | 01:32 | |
*** ssurana has quit IRC | 01:37 | |
*** neilus has joined #kolla | 01:39 | |
*** hr_ has joined #kolla | 01:39 | |
SamYaple | dmsimard: that goes down more than github :P but its normally up when github is down | 01:44 |
*** weiyu has joined #kolla | 01:44 | |
*** mbound has joined #kolla | 01:45 | |
*** frzn_ has joined #kolla | 01:46 | |
*** salv-orlando has joined #kolla | 01:47 | |
*** shakamunyi has joined #kolla | 01:48 | |
*** sdake_ has quit IRC | 01:49 | |
*** neilus has quit IRC | 01:49 | |
*** mbound has quit IRC | 01:50 | |
*** salv-orlando has quit IRC | 01:57 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Cleanup mariadb and make compatiable with mesos https://review.openstack.org/272160 | 02:06 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Add labels to docker module https://review.openstack.org/273309 | 02:06 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Update all bootstraps to use Label https://review.openstack.org/273310 | 02:06 |
*** shakamunyi has quit IRC | 02:10 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Add labels to docker module https://review.openstack.org/273309 | 02:10 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Update all bootstraps to use Label https://review.openstack.org/273310 | 02:10 |
*** shakamunyi has joined #kolla | 02:11 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Update all bootstraps to use Label https://review.openstack.org/273310 | 02:12 |
weiyu | when I deploy ceph, meet a question | 02:17 |
*** RA_ has joined #kolla | 02:17 | |
weiyu | docker/ceph/ceph-mon/extend_start.sh must use $(hostname -s) replace $(hostname) | 02:19 |
weiyu | :SamYaple | 02:21 |
*** tobe has joined #kolla | 02:22 | |
*** unicell has quit IRC | 02:23 | |
*** sdake has joined #kolla | 02:29 | |
*** sdake has quit IRC | 02:29 | |
*** neilus has joined #kolla | 02:33 | |
*** neilus has quit IRC | 02:37 | |
*** kjelly has quit IRC | 02:41 | |
*** kjelly has joined #kolla | 02:42 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Rework log output if pull fails https://review.openstack.org/272099 | 02:46 |
*** rhallisey has quit IRC | 02:48 | |
*** hr_ has quit IRC | 02:52 | |
SamYaple | weiyu: im confused, what is your question again? | 02:55 |
*** sdake has joined #kolla | 02:55 | |
weiyu | $(hostname) return host's fqdn in centos7 | 02:58 |
SamYaple | for the ceph mon, that is desired, no? | 02:59 |
SamYaple | it does the same on ubuntu | 02:59 |
frzn_ | SamYaple: do you sleep ? | 02:59 |
SamYaple | frzn_: yea. sometimes | 02:59 |
frzn_ | gotcha. | 02:59 |
weiyu | but ansible/roles/ceph/templates/ceph-mon.json.j2 ansible_hostanem = host's short name | 02:59 |
*** signed8b_ has quit IRC | 02:59 | |
*** Jeffrey4l has joined #kolla | 03:00 | |
weiyu | This make my ceph_mon don't start | 03:02 |
frzn_ | weiyu: you can use ansible_fqdn | 03:02 |
frzn_ | instead of ansible_hostname | 03:02 |
frzn_ | br.. | 03:02 |
frzn_ | brb* | 03:02 |
weiyu | I used ansible_nodename | 03:03 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Add labels to docker module https://review.openstack.org/273309 | 03:05 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Update all bootstraps to use Label https://review.openstack.org/273310 | 03:05 |
*** mdnadeem has joined #kolla | 03:08 | |
SamYaple | weiyu: would your suggested fix be to use the FQDN or to set hostname -s? | 03:09 |
*** frzn_ has quit IRC | 03:09 | |
weiyu | I suggest set hostname -s | 03:09 |
*** RA_ has quit IRC | 03:09 | |
SamYaple | weiyu: ok lets do that. | 03:10 |
SamYaple | thats probably best for backwards compatibility | 03:10 |
weiyu | $(hostname -s) equal to ansible_hostname, but $(hostname) isn't | 03:12 |
SamYaple | weiyu: yes. i agree | 03:12 |
weiyu | :SamYaple I will do that | 03:13 |
Jeffrey4l | About this https://bugs.launchpad.net/kolla/+bug/1538136 this issue really exist. Why you mark it Invalid? | 03:20 |
openstack | Launchpad bug 1538136 in kolla "mariadb can not start from a host reboot" [Undecided,Invalid] - Assigned to Jeffrey Zhang (jeffrey4l) | 03:20 |
Jeffrey4l | SamYaple, ^ | 03:20 |
*** neilus has joined #kolla | 03:34 | |
*** Marga_ has quit IRC | 03:38 | |
*** neilus has quit IRC | 03:39 | |
*** salv-orlando has joined #kolla | 03:43 | |
*** hr_ has joined #kolla | 03:43 | |
*** dave-mccowan has quit IRC | 03:49 | |
*** sdake_ has joined #kolla | 03:50 | |
*** sdake has quit IRC | 03:51 | |
*** coolsvap|away is now known as coolsvap | 03:54 | |
*** salv-orlando has quit IRC | 03:56 | |
sdake_ | Jeffrey4l i fixed that bug state up for you | 03:58 |
sdake_ | note you are welcome to do tht yourself i you want | 03:58 |
*** ssurana has joined #kolla | 04:00 | |
*** ssurana has left #kolla | 04:02 | |
Jeffrey4l | thanks sdake_ , I know I can change the state. But I just want to know the reason why SamYaple change it. | 04:03 |
sdake_ | it is a valid bug | 04:03 |
sdake_ | he used yoru bug id in his fix | 04:04 |
sdake_ | which looks correct to me | 04:04 |
sdake_ | read hte last comment he made on the gerrit tracker | 04:04 |
sdake_ | he just forgot to change the bug id when he wsa done fixing it :) | 04:04 |
sdake_ | or rather change the bug states | 04:04 |
sdake_ | i think what happened is he nerd raged because popele submit that patch every couple of weeks :) | 04:05 |
sdake_ | using labels is the way to go | 04:05 |
*** sdake_ is now known as sdake | 04:05 | |
sdake | but then again who knows what goes on in sam's head :) | 04:06 |
sdake | or mine or yours :) | 04:06 |
Jeffrey4l | Got it . Thanks a lot. | 04:06 |
sdake | sure happy to help resolve that conflcit ;) | 04:06 |
sdake | no I AM YOUR FATHER | 04:19 |
*** Liuqing has quit IRC | 04:20 | |
*** Liuqing has joined #kolla | 04:22 | |
*** unicell has joined #kolla | 04:29 | |
SamYaple | Jeffrey4l: I don't recall marking that invalid so it was probably a slip of the mouse | 04:30 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Rework log output if pull fails https://review.openstack.org/272099 | 04:33 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Enable find_disks module to match by filesystem label https://review.openstack.org/272409 | 04:34 |
Jeffrey4l | SamYaple, roger that. I have found your PS. | 04:36 |
openstackgerrit | Merged openstack/kolla-mesos: Add api_interface and service_name to hostvars https://review.openstack.org/272776 | 04:40 |
*** tpot has quit IRC | 04:46 | |
*** frzn_ has joined #kolla | 04:47 | |
*** weiyu has quit IRC | 04:51 | |
*** coolsvap is now known as coolsvap|away | 04:51 | |
*** Marga_ has joined #kolla | 04:58 | |
*** coolsvap|away is now known as coolsvap | 04:59 | |
*** Marga_ has quit IRC | 05:02 | |
*** unicell has quit IRC | 05:08 | |
*** RA_ has joined #kolla | 05:09 | |
*** tpot has joined #kolla | 05:10 | |
*** weiyu has joined #kolla | 05:19 | |
sdake | samyaple about? | 05:24 |
openstackgerrit | weiyu proposed openstack/kolla: reset $(hostname) to $(hostname -s) in ceph extend_start.sh https://review.openstack.org/273340 | 05:24 |
SamYaple | sdake: yea | 05:29 |
*** reddy has joined #kolla | 05:30 | |
*** tpot has quit IRC | 05:46 | |
weiyu | hello,sdake and SamYaple what's mean that "please file a techdebt bug " | 05:47 |
*** sdake_ has joined #kolla | 05:48 | |
*** unicell has joined #kolla | 05:48 | |
*** sdake has quit IRC | 05:49 | |
SamYaple | weiyu: Don't worry about it. sdake was asking you to file a bug but its not needed since you are preseving existing behaviour | 05:56 |
*** tpot has joined #kolla | 05:57 | |
weiyu | Oh ,I see | 05:57 |
weiyu | Tanks | 05:57 |
*** sdake_ has quit IRC | 06:00 | |
*** sdake has joined #kolla | 06:01 | |
*** sdake_ has joined #kolla | 06:08 | |
*** sdake has quit IRC | 06:10 | |
*** tpot has quit IRC | 06:16 | |
*** Marga_ has joined #kolla | 06:17 | |
*** Marga_ has quit IRC | 06:17 | |
*** Marga_ has joined #kolla | 06:18 | |
*** frzn_ has quit IRC | 06:19 | |
*** tfukushima has joined #kolla | 06:29 | |
*** stvnoyes has quit IRC | 06:33 | |
*** stvnoyes has joined #kolla | 06:34 | |
*** salv-orlando has joined #kolla | 06:38 | |
sdake_ | it was a "I dont know if its necessary" request fwiw :) | 06:44 |
sdake_ | weiyu ^^ | 06:44 |
*** RA_ has quit IRC | 06:48 | |
sdake_ | well time to ptfo | 06:48 |
sdake_ | night folks | 06:48 |
*** salv-orlando has quit IRC | 06:49 | |
weiyu | sdake_ It is backwards compatibility ^^ | 06:55 |
sdake_ | roger | 06:55 |
sdake_ | i just dont want upgrades to turn into a charlie foxtrot of a bunch of incompatible changes so i'd like to track those things which will make upgrades not work | 06:56 |
*** unicell has quit IRC | 06:56 | |
weiyu | I see | 07:01 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix swift build issue https://review.openstack.org/272864 | 07:07 |
openstackgerrit | Merged openstack/kolla: reset $(hostname) to $(hostname -s) in ceph extend_start.sh https://review.openstack.org/273340 | 07:08 |
*** unicell has joined #kolla | 07:09 | |
sdake_ | SamYaple ping | 07:10 |
sdake_ | how do i pull the registery v2 dev version | 07:10 |
sdake_ | all i see is 2.2 | 07:10 |
sdake_ | and 2.21 | 07:11 |
SamYaple | sdake_: just build the dockerfile | 07:16 |
SamYaple | from master | 07:16 |
*** salv-orlando has joined #kolla | 07:26 | |
*** unicell has quit IRC | 07:44 | |
*** coolsvap is now known as coolsvap|away | 08:00 | |
*** fgimenez has joined #kolla | 08:03 | |
*** fgimenez has quit IRC | 08:03 | |
*** fgimenez has joined #kolla | 08:03 | |
*** alisonh has quit IRC | 08:08 | |
*** salv-orlando has quit IRC | 08:10 | |
*** weiyu has quit IRC | 08:14 | |
*** barra204 has joined #kolla | 08:17 | |
*** kejlly_ has joined #kolla | 08:17 | |
*** weiyu has joined #kolla | 08:17 | |
*** dmsimard1 has joined #kolla | 08:18 | |
*** akwasnie has joined #kolla | 08:18 | |
*** athomas_ has joined #kolla | 08:18 | |
*** bmace_ has joined #kolla | 08:19 | |
*** kjelly has quit IRC | 08:19 | |
*** dmsimard has quit IRC | 08:19 | |
*** shakamunyi has quit IRC | 08:19 | |
*** athomas has quit IRC | 08:19 | |
*** mdnadeem has quit IRC | 08:19 | |
*** bmace has quit IRC | 08:19 | |
*** sdake_ has quit IRC | 08:19 | |
*** mdnadeem has joined #kolla | 08:20 | |
*** sdake has joined #kolla | 08:21 | |
akwasnie | morning guys;) | 08:21 |
*** bmace_ has quit IRC | 08:23 | |
*** alisonh has joined #kolla | 08:26 | |
nihilifer | eo/ | 08:30 |
*** gfidente has joined #kolla | 08:35 | |
*** sdake has quit IRC | 08:40 | |
*** kproskurin has joined #kolla | 08:41 | |
*** diga has joined #kolla | 08:42 | |
*** sdake has joined #kolla | 08:47 | |
*** ppetit has joined #kolla | 08:48 | |
*** rmart04 has joined #kolla | 08:52 | |
*** mbound has joined #kolla | 08:54 | |
*** shardy has joined #kolla | 09:00 | |
*** achanda has quit IRC | 09:01 | |
*** ppetit_ has joined #kolla | 09:06 | |
*** ppetit has quit IRC | 09:07 | |
*** ppetit_ is now known as ppetit | 09:07 | |
*** tzn has quit IRC | 09:10 | |
*** salv-orlando has joined #kolla | 09:11 | |
*** ppetit has quit IRC | 09:15 | |
*** egonzalez90 has joined #kolla | 09:16 | |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Improve logging in deploy.py https://review.openstack.org/272992 | 09:20 |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Improve logging in deploy.py https://review.openstack.org/272992 | 09:22 |
*** egonzalez90 is now known as egonzalez | 09:47 | |
*** alisonh has quit IRC | 10:06 | |
openstackgerrit | Merged openstack/kolla: Remove the libvirt socket https://review.openstack.org/273216 | 10:06 |
*** salv-orl_ has joined #kolla | 10:06 | |
*** ppetit has joined #kolla | 10:08 | |
*** salv-orlando has quit IRC | 10:09 | |
*** alisonh has joined #kolla | 10:10 | |
*** Liuqing has quit IRC | 10:13 | |
openstackgerrit | Merged openstack/kolla: Fix swift build issue https://review.openstack.org/272864 | 10:14 |
openstackgerrit | Merged openstack/kolla: Add the missed stub upgrade.yml https://review.openstack.org/272897 | 10:15 |
openstackgerrit | Michal Rostecki proposed openstack/kolla: Use trusts in heat.conf https://review.openstack.org/236198 | 10:15 |
*** weiyu has quit IRC | 10:29 | |
*** weiyu has joined #kolla | 10:30 | |
*** akwasnie has quit IRC | 10:42 | |
*** fgimenez has quit IRC | 10:48 | |
*** fgimenez has joined #kolla | 10:48 | |
*** fgimenez has quit IRC | 10:48 | |
*** fgimenez has joined #kolla | 10:48 | |
*** weiyu has quit IRC | 11:01 | |
*** dims has joined #kolla | 11:07 | |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add heat config https://review.openstack.org/271978 | 11:14 |
*** jmccarthy has joined #kolla | 11:26 | |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Improve logging in deploy.py https://review.openstack.org/272992 | 11:29 |
*** tzn has joined #kolla | 11:30 | |
*** salv-orl_ has quit IRC | 11:34 | |
*** tobe has quit IRC | 11:41 | |
*** tfukushima has quit IRC | 11:47 | |
*** athomas_ has quit IRC | 11:53 | |
*** rhallisey has joined #kolla | 11:54 | |
*** pbourke has quit IRC | 12:03 | |
*** pbourke has joined #kolla | 12:03 | |
*** neilus has joined #kolla | 12:07 | |
*** neilus has quit IRC | 12:11 | |
*** egonzalez has quit IRC | 12:13 | |
*** ppetit has quit IRC | 12:13 | |
*** iceyao has quit IRC | 12:14 | |
*** neilus has joined #kolla | 12:15 | |
*** neilus has quit IRC | 12:19 | |
*** salv-orlando has joined #kolla | 12:21 | |
*** weiyu has joined #kolla | 12:22 | |
Jeffrey4l | sdake, I am sure that the qemu process will be killed even when using pid=host mode. | 12:28 |
Jeffrey4l | when restart docker service. | 12:28 |
Jeffrey4l | we talked this on last meeting. | 12:28 |
*** ppetit has joined #kolla | 12:37 | |
*** dave-mccowan has joined #kolla | 12:41 | |
*** tobe has joined #kolla | 12:41 | |
*** dwalsh has joined #kolla | 12:41 | |
*** dwalsh has quit IRC | 12:43 | |
*** hr_ has quit IRC | 12:43 | |
*** dwalsh has joined #kolla | 12:44 | |
*** dims has quit IRC | 12:47 | |
*** diga has quit IRC | 12:48 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the machine-id which cause nova-compute raise exception https://review.openstack.org/273508 | 12:48 |
*** Allen_Gao has quit IRC | 12:53 | |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add heat config https://review.openstack.org/271978 | 12:54 |
*** fgimenez has quit IRC | 12:55 | |
*** fgimenez has joined #kolla | 12:57 | |
*** fgimenez has quit IRC | 12:57 | |
*** fgimenez has joined #kolla | 12:57 | |
sdake | morning | 12:59 |
sdake | ya i didn't try docker restart | 12:59 |
sdake | just restart of services under docker control | 12:59 |
sdake | the fact that docker kills everyhting it starts, nothing can be dne about that | 12:59 |
sdake | docker is meant not to be restarted | 12:59 |
Jeffrey4l | yep | 13:02 |
Jeffrey4l | So when upgrade the services, this should be needed more consideration to the case. | 13:03 |
*** tobe has quit IRC | 13:09 | |
*** Allen_Gao has joined #kolla | 13:10 | |
*** sdake has quit IRC | 13:16 | |
*** sdake has joined #kolla | 13:18 | |
sdake | some cat sent me a job offer for "between 42000 euro and 80000 euro" | 13:18 |
sdake | i'm thrillled! | 13:18 |
sdake | Jeffrey4l i dont have a good answer for upgrading docker without blowing up the cluster | 13:19 |
sdake | i am aware of the problem | 13:20 |
sdake | but i am not taking care of it | 13:20 |
sdake | because there is ntohing i can really do aout it | 13:20 |
sdake | its a fundamental flaw in docker at this point in time | 13:20 |
*** iceyao has joined #kolla | 13:21 | |
*** signed8b_ has joined #kolla | 13:21 | |
*** weiyu has quit IRC | 13:22 | |
Jeffrey4l | Yep. btw, upgrade the nova-libvirt container(libvirt service) should also encounter this issue, too. | 13:23 |
sdake | disagree | 13:24 |
sdake | test it and prove it | 13:24 |
sdake | sdake.io check out my proof in my blog post | 13:24 |
sdake | granted that was on docker 1.4.1 ;) | 13:25 |
Jeffrey4l | I tested. qemu process is start by libvirt in nova-libvirt containers. restart the nova-libvirt container, the qemu process will be killed. | 13:25 |
sdake | how are you rstarting the nova-libvirt container? | 13:25 |
Jeffrey4l | `docker restart nova-libvirt` | 13:26 |
sdake | i tested at late as 1.8.3 with a redeploy and that works | 13:26 |
sdake | ok well docker restart is not wht we do in kolla | 13:26 |
sdake | we kill the container | 13:26 |
sdake | the nwe start it | 13:26 |
sdake | test real world redeploy - we need to know if that broke | 13:27 |
Jeffrey4l | Isn't it the same? Kill the container meanins all the processes in the container being killed. | 13:27 |
sdake | deploy once, deploy ag ain | 13:27 |
sdake | no docker can't kill processes in qemu space because it doesn't know about them | 13:27 |
sdake | it can kill libvirt | 13:27 |
sdake | because it knows about that | 13:27 |
sdake | qemu it doens't know about because qemu are not children of libvirt on libirt death, they are inherited by init | 13:28 |
Jeffrey4l | re-deploy won't hurt. But upgrade will. | 13:28 |
sdake | libvirt doen't | 13:28 |
Jeffrey4l | But qemu and libvirt are in the same mount namespace. | 13:28 |
sdake | how is upgrade different then redeploy | 13:28 |
sdake | in redeploy we kill and start | 13:28 |
sdake | in upgrade we kill and start | 13:28 |
Jeffrey4l | ok. let me try the kill and start first. Maybe it is different with my thinking. | 13:29 |
sdake | now maybe container rm -f will destroy qemu, i haven't tried that | 13:29 |
sdake | i was focused in my blog post can libvirt be upgraded without interrupting existing vms, and the answer was yes | 13:30 |
Jeffrey4l | sdake, http://paste.openstack.org/show/485266/ see qemu process is killed when the nova-libvirt container is killed. | 13:31 |
*** RA_ has joined #kolla | 13:31 | |
*** Liuqing has joined #kolla | 13:32 | |
Jeffrey4l | sdake, if there is not docker container stuff. stop/upgrade the libvirt is OK and the qemu process will be remain. | 13:32 |
sdake | well that is a change in how docker operates | 13:33 |
sdake | have you read my blog post? | 13:33 |
Jeffrey4l | sdake, no. I read it now. | 13:33 |
sdake | maybe there is a different command besides kill | 13:33 |
sdake | just waking up here, give me a little bit to get brain booted | 13:33 |
sdake | which version of docker are on | 13:34 |
sdake | Jeffrey4l ^^^ | 13:34 |
Jeffrey4l | 1.9.1 | 13:34 |
*** signed8__ has joined #kolla | 13:41 | |
*** signed8b_ has quit IRC | 13:42 | |
*** v1k0d3n has quit IRC | 13:43 | |
*** Liuqing has quit IRC | 13:44 | |
*** weiyu has joined #kolla | 13:44 | |
*** Liuqing has joined #kolla | 13:47 | |
*** v1k0d3n has joined #kolla | 13:48 | |
*** weiyu has quit IRC | 13:49 | |
*** RA_ has quit IRC | 13:52 | |
*** Liuqing has quit IRC | 13:52 | |
sdake | Jeffrey4l is libvirt still running in pid=host mode? | 13:53 |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 13:53 |
Jeffrey4l | sdake, I deploy by using kolla-ansible. And I double checked it just now. It is using pid=host mode. | 13:54 |
sdake | Jeffrey4l i see kolla docker says so | 13:54 |
sdake | rather the ansible code | 13:54 |
sdake | can you introspect th econtainer and mmake sure | 13:54 |
Jeffrey4l | yep. I checked it using the `docker inspect` | 13:55 |
Jeffrey4l | "PidMode": "host", | 13:55 |
Jeffrey4l | sdake, & | 13:55 |
sdake | docker odesn't reap zombie processes or send children kill signals | 13:55 |
sdake | therefore qemu shoudl not be affected | 13:55 |
sdake | try docker kill -s KILL libvirt | 13:56 |
sdake | perhaps the default was changed | 13:56 |
* Jeffrey4l trying | 13:57 | |
*** iceyao has quit IRC | 13:58 | |
Jeffrey4l | sdake, the same. qemu is kill, too. | 13:58 |
sdake | ok well i'm out of ideas | 13:59 |
sdake | perhaps there is a way to start a container with protected attributes | 13:59 |
sdake | such that its children rent killed | 13:59 |
sdake | this is a behavior change in docker | 13:59 |
sdake | i spent 2 weeks making this work in liberty | 14:00 |
Jeffrey4l | ok. I will search more info about this. | 14:00 |
sdake | we may need to patch docker in some way to add a f eature to avoid killing processes | 14:01 |
sdake | dwalsh ping ^^ | 14:01 |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Update cleanup-containers to remove manila containers https://review.openstack.org/273558 | 14:02 |
*** salv-orlando has quit IRC | 14:09 | |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 14:11 |
*** Allen_Gao has quit IRC | 14:14 | |
Jeffrey4l | sdake, event though, I have no much knowledge about the docker. But I still think it should be a bug that the child process remain after killing the container. Docker should avoid this. (just some thoughts) | 14:14 |
sdake | i agree but that is not how it used to behave | 14:15 |
sdake | perhaps we can get it to behave as it used to with a config option | 14:15 |
sdake | Jeffrey4l can you tell me what patch had you and sam fixed with the label | 14:17 |
*** iceyao has joined #kolla | 14:17 | |
sdake | either the review or the bug # | 14:17 |
Jeffrey4l | sdake, you want this https://launchpad.net/bugs/1538136 | 14:18 |
openstack | Launchpad bug 1538136 in kolla "mariadb can not start from a host reboot" [Critical,In progress] - Assigned to Jeffrey Zhang (jeffrey4l) | 14:18 |
openstackgerrit | Steven Dake proposed openstack/kolla: check out the review in this patch https://review.openstack.org/273565 | 14:22 |
nihilifer | wat | 14:23 |
sdake | damnit | 14:23 |
sdake | Jeffrey4l or nihilifer i subitted a full review for patch set 3 in manilla and cant' save my comments | 14:23 |
sdake | any tips on how to do that? | 14:24 |
sdake | since patch set 4 is avaiable | 14:24 |
nihilifer | sdake: do you havew this 3rd patched picked in "diff against"? | 14:25 |
nihilifer | have* | 14:26 |
sdake | nm had to maximize browser for reply butotn to appear | 14:26 |
sdake | wierd bug | 14:26 |
*** salv-orlando has joined #kolla | 14:26 | |
sdake | which is maurico lima's nick? | 14:26 |
*** reddy has quit IRC | 14:27 | |
*** neilus has joined #kolla | 14:27 | |
sdake | if your out there nice work dude | 14:27 |
Jeffrey4l | sdake, this https://review.openstack.org/#/c/269688/3 ? | 14:27 |
sdake | yes | 14:27 |
sdake | i figured ito ut jefrey4l | 14:27 |
Jeffrey4l | what's mean by "can't save comments"? | 14:28 |
sdake | the repely button didn't appaer in my browsser window | 14:28 |
Jeffrey4l | I saw your comments. | 14:28 |
sdake | until i maximized my browser | 14:28 |
Jeffrey4l | lol | 14:28 |
*** diogogmt has quit IRC | 14:28 | |
Jeffrey4l | yes. I reproduced it.. | 14:29 |
nihilifer | well, i'll repeat myself again, but gertty is the best solution here ;) | 14:29 |
sdake | what is gertty | 14:29 |
nihilifer | gerrit cli | 14:29 |
limamauricio | sdake: I saw your rewiers, I'll fix it | 14:29 |
limamauricio | Thanks you | 14:29 |
*** salv-orlando has quit IRC | 14:29 | |
sdake | limamauricio rockin work dude | 14:29 |
sdake | can i ask you, how long did you think you spent on that service | 14:29 |
dwalsh | sdake, Use runc. :^) | 14:29 |
nihilifer | sdake: https://github.com/openstack/gertty | 14:29 |
*** salv-orlando has joined #kolla | 14:29 | |
sdake | trying to get a handle on how long it takes people to come up to speed on implementing anew service in kolla | 14:30 |
sdake | dwalsh runc is what | 14:30 |
Jeffrey4l | didn't know this tool. thanks nihilifer | 14:30 |
limamauricio | 3 days, but I just updated the mkoderer path. | 14:30 |
*** neilus has quit IRC | 14:31 | |
limamauricio | It was hard for me, but I am beginner in this, so I think I did well | 14:31 |
*** diogogmt has joined #kolla | 14:32 | |
sdake | dwalsh is runc like, a legit project | 14:32 |
sdake | or are you kidding around | 14:32 |
*** dmsimard1 is now known as dmsimard | 14:34 | |
limamauricio | sdake: I'm having lunch now, I'll commit your changes when I come back | 14:38 |
*** jtriley has joined #kolla | 14:43 | |
*** neilus has joined #kolla | 14:45 | |
dwalsh | sdake, Depends on what you mean by legit. It is used heavily in upcoming docker, but it is still libcontainer. When docker moves to executing runc directly versus using libcontainer, then I guess you could have more confidence. | 14:47 |
dwalsh | Problem with runc, is you don't get COW file system. But I believe we should not use COW file sytems in production, just start using R/Only traditional chroot OS. | 14:48 |
*** inc0 has joined #kolla | 14:49 | |
*** diogogmt has quit IRC | 14:49 | |
sdake | dwalsh cool well its too late in our cycle to switch to that | 14:49 |
sdake | but really my quesiton was more about theprocess handling semantic of kill with pid=host | 14:49 |
sdake | previously qemu processes weere not killed | 14:49 |
sdake | with docker 1.9.1 all processes are killed when libvirtd is sent a sigkill | 14:50 |
sdake | this is totally not suitable | 14:50 |
sdake | i get runc can handle the docker daemon restart problem | 14:50 |
sdake | that seems super cool | 14:50 |
kproskurin | Guys, I trying to deploy mesos infra from kolla master and getting this: http://fpaste.org/315781/53992636/ | 14:51 |
*** neilus has quit IRC | 14:55 | |
sdake | kproskurin are you using docker 2.0.0? | 14:56 |
*** neilus has joined #kolla | 14:56 | |
*** akwasnie has joined #kolla | 14:56 | |
*** mdnadeem1 has joined #kolla | 14:56 | |
kproskurin | Oh I got 1.9.1 | 14:56 |
sdake | 1.9.4 | 14:56 |
sdake | sorry not docker | 14:56 |
sdake | i mean ansible | 14:56 |
sdake | docker 1.9.1 is confirmed to work | 14:57 |
sdake | ansible 2.0.0 does not work | 14:57 |
sdake | your bcktrace looks like an ansible 2.0.0 problem to me | 14:57 |
kproskurin | ops, yeah, I have 2.0 | 14:57 |
sdake | use 1.9.4. please | 14:57 |
kproskurin | ok, ty | 14:57 |
sdake | we have not ported to 2.0.0 and will not be doing so for mitaka | 14:57 |
dwalsh | sdake, Then something got broken in docker-1.9. You mean the vms are not reparented else where out of the container? | 14:57 |
dwalsh | It is using pid=host? | 14:58 |
dwalsh | correct? | 14:58 |
sdake | dwalsh roger | 14:58 |
sdake | dwalh i tested this ion docker 1.6ish and it worked like a champ | 14:58 |
dwalsh | Then that is a new bug. Please open a bugzilla at redhat.com and I will have someone look at it. | 14:58 |
sdake | and docker 1.8ish | 14:58 |
sdake | Jeffrey4l can you open a bug please and copy dwalsh on it | 14:58 |
Jeffrey4l | np | 14:59 |
dwalsh | I have not seen a bug, so I did not know. I will ping rhallisey to test it and will see about getting a fix. | 14:59 |
sdake | dwalsh should we not open a issue in he issue tracker upstream as wel? | 14:59 |
dwalsh | Sure. Do both. | 14:59 |
dwalsh | I get yelled at about bugs, though. :^) | 14:59 |
sdake | Jeffrey4l there is your course of action from the one of3 docker experts i know :) | 14:59 |
sdake | dwalsh 1.9.1 not reparting and rather killing seems to make more sense to me | 15:00 |
sdake | but its not what we need for libvirt! | 15:00 |
Jeffrey4l | cool | 15:00 |
*** fgimenez has quit IRC | 15:00 | |
Jeffrey4l | btw, i will start a docker 1.8 to test this case. | 15:00 |
sdake | Jeffrey4l use the word "regression" in the title | 15:01 |
sdake | Jeffrey4l that would rock bu tyou may have to ue liberty for that, i think our anible module is not compatible with docker .18 and our named volumes definately are not | 15:01 |
sdake | Jeffrey4l please add me to the bugizilla cc as well as the issue tracker list | 15:02 |
sdake | this is a critical critical issue to solve | 15:02 |
Jeffrey4l | I can start a simple test. by using `nohub sleep 3000&` to simulate the qemu. I think it should be the same | 15:02 |
sdake | Jeffrey4l note runc as dwalsh pointed out solves the docker restart problem | 15:02 |
sdake | it uses a separate process to create teh container | 15:02 |
sdake | so if docker restarts, runc stays intact | 15:02 |
sdake | so, it hink that is future tech | 15:03 |
Jeffrey4l | roger. | 15:03 |
sdake | so there is our answer to that problem | 15:03 |
sdake | "its coming" :) | 15:03 |
Jeffrey4l | Good news. | 15:03 |
sdake | i wish i kenw about runc in july 2015 when it was announced | 15:03 |
sdake | i got asked that same fn question 20 times at openstack summit | 15:03 |
sdake | it was really frustrating that I had to answer with the lame "docker is not supposed to be restarted in our model" | 15:04 |
sdake | dwalsh is runc and hte docker daemon going to be integrated? | 15:04 |
sdake | dwalsh atm we are highly reliant on the docker api, which is loosly stable in the current docker series | 15:05 |
sdake | dwalsh second quesiton, runc is based entirely on libcontainer in the backend, so it has the same functioanl behavior as docker? | 15:05 |
*** fgimenez has joined #kolla | 15:05 | |
*** fgimenez has quit IRC | 15:05 | |
*** fgimenez has joined #kolla | 15:05 | |
sdake | limamauricio enjoy lunch, when ou return, you aid it took you 3 days, but how much "code time" did it take you, not including review turnarounds | 15:06 |
*** neilus has quit IRC | 15:07 | |
sdake | limamauricio aid/said | 15:08 |
*** dims has joined #kolla | 15:08 | |
*** fthiagogv has quit IRC | 15:10 | |
*** diogogmt has joined #kolla | 15:21 | |
*** diogogmt has quit IRC | 15:23 | |
*** ppetit has quit IRC | 15:23 | |
*** iceyao has quit IRC | 15:25 | |
Jeffrey4l | sdake, what docker version are you using in you article? | 15:28 |
*** akwasnie has quit IRC | 15:29 | |
*** daneyon has joined #kolla | 15:30 | |
*** achanda has joined #kolla | 15:32 | |
*** daneyon_ has quit IRC | 15:32 | |
*** blahRus has joined #kolla | 15:33 | |
dwalsh | sdake, Supposedly docker daemon will generate a OCI json data and then execute runc to act on it, in docker-1.11 or later. We shall see. Their goal is to allow you to specify other backends like clearcontainers, lxc, Windows? Any back end that supports the oci json should be able to run. | 15:42 |
dwalsh | runc is an OCI back end based on libcontainer. | 15:43 |
*** akwasnie has joined #kolla | 15:43 | |
*** achanda has quit IRC | 15:43 | |
dwalsh | sdake, It has the same functionality as the libcontainer execdriver. Obviously not all of the docker daemon functions. Like graphc/pull/push etc. | 15:44 |
*** neilus has joined #kolla | 15:44 | |
*** neilus has quit IRC | 15:49 | |
*** Jeffrey4l has quit IRC | 15:52 | |
*** pbourke has quit IRC | 15:59 | |
*** daneyon_ has joined #kolla | 15:59 | |
*** daneyon has quit IRC | 16:01 | |
*** diogogmt has joined #kolla | 16:02 | |
*** salv-orl_ has joined #kolla | 16:06 | |
*** Jeffrey4l has joined #kolla | 16:06 | |
sdake | damn we need pull | 16:08 |
*** salv-orlando has quit IRC | 16:09 | |
*** akwasnie1 has joined #kolla | 16:11 | |
*** akwasnie has quit IRC | 16:14 | |
*** pbourke has joined #kolla | 16:15 | |
*** akwasnie1 has quit IRC | 16:15 | |
*** neilus has joined #kolla | 16:21 | |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add memcached support https://review.openstack.org/266477 | 16:21 |
*** rmart04 has left #kolla | 16:21 | |
*** neilus has quit IRC | 16:25 | |
*** unicell has joined #kolla | 16:26 | |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add memcached support https://review.openstack.org/266477 | 16:27 |
*** ppetit has joined #kolla | 16:29 | |
SamYaple | morning | 16:29 |
*** Jeffrey4l has quit IRC | 16:33 | |
*** Jeffrey4l has joined #kolla | 16:34 | |
nihilifer | any european folks are going to fosdem? ;) | 16:34 |
pbourke | nihilifer: ive been meaning to for about the past 3 years | 16:41 |
pbourke | nihilifer: keep forgetting about it till its too late | 16:41 |
SamYaple | morning pbourke | 16:41 |
pbourke | SamYaple: howdy | 16:41 |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Support deploy without configs https://review.openstack.org/266476 | 16:43 |
Jeffrey4l | sdake, do not know how to add you to the cc list. But here is the bug url https://bugzilla.redhat.com/show_bug.cgi?id=1302807 | 16:46 |
openstack | bugzilla.redhat.com bug 1302807 in docker "regression: Unattached child process should exist when the container is killed in docker container with pid=host" [Unspecified,New] - Assigned to andreas.bierfert | 16:46 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the machine-id which cause nova-compute raise exception https://review.openstack.org/273508 | 16:48 |
Jeffrey4l | sdake, but I tested this with docker 1.6.2 , and the child process is killed the same. :( | 16:50 |
sdake | dwalsh can you comment on that ^ | 16:51 |
sdake | aare you sure pid mode is being set? | 16:52 |
sdake | as in are you launching from the command line | 16:52 |
openstackgerrit | Qiu Yu proposed openstack/kolla: Detect and use physical disks for Swift storage https://review.openstack.org/272410 | 16:52 |
Jeffrey4l | sdake, docker run -it --privileged --pid=host --net host centos bash | 16:52 |
sdake | and the nwhat? | 16:53 |
* Jeffrey4l is writing | 16:53 | |
SamYaple | Jeffrey4l: i see the same thing you see, but this wasnt always the case | 16:54 |
SamYaple | Jeffrey4l: ill see if i can figure it out | 16:54 |
Jeffrey4l | SamYaple, you mean the docker? | 16:54 |
dwalsh | sdake, I will have my guys look at this. | 16:55 |
SamYaple | Jeffrey4l: yea | 16:56 |
*** neilus has joined #kolla | 16:57 | |
SamYaple | Jeffrey4l: does the machine-id bug also affect liberty? (sounds like it should) | 16:57 |
kproskurin | SamYaple: Sorry I didnt answered for your comment in memcached review. I did it now. | 16:58 |
Jeffrey4l | SamYaple, yep it should. ( i am thinking it should be introduced from centos:7.2 ) | 16:58 |
SamYaple | Jeffrey4l: i updated the bug to target liberty too | 16:58 |
*** mdnadeem1 has quit IRC | 16:58 | |
SamYaple | kproskurin: ok thanks! | 16:59 |
Jeffrey4l | thanks a lot. | 16:59 |
Jeffrey4l | sdake, dwalsh SamYaple see this small test about the docker issue http://paste.openstack.org/ | 17:00 |
*** akwasnie has joined #kolla | 17:02 | |
Jeffrey4l | sorry, here is the link http://paste.openstack.org/show/485293/ | 17:02 |
*** mdnadeem1 has joined #kolla | 17:02 | |
Jeffrey4l | sdake, dwalsh SamYaple ^ | 17:02 |
sdake | dwalsh thanks , fwiw it could bea problem on our end | 17:03 |
SamYaple | pbourke: Jeffrey4l I believe I have answered your questions about the LABEL mechanism in kolla_docker | 17:05 |
*** diogogmt has quit IRC | 17:06 | |
sdake | Jeffrey4l what process is 4320 | 17:06 |
SamYaple | dwalsh Jeffrey4l It looks like even though the pid is in the host pid namespace the parent process is still under the docker daemon | 17:08 |
Jeffrey4l | sdake, it is not the process id. | 17:08 |
sdake | Jeffrey4l sorry wron question, what proess is the parent of 4320 | 17:08 |
sdake | sorry not 4320 the sleep process | 17:08 |
Jeffrey4l | sdake, 4320 should be the VSZ( from ps, I have no idea for this) | 17:09 |
SamYaple | sdake: don't worry qemu still survives | 17:10 |
*** akwasnie1 has joined #kolla | 17:10 | |
SamYaple | im just trying to figure out a test to reproduce qemu launching | 17:10 |
SamYaple | probably need to disown | 17:10 |
Jeffrey4l | SamYaple, tried. not work. (may be i am wrong), pls test it. | 17:10 |
SamYaple | Jeffrey4l: i have | 17:11 |
*** neilus has quit IRC | 17:11 | |
SamYaple | killed the docker daemon entirely, qemu still running | 17:11 |
SamYaple | but thats because its parent is pid 1 | 17:11 |
Jeffrey4l | :( | 17:11 |
sdake | who is the parent id of the sleep jeffrey4l | 17:11 |
sdake | with host=pid, sleep shoud lbe reparented to 1 when baah is killed | 17:12 |
*** akwasnie has quit IRC | 17:12 | |
SamYaple | sdake: this is not the case | 17:12 |
*** akwasnie1 has quit IRC | 17:12 | |
SamYaple | this is probably a thing libvirt is doing and im trying to figure out what its doing | 17:13 |
Jeffrey4l | sdake, 1 | 17:13 |
sdake | oh ya could be a libvirt change | 17:13 |
sdake | jeffrey1 good thats whati t should be | 17:13 |
sdake | that means sdocker hansn't mucked wit h the zombie repeaing proess problem | 17:13 |
sdake | althoug hthe fact that Jeffrey4l's sleep thing also kills the sleep is worrisome SamYaple | 17:14 |
*** neilus has joined #kolla | 17:15 | |
SamYaple | sdake: yea I can produce the same beahviour | 17:16 |
SamYaple | i wonder if qemu is only sticking around because it ignored the kill | 17:16 |
SamYaple | I mean the end result is it sticks around | 17:16 |
SamYaple | but still | 17:16 |
sdake | sigkill cannot be ignored | 17:16 |
SamYaple | i didnt say sigkill | 17:16 |
sdake | see man 7 signal | 17:16 |
SamYaple | and it can kinda be ignore | 17:17 |
SamYaple | not by the process but it will zombie of there is certain things waiting (like IO) | 17:17 |
limamauricio | sdake: I did your changes, I'm thinking of changing the init-runonce to make some necessary settings for the manila work. What do you think? | 17:17 |
sdake | SamYaple agree re zombie | 17:17 |
limamauricio | get if enable_manila = yes | 17:17 |
limamauricio | and do it | 17:17 |
sdake | limamauricio sure get em in and we will review them - but needs to be optional | 17:17 |
*** mdnadeem1 has quit IRC | 17:18 | |
sdake | but if a process is not zombie it cannot ignore sigkill since kernel terminates it with prejudice | 17:18 |
SamYaple | Jeffrey4l: if you launch a container with pid=host and run this command `( sleep infinity & )` then kill -9 the docker daemon the process sticks around | 17:18 |
SamYaple | stoping the container kills the process though | 17:19 |
* Jeffrey4l is trying | 17:19 | |
limamauricio | I'll change the init-runonce and commit for review in the same path | 17:19 |
limamauricio | ok? | 17:19 |
sdake | wfm | 17:19 |
*** fgimenez has quit IRC | 17:19 | |
SamYaple | limamauricio: init-runonce doesn't read enable_manila=yes | 17:19 |
limamauricio | :/ | 17:20 |
*** unicell has quit IRC | 17:20 | |
SamYaple | limamauricio: we only do basic environment init with that | 17:20 |
SamYaple | ie, neutron glance nova keystone | 17:20 |
SamYaple | no heat or cinder or anythign else | 17:20 |
SamYaple | you would be better off creating an init-manila script | 17:20 |
Jeffrey4l | SamYaple, yes, you are right. but it should be dangoure to send kill -9 to docker daemon. | 17:21 |
SamYaple | Jeffrey4l: im not suggesting it, im pointing it out as we map behaviour | 17:22 |
Jeffrey4l | hmm.. | 17:22 |
SamYaple | Jeffrey4l: but keep in mind, I have a qemu process running and i stop the nova_libvirt container like normal | 17:23 |
SamYaple | so libvirt is doing _something_ helpful here | 17:23 |
*** neilus has quit IRC | 17:23 | |
Jeffrey4l | sure. | 17:23 |
SamYaple | we need to identify how it launches the process | 17:23 |
*** diogogmt has joined #kolla | 17:24 | |
Jeffrey4l | some thoughs | 17:24 |
Jeffrey4l | 1. how about start every one qemu process in a docker container? | 17:25 |
SamYaple | Jeffrey4l: we cant do that | 17:25 |
SamYaple | we dont start qemu processes | 17:25 |
SamYaple | libvirt does | 17:25 |
Jeffrey4l | 2. we can not exec in a stop container, so what will be happend when the container is killed, but the qemu binary is still used from that container. | 17:26 |
SamYaple | Jeffrey4l: so when the qemu is dangling when the nova_libvirt process starts again it can reattach and remanage that process | 17:27 |
SamYaple | but assuming its not coming backup, you can kill it from the host | 17:27 |
Jeffrey4l | I know that. Just thoughts.(may be some hack method could be used.) Just throw these for brain storm :D | 17:28 |
SamYaple | yea keep it coming! | 17:29 |
SamYaple | this is how problems get solved | 17:29 |
Jeffrey4l | yep | 17:29 |
Jeffrey4l | Based on 2: I still think it is impossible to stop the container and start it again with the child process unchanged. | 17:30 |
*** akwasnie has joined #kolla | 17:31 | |
Jeffrey4l | even, I think 1 is more easy to 2 :D | 17:31 |
SamYaple | Jeffrey4l: no i mean that is what happens | 17:31 |
SamYaple | it actually happens that way | 17:31 |
sdake | what is th eprent pid of qemu | 17:32 |
sdake | Jeffrey4l | 17:32 |
sdake | lets see if that matches sleep | 17:32 |
*** athomas has joined #kolla | 17:32 | |
sdake | as launched from virsh | 17:32 |
Jeffrey4l | just stop the container then start it is meanless. We really need is upgrade, stop the container, start another new container. | 17:33 |
* Jeffrey4l starting my test env to check that. | 17:33 | |
Jeffrey4l | IIRC, the ppid of qemu is 1. | 17:33 |
sdake | samyaple are you proposing killilng docker -9 on the compute nodes? | 17:35 |
*** akwasnie has quit IRC | 17:35 | |
sdake | during upgrade process | 17:35 |
SamYaple | sdake: hell no | 17:35 |
SamYaple | im just observing | 17:35 |
sdake | ok | 17:35 |
sdake | got it | 17:36 |
sdake | well this shit worked | 17:36 |
sdake | i spent 2 weeks on it at endo f liberty | 17:36 |
SamYaple | sdake: it still does | 17:36 |
sdake | fixing like 12 different bugs | 17:36 |
SamYaple | but i cant figure out how | 17:36 |
SamYaple | whereas before it was easily reproduceable | 17:36 |
SamYaple | it may be because of qemus ties into the kernel | 17:36 |
SamYaple | i wonder if it breaks with pure software qemu... | 17:37 |
sdake | are you guys runnign on virt? | 17:37 |
SamYaple | i am, but im running kvm-on-kvm | 17:37 |
sdake | i never tested virt only baremetal | 17:37 |
Jeffrey4l | qemu on kvm | 17:38 |
sdake | well there re some deltas | 17:38 |
sdake | my labis torn apart or i'd try baremetal | 17:38 |
sdake | shoud have the tile done next wed or thur in my office | 17:38 |
sdake | (big office) | 17:38 |
Jeffrey4l | sdake, the qemu ppid is 1, for sure. | 17:38 |
sdake | doing bedroom monday | 17:38 |
sdake | ok well that is good, that means docker and libvirt haven't done any mucking around with the reparenting | 17:39 |
openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Support deploy without configs https://review.openstack.org/266476 | 17:39 |
sdake | which begs the question how the hell does the process get killed | 17:39 |
sdake | its parent process is init, init will not send it any signal | 17:39 |
Jeffrey4l | SamYaple, even the child process dettached. But the pid is not change and the old parent process should still know it. When using graceful stop, the parent will stop the all the children process he know. When stop using `-9`, the parent process has no time to do this. | 17:41 |
Jeffrey4l | may be this is the case. | 17:41 |
SamYaple | Jeffrey4l: not when you disown | 17:41 |
SamYaple | if the parent is pid 1, there is nothing to know | 17:42 |
SamYaple | but i think docker made some changes and is now tracking all pids that get launched in the container | 17:42 |
*** sdake_ has joined #kolla | 17:42 | |
sdake_ | [10:39:38] <sdake>which begs the question how the hell does the process get killed | 17:42 |
sdake_ | [10:39:55] <sdake>its parent process is init, init will not send it any signal | 17:42 |
sdake_ | [10:40:51] <sdake>a debug printk of signal in kernel.org would be extremely helpful now | 17:42 |
sdake_ | [10:41:30] <sdake>Jeffrey4l add a link to this irc session in that bug at red hat so they can read through our discussion | 17:42 |
SamYaple | sdake: see above | 17:43 |
SamYaple | 17:42:44 < SamYaple> but i think docker made some changes and is now tracking all pids that get launched in the contain | 17:43 |
sdake_ | i got disconnect sam | 17:43 |
Jeffrey4l | sdake_, ok. | 17:43 |
sdake_ | SamYaple apparently Jeffrey4l said this problem exists in 1.6.2 as well | 17:43 |
SamYaple | if its tracking external to kernel, then the child/parent relationship doesnt matter | 17:43 |
SamYaple | sdake_: 1.6.2 is when they "fixed" pid stuff ;) | 17:43 |
sdake_ | jeff try 1.6.0 | 17:44 |
Jeffrey4l | yes. the paste.openstack.org test is using 1.6.2 | 17:44 |
sdake_ | 1.5.0 pid host mode was busted | 17:44 |
*** kproskurin has quit IRC | 17:44 | |
sdake_ | i dont know what version i tested docker with liberty but it could have been 1.6.0 | 17:44 |
*** sdake has quit IRC | 17:45 | |
Jeffrey4l | i will try 1.6.0 soon. | 17:45 |
sdake_ | liberty docs said 1.7.0 | 17:46 |
sdake_ | but its possible iteseted with 1.6.0 | 17:47 |
*** salv-orl_ has quit IRC | 17:48 | |
sdake_ | i guess we need a docker upgrade playbook for upgrades | 17:48 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Update ceph-mon keyrings https://review.openstack.org/273687 | 17:49 |
SamYaple | sdake_: no the daemon is for the operator to upgrade | 17:50 |
SamYaple | we dont touch the host | 17:50 |
sdake_ | i think we can provide a reference implementation | 17:51 |
sdake_ | outside the current playbooks | 17:51 |
SamYaple | We can provide documentation out the wazoo for sure | 17:51 |
sdake_ | the problem with docs is i want to do an automated update at summit | 17:51 |
SamYaple | and you can upgrade teh docker daemon with liberty containers.... you just cant run the playbooks after that | 17:51 |
SamYaple | liberty playbooks that is | 17:51 |
sdake_ | named volumes makes that impossible | 17:51 |
SamYaple | has nothing to do with that | 17:52 |
sdake_ | you mispoke so i mospoke | 17:52 |
SamYaple | i did not mispeak | 17:52 |
SamYaple | you can run dokcer 1.9+ with the liberty containers, you just cant run the liberty playbooks with docker 1.9 | 17:52 |
sdake_ | ya wfm but curious why | 17:53 |
SamYaple | ansible docker module | 17:53 |
sdake_ | i want a reference implemtnation taht upgrades docker on all nodes i the inventory | 17:54 |
*** neilus has joined #kolla | 17:54 | |
sdake_ | it can live outside ansible if need be | 17:54 |
sdake_ | possibly contrib | 17:54 |
Jeffrey4l | I havn't find the docker 1.6.2 binary. May be need build from source. But I really need to go to bed. | 17:54 |
Jeffrey4l | I wil test it tomorrow. | 17:55 |
sdake_ | Jeffrey4l thanks foryour hlep on this one | 17:55 |
sdake_ | Jeffrey4l shoot me an email stdake@cisco.com with results please in case i'm not around | 17:55 |
SamYaple | sdake_: this is 180 from the original goal. there is a reason we ddont have a plybook to install docker on the hosts either | 17:55 |
*** ssurana has joined #kolla | 17:55 | |
Jeffrey4l | I will. | 17:55 |
SamYaple | that conversation has been had | 17:55 |
sdake_ | did you miss contrib directory | 17:55 |
Jeffrey4l | Bye guys. | 17:55 |
SamYaple | nope. weve had _that_ conversation | 17:55 |
SamYaple | by Jeffrey4l! | 17:55 |
SamYaple | so maybe its time to have it again | 17:55 |
sdake_ | yes its a new conversation i just started :) | 17:55 |
SamYaple | its an old coversation we are going to rehash | 17:56 |
SamYaple | its not new | 17:56 |
sdake_ | contrib is new to hte conversation | 17:56 |
SamYaple | it is not... | 17:56 |
SamYaple | unless you mean a new repo | 17:56 |
sdake_ | new directory under tld | 17:56 |
*** unicell has joined #kolla | 17:56 | |
SamYaple | its not using the inventory then | 17:57 |
sdake_ | what is the objection just so i understand your position | 17:57 |
SamYaple | all the ansible stuff goes under ansible/ tld | 17:57 |
sdake_ | contribansible | 17:57 |
sdake_ | contrib/ansible | 17:57 |
SamYaple | wont work | 17:57 |
SamYaple | and anyway what you are purposing is going to tie the ansible stuff into the kolla repo and make it much harder to seperate kolla-ansible which is the stated goal of everyone here | 17:58 |
sdake_ | ok first objection noted | 17:58 |
sdake_ | others? | 17:58 |
*** neilus has quit IRC | 17:58 | |
SamYaple | the original agreement from the community was no tasks to touch the host | 17:59 |
SamYaple | that was the first object ^ | 17:59 |
sdake_ | i think its time to have that discussion again possibly at the mimdcycle | 17:59 |
*** dwalsh has quit IRC | 17:59 | |
SamYaple | I agree | 17:59 |
sdake_ | i am getting requests from various sources to touch the host in various ways | 17:59 |
sdake_ | perhaps it shouod lbe a separate repository given your first objection | 18:00 |
sdake_ | kolla shoudl contain docs and containers i think | 18:00 |
*** Jeffrey4l has quit IRC | 18:00 | |
sdake_ | rather your second objection | 18:00 |
sdake_ | we need it to do one click upgrades | 18:01 |
SamYaple | host management was discussed in tokyo (with the talk of bifrost and what not) | 18:01 |
sdake_ | agree total host deployment possibly out of scope | 18:01 |
SamYaple | its not a one click setup, its not a one click upgrade | 18:01 |
sdake_ | it can be made so | 18:02 |
SamYaple | we dont setup docker, we shouldnt manage it | 18:02 |
SamYaple | if we setup the host (see bifrost) then sure | 18:02 |
SamYaple | but thats already goign to be a second playbook | 18:02 |
SamYaple | in a seperate area | 18:02 |
sdake_ | i could live with two click upgrade | 18:02 |
SamYaple | well then yo uhave it | 18:02 |
*** mbound has quit IRC | 18:02 | |
SamYaple | there are many ansible galaxy playbooks for managing packages | 18:03 |
SamYaple | including specifc for docker | 18:03 |
SamYaple | all code Kolla doesn't need to touch or maintain | 18:03 |
sdake_ | mind pointing me at one that looks sanitary so i can play aorund with it | 18:04 |
sdake_ | i am not a galaxy expert ;) | 18:04 |
sdake_ | SamYaple my wife says there is an ansible-galaxy tool | 18:05 |
sdake_ | do you know anything about it? | 18:06 |
sdake_ | shedoesn't recallt he details of how it works | 18:06 |
SamYaple | not at all. cloudnull does for sure | 18:06 |
sdake_ | ok well i'll add to agenda for midcycle, its part of the upgrade problem | 18:06 |
sdake_ | because we require docker 1.10 for mitaka | 18:06 |
sdake_ | and docker 1.7.0 for liberty | 18:06 |
sdake_ | and i dont really want to confuse operators with 'go login to all your nodes and upgrade docker" | 18:07 |
SamYaple | 1.8.2 for liberty | 18:07 |
SamYaple | we tell them "go login to all your nodes and install docker" right now... | 18:07 |
sdake_ | the docs say 1.7.0 - i guess they shoudl be updated | 18:07 |
SamYaple | maybe 1.7 is lowend, i dont recall | 18:07 |
sdake_ | ya that kind of sucks in my opinion | 18:08 |
sdake_ | we never finished the job on bifrost experimentation | 18:08 |
SamYaple | we never started | 18:09 |
sdake_ | i know inc0 has 2-3 people working on bare metal deploy - not ssure if related to bifrost or not | 18:09 |
SamYaple | there has been no code or reviews related to this | 18:09 |
SamYaple | i dont know if there is even a blueprint | 18:09 |
sdake_ | there isn't its a problem i just considered while we were jerking around with this qemu problem | 18:09 |
SamYaple | but as an operator who has lots of operator friends I can tell you none of them care about what you are saying | 18:09 |
sdake_ | my brain works in mysterious ways ;) | 18:09 |
sdake_ | saying where | 18:10 |
SamYaple | installing and managing the docker package on the host is perfectly fine | 18:11 |
SamYaple | most have there own system for packages and package control as it is | 18:11 |
SamYaple | its a non-issue | 18:11 |
unicell | SamYaple: https://review.openstack.org/#/c/272410/ you mentioned that there's a better way for the param variable? | 18:16 |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 18:21 |
cloudnull | o/ all -- if i can help out with ansible + galaxy things let me know | 18:24 |
*** sdake_ has quit IRC | 18:26 | |
*** dwalsh has joined #kolla | 18:29 | |
*** dwalsh has quit IRC | 18:29 | |
*** dwalsh has joined #kolla | 18:30 | |
*** dwalsh has quit IRC | 18:34 | |
SamYaple | unicell: having the params wrapped in a variable like that isnt good. just follow the other tasks and have the params in the actual task | 18:36 |
unicell | SamYaple: that means there's no way to customize the param if needed to? | 18:36 |
SamYaple | thanks cloudnull sdake was asking about galaxy. im sure he would benefit from your galaxy knowlege | 18:37 |
SamYaple | unicell: correct. because there is no need to customize the param name | 18:37 |
SamYaple | use a variable for the customizable part | 18:37 |
SamYaple | i think you did that in PS1 or PS2 | 18:37 |
cloudnull | ping me whenever if i can help. | 18:37 |
*** rhallisey has quit IRC | 18:38 | |
unicell | ok, I got what you mean | 18:39 |
limamauricio | SamYaple, sdake I remove "/dev/:/dev/" and "/run/:/run/" | 18:43 |
openstackgerrit | Qiu Yu proposed openstack/kolla: Detect and use physical disks for Swift storage https://review.openstack.org/272410 | 18:45 |
*** CheKoLyN has joined #kolla | 18:49 | |
*** dwalsh has joined #kolla | 18:50 | |
SamYaple | limamauricio: I saw that, ive done a full review of the config file now and mentioned some changed needed | 18:54 |
SamYaple | limamauricio: but its close! | 18:54 |
*** ppetit has quit IRC | 18:55 | |
limamauricio | Thank you, your reviews are very helpful. | 18:56 |
*** tzn has quit IRC | 19:00 | |
SamYaple | unicell: ok im going to do a full fresh review of the swift patch, see what pops up. thank you for working with the review process! | 19:00 |
*** dwalsh has quit IRC | 19:03 | |
*** neilus has joined #kolla | 19:04 | |
unicell | SamYaple: thanks you so much for the helpful comments! | 19:05 |
unicell | *thank* | 19:06 |
SamYaple | unicell: i always feel like I come off to opinionated, but im really trying to think for the future | 19:08 |
unicell | I agree, even the patch works for me doesn't mean it works for everybody else' deployment | 19:11 |
openstackgerrit | Merged openstack/kolla-mesos: Make the start tests cleaner by wrapping the globals https://review.openstack.org/269882 | 19:12 |
*** neilus has quit IRC | 19:19 | |
*** neilus has joined #kolla | 19:26 | |
limamauricio | SamYaple, can I only comment the default lines in manila.conf.j2? | 19:32 |
SamYaple | limamauricio: no it should be completely removed | 19:32 |
limamauricio | ok | 19:32 |
*** dwalsh has joined #kolla | 19:33 | |
SamYaple | limamauricio: we have a streamlined approach to configs in Kolla, as stripped down as possible. we set only the variable that must be set to have a proper deployment | 19:33 |
limamauricio | even if someone wanted to use later? | 19:34 |
limamauricio | I'll remove :) | 19:34 |
SamYaple | limamauricio: you can still use it | 19:35 |
SamYaple | limamauricio: you can use the augment files | 19:35 |
SamYaple | you can add any params or variables you want there | 19:35 |
limamauricio | Ok | 19:35 |
limamauricio | make sense | 19:35 |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 19:41 |
limamauricio | SamYaple ^ | 19:41 |
*** salv-orlando has joined #kolla | 19:42 | |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Update cleanup-containers to remove manila containers https://review.openstack.org/273558 | 19:48 |
openstackgerrit | Qiu Yu proposed openstack/kolla: Detect and use physical disks for Swift storage https://review.openstack.org/272410 | 19:49 |
*** salv-orlando has quit IRC | 19:50 | |
*** neilus has quit IRC | 19:51 | |
*** daneyon_ has quit IRC | 19:51 | |
*** neilus has joined #kolla | 19:55 | |
*** neilus has quit IRC | 19:56 | |
*** neilus has joined #kolla | 19:57 | |
openstackgerrit | Qiu Yu proposed openstack/kolla: Detect and use physical disks for Swift storage https://review.openstack.org/272410 | 20:00 |
*** neilus has quit IRC | 20:12 | |
*** salv-orlando has joined #kolla | 20:17 | |
*** neilus has joined #kolla | 20:18 | |
*** daneyon has joined #kolla | 20:21 | |
*** Marga_ has quit IRC | 20:24 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Add labels to docker module https://review.openstack.org/273309 | 20:26 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Update all bootstraps to use Label https://review.openstack.org/273310 | 20:26 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Cleanup mariadb and make compatiable with mesos https://review.openstack.org/272160 | 20:26 |
*** salv-orlando has quit IRC | 20:30 | |
*** dwalsh has quit IRC | 20:49 | |
*** tzn has joined #kolla | 20:50 | |
*** tzn has quit IRC | 20:50 | |
*** tzn has joined #kolla | 20:50 | |
*** dwalsh has joined #kolla | 20:56 | |
*** dtturner has joined #kolla | 20:57 | |
*** mbound has joined #kolla | 21:00 | |
*** dmsimard_ has joined #kolla | 21:02 | |
*** dmsimard_ has quit IRC | 21:02 | |
*** tpot has joined #kolla | 21:07 | |
dtturner | Hi folks. A question I'm sure has been asked countless times but.... are there any resources out there covering the setup/execution of multinode deployments in more detail? | 21:10 |
*** salv-orlando has joined #kolla | 21:11 | |
SamYaple | dtturner: actually no it doesnt get asked alot or we would add more detailed docs! | 21:12 |
SamYaple | dtturner: what is it you feel is lacking? | 21:12 |
SamYaple | the difference between multinode and AIO is only in the inventory file | 21:12 |
dtturner | Ha! Well, I've been using Oracle's kollacli for multinode deployments of Kilo up to this point, which performs alot of the heavy lifting. I'm now wanting to run some tests closer to trunk, so I need to cut the cli cord and run deploys manually for now. | 21:15 |
SamYaple | ive not used oracles fork of Kolla, however pbourke is an Oracle dev | 21:15 |
SamYaple | if you are saying you want to run vanila kolla there is a kolla-ansible tool that makes that as simple as `kolla-ansible deploy` | 21:16 |
*** neilus has quit IRC | 21:17 | |
dtturner | I'm with Oracle as well and have chatted with Borne. I'm on a different team that plans to test closer to trunk, and so we are playing around with upstream vanilla kolla now.. working through the differences. | 21:19 |
dtturner | OK- thanks. I'll give that a try. I wasn't sure if kolla-ansible was for AIO testing only. | 21:21 |
*** jasonsb has quit IRC | 21:21 | |
*** macsz1 has joined #kolla | 21:26 | |
*** neilus has joined #kolla | 21:27 | |
openstackgerrit | Eric Lemoine proposed openstack/kolla: [POC] Heka proof-of-concept tests https://review.openstack.org/269745 | 21:30 |
*** neilus has quit IRC | 21:31 | |
*** athomas has quit IRC | 21:33 | |
*** neilus has joined #kolla | 21:34 | |
*** akwasnie has joined #kolla | 21:35 | |
*** neilus has quit IRC | 21:38 | |
*** shardy has quit IRC | 21:38 | |
*** CheKoLyN has quit IRC | 21:43 | |
*** sdake has joined #kolla | 21:45 | |
*** sdake has quit IRC | 21:45 | |
*** sdake has joined #kolla | 21:48 | |
*** neilus has joined #kolla | 21:49 | |
*** sdake_ has joined #kolla | 21:50 | |
*** sdake has quit IRC | 21:52 | |
*** macsz1 has quit IRC | 21:52 | |
*** neilus has quit IRC | 21:55 | |
*** neilus has joined #kolla | 21:56 | |
*** CheKoLyN has joined #kolla | 21:56 | |
sdake_ | SamYaple ping soryr i dropped off earlier contrctor showed up and i've been working like a maniac getting him prepped | 22:00 |
*** akwasnie has quit IRC | 22:00 | |
*** neilus has quit IRC | 22:00 | |
*** athomas has joined #kolla | 22:00 | |
*** akwasnie has joined #kolla | 22:01 | |
*** athomas has quit IRC | 22:01 | |
*** athomas has joined #kolla | 22:01 | |
*** neilus has joined #kolla | 22:02 | |
*** inc0 has quit IRC | 22:02 | |
*** iNeilus has joined #kolla | 22:03 | |
*** Marga_ has joined #kolla | 22:05 | |
*** neilus has quit IRC | 22:06 | |
*** salv-orl_ has joined #kolla | 22:06 | |
*** akwasnie has quit IRC | 22:07 | |
*** iNeilus has quit IRC | 22:08 | |
openstackgerrit | Eric Lemoine proposed openstack/kolla: [POC] Heka proof-of-concept tests https://review.openstack.org/269745 | 22:08 |
*** salv-orlando has quit IRC | 22:08 | |
*** neilus has joined #kolla | 22:10 | |
SamYaple | sdake_: i dont know what you are saying | 22:10 |
*** iNeilus has joined #kolla | 22:12 | |
*** neilus has quit IRC | 22:14 | |
*** iNeilus has quit IRC | 22:16 | |
*** gfidente has quit IRC | 22:17 | |
*** akwasnie has joined #kolla | 22:26 | |
*** jtriley has quit IRC | 22:28 | |
*** RA_ has joined #kolla | 22:32 | |
*** athomas has quit IRC | 22:33 | |
*** akwasnie has quit IRC | 22:42 | |
*** frzn_ has joined #kolla | 22:44 | |
*** CheKoLyN has quit IRC | 22:45 | |
*** frzn_ has quit IRC | 22:45 | |
*** frzn_ has joined #kolla | 22:46 | |
*** tzn has quit IRC | 22:49 | |
*** CheKoLyN has joined #kolla | 23:01 | |
*** RA__ has joined #kolla | 23:06 | |
*** RA_ has quit IRC | 23:06 | |
*** CheKoLyN has quit IRC | 23:11 | |
*** tzn has joined #kolla | 23:13 | |
*** RA__ has quit IRC | 23:13 | |
sdake_ | samyaple we were talking earlier and i closed my laptop because the contractors were here to do my flooring | 23:14 |
sdake_ | and i had to get them prepped for about 3-4 hours | 23:14 |
sdake_ | my office wont be ready for atleast 8 more days | 23:15 |
* sdake_ growls | 23:15 | |
*** mbound has quit IRC | 23:15 | |
sdake_ | then two days to slep all my shit back into it | 23:15 |
*** blahRus has quit IRC | 23:19 | |
*** RA_ has joined #kolla | 23:20 | |
*** dwalsh has quit IRC | 23:28 | |
*** frzn_ has quit IRC | 23:45 | |
*** tzn has quit IRC | 23:47 | |
*** daneyon has quit IRC | 23:48 | |
sdake_ | wow 13 minute push for all containers | 23:51 |
sdake_ | docker v2 is pretty fast | 23:51 |
sdake_ | registry that is | 23:51 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!