Tuesday, 2016-02-02

openstackgerritAllen Gao proposed openstack/kolla: Move the install base type check to kolla-build  https://review.openstack.org/27462900:03
*** Allen_Gao has joined #kolla00:05
*** salv-orl_ has quit IRC00:08
*** dims has quit IRC00:10
*** dave-mcc_ has joined #kolla00:27
*** dave-mccowan has quit IRC00:28
*** ajafo has quit IRC00:32
*** ajafo has joined #kolla00:32
*** absubram has quit IRC00:41
*** unicell has joined #kolla00:52
*** jasonsb has joined #kolla00:53
*** unicell1 has joined #kolla01:02
*** unicell has quit IRC01:04
*** kjelly has joined #kolla01:11
*** diogogmt has joined #kolla01:18
*** daneyon_ has quit IRC01:36
*** daneyon has joined #kolla01:36
*** kjelly has quit IRC01:39
openstackgerritAllen Gao proposed openstack/kolla: Move the install base type check to kolla-build  https://review.openstack.org/27462901:39
*** ajafo has quit IRC01:40
*** ajafo has joined #kolla01:40
*** ssurana has quit IRC01:41
*** weiyu has joined #kolla01:49
*** tfukushima has joined #kolla01:54
*** Jeffrey4l has joined #kolla02:03
*** dave-mcc_ has quit IRC02:09
*** dave-mccowan has joined #kolla02:10
*** dims has joined #kolla02:11
*** Marga_ has quit IRC02:20
*** Marga_ has joined #kolla02:22
*** coolsvap|away is now known as coolsvap02:26
*** Marga_ has quit IRC02:27
*** kjelly has joined #kolla02:29
openstackgerritAllen Gao proposed openstack/kolla: Move the install base type check to kolla-build  https://review.openstack.org/27462902:29
*** kejlly_ has joined #kolla02:32
*** kjelly has quit IRC02:32
*** kejlly_ is now known as kjelly02:32
*** ssurana has joined #kolla02:46
*** ssurana has quit IRC02:48
*** chandankumar has joined #kolla02:50
*** thumpba has quit IRC02:51
*** coolsvap is now known as coolsvap|away02:59
*** chandankumar has quit IRC03:04
*** kjelly has quit IRC03:14
*** tfukushima has quit IRC03:14
*** tfukushima has joined #kolla03:16
*** chandankumar has joined #kolla03:19
*** yuanying_ has joined #kolla03:21
*** yuanying has quit IRC03:24
openstackgerritJeffrey Zhang proposed openstack/kolla: Playbook for horizon upgrade  https://review.openstack.org/27482703:29
openstackgerritJeffrey Zhang proposed openstack/kolla: Playbook for horizon upgrade  https://review.openstack.org/27482703:30
*** unicell1 has quit IRC03:31
*** chandankumar has quit IRC03:32
*** ssurana has joined #kolla03:34
*** sdake has joined #kolla03:35
*** yuanying has joined #kolla03:41
*** Marga_ has joined #kolla03:42
*** achanda has joined #kolla03:43
*** Marga_ has joined #kolla03:43
*** yuanying_ has quit IRC03:44
*** Marga_ has quit IRC03:45
*** Marga_ has joined #kolla03:46
*** chandankumar has joined #kolla03:48
*** dims has quit IRC03:53
*** ssurana has quit IRC03:53
*** klint has joined #kolla03:54
*** CheKoLyN has joined #kolla03:55
*** achanda has quit IRC04:04
*** yuanying has quit IRC04:05
*** yuanying has joined #kolla04:06
*** coolsvap|away is now known as coolsvap04:07
*** yuanying_ has joined #kolla04:07
*** yuanying has quit IRC04:07
*** achanda has joined #kolla04:11
*** chandankumar has quit IRC04:12
*** Marga_ has quit IRC04:25
*** Marga_ has joined #kolla04:25
*** chandankumar has joined #kolla04:26
*** chandankumar has quit IRC04:37
*** weiyu has quit IRC04:43
*** tfukushima has quit IRC04:43
*** sdake has quit IRC05:05
*** unicell has joined #kolla05:07
*** dave-mccowan has quit IRC05:10
*** weiyu has joined #kolla05:16
*** chandankumar has joined #kolla05:22
*** chandankumar has quit IRC05:25
*** tfukushima has joined #kolla05:34
*** weiyu has quit IRC05:41
*** chandankumar has joined #kolla05:42
*** dratushnyy has joined #kolla05:43
dratushnyyHi guys05:44
*** Guest51478 has quit IRC05:48
*** chandankumar has quit IRC05:49
*** kjelly has joined #kolla05:50
*** weiyu has joined #kolla05:52
*** CheKoLyN has quit IRC06:00
*** Marga_ has quit IRC06:00
*** dwalsh has joined #kolla06:11
*** Allen_Gao has quit IRC06:12
*** chandankumar has joined #kolla06:22
*** chandankumar has quit IRC06:23
*** chandankumar has joined #kolla06:23
*** ajafo has quit IRC06:24
*** ajafo has joined #kolla06:25
*** Allen_Gao has joined #kolla06:28
*** Marga_ has joined #kolla06:30
openstackgerritAllen Gao proposed openstack/kolla: Playbook for upgrade memcached  https://review.openstack.org/27503506:54
openstackgerritAllen Gao proposed openstack/kolla: Playbook for rabbitmq upgrade  https://review.openstack.org/27503606:54
openstackgerritJeffrey Zhang proposed openstack/kolla: Upgrade the ansible version to 2  https://review.openstack.org/27503706:54
openstackgerritAllen Gao proposed openstack/kolla: Playbook for upgrade murano  https://review.openstack.org/27503807:04
*** dwalsh has quit IRC07:05
*** Allen_Gao has quit IRC07:13
*** coolsvap is now known as coolsvap|away07:16
*** Allen_Gao has joined #kolla07:31
*** ppetit has joined #kolla07:31
*** achanda has quit IRC07:40
*** achanda has joined #kolla07:43
*** tfukushima has quit IRC07:47
*** tfukushima has joined #kolla07:49
*** achanda has quit IRC07:55
*** kexiaodong has joined #kolla07:59
*** ppetit has quit IRC08:09
*** sdake has joined #kolla08:09
openstackgerritJeffrey Zhang proposed openstack/kolla: Upgrade the ansible version to 2  https://review.openstack.org/27503708:12
*** fgimenez has joined #kolla08:13
*** akwasnie has joined #kolla08:14
kjellyHi guys, it seems that kolla-build script never read `/etc/kolla/kolla-build.conf`. Does anyone have the same issue?08:15
*** chandankumar has left #kolla08:17
*** Marga_ has quit IRC08:24
*** dwalsh has joined #kolla08:25
*** tobe has joined #kolla08:28
*** kproskurin has joined #kolla08:29
*** athomas has joined #kolla08:31
openstackgerritJeffrey Zhang proposed openstack/kolla: Validate that all the containers are started  https://review.openstack.org/27506708:33
*** akwasnie has quit IRC08:41
*** ajafo has quit IRC08:48
*** ajafo has joined #kolla08:48
*** dwalsh has quit IRC08:50
*** akwasnie has joined #kolla08:53
*** weiyu has quit IRC08:55
*** dwalsh has joined #kolla08:58
*** tfukushima has quit IRC08:58
*** tfukushima has joined #kolla08:58
*** weiyu has joined #kolla09:02
*** mbound has joined #kolla09:02
*** tfukushima has quit IRC09:03
*** tobe has quit IRC09:16
*** dratushnyy has quit IRC09:24
*** achanda has joined #kolla09:25
*** achanda has quit IRC09:30
*** rmart04 has joined #kolla09:34
*** phong_ has joined #kolla09:35
*** ppetit has joined #kolla09:50
*** mbound has quit IRC09:52
*** mbound has joined #kolla09:52
*** weiyu has quit IRC09:53
*** rhallisey has joined #kolla09:53
*** weiyu has joined #kolla09:54
*** mbound_ has joined #kolla09:55
*** mbound has quit IRC09:55
phong_I am trying to deploy, developing multi-node Kolla on an OpenStack cloud environment that supports Heat, is there any step-by-step guide? I found it is hard to image deployment model10:04
phong_thanks10:04
*** kproskurin has quit IRC10:09
ccesariohello.... following this page... http://docs.openstack.org/developer/kolla/deploy-all-in-one-node.html ... when I try deploy I'm getting this error10:09
ccesarioeploying Playbook : ansible-playbook -i ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml ansible/site.yml10:09
ccesarioERROR: merge_configs is not a legal parameter in an Ansible task or handler10:09
*** dwalsh has quit IRC10:09
ccesariodoes someone have any idea about fix it ?10:10
Allen_Gaoccesario: what your ansible version?10:12
*** ajafo has quit IRC10:14
*** openstackgerrit has quit IRC10:17
*** openstackgerrit has joined #kolla10:17
*** akwasnie has quit IRC10:18
pbourkeccesario: please check your software versions against the table on this page http://docs.openstack.org/developer/kolla/quickstart.html10:18
ccesariolet me check...10:18
*** akwasnie has joined #kolla10:18
*** akwasnie1 has joined #kolla10:20
*** akwasnie has quit IRC10:20
*** akwasnie1 has quit IRC10:22
openstackgerritAntoine Roux proposed openstack/kolla: Fix wrong behavior when the source is a directory  https://review.openstack.org/27510410:24
*** macsz has left #kolla10:26
*** macsz has joined #kolla10:27
*** akwasnie has joined #kolla10:31
*** ajafo has joined #kolla10:31
ccesariowell... ansible is 1.9.210:32
ccesarioansible --version10:32
ccesarioansible 1.9.210:32
pbourkeJeffrey4l: what functionally are you changing here? https://review.openstack.org/#/c/275067/10:33
pbourkeccesario: min is 1.9.410:34
ccesariodoes the git trunk branch is is compatible with ubuntu-14.04.3 and liberty version!?10:34
ccesariopbourke, yes... I ḿ upgrading it10:34
openstackgerritweiyu proposed openstack/kolla: fix miss rabbitmq config when hostname include special character  https://review.openstack.org/27511110:44
*** akwasnie has quit IRC10:45
*** kproskurin has joined #kolla10:49
*** akwasnie has joined #kolla10:53
*** weiyu has quit IRC10:57
*** ppetit has quit IRC10:58
*** dave-mccowan has joined #kolla10:59
*** dims_ has joined #kolla11:02
Jeffrey4lpbourke, old logical is: if the ansible deploy success, then it is succe. new logical is: not only the ansible deploy success, and the docker containers are all up status(not exited, not created status)11:08
*** fthiagogv has quit IRC11:09
Jeffrey4lpbourke, check this log http://logs.openstack.org/38/275038/1/check/gate-kolla-dsvm-deploy-centos-source/caa9df5/console.html#_2016-02-02_07_36_07_18411:10
Jeffrey4lneutron_openvswitch_agent container exited. But the test is success.11:10
*** ajafo has quit IRC11:12
*** ajafo has joined #kolla11:13
*** dave-mccowan has quit IRC11:17
pbourkeany idea why the --filter=exited didn't pick that up?11:17
*** fgimenez has quit IRC11:19
*** fgimenez has joined #kolla11:20
Jeffrey4lpbourke, what? the filter is should be used like ` --filter status=exited`11:25
pbourkeJeffrey4l: sorry, whatever is in the existing code11:26
*** phong_ has quit IRC11:26
pbourkeJeffrey4l: it seems to me it should pick it up?11:26
pbourkeJeffrey4l: you have just added an extra filter for created11:26
kproskurinpbourke: Do you think we could merge heat trusts? https://review.openstack.org/#/c/236198/ Or we need someone else vote there?11:26
Jeffrey4lI don't know what do your mean11:26
pbourkekproskurin: I was going to wait for sdake to approve as he's more familiar with heat11:28
Jeffrey4lyou want to know why I add `--filter status=created`?11:29
pbourkeyes11:29
*** jmccarthy has quit IRC11:29
*** achanda has joined #kolla11:29
kproskurinpbourke: got it, tx11:29
*** jmccarthy has joined #kolla11:30
Jeffrey4lI just see this on my laptop. It seems the container stick at the create stage. I have no idea why and how it happend.11:30
Jeffrey4lAdding it just in case.11:31
Jeffrey4lpbourke, ^11:31
pbourkeJeffrey4l: I understand, but here's my question11:32
pbourkein the log you linked, the openvswitch container is not stuck in created, its exited11:32
pbourkeso the created filter wont catch this11:32
pbourke?11:32
*** fthiagogv has joined #kolla11:33
Jeffrey4lpbourke, the log linked is another PS. without my PS.11:33
pbourkeI know, but I dont see how your PS will fix it11:34
*** achanda has quit IRC11:34
Jeffrey4lpbourke, check this http://logs.openstack.org/67/275067/1/check/gate-kolla-dsvm-deploy-ubuntu-source/ff693b2/console.html#_2016-02-02_10_14_12_68211:35
Jeffrey4lthis is the log with my PS.11:35
Jeffrey4lThe root cause is the this line `tools/kolla-ansible deploy || print_failure` . The print_failure never be executed if the deploy success.11:36
pbourkeah11:37
Jeffrey4lSo seperate it into two line. And print the failed container and exit with code 111:37
pbourkeI see11:37
pbourkethanks!11:37
Jeffrey4l:D11:37
Jeffrey4lMaybe I should file a bug to desc this?11:38
pbourkeJeffrey4l: I think a better commit message would be great, e.g. "...bash uses lazy evaluation so if the deploy succeeded the print_failure never runs" or something like that11:39
pbourkeJeffrey4l: these things help when you're reviewing 20 patches at a time :)11:39
Jeffrey4lpbourke, yes. You are right. I will fix this. Thanks a lot.11:40
openstackgerritJeffrey Zhang proposed openstack/kolla: Validate that all the containers are started  https://review.openstack.org/27506711:47
ccesariopbourke, after upgrade..... the deploys process pass.... but now... with other error11:50
ccesariohttp://pastebin.com/0q1Rk5ZG11:50
*** tzn has joined #kolla11:52
pbourkeccesario: can you paste the versions of docker and docker-py11:54
ccesario# docker --version12:00
ccesarioDocker version 1.9.1, build a34a1d512:00
ccesarioName: docker-py12:00
ccesarioVersion: 1.6.012:00
pbourkehmm seems correct12:01
akwasnieccesario: it looks that you do not have this image  (192.168.16.228:4000/kollaglue/ubuntu-source-data:1.0.0-liberty)12:01
akwasnieccesario: if you do docker images, can you find it there?12:02
pbourketry: 'docker pull 192.168.16.228:4000/kollaglue/ubuntu-source-data:1.0.0-liberty'12:02
ccesariodocker pull 192.168.16.228:4000/kollaglue/ubuntu-source-data:1.0.0-liberty12:04
ccesarioPulling repository 192.168.16.228:4000/kollaglue/ubuntu-source-data12:04
ccesarioError while pulling image: Get http://192.168.16.228:4000/v1/repositories/kollaglue/ubuntu-source-data/images: dial tcp 192.168.16.228:4000: connection refused12:04
*** chandankumar has joined #kolla12:04
ccesariops aux | grep dock12:04
ccesarioroot      1639  0.2  0.9 317596 38484 ?        Ssl  10:00   0:00 /usr/bin/docker daemon --insecure-registry 192.168.16.228:400012:04
pbourkeis your registry running?12:09
*** rhallisey has quit IRC12:13
*** weiyu has joined #kolla12:16
ccesariohmmm... let me check12:17
openstackgerritAntoine Roux proposed openstack/kolla: Fix wrong behavior when the source is a directory  https://review.openstack.org/27510412:23
*** fthiagogv has quit IRC12:26
*** fthiagogv has joined #kolla12:27
*** kproskurin has quit IRC12:36
*** macsz has quit IRC12:36
*** fgimenez has quit IRC12:36
*** macsz1 has joined #kolla12:36
*** fgimenez has joined #kolla12:38
*** fgimenez has quit IRC12:38
*** fgimenez has joined #kolla12:38
openstackgerritAllen Gao proposed openstack/kolla: Move the install base type check to kolla-build  https://review.openstack.org/27462912:41
ccesariopbourke, it is running ... http://pastebin.com/Eb0kUMW912:44
ccesariopbourke, what branch git is recommended to use ? master ? stable/liberty!?12:44
pbourkeccesario: ok, and did you push your images to it? via the --push arg12:44
pbourkeccesario: either should be fine12:44
ccesarioI 'm removing all and re-start the build process12:45
ccesariomaster or stable/liberty branch  ?12:52
openstackgerritAllen Gao proposed openstack/kolla: Playbook for upgrade mistral  https://review.openstack.org/27517612:58
*** anteaya has joined #kolla13:00
*** logan- has quit IRC13:00
*** logan- has joined #kolla13:00
*** weiyu has quit IRC13:02
*** anteaya has quit IRC13:06
*** ajafo has quit IRC13:18
*** ajafo has joined #kolla13:24
*** Marga_ has joined #kolla13:25
*** achanda has joined #kolla13:33
*** dwalsh has joined #kolla13:37
*** dwalsh has joined #kolla13:37
*** achanda has quit IRC13:38
*** thumpba has joined #kolla13:42
*** tzn has quit IRC13:44
*** tzn has joined #kolla13:47
*** dwalsh has quit IRC13:51
*** dwalsh has joined #kolla13:51
*** Allen_Gao has quit IRC13:52
*** Marga__ has joined #kolla13:53
*** ppetit has joined #kolla13:57
*** Marga_ has quit IRC13:57
*** klint has quit IRC13:59
*** salv-orlando has joined #kolla14:01
*** chandankumar has quit IRC14:02
*** thumpba has quit IRC14:06
*** tzn has quit IRC14:07
*** tzn has joined #kolla14:07
*** sdake has quit IRC14:11
*** thomasem-away is now known as thomasem14:13
*** rhallisey has joined #kolla14:13
*** dave-mccowan has joined #kolla14:15
*** dmsimard has quit IRC14:15
*** fgimenez has quit IRC14:16
*** achanda has joined #kolla14:18
*** fgimenez has joined #kolla14:19
*** fgimenez has quit IRC14:19
*** fgimenez has joined #kolla14:19
*** diogogmt has quit IRC14:26
*** achanda has quit IRC14:27
openstackgerritAlicja Kwasniewska proposed openstack/kolla: Added Elasticearch and its deployment.  https://review.openstack.org/26771414:29
pbourkeccesario: I think you will have best results with master but stable/liberty is perfectly fine14:31
pbourkeccesario: so whatever suits you14:31
*** akwasnie has quit IRC14:32
*** rmart04 has quit IRC14:32
*** rmart04_ has joined #kolla14:32
*** rmart04_ is now known as rmart0414:32
openstackgerritMerged openstack/kolla: Fix wrong behavior when the source is a directory  https://review.openstack.org/27510414:33
*** kproskurin_ has joined #kolla14:37
*** absubram has joined #kolla14:41
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers  https://review.openstack.org/27021314:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles  https://review.openstack.org/27015114:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use private network in default settings  https://review.openstack.org/26954514:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB  https://review.openstack.org/27025214:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml  https://review.openstack.org/27456014:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510914:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove named volumes  https://review.openstack.org/27523714:42
openstackgerritEric Lemoine proposed openstack/kolla: Remove unexisting data_files in setup.cfg  https://review.openstack.org/27523814:42
*** rmart04 has quit IRC14:42
*** absubram_ has joined #kolla14:42
*** rmart04 has joined #kolla14:46
*** absubram has quit IRC14:46
*** absubram_ is now known as absubram14:46
*** jeh has joined #kolla14:49
*** jtriley has joined #kolla14:50
*** rmart04 has quit IRC14:50
*** dwalsh has quit IRC14:52
openstackgerritMerged openstack/kolla: fix miss rabbitmq config when hostname include special character  https://review.openstack.org/27511114:55
*** inc0 has joined #kolla15:00
inc0good morning15:01
*** dmsimard has joined #kolla15:02
pbourkehey inc015:04
pbourkeinc0: can I ask you a few questions on your nova upgrade patch? Or would you rather just add them all to the review15:05
inc0pbourke, shoot15:06
*** fthiagogv has quit IRC15:06
*** fthiagogv has joined #kolla15:07
pbourkeinc0: ok. first of all, why are you using docker labels in upgrade/conductors.yml, but not the others15:07
inc0pbourke, so conductors are stopped, all of them15:08
inc0this is only service that is not rolled-upgraded15:08
*** mbound_ has quit IRC15:08
inc0and I want to make sure that if I stop every conductor, I have damn good reason to do it15:08
inc0and don't do it every time I run upgrade play15:08
inc0I am ok with rebuilding and reruning every other container, as it is rolled upgrade15:09
pbourkecan you have more than one conductor running on a node?15:11
*** ppetit has quit IRC15:11
inc0technically you can, but we don't do this with kolla15:11
inc0I mean you should be able to, I'm not 100% sure, in any case that would be antipattern at best15:12
*** rmart04 has joined #kolla15:12
pbourkeso why not just upgrade it the usual way15:12
inc0because nova is wired in a way that conductors, all of them, has to be up-to-date15:12
inc0they do all the message backporting and everything15:12
*** thumpba has joined #kolla15:13
inc0so if one conductor is newer than other, if it projects a message and other consume it, finding out that it's too new for him, he will raise an error15:13
inc0thats why stop all the conductors and start new ones15:13
pbourkeok I get that, but what part does the label play in that15:13
inc0so we had to have some place to put version in15:14
inc0because minor upgrades doesn't need full stop on conductors15:14
inc0only major ones15:14
inc0so I need to be sure that upgrade I'm doing is a major one, and I can't do this by simply inspecting image15:15
inc0this is related to record-version bp15:15
pbourkethe final part of the task though is going to restart if there's a new image / openstack_release var set regardless15:16
pbourkejust for reference, I'm talking about this file: https://review.openstack.org/#/c/254395/13/ansible/roles/nova/tasks/upgrade/conductors.yml15:16
inc0yes but I don't stop it anywhere15:17
inc0which means I'm rebuilding running container15:17
pbourkeso stop+start is different to restart?15:18
inc0yes because I'm stopping all of them15:18
inc0remember that task is run on every node before moving to next one15:19
inc0whereas restarting is somewhat async15:19
*** jmccarthy has quit IRC15:19
*** jmccarthy has joined #kolla15:19
inc0there is still slight chance that at very short time there will be downtime of all conductors, but that's acceptable imho15:20
*** blahRus has joined #kolla15:20
*** dmsimard has quit IRC15:21
*** rmart04 has quit IRC15:22
*** mbound has joined #kolla15:23
inc0does that make sense pbourke ?15:24
pbourkenot yet :( Maybe I dont understand fully how Ansible executes tasks15:24
inc0ok15:24
inc0well let me put it this way15:25
ccesariopbourke,  memcached container error ..... http://pastebin.com/37GzHryq15:25
inc0if you run play, it won't go to next task before previous task is finished right?15:25
pbourkeok15:25
pbourkeacross all nodes15:26
inc0on the other hand, thanks to thread option in play (default 8) it will process more than one host during each run15:26
inc0across all nodes15:26
inc0but inside single task, there is async15:26
inc0so if you run less threads than number of conductors, you are 100% sure that there will be no downtime15:26
inc0because it will restart one at the time15:27
inc0if you run more, there is still good chance that downtime will now overlap, because container restart is so fast that it will be less than a second of effective downtime15:27
pbourkeok15:28
inc0and stop->start ensures that there is downtime and there is no conflicting conductors running at the same time all across cluster15:28
pbourkeyeah I see it now15:28
inc0because it will stop first and only after all nodes are stopped, it start new ones15:28
inc0hence "when" on stop, we don't want to do this unless we have good reasons to15:28
*** fgimenez has quit IRC15:28
pbourkemakes sense15:29
pbourkethanks15:29
*** fgimenez has joined #kolla15:29
*** fgimenez has joined #kolla15:29
inc0np15:30
pbourkeso maybe this feeds into the other thing I was asking before15:30
*** jmccarthy has quit IRC15:30
pbourkewhich is why you've split the restart out into multiple tasks15:30
pbourkeit makes sense for the conductor but the rest seem vanilla15:31
inc0because only conductors have to be up-to-date15:31
inc0because conductors does backporing of objects and they are "translators" between versions15:31
pbourkeright, but for the other services15:31
pbourkewhy not just include start.yml as in keystone15:31
inc0other nova services? hmm...good point15:32
*** rmart04 has joined #kolla15:33
inc0only reason I can think of right now is order of upgrade15:33
inc0we need to start from conductor15:33
pbourkeyeah I was thinking that but I think it could be done with some refactoring of start.yml15:33
pbourkebut its not a major deal right now15:34
inc0yeah it should be as simple as changing order of start15:34
inc0I'll look into it, thanks15:34
pbourkeso the final question is around the sighup15:34
pbourkeis it just premature optimisation15:34
pbourkeit feels hacky15:34
inc0well, it's what nova guys told me to do15:34
inc0nova have upgrade_version=auto15:35
inc0(or whatever this config is called)15:35
inc0this means it will autodetect lowest version running and backport messages to it15:35
pbourkeumm15:35
inc0sighup tells nova to re-read this and hopefully notice that it's newer than it used to be15:35
pbourkeyou're talking about this file? https://review.openstack.org/#/c/254395/13/ansible/roles/nova/tasks/upgrade/reload.yml15:35
inc0yeha15:36
inc0http://lists.openstack.org/pipermail/openstack-dev/2015-December/081591.html15:37
pbourkeI would be surprised if restarting the container wouldnt have the same effect15:38
*** JoseMello has joined #kolla15:43
*** jmccarthy has joined #kolla15:45
*** rmart04 has quit IRC15:45
*** rmart04 has joined #kolla15:47
*** dims has joined #kolla15:54
*** dims_ has quit IRC15:54
SamYaplemorning all15:55
SamYapleinc0: you know I didn't mention this last night, with a few very small tweaks we would be able to allow the user to choose between containerized libvirt and libvirt on the host system (they would do all host configuration)15:57
SamYaplethat would fit kollas motto still (dont touch the host)15:57
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Add memcached support  https://review.openstack.org/26647715:57
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Support deploy without configs in tests  https://review.openstack.org/27528715:57
*** ssurana has joined #kolla15:58
inc0SamYaple, that might be good idea to do as well15:58
inc0I have few ideas how to deal with this problem now, night sleep helped15:58
inc0however give me some more time to test/check them out15:58
SamYapleinc0: yea sure. but just letting you know. since we go over tcp its even easier. but we would need a single option (libvirt_on_host: False) and that would be _it_. We can just skip the nova_libvirt container and it would be on the user to setup libvirtd.conf correctly (we can provided needed options in docs)16:00
SamYapleill submit a patch to this effect for you today16:00
SamYapleit doesnt fix the issue, but some people might prefer that16:01
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove named volumes  https://review.openstack.org/27523716:01
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers  https://review.openstack.org/27021316:01
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles  https://review.openstack.org/27015116:01
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use private network in default settings  https://review.openstack.org/26954516:02
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB  https://review.openstack.org/27025216:02
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml  https://review.openstack.org/27456016:02
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510916:02
*** sdake has joined #kolla16:21
sdakeinc0 ping16:21
sdakewhen on earth did anyone agree we would be oding rolllbacks16:22
sdakeroll forwrd is hard  enough16:22
sdakepersonallly i think rollbacks are a near impossibility16:22
SamYaplesdake: morning16:22
sdakebut maybe down the road it can happen16:22
sdakemorning SamYaple16:22
SamYaplesdake: please for the love of god merge this! https://review.openstack.org/#/c/236198/16:22
sdakei am on half day pto today16:22
SamYapleits being held for you ::)16:23
sdakeSamYaple it has 2 +2s16:23
SamYaplesdake: waiting on you16:23
sdakelet me read full log16:23
SamYapleyou are heat guy16:23
sdaketesst it16:24
sdakegot it16:24
SamYapleI (try) to hold patches for those most familiar with the issue16:24
sdakeok well put a workflow -1 on the thing16:24
SamYapleits not my patch and it works for me16:24
sdakei think i am interruption free for the rest of th work16:24
sdakedid you use wait conditions?16:24
SamYaplebut me and you disagree on what works in heat :)16:24
SamYapleprobably not, im not a heat guy16:25
sdakeI am not usre if I can test it with my lab torn apart16:25
SamYaplebarely got a template from the interwebs working16:25
sdakei am not sure i can get wcs to work in a vm16:25
sdakewcs=waitconditions16:25
SamYaplewell nihilifer says its working. we can merge it. its not any worse than it is now16:25
sdakethat is why trusts are needed16:25
SamYapleif you cant test it really dont want to hold longer (look at age of patch)16:25
sdakelets confirm with nihilifer if wcs work16:25
sdakeif not we can get a confirmation from asalkeld16:26
SamYaplenihilifer: ^^16:26
sdakehe is also a heat dude16:26
SamYaplefair enough16:26
SamYaplejust so its on your radara16:26
sdakehis labi isn't afu atm16:26
sdakesory guys i knwo i've been out a bit in january as well as dec was a total loss16:28
sdakethe pain will end soon with the remodel16:28
sdakeand hten i'll be back at 18 hour days just like you like it :)16:28
SamYaplesdake: named volumes are done btw so thats out of your mind16:29
sdakesweet16:29
sdakenice job guys16:29
*** daneyon_ has joined #kolla16:29
sdakear enamed volumes also slow like container ovlluems?16:30
sdakelike data containers16:30
SamYapleits the same mechanism16:30
elemoinesdake, are you available for the IRC meeting tomorrow?16:30
SamYaplea mechanism ive never found slow16:30
sdakeelemoine i will make time16:30
SamYaplebut yea it would be the same16:30
elemoinethere's currently no agenda on https://wiki.openstack.org/wiki/Meetings/Kolla16:30
sdakeSamYaple if you try booting nova from the data container its alot slower16:30
sdakeelemoine whoever wants an agenda item fixes the agenda meeting calendar :)16:31
sdakeit typically just takes care of itself16:31
elemoinesdake, makes sense16:31
sdakei never touch it and its always ready to rock :)16:31
sdakeok i think our vote is done for hte time to select our new meeting16:31
sdakebut i need some time toe coordinate that with the infra team, so that wont start until next week16:31
openstackgerritMerged openstack/kolla: Playbook for upgrade murano  https://review.openstack.org/27503816:32
sdakeok i actually have to prep for this webinar interview for hte foundation and get a shower and get my contractors sorted out and maybe fit in a hair cut!16:32
inc0sdake, pong16:32
sdakeinc0 rollback - we ren't doing it16:32
sdakei wasn't clear on your email, were ou suggesting not doing thin containers as well16:33
*** daneyon has quit IRC16:33
elemoinesdake, I updated the wiki page16:33
inc0yeah, what I mean is to remove code that is not docker 1.8 compatible16:33
elemoineI kept SamYaple's items as he wasn't there last time IIRC16:33
sdakeSamYaple plesae try to be at next meeting yu had some agenda items16:34
inc0quick question guys16:35
inc0if we rebuild contrainer with vms16:35
inc0vms will get not only restarted, but actually removed right?16:36
sdakeso there is a bug16:36
sdakevms are restartedi if the container is restarted16:36
sdakebut in liberty i tested it iand the vms for certain did not restart16:36
inc0and container is restarted when docker is restarted16:36
sdakethey stayed active and libvirt connected to them16:36
sdakei have multiple people owkring on this problem in the docker community16:36
inc0which version of docker?16:36
sdakethey recognize the implications of vm downtime16:37
sdakeinc0 the docs for liberty say 1.7.0 and i was doing the 2 weeks of making that vm shit work at the end of hte cycle16:37
sdakemostly l3/rc116:37
*** daneyon has joined #kolla16:37
sdakeit otok me 2 weeks16:37
sdakewe have identified a regression in the new version of docker16:37
sdakeor in how we are using it16:37
inc0so currently I can't get pid=host stuff work on docker restart16:37
sdakesomething is broken and htat needs to be fixed, but we can upgrade docker daemon without vm downtime and upgrade libvirt container without vm downtime in liverty16:38
inc0and we don't have migration plan from data container to named volume16:38
inc0sdake, not the case any more it seems16:38
sdakeinc0 we can get a migration plan for data container to named volume16:38
sdakeinc0 i know as i said i have multiple cats working on it16:38
sdakewe haev to trust they will get the job done16:38
sdakethey are seriously the #1 experts in all this pid=host space16:39
sdakeand they agreed current behavior is broken16:39
inc0ok, will we get this before mitaka release?16:39
sdakegod i hope so16:39
sdakeotherwise we wont be upgrading the libvirt container16:39
inc0and will we get 1.8 ->1.11 without restarting vms?16:39
sdakekill -9 docker still keeps the vms intact16:39
sdakesamyaple tested and confirmed that16:39
*** daneyon_ has quit IRC16:40
inc0I'll check out kill -916:40
inc0service docker restart breaks stuff16:40
sdakekill -9 works16:40
sdakeeverything else not sure16:40
sdakeour upgrade is kill -9 all nodes of docker, upgrade rpm, start docker daemons16:41
sdakesorry all docker dameon nodes16:41
inc0if that will keep vm intact, I'm ok16:42
openstackgerritMerged openstack/kolla: Playbook for upgrade memcached  https://review.openstack.org/27503516:42
inc0I mean we still won't be able to work with data container16:42
inc0but that's different issue to tackle16:42
inc0ok, so kill -9 does preserve vms, but starting docker removes it16:44
*** intr1nsic has quit IRC16:44
inc0which can be solved with 1.11 if your docker ppl will take that into account16:44
inc0so being able to run container on pid that's running16:44
SamYaplesdake: im -2 on kill -9 period16:45
SamYaplethat can corrupt the docker database16:45
SamYapleinc0: the issue here is not solved with docker 1.1116:46
inc0and docker start breaks stuff anyway16:46
SamYapleyou still need to upgrade docker contaienr for qemu16:46
SamYaplenova_libvirt16:46
sdakeok well lets figure out what the hell the requireements are and give them to me and i'll get them done in docker16:46
openstackgerritEric Lemoine proposed openstack/kolla: Remove unexisting data_files in setup.cfg  https://review.openstack.org/27523816:46
inc0but we can decouple qemu upgrade from kolla upgrade16:46
SamYaplefrom nova upgrade, not kolla upgrade16:46
SamYaplebut i agree16:46
SamYapleit can be decoupled16:46
SamYaplemaybe16:47
sdakeyes decouple16:47
SamYaplenova does have libvirt requirements, and those change16:47
inc0it can be decoupled from kolla upgrade as kolla code16:47
openstackgerritMerged openstack/kolla: Validate that all the containers are started  https://review.openstack.org/27506716:47
SamYaplei know the supported list of libvirt and qemu changes16:47
inc0yes but unless it changes we don't have to do it16:47
SamYaplebut it does every release16:47
*** dims_ has joined #kolla16:47
inc0but you can do it later16:47
inc0not all at once16:47
inc0and that's important16:48
SamYapleright, but you wont be in a supported package state until you do16:48
SamYapleand that might break service agreements16:48
SamYaplejust something to keep in mind16:48
inc0all I'm saying, let's provide option not force it on ops16:48
inc0we need to upgrade qemu and docker at some point - agree16:49
SamYapledecouple i am ok with, but its still a requirement for a full upgrade from one version to another of kolla16:49
SamYapleare we in agreement on these points?16:49
inc0but what I don't agree with is to upgrade it at the very same day we upgrade openstack16:49
*** dims has quit IRC16:49
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Add memcached support  https://review.openstack.org/26647716:49
inc0problem is, due to named volumes and thin containers, you can't do this16:49
SamYaplecant do what?16:49
inc0upgrade kolla without vm downtime16:50
sdakeotp brb16:50
SamYaplei know16:50
inc0because new kolla won't work16:50
inc0at all16:50
inc0so no, you can't decouple it16:50
kproskurin_sdake: Could you pls look into heat trusts patch? https://review.openstack.org/#/c/236198/ I think it ready to be merged16:50
SamYaplewell you can16:50
inc0and old kolla won't work with new docker so you can't upgrade docker and then upgrade kolla16:50
SamYapleyou just have to decouple the qemu named volume switch too16:50
SamYaplebut youre right, this is a full stop of the vms for the named volume and qemu switch at this time16:51
sdakesounding good sam16:51
inc0there is no version of kolla that supports 1.8 and 1.1016:51
inc0at the same time16:51
SamYapleuhhh16:51
SamYapleyea there is16:51
SamYapleliberty runs just fine on 1.1016:51
SamYapleyou just cant run our playbooks16:51
SamYapleyou can upgrade docker in liberty16:52
SamYaplethen run the new playbooks for mitaka16:52
inc0but during docker upgrade you don't have kolla plays to help you16:53
sdakeyes agree this is the approach we should tkae as we discussed friday16:53
SamYapleinc0: this is true, but the way we have built the containers by default they will restart16:53
*** rhallisey has quit IRC16:53
SamYapleif the default restart policy is changed then its on the operator to restart them16:53
SamYaplebut they should be aware of that as they changed teh option16:54
*** absubram has quit IRC16:55
sdakeok so what do we need16:55
sdakewe need a docker that doesn't restart vms on a container upgrade16:56
sdakewhich used to work16:56
sdakeand no longer does16:56
sdakewhat ELSE do we need16:56
SamYaplesdake: that is an option in docker that doesnt exist16:56
SamYapleim working that part out16:56
sdakeSamYaple it WAS working16:56
SamYapleill hit up the apprpriate channel16:56
sdakei will get my people to getit working again16:56
SamYaplesdake: actually it _wasnt_ according to docker16:56
*** mbound has quit IRC16:56
SamYaplethat was a bug16:56
sdakeoh that was a bug?16:56
SamYaplethats why the pid=host containres couldn't fully stop16:56
*** achanda has joined #kolla16:57
sdakeand they fixed the bug and broke the function we depended on?16:57
sdakehae a issue tracker #?16:57
SamYaplegoogle knows i do not16:57
sdakedid you see it though or just guessing16:57
SamYaplebut anyway, we need that as a feature _and_ we need docker daemon restart without stopping containers16:57
SamYaplesdake: no thats what happened16:57
sdakeso your speculating? :)16:57
SamYapleno thats what actually happened16:58
SamYaplethere are bugs out there for this16:58
sdakeso you saw evidence16:58
sdakeok16:58
SamYaplei dont know them its been like a year16:58
sdakegood you saw evidence of a bug, i can work from there16:58
sdakeok docker daemon restart without topping containers, the answer to that is runc16:58
SamYapleinc0: so what we need is a shared pid namespace. and docker is working on that16:58
sdakethat wont be ready for mitaka16:58
SamYapleinc0: with a shared pid namepsace we will be solid16:59
SamYaplesdake: the docker daemon restart will be ready for mitaka16:59
*** blahRus1 has joined #kolla16:59
SamYaplesdake: the "qemu process survives container stop and recreate" will not be ready16:59
sdakeSamYaple docker palns to solve this problem with shared pid namespace?17:00
sdakedwalsh ^^17:00
sdakedamn17:00
sdakenot here17:00
SamYapleno it doesnt plan anything to "solve" this issue17:00
inc0they have "hot upgrades" in roadmap17:00
inc0not sure what that means17:00
SamYapleinc0: thats teh docker daemon restart17:00
sdakeok with the host=pid thing here is what is going to happen17:00
sdakemark my words, mark them well17:01
sdakewe will get an option in docker17:01
inc0I know, not sure if that's about shared pid namespace17:01
*** blahRus has quit IRC17:01
inc0but that still needs kill -9 mind you17:01
sdakethe option will be "let all the processes be managed by pid=117:01
inc0and Sam has point - this can break things17:01
inc0otherwise docker explicitly kills container17:01
sdakenot kill -917:01
inc0yes, but having option in new docker doesnt help us17:02
inc0because we still need to turn off 1.817:02
sdakeso if this option is used, docker will give up responsibility for killing pids with pid=host17:02
inc0and that kills containers17:02
sdakeif as SamYaple claims they fixed bugs to make that happen17:02
sdakeinc0 got it17:03
sdakekill -9 wfm ;)17:03
SamYaplesdake: nevertheless im -217:03
sdakethe docker database is consistent17:03
SamYaplethats a good way to loose all of your data17:03
sdakeif docker can't handl a process crash we are doome17:03
sdaked17:03
sdakea kill -9 is a process crash just different uncatchable signal17:04
sdakeif segv hits docker17:04
sdakei highly highlly doubt the docker db woudl be corrupted17:04
sdakein such a way as to totally destroy the system17:04
sdakeif it is, then we are fucked17:04
inc0if it does, we have much bigger issue to solve here17:05
sdakeok gotta do some stuff17:05
inc0ok17:05
SamYaplewell im telling you that _can_ happen17:05
SamYaplewe don't want it to17:05
sdakei'll be back but not to the channel until after 3-4pm17:05
openstackgerritMerged openstack/kolla: Playbook for upgrade mistral  https://review.openstack.org/27517617:05
inc0ok I'll work on migrating data container to named volume17:05
*** achanda has quit IRC17:06
sdakei hae to prep for haircutting and taking lucnh to my kid at school and webinar17:06
inc0I might have super-hacky idea17:06
sdakeinc0 good that is something we need17:06
sdakesuper hhacky is good17:06
sdakewe are doing it one time17:06
sdakewe may carry it cuple releases17:06
sdakeopenstack recommends 2 releasses before deprecating old stuff17:06
sdakeatleast that is how it used to be not sure now17:06
*** dwalsh has joined #kolla17:06
inc0I guess if we have 1 release for us it's ok17:07
inc0but anyway, we need deprecation process and docker migration process17:07
inc0kill -9 keeps vm, we will test it out to make sure it won't destroy your data17:07
inc0then we wait till docker enables shared pid namespace17:07
inc0and attach to running pids17:08
inc0and we'll do this along with migration to data container17:08
SamYapleinc0: i have the named migration covered17:08
Jeffrey4lBut the binary are still in the old container, right?17:08
inc0SamYaple, do share plz17:08
SamYapleinc0: as i stated in the initial named volume patch17:08
SamYapleinc0: so there are two paths17:09
SamYaple1) lookup old path via docker inspect and do a host-level `mv` operation17:09
SamYaple2) new container with old volume and new named volume and do mv inside container17:10
inc0that was hacky way I had in mind17:10
SamYaplei like 2)17:10
inc0yeah 2 seems better17:10
*** mbound has joined #kolla17:10
inc0we can do "migration volume"17:10
inc0"migration container"17:10
SamYapleno we dont actually need a new container for this17:10
inc0however that will also cause downtime17:10
SamYaplebut yes it will cause downtime17:10
inc0yeah it's just ubuntu container with one command17:10
SamYapleright17:11
SamYapleill just have to introduce the command option back to kolla_docker, not an issue17:11
inc0so if we do symlink instead17:11
inc0it might not cause downtime17:11
SamYaplewont work on all underlying filysystems17:11
inc0symlink on host level17:11
SamYaplealso i worry about the overlyafs stuff too17:11
SamYapleso i want to only use 1) as a last resort17:12
inc0yeah, well, this problem should be solvable tho17:12
inc0we just need to figure out how to pretend that container data is volume17:12
SamYaplehost level things are going to make that hard/impossible17:12
SamYaplewe cant account for every host level change and symlink doesnt atually move the data17:13
SamYaplewhat if docker doesn't allow symlink (or worse, they allow it now and it breaks in the future)17:13
inc0how about we ask docker to introduce yet another thing - make volume out of existing data17:13
SamYapleyea but thats not going to happen for Mitaka17:13
SamYapleand then it doesnt matter anyway17:13
SamYaplebtw what you are asking for is volume rename17:13
SamYapleand it has been asked before17:13
inc0SamYaple, how angry would you be at me if I'd move us back to data containers?17:14
inc0because if we do that17:14
inc0we can say that "run mitaka on 1.8"17:14
sdakewe are not going back to data container17:14
*** salv-orlando has quit IRC17:14
sdakeok this is definately midcycle topic17:15
inc0and by the next release we'll have all the docker stuff we need17:15
sdakei really have to run guys17:15
inc0ok, let's discuss it in midcycle17:15
sdakei will bbe back in about 5 hours17:15
sdakefull blast rest of day17:15
sdakekeep discussing now17:15
sdakeget the ideas out there17:15
SamYapleinc0: not about angry, data contaienrs dont solve any issues17:15
inc0don't bring weapons, bare hands will have less chance to kill us17:15
sdakebut just pointing ut we will need to rehash this at midcycle17:15
inc0SamYaple, will allow us to keep docker 1.8 till next release17:15
*** sdake has quit IRC17:15
SamYapleno it wont17:16
SamYaplewe need 1.1017:16
inc0for thin containers17:16
inc0I know17:16
SamYaplefor thin container17:16
inc0but we can hold this till next release as well17:16
inc0what I'm asking is deprecation period for 1.817:16
inc0docker 1.817:16
SamYaplelol like i said reverting an entire cycle17:16
inc0of a single release17:16
SamYapleand you still dont solve the data contaienr issue17:17
inc0no, kolla_docker will actually save our asses17:17
inc0because it works on all of these17:17
SamYapleinc0: https://bugs.launchpad.net/kolla/+bug/152123117:17
openstackLaunchpad bug 1521231 in kolla "rabbitmq fail to restart on image update/upgrade" [Critical,In progress] - Assigned to Vladislav Belogrudov (vlad-belogrudov)17:17
SamYapledata contaienrs are a flawed idea we should have never used17:17
inc0ok, host bind instead of named volume then17:17
inc0for 1 release17:17
inc0I know it's against our policy of clean host17:18
SamYaplewe still need to move the data...17:18
SamYaplesame issue17:18
inc0well, not exactly17:18
SamYaplehow so?17:19
inc0you can do it slowly and one container at the time17:19
inc0whereas we propose small revolution17:19
SamYaplesame with named volumes though17:19
SamYapleslowly move some data17:19
SamYaplenot all at once17:19
inc0but you have to upgrade docker17:19
inc0in between17:19
*** fgimenez has quit IRC17:19
SamYapleno you update docker then pull mitaka playbooks17:20
SamYapleupdating docker is done on liberty17:20
SamYapleso docker is updated and ready for this17:20
inc0so bottom line, we make sure that docker can reattach itself to running pid17:20
SamYaplewe cant get around these downtimes. but honestly downtimes were _always_ expected. we said hard downtimes during tokyo was expected17:21
SamYapleinc0: its not docker its libvirt17:21
SamYaplelibvirt can do that17:21
inc0well my point is next containers we run will have same VMs we had17:21
inc0before17:21
SamYaplewe want docker to disown the processes17:21
SamYapleand forget about them17:21
inc0which can happen if we kill -9 it17:21
SamYaplebut not on a start again17:21
SamYaplelike you said17:21
inc0but that can be done with docker part17:22
SamYaplereally what is wanted is a complete disowning of the process17:22
SamYapleno tracking. completely forget it17:22
inc0or being able to take on running process from previous docker17:22
inc0either one works17:22
SamYapledoesnt solve the liberty-to-mitaka thing at all17:22
inc0not killing it - tacking ownership17:22
SamYapleso well save that for conversation for another time17:23
inc0midcycle17:23
SamYaplemidcycle should be problems at hand mostly17:23
inc0remember, no weapons or sharp objects17:23
SamYaplei think we have enough of those17:23
inc0that is problem at hand17:23
inc0that is upgrade path17:23
inc0discussion17:23
SamYapleok but im talking liberty-mitaka upgrade17:23
SamYaplewe do that discussion first17:23
SamYaplethen if time mitaka-newton17:23
SamYaplebut well have time at summit for that too17:24
SamYapleok im going to lunch17:24
SamYapleinc0: ill get you that libvirt_on_host patch today17:24
inc0I'm ok with moving libvirt stack to host17:25
SamYaplenot by defaut17:25
SamYapleas an option17:25
SamYapleif you want it on host, go for it17:25
inc0ok, whatever, as upgrade17:25
SamYapleno17:25
inc0then later we can try to attach to it17:25
SamYapleno im saying an option to allow you to run livirt_on_host if you choose17:25
SamYaplebut not the default17:26
inc0as upgrade with option I personally will recommend, may not be default but I'm fairly sure will be pretty populart17:26
SamYapleyou might be right17:26
SamYaplei dont disagree17:26
*** salv-orlando has joined #kolla17:26
SamYapleits been asked before17:26
inc0also I'm ok with having new vms running in container as we do now17:26
inc0just...we need docker to solve this stuff17:26
SamYaplemy only point is it wont be default, options are great17:26
SamYaplebut by default we dont touch the hose17:27
SamYaplehost*17:27
inc0what I'd like to see is also migration plan container->host17:27
SamYaplethats easy too17:27
SamYapleif we do 2) of the migration plan17:27
inc0it should be17:27
SamYapleits time consuming potentially though17:27
inc0well we won't really have to17:27
SamYaplepotentialyl thats moving cross file-system17:27
inc0but we can do this lm-style17:27
inc0and somehow reconnect qemu to new file after we're done17:28
inc0and if we use ceph, we're golden17:28
SamYapleceph is golden, moving qcow2 is slow17:28
SamYaplewe arent going to send qemu commands over qmp to move the files cross file-system though17:28
inc0well ceph is default anyway so I don't really care that much17:28
inc0ephemeral is emphemeral, disks die17:28
SamYapleephemeral is on ceph too ;)17:28
inc0it's not that ephemeral17:29
SamYapleanyway moving on to other issue. this has solutions17:29
inc0I mean if you use ephemeral in ceph, they're not files in fs17:29
SamYaplewhich is the default17:29
inc0so if you use ceph for ephemerals, which we do by default and suggest17:29
*** thumpba has quit IRC17:29
inc0there is no data to migrate17:30
SamYaplenova backed by ceph IS ephemeral backed by ceph17:30
inc0I know, so we zombify qemu processes, move libvirt to host and reattach to qemu17:30
inc0and then upgrade docker with API downtime but VMs stays alive17:30
inc0sounds like a plan?17:31
SamYaplethat wont work17:31
SamYaplebut you can try17:31
inc0how so?17:32
SamYapledocker wil lstill kill qemu17:32
*** jasonsb has quit IRC17:32
*** openstackgerrit has quit IRC17:32
*** openstackgerrit has joined #kolla17:32
inc0well I assume old version will not (kill -9) and new version will not as we'll push sdake to use hardest whip he can to make sure it won't17:32
inc0I wanted to learn golang and I'll just start writing this stuff if that's the case17:33
*** Jeffrey4l has quit IRC17:33
SamYaplefocus man17:33
SamYapleyou have no focus17:33
inc0I'm ok with saying on summit "this is docker release preview, to do upgrades we need features from it so please hold on till it's released"17:34
SamYapleit wont be in 1.1117:34
inc0noone deploy openstack on day 1 anyway17:34
ccesariopbourke,  it is running :)17:35
pbourkeccesario: great :)17:35
pbourkeccesario: did you figure out what was your original problem?17:35
*** kproskurin_ has quit IRC17:38
ccesariothe first one it was the ansible version...  the second it was the registry service.... and the last one kolla_install_type: "source"  ... by default binary it was used ....  I'm still  validating all services....17:38
ccesarioonly mismatches caused by user side17:40
*** unicell has quit IRC17:41
*** mbound has quit IRC17:42
*** sdake has joined #kolla17:48
*** sdake has quit IRC17:50
*** salv-orl_ has joined #kolla17:50
*** rhallisey has joined #kolla17:53
*** salv-orlando has quit IRC17:54
limamauricioSamYaple, i'm doing the last tests with the Manila and now I get this error -> OSError: [Errno 13] Permission denied: '/var/lib/manila/tmp'17:57
SamYaplelimamauricio: can you post a patch17:58
SamYaplelimamauricio: that is for the oslo_lock stuff17:58
limamauricio[oslo_concurrency] lock_path = /var/lib/manila/tmp17:59
SamYapleyea but the containter clearly has no perms there. if you are working on the most recent code, that would make sense18:00
*** pbourke has quit IRC18:00
*** pbourke has joined #kolla18:00
limamauriciotmp is a dir or a file?18:01
SamYapledir18:02
SamYaplebut manila user should have /var/lib/manila perms18:02
*** jmccarthy has quit IRC18:02
limamauricioI add a manila_sudoers file on manila-base dir18:02
limamauriciothis error happens in share_container18:03
SamYaplehold on limamauricio, ill submit the patch for you to fix this issue (rather than trying to describe it further)18:03
limamauriciook18:04
*** rmart04 has quit IRC18:04
ccesariopbourke, does kolla support upgrades ? Kilo->Liberty by example ? if yes ... is there any document about it ?18:04
*** rhallisey has quit IRC18:05
openstackgerritSam Yaple proposed openstack/kolla: Fix manila perms  https://review.openstack.org/27535918:05
SamYaplelimamauricio: https://review.openstack.org/27535918:06
*** tzn has quit IRC18:06
*** athomas has quit IRC18:10
*** athomas has joined #kolla18:11
*** athomas has quit IRC18:12
openstackgerritSam Yaple proposed openstack/kolla: Rename kolla_ansible to kolla_toolbox  https://review.openstack.org/27415418:13
*** intr1nsic has joined #kolla18:13
*** unicell has joined #kolla18:16
*** achanda has joined #kolla18:19
*** thumpba has joined #kolla18:21
*** thumpba has quit IRC18:27
*** salv-orl_ has quit IRC18:28
*** salv-orlando has joined #kolla18:28
*** salv-orlando has quit IRC18:31
*** salv-orlando has joined #kolla18:32
*** dwalsh has quit IRC18:36
*** thumpba has joined #kolla18:42
*** salv-orlando has quit IRC18:43
*** dwalsh has joined #kolla18:45
*** salv-orlando has joined #kolla18:45
*** absubram has joined #kolla18:46
*** rmart04 has joined #kolla18:46
*** tzn has joined #kolla18:47
*** tzn has quit IRC18:48
SamYaplehey inc0 user namespace remapping has landed in docker 1.1018:49
SamYapleall the drop-root work we did was for naught!18:49
*** rmart04 has quit IRC18:56
*** whydidyoustealmy has joined #kolla19:03
limamauricioSamYaple: I have the same error19:05
SamYaplelimamauricio: did you fully rebuild your contianers?19:05
limamauricioyes :/19:06
SamYaplehmm im not sure. was there a typo? the fact is if the container is running as the manila user it should have access to /var/lib/manila with my patch19:06
*** vincent_1dk has joined #kolla19:08
limamauricioI accessed the container and the tmp directory does not exist.19:10
*** Kennan has quit IRC19:12
*** barra204 has quit IRC19:12
*** kbyrne has quit IRC19:12
*** Slower has quit IRC19:12
*** vincent_vdk has quit IRC19:12
*** Slower has joined #kolla19:14
*** kbyrne has joined #kolla19:15
*** Kennan has joined #kolla19:16
*** dwalsh has quit IRC19:25
*** salv-orlando has quit IRC19:25
*** dmsimard has joined #kolla19:31
*** ppetit has joined #kolla19:34
*** dmsimard has quit IRC19:37
*** dmsimard has joined #kolla19:41
*** dmsimard has quit IRC19:41
*** dmsimard has joined #kolla19:46
openstackgerritQiu Yu proposed openstack/kolla: Update register playbooks to use per-role endpoint address  https://review.openstack.org/27538619:48
*** whydidyoustealmy is now known as barra20419:53
*** Marga__ has quit IRC20:00
*** thumpba has quit IRC20:00
*** ppetit has left #kolla20:01
SamYaplelimamauricio: is that the error you get?20:05
SamYapleit should be creaintg the tmp dir for you20:05
*** achanda has quit IRC20:11
*** salv-orlando has joined #kolla20:28
*** sdake has joined #kolla20:47
*** thumpba has joined #kolla20:51
*** achanda has joined #kolla20:52
*** thumpba has quit IRC20:56
*** thumpba has joined #kolla20:59
*** salv-orl_ has joined #kolla21:04
*** salv-orlando has quit IRC21:04
*** salv-orl_ has quit IRC21:08
*** ssurana has quit IRC21:08
*** salv-orlando has joined #kolla21:08
*** CheKoLyN has joined #kolla21:15
*** tzn has joined #kolla21:16
*** Marga_ has joined #kolla21:22
*** Guest40848 has joined #kolla21:23
openstackgerritMerged openstack/kolla: Add an ansible role for mongodb  https://review.openstack.org/26939321:25
*** sdake has quit IRC21:29
*** kproskurin has joined #kolla21:31
openstackgerritMerged openstack/kolla: Fix Swift for multinode P+ACO deployment  https://review.openstack.org/27421521:39
*** inc0 has quit IRC21:48
*** achanda has quit IRC21:48
*** thumpba has quit IRC21:54
*** sdake has joined #kolla21:57
openstackgerritSam Yaple proposed openstack/kolla: Create nova-api database  https://review.openstack.org/27492821:57
sdakeyo21:57
SamYapleyo sdake21:59
SamYaplebtw nova has been broken for a week if you werent aware21:59
*** achanda has joined #kolla21:59
kproskurinHi sdake, could you pls look into heat trusts review? https://review.openstack.org/#/c/236198/22:07
SamYaplekproskurin: sdakes lab is down. I have tested it but i did not excersize it22:08
SamYaplekproskurin: sdake said if you get asalkeld sign off on the patch we can merge it22:08
SamYaplejust let me know and ill +W22:08
SamYaplesince asalkeld has +1'd the patch I would assume its good, but i just want to ask the question to him directly first22:09
kproskurinSamYaple, well asalkeld gave his +1 and I did some testing too, for kolla mesos...22:09
kproskurinok, no problme22:09
SamYapledid you test wait conditions?22:09
* SamYaple is not a heat guy, im just repeating sdakes questions22:09
kproskurinIm not a heat guy too, so I not really sure what wait condition is. I deployed it like 20 times, check some heat cli calls and so on22:10
SamYapleyea i did too22:10
SamYapleapparently thats not enough (or something) so i was going to ask asalkeld about wait conditions and merge22:11
kproskurinOk, so lets wait for asalkeld response than, no problem22:11
SamYaplefyi kproskurin https://bugs.launchpad.net/kolla-mesos/+bug/154057922:12
openstackLaunchpad bug 1540579 in kolla-mesos "nova-api -- No sql_connection parameter is established" [Undecided,New]22:12
SamYaplethe kolla-ansible fix is here https://review.openstack.org/#/c/274928/22:13
*** inc0 has joined #kolla22:13
SamYaplethe kolla-ansible fix will need to land then you can fix mesos22:13
inc0sdake, around?22:13
SamYapleyo inc022:13
sdakeotp but will be soon22:13
inc0day of meetings \o/22:13
kproskurinSamYaple, nova is in merge conflict right now, waiting for asalkeld to fix it, so I didnt test it yet for mesos22:14
SamYapleok kproskurin ill let asalkeld know then by commenting on review22:14
kproskurinok, tx22:15
*** bmace has joined #kolla22:18
*** ssurana has joined #kolla22:19
*** sdake has quit IRC22:24
*** sdake has joined #kolla22:26
sdakeinc0 ok i'm off the phone22:26
sdakewhats up22:26
*** kproskurin has quit IRC22:27
inc0so, how good are these docker folks of yours? because we might have idea22:29
inc0SamYaple, correct me plz22:29
SamYapleinc0: me correct you?22:29
inc0but if we kill -9 docker 1.8 vm keeps going22:29
SamYaplepffff22:29
inc0we end up with pid on host and some data22:30
inc0in container22:30
inc0problem is, if we start docker again, it will kill running pids22:30
*** achanda has quit IRC22:30
SamYapleinc0: this may be of interest to you22:31
inc0so if we explicitly tell docker not to do it, we'll have effectively qemu on host, outside of container22:31
SamYaple"20:35:48 < cpuguy83> SamYaple: As for disowning a process like this, I don't think that will ever be supported... for being able to restart a daemon w/o having to kill all the processes in a container, yes, most definitely that should be supported at some point."22:31
SamYaplecpuguy83 said no22:31
SamYaplealready asked this question22:31
inc0ok, but if we teach docker to, instead of killing, restore containers from pids22:31
inc0we're in even better position22:31
SamYapleconversation was a bit longer than that, but pid existing outside of docker daemon is a bug. why would they reattach to existing ones rather than kill22:32
inc0bottom line, docker will kill vms on startup (if we previously kill -9 it), this can't happen22:32
SamYaple(according to them)22:32
inc0which is way to go in my opinion as well22:32
SamYaple"20:39:08 < cpuguy83> SamYaple: I think it's outside the Docker use-case. You could look at runc for this."22:32
SamYapleagain this was asked and rejected. maybe presure will help but i doubt it22:33
inc0well we are where we are22:33
SamYaplethis _is_ outside teh docker use-case22:33
inc0docker startup kills vm22:33
SamYaplethis is the desired affect inc022:33
inc0not by us it's not22:33
SamYaplethe only hope I can think of is shared pid-namespace22:33
SamYaplewell we can "look at runc for this" ;)22:33
SamYapleinc0: with shared pid namespaces we get "20:40:24 < SamYaple> so instead of host its another docker container holding on the qemu processes like shared volumes_from22:35
SamYapleinc0: sdake see https://github.com/docker/docker/issues/1016322:35
SamYaplethats the issue that may solve this for us long term22:35
SamYapleso i would think preassure is best applied there22:35
SamYaplei was shutdown pretty hard suggesting the disown thing :/22:36
inc0I also thing that might be way to go22:36
inc0reattaching to running containers will make our job easier22:36
SamYapleshared pid namespace?22:37
inc0yeah22:37
inc0with reattach option22:37
SamYapleok that will work if we can guide development of that feature22:37
SamYaplebecause it could very well not work too22:37
SamYaplelike each container tracks its own pid and there is no ownership sharing22:37
inc0I agree, it might work, that's where sdake's guys come in22:38
inc0also there is that: https://github.com/docker/docker/issues/265822:38
inc0this is roadmap, and that implies that docker will have to have some sort of re-attach22:38
SamYapleyea but thats happening22:38
SamYaplereattach to namespaces is different22:38
SamYaplewe are talking processes inside the naemspace22:39
inc0wonderful, we can help in a way so we can laverage reattach22:39
inc0ok, what I mean is22:39
inc0to achieve hot restart of docker22:39
SamYapleyea but thats happening. that is supposed to be in 1.1122:39
inc0docker daemon after restart needs to be able to reconnect to running containers22:39
inc0SamYaple, great, I know, it is possible to have it in Mitaka at least in form of RC22:39
SamYaplethats not _inside_ the namespace though, its just not killing the namespace on stop/start22:39
inc0which is exactly what we need really22:40
*** dwalsh has joined #kolla22:40
SamYaplethe pid sharing feature would need to be guided to allow non-watched pids to be transient between the two containers i think22:40
SamYaplethats only part of the issue inc022:40
SamYaplethe qemu container restart is the bigger one22:40
SamYapleno getting around that one ever22:40
inc0point is, it potentially might help us22:41
asalkeldo/22:41
SamYapledocker upgarde is a great solution to a problem. but thats not _the_ problem22:41
SamYapleyo yo asalkeld22:41
inc0it very well may not, but it can, and we should move heaven so it will22:41
SamYapleasalkeld: if you could just confirm trusts are _actually_ working with this patch ill merge it https://review.openstack.org/#/c/236198/22:41
inc0SamYaple, problem for us right now is that docker upgrade causes qemu restart22:41
inc0upgrades of qemu is different problem I don't want to tackle now22:42
SamYapleinc0: we have different opinions on that. we have a hard stop of qemu for file moves anyway. period22:42
inc0I want to be able to upgrade openstack with kolla without ever touching qemu22:42
asalkeldSamYaple: looking22:42
SamYapleany upgrade,ever point releases, are going to upgrade nova_libvirt22:42
inc0with ceph you don't22:42
inc0not playbook I've written22:42
SamYaplethen you aren't doing upgrade and oyu need to add that22:43
inc0and I will argue against it22:43
inc0no I don't22:43
SamYapleits a requirement...22:43
SamYapleand you will yes22:43
inc0I want to do it in separate thing22:43
SamYapleits needed for upgrades22:43
inc0infra upgrades may be separate22:43
SamYapleits not an infra upgrade.22:43
SamYapleits a container we built22:43
SamYapleit has to be upgraded22:43
inc0it does not22:44
SamYapleyou are absolutely wrong22:44
inc0it might at some point if nove requires it22:44
SamYapleall containers are on the same base images for a reason22:44
inc0wonderful, you can just not touch it22:44
inc0or you can upgrade it PRIOR to openstack upgrade22:44
inc0or after22:44
SamYapleyou can do that if _you_ want, but the default is to upgrade the container22:45
inc0my point, you dont have to do it at this very moment22:45
SamYapleyou do have to do it for the upgrade to be complete22:45
inc0no, default is 2 separate plays22:45
inc0I'm -2 on forcing upgrade qemu along with openstack22:45
inc0but you can do it in separate time-scale22:45
SamYapleyou arent going to block upgrade by being stuborn inc022:45
*** jtriley has quit IRC22:45
inc0I'm going to block imposing forced downtime22:46
SamYapleqemu containres must change when we build new base images22:46
inc0but running ones don't have to be touched22:46
*** asalkeld has quit IRC22:46
SamYapleyou cannot unilaterally make a change to what Kolla has always been22:46
inc0well, upgrades are part of it22:46
inc0with minimal downtime22:47
inc0I think less downtime is worth breaking from policy22:47
SamYaplei dont care man, you can argue all day long but we are keeping all containers on the same base images22:47
inc0if we provide good way to get consistent with it22:47
*** asalkeld has joined #kolla22:47
SamYapleuntil that happens upgrade is not complete22:47
inc0at *separate* timescale22:47
inc0even if that means 6 months22:47
SamYaplenope22:48
SamYaplewont work22:48
SamYapleneeds to stay on same base image22:48
SamYapleyou are diverging from the same base image22:48
inc0why is that exactly?22:48
asalkeld:headdesk: updated distro, "do you want to reboot now or later" - /me presses the wrong button - nooo22:48
inc0having separate requirement space is exactly what contianers give youy22:48
SamYapleyou feel free to lock in nova_libvirt to a certain version in your globals.yml, thats why these options exist22:48
SamYapleits not teh default22:48
inc0thats exactly why you want containers in upgrade22:49
inc0but I don't have to include redeploy of nova_libvirt in upgrade play22:49
inc0and I will not include it there22:49
inc0I will prepare sepratate strategy for this one22:49
SamYapleyou will22:49
SamYapleyou just need to pin it22:49
SamYapleyou don't understand kolla man22:50
SamYapletheres one option for this22:50
SamYaplebut the container will have the nova_libvirt start22:50
inc0upgrade play won't22:50
inc0nova-upgrade play anyway22:50
SamYapleit will22:50
SamYapleits a requirement22:50
inc0there will be libvirt upgrade play22:50
inc0later on, called per node22:51
SamYaplebut again you dont understand is my guess22:51
SamYapleugh you just dont get it22:51
SamYapleif you dont do it in the upgrade play, it will just happen next time you run deploy22:51
SamYapleyou just dont understand man22:51
inc0I know it will22:51
inc0that's precisely why you won't do this22:51
SamYapleholy crap22:51
SamYapleit doesnt need to be this way22:51
SamYaplewe already have the infrastrucute in place to fix the issue you are talking about22:52
SamYaple_you_ need to set one option22:52
inc0if you want to restart all the vms everytime ANYTHING in base image version of anything change...well22:52
SamYaplejeez dude22:52
inc0I'm taling about default22:52
SamYapleno22:52
inc0we have bigger issue then it seems22:52
SamYaplenova_libvirt is upgraded by default22:52
SamYapleyou want to change kolla all by yourself22:52
SamYaplethats the issue22:52
SamYaplewe have options for a reason22:52
inc0you did multiple of times22:53
SamYapleyou can deploy and manage it the way you want22:53
SamYaplenot by myself22:53
SamYaplei got the community behind it first22:53
inc0let's discuss it on midcycle22:53
SamYapledo you know which option to set to do what you want (AND STILL MAKE deploy WORK!)22:53
inc0because don't worry, I won't +2 my own patches, so by definition I need community behind this22:53
inc0yes, but I KNOW KOLLA and 99% people DONT22:54
inc0people run default22:54
inc0and they cry22:54
inc0I don't want to see people cry22:54
SamYapleok if we are on the same page, what is the option im talking about?22:54
inc0you can specify image22:54
SamYaplespecifcally just the tag...22:55
SamYapleto pin the image...22:55
SamYaplethis is why this option exists22:55
SamYapleits for rabbitmq. and mariadb. and all the rest22:55
inc0if you have doubts that I know this kind of basics, why did you vote me for core in the first place?22:55
SamYapleits why i put the option there in the first place22:55
SamYaplebecause you want to change the fundementals of kolla and think 2 core members +2 is the "community behind it"22:56
inc0I think that's why core members have +2 power22:56
SamYaplethat doesnt mean the community is behind it22:56
SamYaplethats my point22:56
inc0how many +2 it takes to "have community behind"?22:56
inc0I'm willing to ask cores to wait with +2 for that many _222:57
inc0+2s22:57
SamYapleyou said ive changed kolla many times yes? the biggest was the ansible change that i brought form yaodu before your time22:57
SamYaplethat required a spec22:57
SamYaplego see the "community behind" that one22:57
SamYaplethats what im talking about22:57
SamYaplethis beviour is in place for a reason22:57
inc0but we did agree for spec-less process22:57
inc0I actually would like to reintroduce specs22:57
SamYapleyou are changing that basic behaviour under the radar man22:58
SamYapleyou need to for this22:58
inc0because forced upgrade we found ourselves in, I hate that22:58
inc0no, I simply add new playbook22:58
inc0deploy will stay EXACTLY the same22:58
inc0and will restart VMs every time base image got rebuilt22:58
SamYaplejust submit a patch so i can -1 you there. you arent going to change this behaviour with jsut two cores approval22:59
inc0which means you really don't want to re-run deploy now when I think about it22:59
SamYaplei jsut said that22:59
SamYapleglad you are catching up22:59
inc0which renders your argument invalid23:00
SamYapleyou are wrong23:00
inc0because I don't care about deploy play23:00
inc0upgrade play MAY not touch nova_libvirt at all23:00
inc0it wont unless I write task for it, which I won't23:00
SamYapleit has to to be complete23:00
SamYapleotherwise when you rerun deploy you change things23:00
SamYaplethats bad23:00
inc0reruning deploy at all will break stuff unless you don't touch your images at all23:01
SamYapleno, you should be able to run upgrade, then deploy and deploy change nothing23:01
SamYaplethats the whole point23:01
SamYapleupgrade is finished23:01
inc0which basically mean that I'll put huge red letter "don't run deploy unless you a. upgraded openstack and b. upgraded infra underneath"23:01
SamYapleor just good documentation on the deploy process (like every other openstack project does with release notes)23:02
*** CheKoLyN has quit IRC23:02
*** dwalsh has quit IRC23:02
inc0admins don't care if upgrade is finished in 3 months if that what it takes, unless cloud stays up all the time23:02
SamYapleif you want to pin nova_libvirt to prevent undesireable behaviour here is how "set one option in globals.yml"23:02
inc0so they can upgrade services23:02
inc0and then qemu one by one23:02
* SamYaple sigh23:03
inc0I'm not pinning nova libvirt, I'm just not upgrading it in same play as nova_compute23:03
SamYaplethen you don't have a competeled upgrade23:03
SamYaplebecause as soon as you run deploy you are screwed23:03
SamYapleand thats what operators will do23:03
inc0I'm out, cya23:03
*** inc0 has quit IRC23:03
*** unicell1 has joined #kolla23:09
*** unicell has quit IRC23:10
*** salv-orlando has quit IRC23:13
*** salv-orlando has joined #kolla23:13
*** tzn has quit IRC23:18
*** alisonh has quit IRC23:26
SamYapleasalkeld: did you say that the heat patch was good? i lost the scrollback23:32
asalkeldSamYaple: i haven't tested23:32
SamYapleok23:33
SamYaplewell just ping me when you say its solid and ill merge23:33
asalkeldi am just busy beating a patch into submission23:33
SamYapleno rush23:33
SamYapleasalkeld: i left a comment on your nova mesos patch. you need the nova database and the nova_api database (thats a thing!)23:34
asalkeldok, thanks23:34
*** thumpba has joined #kolla23:34
*** stvnoyes has quit IRC23:40
*** stvnoyes has joined #kolla23:41
*** tzn has joined #kolla23:44
*** tzn has quit IRC23:45
*** alisonh has joined #kolla23:45
*** sdake has quit IRC23:48
*** blahRus1 has quit IRC23:52
*** jeh has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!