Monday, 2016-02-15

*** jmccarthy has quit IRC00:15
*** jmccarthy has joined #kolla00:16
openstackgerritSam Yaple proposed openstack/kolla: Reduce gate resource usage  https://review.openstack.org/28002900:18
sdakeSamYaple around?00:19
sdakeSamYaple did a charger get left in the office there?00:19
sdakeusb charger type thing00:20
sdake3 plugs iirc00:20
sdakeanker iirc00:20
*** salv-orlando has quit IRC00:21
*** tzn has quit IRC00:25
openstackgerritSam Yaple proposed openstack/kolla: Reduce gate resource usage  https://review.openstack.org/28002900:39
SamYaplesdake: doubtful00:40
SamYaplewe cleaned up the office00:40
*** sdake has quit IRC00:42
*** iNeilus has quit IRC00:42
openstackgerritMerged openstack/kolla-mesos: Zookeeper expects strings as values  https://review.openstack.org/27956600:45
*** XuChao918 has joined #kolla00:49
*** tzn has joined #kolla00:50
*** iceyao has joined #kolla00:51
*** tzn has quit IRC00:57
*** sdake has joined #kolla00:58
asalkeldSamYaple: i am testing that netns:shared and am getting an error from docker, just wondered if you have seen this01:00
asalkelddocker: Error response from daemon: Cannot start container db594243927bb066f788eb19e91b286aa6f28f0d7499c3737ea914c76e1cac43: Path /run/netns is mounted on /run but it is not a shared mount..01:00
asalkeldthis has both /run and /run/netns:shared01:01
SamYapleasalkeld: yes i have seen that01:01
SamYapleyou need to update the docker daemon (systemctl)01:01
asalkeldok, I did a yum update, guess i need to restart it too01:02
SamYaplehold on asalkeld01:02
SamYapleill get the patch01:02
sdakeyum update restarts docker daemon asalkeld01:02
SamYapleasalkeld: https://review.openstack.org/#/c/272715/01:03
asalkeldlooking01:03
asalkeldbtw: http://paste.openstack.org/show/486941/ (marathon file)01:03
SamYapleasalkeld: look at the commit message01:03
asalkeld[Service]01:04
asalkeld    MountFlags=shared01:04
asalkeldthanks!01:04
SamYapleanytime!01:04
SamYapledocs patch tonight for it01:04
*** alisonh has left #kolla01:16
*** alisonh has joined #kolla01:26
*** salv-orlando has joined #kolla01:29
*** salv-orlando has quit IRC01:31
*** sdake_ has joined #kolla01:34
*** neilus has joined #kolla01:36
*** sdake has quit IRC01:36
*** neilus has quit IRC01:40
*** Jeffrey4l has joined #kolla01:47
*** sdake_ has quit IRC01:53
*** achanda has joined #kolla01:58
*** tzn has joined #kolla02:13
*** tzn has quit IRC02:17
*** sdake has joined #kolla02:31
sdakeSamYaple is that mountflags=shared the same problem as the kolla toolbox problem?02:53
SamYaplesdake: different issues03:07
sdakethanks03:10
*** coolsvap|away is now known as coolsvap03:36
openstackgerritMerged openstack/kolla: Make pep8 *the* linting interface  https://review.openstack.org/27938403:47
*** klint has joined #kolla03:48
openstackgerritMerged openstack/kolla: elevates privileges and grants access to /dev/mapper to cinder backup  https://review.openstack.org/27699003:52
*** cfarquhar has quit IRC03:52
*** cfarquhar has joined #kolla03:54
*** cfarquhar has quit IRC03:54
*** cfarquhar has joined #kolla03:54
*** coolsvap is now known as coolsvap|away03:59
*** dave-mccowan has quit IRC04:00
openstackgerritSam Yaple proposed openstack/kolla: Reduce gate resource usage  https://review.openstack.org/28002904:07
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510904:08
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Respect the enable_<service> variables  https://review.openstack.org/28005804:08
*** coolsvap|away is now known as coolsvap04:13
openstackgerritMerged openstack/kolla: Designate ubuntu binary container  https://review.openstack.org/27667604:16
*** obuisson has quit IRC04:17
*** neilus has joined #kolla04:18
*** kbyrne has quit IRC04:19
*** kbyrne has joined #kolla04:19
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Respect the enable_<service> variables  https://review.openstack.org/28005804:22
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510904:22
asalkeldnihilifer: that should be working ^, the mounting of volumes works. I had an issue connceting to rabbit but i think it's my setup issue04:24
*** neilus has quit IRC04:24
*** Marga_ has joined #kolla04:24
*** Marga_ has joined #kolla04:25
asalkeldnihilifer: note that "cleaning containers" part of the cleanup script is getting stuck04:26
*** dims has joined #kolla04:28
*** dims has quit IRC04:30
openstackgerritSam Yaple proposed openstack/kolla: Reduce gate resource usage  https://review.openstack.org/28002904:43
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510904:44
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Move mariadb mount from default file  https://review.openstack.org/28006104:44
*** salv-orlando has joined #kolla04:44
*** salv-orlando has quit IRC04:49
*** sdake has quit IRC04:57
*** daneyon_ has joined #kolla05:01
*** daneyon has quit IRC05:02
nihiliferasalkeld: ok05:02
openstackgerritSam Yaple proposed openstack/kolla: Reduce gate resource usage  https://review.openstack.org/28002905:04
SamYapleman i am out of it tonight05:04
SamYaplelook at the diff history for that patch. i know all of those things :/05:04
openstackgerritMerged openstack/kolla-mesos: Respect the enable_<service> variables  https://review.openstack.org/28005805:06
*** chandankumar has joined #kolla05:19
mdnadeemajafo, SamYaple , nihilifer , did any one notice this error while kolla-ansible deploy "msg: The requested image does not exist: localhost:4000/kollaglue/centos-binary-keystone:2.0.0"05:28
SamYaplemdnadeem: thats a pretty clear message05:28
nihilifermdnadeem: this message means that it cannot pull the image from registry05:28
SamYaplethe image doesnt exist in the registry05:28
nihiliferplease check whether05:28
mdnadeemSamYaple, Yes, i am thinking is this a bug?05:29
nihilifer1) registry works05:29
nihilifer2) the image is present here05:29
SamYaplemdnadeem: no, its user error in this case05:29
nihilifernope, it's not a bug...05:29
mdnadeemeven this image does not exist in kollaglue repo05:29
SamYaplecorrect05:29
SamYapleyou have to build it05:29
mdnadeemSamYaple, ok05:31
*** achanda has quit IRC05:32
mdnadeemSamYaple, I had run kolla-build before may be due to some error it is not build keystone image, However i cant get the log to debug the build error for keystone image05:40
*** achanda has joined #kolla05:41
mdnadeemdid any one know, from where i can get kolla-build log to debug the error for the failed images?05:41
SamYaplemdnadeem: it outputs it when you build05:45
*** tzn has joined #kolla05:45
SamYaplemake sure to pull the latest master05:45
mdnadeemSamYaple, Now i lost the ouput screen, however as i remember in output it shows only image failed , but not give any usefull info to debug it, like why it is failing05:47
SamYapleit gives all of the output that docker returns05:50
*** tzn has quit IRC05:50
mdnadeemSamYaple, ok, i will try to freshly build keystone image alone.05:50
mdnadeemSamYaple, It will be nice if error messages is properly logged in some log file. May be in future we can do this :)05:51
mdnadeemSamYaple, Now i get the actual error which cause keystone image build fail --> Error unpacking rpm package httpd-2.4.6-40.el7.centos.x86_6405:56
mdnadeem which05:56
kejlly_d o05:57
kejlly_sorry, type in wrong place.05:57
*** severion has joined #kolla06:05
*** alisonh has quit IRC06:05
*** v1k0d3n_ has quit IRC06:07
*** mdnadeem_wb has joined #kolla06:08
*** alisonh has joined #kolla06:08
*** mdnadeem has quit IRC06:10
*** mdnadeem has joined #kolla06:11
*** sdake has joined #kolla06:25
openstackgerritMerged openstack/kolla-mesos: Move mariadb mount from default file  https://review.openstack.org/28006106:26
*** salv-orlando has joined #kolla06:30
*** Allen_Gao has quit IRC06:36
*** aginwala has joined #kolla06:38
*** sdake has quit IRC06:39
*** aginwala has quit IRC06:43
*** jasonsb has quit IRC06:43
*** Allen_Gao has joined #kolla06:49
*** achanda has quit IRC06:58
*** Marga_ has quit IRC07:01
*** neilus has joined #kolla07:01
*** dolpher has quit IRC07:03
*** vincent_vdk has quit IRC07:03
*** vincent_vdk has joined #kolla07:03
*** nihilifer has quit IRC07:03
*** Kennan_Vacation has quit IRC07:03
*** dolpher has joined #kolla07:04
*** nihilifer has joined #kolla07:04
*** Kennan_Vacation has joined #kolla07:04
*** neilus has quit IRC07:07
nihiliferasalkeld: i see that new service format doesn't include "volumes" parameter in marathon. am i right?07:11
nihiliferhttps://mesosphere.github.io/marathon/docs/native-docker.html like in the 1st listing07:11
nihiliferif not, i'm going to add it. passing volumes directly as a docker parameter doesn't make them cleaned up by mesos, generally07:12
nihiliferso passing volumes as a docker param should be imo used only as a woekaround for named volumes07:12
asalkeldnihilifer: it does07:26
openstackgerritJeffrey Zhang proposed openstack/kolla: fix the custom profile raise exception issue  https://review.openstack.org/27202207:26
asalkeldjust it doesn't support "shared"07:26
asalkeldso we use parameters section for the shared one07:26
*** chandankumar has quit IRC07:28
*** achanda has joined #kolla07:34
*** tzn has joined #kolla07:34
*** chandankumar has joined #kolla07:35
*** tzn has quit IRC07:39
*** salv-orlando has quit IRC07:47
*** mdnadeem_ has joined #kolla07:51
*** iceyao has quit IRC07:52
*** iceyao has joined #kolla07:52
*** mdnadeem has quit IRC07:55
*** mdnadeem has joined #kolla07:55
*** mdnadeem_ has quit IRC07:57
*** fgimenez has joined #kolla08:01
*** fgimenez has quit IRC08:01
*** fgimenez has joined #kolla08:01
nihiliferasalkeld: ah, ok08:09
ajafoo/08:10
*** iceyao_ has joined #kolla08:11
*** subscope has joined #kolla08:13
*** iceyao has quit IRC08:14
*** mbound has joined #kolla08:20
*** achanda has quit IRC08:26
*** mikelk has joined #kolla08:27
*** achanda has joined #kolla08:28
*** chandankumar has quit IRC08:30
*** Jeffrey4l has quit IRC08:30
*** neilus has joined #kolla08:32
*** achanda has quit IRC08:42
*** Jeffrey4l has joined #kolla08:42
ajafoanyone had error like this 500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/kolla_toolbox/start'    ?08:43
*** Serlex has joined #kolla08:44
*** chandankumar has joined #kolla08:45
*** neilus has quit IRC08:48
*** salv-orlando has joined #kolla08:59
*** salv-orlando has quit IRC08:59
*** salv-orlando has joined #kolla08:59
*** kproskurin has joined #kolla09:04
*** athomas has joined #kolla09:13
*** fgimenez has quit IRC09:13
*** fgimenez has joined #kolla09:14
*** dolpher has quit IRC09:16
*** dolpher has joined #kolla09:16
*** chandankumar has quit IRC09:17
*** chandankumar has joined #kolla09:17
*** athomas has quit IRC09:26
*** athomas has joined #kolla09:26
*** v1k0d3n_ has joined #kolla09:27
*** iNeilus has joined #kolla09:28
*** kproskurin has quit IRC09:28
*** severion has quit IRC09:28
*** elemoine- has quit IRC09:28
*** subscope has quit IRC09:29
*** elemoine has joined #kolla09:30
*** iNeilus has quit IRC09:33
*** subscope has joined #kolla09:37
*** akwasnie has joined #kolla09:51
*** tzn has joined #kolla09:58
*** tzn has quit IRC09:59
*** tzn has joined #kolla09:59
Jeffrey4lajafo, are u using pull action?10:01
*** akwasnie1 has joined #kolla10:01
ajafoJeffrey4l: just sudo kolla-ansible deploy10:03
*** salv-orl_ has joined #kolla10:03
ajafoJeffrey4l: and just reproduce it on second env10:03
*** akwasnie has quit IRC10:04
ajaforeproduced*10:04
*** salv-orlando has quit IRC10:06
Jeffrey4lajafo, what's the docker-py and docker version?10:07
ajafodocker-engine-1.10.1-1.el7.centos.x86_64  and docker-py 1.7.010:07
Jeffrey4lhmm..10:09
Jeffrey4loh.10:09
Jeffrey4lgot it.10:10
Jeffrey4lajafo, it should be this issue. https://review.openstack.org/#/c/279064/10:10
ajafoJeffrey4l: checking10:11
Jeffrey4lajafo, manul fix should: `rm -f /run/kolla/log && docker restart rsyslog && docker rm kolla-toolbox`10:11
Jeffrey4lthen run kolla-ansible deploy again.10:12
*** jmccarthy has quit IRC10:12
*** jmccarthy has joined #kolla10:14
ajafoJeffrey4l: it works :) thanks!10:15
Jeffrey4lajafo, :D10:15
*** gfidente has joined #kolla10:18
*** macsz has joined #kolla10:21
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510910:33
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Make Mesos resources configurable  https://review.openstack.org/28014410:33
nihiliferasalkeld: fyi ^^10:33
nihiliferhardcoded 512 mb ram for everything doesn't fit well for standard vm size10:33
*** alyson_ has joined #kolla10:38
*** akwasnie1 has quit IRC10:40
*** Jeffrey4l has quit IRC10:45
*** subscope has quit IRC11:02
*** Jeffrey4l has joined #kolla11:08
*** akwasnie has joined #kolla11:11
mdnadeem@anyone , deploy fail with Error "response from daemon: Container mariadb is restarting, wait until the container is running"11:19
mdnadeemdocker logs mariadb  shows-->  /usr/bin/mysqld_safe: line 139: /var/lib/mysql//kolla.err: Permission denied11:20
mdnadeem^^ Any idea guys ??11:21
mdnadeemnihilifer, jeffrey4l  ^^11:26
*** subscope has joined #kolla11:30
ajafojust notice similar error half hour ago now trying to recreate it11:31
mdnadeemajafo, :(, i always stuck at some point while deploying kolla, hope fully i get success soon :)11:33
ajafomdnadeem: ubuntu or centos?11:34
mdnadeemajafo, ubuntu, source11:34
mdnadeemajafo, u?11:34
nihilifermdnadeem: you can try to do11:35
nihiliferdocker exec -t <name_of_mariadb_container> /bin/bash11:36
nihiliferand check this command inside11:36
nihiliferstat -c %U:%G /var/lib/mysql11:36
ajafoubuntu but few minutes ago notice some digitalocean mirror conecting problem11:36
ajafonihilifer: it probably doesn't go in my case container was just restarting11:36
mdnadeemnihilifer, ok, i will try11:37
nihilifermaybe "-R" is needed in chown in mariadb's extend_start...11:38
mdnadeemnihilifer, can not start mriadb container , docker exec ----- fails --> Error response from daemon: Container mariadb is restarting, wait until the container is running11:41
mdnadeemnihilifer, time to catch my bus, will debug it more  from home11:43
*** limamauricio has joined #kolla11:43
*** fgimenez has quit IRC12:03
*** fgimenez has joined #kolla12:03
limamauricioHello there, I ran kolla-ansible deploy and I have a new error now in TASK: [keystone | Creating the admin project, user and role]12:05
limamauriciologs ->http://pastebin.com/rd5H0Ktb12:05
limamauricio:/12:06
*** rhallisey has joined #kolla12:06
*** iceyao_ has quit IRC12:08
*** iceyao has joined #kolla12:08
openstackgerritRyan Hallisey proposed openstack/kolla: Upgrade cinder playbook  https://review.openstack.org/27804912:10
*** sdake has joined #kolla12:21
*** openstackgerrit_ has joined #kolla12:27
*** akwasnie has quit IRC12:28
*** coolsvap is now known as coolsvap|away12:36
openstackgerritvenkatamahesh proposed openstack/kolla: Use uppercase 'S' in word "OpenStack"  https://review.openstack.org/27623012:37
*** dims_ has joined #kolla12:42
*** jtriley has joined #kolla12:43
*** akwasnie has joined #kolla12:55
*** salv-orl_ has quit IRC13:04
*** Serlex has quit IRC13:05
*** iNeilus has joined #kolla13:05
*** athomas has quit IRC13:06
*** subscope has quit IRC13:10
*** iNeilus has quit IRC13:10
*** athomas has joined #kolla13:12
*** fgimenez has quit IRC13:12
*** JoseMello has joined #kolla13:13
*** dave-mccowan has joined #kolla13:14
ajafolimamauricio: what kind of build have you got? ubuntu/centos binary/source?13:14
*** fgimenez has joined #kolla13:15
*** fgimenez has quit IRC13:15
*** fgimenez has joined #kolla13:15
*** subscope has joined #kolla13:17
*** shakamunyi has quit IRC13:20
*** jtriley has quit IRC13:21
*** sbezverk has joined #kolla13:22
sbezverkHello, I am trying to deploy AIO on rhel 7.2 and I am getting this prechecks error. fatal: [192.168.80.25] => One or more undefined variables: 'dict object' has no attribute u'ansible_tenants-b0'13:24
sbezverkany suggestions where could be a misconfiguration?13:24
*** dims_ has quit IRC13:33
sdakeSamYaple13:33
sdakere PDE13:33
sdakeyou had asked what it meant13:33
sdakeI looked into it a bit13:33
*** dims has joined #kolla13:33
sdakeone problem with PDE is it skews dashboarding to companies that have multiple contributors versus just one (like you with Servosity or me with Cisco)13:34
sbezverkHere is globals.yml, appreciate if somebody could check. http://paste.openstack.org/show/486981/13:38
*** salv-orlando has joined #kolla13:40
*** _tzn has joined #kolla13:45
*** tzn has quit IRC13:48
*** dwalsh has joined #kolla13:49
limamauricioajafo: centos/source13:51
*** dims has quit IRC13:53
*** dims has joined #kolla13:57
*** klint has quit IRC13:57
sdakemorning cats13:57
sdakelimamauricio still having trouble deploying?13:58
limamauricioyes13:58
limamauricio:/13:58
sdakewe fixed a critical (non-functional) bug in master for centos13:58
limamauricioTASK: [keystone | Creating the admin project, user and role] ?13:58
sdakeif your on  centos/source please git pull13:58
sdakesbezverk same goes13:58
sdakelimamauricio yes that is the one we fixed13:58
limamauricioWow!!!!13:58
limamauricio:)13:59
sdakethe deal is keystone deprecated an option in their middleware13:59
sdakeand changed the default middleware auth in the process of the deprecation13:59
sdakewith a deprecation your not supposed to change default behavior13:59
sdakebut projects do what they want :)13:59
sdakeand we roll with it13:59
sdakestill it is a bit disruptive14:00
sdakelimamauricio just clarifying here, your on master correct?14:00
limamauricioyes14:00
*** fthiagogv has joined #kolla14:00
limamauriciojust a git pull?14:00
sdakesbezverk did you build centos/binary to deploy on RHEL 7.2?14:00
*** dwalsh has quit IRC14:00
*** dims has quit IRC14:00
sdakesbezverk your issue is brand new, never been reported (this prechecks problem)14:01
sdakesbezverk I'm not quite sure where to begin, just woke up about 30 mins ago - let me get my day going and we can debug together if you have time14:01
*** chandankumar has quit IRC14:02
*** dims has joined #kolla14:02
sdakesbezverk  just got through your globals.yml14:02
sdakesbezverk  can you ip link show | fpaste please14:03
sdakesbezverk ansible --version | fpaste14:05
sdakesbezverk pip docker-py show | fpaste14:05
sdakepip show docker-py | fpaste that is14:05
*** dwalsh has joined #kolla14:06
sdakelima git pull --rebase will do the job14:06
sdakelimamauricio  ^^14:06
sbezverksdake I am making a backup of "deployment" vm to have something to roll back as soon as it is completed I will be able to debug it. Just to give an idea of a scenario here. I have a server, which is supposed to become AIO server, on that server I am already running qemu kolla deployment VM.14:06
sdakeso you already running a deployment and want a second deployment?14:07
sdakelet me reread tha tfor a minute - its not clear to me ;)14:08
ajafomorning sdake14:09
*** iceyao has quit IRC14:09
sdakeajafo morning14:10
sdakesbezverk i gave you some fpaste commands above to run - I need to see those please14:10
*** kproskurin_ has joined #kolla14:11
sdakesbezverk so if I understand correctly, you want to deploy openstack AIO using Kolla in a VM, then once its deployed, keep a copy of the qcow2 file around, and restore it immediately after testing openstack services14:13
sbezverksdake requested commands: http://paste.openstack.org/show/486986/14:18
ajafosdake: I found solution on my last problems it was authplugin for ubuntu I saw that sby made patch for it too but it will be discussed because of compatibility with ubuntu/liberty, the same problem we have here https://bugs.launchpad.net/kolla/+bug/1533620 imho if's for liberty to have compatibility could be ok14:18
openstackLaunchpad bug 1533620 in kolla "Nova-compute upgrade level to auto doesn't work on Ubuntu" [Undecided,New] - Assigned to Artur Zarzycki (azarzycki)14:18
sdakeyour prechecks are failing because eht0 is the only network interface yoru system has sbezverk14:18
*** athomas has quit IRC14:19
*** mbound_ has joined #kolla14:20
*** JoseMello_ has joined #kolla14:21
*** Serlex has joined #kolla14:23
sbezverksdake right, kolla runs inside of a vm with FS located on a san storage, so any server can be used to host kolla vm for AIO installation.14:23
*** athomas has joined #kolla14:23
sdakeajafo i updated bug14:24
*** dave-mccowan has quit IRC14:24
*** JoseMello has quit IRC14:24
*** mbound has quit IRC14:24
sdakeajafo does that answer your question?14:24
*** dave-mccowan has joined #kolla14:24
*** JoseMello_ is now known as JoseMello14:25
sdakemorning daneyon_14:25
sdakedave-mccowan14:25
dave-mccowangood morning sdake14:25
sdakewtb implementation of https://blueprints.launchpad.net/kolla/+spec/diag-env14:26
sdakeshould take 5 minutes :)14:26
sdakeplus days of review time :)14:26
*** fgimenez has quit IRC14:27
sdakedave-mccowan can you rebase your patch on master 27970714:27
sdakeso we can see if the gate works correctly with it14:27
*** salv-orlando has quit IRC14:27
sdakedave-mccowan https://review.openstack.org/#/c/279707/14:27
sdakegit git pull --rebase14:28
sdakegit review will do the trick ;)14:28
dave-mccowanyep.  will do.14:28
sdakei'm not sure when our master fixes merged to fix the gate14:28
sdakebut it may ahve been less then 24 hours ago14:28
sdakei think it was about 12 hours ago14:28
*** dims has quit IRC14:28
sdakesbezverk so basically your globals.yml you are specifying a bunch of network interfaces that don't exist in the system14:29
sdakekolla doesn't create network interfaces on its own14:29
sdakeif your using a vm, add a network interface for vm214:30
*** fgimenez has joined #kolla14:30
sdakerather for vm14:30
sdakethen you will have eth0 and eth114:30
sdakeuse one for your mgmt network and one for your provider network14:30
sdakechange provider_b0 or whateveryou have in globals.yml to eth114:30
sdakeand mgmt_b0 to eth014:30
sdakeor alternatively change eth0/eth1 to mgmt_b0, however that is done in ubuntu14:31
openstackgerritAllen Gao proposed openstack/kolla: Remove the role path from the template src  https://review.openstack.org/28026214:31
*** Allen_Gao has quit IRC14:33
ajafosdake: yes, thanks14:33
*** dmsimard has joined #kolla14:33
*** jtriley has joined #kolla14:36
*** dims has joined #kolla14:36
*** macsz has quit IRC14:37
*** akwasnie has quit IRC14:38
*** akwasnie has joined #kolla14:39
sdakeelemoine are you around?14:40
d_codemorning sdake14:41
d_codelooks like over the weekend source builds were fixed?14:41
sdaked_code morning fine sir14:41
d_codethe Keystone issue?14:41
sdakeroger14:41
d_codew00t14:41
d_codebinary?14:41
d_codestill debugging the dhcp?14:41
sdakehang tight14:41
sdaketalkingwith wife14:41
sbezverksdake but these interface do exist on a target physical server, what os the point of kolla checking local VM for these interfaces, I am not deploying AIO in VM, I guess I am missing something here..14:41
d_codekk14:42
*** banix has joined #kolla14:42
*** akwasnie has quit IRC14:42
*** akwasnie has joined #kolla14:44
dave-mccowansdake spinning up kolla is on my to-do list today.  i have a centos7 vm and an ubuntu 14.04 vm.  should either of those work for a one node install?14:45
sdakedave-mccowan sounds good either shoudl work note i am having trouble with centos atm14:48
sdakesbezverk they need to exist in the vm14:48
*** akwasnie has quit IRC14:48
sdaked_code we think there is an issue remaining possiblly with dhcp - not sure, I cna't g et past toolbox deploy on my gear14:48
sdakei intended to debug that today14:48
sdakebut centos is deploying in the gate correctly14:49
openstackgerritDave McCowan proposed openstack/kolla: Use variables to specify http or https when constructing URLs  https://review.openstack.org/27970714:49
d_codecool. I’m just gonna start a fresh build and wanted to see my options14:49
sdakeso if you got as far as dhcp it may deploy for for14:49
d_codeI’ll rock the source14:49
d_code:)14:49
sdaked_code ^^14:49
sdakegit pull --rebase14:49
sdakefor you that is14:49
d_codeyup14:49
sdakeok folks I need to head to airport for 45 mins to drop wife off for ansible fest or whatever she is going to14:49
sdakei'll be back14:49
*** Jeffrey4l has quit IRC14:53
*** iNeilus has joined #kolla14:54
*** chandankumar has joined #kolla14:58
*** iNeilus has quit IRC14:58
*** dims has quit IRC14:59
*** sdake has quit IRC15:02
*** cloudnautique has joined #kolla15:05
*** haplo37 has joined #kolla15:07
*** dims has joined #kolla15:10
sbezverkprecheck docker version check fails. All version looks right, could someobdy check: http://paste.openstack.org/show/487000/15:14
*** cloudnautique has quit IRC15:15
*** cloudnautique has joined #kolla15:22
*** cloudnautique has quit IRC15:22
*** cloudnautique has joined #kolla15:23
*** cloudnautique has quit IRC15:23
*** cloudnautique has joined #kolla15:23
ajafosbezverk: you've probably old docker version 1.8.2?15:26
ajafomaster is compatible with 1.1015:26
sbezverkajafo ok, then please change to doc page, it explicitely ask to downgrade to 1.8.215:28
*** subscope has quit IRC15:30
ajafosbezverk: as I see in table is docker 1.10 (https://github.com/openstack/kolla/blob/master/doc/quickstart.rst) but later I see section with downgrade - it fresh change so probably nobody changed it, if nobody do this earlier I'll make changes today later15:32
*** chandankumar is now known as chandankumar|out15:33
rhalliseyajafo, downgrade is only for stable liberty15:33
*** chandankumar|out is now known as chandan|brb15:33
rhalliseyI can fix the doc to make that more clear15:33
rhalliseymaster use 1.9 or higher15:33
*** alyson_ has quit IRC15:34
*** blahRus has joined #kolla15:34
*** alyson_ has joined #kolla15:37
sbezverkrhallisey with 1.10.1 docker, precheck has completed sucessfully. Now I need to deploy AIO on physical server,using kolla VM. For this scenario, would it be sufficient just to add physcial server ip to ansible inventory file?15:38
rhalliseyya I think for AIO that should be fine15:39
sbezverk<rhallisey> it is strange, it tries to install AIO on a local host which is my kolla VM, instead of deploying it on a physical server. What am I missing here?15:41
*** subscope has joined #kolla15:41
dmsimardSamYaple: You might have to do this earlier than you think: https://github.com/openstack/kolla/blob/d346545922fa3dfda742a9acc6ca6e9f140786ab/docker/keystone/Dockerfile.j2#L6615:47
*** thomasem has quit IRC15:50
*** thomasem has joined #kolla15:52
*** daneyon_ has quit IRC15:56
*** daneyon has joined #kolla15:56
rhalliseysbezverk, maybe I'm confused with what your doing then15:59
*** daneyon has quit IRC15:59
*** daneyon has joined #kolla16:00
*** jmccarthy has quit IRC16:03
SamYapledmsimard: looking16:04
SamYapledmsimard: im doing it right now16:04
*** daneyon_ has joined #kolla16:04
SamYapledmsimard: that was just a temp patch to get the gate and dev rolling again16:04
dmsimardSamYaple: I pulled the trigger a bit too quickly -- but keystone actually removed admin_auth_token from the default pipeline and broke a buncha things16:04
SamYapleyea dmsimard that sed lines _adds_ it back to the efault pipeline16:05
dmsimardyeah16:05
SamYapleits just deprecated not fully removed16:05
*** jmccarthy has joined #kolla16:05
dmsimardSamYaple: For RDO we'll do that right inside the packaging for this cycle16:07
dmsimardWe submitted a partial revert to add it back into the default pipeline but I'm not sure it will go through at this point16:07
*** daneyon has quit IRC16:07
*** mbound_ has quit IRC16:08
*** stvnoyes has quit IRC16:09
*** stvnoyes has joined #kolla16:09
sbezverk<rhallisey> I have VM with kolla, I want to initiate AIO deployment from kolla VM onto a physical server.16:09
SamYapledmsimard: honestly? Its deprecated. That means not-default in my opinon16:10
SamYapledmsimard: im very happy with the change, but i understand your concern16:10
*** jtriley has quit IRC16:11
dmsimardSamYaple: it's just not a proper deprecation if it's removed in our opinion16:11
dmsimardSamYaple: "fix your stuff in packaging/installer" is the usual answer we get but this one didn't roll very good with us16:11
dmsimardlike, okay, we'll create a nova api database now because you've been telling us since kilo16:12
openstackgerritMauricio Lima proposed openstack/kolla: Add manila perms to /var/cache/manila  https://review.openstack.org/28030616:12
*** sdake has joined #kolla16:12
dmsimardbut devstack switched to bootstrap back in december ..16:12
*** jtriley has joined #kolla16:14
SamYapledmsimard: fair enough, but a day of breakage in the gate isn't worth a revert in my opinion16:14
dmsimardSamYaple: it broke everything leveraging puppet modules16:15
dmsimardusers and installers16:15
SamYaplei understand what yo uare saying dmsimard16:15
SamYapleit broke kolla too16:15
SamYaplewe fixed it and moved on. it is master after all16:15
dmsimardWe pick our battles :)16:15
SamYaplei get it. i think im on the other side of the fence on this battle though16:16
SamYaplerhallisey: master is 1.1016:16
dmsimardYeah, *shrugs* it looks like there is disagreement even between the keystone cores on that change16:17
dmsimardThe topic has been going for almost an hour now.16:17
rhalliseySamYaple, oh right because of the neutron thin containers16:17
SamYaplerhallisey: indeed16:17
rhalliseyforgot about that16:17
SamYapledmsimard: ive logged in. i like reading the drama!16:18
dmsimardSamYaple: you've missed a lot, look at the eavesdrop lol16:18
dmsimardseriously though, distros are getting fed up with upstream shovelling issues in our yard16:19
SamYapledmsimard: eh its a two edged sword16:21
SamYaplenew features new changes16:21
SamYapleit happens16:21
SamYaplethis keystone-manage bootstrap thing is something I actually asked for a year ago16:22
SamYapledmsimard: check these logs16:23
SamYaple##ansible-shade.2015-07-07.log:03:16 < SamYaple> mordred: I was speaking about the SERVICE_TOKEN (its called admin_token in the conf file). The one that lets you setup the initial endpoints and users. Ill look into it, but its no rush on that now. Thanks for the info16:23
SamYaple##ansible-shade.2015-07-07.log:03:49 < SamYaple> mordred: from a security standpoint, I went a different direction. I used the token to bootstrap just once from withinside the container. I dont wnt ansible spitting out that service token16:23
SamYaple##ansible-shade.2015-07-07.log:03:58 < mordred> SamYaple: yah. actaully, morgan was talking about making keystone-manage able to handle this without needing the admin token middleware at all16:23
openstackgerritMerged openstack/kolla: Use uppercase 'S' in word "OpenStack"  https://review.openstack.org/27623016:23
rhalliseysbezverk, so wait your using kolla to create kolla on a physical server?16:24
rhalliseyfrom a vm16:24
*** _tzn has quit IRC16:27
*** guillaume_ has joined #kolla16:27
d_codeI’m still getting the keystone errors on centos source build16:28
sbezverkrhallisey yes, basically at the end I want to have physical server running bunch of docker containers and offerring openstack services16:28
d_codei did a “git pull —rebase” this morning16:28
guillaume_hello all, I got an error about when starting neutron_dhcp_agent : Path /run/netns is mounted on /run but it is not a shared mount.16:28
rhalliseysbezverk, what I don't understand though is what the kolla vm has to do with getting to that16:28
d_codeon “Creating the admin project, user and role”16:28
SamYapleguillaume_: moment16:30
*** chandan|brb is now known as chandankumar16:30
SamYapleguillaume_: i have a docs patch started around that issue16:30
guillaume_thx16:30
SamYapleguillaume_: for now, read this commit message https://review.openstack.org/#/c/272715/16:31
*** mbound has joined #kolla16:31
*** jasonsb has joined #kolla16:33
sbezverk<rhallisey> Kolla VM will be re-used for other deployments. Example: I need to bring up openstack AIO, do something, then clean it and bring another AIO. I was hoping to Kolla VM as a deployment engine. So when I need to bring up new physical server, I just install OS, install docker container engine and the initiate deployment from Kolla VM, make sense?16:33
rhalliseysbezverk, so your talking about using that vm like an undercloud16:34
*** tzn has joined #kolla16:35
*** tzn has quit IRC16:35
SamYaplesbezverk: this is how I use kolla16:36
sbezverkSamYaple Glad to hear that it make sense to other folks.16:38
guillaume_SamYaple : I dont understand where I need to put the [service] thing16:38
guillaume_in /etc/init/docker.conf?16:38
*** fgimenez has quit IRC16:39
rhalliseysbezverk, cool16:39
SamYapleguillaume_: /usr/lib/systemd/system/docker.service16:39
SamYaplesudo sed -i 's|^MountFlags=.*|MountFlags=shared|' /usr/lib/systemd/system/docker.service16:39
SamYaple^ that will do it for you guillaume_16:39
guillaume_thx a lot16:39
SamYapleyoull need to reload systemctl and restart the service16:40
guillaume_in ubuntu it is : /lib/systemd/system/docker.service16:41
sbezverkSamYapple could you share how you initiate deployment from VM to a physical host. When I try, it always try to deploy on a local host.16:41
d_codeSamYaple: I’d recommed using /etc/systemd/system/docker.service instead. When you reinstall the RPM/DEB there’s no guarantees that your changes will be preserved16:41
d_codethe /etc/ dir masks the /lib dir…  this is its purpose16:41
*** subscope has quit IRC16:42
*** iNeilus has joined #kolla16:42
SamYapled_code: i cant speak for centos, but on ubuntu that file does not exist16:43
SamYapleand the multi-user target is a symlink16:43
*** fgimenez has joined #kolla16:43
d_codehmm…  weird16:43
d_codeon CentOS…that’s the way to do it…it’s in the freedesktop spec…there should be a local directory that is intended for administrator modification of the service/socket/whatever files16:44
SamYapled_code: may be bad packaging... what you suggest works on ubuntu too (its a systemd thing) but its not setup by default16:44
d_codeah…makes sense16:44
*** mbound has quit IRC16:45
guillaume_SamYaple : in ubuntu it is : /lib/systemd/system/docker.service16:45
*** jtriley has quit IRC16:46
*** jtriley has joined #kolla16:46
*** subscope has joined #kolla16:46
SamYapleguillaume_: it is in centos too, what d_code was saying is /etc/systemd/system/docker.service survives upgrades (and he is right)16:46
SamYaplebut you would have to add that file yourself on ubuntu16:47
guillaume_ok16:47
*** iNeilus has quit IRC16:47
guillaume_so I copy the file in /lib to this file?16:47
openstackgerritSam Yaple proposed openstack/kolla: Reduce gate resource usage  https://review.openstack.org/28002916:50
SamYapleguillaume_: you can do that. or once youve made the appropriate change, you can just leave it alone16:51
SamYapleremember, that service file can change on upgrades16:51
SamYapleyou don't want a stale service file around16:51
guillaume_SamYaple, I now have this : docker start kolla_toolbox16:51
guillaume_Error response from daemon: Cannot start container 6383443305d2335305924e1b708fcc45d6a0aff3b9b8d8e916bb595ac9830b58: [9] System error: not a directory16:51
guillaume_Error: failed to start containers: kolla_toolbox16:51
SamYapleguillaume_: _that_ is an entirely different issue.16:51
SamYaplewe know about this, it just started a few days ago. we have no confirmed causes yet16:52
guillaume_ok  strange16:52
guillaume_For me it appears to be there since the change16:52
SamYaplewhat I would suggest is to stop the docker service and then restart your machine16:52
guillaume_of the file16:52
guillaume_before I was able to go further than that16:52
*** ByPasS has quit IRC16:52
SamYapleyea we are still tracking down the cause16:52
guillaume_ok16:52
*** ByPasS has joined #kolla16:54
sdakemorning16:55
sdakeback16:55
sdakereading scrollback16:55
*** mikelk has quit IRC16:57
sdaked_code you need to rebuild your containers16:57
d_codesdake: I did cleanup-images16:57
d_codeI’ll give it another go16:57
sdakeSamYaple the docs shoudl prboably just have us overrride /etc16:59
sdakeguilaume_ that problem is known and unsolved, we aree working on it16:59
sdaked_code sorry was reading scrollback16:59
sdakeok justa minute17:00
sdakelet me think plz :)17:00
sdaked_code I have a suspicion there is some kind of container leak (atleast on centos) in the build system17:01
*** Marga_ has joined #kolla17:01
sdakecan you run docker ps -a | fpaste17:01
sdaked_code (what would solve this short term is to use the nuclear option on docker cleanup :)17:01
sdaked_code but if you could put out the odcker ps -a output that would be helpful so we can confirm it17:02
SamYaplesdake: overriding /etc is the issue17:02
SamYapleubuntu doesnt package /etc17:02
SamYaplebut ill post some docs stuff and well get it figured out in review17:03
sdakesounds good17:03
*** dims_ has joined #kolla17:03
sdakeif you could get somethign in queue soon that would be helpful to point people at :)17:03
sdake(re docs change)17:03
SamYaplesdake: yea thatll be tonight17:03
sdakesbezverk are you sorted?17:03
sbezverksdake nope17:04
SamYaplewas going to be last night, but then it didnt happen sinec it was sunday17:04
sdakeright17:04
SamYaplesdake: btw, i had that mount issue on ubuntu17:04
SamYaplei rebooted and it worked17:04
sdakeinteresting17:04
*** dims has quit IRC17:04
SamYaplei think it may be a problem with changing the shared mount flag thing without a hard reboot17:04
sdakei have a debug log  around here on one of my vms of docker running17:04
sdakei'll try that SamYaple17:05
sdakeand see if i can get a  deploy in my vm17:05
SamYapleif that _is_ the problem, i bet we can figure out a fix without a reboot17:05
sdakei can see why a reboot there would work using your speculation ;)17:06
*** salv-orlando has joined #kolla17:06
*** Marga_ has quit IRC17:07
d_codesdake: I blew it up…I’ll rebuild the source and see where I get17:09
SamYaplegoing to lunch17:10
SamYaplebbiab17:10
*** salv-orlando has quit IRC17:11
sdakerestarting docker with mountflags shared doesn't fix toolbox problem trying reboot now17:12
sdaked_code any chance you can get your centos patch in the queue?17:12
sdakethe host setup script17:12
d_codeyeah, no problem. I was gonna wait until it worked :)17:13
sdake;-)17:13
d_codecan I just fork on github and do a pull request?17:13
sdakenope17:13
d_codeor is there something different17:13
d_codelol17:13
sdakewe use gerrit17:13
d_codek17:13
sdakedoc links coming17:13
d_codek17:13
sdakegroan can't find docs17:15
sdakejust a moment need to use special google skillset ;)17:15
sdakehttp://docs.openstack.org/infra/manual/developers.html17:15
sdaked_code ^^17:17
sdakewhen your ready17:17
sdakeits pretty easy to setup - takes about 15 mins, if you need help ping here or openstack-infra17:17
sdakeonce your setup your setup forever17:17
d_codecool17:17
sdakewell until the robots take over17:18
*** thomasem has quit IRC17:18
sdakeSamYaple a reboot with sahred flag does not fix the kolla toolbox issue17:20
*** kproskurin_ has quit IRC17:20
sbezverksdake it is still not clear how to initiate a deployment from Kolla VM onto a physical server. SamYaple mentioned that he is using this approach..17:20
sdakesbezverk you deploy either on either a bare metal server or onto a vm17:20
sdakeyou could deploy from a vm to a bare metal server if you want17:20
sdakefor that you will need a registry17:20
SamYaplesbezverk: oh perhpas i misunderstood your issue. can you elaborate?17:21
SamYapleI am fairly certain you can do what you want... but just to be clear17:21
sdakesbezverk here is an easy question to answer, do you want openstack running on the bare metal, or on the vm?17:21
* sdake thinks we need a non-vagrant non-heat use-a-vm guide ;)17:22
sbezverkOk, I have VM hosted on my lab network which is running kolla and registery. I want to be able to deploy OpenStack (in a form of docker containers) on a physical server running ONLY OS and Docker engine.17:22
sdakeor make vagrant more usable alteneratively17:23
sdakesbezverk roger got it17:23
d_codesdake: http://ur1.ca/oiv9317:23
*** thomasem has joined #kolla17:23
d_codek. just finished source build17:23
SamYaplesbezverk: yea just setup your inventory with the physical hosts (not your kolla vm) and you should be good17:23
*** fgimenez has quit IRC17:23
SamYaplesbezverk: now this depends on your networking, obviously your vm and phsycial hosts need to talk to each other17:23
SamYaplebut its that straight forward17:23
sdaked_code good to see nothign leaked there - that was with the ps -a flag right?17:24
d_codeyes17:24
sdakecool mabye it was during deploy it leaks17:24
sbezverkSamYaple connectivity is not an issue, when I initiate deployment Kolla tries to deploy on a local host, even though it does not exist in ansible inventory file17:24
sdakeI only saw it the one time i deployed on sunday (a non work dya for me :)17:25
d_codeso, gonna deploy17:25
d_code`time kolla-ansible deploy -vvv`17:25
SamYaplesbezverk: how are you specifying your inventory? (exact kolla-ansible command)17:25
SamYapled_code: that command for me deploys in under 2 minutes on multinode ;)17:25
SamYaplesdake: i got the gate down to an 8 minute deploy with ubuntu (though it averages 12m)17:26
sdakesbezverk there is an inventory file in ansible/inventory/all_in_one17:26
sdakeSamYaple how17:26
sdakesbezverk it contains "localhost"17:26
SamYaplesdake: magic17:26
sdakeyou need to modify it or provide your own of the same format with the hosts specified17:26
SamYaplesdake: mirrors and dropping some non-used build containers and services in the deploy gate (ceph horizon heat)17:27
d_codefailing on admin project again, looks like17:27
sdakesbezverk kolla-ansible takes an -i flag17:27
sdakeya mirrors probaby make a big difference17:27
sdakeI did notice the huge delta between ubuntu gating speed and centos gating speed17:27
SamYaplemirrors got me to 12, the other changes took me to 817:27
d_codeThe error was: TypeError: Value of unknown type: <class 'keystoneauth1.exceptions.http.Unauthorized'>, The request you have made requires authentication. (HTTP 401) (Request-ID: req-03152288-1ff9-4faa-99eb-73de33c97fd4)"17:27
SamYaplesdake: no rpm mirrors yet. im kkeeping tabs on the issue in the infra17:28
d_code2m19s17:28
SamYapled_code: grab the latest master code and rebuild your images17:28
SamYaplethis was fixed and merged last night17:28
sdake8ef7434770846e81aa04009838bd1df69cc29ac817:28
sdaked_code do you ahve that commit in your repo?17:29
sdakegit pull --rebase should have pulled it in17:29
d_codejust updated to ‘84f4314’17:29
sdakehrm17:29
SamYaplesdake: even if its in the repo doesnt mean his kolla image for keystone has it (old build)17:29
d_codek17:30
d_codegonna nuke this whole box17:30
d_codewell.17:30
sdakeSamYaple he wiped out /var/lib/kolala 20 seconds ago and rebuilt17:30
d_codedocker17:30
sdaked_code hang tight17:30
SamYaple*boom*17:30
sdakerather 20 mins ago17:30
SamYapleah17:30
sdakerather /var/lib/docker17:30
SamYapleprobably wants head of master then17:30
SamYaplewasnt*17:30
sdakethe commit he is on is head of master17:30
sdake84f314 is that OpenStack change thing17:31
SamYaplesdake: 17:29:53 < d_code> just updated to17:31
sdakeI don't know if it acutally works baremetal because I can't get past toolbox17:31
SamYaplehe said he just updated17:31
sdakeby just he menas about an hour ago17:31
SamYaplehe said it 2 minutes ago17:31
sdakehe did it an hour ago17:31
SamYaplealrighty17:31
sdaked_code ?17:31
SamYapleits fine17:31
*** aginwala has joined #kolla17:32
*** subscope has quit IRC17:33
SamYaplesdake: https://jenkins02.openstack.org/job/gate-kolla-dsvm-deploy-centos-binary/284/console17:34
SamYaplesdake: that fails on kolla_toolbox17:34
SamYapleit happens in the gate... SOMETIMES17:34
sdakeinteresting17:34
d_codesdake: I updated when you told me to do the rebase17:35
d_codethen rebuilt17:35
d_codethen I updated again…so there were new changes that came in17:35
sdakei see17:35
*** tzn has joined #kolla17:35
sdakewell when i told you to do the rebase the centos changes wer ein17:35
sdakebut your on tip of master atm17:35
*** salv-orlando has joined #kolla17:36
sdakeSamYaple well at this point I'd go harass the docker devs for advice if  Icould get into the damn channel17:36
sdakebut -ETOOMANYPASSWORDS17:36
SamYaplesdake: already on it. no worries17:37
SamYapleive been talking with lk4kd(?) about it for a few days17:37
sdakeya17:37
*** cloudnautique has quit IRC17:37
sdakehe rocks17:37
SamYaplehe walked me through the initial issues for rsyslog to pump out that fix17:37
*** jtriley has quit IRC17:37
SamYapleindeed17:37
d_codesooo…..  nuke it?17:37
sdakeyou mean the shared fix?17:37
SamYapleim almost positive i know the breaking commit, but why does it work _sometimes_17:37
sdakelet me know the ommit, i'll revert in my vm and let you know for sure ;-)17:38
sdakesometimes = race condition typically17:38
sdakethreads ftw!17:38
sdakeSamYaple if you can point out the commit that you think causes the problem, I can assist in debug17:39
SamYapleno its the same one from rsyslog issue17:39
SamYaplebut i think its modifying /dev perms17:40
sdakedocker has special magic code to mount /dev17:40
*** jtriley has joined #kolla17:40
sdakeasked for it in 1.6.0 time17:40
sdakei guess we got it ;-)17:40
sdakeso which commit SamYaple so I can test if it is indeed the problem17:41
sdakei dont know which rsyslog commit was the problem17:41
sdakeif we can narrow it down to a specific commit - it narrows the search area for the problem17:41
sdakesounds like you think its the bindmounting of /dev17:41
sdakeyup not bindmounting dev fixes the problem17:43
sdakeSamYaple  ^^17:43
SamYaplesdake: thats not it at all17:44
sdakeI removed /dev:/dev from bindmount of kolla-toolbox and we are cooking now17:44
SamYaplei mean the special magic code17:44
SamYapleand you can also make it work without binding in /run17:44
sdakei'll try that17:44
SamYaplebut its a comlicated issue im almost possitive is related to permissions in /dev17:44
*** openstackgerrit has quit IRC17:45
*** blahRus1 has joined #kolla17:46
*** openstackgerrit has joined #kolla17:46
*** neophy has joined #kolla17:46
sdakeSamYaple removing /run:/run has zero effect - internal server error 50017:46
SamYaplesdake: http://paste.openstack.org/show/487028/17:46
SamYaplelike i said, its _probably_ related to that17:47
SamYapleand its _probably_ a permissions issue17:47
SamYaplebut lots of people are out in docker-dev today17:47
sdakegit SamYaple ya i pasted that a couple days ago17:47
*** aginwala has quit IRC17:47
sdakegit/ya17:47
sdakethe sandbox thing is where it begins17:47
*** blahRus has quit IRC17:48
sdakerhallisey can you go bug dan walsh to join channel plz17:48
sdakehe implemented the dev:dev bindmounting17:48
rhalliseyya I'll see if he's around17:48
sdakethx17:49
d_codeSamYaple: the MountFlags=shared addresses the dhcp issue?17:50
SamYapleyea17:50
d_codek…that’s the hurldle I was running into on centos binary…I’ll retry with that17:52
d_codeexcept I just nuked docker, so I’ll rebuild :)17:52
sdakeok it has nothign to do with run17:52
sdakebindmounting /dev:/dev followed by bindmounting /etc/:/dev/log17:53
sdaketriggers problem17:53
sdakeits a double bindmount on the dev fs17:53
sdake100% reproducible on my gear17:53
SamYaplesdake: yea thats what me and inc0 were looking at last night17:53
SamYaplebut the question is _why_ it only breaks sometimes17:54
sdakewell you said it was run, it has nothing to do with run ;)17:54
SamYaplemaybe all the time out your gear, but its a race looking at the gate failures17:54
sdakeyup roger on the race17:54
sdakedefinately looks like a race condition17:54
sdakebut could also be environmental17:54
SamYaple17:45:00 < SamYaple> and you can also make it work without binding in /run17:54
sdakewhat are gate permissions on /dev/log - any idea?17:54
SamYapleoops */dev17:55
SamYaplelook at context, i meant /dev sorry17:55
sdakegot it17:55
sdakeok well i'm up to speed now17:55
SamYapleso we have another option17:55
SamYaplei didnt want to mad rush into it17:55
SamYaplebut we dont _have_ to bind to /dev/log17:55
SamYaplewe can configure most (all?) service to log to any other socket17:55
SamYaplewanted to flesh out the cause first though17:56
sdakewhen dan comes around we can get some free consulting on why double dev bindmounts dont work ;-)17:56
SamYaplewont work sometimes17:56
sdakewhich version of docker are you running SamYaple ?17:57
SamYaple1.1017:57
sdakei've got 1.10.117:57
SamYaple1.10.117:57
SamYaple*17:57
sdakewell its clearly a docker problem17:57
sdakeeither docker doesn't support that model of bindmounting (not intended in the design) or its broken in its implementation17:58
*** blakec has joined #kolla17:58
sdakehow long has this problem been around?17:58
sdakejust on docker 1.10?17:58
*** cloudnautique has joined #kolla17:59
*** cloudnautique has quit IRC17:59
*** cloudnautique has joined #kolla17:59
SamYapleagain, its a race17:59
*** cloudnautique has quit IRC17:59
SamYaplelikely the race is trying to bind to /dev/log before /dev is bound18:00
SamYapleand I would be willing to bet I can fix that18:00
SamYapleinfact im almost positive I can now18:00
*** cloudnautique has joined #kolla18:00
SamYaplepatch incoming18:00
SamYapleits not a race18:00
SamYapleits an unsorted dict!18:00
sdakeoh well that would explain it18:00
SamYaplesometimes /run/kolla/log:/dev/log is before /dev/:/dev/18:00
SamYaplei bet you thats it actually18:01
SamYapleerror message makes sense too18:01
sdakepaste patch I'll test in my 100% reprodubiel environemnt :)18:01
sdakedocker shouldn't implode as it does - it leaks a container in this scneario18:03
sdakebut thats a minor problem if its just a ordering issue on the mounting operation18:03
SamYaplesdake: check that patch18:04
openstackgerritSam Yaple proposed openstack/kolla: Fix sandbox issues on some hosts  https://review.openstack.org/28034718:04
SamYaplesdake: that _may_ not fix it18:05
SamYaplesince im reading and ansible dict18:05
SamYapleactually18:05
SamYapleno its a list18:05
SamYapleanyway i think this is the right path18:06
sdakeso i think sort should be after the :18:07
*** Marga_ has joined #kolla18:07
sdakerather the nthe full string18:07
sdakedoes this patch do that?18:08
SamYapleim not sure what you are asking. this keeps the dict ordered18:10
sdake doesn't work18:10
SamYapleso when docker-py loops over it it sets it up in the right order18:10
*** tzn has quit IRC18:10
SamYapleok sdake, lets go another direction here. do it break with _only_ dev?18:10
sdakegit right18:10
sdakeright18:10
SamYaplewhat combination breaks it?18:10
sdakei dont know let me try that line of thinking18:10
sdakels18:11
*** achanda has joined #kolla18:11
SamYapleif its /dev and /run/olla/log:/dev/log then think its still an ordering issue18:11
sdakeright18:11
sdakejust needs to be ssorted after the : ;)18:11
SamYaplesorted is the wrong approach18:11
SamYapleordered is the right18:12
sdakeright18:12
SamYapleordered keeps them in order as they get added, sorted rearrages18:12
SamYapleim heading back to office will check back in in a minute18:12
sdakechagning order of dev moutns in ansible file has no effect18:13
sdakepossible ansible sorts it18:13
*** dwalsh has quit IRC18:14
SamYaplesdake: no ansible has a list18:19
SamYaplelists arent sorted18:19
SamYapledocker-py gets a dict18:20
SamYapledicts are based on hashes18:20
SamYapleits possible its chaning there18:20
sdakeyup18:21
sdakecan you try sorting from the : onwards18:21
sdakeand i'll tes tthat out18:21
sdakeso rax just announced RHEL OSP as a service18:22
sdake99.99% uptime18:22
sdakelol18:22
sdake4 9's availability unlikely imo ;-)18:22
sdakeSamYaple i've got a 30 min meeting but i'll be back, if you could work up a patch with a sort after the first : if dev is matched in the string, I'll try that18:23
sdakeif not, i'll write it myself and try it when I get back ;)18:23
sdakekind of hacky but dev has special magic for moutning and clearly we are doing something wrong ;-)18:24
sdaketoo bad danwalsh isn't around18:25
sdakehe could explain better the dev bindmounting process - its different then other filesystems18:25
sdakei just dont have the details to offer up the differences18:25
sdakebut i am certain its different because i looked at the patch the docker cats and  rht put together for it in libcontainer18:26
sdakeand said18:26
sdake"go"18:26
sdake"dont have time for that" :)18:26
*** jtriley has quit IRC18:26
*** dwalsh has joined #kolla18:31
d_codeguess who’s got two thumbs and the latest build of centos binary kolla running on his box?18:32
d_codethis guy18:32
sdakeit ran?18:34
sdakewish it worked for me ;-)18:35
sdakewhat did you do differently ;)18:35
sdakeyou didn't get stuck on the kolla_toolbox thing?18:35
sdaked_code this is a normal state of kolla (master always runs) its this recent change to 1.10 which has been a little disurptive but necessary ;)18:36
d_code:)18:36
sdaked_code the main problem is our gates are not voting18:36
sdakethe reason we cna't make our gates voting is because the upstream repos go down and then our gate gets blocked for days on end18:36
d_codeI added the MountFlags=shared to the /etc/systemd/system/docker.service file18:36
sdakewe are working on solving that18:36
d_codecan you setup like a Pulp instance that mirrors upstream?18:37
d_codewould make your builds much faster too18:37
d_codesdake: now that it’s working…  I’ll try to replicate it using my instructions and work on submitting a patch18:39
d_codefor the build script18:39
sbezverkSamYaple I got my VM back on track. For the deployment we talked earlier, in globals.yml, do I specify values for my remote physical server or for VM?18:40
sdakesbezverk the internal_address should be a free VIP on your network18:41
sdakesbezverk the network interface names should be the names on your physical vm18:41
sbezverksdake but there is no physical vm, there is physical server OR VM, what is it then?18:43
d_codesdake: so…neutron_external_interface… by default that is NATing public traffic out?18:43
sdakereplace physical vm in above line with physical bar emetal18:43
*** Marga_ has quit IRC18:43
sdakehowever neutron works, I really dont know ;)18:44
sdakeits the "br-ex" in devstack18:44
sdakeiirc18:44
d_codethe network for that interface has a gateway of 192.168.1.1/2418:44
d_codeyes…that’s correct18:44
sdakeinit-runonce18:44
sdakeis what I use to setup my network18:44
d_codeah18:44
d_codeI’ll poke into that18:44
sdakethats tailored to my environment18:44
sdakeyou will have to tailor to your personal network18:44
sdakebut its pretty close18:44
rhalliseysdake, that /dev issue is solved?18:45
rhalliseythought I read you guys figured it out18:45
*** Marga_ has joined #kolla18:49
*** Marga_ has quit IRC18:49
d_codesdake: cool. just had to create the right floating IPs and it worked18:51
d_codeit…lunch then back to tinkering18:51
sdakerhallisey unsolved thus far18:52
sdakewe know that moutnign /dev and /dev/log at same time result in docker cratering18:52
sdakewe are not sure if ordering of the moutning matters18:52
sbezverksdake I adjusted globals.yml. now /usr/share/kolla/ansible/inventory/all-in-one, there is line localhost       ansible_connection=local18:52
*** Marga_ has joined #kolla18:52
sbezverksdake do I change it for my server ip address and connection ssh?18:52
*** achanda has quit IRC18:54
*** subscope has joined #kolla18:55
openstackgerritimain proposed openstack/kolla: A few silly grammar fixes.  https://review.openstack.org/28036318:58
*** banix has quit IRC18:59
*** vhosakot has joined #kolla19:00
sbezverksdake Started deployment and it fails: I pasted logs from the server and from VM here: http://paste.openstack.org/show/487032/ Please check.19:03
limamauricioTASK: [neutron | Starting neutron-dhcp-agent container] *********************** failed: [localhost] => {"changed": true, "failed": true} msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)19:04
*** tzn has joined #kolla19:07
sbezverknever mind I figured out.. deployment is still going..19:08
dave-mccowanhello SamYaple19:12
Serlexsbezverk I had a similar error but durin vagrant multi-mode deployment19:14
Serlexlimamauricio - is that a centos AIO deployment?19:15
limamauricioyes19:15
sbezverkSerlex that one I figured out, my baremetal server did not have insecure access to registry19:15
SerlexIf so,  myself and few others here have come across same problem19:15
SerlexIs it resolved? I'm not sure, sdake knows best19:15
sbezverkSerlex but now the deployment is failing because mariadb does not start on the server19:15
Serlexoh yeah, I remember now sbezverk, you are correct! It took me few hours to figure it out19:16
sbezverkfailed: [192.168.80.25] => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "ls", "/var/lib/mysql/mariadb.pid"], "delta": "0:00:00.051707", "end": "2016-02-15 14:11:32.886853", "failed": true, "rc": 1, "start": "2016-02-15 14:11:32.835146", "stdout_lines": [], "warnings": []}19:16
sbezverkstderr: Error response from daemon: Container mariadb is restarting, wait until the container is running19:16
Serlexwhats the status in docker ps -a19:16
*** fthiagogv_ has joined #kolla19:16
Serlexlimamauricio - wait for sdake to confirm a work around. Did you build using binary or source?19:17
sbezverkSerlex http://paste.openstack.org/show/487035/19:18
Serlexhmm I'm not an expert on docker or kolla I'm afraid19:19
sdakewas in meeting19:19
sdakewhats up19:19
*** fthiagogv has quit IRC19:19
*** limamauricio has quit IRC19:19
Serlexhi19:19
sdakelet me read scrollback19:19
sbezverksdake mariadb container is flapping like a toilet seat19:19
sdakeso d_code was able to deploy with the shared workaround19:20
Serlexlimamauricio - is having the same ' Starting neutron-dhcp-agent container'19:20
sdakesbezverk haven't seen that before19:20
sdakeSerlex ya thats fixed19:20
Serlexahh awesome19:20
Serlexgreat, i can get back to playing around with kolla19:20
Serlexhow do I apply the fix?19:20
sdakepull master19:20
SamYapledave-mccowan: hey19:20
sdakeblow away /var/lib/docker19:20
sdakestop docker19:21
sdakeblow away /var/lib/docker19:21
sdakestart docker19:21
sdakeand start a fresh would be my recommendation ;)19:21
Serlexahhhhh19:21
d_codeSerlex: I had to  edit the openstack.service file too19:21
Serlex:'(19:21
sdakeif you need to keep /var/lib/docker19:21
sdaked_code which one?19:21
sdakethen just delete all containers shown in docker ps -a19:21
sdakedocker rm -f ab19:21
dave-mccowanSamYaple I have some time over the next couple weeks to help out with TLS endpoints.  do you have a design in mind, or a proposal on how to collaborate?19:21
d_codesdake: the change seems to have been the mount options19:21
sdakewhere ab is teh first 2 or 3 characters of the id19:21
SerlexI will try again, sorry for more dump questions, maste you mean clone again using git?19:21
SamYaplesdake: re: "after the split" thats not what happening19:22
SamYapleordereddicts need ot be setup this way19:22
d_codemy /etc/systemd/system/docker.service (on CentOS 7.2) http://ur1.ca/oivwz19:22
SamYaplewhat do you think is going to occur?19:22
sdakeserlex that works or git pull --rebase will do it  too19:22
d_codethat was the fix for the neutron dhcp thing19:22
Serlexok cheers19:22
d_codeI also rebased19:22
sdakeok time for a break19:22
sdakebbiaf19:22
d_codeSerlex: so that was a binary install too19:22
*** chandankumar has quit IRC19:22
SamYapledave-mccowan: its fairly simply, if external_ssl is enabled then we push a user provided ssl cert to haproxy and set that up19:23
d_codethe source one was still giving me issues with Keystone19:23
Serlexd_code openstack.service  or docker.service?19:23
SamYapleall in all im thinking under 20 lines of code19:23
d_codeoh19:23
SerlexI will do a fresh binary bare-metal install tomorrow19:23
d_codedocker.service19:23
Serlexok19:23
SerlexI will keep a record of your copy19:23
SamYaplei think the biggest patch was the one you already submited dave-mccowan19:23
SamYaplewell no. because we need the second public vip for haproxy. but again not to much work19:24
d_codeSerlex: here’s the steps I did -> https://gist.github.com/dcode/aea450c164803406763719:24
dave-mccowanyea, in lines of code that's the bulk.  there's a few more tweaks needs for some services, especially horizon.19:24
d_codeI’m gonna verify those work on a clean OS install (CentOS 7.2 minimal) and I’ll work on submitting a patch to Kolla19:24
*** neilus has joined #kolla19:24
d_codein the case of that gist, 192.168.4.210 is my interface IP and .200 is the VIP19:25
dave-mccowanalso, we need a way to test it.  do you think a new gate job would be in order?19:26
sdakedave-mccowan our gate doens't do functiona ltesting atm19:27
sbezverkany chance the issue I see with mariadb might be related to docket 1.10.1??19:27
sdakeso the answer is yes we need gating ;)19:28
dave-mccowanSamYaple  haproxy.cfg.j2 also needs some work.  this is where the 2 VIPs discussion comes in.19:28
*** neilus has quit IRC19:29
*** salv-orlando has quit IRC19:30
*** jtriley has joined #kolla19:30
dave-mccowansdake samyaple line 38 of https://etherpad.openstack.org/p/kolla-mitaka-midcycle-ssl is how i expect haproxy.cfg to look.  thoughts on this? (line 26 is how it looks now)19:32
sdakeserlex is this --insecure-registry flag not documented?19:33
sdakethe error lima sees is what SamYaple and I are workign on atm19:34
*** achanda has joined #kolla19:35
d_codesdake: https://github.com/openstack/kolla/blob/master/doc/quickstart.rst#deploy-a-registry-required-for-multinode19:36
d_codeit’s documented19:36
sdakeyap retty sure i put that in ;)19:36
*** banix has joined #kolla19:36
sdakei really dont know what to do when people don't read the quickstart ;)19:37
sdakei would get it if the quickstart were wrong, which it is at the moment19:37
sdakebecause it is mising the docker service file seding19:37
sdakesbezverk  ok lets debug mariadb container plz19:38
*** jtriley has quit IRC19:38
SamYapledave-mccowan: haproxy.cfg.j2 is super simple to tweak19:39
SamYapleyou can listen on multiple address19:39
*** tzn has quit IRC19:39
sdakesbezverk ping me when your ready for that19:40
sdakeso d_code is in good shape19:40
sdakewho else is suffering :)19:40
d_code:thumbsup:19:40
sdakeSerlex your going to try a fresh install then?19:41
sdakeand lima is jammed up on the same issue i am19:41
sdakewhich is blocking development19:41
sdaked_code you just change the docker.service file, removed /var/lib/docker and started fresh?19:42
SamYaplesdake: can you review https://review.openstack.org/#/c/280029/ (note that designate-base is failing for centos-binary, unrelated to this patch)19:42
sdaked_code centos source or centos_binary?19:42
sbezverksdake any time19:42
sbezverkI removed all containers and tried to deploy again19:43
sdakesbezverk run docker ps -a | fpaste19:43
sdakeor paste it19:43
sbezverkand now kolla-toolbox is misbehaving19:43
sbezverkTASK: [common | Starting kolla-toolbox container] *****************************19:44
sbezverkfailed: [192.168.80.25] => {"changed": true, "failed": true}19:44
sbezverkmsg: APIError(HTTPError('500 Server Error: Internal Server Error',),)19:44
*** limamauricio has joined #kolla19:44
sbezverkdocker ps -a19:44
sbezverkCONTAINER ID        IMAGE                                                             COMMAND                 CREATED              STATUS              PORTS               NAMES19:44
sbezverk550d4b451d2c        192.168.80.240:4000/kollaglue/centos-source-kolla-toolbox:2.0.0   "/bin/sleep infinity"   About a minute ago   Created                                 kolla_toolbox19:44
sbezverk463373e04640        192.168.80.240:4000/kollaglue/centos-source-rsyslog:2.0.0         "kolla_start"           About a minute ago   Up About a minute                       rsyslog19:44
d_codesdake: I did binary19:44
d_codesource failed19:44
limamauricioSerlex19:44
d_codew/ the Keystone API issue19:44
d_codesdake: ^^19:44
d_codesdake: yes, I started fresh with the latest checkout of master (at the time), and the updated service file19:45
sdaked_code interesting19:45
sdakesbezverk please use a paste service, if your on centos you can install fpaste from epel-release19:45
sdakeand just pipe it right into fpaste19:45
sdakesbezverk your stuck on the ssame point as i am koolla-toolbox failing to start19:46
sdakenotice "Created" but no uptime on the toolbox container19:46
sdakesbezverk we are working on this problem, not sure if its a regression or what but its clearly creating alot of chaos ;)19:47
sdake5+ people have reported in last couple days19:47
sbezverkhttp://paste.fedoraproject.org/323169/5565715119:48
sbezverksdake interestigly it was working for the first time as you can see from my previous paste19:49
sdakeyup19:49
sdakeit would probably work again further if you reboot your baremetal19:49
sdakebut clearly that is not an acceptable solution long term19:50
sbezverksdake let me try, but it does not explain mariadb issue right?19:50
sdakemariadb issue is new to me19:50
*** blakec has quit IRC19:52
SamYaplesbezverk: what mariadb issue?19:53
sbezverkSamYaple mariadb was kept restarting19:54
SamYaplesbezverk: you most likely had an unclean environment19:55
SamYaplewhen redeploying you need to make sure you clean up previous containers and volumes19:56
*** Slower is now known as Slow19:57
sdakeit was cratering on something related to pid checking19:57
*** Slow is now known as Slower19:57
sdakeslowest19:58
openstackgerritRyan Hallisey proposed openstack/kolla: Upgrade cinder playbook  https://review.openstack.org/27804919:58
sdakebbiaf need to recharge laptop and care for children (today is day off from school)19:59
limamauricioTASK: [neutron | Starting neutron-dhcp-agent container] ******** msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),)20:00
limamauricioCentos, Source, Docker version 1.10.1, build 9e83765, ansible 1.9.420:00
SamYaplelimamauricio: docker 1.10? MountFalgs=shared?20:00
*** vhosakot has quit IRC20:01
limamauriciodocker 1.1020:01
limamauricioMountFlags i don'i know20:01
SamYapleyea ive got docs coming tonight to explain that20:01
limamauricioMountFlags i don't* know20:01
SamYaplecheck this commit message for now https://review.openstack.org/#/c/272715/20:01
sbezverksdake rebooting baremetal did not help with toolbox20:03
*** Serlex has quit IRC20:03
SamYaplesbezverk: we are working on toolbox issue. unfortunately I do not have access to a box with this issue to test patches for it20:04
SamYapleso im reduced to submitting patches that others then pull and test :/20:04
SamYaplereally slows troubleshooting20:04
limamauricioSamYaple, thanks20:04
limamauricioI add MountFlags=shared :)20:04
SamYaplelimamauricio: anytime. docs patch coming tonight20:05
limamauricio:)20:05
sbezverkSamYaple let me know if I can help, I do not mind to test it20:05
sbezverkSamYaple, if you want access my server via web conferencing, that could work too..20:06
*** tzn has joined #kolla20:07
*** Serlex has joined #kolla20:09
openstackgerritRyan Hallisey proposed openstack/kolla: Upgrade cinder playbook  https://review.openstack.org/27804920:12
*** mbound has joined #kolla20:12
rhalliseySamYaple, ^ that should be good.  I don't what was up with that rebase..20:12
limamauricioSamYaple, one doubt, i must install httpd?20:13
*** achanda has quit IRC20:15
limamauricioI've seen somewhere a command that install it, but it does not have seen it in official documentation20:15
*** achanda has joined #kolla20:16
*** openstackgerrit has quit IRC20:17
*** openstackgerrit_ is now known as openstackgerrit20:17
*** openstackgerrit has quit IRC20:17
*** openstackgerrit_ has joined #kolla20:17
*** openstackgerrit_ is now known as openstackgerrit20:18
*** openstackgerrit_ has joined #kolla20:18
SamYaplelimamauricio: for what?20:20
SamYapleim not sure where youll need httpd20:20
SamYapleelemoine_: ping20:20
limamauricioI just saw20:20
limamauricio:)20:20
limamauriciosam20:20
limamauricioafter add MountFlags=shared i must run systemctl restart docker.service20:21
SamYapleyes20:21
limamauricio?20:21
limamauricio:)20:21
limamauriciook20:21
limamauriciothis should work right? if you give the same error, I must rebuild it?20:22
SamYapleyou should not need to rebuild20:22
limamauriciook20:22
SamYapleyou may need to remove the containers/rekick20:22
limamauricioi take a snapshot without containers20:23
limamauricio:)20:23
SamYaplerhallisey: centos binary is failing on designate-base20:23
SamYaplerhallisey: https://review.openstack.org/27804920:23
SamYaplethat patch is ready but you see gate..20:23
d_codelimamauricio: after you change the mountflags, you have to run `systemctl daemon-reload` first20:24
d_codethat makes systemd reparse the service file20:24
limamauriciook20:24
sbezverksdake does neutron or any oher opensatck containers expose in any way its .conf files so it can be changed and the container restarted for changes to become active?20:24
limamauricio systemctl daemon-reload and systemctl restart docker.service20:25
limamauricio?20:25
SamYapleyes limamauricio20:25
SamYaplesbezverk: you are asking if you can make a config change and have the configs take affect?20:25
SamYapleif so we are working on some playbooks for that. our midcycle (last week) covered that topic20:26
*** neilus has joined #kolla20:26
rhalliseySamYaple, I see...20:26
sbezverkSamYaple, yes, example if I add to cinder.conf new backstore or other possible changes20:26
SamYaplesbezverk: as of right now, you can remove teh cinder-api container and rerun the playbooks for the changes to take affect20:27
sbezverkhow I can access cinder.conf within a container and would restarting a container do the trick20:27
SamYapleit depends20:27
SamYapleif you have the config set to COPY_ALWAYS you can change it external to the container20:27
SamYaplethe default is the container itself will only ever hold one container (immutability)20:28
SamYaplebut its configurable to allow it to change its contfig on container restart20:28
SamYaple20:28:47 < SamYaple> the default is the container itself will only ever hold one container (immutability)20:28
SamYaplewill only ever hold one config*20:28
sbezverkSamYaple I see, but how one would access cinder.conf if we taker cinder container as an example?20:29
SamYaplesbezverk: on the cinder-api host, /etc/kolla/cinder-api/cinder.conf20:30
SamYaplewith kolla_config_strategy set to COPY_ALWAYS changes you make there will take affect in hte container on contianer restart20:30
sdakesbezverk yup didn't think it would20:33
sbezverkSamYaple, I think I got it. so COPY_ALWAYS is set, then the container check everytime when it restart for possible changes in the configuration, correct?20:33
sdakesbezverk I am starting to suspect changing mountflags requires a complete removal of docker /var/lib/docker20:34
sdakebut I don't want to damage my vm so I can actually get to the bottom of the problem20:34
sdakesbezverk d_code did an rm -rf of /var/lib/docker after stopping docker and rebooting and that got his centos binary going20:34
d_codewell…I didn’t reboot, but the rest is true20:34
SamYaplesbezverk: correct20:35
SamYaplesdake: MountFlags is not a docker thing20:35
SamYapleits a systemd thing20:35
sdakei see20:35
SamYapleubuntu 14.04 isn't involved here20:35
SamYaplebut, say, 14.10 would need to set that20:35
sdaketaht is the delta between my gear and d_code's gear20:36
sdake(the removal of /var/lib/docker)20:36
sdakebut i guess it still happens randomly after that point20:36
sdakejust got back from break, i'm digging into it now20:37
SamYapleits probably not random, but im going the opposite way20:37
SamYapleim removing the /dev/log binding20:37
SamYapleit was enevitable just annoying20:37
sbezverksdake just removing /var/lib/docker did not help me, trying wit reboot20:39
*** salv-orlando has joined #kolla20:42
SamYaplesdake: can I jump onto your box to check this snadbox thing?20:52
SamYaple(you said its 100% reproduceable?20:52
sdakeits a vm20:52
*** vhosakot has joined #kolla20:52
sbezverksdake rm and reboot did the trick, so I am back to my flapping mariadb container20:52
sdakei'm not sure how I can make it happen SamYaple20:52
SamYapleso?20:52
SamYapleoh20:52
SamYapleyou can port forward20:53
SamYaplebbut i dont feel like walking you through that20:53
sdakefrom my mac laptop?20:53
SamYapleyea but its ok20:53
sdakei dont need walking through - I c an set it up if you like20:53
SamYaplemore trouble than its worth20:53
sdakejust rather try this sorting  thing first20:53
sdakesee if that resolves the problem20:53
SamYaplesdake: its dorting properly20:53
SamYaplesorting*20:53
SamYaplewhat you are suggesting does not do what you think it does20:53
sdakebut sbezverk above rebooted and rm rf20:54
SamYaplerather its ordering correctly, not sorting20:54
SamYapleyou are suggesting sorted (which is bad since that reorders)20:54
sdakei was planning to keep everything intact unless it had a dev target20:54
SamYaplei think we are talking about two different things20:56
SamYaplethe patch i submitted does what ithink it should do, that doesnt work according to you20:56
SamYapleyou should submit one if you have a different vision20:56
SamYaplei cant reproduce20:56
SamYapleor at least not consistently to test20:57
sdakeSamYaple if I had one written I would submit ;)20:57
sdakehard to irc and code at same time ;-)20:58
vhosakotHey all, I'm a dev looking to get involved in kolla, where do I get started ?  I have experience with OpenStack, Docker, Ansible, Python, Vagrant, Shell Scripting, etc... I have read about kolla and really like what is does.. Let me know how I can get started!20:58
SamYaplevhosakot: first things first, deploy kolla!21:00
sdakeif you can ;)21:00
SamYaplethats a good way to get involved, deploying and figuring out what works/doesnt and submitting patches based on that21:00
SamYapleright now there is a fairly crippling issue for some issues due to docker 1.1021:00
SamYapleso we are working on that ATM21:00
vhosakotso, Should I get going with an all-in-one kolla dev environment with Vagrant ?   http://docs.openstack.org/developer/kolla/vagrant-dev-env.html21:01
sdakepeople complain alot about vagrant and end up deploying in their own vm of their own creation21:01
sdakei dont know what hte issue is with vagrant since i dont use it21:01
sdakeeither the docs or implementation need imrprovement or both21:01
sdakethat would probably be a good way to get started if you a a vagrant dude :)21:02
sdakewe have a million things to do after you get up to speed on how kolla deploys21:02
sdakenot really a mllion21:02
sdakeless then 1,000 :)21:03
SamYaplevhosakot: dont listen to him. the overhead isn't high21:03
sdake107 open bugs in bug tracker ;)21:03
sdakehundreds of blueprints21:03
sdakebusy busy project :)21:03
sdakewear a seat belt ;)21:03
sdakeSamYaple you use vagrant?21:03
SamYaplesdake: i went through it. its not work creating your own vm in my opinion21:04
sdakeafter d_code submits his host setup patch, I agree, the vm setup shoudl be pretty easy21:05
sdakeatm following the quickstart guide seems challenging ;)21:05
vhosakotSo, doesn't kolla work out of the box on day 1 for a developer/operator.... I'd like to fix it if the team thinks it needs to be fixed soon :)21:06
SamYaplevhosakot: it does work out of the box, youve come on a bad day21:06
sdakevhosakot unfortunately I think only a super experienced kolla developer can fix this problem21:06
SamYaplethis probably popped up last night21:06
sdakebut once its fixed you will be golden21:06
vhosakot:)21:06
SamYapleyea its been fun. keysotne broke us, docker broke us, we all get broken!21:06
SamYaplebut we fixed the keystoen problem, working through the docker one now21:07
sdakeya dependency management FTW21:07
sdakeget the anestaphine from the medlock!!21:07
vhosakotI understand... totally.. dependent code.... ok, so, then, shall I get started with deploying manually then (non-Vagrant way) ?21:07
vhosakotI have a Ubuntu now that I use for DevStack21:08
sdakeyou can't deploy kolla on the same node as devstack21:08
sdakeor i guess you could but you really wouldn't want to21:08
sdakeyou would damage your devstack, which is painful to fix21:08
SamYaplevhosakot: perfect21:08
sdakeyou will want a new vm to start with21:08
SamYaplevhosakot: ubuntu works well21:08
SamYaple14.04 esspecially at this time21:09
sdakeya no systemd ftw21:09
vhosakotcool, I can create new VM.. no prob... I do it daily :) using vagrant :)21:09
vhosakotyes... I'm on 14.0421:09
sdakevhosakot if you really ar e rockin at vagrant try the vagrant docs21:09
SamYaplevhosakot: yup new vm just follow the deploy guide and it _should_ just work21:09
sdakemaybe second21:09
sdakeafter deploying manually with the quickstart guide21:09
sdakevagrant is meant more for devleopment then eval21:10
sdakequickstart is meant for eval21:10
SamYaplevhosakot: couple of things to remember, need ansible 1.9.4, docker 1.10+, docker-py 1.6+21:10
SamYapleafter that we handle the rest21:10
sdaketo build you need a  bunch of deps as well21:10
vhosakotSamYaple: deploy guide meaning  http://docs.openstack.org/developer/kolla/vagrant-dev-env.html ??21:10
*** blakec has joined #kolla21:10
sdakeinstall test-requirements.txt and requirements.txt on your host21:10
sdakerather on your vm21:10
guillaume_failed: [localhost] => {"changed": true, "failed": true}21:10
guillaume_msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/kolla_toolbox/start',),)21:10
guillaume_FATAL: all hosts have already failed -- aborting21:10
guillaume_Still no solution?21:10
sdakevhosakot those are the docs, there is also a quickstart21:10
sdakeguillaume_ yes we know21:11
vhosakotsdake: please point me to the doc to install kolla... I am searching the repo as well...21:11
SamYapleguillaume_: we are working on it. if you _really_ want a solution, use ubuntu 14.0421:11
SamYapleit wont have this issue21:11
SamYapleotherwise give it a day at most21:11
guillaume_we alredy use 14.04 and have the problem21:11
SamYaplereally?21:11
sdakevhosakot try http://docs.openstack.org/developer/kolla/quickstart.html21:11
SamYaplei cannot reproduce this on ubuntu at all21:11
SamYaplei dont have a machine where I can reproduce this either :/21:12
guillaume_root@kolla-guillaume:~# uname -a21:12
guillaume_Linux kolla-guillaume 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux21:12
sdake3.1321:12
SamYaple3.13, i think we had a reason why that didnt work21:12
vhosakotsdake: thanks!  so, is that link about installing kolla manually from scratch ?21:12
SamYapleguillaume_: whats your backend for docker ? `docker info`21:12
sdakevhosakot right - on bare metal or in a vm21:12
guillaume_root@kolla-guillaume:~# docker info21:12
guillaume_Containers: 3421:12
guillaume_ Running: 2021:12
guillaume_ Paused: 021:12
guillaume_ Stopped: 1421:12
guillaume_Images: 41321:12
guillaume_Server Version: 1.10.121:13
guillaume_Storage Driver: aufs21:13
sdakei'm using overly atm21:13
guillaume_ Root Dir: /var/lib/docker/aufs21:13
guillaume_ Backing Filesystem: extfs21:13
guillaume_ Dirs: 29721:13
sdakegroan don't paste in channnel plz21:13
guillaume_ Dirperm1 Supported: false21:13
guillaume_Execution Driver: native-0.221:13
guillaume_Logging Driver: json-file21:13
sdakeunless its a 1-2 liner21:13
guillaume_Plugins:21:13
guillaume_ Volume: local21:13
guillaume_ Network: bridge null host21:13
vhosakotsdake: nice! ok, I know what to do now.... thanks!  I will let the team know what happens21:13
guillaume_Kernel Version: 3.13.0-77-generic21:13
guillaume_Operating System: Ubuntu 14.04.3 LTS21:13
guillaume_OSType: linux21:13
guillaume_Architecture: x86_6421:13
SamYaplesdake: ahhhhh i think i know the delta. im using btrfs. i saw this on aufs21:13
guillaume_CPUs: 821:13
SamYapleits the overlay21:13
guillaume_Total Memory: 15.67 GiB21:13
guillaume_Name: kolla-guillaume21:13
guillaume_ID: WUWT:AWGH:JP4A:XOPF:PVXA:NH4S:WX3L:2LAL:3UFC:ESOS:JLBU:HJPU21:13
guillaume_WARNING: No swap limit support21:13
guillaume_sry21:13
vhosakotonce I'm up with kolla, I can get started... you guys can let me know what to implement/fix21:13
sdakeinteresting, so graph driver is the culprit21:14
sdakevhosakot sounds like  a plan21:14
sdakethis channel runs 24/721:14
SamYaplesdake: maybe21:14
vhosakotsdake: cool!21:14
sdakesomeone will be around to help you ;)21:14
vhosakotI will be here in the channel everyday21:14
vhosakotcool!21:14
*** rajathagasthya has joined #kolla21:15
*** blakec has quit IRC21:15
sdakei guess i could try a different storage backend21:15
sdaked_code what graph driver are you using?21:15
sdakeI'd like to actually fix the problem rather then hack around it21:15
d_codenovnc?21:15
d_codewait21:16
d_codestorage21:16
sdakegraph driver is a docker thing21:16
d_codedoing the default21:16
sdakeon centos?21:16
d_codeyup21:16
d_codeso…filesystem?21:16
sdakethat is lvm fat driver21:16
d_codewhich, we talked about is a bad idea21:16
d_codebut it worked21:16
sdakeSamYaple you ought to give overlayfs a spin21:16
sdakemight get you a reproducible environment21:17
sdakethe thing is d_code everyone else in here with this problem is using the same graph driver as you21:17
sdakeSamYaple the graphdriver in the gate is diffferent then what everyone else is using as well for centos21:19
sdakewhich may explain why centos in the gate works most of th etime21:20
SamYaplebtrfs in the gate21:20
sdakeyup21:20
SamYaplebut if its a race btrfs is _much_ faster than overlayfs when it comes to mounting21:20
sdakeit probably doesn't matter what order we specify to api, its probably going  to roll in different threads21:21
sdakethreading = evil21:21
sbezverksdake figured out mariadb issue, was misconfig. used the same ip address, so mariadb failed to start the service..21:21
sdakesbezverk you mean you VIP was your host's IP?21:22
sdakesounds like a good precheck21:22
SamYaplesdake: its in the prechecks21:22
sdakeif kolla_internal_address in list_of_nodes - explode21:22
sdakeok folks i am going to focus for a bit on this problem, dropping away from irc21:23
sdakeif I seem non-responsive, thats why :)21:23
*** cloudnautique has quit IRC21:27
vhosakotIs the heading "Bare Metal Deployment of Kolla" in http://docs.openstack.org/developer/kolla/quickstart.html.... shouldn't it be "Deployment of kolla on a bare metal node or a virtual machine" ? :)21:28
SamYaplevhosakot: docs patches are always welcome21:28
vhosakotI will find all issues in the quickstart (if any) and combine all of them into a single doc-fix merge21:28
vhosakotok, cool21:28
*** achanda has quit IRC21:29
sdakeSamYaple just struck gold21:32
sdakehang tight log incoming21:32
*** aginwala has joined #kolla21:34
sdakehttp://paste.fedoraproject.org/323210/14555720/21:34
sdakefirst order works21:35
sdakesecond order doesn't work21:35
sdakerepeated several times same results21:35
sdakelogs same result about the sandbox container nonsense21:35
sdakethat atleast proves its an ordering problem21:37
guillaume_i changed my kernel to 4.2.0-27 and still have the issue with the kolla-toolbox21:37
sdakethe gate flakiness may demonstrate its a race condition21:37
sdakebut that is a bit of speculation21:38
SamYapleyea but not an ordering problem i think we can fix unfortunately21:38
SamYapleits not a race21:38
SamYaplethats wrong21:38
SamYapleits a... hashing? problem21:38
SamYaplewhen a dict is created its keys are hashed21:38
openstackgerritShaun Smekel proposed openstack/kolla: Adds support for cleaning up bootstrap containers in the toos/cleanup-containers script.  https://review.openstack.org/27996121:38
*** cloudnautique has joined #kolla21:38
*** cloudnautique has joined #kolla21:39
*** cloudnautique has quit IRC21:39
sdakeasalkeld you around21:39
openstackgerritShaun Smekel proposed openstack/kolla: Adds support for cleaning up bootstrap containers in the toos/cleanup-containers script.  https://review.openstack.org/27996121:39
*** cloudnautique has joined #kolla21:40
sdakeSamYaple join #heat21:41
sdakezaneb is super expert at python21:41
sdakehe will e able to give us a concrete answer if there is  a solution21:41
*** aginwala has quit IRC21:46
*** subscope has quit IRC21:48
*** aginwala has joined #kolla21:48
sdakeok sam, your patch looks right21:50
sdakebut doesn't work21:50
*** mdnadeem has quit IRC21:50
sdakecould we try in reverse order see if that changes anything?21:50
*** mdnadeem has joined #kolla21:50
*** zaneb has joined #kolla21:53
zanebsdake, SamYaple: o/21:53
SamYapleas i said before, dont worry about it, ive got the long way around patch comming21:53
SamYaplethanks zaneb21:53
sdakehey zaneb21:53
sdakeya thanks zaneb for helping clear that up21:54
zanebI lurk here so we can avoid spamming the other channel21:54
zanebs/I/I'll/21:54
SamYaplesdake: this is broken because of the 1.10 patch i submitted which was required to fix /dev bindmount21:54
SamYaplefor whatever reason on my ubuntu it hashes in the right order (out of luck) most of the time21:54
SamYapleive got a bigger fix coming that ignores this whole reordering thing21:55
sdakei guess the question i have is what is hasing ;)21:55
* zaneb is still confused but if y'all have it figured out that's great ;)21:55
SamYaplesdake: https://wiki.python.org/moin/DictionaryKeys21:55
zanebsdake: a dict is just a hash table21:55
sdakeyes i know that21:55
sdakei did not know it could be ordered21:56
sdakethat is what the question was about21:56
SamYaplezaneb: so the issue is we have a list of mounts that need to happen in a certain order, between kolla and docker-py that list is becoming unordered21:56
SamYapleeven when using ordered dicts21:56
SamYapleso the fix im doing is to remove the dependacy on those mounts being in a certain order21:56
sdakewfm21:57
zanebare you sure it's between kolla and docker-py, or between docker-py and docker?21:57
sdakezaneb not sure21:57
SamYaplezaneb: havent dug in, looks like docker-py to docker to be honest21:57
sdakei'd like to find out to get the bug fixed21:57
SamYapleim passing an ordered dict (confirmed that) so it might be docker-py to docker21:57
sdakebut master is broken so that  takes priority atm21:57
*** rajathagasthya has quit IRC21:58
SamYaplemaster is broken on some systems21:58
SamYaplenot mine for some reason :/21:58
SamYaplehard to test fies21:58
SamYaplefixes*21:58
*** Serlex has quit IRC21:58
sdakenow that I understand the issue, I am happy to test any fix you have SamYaple21:58
zanebSamYaple: wait, what is self.module.params in kolla_docker.py? is that a dict too?21:58
*** macsz has joined #kolla21:59
zanebSamYaple: self.module.params['volumes'] I mean21:59
*** gtt116_ has joined #kolla22:00
SamYaplezaneb: list22:00
SamYaplethat was my thought earlier too22:01
*** alyson_ has quit IRC22:02
SamYaplezaneb: found it. think its a docker-py bug22:02
SamYapleat this point in the code the volumes will be a list (no matter if passed in as a dict) and this code runs every time22:03
SamYaplehttps://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L84122:03
*** salv-orl_ has joined #kolla22:03
*** macsz has quit IRC22:03
*** gtt116__ has quit IRC22:04
SamYaplesdake: you willing to test a docker-py code change?22:04
sdakeabsolutely22:05
sdakefpaste the diff please22:05
sdakeor paste22:05
sdakeor whatever22:05
zanebewww22:05
*** salv-orlando has quit IRC22:05
sdakewhats the eww about zaneb ;)22:06
zanebabout https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L84122:06
SamYaplesdake: make it look like this http://paste.openstack.org/show/487043/22:06
SamYapleim not going to diff it because i didnt build a patch22:06
SamYapleill be forking docker-py in a moment though22:06
sdakewill do22:06
sdakebbiaf22:06
SamYapleyou can test my fork i suppose22:06
sdakei'll ust hack my env up - its no big deal easy to fix22:06
SamYaplezaneb: at that point the volumes stuff could have looked like this, string->list->dict->string->list->dict22:07
SamYaplezaneb: how awesome is that!22:07
SamYaplebefore it was ever used!22:07
zaneblol22:07
zanebSamYaple: that code only runs on a list, so you could pass an OrderedDict to it and not have to fork docker-py22:08
zanebif you have control over the code that calls that, at least22:09
SamYaplezaneb: it gets converted to a list before then :(22:09
SamYaplein docker-py22:09
SamYaplezaneb: it gets converted to list here https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L29522:09
SamYaplethen returned _back_ to kolla_docker, then its expected to be either a string or list22:10
SamYaplei suppose I could override it... but then i lose the ability for docker-py to validate/change my dict to work with docker22:10
*** banix has quit IRC22:10
d_codeso, while my clean build is running…  tunnel_interface is globals.yml is for the vxvlan traffic between compute/controller hosts, right?22:12
SamYapled_code: correct, it defaults to network_interface variable22:12
d_codeapi_interface is where the actual web ui and API endpoints for running openstack cli cients, right?22:12
SamYapleyes, and a few other things like mysql traffic22:13
d_codeand storage, is obviously for storage…in the default install, is that ceph?22:13
sdakezaneb thoughts on SamYaple's analysis?22:13
SamYapleit should be ceph, but i fear that variable is entirely unused atm22:13
SamYapled_code: that code is untested by most, but thats the idea behind the variables (ill be gating them at some point)22:14
d_codeso, it would go over api_interface, probably?22:14
SamYapleprobably22:14
*** neophy has quit IRC22:14
zanebSamYaple: host_config['Binds'] = OrderedDict((k, dict()) for k in host_config['Binds'])?22:14
SamYaplezaneb: yes that would work sure, but my point was docker-py validates that stuff22:14
SamYapleit has all the code about deprecation and what not22:14
SamYaplewe would have to give it up22:14
SamYaplealso stuff like https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L295 zaneb22:14
zanebah, ok. I am missing the big picture of the data flow here so just spitballing22:15
SamYaplebasically docker-py sanatizes it into a list. while I _could_ pass a dict, I would have to sanatize it in kolla_docker22:15
SamYaplebut yes, technically possible22:15
openstackgerritSam Yaple proposed openstack/kolla: [WIP] Move socket binding to named_volume  https://review.openstack.org/28042922:16
SamYaplesdake: theres a big patch to hopefully sidestep this whole issue22:16
sdakebe cool if we can fix in docker-py for other folks too :)22:17
sdakei'll get  to teting your patch in a moment22:17
sdakeever seen south park?22:17
sdakethey are coming right for me!22:17
vhosakotI need the VM to have two interfaces, right ? eth0 (network_interface) and eth1 (neutron_external_interface) /etc/kolla/globals.yml22:18
sdakeso I think a plan of action is to fix this in docker-py so we aren't afflicted by it in the future, possibly workaround in whatever way is most sanitary, and get a patch merged asap :)22:18
SamYaplevhosakot: yes. it is technically possible to do kolla with one interface but it is complicated and not-documented22:19
sdakevhosakot yes 2 ips ;)22:19
SamYaplesdake: you confirm it works and ill submit a PR for docker-py22:19
sdakewill do sam, working on getting my env going22:19
vhosakotSamYaple: ok, two interfaces, two IPs, two networks22:19
sdakeno ip on the second interface22:19
vhosakotok22:19
vhosakotwhat tool do you guys use ? VirtualBox ?  I use virt-manager and it does not allow me to add more than one interface per guest VM22:20
sdakeatleast we are to the root cause of the issue after two days :)22:20
sdakei use bare metal when its available, in remodel atm so using virutalbox22:20
sdakevhosakot wwith virt-manager you ahve to creat a second network22:21
vhosakotI dont have a bare-metal server22:21
SamYaplevhosakot: virt-manager you can hot add a second interface22:21
sdakethat is a little complicated and undocumented :)22:21
vhosakotsdake: ah, second network in virt-manager22:21
vhosakotthought so.. ok, thanks!22:21
sdakesam have your pate open22:22
sdakewher edoes line 1 go22:22
SamYaplei dont know what version of docker you have so the line numbers change22:23
sdakewhich file/function22:23
SamYaplebut its in docker/utils/utils.py22:23
sdakethanks22:23
*** bmace has quit IRC22:23
*** bmace has joined #kolla22:23
SamYaplethe function is create_container_config() I *think*22:24
sdakegot it22:24
sdakei'll give ita  go22:24
vhosakotsdake: I see I can create a second virtual network, but, when creating the VM, I can attach it to one one network... Shouldn't I attach the VM to both the networks..22:24
vhosakotin virt-manager22:24
*** mbound has quit IRC22:26
*** blahRus1 has quit IRC22:26
*** dwalsh has quit IRC22:26
*** JoseMello has quit IRC22:28
*** achanda has joined #kolla22:30
sdakecd SamYaple that 2 liner failed22:30
sdakesame results22:30
sdakelooked good though22:30
SamYaplehmm. i wish I could be in the environemnt22:31
sdakei'll see if I can figure that out22:31
SamYaplesdake: did you use the docker-py with the kolla patch i submited earlier?22:31
sdakejust keep sorting out your workaround22:31
sdakewhich one?22:31
sdakeoh no22:31
sdakeduh22:31
sdakelet me do that22:31
sdakesame result22:32
sdakei can either debug this or setup my env for you to get in22:33
SamYaplewell idk. i need to pdb at this point22:33
sdakewhich would you prefer22:33
SamYapleif you setup your environment ill log in22:33
SamYaplebut i cant give anymore pointers22:33
sdakeanymore pointers?22:34
SamYapleas to what the issue might be or possible fixes22:34
sdakeyou mean about environnmental setup22:34
SamYapleabout the issue at hand22:34
SamYaplethis patch should make it better though https://review.openstack.org/28042922:34
SamYapleavoids the whole issue22:34
sdakeright22:35
SamYapleif you _still_ hit the issue, this is a different problem22:35
sdakelet me drop off the vpn so i can get you access if I can figure out how to get my vm an ip address ;)22:35
SamYapleit might be worth it to give it a spin22:35
sdakeok i'll try it now moment22:35
SamYapleyoull need to rebuild22:36
sdakeya that works22:37
sdakedeploys atleast22:38
SamYapleyou rebuild already?22:38
sdakelet me rebuild didn't notice the docker changes22:38
SamYaplewell yea22:38
SamYaplei mean that will _work_ but logging is broken until you rebuild22:38
sdakeya i see that22:38
SamYaplethis is much cleaner anyway22:38
SamYaplename_volumes to share teh socket22:39
sdakeya i liek your change22:39
sdakehave you tested it?22:39
SamYapleyea22:39
sdakeworks for you?22:39
SamYaplefull deploy with logging22:39
SamYapleyea22:39
sdakeok22:39
sdakebuilding - t-45 minutes out22:39
SamYapleits WIP because no bug/bp22:39
sdakeright22:39
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Make Mesos resources configurable  https://review.openstack.org/28014422:42
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510922:42
*** athomas has quit IRC22:44
sdakeok SamYaple need to eat - bbiaf afte rbuild done with results22:47
*** tzn has quit IRC22:52
*** aginwala has quit IRC22:56
*** haplo37 has quit IRC22:58
*** aginwala has joined #kolla22:58
sdakenamed volumes are pretty sweet tech22:59
asalkeldnihilifer: https://review.openstack.org/#/c/280144/223:02
asalkeldmem/cpu are supported23:02
vhosakotsdake: ok, I created a VM with two network interfaces (eth0 and eth1) using virt-manager... I expected virt-manager allows me to attach two networks when creating teh VM... it does not.. so, I created the VM with one interface, added the second interface, and rebooted the VM..23:03
sdakethat should do it23:03
vhosakotsdake: may be I'm used to how OpenStack horizon allows the user add a VM to multiple networks when creating the VM.. :)23:04
vhosakotsdake: cool23:04
sdakeymmv with libvirt and two networks and actually getting neutron to work23:04
sdakeI never operate kolla that way23:04
sdakeI always operate on bare metal23:04
sdakebeen doing vms for about a month since my house is disassembled23:05
sdakeits painful23:05
sdakei'm almsot to the point of moving my lab into my bedroom23:05
vhosakotso, on the bare metal, you have two physical NICs or virtual NICs ?23:05
sdakeexcept my 1.2tb pcie ssd is busted and needs to be shipped to intel for repair23:05
vhosakotwow23:05
sdakei have two 10gige nics23:05
vhosakotnice23:05
nihiliferasalkeld: ok, i see23:12
d_codesdake: so, if I want compute services on two hosts, do I need to change the mariadb role to just be one?  I tried to deploy with two control  hosts and it died on mariadb23:12
sdakeha requires 3 hostss23:13
d_codeah23:14
d_codethat makes sense23:14
sdakealthough it can be dpeloyed with 223:14
sdakepretty sure SamYaple deploys with 223:14
sdakebut tbh I only test with 323:14
d_codewell…I just fell back to 1 control host for now, 2 compute hosts23:14
sdakeif you paste your inventory file i could take a look23:14
sdakeit could be a bug in master23:14
sdakeonly a couple people in the community have gear to test ha23:15
d_codek. holding out fo the deploy to complete23:15
sdakeits my default setup23:15
sdakethe mariadb role shoudl not be on a compute host though23:15
sdakerereading your question, I'm not sure i parsed it correctly the first time23:15
sdakebeen a long long day23:15
d_codeI’ve got….a unique deployment scenario23:15
sdakeyou can run control and compute on two nodes23:16
sdakeand sort of get ha ;)23:16
dave-mccowani'm following the quick start guide and i'm at the tools/build.py step.  some of my containers are failing to build.  the doc just says keep trying...    any other advice?  looks like the centos httpd package is related to the root cause.23:16
sdakedave-mccowan httpd should pull in fine23:16
d_codesdake: http://ur1.ca/oiwym23:16
sdakedave-mccowan what container ?23:16
d_codeI had ddi001 and ddi002 on all roles but storage23:17
dave-mccowanseveral, including keystone and horizon23:17
d_codeand maria failed23:17
sdaked_code that should absolutely work23:17
sdakedave-mccowan it is possible the mirrors are busted23:18
* d_code shrugs23:18
d_codeI’ll circle back to it23:18
sdakewhat I would recommend is to rebuild your base container and keystone and horizon23:18
sdakebuidl.py keystone horizon will do the job23:18
sdakerather23:18
sdakebuild.py --no-cache keystone horizon23:18
sdakesometimes the upstream repos change and files become unavailable from the repo that is installed in teh base image23:19
sdakerebuilding the base image fixes this problem23:19
sdakesince the mirrors are propogating over the internets all the time23:19
*** jtriley has joined #kolla23:19
sdakeand basically your mirror has either been deleted or is repropogating23:19
sdakedave-mccowan does that make sense?23:20
d_codehmmm…. what provides the neutron command?23:20
d_codeb/c….that’s aint workin23:20
d_codepython-openstack?23:20
dave-mccowanyep.  i'll do that.  if it still fails, i should wait and try agin?  maybe the mirrors will be less busy later tonight.23:21
d_codeclient23:21
sdakepython-neutronclient23:21
sdakepython-openstackclient may ahve equivalents23:21
sdakeit is in rapid development23:21
sdakei use the old school clients because old dog new tricks cant teach and all that ;)23:22
sdaked_code ^^23:22
d_codethanks23:22
dave-mccowansdake i'm not clear on the resolution of the 2 VIP direction.  can you give you your view of next steps on that?23:22
d_codeweird…they were there when I did the allinone…but not now23:22
sdakedave-mccowan SamYaple agreed that 2 VIPs managed by HAproxy is the way to go from my parsing23:23
sdakeand apparentl ythis was discussed at the midcycle23:23
sdakei only type 110 WPM :)23:24
*** vhosakot has quit IRC23:24
*** achanda has quit IRC23:25
dave-mccowansdake seems reasonable.  tls changes will be easier once that is done.23:25
d_codesdake: yeah, so iptables is totally in the way :p23:27
d_codeor did I screw myself and kolla attempts to configure firewalld?23:27
sdakekolla doesn't configure firewalld23:28
sdakeif you have host ip tables, I generally turn my rules off23:29
sdakewe probably need a document that describes how to setu pa host23:29
sdakei wanted this this cycle23:29
sdakebut it hasn't happened yet23:29
d_codek.  I’ll have to automate that for my deployment23:29
*** yuanying_flu is now known as yuanying23:29
d_codesdake: the default init networks23:31
d_codeis the idea the vm gets an IP from “demo-net” (10.0.0.0/24)23:31
*** neilus has quit IRC23:31
d_codeand 10.0.2.0/24 are “floating IPs”?23:31
d_codewhich would be available on the nova_external interface?23:32
sdaked_code yes23:32
d_codecool23:32
d_codethanks23:32
sdakedemo-net is a private network23:33
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Make Mesos resources configurable  https://review.openstack.org/28014423:33
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config  https://review.openstack.org/26510923:33
sdakeprivate as in neutron private23:33
*** achanda has joined #kolla23:33
d_codesdake: https://gist.github.com/dcode/aea450c164803406763723:35
d_codethat’s what finally worked for me23:35
d_codeI’ll get around to submitting it as you asked23:35
openstackgerritAngus Salkeld proposed openstack/kolla-mesos: Update docs on adding a new service  https://review.openstack.org/28044423:35
asalkeldnihilifer: ^23:35
d_codeI’m out…time to make dinner23:36
sdaked_code I'd remove line 106+23:36
sdakebut the rest looks beutiful23:36
sdakeI shoudl have done that long ago23:36
*** Jeffrey4l has joined #kolla23:39
d_code:)23:39
sdakedave-mccowan fwiw I just rebuilt and onl ythe containers that shouldn't build dont on centos binary23:39
sdakethese include designate, mistral, gnocchi, and murano23:40
sdakethese don't build because there is no upstream RPM for them23:40
sdakenotice there is one for magnum ;-)23:40
*** dims_ has quit IRC23:41
asalkeldsorry, nihilifer getting a bit picky :-(23:42
openstackgerritMichal Rostecki proposed openstack/kolla: Ensure that /var/lib/zookeeper is owned by user  https://review.openstack.org/28044523:42
dave-mccowansdake those failed for me too.  but also keystone and horizon?23:43
sdakekeystone should work and horizon should work23:44
sdakedid you tyr the no-cache thing?23:44
*** banix has joined #kolla23:45
sdake./build.py --no-cache keystone horizon23:45
sdakethat will buid only keystone and horizon23:45
sdake(and its dependencies)23:45
*** jtriley has quit IRC23:46
nihiliferasalkeld: good idea with conf profiles23:47
dave-mccowanyep, that failed.   httpd.x86_64 0:2.4.6-40.el7.centos   is the reported failing package.23:48
sdakeSamYaple your patch is deploying atm, i'll let you know23:48
sdakedave-mccowan is there any more data available?23:48
sdakeif you run script23:48
sdakeyou can run the build23:49
sdaketype eixt23:49
sdakea file typesciript will be created23:49
sdakesamyaple http://paste.fedoraproject.org/323244/8031514523:52
*** jtriley has joined #kolla23:52
sdakeneed more anestaphine from the medlock!!23:52
*** tzn has joined #kolla23:52
dave-mccowansdake http://paste.openstack.org/show/487051/23:52
*** banix has quit IRC23:55
SamYaplesdake: so it deployed, but broke with mariadb?23:55
sdakekolla-toolbox looks like it is missing jinja223:56
sdakedave-mccowan fwiw my keystone container has that httpd in it23:56
sdakefrom your base image, remove line cd23:56
SamYaplesdake: _binary_ _centos_ kolla-toolbox is busted for you. thats important23:56
*** dims has joined #kolla23:57
SamYapleits a big difference in that container23:57
sdakeit works for d_code23:57
asalkeldthis is irritating bug 154428723:57
openstackbug 1544287 in kolla-mesos "unable to launch containers, Unexpected Env format for 'ContainerConfig.Env'" [Critical,Confirmed] https://launchpad.net/bugs/154428723:57
*** tzn has quit IRC23:57
SamYapleasalkeld: what version of docker-py?23:58
asalkeldmesos is using docker inspect to validate ENV23:58
sdakedave-mccowan throw in RUN rpm -e yum-plugin-fastestmirrors23:58
asalkeldthe issue seems to be a newer version of mesos23:58
sdakedave-mccowan in the keystone container23:58
sdakethen build.py keystone23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!