Thursday, 2016-02-18

*** aginwala has joined #kolla00:00
*** ssurana has joined #kolla00:05
*** jtriley has joined #kolla00:12
*** spisarski has quit IRC00:15
*** tzn has joined #kolla00:20
openstackgerritSteven Dake proposed openstack/kolla: Tidy up README.rst  https://review.openstack.org/28159100:21
*** tzn has quit IRC00:26
asalkeldany one ever seen docker fork errors during the image build?00:35
asalkeldalso: "failed to register layer: Untar re-exec error: exit status 2: output: runtime/cgo: pthread_create failed: Resource temporarily unavailable"00:36
asalkeldDocker version 1.10.1, build 9e8376500:37
asalkeldtried docker-py version 1.6 and 1.700:37
*** RA__ has joined #kolla00:38
*** jtriley has quit IRC00:40
*** jasonsb has joined #kolla00:50
SamYapleasalkeld: ... no00:51
SamYapletry fewer threads00:51
SamYaplewhats your backend?00:51
openstackgerritMerged openstack/kolla: Fix the mistral services name typo  https://review.openstack.org/28119900:51
asalkeldhow do i figure that out?00:51
SamYapledocker info00:52
asalkeldaufs00:53
SamYaplenope00:53
SamYaplehaent seen it00:53
asalkeldhttp://paste.openstack.org/show/487358/00:53
asalkeldok, don't worry00:53
SamYapleoh asalkeld i know00:53
*** Jeffrey4l has joined #kolla00:54
SamYaplei bet thats a similiar problem with rabbit00:54
asalkeldyeah?00:54
SamYaplerabbit runs out of process and you have to tell the kernel it can start more00:54
SamYaplesome limit00:54
SamYapleplimit?00:54
SamYapleulimit!00:54
asalkeldok, looking00:55
SamYaplethats what i tweak with rabbitmq00:55
SamYaplewhats ulimit -a have for open files?00:55
SamYaple`ulimit -a`00:55
SamYaple`ulimit -n 10000`  sets it to 10000 open files00:56
asalkeldopen files                      (-n) 102400:56
* SamYaple points out sockets and handles are open files00:56
asalkeldcool thx00:56
openstackgerritMerged openstack/kolla: Fix missing 14.04 info  https://review.openstack.org/28153900:57
*** iceyao has joined #kolla01:00
*** achanda has quit IRC01:03
*** sdake_ is now known as sdake01:05
*** iceyao_ has joined #kolla01:07
*** tpot has quit IRC01:07
*** iceyao has quit IRC01:10
openstackgerritJeffrey Zhang proposed openstack/kolla: also remove the volumes when cleaning up containers  https://review.openstack.org/28072201:13
openstackgerritMerged openstack/kolla: Adds support for cleaning up bootstrap containers in the toos/cleanup-containers script.  https://review.openstack.org/27996101:14
*** achanda has joined #kolla01:20
*** ssurana has quit IRC01:28
*** sdake has quit IRC01:34
openstackgerritMerged openstack/kolla: Tidy up README.rst  https://review.openstack.org/28159101:35
*** diogogmt has joined #kolla01:42
*** diogogmt has quit IRC01:42
*** diogogmt has joined #kolla01:44
*** diogogmt has quit IRC01:49
*** sdake has joined #kolla02:03
*** sdake has quit IRC02:03
*** sdake has joined #kolla02:04
*** rhallisey has joined #kolla02:05
*** aginwala has quit IRC02:06
SamYaplehey mandre02:08
*** aginwala has joined #kolla02:09
*** aginwala has quit IRC02:09
*** aginwala has joined #kolla02:10
*** Marga__ has joined #kolla02:10
mandrehey02:11
mandreSamYaple, doesn't check_failure() handle all cases?02:11
SamYapleall failure cases sure02:12
SamYaplewe still want to copy the logs on success02:12
*** Marga_ has quit IRC02:12
mandreI was mistaken because of the trap, I though it was called when the script exited02:14
SamYapleoh right. yea it used to be02:14
SamYapleJeffrey4l fixed that up02:14
* mandre needs to revise the basics02:14
*** Marga__ has quit IRC02:14
*** vhosakot has joined #kolla02:15
*** d_code has quit IRC02:18
*** d_code has joined #kolla02:19
*** tzn has joined #kolla02:24
mandreSamYaple, check_failure is also called when the script exits properly without error02:25
mandream I missing something?02:25
*** diogogmt has joined #kolla02:26
*** tfukushima has joined #kolla02:28
*** tzn has quit IRC02:29
SamYaplemandre: no but i might be02:29
SamYaplei thought it only caught failures02:29
SamYaplebut reading it again it definetely catches everything02:29
SamYaplei think it intially was only failures and i just never updated my head code02:29
mandrenah, initially it could be missed when there was a error in the script because of set -o errexit02:31
mandrenow the trap makes sure it's always called when the script exits02:32
SamYaplewell either way the way it works now you are right, its redundant02:32
Jeffrey4lwith trap, the check_failure is called when script exit( including script exit with error).02:37
*** alger has joined #kolla02:37
openstackgerritMartin André proposed openstack/kolla: Make Mistral deployable via ansible  https://review.openstack.org/28161002:39
*** aginwala has quit IRC02:40
ayoungSamYaple, Was able to run a python-keystoneclient functional test against Kolla...getting closer!02:41
*** vhosakot has quit IRC02:43
SamYaplew00t ayoung !02:44
SamYaplei got in my new processor02:44
SamYaplethat completes my cluster!02:44
ayoungSamYaple, Need to make a fix to Keystone.  Link to deploying Keystone On Kolla out of a git repo?02:45
ayoungSamYaple, its one of those things that works when executed against SQLite, which does not enforce integrity constraints.  Need to test against MySQL...02:45
SamYapleayoung: did you generate a kolla-build.conf yet?02:45
ayoungnope02:46
SamYapletox -e genconfig02:46
SamYapleayoung:02:47
SamYaple# The location for source install (string value)02:47
SamYapleoops02:47
*** tpot has joined #kolla02:47
SamYapleayoung: http://paste.openstack.org/show/487359/02:47
ayounglooking02:47
SamYaplebe warned, i did not implement this nor have I used it. But the oracle cats use it exlusively02:48
ayoungSamYaple, what about a local git repo?02:48
SamYapleI do not know the answer to that02:48
ayounglocation = /opt/stack/keystone02:49
SamYapleit *might* work02:49
SamYaplepeople have talked about taht before02:49
SamYapleayoung: looks like no. give me a second ill throw up a patch for you02:50
ayoungcool02:50
dtturnerHi - I'm hitting the following error starting rsyslog container during deployment: msg: AttributeError("'Client' object has no attribute 'create_host_config'",)02:55
dtturnerOn my deploy host I’m running docker version 1.9.1, and docker-py 1.7.002:55
dtturnerI hit this building / deploying from trunk a couple of weeks back as well as with a build from trunk today.   Any ideas?02:56
SamYapledtturner: docker 1.10 is require for trunk02:57
SamYapleayoung: got a patch, testing02:57
SamYapleayoung: be warned, it will probably change before merging. people have opinions02:57
dtturnerSamYaple: Eek.  OK.  I missed that.  Thanks.02:57
ayoungSamYaple, not a problem.  I'll give it a test run once its posted02:58
*** tpot has quit IRC03:00
*** tpot has joined #kolla03:02
openstackgerritSam Yaple proposed openstack/kolla: [WIP] local git repo build  https://review.openstack.org/28161503:02
SamYapleayoung: tested ^^03:02
ayoungSamYaple, thanks03:02
SamYapleayoung: http://paste.openstack.org/show/487360/03:03
SamYapleno reference tag yet, so it just tars up your git directory03:03
SamYaplemandre: im a bit overloaded, do you have time to take over https://review.openstack.org/281615 ?03:03
ayoungSamYaple, not file:/  ?03:03
SamYapleayoung: nah this was quick and dirty03:04
ayoungI guess no need03:04
SamYaplelong term it will be file:// i bet03:04
SamYapleno need for a "localgit" type03:04
ayoungSamYaple, did the non-root deploys merge already?03:04
SamYapleyea03:04
*** salv-orl_ has quit IRC03:04
mandreSamYaple, np03:05
ayoungalso I have the edit03:05
ayoung--- a/requirements.txt03:05
mandrelet me see what you've done...03:05
ayoung-docker-py>=1.4.0 # Apache-2.003:05
ayoung+docker-py<1.7 # Apache-2.003:05
ayoungcan that go?03:05
ayoungguess I can git stash that one03:05
SamYapleayoung: you _have_ to edit those reqs?03:05
SamYaplebtw master should have docker-py 1.6.003:05
ayoungSamYaple, I don;t remember03:05
ayoungso that merged03:05
SamYapleyea03:06
SamYaple1.6.0 is up03:06
SamYapleits in master kolla03:06
*** iceyao_ has quit IRC03:09
*** iceyao has joined #kolla03:09
ayoungSo far so good03:14
ayoungSamYaple, does that work for libraries as well, or just the servers?03:15
SamYaple?03:15
SamYapleoh you mean like a certain version of a certain library?03:15
ayoungSamYaple, python-keystoneclient specifically03:15
ayoungand keystoneauth03:15
SamYaplewell we install based on constraints from master requirements repo03:16
SamYaplehttps://github.com/openstack/kolla/blob/master/docker/keystone/Dockerfile.j2#L5503:16
SamYapletell me what specifically you need and ill see what needs to happen to make that work03:16
*** iceyao has quit IRC03:18
*** iceyao has joined #kolla03:18
*** unicell has quit IRC03:19
ayoungSamYaple, I'll loop back around with you on that. Don;'t need it tonight03:19
*** sdake has quit IRC03:20
SamYapleawesome03:20
*** yuanying_ has quit IRC03:21
*** tzn has joined #kolla03:25
openstackgerritMartin André proposed openstack/kolla: [WIP] Add local directory or file for source build  https://review.openstack.org/28161503:29
*** tzn has quit IRC03:29
mandrebrb, lunchtime03:30
*** dims_ has quit IRC03:30
*** dims has joined #kolla03:32
*** iceyao has quit IRC03:35
*** iceyao has joined #kolla03:35
*** jtriley has joined #kolla03:36
*** yuanying has joined #kolla03:39
*** inc0 has joined #kolla03:41
inc0hey03:41
SamYaplehey inc003:42
SamYapleyoure on late03:42
inc0well, happends03:42
inc0I think we need second opinion on nova upgrades:P03:42
inc0because we are in tie03:42
inc0and need someone to break it03:42
SamYaplei stil ldont understand what you are complaining about03:42
inc0let me put it this way03:42
inc0if I would change something in start task for nova_conductor03:43
SamYapleif you want to do the work to split out the start tasks now to not violate DRY03:43
SamYaplethats fine03:43
inc0currently I'd probably grep for start task03:43
SamYapleits a bit more work but it sfine03:43
inc0we need to do this03:43
inc0but not in this patchset03:43
SamYaplethats what i said03:43
inc0and even then it wont help this case03:43
SamYapleit will03:44
inc0because if we split it03:44
SamYapleonly one place for nova-conductor to start03:44
inc0we still need to change common start AND check image03:44
*** iceyao has quit IRC03:44
inc0every upgrade play have duplicate start03:44
*** iceyao has joined #kolla03:44
SamYaplewhat are you talkign about03:44
inc0so if you make a project-wide change of mounts or whatever03:44
inc0I bet you will forget about this place03:45
SamYapleif yo uare talkign about the duplicated start nova-conductor tasks, then yes thats violating DRY03:45
inc0I'm talking about copypaste from start.yml in yupgrade plays03:45
SamYaplewe have discussed this03:45
SamYapleeveryone should be aware of this03:45
SamYaplewe said we would refactor after upgrades03:45
SamYaplewe didnt want to slow or complicate upgrades03:45
SamYapleif you are saying that _HAS_ to be done now, im ok with that03:46
inc0and if we refactor we probably will forget about this one03:46
SamYaplelike i said, you want to do it now im not going to _block_ that03:46
SamYapleinc0: you pick the wierdest battles. I jsut don't want to violate DRY03:49
SamYaple(also the underscore naming is just bad practice here)03:49
*** alger has quit IRC03:54
*** yuanying has quit IRC03:55
*** unicell has joined #kolla03:55
inc0SamYaple, well, DRY exists to avoid bugs caused by changing stuff in one place and forget to do it in other places03:57
inc0there is little space for it in my code and this file changes raretly03:57
inc0on the other hand if I follow by your decision, I introduce new, less obvious and more likely to change place for that03:58
inc0and bugs there are harder to debug and much less obvious03:58
inc0and yeah, we need to refactor dry in plays, but that's ansible problem and that's not an easy fix03:59
inc0as for underscore names, this is cosmetics, I'm ok with change but I would actually refactor whole thing so each of this function is standalone04:00
SamYapleim sorry i dont think you understand04:01
inc0one of us don't04:01
SamYapleyou need to split up start.yml04:01
SamYaplethen there is no duplicated start code04:01
SamYapleno DRY violation04:01
inc0if you can't do simple import, there have to be04:02
inc0and in this case I don't think you can04:02
SamYaplewhat are you talking about?04:02
SamYapleimport?04:02
inc0include start.yml in upgrade plays04:02
SamYapleno i said splut up start.yml04:03
SamYapleas in start_conductor.yml04:03
SamYaplein the start play you include start_conductor.yml04:03
SamYaplesame with upgrade play04:03
SamYaplewe talked about doign that _after_ the upgrade play so as not to slow down upgrades, but if you want to do it now im fine with that04:03
inc0but that won't help with case at hand04:03
SamYapleit will absolutely04:04
inc0because if you change mounts in start_conductor.yml04:04
SamYaplethen there is no chance t odiffer04:04
inc0it won't affect check task04:04
SamYaple...... if you change the mounts in start_conductor.yml you are going to be starting a new container anyway04:04
inc0so if you forget to make same change in check task04:04
*** rhallisey has quit IRC04:04
inc0wonderful, but then if we forget to make this change there04:04
SamYaplethere is no mounts in check task04:05
inc0each time you run upgrade task it will stop04:05
SamYaplewhere are you getting this info04:05
inc0not-idempotently04:05
SamYapleyou are absolutely wrong04:05
SamYaple100%04:05
*** yuanying has joined #kolla04:05
*** alger has joined #kolla04:07
*** yuanying_ has joined #kolla04:07
SamYapleoh man inc004:07
SamYapleim so sorry04:07
SamYapleoh man i feel so bad04:07
inc0https://github.com/openstack/kolla/blob/master/ansible/library/kolla_docker.py#L246 we change env and forget to do it in check task == stop every time04:07
SamYaplei see what you mean04:07
SamYapledude i was in the wrong way04:08
SamYaplejeez i get what you are saying04:08
inc0yeah, I thought you might04:08
SamYaplei was focusing on the start04:08
inc0I did start from check_container_differs04:08
*** Marga_ has joined #kolla04:08
SamYaplei was thinking you were talking about the starting of the container04:08
SamYaplei was so confused04:08
inc0that's most obvious way, but it didnt work as intended04:08
inc0separation start task is acutally good idea, I'll do that04:09
SamYapleok so here is somethign we can actually do04:09
inc0(tomorrow)04:09
SamYapleive built kolla_docker to support the check mode04:09
SamYaplewhat we can do is set checkmode on the tasks and registry if it changes04:09
*** yuanying has quit IRC04:09
*** Marga_ has quit IRC04:10
inc0how does it work?04:10
SamYaplecheck mode?04:10
inc0yeah04:10
inc0low level04:10
SamYapleit makes no changes04:10
inc0because method for stopping is single flow04:10
SamYapleit just reports if there was achange04:10
*** Marga_ has joined #kolla04:10
SamYaplebut lets table that04:11
SamYaplethats different work04:11
SamYaplelets not hold up upgrades for that work04:11
inc0https://review.openstack.org/#/c/254395/16/ansible/library/kolla_docker.py line 50804:11
inc0how does it know which line is changing stuff04:11
inc0?04:11
inc0or I have to write it in different way to support it?04:12
SamYapleinc0: the module needs a bit of tweaking for that thats what i said tabling it04:12
inc0ok04:12
inc0so for now we're good to keep logic like I written it?04:12
SamYaplei mean it just needs a check and exit call for each "changed = true" flag, but lets not do it now04:12
inc0I can change underscore naming and separate start.yml to increase dry04:12
SamYaplei dont like the underscore renaming04:12
SamYaplebut thats it04:13
SamYapledude i feel so bad right now.04:13
*** alger has quit IRC04:13
inc0ok, I'll figure it out tomorrow04:13
SamYaplei had thirty things going on in my head04:13
inc0don't worry too much04:13
SamYaplei should have listened more04:13
SamYaplesorry man04:13
*** alger has joined #kolla04:13
inc0I'm glad we got that cleared out, that's all04:13
SamYapleyea. well heres to moving forward04:14
SamYapleinc0: alternatively you could just add a "check_image_only" flag to check_contaier_differs04:16
inc0that will complicate whole thing04:16
SamYaplethat would be no DRY violation, and no function renaming04:16
SamYaplehow so?04:16
inc0I'd rather refactor all checks to either accept container_info or if not provided get container by themselves04:17
inc0so in effect all of them will be standalone04:17
SamYapleyea but they wont all be useable04:17
SamYaplesince you have to update the acceptable choices04:18
inc0yeah, other will just be to create a standard04:18
SamYapleyea thats true04:18
inc0I can do this with just an image04:18
inc0might be actually good for claridy04:18
inc0clarity04:18
inc0since it is somewhat special04:19
SamYapleeach function starts with 'container = container_info or self.check_container()04:19
inc0yeah04:19
SamYapleand function accepts (self, container_info=None)04:19
SamYapleyea that works04:19
*** iceyao has quit IRC04:19
SamYaplei like that04:19
inc0I wanted to refactor it this way anyway04:19
inc0will do that tomorrow04:19
SamYaplecool04:19
*** iceyao has joined #kolla04:19
inc0would you plz do me a favor and comment patchset so others will see that we agreed on this one?04:20
inc0and add note for splitting start.yml plz so it wont skip my mind04:20
SamYapleLET THE RECORD SHOW MICHAL AND SAM HAVE REACHED AN AGREEMENT04:20
inc0we should clad it in stone or something04:20
SamYaplelol we always come to an agreement eventually04:20
inc0fund a bench in waterfall park to remember it04:20
SamYapleill carve it at table rock04:21
SamYapleinc0: ok commented04:25
*** tzn has joined #kolla04:26
inc0thanks04:26
inc0these should be fairly easy04:26
SamYapleyup04:26
inc0I'll ping you later to discuss best approach for keepalived upgrades04:28
*** iceyao has quit IRC04:28
SamYaplei think those need full stop too but i cant remember. i looked it up at one point04:28
SamYaplebut vip can stick around, so no worries04:28
inc0well if you stop keepalived container it will migrate04:29
inc0can you force it not to?04:29
SamYaplenah you stop the non-active one04:29
SamYaplethen you stop the active and VIP will stick around04:29
SamYaplewell you need -V on the daemon i think...04:29
inc0ahh...that might work04:29
*** iceyao has joined #kolla04:29
SamYaplebut we can do a rolling way so it only changes at max once04:29
inc0we need to figure out best way to find currently active04:29
inc0might not be trivial04:30
inc0I'd rather do full stop without change04:30
SamYaplei mean ansible knows04:30
*** tzn has quit IRC04:30
SamYapleit has all ip info04:30
inc0hmm...yeah that might work, if we do it correctly04:30
inc0I'll play around it04:31
SamYaplewhen [api_interface] == {{ kolla_internal_address }}04:31
SamYaplesomething like that04:31
SamYaplei mean obviously thats wrong, but you get it04:31
inc0I get it, hacky but might work04:31
inc0hacky is ok for this release04:31
inc0then we'll see if it needs changing04:31
SamYapleidk seems pretty clean to me04:31
SamYaplebut anyway04:31
SamYaplefor tomorrow04:31
inc0yeah04:31
inc0I'm going off, cyas04:31
SamYaplebye04:32
*** inc0 has quit IRC04:32
*** RA__ has quit IRC04:37
*** vhosakot has joined #kolla04:41
*** alger has quit IRC04:49
*** alger has joined #kolla04:52
*** dims_ has joined #kolla04:53
*** dims has quit IRC04:55
*** dims_ has quit IRC05:00
*** dims has joined #kolla05:02
*** dims has quit IRC05:02
mdnadeemSamYaple, ping05:03
*** ccesario has quit IRC05:05
mdnadeemhad any one face issue with TASK: [neutron | Starting neutron-dhcp-agent container] while deploying kolla on ubuntu os ?05:10
mdnadeemPlease have a look on http://pastebin.com/bMgnMz0405:10
mdnadeem^^ jeffrey4l , nihilifer05:12
*** mdnadeem_web has joined #kolla05:15
*** Allen_Gao has quit IRC05:16
*** ccesario has joined #kolla05:18
*** ccesario has quit IRC05:26
*** tzn has joined #kolla05:26
*** tzn has quit IRC05:31
openstackgerritMartin André proposed openstack/kolla: Set nova hypervisor to qemu in Vagrant  https://review.openstack.org/28163705:32
*** Allen_Gao has joined #kolla05:33
*** dave-mccowan has quit IRC05:38
*** ccesario has joined #kolla05:39
*** iceyao has quit IRC05:45
*** iceyao has joined #kolla05:45
mdnadeemoh now i see , i have hit this bug https://bugs.launchpad.net/kolla/+bug/154679805:54
openstackLaunchpad bug 1546798 in kolla "neutron failed to deploy in ubuntu binary" [Low,Triaged] - Assigned to Sam Yaple (s8m)05:54
*** Marga_ has quit IRC06:01
*** aginwala has joined #kolla06:24
*** Marga_ has joined #kolla06:25
*** tzn has joined #kolla06:27
*** tzn has quit IRC06:32
*** vhosakot has quit IRC06:34
*** iceyao has quit IRC06:38
*** iceyao has joined #kolla06:39
*** vhosakot has joined #kolla06:41
openstackgerritJeffrey Zhang proposed openstack/kolla: also remove the volumes when cleaning up containers  https://review.openstack.org/28072206:53
*** Serlex has joined #kolla07:03
*** vhosakot has quit IRC07:14
*** achanda has quit IRC07:14
*** achanda has joined #kolla07:17
*** chandankumar has joined #kolla07:22
*** intr1nsic has quit IRC07:25
*** iceyao has quit IRC07:26
*** iceyao has joined #kolla07:27
*** alisonh has quit IRC07:27
*** intr1nsic has joined #kolla07:28
*** iceyao_ has joined #kolla07:28
*** tzn has joined #kolla07:28
*** iceyao has quit IRC07:31
*** tzn has quit IRC07:32
*** iceyao_ has quit IRC07:39
*** alisonh has joined #kolla07:42
openstackgerritEric Lemoine proposed openstack/kolla: Fix typo in quickstart  https://review.openstack.org/28168107:43
*** salv-orlando has joined #kolla07:44
openstackgerritJeffrey Zhang proposed openstack/kolla: also remove the volumes when cleaning up containers  https://review.openstack.org/28072207:45
*** iceyao has joined #kolla07:48
openstackgerritJeffrey Zhang proposed openstack/kolla: Ensuring the rsyslog service to be ready  https://review.openstack.org/27906407:50
*** alisonh has quit IRC07:50
*** jtriley has quit IRC07:54
*** alisonh has joined #kolla07:56
*** alger has quit IRC07:56
*** alger has joined #kolla08:00
*** dtturner has quit IRC08:02
*** iceyao_ has joined #kolla08:02
Jeffrey4l elemoine do you have any idea why the link is not create when using `sudo ln -sf /var/lib/kolla/rsyslog/log /dev/log`  in Sam's PS,now.08:04
*** iceyao has quit IRC08:05
*** jmccarthy1 has joined #kolla08:06
openstackgerritEric Lemoine proposed openstack/kolla: Also remove the volumes when cleaning up containers  https://review.openstack.org/28072208:07
*** Marga_ has quit IRC08:07
*** fgimenez has joined #kolla08:10
*** jtriley has joined #kolla08:12
SerlexAnyone seen08:18
SerlexERROR:kolla.cmd.build:Error: Status 502 trying to push repository lokolla/centos-binary-haproxy:08:18
*** mdnadeem_web has quit IRC08:20
*** mikelk has joined #kolla08:21
*** tzn has joined #kolla08:29
*** mikelk has quit IRC08:30
*** mikelk has joined #kolla08:31
*** tzn has quit IRC08:33
*** Jeffrey4l has quit IRC08:45
*** alger has quit IRC08:46
*** alger has joined #kolla08:46
*** Jeffrey4l has joined #kolla08:58
*** akwasnie has joined #kolla09:02
*** alger has quit IRC09:02
*** alger has joined #kolla09:03
*** tzn has joined #kolla09:04
*** macsz has joined #kolla09:05
*** gfidente has joined #kolla09:10
*** tzn has quit IRC09:10
*** mbound has joined #kolla09:12
*** alger has quit IRC09:21
*** achanda has quit IRC09:25
*** Allen_Ga_ has joined #kolla09:28
*** achanda has joined #kolla09:28
*** achanda has quit IRC09:29
*** Allen_Ga_ has quit IRC09:29
*** aginwala has quit IRC09:33
*** alger has joined #kolla09:33
*** tpot has quit IRC09:35
*** achanda has joined #kolla09:46
akwasnieelemoine: ping09:47
akwasnieelemoine, sdake, inc0 : https://www.elastic.co/blog/heya-elastic-stack-and-x-pack09:48
*** tfukushima has quit IRC09:49
akwasniegood to know that there are some changes coming soon, hope it won't be a problem for our logging solution09:50
elemoine_akwasnie:  hie09:50
elemoine_thanks for the link09:52
*** tfukushima has joined #kolla09:54
*** tfukushima has quit IRC09:54
*** pbourke has joined #kolla09:55
nihiliferbtw, can the other cores look at this? https://review.openstack.org/#/c/267714/09:55
nihiliferit'd be nice to merge it soon09:55
nihilifer(to be honest, i don't get why we need "more eyes" and more than 2 cores on this patch, but ok... ;) )09:58
akwasnienihilifer hi:)10:00
akwasnieand thanks10:01
nihilifernp akwasnie ;)10:01
*** fgimenez has quit IRC10:02
*** openstackgerrit has quit IRC10:02
akwasnienihilifer are you going to attend summit?10:03
*** openstackgerrit has joined #kolla10:03
*** salv-orl_ has joined #kolla10:03
nihiliferakwasnie: probably yes10:05
*** salv-orlando has quit IRC10:06
akwasniegreat;)10:06
*** fgimenez has joined #kolla10:06
*** pbourke has quit IRC10:10
*** pbourke has joined #kolla10:11
*** achanda has quit IRC10:12
*** achanda has joined #kolla10:14
*** achanda has quit IRC10:16
*** Jeffrey4l has quit IRC10:25
*** tzn has joined #kolla10:31
*** kproskurin has joined #kolla10:36
*** Jeffrey4l has joined #kolla10:42
*** Allen_ has joined #kolla10:54
*** Allen_ is now known as Allen_Gao_10:54
*** Allen_Gao_ is now known as AllenGao10:56
*** dims has joined #kolla10:59
*** Allen_Gao has quit IRC10:59
*** asalkeld has quit IRC11:00
*** AllenGao is now known as Allen_Gao11:02
*** athomas has quit IRC11:09
*** tobe has joined #kolla11:11
*** tfukushima has joined #kolla11:11
*** athomas has joined #kolla11:14
*** athomas has quit IRC11:16
*** achanda has joined #kolla11:17
*** pbourke has quit IRC11:18
*** athomas has joined #kolla11:18
*** alger has quit IRC11:19
*** Jeffrey4l has quit IRC11:19
*** iceyao_ has quit IRC11:21
*** mdnadeem has quit IRC11:22
*** tfukushima has quit IRC11:23
*** achanda has quit IRC11:24
*** Jeffrey4l has joined #kolla11:29
*** dims_ has joined #kolla11:30
*** dims has quit IRC11:31
*** fgimenez has quit IRC11:32
*** fgimenez has joined #kolla11:32
limamauricioTASK: [common | Starting rsyslog container] *********************************** failed: [localhost] => {"changed": true, "failed": true} msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/rsyslog/start',),)11:34
limamauricioSometimes this error happens, but I do not know why11:34
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ  https://review.openstack.org/27666811:36
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder infrastructure  https://review.openstack.org/27666711:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Murano logs  https://review.openstack.org/28132811:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Magnum logs  https://review.openstack.org/28132911:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs  https://review.openstack.org/27584411:36
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack  https://review.openstack.org/27584511:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Nova logs  https://review.openstack.org/27584611:36
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB  https://review.openstack.org/27584711:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Heat logs  https://review.openstack.org/28132711:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Cinder logs  https://review.openstack.org/28125911:36
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for Keystone/Apache  https://review.openstack.org/27919211:36
openstackgerritEric Lemoine proposed openstack/kolla: Add a Dockerfile for Heka  https://review.openstack.org/27584011:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Keystone logs  https://review.openstack.org/27919311:36
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka to common role  https://review.openstack.org/27584111:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect HAProxy and Keepalived logs  https://review.openstack.org/27584211:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Swift logs  https://review.openstack.org/27949011:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Neutron logs  https://review.openstack.org/27948911:36
openstackgerritEric Lemoine proposed openstack/kolla: Make Ansible write its logs into "kolla_logs"  https://review.openstack.org/27919111:36
*** Allen_Gao has quit IRC11:39
*** jmccarthy has quit IRC11:40
limamauricioTASK: [common | Starting rsyslog container] *********************************** failed: [localhost] => {"changed": true, "failed": true} msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/rsyslog/start',),)11:41
limamauricioSometimes this error happens, but I do not know why11:41
*** jmccarthy has joined #kolla11:42
*** Allen_Gao has joined #kolla11:42
*** mdnadeem has joined #kolla11:43
*** pbourke has joined #kolla11:49
*** Jeffrey4l__ has joined #kolla11:50
*** Jeffrey4l has quit IRC11:52
*** bmace has quit IRC11:53
*** pbourke has quit IRC11:55
*** bmace has joined #kolla11:56
*** salv-orl_ has quit IRC12:03
*** Allen_Ga_ has joined #kolla12:04
*** Allen_Gao has quit IRC12:08
*** Jeffrey4l_ has quit IRC12:09
*** rhallisey has joined #kolla12:10
*** Allen_Ga_ has quit IRC12:18
*** Allen_Gao has joined #kolla12:21
*** salv-orlando has joined #kolla12:26
*** dave-mccowan has joined #kolla12:26
*** Allen_Gao has quit IRC12:26
*** Allen_Gao has joined #kolla12:27
*** JoseMello has joined #kolla12:28
*** Allen_Gao has quit IRC12:29
*** stvnoyes has quit IRC12:33
*** Allen_Gao has joined #kolla12:33
*** Allen_Gao has quit IRC12:33
*** stvnoyes has joined #kolla12:33
*** Allen_Gao has joined #kolla12:35
*** iceyao has joined #kolla12:37
*** fgimenez has quit IRC12:38
*** Allen_Gao has quit IRC12:39
*** fgimenez has joined #kolla12:41
*** fgimenez has quit IRC12:41
*** fgimenez has joined #kolla12:41
*** dwalsh has joined #kolla12:42
*** pbourke has joined #kolla12:43
*** Allen_Gao has joined #kolla12:46
*** pbourke_ has joined #kolla12:46
*** pbourke has quit IRC12:48
*** limamauricio has quit IRC13:02
*** limamauricio has joined #kolla13:02
*** liyi has joined #kolla13:12
liyianyone know if there is a "GOOD" commit that I could deploy multiple nodes?13:13
*** thumpba has joined #kolla13:15
*** chandankumar has quit IRC13:16
pbourke_liyi: master13:17
limamauriciopbourke_ kolla not works with ansible 2.0.0 right?13:19
pbourke_limamauricio: correct13:19
limamauriciook13:19
limamauriciopbourke_ msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/rsyslog/start',),)13:20
limamauricioansible 1.9.4 docker 1.10.113:20
pbourke_docker-py version?13:21
limamauricio1.713:21
*** achanda has joined #kolla13:22
pbourke_perhaps check journalctl and see what docker is complaining about13:22
liyii have the same problem like limamauricio,13:22
liyion master brancher I mean13:22
limamauriciome too liyi13:22
liyidocker start fc0b7baaede2 Error response from daemon: Cannot start container fc0b7baaede2684c834e5997700cd0623b45d800f554d95e6c63996e53a3286e: Path /run/netns is mounted on /run but it is not a shared mount. Error: failed to start containers: fc0b7baaede213:23
*** chandankumar has joined #kolla13:23
liyithis is what I got from docker log if I try to manually start kollaglue/ubuntu-binary-neutron-dhcp-agent:2.0.013:24
limamauricioMountFlags=shared13:24
limamauricioset it13:24
elemoine_pbourke_: are you sure kolla support ansible 2?13:25
pbourke_elemoine_: kolla does *not* support ansible 213:25
liyiI have ubuntu, have tired: mount --make-shared /run13:25
elemoine_I misread13:25
elemoine_sorry13:25
liyimake no difference13:25
limamauricio:/13:25
pbourke_liyi: can you run any docker containers? (docker run -it ubuntu ls)13:26
elemoine_limamauricio wrote "kolla not works with ansible 2.0.0" which  I read as "kolla now works…"13:26
pbourke_elemoine_: :)13:26
limamauriciohow i check journalctl?13:27
*** achanda has quit IRC13:27
liyiyes, other container runs nicely13:27
liyibut when it try to start this particular container, it failed :(13:28
pbourke_actually, this could be a problem with the new thin neutron containers13:28
pbourke_so others start up, just not neutron?13:28
pbourke_s/so/do13:28
liyijust not kollaglue/ubuntu-binary-neutron-dhcp-agent:2.0.0, others like kollaglue/ubuntu-binary-neutron-openvswitch-agent:2.0.0 and kollaglue/ubuntu-binary-neutron-server:2.0.0  works OK13:29
limamauriciopbourke_ i should set docker_insecure_registry: "True" and  docker_pull_policy: "missing" in globals?13:30
pbourke_limamauricio: no Im pretty sure this is a bug13:30
pbourke_limamauricio: would you mind logging something in launchpad? till then you may like to try the stable/liberty branch13:31
pbourke_SamYaple: when you're around if you see this ping me13:31
limamauricioi'm in master13:31
limamauriciomaster/centos/source13:31
pbourke_limamauricio: git checkout stable/liberty13:31
limamauricioi will not do that because the master deploy was working yesterday13:32
limamauriciobut i rebuild and now i have an error13:32
limamauricio:/13:32
pbourke_:/13:33
limamauriciowhat I did differently was stop and disable firewalld.service13:33
pbourke_will check logs and see if anything suspect merged since yesterday13:33
pbourke_yeah it may be something changed on your host that the namespace propagation feature doesn't like13:34
limamauriciobefore I had not disabled it13:34
limamauriciobut i have to disable firewalld.service rigth?13:35
limamauricioright*13:35
*** Liuqing has joined #kolla13:38
openstackgerritMerged openstack/kolla-mesos: Add memcached dep to horizon  https://review.openstack.org/28123613:40
openstackgerritMerged openstack/kolla: Fix typo in quickstart  https://review.openstack.org/28168113:41
*** sdake has joined #kolla13:41
liyi@pbourke, anything else could I try?13:41
openstackgerritMerged openstack/kolla: Make Mistral deployable via ansible  https://review.openstack.org/28161013:42
*** reddy has joined #kolla13:44
pbourke_liyi: unfortunately I dont have any other suggestions right now. I haven't looked at the thin neutron containers yet13:45
pbourke_liyi: when SamYaple is online in an hour or two he will likely have suggestions13:45
liyi@thanks a lot Pbourke!13:46
liyibtw, I have this mount output on my node13:46
liyisudo mount | grep run none on /run type tmpfs (rw,relatime) nsfs on /run/docker/netns/default type nsfs (rw) none on /run type tmpfs (rw,relatime) nsfs on /run/docker/netns/default type nsfs (rw)13:46
liyiif everyone else has a hint, please help13:47
*** Allen_Gao has quit IRC13:47
SerlexI thought the issue was resolved with shared mount13:48
Serlexmake sure you've reloaded and restarted the service13:48
limamauricioSerlex, msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/rsyslog/start',),)13:48
limamauricioyou known this error?13:48
liyi@Serlex, I did restart docker serivce, but not the OS.13:49
pbourke_probably, its not related to your issue though13:49
limamauricioknow*13:49
pbourke_ignore above message, wrong window13:49
SerlexI was referring to neutron-agent problem13:49
Serlexmy bad13:49
pbourke_liyi: limamauricio: I was about to suggest something similar to serlex, perhaps try removing all running containers and then restarting docker13:50
pbourke_bit dumb but it may reset the problematic mount13:50
limamauriciook13:51
liyithanks, I will give it a try.13:51
fthiagogvGuys, do someone know about this error msg? msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/kolla_toolbox/start',),)13:52
liyiwill report back later. I have 3 controller nodes, will take a while for doing so13:52
liyi@fthiagogv it is hot issue now13:53
liyiif u have not set MountFlags, might give it a try first. hope u luckier than me13:53
pbourke_fthiagogv: please check journalctl for docker error msgs13:54
rhalliseyfthiagogv, have you updated you repo recently?13:58
rhalliseyfthiagogv, there was a recent commit that fixed that13:59
rhalliseyI would update your repo and rebuild kolla-toolbox13:59
pbourke_liyi: are you 100% sure the mountflags are getting picked up by the daemon14:00
fthiagogvI think so, but I will update again and try.14:00
limamauriciorhallisey msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/rsyslog/start',),)14:01
limamauricio:/14:01
liyi@pbourke i don't know how to verify it, do u know how?14:01
limamauricioyesterday was working, but today...14:01
pbourke_liyi: trying to find that now. I just ask as I have found docker fiddly to configure14:02
*** Liuqing has quit IRC14:02
rhalliseyliyi, systemctl daemon-reload14:02
rhalliseysystemctl restart docker14:02
rhalliseylimamauricio, I haven't run into that.. Will rebuild and deploy and get back to you14:02
liyii run ubuntu, but did try something similar14:03
rhalliseyare you in 14.04?14:03
liyiyes14:03
rhalliseylook in /proc/self/mountinfo14:04
rhalliseyI think you can grep for 'shared' in there14:04
liyi349 335 0:17 / /run rw,relatime shared:1 - tmpfs none rw 285 349 0:4 / /run/docker/netns/default rw shared:2 - nsfs nsfs rw14:05
*** Allen_Gao has joined #kolla14:05
liyihere is what I got when grep shared in the file14:05
Serlexcan't you do something like this sudo systemctl show docker | grep Mount14:06
rhalliseyubuntu 14.04 uses upstart14:06
*** fgimenez has quit IRC14:07
pbourke_what file are you setting the mountflags liyi14:07
Serlexstill upstart must have a way no showing what the service is running with14:07
ajafoin ubuntu dont set shared in file it need to make mount command on host14:07
liyiI run "mount --make-shared /run" as suggested in the doc14:08
*** fgimenez has joined #kolla14:08
*** fgimenez has quit IRC14:08
*** fgimenez has joined #kolla14:08
ajafo mount --make-shared /run14:08
liyiyep, there is what i did14:08
liyino help14:08
liyidelete all containers and re-deploy14:09
liyiendup with errors14:09
liyiTASK: [mariadb | Waiting for MariaDB service to be ready] *********************  failed: [k-blade-1] => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "ls", "/var/lib/mysql/mariadb.pid"], "delta": "0:00:00.212851", "end": "2016-02-18 13:57:24.275956", "failed": true, "rc": 2, "start": "2016-02-18 13:57:24.063105", "stdout_lines": [], "warnings": []}14:09
*** thumpba has quit IRC14:09
*** dave-mccowan has quit IRC14:09
rhalliseyI'm wondering if there needs to be some ordering or an addition step here14:09
rhalliseyI don't run ubuntu.  SamYaple would be able to answer this14:10
*** dave-mccowan has joined #kolla14:10
liyiOK, I wait for Sam then14:10
sdakeSamYaple ping me when you get in plz14:11
sdakellyi we have seen that report 3 separate times14:11
sdakeplease file a bug14:11
sdakei think it has to do with selinux possibly14:11
sdakebut i am not certain14:11
sdakeliyi just file a bug based upon what you know14:12
sdakeliyi that fact that people keep running into it is a problem, but bug reports over irc arenot  helpful because they get lost ;)14:12
*** pbourke_ has quit IRC14:13
*** pbourke has joined #kolla14:13
liyithanks for suggestion sdake. I now clean up all nodes and re-deploy from re-install os14:13
liyiwill try to get a report after the new test14:13
sdakejust report what you got ;)14:14
sdakeif you clean up your environment it may never happen again14:14
ajafolimamauricio: I had this error until get https://review.openstack.org/#/c/279064/14:14
liyia little too late, already wipe off logs :)14:14
sdakei see14:14
sdakewell we cna't fix things that aren't in bug tracker ;)14:15
limamauricioajafo, sometimes i have this erro too msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/kolla_toolbox/start',),)14:16
ajafolimamauricio: as I wrote I had this error before apply patch from link, but I see it's abandoned14:16
*** britthou_ has joined #kolla14:17
limamauricioajafo14:18
limamauricioyou use centos?14:18
*** pbourke has quit IRC14:18
ajafoI get this error on centos vagrant machine and I run there ubuntu images14:19
*** britthouser has quit IRC14:19
limamauricioajafo, u disabled firewalld.service?14:20
Allen_Gaolimamauricio: it's abandoned because the rsyslog is moved to named_volume https://review.openstack.org/#/c/28042914:20
limamauricioI'm doing everything from scratch again14:20
limamauriciocentos/source14:20
*** pbourke has joined #kolla14:21
limamauricioand i following this guide http://docs.openstack.org/developer/kolla/quickstart.html14:22
limamauriciostep-by-step14:22
limamauricio:)14:22
limamauricioI saw several different ways to create a registry14:23
Serlexwhy don't you do binary?14:23
limamauricioIt should work both, right?14:23
limamauricioExecStart=/usr/bin/docker daemon -H fd:// $other_args or ExecStart=/usr/bin/docker daemon -H fd:// --insecure-registry localhost:400014:24
limamauricioI'll try binary later14:25
*** jtriley has quit IRC14:25
*** diogogmt has quit IRC14:26
*** diogogmt has joined #kolla14:28
Allen_Gaosdake: did you see this bug https://review.openstack.org/#/c/281159/ ?14:33
*** jtriley has joined #kolla14:33
*** Marga_ has joined #kolla14:40
*** kproskurin has quit IRC14:41
*** kproskurin has joined #kolla14:42
*** pbourke has quit IRC14:44
*** Allen_Ga_ has joined #kolla14:45
*** pbourke has joined #kolla14:45
*** Marga__ has joined #kolla14:46
*** Allen_Gao has quit IRC14:46
*** Allen_Gao has joined #kolla14:46
*** Allen_Ga_ has quit IRC14:47
sdakeAllen_Gao comment left in bug tracker14:47
sdakei'll leave one in the review14:47
*** dave-mccowan has quit IRC14:48
*** Marga_ has quit IRC14:49
Allen_Gaosdake: I see your comment in traker, thank you for your quick response, then the "rpm -e " is the right fix? I can't understand why Sam -2 on that ?14:51
*** tobe has quit IRC14:57
SamYaplesdake: ping14:58
SamYaplepbourke: ping14:58
pbourkeSamYaple: hey14:58
SamYaplewhats up14:58
pbourkeSamYaple: my connection has been on and off today so I may have missed some messages. but basically limamauricio and liyi are both having the same issue with thin neutron on ubuntu14:58
*** diogogmt has quit IRC14:58
SamYapleis /run a shared mountpoint?14:59
pbourkeSamYaple: ill see if I can find the error, basically they have run the mount command as in docs but docker still complains14:59
SamYapleill need more info than that. but just fyi our gate is ubuntu 14.04 and i run 15.10 so all the ubuntu bases are covered15:00
pbourkedocker start fc0b7baaede2 Error response from daemon: Cannot start container fc0b7baaede2684c834e5997700cd0623b45d800f554d95e6c63996e53a3286e: Path /run/netns is mounted on /run but it is not a shared mount. Error: failed to start containers: fc0b7baaede215:00
pbourkejust wondering if that rings any bells15:00
SamYapleit sounds like /run is not a shared mountpoint15:00
pbourkeyeah15:00
*** tzn has quit IRC15:01
pbourkeok I dont have a whole pile more info, maybe the guys will reappear and let us know if its still an issue15:01
SamYapleif you do run this:15:01
SamYapleawk '$5 == "/run" {print $7}' /proc/self/mountinfo15:01
SamYapleif that doesnt say shared, its not shared15:02
liyihi Sam, nice to see u here15:02
*** dave-mccowan has joined #kolla15:02
liyiI just post my errior into https://bugs.launchpad.net/kolla/+bug/154679815:02
openstackLaunchpad bug 1546798 in kolla "neutron failed to deploy in ubuntu binary" [Low,Triaged] - Assigned to Sam Yaple (s8m)15:02
openstackgerritMerged openstack/kolla: Also remove the volumes when cleaning up containers  https://review.openstack.org/28072215:03
SamYapleliyi: did you remove the neutron container after you got tht message?15:03
*** tzn has joined #kolla15:03
liyiif running your magic command, i got this:15:03
liyiawk '$5 == "/run" {print $7}' /proc/self/mountinfo - shared:115:03
liyii actual wipe off everything, include OS, and re-deploy from scratch.15:04
liyigot that error15:04
liyii mean the error I just reported in https://bugs.launchpad.net/kolla/+bug/154679815:05
openstackLaunchpad bug 1546798 in kolla "neutron failed to deploy in ubuntu binary" [Low,Triaged] - Assigned to Sam Yaple (s8m)15:05
*** sdake has quit IRC15:05
SamYapleliyi: you ran that command on k-blade-1? or on the host where you run ansible?15:06
liyiyou mean your magic command, I run it on k-blade-115:06
liyiliyi@k-blade-1:~$ awk '$5 == "/run" {print $7}' /proc/self/mountinfo - shared:115:07
*** sdake has joined #kolla15:08
*** tzn has quit IRC15:08
limamauricioi'm not using ubuntu15:08
*** Allen_Gao has quit IRC15:08
SamYapleso it looks like you aren't removing your neutron-dhcp-agent container between tests15:08
SamYapleif you get that errror, correct the mountpoint to shared15:08
SamYaplethen docker rm -v -f neutron_dhcp_agent15:08
SamYapleand run the playbooks again15:08
liyiSam, I wipe off everything my disk and re-install the system then test15:09
liyii am very sure everything is gone before I deploy.15:09
SamYaplebut did you remember to make the mountpoint shared before you ran the playbooks again15:10
liyiyes i do15:10
SamYapleim just not sure how to replicate your issue then. I haven't seen that. The gate doesnt see that. Cna you give me steps to replicate it?15:11
SamYaplethe only time ive seen this is when someone forgot something15:11
liyii make a simple playbook and run "sudo mount --make-shared /run && sudo service restart docker"15:11
*** britthouser has joined #kolla15:12
liyiafter that, i run the deployment as usual15:12
*** britthou_ has quit IRC15:12
openstackgerritMerged openstack/kolla: Fix the incorrectly removed dependent packages  https://review.openstack.org/28115915:13
openstackgerritMerged openstack/kolla: Add double quotes in kolla_enable_sanity_checks  https://review.openstack.org/28182315:13
liyiHere is what I did: re-format my disk -> install new OS -> run  "sudo mount --make-shared /run && sudo service restart docker" on each node -> run kolla playbook15:14
SamYapleliyi: you could attempt to create a seperrate mountpoint for /run/netns15:14
SamYapleits just tmpfs anyway15:14
liyihmm, on my k-blade-1, I don't see /run/netns15:15
liyiI have this k-blade-1:~$ sudo mount | grep run none on /run type tmpfs (rw,relatime) nsfs on /run/docker/netns/default type nsfs (rw) none on /run type tmpfs (rw,relatime) nsfs on /run/docker/netns/default type nsfs (rw)15:15
SamYapleliyi: you wont see it until its used15:16
liyi none on /run type tmpfs (rw,relatime)15:16
liyinsfs on /run/docker/netns/default type nsfs(rw)15:16
liyiOK15:16
SamYaplemkdir /run/netns && mount -o rw,relatime,shared -t tmpfs tmpfs /run/netns15:17
liyimkdir: cannot create directory '/run/netns': File exists15:17
liyisudo mkdir -p /run/netns && sudo mount -o rw,relatime,shared -t tmpfs tmpfs /run/netns mount: wrong fs type, bad option, bad superblock on tmpfs,        missing codepage or helper program, or other error        (for several filesystems (e.g. nfs, cifs) you might        need a /sbin/mount.<type> helper program)        In some cases useful info is found in syslog - try        dmesg | tail  or so15:18
*** spisarski has joined #kolla15:18
SamYapleyou said you didnt see /run/netns15:18
liyii don't see it when in the output of mount15:18
SamYapleok docker just creates a folder there15:19
liyiwhen it is actually in the filesystem, strange15:19
SamYaplefeel free to remove it15:19
SamYapleits not a mountpoint15:19
SamYapleim saying you can turn it into one15:19
liyimount: wrong fs type, bad option, bad superblock on tmpfs,        missing codepage or helper program, or other error        (for several filesystems (e.g. nfs, cifs) you might        need a /sbin/mount.<type> helper program)        In some cases useful info is found in syslog - try        dmesg | tail  or so15:19
liyiwhen "sudo mkdir /run/netns && sudo mount -o rw,relatime,shared -t tmpfs tmpfs /run/netns"15:20
liyiI got that error, it seems faile to mount15:21
*** iceyao has quit IRC15:21
SamYapleliyi: can you post the output of `uname -a; cat /etc/issue`15:21
liyiit seem my system dose not support shared flag15:22
liyiafter I remove the shared flag, it works15:22
*** pbourke has quit IRC15:22
liyiLinux k-blade-1 4.2.8-ubuntu #1 SMP Wed Feb 3 22:19:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux15:24
liyiUbuntu \n \l docker-sys: 172.18.42.1 eth0: 10.1.1.11 lo: 127.0.0.115:24
*** inc0 has joined #kolla15:25
*** achanda has joined #kolla15:25
SamYapleliyi: im not sure what to make of that. what distro and version is on k-blade-1?15:26
inc0good morning15:26
akwasniehi inc015:26
inc0good morning Ala, long time no see:)15:26
liyii build my old kernel and pull down ubuntu 14.04 as root fs15:27
liyii build my own kernel and pull down ubuntu 14.04 as root fs15:27
liyifor some reason, the shared flag is not supported in my system.15:27
akwasnieinc0: yes, quite long...what's up in SA? Poland better?:D15:28
inc0nope, I like these winters15:28
liyiI will try to have a look into. Thanks for the help Sam!15:28
*** pbourke has joined #kolla15:28
inc0today is 25C15:28
akwasnie'winters'15:28
SamYapleI would recommend testing a vanilla install of ubuntu liyi. see if the same issue occurs15:28
SamYaple'25C'15:28
akwasniexD15:29
inc0I should move to Kelvin degrees15:29
SamYapleKelvin is more accurate15:30
SamYapleI would support a switch15:30
*** achanda has quit IRC15:30
inc0it's just as acurate as celsous15:30
inc0just with offset15:30
akwasniein Poland we have 3 C...so it would be sth like 276 K?15:31
*** tzn has joined #kolla15:31
akwasniein Poland I mean in my city...Poland is not so small that we have only one temperature... ;p15:32
liyiSam, I was expect the different kernel should not be an issue. Maybe not the case as it turns out :(15:32
SamYapleinc0: F is just as accurate as C with a different offset!15:32
inc0it is not SamYaple, 1 degree C != 1 degree F15:33
inc0but truth be told, it's slightly more accurate15:33
SamYaplei got my hard drives in15:34
inc0cool15:36
*** chandankumar has quit IRC15:36
SamYapleelemoine_: looking good, correct a small issue in common role patch and a full rebase and i think we are ready to merge!15:40
*** tobe has joined #kolla15:40
elemoine_SamYaple: I had the detach: false change locally already :)15:41
SamYapleelemoine_: oh im suuuuure you did!15:41
elemoine_ SamYaple I did not test Swift yet15:41
*** vhosakot has joined #kolla15:41
elemoine_I also want to add a commit that removes rsyslog15:42
liyihi again Sam, what do u get in when run "sudo mkdir /run/netns && sudo mount -o rw,relatime,shared -t tmpfs tmpfs /run/netns" in your OS? I got the same error in both my home built OS and Ubuntu 14.04.3 LTS15:42
SamYapleelemoine_: I would prefer the Heka stuff land in before we remove rsyslog15:42
SamYapleso we can revert if needed15:42
elemoine_SamYaple: fine with me15:43
elemoine_for Swift, I am not sure how I can test (I use vagrant)15:43
SamYapleliyi: I iddnt actually run that command but it probably errors teh same way15:43
SamYapleliyi: mount binary is old on 14.04, may not recogized shared15:43
SamYaplebut mount --make-shared /run is stil lvalid15:43
SamYapleeven on 14.0415:43
elemoine_SamYaple: pushing now…15:43
liyiyep15:43
liyithat works for both in my case15:44
liyibut I still end up in problems.15:44
SamYapleare yo urunning on bare metal? in a vm?15:44
liyiin a VM15:44
SamYapledid you install vanilla ubuntu 14.04?15:45
liyiyes, but I have not tried to deploy on vailla one yet. because that take hours to install the OS15:45
SamYapleI dont know what to suggest other than that. If it works on vanilla there is likely some difference that is breaking things15:47
liyimy customized built ubuntu only take minutes to installed, so I have been happily using it instead of vanilla one, until i hit on this problem15:47
SamYaplei deploy vanilla in minutes, its all about that apt-cacher15:47
liyiwill test on vanilla ubuntu later on when get time15:48
liyiyep, u r right.15:48
*** thumpba has joined #kolla15:49
vhosakotfailed: [localhost] => {"changed": true, "failed": true}15:49
vhosakotmsg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/images/create?tag=2.0.0&fromImage=192.168.122.45%3A4000%2Fkollaglue%2Fubuntu-source-rsyslog’,),)”.   Is the conainter/image broken ?15:49
vhosakotI see this when I run "kolla-ansible deploy"15:50
vhosakotI filed the bug https://bugs.launchpad.net/bugs/1546863 last night15:50
openstackLaunchpad bug 1546863 in kolla ""kolla-ansible deploy" throws HTTPError when starting rsyslog container" [Undecided,In progress] - Assigned to Vikram Hosakote (vhosakot)15:50
vhosakotany idea how to resolve this... can I ignore this error ?15:51
pbourkevhosakot: can you check the docker logs to see what error its throwing?15:54
*** pbourke has quit IRC15:54
*** pbourke has joined #kolla15:56
*** diogogmt has joined #kolla15:57
*** Serlex has quit IRC15:59
fthiagogvvhosakot what OS? Centos or Ubuntu?16:01
vhosakotUbuntu 14.0416:01
vhosakotpbourke: I cannot check logs as the container rsyslog is not running16:02
pbourkevhosakot: I mean docker daemon logs16:02
pbourkeunder /var/log16:02
*** salv-orl_ has joined #kolla16:03
*** zigo has quit IRC16:03
fthiagogvvhosakot do you set insecure-registry?16:03
vhosakotyes16:04
vhosakotok16:04
openstackgerritSam Yaple proposed openstack/kolla: Add docs to explain why we disabling selinux  https://review.openstack.org/28099516:04
vhosakotin /var/log ok16:04
*** zigo has joined #kolla16:05
*** blahRus has joined #kolla16:05
vhosakotI see this error for the "Starting rsyslog container" task when running "kolla-ansible deploy"16:05
* vhosakot is checking /var/log16:05
fthiagogvvhosakot are you doing single or multinode?16:06
*** salv-orlando has quit IRC16:06
*** coolsvap|away has quit IRC16:06
SamYaplevhosakot: /var/log/upstart/docker.log16:06
vhosakotfthiagogv: all-in-one single16:06
SamYaplecan I get an ack on this patch? https://review.openstack.org/#/c/280738/16:07
vhosakotwow16:07
vhosakotI see this16:07
vhosakot[37460] Handler for GET /v1.22/containers/kollaglue/ubuntu-source-rsyslog/json returned error: No such container: kollaglue/ubuntu-source-rsyslog16:07
vhosakotIs /var/log/upstart/docker.log docker daemon's logfile ?16:07
SamYaplevhosakot: did you set your registry correctly in gobals.yml?16:08
SamYapleglobals*16:08
SamYaplealright I need another core to volunteer to go through all the Heka patches.16:08
SamYapleI have +2 them all after many patchsets16:08
SamYaplethey are ready for merge, but they need to all my gone through again16:08
*** chandankumar has joined #kolla16:09
SamYaplestarting here https://review.openstack.org/#/c/27584016:09
vhosakotI have "docker_registry: "192.168.122.45:4000" in globals.yml16:09
elemoine_thanks SamYaple, I'll try to test Swift in the mean time16:09
vhosakotI have this in /etc/init.d/docker16:09
vhosakotDOCKER_OPTS="-H fd:// --insecure-registry 192.168.122.45:4000"16:09
vhosakotI also did    docker run -d -p 4000:5000 --restart=always --name registry registry:216:10
pbourkeSamYaple: Ive started going through them, but swamped with other stuff. I want to get swift upgrade tested before I look at anything else16:10
SamYaplepbourke: i get that. this logging this is just a big change and likely full of bugs (no offense elemoine_) and I want to see it merged well before mitaka-316:11
SamYapleso if you can carve out time that would be greatly appreciated16:12
pbourkeyeah i'll do my best. Id def like to test it before merge16:12
elemoine_SamYaple: that makes totally sense to me (merging this well before mitaka-3)16:12
*** inc0 has quit IRC16:12
pbourkeswitching between liberty and master is a killer16:12
fthiagogvvhosakot try to push some image using your registry, "docker push 192.168.122.45:4000/kollaglue/..." When i had this error was because that registry was wrong. However was in Centos.16:12
SamYaplepbourke: I did not test all of the services, but the core ones are working fwiw16:13
vhosakotfthiagogv: cool, let me push16:14
fthiagogvDo someone know about this? fatal: [control] => One or more undefined variables: 'dict object' has no attribute 'stdout'16:15
fthiagogvtask: neutron | reading json from variable16:15
openstackgerritSam Yaple proposed openstack/kolla: Launch VM in gate  https://review.openstack.org/28074916:17
SamYaplemandre: i think i figured it out16:17
SamYaplemandre: turns out some of the hypervisors have HVM enabled for nested KVM. who would have guessed!16:17
*** coolsvap|away has joined #kolla16:20
*** reddy has quit IRC16:22
*** akwasnie has quit IRC16:23
*** daneyon has joined #kolla16:30
*** daneyon__ has joined #kolla16:31
*** macsz has quit IRC16:31
limamauriciomanila needs to use openvswitch16:32
*** fgimenez has quit IRC16:33
limamauricioSamYaple, manila needs openvswitch, but i can't create openvswitch dir on manila_share container16:34
limamauricio:/16:34
*** daneyon_ has quit IRC16:34
*** daneyon has quit IRC16:34
*** fgimenez has joined #kolla16:34
SamYaplelimamauricio: i dont follow16:35
vhosakot"docker push 192.168.122.45:4000/kollaglue/ubuntu-source-ceph-mon"  returns16:35
vhosakotRepository does not exist: 192.168.122.45:4000/kollaglue/ubuntu-source-ceph-mon16:35
vhosakot"curl http://192.168.122.45:4000/v2/"  returns16:36
vhosakotcurl: (7) Failed to connect to 192.168.122.45 port 4000: No route to host16:36
vhosakotshould I be able to ping the registry IP 192.168.122.45.. I can't ping16:36
*** chandankumar has quit IRC16:38
SamYaplevhosakot: i would recommend setting up your registry and attempting to push/pull images outside of kolla first16:39
*** haplo37 has joined #kolla16:39
SamYaplethere are many documents online about how to do this16:39
pbourkevhosakot: are you running a registry16:39
pbourkeon 192.168.122.4516:39
*** pbourke has quit IRC16:39
vhosakotyep.. doing it... I think I need to tag first16:39
vhosakotyes16:39
vhosakoton 192.168.122.4516:39
vhosakoton port 400016:40
*** pbourke has joined #kolla16:41
fthiagogvvhosakot, when you build, you use the option --push? And didn't show any error? When your registry are broke, build --push does not works good and many images named <none> appears.16:42
vhosakotfthiagogv: no, I did not use the option --push when building16:43
*** prithiv has joined #kolla16:45
prithivI am having trouble deploy kolla on bare metal and facing issue16:46
prithivcan someone help me..16:46
g3ekhello, is kolla have a way to test the installation of a container appart of ansible or should I use ansible unit test to test it?16:46
rhalliseyprithiv, ya sure what's up?16:46
prithivi am just tryin got build on a baremetal AIO node. i am getting error with memcached container16:48
rhalliseyg3ek, not sure what you mean.  Do you want to test if a service is working?16:48
prithiv500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/images/create?tag=2.0.0&fromImage=10.237.214.35%3A5000%2Fkollaglue%2Fubuntu-source-memcached'16:48
rhalliseyg3ek, you can look at the containers status.  Docker ps -a16:48
*** mikelk has quit IRC16:49
prithivhow would i find which one is memcached container16:49
prithivi ddint try to look at providing names for the container when pulling16:50
g3ekrhallisey, I have to use kolla to deploy a whole openstack environment and some of our own service. I need to check if everything have been correctly installed and is working after the kolla-ansible deploy. I think the best way to check this will be with ansible, but I don't know if kolla have another way to make those check.16:50
*** dims has joined #kolla16:50
*** aginwala has joined #kolla16:51
rhalliseyprithiv, docker logs memcached16:51
rhalliseyneed some more info16:51
prithivi dont see any containers in that name16:51
*** dims_ has quit IRC16:51
rhalliseyok so it's just failing to show up hmm16:51
g3ekprithiv, maybe the container is down try with docker ps -a16:52
rhalliseyg3ek, ./tools/kolla-ansible post-deploy16:52
rhalliseyI think that's what you may be looking for16:52
g3ekrhallisey, this is not for the admin_openrc file?16:52
prithivi tried with ps -a…16:53
prithivthe thing is i have container name auto-assigned by  docker. i did not know if there is a way to specify container name when doing kolla-build16:54
g3ekrhallisey, ohhhh this run the file ./ansible/post-deploy.wml cool, thanks16:54
rhalliseyg3ek, ya I think that's what your looking for16:55
rhalliseyprithiv, there is.  `kolla-build memcached <flags>16:55
limamauriciosomeone know how i do to install openvswitch in manila?16:57
limamauriciojust add yum -y install openvswitch on manila-base dockerfile?16:58
rhalliseyya that would do it16:59
SamYaplelimamauricio: so manila needs the ovs binaries is what you are saying?16:59
limamauricioyes17:00
SamYapleyea then thats the way to do it17:00
SamYaplejust mount /run to get the correct socket too17:00
limamauricioopenvswitch is required to manila17:00
limamauriciomount /run17:01
limamauriciohm...17:01
limamauriciook17:01
SamYaplelimamauricio: no install openvswitch and then rememeber to add /run:/run to the mounts for the container17:03
SamYapleso it can talk to openvswitch server17:03
*** alisonh has quit IRC17:03
*** fgimenez has quit IRC17:04
limamauricioSo, i shouldn't add yum install openvswitch in manila-base dockerfile?17:04
limamauriciojust add /run:run?17:04
limamauriciojust add /run:/run?17:04
*** tzn has quit IRC17:05
SamYaplelimamauricio: install openvswitch in manila-base (or whatever subset application needs it)17:07
SamYaplethen add /run:/run17:07
*** aginwala has quit IRC17:07
openstackgerritMerged openstack/kolla: Ceilometer ubuntu binary container  https://review.openstack.org/27728617:10
*** tzn has joined #kolla17:11
*** dmsimard is now known as rdobot17:11
sdakeits a shame openvswitch sin't packaged into the cleint and sever as seprate packages17:12
sdakeall that server bloat for no good reason17:12
sdakeoh well containers ftw ;)17:12
*** rdobot is now known as dmsimard17:13
*** tobe has quit IRC17:13
*** daneyon has joined #kolla17:16
*** daneyon_ has joined #kolla17:17
*** daneyon__ has quit IRC17:19
*** daneyon has quit IRC17:21
openstackgerritDaniel Gonzalez Nothnagel proposed openstack/kolla: Change ubuntu base_image for vagrant to wily64  https://review.openstack.org/28196817:21
*** chandankumar has joined #kolla17:22
openstackgerritDaniel Gonzalez Nothnagel proposed openstack/kolla: Change ubuntu base_image for vagrant to wily64  https://review.openstack.org/28196817:23
SamYaplesdake: there was openvswitch talk about seperate packaging back in 2.117:26
SamYaplei think its landing/landed in 2.517:26
*** achanda has joined #kolla17:28
prithivnope it didnt help17:29
prithivdocker ps -a doesnt show any of the containers with openstack services name in it17:30
*** achanda has quit IRC17:33
liyican anyone explains a little bit what is thin container about? I am lost after we introduce thin container with docker 1.1017:35
*** ssurana has joined #kolla17:39
*** unicell has quit IRC17:40
*** kproskurin has quit IRC17:40
SamYapleliyi: "thin container" is a container running a single process17:41
SamYaplekolla has always been thin containers for most services17:41
*** kproskurin has joined #kolla17:41
*** alisonh has joined #kolla17:46
*** jmccarthy1 has quit IRC17:48
liyiThanks Sam!17:49
sdakeactually our thi ncontainers run multiple processes17:52
sdakei'd define thin container as somehting that does "one type of job"17:52
sdakein the cae of neutron, the neutron agent container did 4 (?) different jobs17:53
sdakeSamYaple made them all do one type of job17:53
*** limamauricio has quit IRC17:54
*** athomas has quit IRC17:54
*** limamauricio has joined #kolla17:54
liyiI though it was a new way to minimise the size of the container, but now I got the idea17:55
sdakeits a way to minimize the number of jobs to 1 that a container does17:58
sdakethin container bit of a misnomer i guess liyi :)17:58
*** athomas has joined #kolla18:00
liyi@sdake, thanks for making it clear. be offline now, Have a nice evening!18:02
*** liyi has left #kolla18:02
*** Marga__ has quit IRC18:02
*** Marga_ has joined #kolla18:03
*** pbourke has quit IRC18:03
*** achanda has joined #kolla18:03
*** Jeffrey4l__ has quit IRC18:06
*** Marga_ has quit IRC18:12
*** aginwala has joined #kolla18:13
*** mbound has quit IRC18:14
openstackgerritMerged openstack/kolla: Add docs to explain why we disabling selinux  https://review.openstack.org/28099518:14
*** Marga_ has joined #kolla18:18
sdakeSamYaple the selinux doc is pretty solid18:19
*** unicell has joined #kolla18:20
*** prithiv has quit IRC18:22
*** aginwala has quit IRC18:23
SamYaplesdake: technically its "tracking one process" that defines thin containers18:24
*** chandankumar has quit IRC18:24
*** fthiagogv_ has joined #kolla18:24
sdakei think the definition is open to intereptation18:24
sdakewhat we know it is not, is doing multiple different types of jobs in the same container ;)18:25
sdakei prefer to bang on fat containers, because those are easy to define :)18:25
*** fthiagogv__ has joined #kolla18:26
*** achanda has quit IRC18:26
*** aginwala has joined #kolla18:27
*** fthiagogv has quit IRC18:27
*** fthiagogv_ has quit IRC18:29
sdakeits a shame with heka we can't make all these xtend_starts go in the base file18:30
openstackgerritMerged openstack/kolla: Set nova hypervisor to qemu in Vagrant  https://review.openstack.org/28163718:31
sbezverksdake I noticed weird behavior with openvswitch, when I reboot my server and then run ovs-vsctl show, it shows wrong bridge configuration, but if I restart openvswitch service, everything becomes correctly plugged.18:33
sdakesbezverk I think we have some work to do on node restart18:33
sdakesbezverk please file a bug18:33
sdakeunfortunately docker doesn't have start ordering18:34
sbezverksdake got it..18:34
sdakethis is probably something we can fix in the containers themselves18:34
sdakewould be nice if it was surgical rather then all over the place though ;)18:34
*** prithiv has joined #kolla18:35
*** achanda has joined #kolla18:37
sdakewhats with zuul18:41
sbezverksdake https://bugs.launchpad.net/kolla/+bug/154716818:41
openstackLaunchpad bug 1547168 in kolla "wrong connections with openvswitch after server reboot" [Undecided,New]18:41
sdakesbezverk thanks it has been triaged18:42
sdakeelemoine_ ping18:44
*** Marga_ has quit IRC18:45
*** gfidente has quit IRC18:47
*** Marga_ has joined #kolla18:48
sbezverksdake it also requires to restart l3-agent container to refresh namespaces info18:51
*** Marga_ has quit IRC18:54
*** Marga_ has joined #kolla18:55
vhosakotcan someone send the output of "px aux | grep docker".. in Ubuntu 14.04, in which file should I add DOCKER_OPTS="-H fd:// --insecure-registry 192.168.122.45:4000"  ?18:56
vhosakot /etc/init.d/docker  or  /etc/init/docker.conf  or  /etc/default/docker ?18:56
*** achanda has quit IRC19:03
*** tzn has quit IRC19:03
vhosakotis  "-H fd://"  the right docker daemon socket to use for kolla ?19:06
*** tzn has joined #kolla19:14
*** achanda has joined #kolla19:17
*** tzn has quit IRC19:18
*** jasonsb has quit IRC19:20
sdaked_code around, can you paste your script for centos setup?19:24
sbezverkAnybody could point to the location of kolla's ansible inventory file? It looks like it does not use default /etc/ansible/hosts?19:24
*** prithiv has quit IRC19:24
*** aginwala has quit IRC19:25
sdakesbezverk ansible/inventory/all-in-one19:25
sdakeor ansible/invetory/multinode19:25
sdakekolla-ansible deploy defaults to all-in-one19:25
vhosakotI resolved it. "-H fd://"  option for docker daemon is just for centos, not for Ubuntu.. after removing "-H fd://",  my registry is up :)19:26
openstackgerritRyan Hallisey proposed openstack/kolla: Add a precheck for MountFlags=shared  https://review.openstack.org/28127919:26
vhosakotI will correct this in the quickstart guide for Ubuntu host19:26
rhalliseySamYaple, can you try that out^^19:26
rhalliseysee if that works for you19:27
rhalliseyon ubuntu19:27
sbezverksdake when I use this command to deploy swift: kolla-ansible deploy --tags swift it skips the ip address I have in all-in-one19:27
sbezverksdake Here is all-in-one I use, appreciate if you could check what is wrong: http://paste.openstack.org/show/487473/19:36
*** aginwala has joined #kolla19:38
*** vhosakot has quit IRC19:42
*** jtriley has quit IRC19:44
*** vhosakot has joined #kolla19:44
*** jtriley has joined #kolla19:48
*** achanda has quit IRC19:51
SamYaplesbezverk: did you set 'enable_swift: "yes"' in globals.yml?19:53
SamYaplerhallisey: will check!19:53
SamYaplebut it looks good19:53
openstackgerritSam Yaple proposed openstack/kolla: Launch VM in gate  https://review.openstack.org/28074919:55
*** achanda has joined #kolla19:58
sbezverkSamYaple that was it, I did not.. thank you20:00
*** gtt116_ has joined #kolla20:06
vhosakotI changed the port.. I get this error now20:07
vhosakotError while pulling image: Get http://192.168.122.41:4000/v1/repositories/v2/repositories/kollaglue/ubuntu-source-rsyslog/images: dial tcp 192.168.122.41:4000: getsockopt: connection refused20:07
dave-mccowandoes anyone use .header and .footer files to poke through an http proxy for tools/build.py? it's not working for me as documented.20:09
*** jmccarthy1 has joined #kolla20:11
*** jmccarthy has quit IRC20:12
*** gtt116 has quit IRC20:12
*** zaneb has quit IRC20:12
*** mandre has quit IRC20:12
*** vincent_vdk has quit IRC20:12
*** zaneb has joined #kolla20:12
*** vincent_vdk has joined #kolla20:12
SamYapledave-mccowan: docker 1.10 actually has proxy support without header and footer files20:13
*** mandre has joined #kolla20:13
dave-mccowanSamYaple  cool.  my environment variable are set and my ansible is 1.10.1.  hmmm, why else would i get a network timeout?  git, yum, pip, etc. all work.20:16
*** aginwala has quit IRC20:16
*** Marga_ has quit IRC20:17
dave-mccowani can wget https://index.docker.io/v1/repositories/library/centos/images, but build.py times out getting it.20:20
SamYapledave-mccowan: hmm i woudl suggest going a small test outside of kolla20:25
SamYapleget it working that way and we can figure out whats wrong20:25
dave-mccowani tried wget to docker.io, and it works through the proxy.  what else to test?20:26
SamYaplea simple dockerfile outside of kolla20:27
*** aginwala has joined #kolla20:27
*** vhosakot has quit IRC20:27
*** vhosakot has joined #kolla20:28
sbezverkSamYaple for swift installation I need to generate account.builder and two other .builder files using swift-ring-builder. I started swift-base container and found this tool there. I generated required files, but they exist within a container, do I have to scp these files back to host or there is another more simple way?20:37
*** athomas has quit IRC20:39
*** athomas has joined #kolla20:40
sdakesbezverk the swift guide details how to do the job20:46
sdakecheck docs20:46
*** Serlex has joined #kolla20:49
*** thumpba has quit IRC20:53
sbezverksdake my question about swift is not covered by usual openstack swift operational guide.. it is more container related and how to get files generated inside of a container back to the host..20:55
openstackgerritDave McCowan proposed openstack/kolla: Update Instructions for Building Behind HTTP Proxy  https://review.openstack.org/28205721:04
SamYaplesbezverk: to transfer files between containers and host you can use the docker cp command `docker cp <container>:/path/to/file /path/on/host`21:05
SamYaplethat works in reverse too21:05
*** dwalsh_ has joined #kolla21:06
vhosakotfixed my port and registry.... still "kolla-ansible deploy" throws the error21:06
vhosakotmsg: The requested image does not exist: 192.168.122.41:4000/kollaglue/ubuntu-source-rsyslog:2.0.021:06
sbezverkSamYaple that is what I was looking for.. btw swift containers are up and running :-)21:07
openstackgerritHarm Weites proposed openstack/kolla: Some fixes specific to running with Vagrant  https://review.openstack.org/27718521:08
*** tzn has joined #kolla21:08
*** tzn has quit IRC21:12
SamYaplesbezverk: awesome!21:12
vhosakotI do see this image  ubuntu-source-rsyslog21:18
*** inc0 has joined #kolla21:25
vhosakotthe registry is looks good.. I can push and pull images... but kolla-ansible fails...21:26
vhosakotcan anyone help... I can paste stuff in pastebin21:26
inc0ayoung, just seen your blog post, thanks:)21:31
inc0http://adam.younglogic.com/2016/02/holla-kolla/21:31
openstackgerritHarm Weites proposed openstack/kolla: Some fixes specific to running with Vagrant  https://review.openstack.org/27718521:35
*** asalkeld has joined #kolla21:35
*** mbound has joined #kolla21:38
openstackgerritHarm Weites proposed openstack/kolla: Some fixes specific to running with Vagrant  https://review.openstack.org/27718521:42
*** tpot has joined #kolla21:50
ayounginc0, your welcome.  I am working on a second one for running the Keystone functional tests against it21:51
inc0fyi we added your selinux issue as one of our priorities for N21:52
inc0so you should have it in repo early N or even earlier21:52
Serlexanyone know where I can set the disk size for vagrant deployment, 20g wasn't enough for operator21:53
Serlexactually nvm21:54
*** salv-orlando has joined #kolla22:03
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Playbook for nova upgrade  https://review.openstack.org/25439522:03
*** inc0 has quit IRC22:04
*** salv-orl_ has quit IRC22:05
*** aginwala has quit IRC22:14
*** jtriley has quit IRC22:16
*** aginwala has joined #kolla22:16
*** dwalsh has quit IRC22:23
*** dwalsh_ has quit IRC22:23
*** dave-mccowan has quit IRC22:26
*** JoseMello has quit IRC22:28
*** Marga_ has joined #kolla22:29
SamYaplesdake: vm functional test passed gate22:38
SamYaplebut... only on ovh22:39
SamYaplewierd right?22:39
openstackgerritSuhail Syed proposed openstack/kolla: Refined documentation regarding registry  https://review.openstack.org/28208422:40
sdakeSamYaple rockin dude!22:40
sdakeonly on ovh?22:44
SamYapleso it seems22:45
SamYaplethe others throw no valid host, but im working on it22:45
sdakeovh means what?22:49
sdakesorry super long week already22:49
SamYapleovh is one of the cloud providers that infra uses22:50
sdakei thought that was what you meant but wasn't sure22:50
sdakeso maybe its a oom problem?22:50
sdakeor somethign else22:50
sdakemaybe virt on virt doesn't work on other cloud provides22:50
sdakeare you doing qemu virt?22:50
sdakeor kvm?22:51
sdakerax i dont think uses kvm22:51
SamYapleqemu22:51
SamYaplebut it doesnt matter if rax uses kvm or not22:51
sdakethat is entirely emulated22:51
SamYaplekvm could work on rax on xen22:51
sdakeit should work22:51
SamYaplebut thats beside the point22:51
sdakedidn't know kvm on xen worked ;)22:51
* sdake wtbs more underlays!!22:52
SamYapleits all about cpu flags22:52
SamYaplenothing else22:52
sdakequestion from daughter "Why does my computer make liquid sounds when it turns on"22:54
sdake"Why its liquid cooled my dear" :)22:55
SamYaplelike my servers!22:55
*** spisarski has quit IRC22:56
asalkeldo/23:00
*** spisarski has joined #kolla23:00
sdakeasalkeld yo bro23:01
sdakei slacked yesterday23:01
sdakesorry23:01
sdakeemail coming23:01
asalkeldnp23:01
sdakedmsimard ping re packstack + magnum23:01
sdakedmsimard is the reason magnum isnt in the current-ci repo because of lack of packstack implementation?23:02
*** aginwala has quit IRC23:02
*** aginwala has joined #kolla23:06
dmsimardNo23:08
dmsimardWe haven't had a promotion to current-passed-ci in a while because trunk keeps breaking23:09
sdakeso packstack is implemented for magnum?23:12
sdakedmsimard ^23:12
sdakei assumed cuurrent-passed-ci is based upon a packstack ci cycle23:13
sdakeby trunk do you mean trunk of magnum or trunk of openstack?23:13
vhosakoti need help o/23:18
vhosakot:)23:18
SamYaplewow!23:19
SamYaplezfsonlinux will be in default repo for 16.0423:19
SamYapleubuntu23:19
vhosakotsdake: how is remote debugging done in the team?  In the neutron channel, I've used https://tmate.io to do remote-debugging23:19
SamYaplevhosakot: someone opens up a port to root....23:20
*** haplo37 has quit IRC23:20
sdakevhosakot yup thats how its done, port ro root through their firewall, which is suboptimal since not alot of people can actually do so :)23:21
sdakelooking at tmate.io23:21
SamYaplerequires trust23:21
SamYaplefor the record i like tmate alot23:21
vhosakotyes, open a port to root, on my machine... wow, I'm on o crop network ;)23:22
vhosakotcorp* == corporate23:22
SamYaplevhosakot: its called a reverse tunnel23:22
SamYapleeven on your orporate netwrk yo ucan do it ;)23:23
vhosakotI see this error23:23
vhosakotTASK: [common | Starting rsyslog container] ***********************************23:23
vhosakotfailed: [localhost] => {"failed": true}23:23
vhosakotmsg: The requested image does not exist: 192.168.122.41:4000/kollaglue/ubuntu-source-rsyslog:2.0.023:23
SamYaplethats a pretty clear cut error23:23
sdakeSamYaple i'm ssure corps sec groups would have a kaniption fit if they knew that was happening :)23:23
vhosakotssh -R ;)23:23
SamYaplei just open a port to my public ip and point it to bash23:24
SamYaplethats just me though23:24
*** tzn has joined #kolla23:24
sdakevhosakot what is on ip .41?23:24
sdakeyour registry?23:24
sdakeand if so, did you build with --push23:24
sdakeand --registery 192.168.22.41:400023:24
vhosakotthe IP address of eth0... the only IP on the Ubuntu box I used to SSH into23:24
*** blahRus has quit IRC23:24
vhosakotyes.. done.. "ps aux | grep docker" returns23:25
vhosakot   /usr/bin/docker daemon --insecure-registry 192.168.122.41:400023:25
sdakerun docker ps -a and paste somewhere not in irc :)23:25
vhosakotsdake: BTW, the "-H fd://" argument mentioned in the quistart guide to setup docker socket is only for CentOS, not for Ubuntu 14.04.. on Ubuntu, the default Unix socket works for docker...23:26
* sdake hopes we get a lab session at summit23:26
sdakevhosakot cool doc fix requested ;)23:27
vhosakotsince the error message is not clean, I opened a bug against docker https://github.com/docker/docker/issues/20468#issuecomment-18597107323:27
vhosakotyes, doc fix coming23:27
vhosakotdoc fix, automated kolla install on Ubuntu, --force-cross-build flag... all are coming23:27
vhosakotthe help must come too... hahaha :)23:27
vhosakotok, will send the output in pastebin :)23:28
*** tobe has joined #kolla23:28
sdakedocker insepct is about the limit of docker diags ;-)23:28
sdaketheir logging is terirble imo23:28
vhosakotyup....  i mean /var/log/upstart/docker.log does not have timestamp!  what package does that ;)23:29
vhosakotI filed two Vagrant bugs too when installing kolla... Vagrant is just for dev work, not prod23:30
sdakeyes that should be clear already in the docs23:30
sdakeThis guide describes how to use Vagrant to assist in developing for Kolla.23:31
sdakeright at top of the vagrant doc23:31
sdakeclunky wording, admittidly23:31
dmsimardsdake: magnum doesn't need to be included in packstack for it to be part of RDO or current-passed-ci23:32
dmsimardhell, there's even cloud kitty in our repos23:32
dmsimardwhat makes a package go from current to current-passed-ci is a pipeline of jenkins jobs that test openstack through various installers (right now: tripleo, packstack, puppet-openstack -- eventually: kolla)23:33
sdakenot sure what a cloud kitten is23:33
vhosakotsdake: here you go :)23:33
sdakemust be fluffy :)23:33
dmsimardsdake: if all of these installers install openstack and test it successfully (tempest) that set of package is promoted23:33
sdakedmsimard cool thats helpful23:34
dmsimardsdake: maybe if you're more visual: http://versiontracker.dmsimard.com/compare/mitaka-master23:34
vhosakotsdake: yes, the doc clearly says Vagrant is for dev... correct23:34
vhosakotsdake: here you go (with the link this time) http://paste.openstack.org/show/487514/  :)23:34
sdakedmsimard so basically magnum wont be promoted until tripleo, packstack, and puppet-openstack are implemented23:34
dmsimardsdake: nooooooooo23:34
dmsimardit doesn't need to be implemented, it just needs to be part of the set of packages that are tested23:35
vhosakotsdake: let me know if you can see what I've pasted in http://paste.openstack.org/show/487514/23:35
dmsimardWe don't have anything that tests magnum right now23:35
dmsimardAs part of that pipeline23:35
dmsimardSo that just means that (exaggerating here) if nova works, magnum will go from current to current-passed-ci23:35
sdakeso it could be promoted without testing?23:35
sdakegot it23:35
sdakethanks23:35
*** tobe has quit IRC23:35
*** dims_ has joined #kolla23:36
dmsimarda lot of things were promoted without being properly tested, we've gotten better at it23:36
dmsimardit's still not perfect but hey, I did a whole talk on that this week :) http://redhat.slides.com/dmoreaus/testing-openstack-is-hard#/23:36
dmsimardfor example, we didn't properly test gnocchi until puppet-openstack had an implementation for it23:37
*** dims has quit IRC23:38
dmsimardso the idea is that we use different installers to try and get as much coverage as we can23:38
dmsimardI even chatted with a guy working on chef cookbooks for openstack today :)23:38
*** tzn has quit IRC23:39
*** dave-mccowan has joined #kolla23:41
vhosakotsdake: did you get a chance to look at http://paste.openstack.org/show/487514/ ?23:45
sdakevhosakot you have some wierd leaked containers23:46
sdakeI haec seen this once befoer on centos on my gear23:46
sdakesince your using ubuntu it may be a global problem23:46
sdakeSamYaple any ideas on above ^^23:46
vhosakotsdake: leaked ? broken ? I'm able to pull and push form the container23:47
vhosakotfrom the registry I meant23:47
*** openstackgerrit has quit IRC23:47
*** openstackgerrit_ is now known as openstackgerrit23:47
sdake7e5562642d95        b1d06b74f1c8        "/bin/sh -c 'chmod 75"   23 hours ago        Exited (0) 23 hours ago                            hopeful_sinoussi23:47
sdakethat is a really odd state for a container to be in23:47
*** openstackgerrit_ has joined #kolla23:47
sdakeall our containers are NAMED23:47
sdakethey dont pick random names23:47
sdakepossibly unless your build failed23:47
sdakemaybe that is why the containers leak - i know we keep intermediate containers around23:48
sdakeor used to at one point23:48
vhosakotah, yes.. the random names are funny23:48
*** openstackgerrit_ is now known as openstackgerrit23:48
SamYaplesdake: what?23:48
sdakeSamYaple see vhosakot's paste of docker ps -a23:48
vhosakotstupefied_golick, loving_kowalevski, cocky_kare haha23:48
SamYapleoh thats most commonly seen on slower backends23:48
*** openstackgerrit_ has joined #kolla23:48
vhosakotSamYaple: http://paste.openstack.org/show/487514/23:48
SamYaplecentos in the gate often had that issue23:49
SamYapleuntil i switched to btrfs23:49
sdakevhosakot which backend are you using?23:49
*** tobe has joined #kolla23:49
SamYapleit can occur with any, ive seen it most commont with btrfs23:49
sdakeI suspect the default - but not sure what that is on ubuntu 14.0423:49
vhosakotstorage backend ?23:49
SamYaple*devicemapper23:49
SamYaplenot btrfs23:49
sdakeyour graph driver23:49
sdakewhat triggers it?23:50
sdakea build failure?23:50
sdakeor is that unknown23:50
vhosakothow can i check it ?23:50
vhosakot  /var/lib/docker ?23:51
sdakevhosakot it sat the top of your log23:51
sdakeit will say graphdriver and have the type23:51
sdakeI run with -s overlay23:51
sdakevhosakot the fact that containers are leaked like that is a bit of a problem23:51
sdakecan you file a bug for that please23:51
sdakewe can sort out how to fix it later - but we will forget if its not tracked23:52
vhosakotyes..23:52
SamYaplerun `docker info`23:52
vhosakotbrb 10 mins23:52
SamYaplevhosakot: ^^23:52
sdakeat pressent I'd use docker rm -f on all the docker containers running at present23:52
sdakeyou can just use the first 2-3 characters of the id23:52
sdakesuch as docker  rm -f 5c523:52
sdakedocker rm -f 73023:52
sdakeetc23:52
SamYaplesdake: its not a problem, jsut means vhosakot needs to build with fewer threads23:53
SamYapleor single threaded23:53
sdakewould still like a bug so poeple that have that problem can understand the issue23:53
sdakebecause you say "build with less threads" I hear "buildign with less threads causes it to happen bu twe dont know why"23:53
sdaketime to ffeed children bbl23:54
sdakevhosakot but it sounds like SamYaple is suggesting build with less threads ;)23:54
*** openstackgerrit_ has quit IRC23:55
*** openstackgerrit_ has joined #kolla23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!