Thursday, 2016-04-14

vhosakoto/00:00
tonybsdake: for reference https://www.openstack.org/summit/austin-2016/summit-schedule/events/947400:01
rhalliseyanyone else see the kubernetes presentation?>00:01
tonybis where we'll talk about how long each stable branch lives for00:01
rhalliseykeynote at summit00:01
sdakeok folks i 've got a meeting now00:01
sdaketonyb ack00:01
sdakeinc0 can you sort this out s you are already doing :)00:01
sdaketonyb would it be possible to add a stable follows policy exception for 1.0.0 release00:02
inc0sdake, tonyb can you discuss tags later on? I don't know much about that stuff00:02
sdaketo allow fresh projects to et in the grove00:02
sdakei mean would you be open to that?00:02
inc0and be vigilant about policy afterwards00:03
tonybsdake: I'd need to think abotu what that means00:03
inc0only reason we do this is to prevent people from really hurting themselves00:03
sdaketonyb o i'll bounce something off the governance queue and open it to discussion, sound good?00:03
inc0and again, we're deployment tool so 1. hurting themselves means losing OpenStack...00:03
sdakeI've really gota jet - lawyers yelling at me :)00:03
tonybsdake: Sure but expect it to get but in the "talk about in Austin" pile :)00:04
inc0and 2. we don't have running service, but we want to be coupled with a release for a reason00:04
sdaketonyb ack00:04
sdaketonyb that is fully what i expect00:04
sdakebut it will be written down00:04
tonybsdake: cool00:04
tonybsdake: go to the meeting ;p00:04
sdakeive already got about 6 gov repo changes that are in that pie :)00:04
inc0tonyb, as for technical side to it00:04
inc0is git co, git reset, git push possible with out infra?00:05
tonybinc0: as to the technical side "go nuts"00:05
tonybinc0: you're the project team and experts and if this is the *only* way you can keep liberty going *and*00:05
inc0yes, that wasn't decision we've taken lightly00:06
tonybther si some kind of going from 1.0.0 -> 1.1.0 will have these extra steps documented etc then it's your call00:06
inc0tonyb, nobody runs 1.0.0 from what we know00:06
inc0right now00:06
inc0ops deploys 1 release back00:07
inc0that's why we want to make sure that if they deploy, they won't be hurt00:07
*** absubram has quit IRC00:07
inc0but yes, we have few ideas how to provide migration00:07
inc0tonyb, how about we discuss it more tomorrow or next week?00:09
inc0so you'll have some time to digest it00:09
tonybinc0: cool.00:09
inc0let's do that then00:09
*** vhosakot has quit IRC00:10
inc0I'll prep stable/mitaka with patch00:10
inc0and reach out to you and then we'll decide how to backport it in a way that nobody gets hurt00:10
inc0would be too much to ask for you to stick around #openstack-kolla?00:11
inc0or which channel you normally are on?00:11
*** achanda has joined #openstack-kolla00:12
inc0well, g2g as well00:16
inc0cya fols00:16
*** inc0 has quit IRC00:16
*** achanda has quit IRC00:17
*** vhosakot has joined #openstack-kolla00:22
sbezverkvhosakot ping00:26
vhosakotpong00:27
sbezverkvhosakot have you played with multinode config?00:27
vhosakotnope.. I use AIO.. ccesario is your man!00:27
vhosakotmlima__ too00:27
vhosakotsdake too00:27
vhosakotdont know who else00:27
sbezverkthanks I was always using AIO, so need to check with folks who knows multinode..00:28
vhosakotcool00:29
*** Jeffrey4l__ has quit IRC00:30
ccesario_hello00:38
*** rhallisey has quit IRC00:41
*** tonyb has quit IRC00:44
*** salv-orlando has joined #openstack-kolla00:44
*** tonyb has joined #openstack-kolla00:44
*** banix has joined #openstack-kolla00:48
*** salv-orlando has quit IRC00:49
openstackgerritHui Kang proposed openstack/kolla: Remove node_templates_directory in ansbile variable file  https://review.openstack.org/30554300:56
*** brad[]` is now known as brad[]00:57
*** vhosakot has quit IRC01:07
*** vhosakot has joined #openstack-kolla01:10
*** achanda has joined #openstack-kolla01:13
*** achanda has quit IRC01:19
*** Marga__ has quit IRC01:22
*** ccesario_ has quit IRC01:34
*** jasonsb has joined #openstack-kolla01:45
*** sdake has quit IRC01:55
*** sdake has joined #openstack-kolla01:56
openstackgerritKuo-tung Kao proposed openstack/kolla: add unit tests for build script  https://review.openstack.org/29042302:03
*** ravig has joined #openstack-kolla02:06
*** sdake has quit IRC02:12
*** ravig has quit IRC02:12
*** weiyu_ has joined #openstack-kolla02:12
*** weiyu_ has quit IRC02:15
*** achanda has joined #openstack-kolla02:15
*** weiyu_ has joined #openstack-kolla02:16
*** weiyu_ has quit IRC02:18
*** unicell1 has quit IRC02:20
*** weiyu has joined #openstack-kolla02:20
*** achanda has quit IRC02:21
*** weiyu has quit IRC02:23
*** weiyu_ has joined #openstack-kolla02:27
*** weiyu_ has quit IRC02:30
*** coolsvap has joined #openstack-kolla02:33
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/30489502:33
sbezverkcoolsvap ping02:37
coolsvapsbezverk: pong02:37
sbezverkcoolsvap qq do you know by any chance why keystone only listens for authentication on api interface?02:37
sbezverkor if separate api is not configured, then on internal interface02:38
*** salv-orlando has joined #openstack-kolla02:40
coolsvapsbezverk: i think its due to the fact that keystone is the entry point for all openstack services02:41
sbezverkcoolsvap agree, but if a person needs to authenticate to opensatck from outside, then this internal network is not visible.02:42
coolsvapyes its always the case, the api interface (which is public) needs to be mapped to the internal interface02:43
coolsvapthe internal interfaces should not be visible to outside world02:44
*** salv-orlando has quit IRC02:44
*** weiyu_ has joined #openstack-kolla02:46
sbezverkcoolsvap but then it is problem, nobody from outside where only public VIP is visible will be able to authenticate against keystone..02:47
sbezverkanyway I will check with fellow operators before filing a new bug02:48
sbezverkthank you for your help..02:48
coolsvapsbezverk: its always the case, the internal network, IPs are never visible/reachable to outside world02:49
coolsvapwhat is the issue you are facing which got you to this question?02:49
*** ravig has joined #openstack-kolla02:49
*** ravig has quit IRC02:50
*** ravig has joined #openstack-kolla02:51
*** weiyu_ has quit IRC02:52
coolsvapvhosakot: around02:54
vhosakotcoolsvap: ping02:54
*** weiyu_ has joined #openstack-kolla02:55
*** ravig has quit IRC02:58
*** ravig has joined #openstack-kolla02:58
*** coolsvap has quit IRC03:00
*** coolsvap has joined #openstack-kolla03:04
coolsvapvhosakot: did you get my messages?03:12
openstackgerritKuo-tung Kao proposed openstack/kolla: add unit tests for build script  https://review.openstack.org/29042303:13
vhosakotcoolsvap: no.. I did not get your message..03:13
vhosakotcoolsvap: can you send again03:13
coolsvapvhosakot: some reviews https://review.openstack.org/#/c/304518/ https://review.openstack.org/304763 https://review.openstack.org/#/c/304783/ https://review.openstack.org/#/c/305073/  https://review.openstack.org/#/c/305588/03:13
patchbotcoolsvap: patch 304518 - kolla - cleanup-host should not remove kolla-build.conf03:13
patchbotcoolsvap: patch 304783 - kolla - Update base dockerfile for formatting03:13
patchbotcoolsvap: patch 305073 - kolla - Update haproxy dockerfile for formatting03:13
patchbotcoolsvap: patch 305588 - kolla (stable/mitaka) - Added help documentation for kolla-ansible upgrade03:13
coolsvapand i wanted to know your views on https://review.openstack.org/#/c/305251/03:13
patchbotcoolsvap: patch 305251 - kolla (stable/mitaka) - check /e/s/s/docker.service in ubuntu03:13
vhosakotcool, will review them..03:14
coolsvapvhosakot: thanks!03:15
*** achanda has joined #openstack-kolla03:17
vhosakotcoolsvap: cool, np03:17
*** Marga_ has joined #openstack-kolla03:20
*** achanda has quit IRC03:22
*** Marga_ has quit IRC03:23
*** Marga_ has joined #openstack-kolla03:24
*** gbraad has joined #openstack-kolla03:25
*** weiyu_ has quit IRC03:31
*** coolsvap has quit IRC03:45
*** jasonsb has quit IRC03:47
*** coolsvap has joined #openstack-kolla03:48
*** weiyu_ has joined #openstack-kolla03:55
*** weiyu_ has quit IRC03:57
*** weiyu_ has joined #openstack-kolla04:01
*** achanda has joined #openstack-kolla04:02
*** weiyu_ has quit IRC04:05
*** achanda_ has joined #openstack-kolla04:05
*** achanda has quit IRC04:06
*** ph03n1x has quit IRC04:15
*** achanda_ has quit IRC04:22
*** alyson_ has quit IRC04:22
*** alyson_ has joined #openstack-kolla04:23
*** ph03n1x has joined #openstack-kolla04:27
*** ruhe2 has joined #openstack-kolla04:30
*** ByPasS has quit IRC04:30
*** ruhe2 is now known as ruhe04:30
*** ByPasS has joined #openstack-kolla04:31
*** banix has quit IRC04:32
*** cfarquhar has quit IRC04:32
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Fixes kolla_toolbox failure due to missing pytz module  https://review.openstack.org/30560604:32
*** cfarquhar has joined #openstack-kolla04:34
*** cfarquhar has joined #openstack-kolla04:34
coolsvapasalkeld: are there any steps listed out to get started with kolla-mesos?04:35
*** ravig has quit IRC04:36
*** banix has joined #openstack-kolla04:36
*** ph03n1x has quit IRC04:37
*** banix has quit IRC04:39
*** weiyu has joined #openstack-kolla04:40
*** ph03n1x has joined #openstack-kolla04:43
*** weiyu has quit IRC04:44
*** ph03n1x has quit IRC04:44
*** ph03n1x has joined #openstack-kolla04:47
*** achanda has joined #openstack-kolla04:53
*** sbezverk has quit IRC04:54
*** sbezverk has joined #openstack-kolla04:54
*** salv-orlando has joined #openstack-kolla05:11
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: check /e/s/s/docker.service.d/kolla.conf in redhat family  https://review.openstack.org/30561105:17
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: check /e/s/s/docker.service in ubuntu  https://review.openstack.org/30525105:20
*** jidar has quit IRC05:24
*** jidar has joined #openstack-kolla05:28
*** Marga_ has quit IRC05:30
*** achanda has quit IRC05:31
*** coolsvap has quit IRC05:35
*** Marga_ has joined #openstack-kolla05:36
*** tfukushima has joined #openstack-kolla05:56
*** dims_ has joined #openstack-kolla06:00
*** v1k0d3n_ has joined #openstack-kolla06:00
*** mdnadeem has joined #openstack-kolla06:01
*** mikelk has joined #openstack-kolla06:02
*** gbraad_ has joined #openstack-kolla06:03
*** harmw_ has joined #openstack-kolla06:04
*** achanda has joined #openstack-kolla06:04
*** mlima_ has joined #openstack-kolla06:04
*** elmiko_ has joined #openstack-kolla06:05
*** Marga__ has joined #openstack-kolla06:06
*** zeus- has joined #openstack-kolla06:07
*** Marga_ has quit IRC06:08
*** sbezverk has quit IRC06:08
*** gbraad has quit IRC06:08
*** vhosakot has quit IRC06:08
*** tonyb has quit IRC06:08
*** mlima__ has quit IRC06:08
*** nihilifer has quit IRC06:08
*** d_code has quit IRC06:08
*** v1k0d3n has quit IRC06:08
*** dims has quit IRC06:08
*** daneyon has quit IRC06:08
*** zeus has quit IRC06:08
*** mdnadeem_ has quit IRC06:08
*** smekel_ has quit IRC06:08
*** cloudnull has quit IRC06:08
*** elmiko has quit IRC06:08
*** nihilifer has joined #openstack-kolla06:09
*** zeus- is now known as zeus06:09
*** zeus is now known as Guest3681206:09
*** intr1nsic has joined #openstack-kolla06:10
*** sbezverk has joined #openstack-kolla06:11
*** tonyb has joined #openstack-kolla06:14
mimizonekolla-build --base ubuntu --type binary --registry 127.0.0.1:4000 --push06:16
mimizonethrows me the following error "ImportError: No module named debtcollector"06:16
mimizoneam I missing a library?06:17
mimizoneactually just calling kolla-build throws the same error06:19
*** weiyu has joined #openstack-kolla06:19
*** coolsvap has joined #openstack-kolla06:19
kjellymimizone: post  traceback in http://paste.openstack.org/06:21
mimizoneI installed debtcollector with pip, now it's complaining about netaddr06:21
mimizonehttp://paste.openstack.org/show/494018/06:22
mimizoneI am following this documentation https://github.com/openstack/kolla/blob/master/doc/quickstart.rst06:22
*** cloudnull has joined #openstack-kolla06:23
mimizoneadding netaddr with pip seems to be enough06:23
kjellymimizone: I install kolla in new virtualenv. All thing works fine. Maybe you have old oslo_config in your env.06:25
mimizoneI think I have done all the other steps with sudo, so most likely that comes from that06:26
mimizonesudo kolla-build is not complaining of anything06:26
kjellymimizone: Ok. Have fun with kolla.06:27
*** jmccarthy has left #openstack-kolla06:31
*** weiyu has quit IRC06:31
*** coolsvap is now known as coolsvap_zzz06:32
*** d_code has joined #openstack-kolla06:32
openstackgerritMD NADEEM proposed openstack/kolla: Remove redundant value  https://review.openstack.org/30562506:34
*** achanda has quit IRC06:36
*** weiyu_ has joined #openstack-kolla06:36
*** jmccarthy has joined #openstack-kolla06:38
*** weiyu_ has quit IRC06:38
*** jmccarthy has left #openstack-kolla06:39
*** vivek__ has quit IRC06:39
mimizonenormal that images for magmum, designate, gnocchi, mistral, zaqar failed be build?06:41
openstackgerritMD NADEEM proposed openstack/kolla: WIP: Added Dockerfile for congress  https://review.openstack.org/30562606:43
openstackgerritMD NADEEM proposed openstack/kolla: WIP: Added Dockerfile for congress  https://review.openstack.org/30562606:48
*** vhosakot has joined #openstack-kolla06:49
*** daneyon has joined #openstack-kolla06:50
*** Serlex has joined #openstack-kolla06:56
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Enable unbuffered output for ansible  https://review.openstack.org/30338907:00
*** vhosakot has quit IRC07:03
*** unicell has joined #openstack-kolla07:04
openstackgerritMerged openstack/kolla: Identify container through  PS1 shell variable  https://review.openstack.org/30539307:06
*** smekel_ has joined #openstack-kolla07:06
*** weiyu_ has joined #openstack-kolla07:09
*** mlima_ has quit IRC07:11
*** sdake has joined #openstack-kolla07:12
*** alyson_ has quit IRC07:12
*** achanda has joined #openstack-kolla07:19
*** unicell has quit IRC07:19
*** unicell has joined #openstack-kolla07:20
*** weiyu_ has quit IRC07:21
*** weiyu has joined #openstack-kolla07:21
*** weiyu has quit IRC07:24
coolsvap_zzzsdake how was the webinar?07:24
*** mbound has joined #openstack-kolla07:24
*** mbound has quit IRC07:25
*** athomas has joined #openstack-kolla07:26
*** chopmann has joined #openstack-kolla07:27
*** weiyu_ has joined #openstack-kolla07:28
*** weiyu_ has quit IRC07:31
*** sdake has quit IRC07:31
*** sbezverk has quit IRC07:38
*** sbezverk has joined #openstack-kolla07:40
*** weiyu_ has joined #openstack-kolla07:46
*** weiyu_ has quit IRC08:00
*** achanda has quit IRC08:01
*** achanda has joined #openstack-kolla08:10
*** achanda has quit IRC08:11
*** mbound has joined #openstack-kolla08:12
*** jmccarthy has joined #openstack-kolla08:16
*** jmccarthy has left #openstack-kolla08:16
*** chopmann has quit IRC08:20
*** daneyon has quit IRC08:20
*** daneyon has joined #openstack-kolla08:21
*** chopmann has joined #openstack-kolla08:23
*** ajafo has left #openstack-kolla08:24
*** openstackstatus has joined #openstack-kolla08:25
*** ChanServ sets mode: +v openstackstatus08:25
*** rmart04 has joined #openstack-kolla08:26
*** coolsvap_zzz has quit IRC08:30
*** weiyu has joined #openstack-kolla08:31
*** gfidente has joined #openstack-kolla08:52
*** v1k0d3n_ has quit IRC08:52
*** v1k0d3n has joined #openstack-kolla08:52
*** chopmann has quit IRC09:03
*** ydeshmukh has joined #openstack-kolla09:10
*** akscram has quit IRC09:30
*** smekel_ has quit IRC09:30
*** chopmann has joined #openstack-kolla09:33
*** bradjones has quit IRC09:33
*** coolsvap has joined #openstack-kolla09:33
*** stvnoyes1 has quit IRC09:34
*** akscram has joined #openstack-kolla09:35
openstackgerritjackning proposed openstack/kolla: Added webroot config to dashboard"  https://review.openstack.org/30570409:36
*** stvnoyes has joined #openstack-kolla09:36
*** chopmann has quit IRC09:38
*** pbourke_ has quit IRC09:38
*** pbourke has joined #openstack-kolla09:39
*** bradjones has joined #openstack-kolla09:43
*** bradjones has joined #openstack-kolla09:43
*** smekel_ has joined #openstack-kolla09:45
*** chopmann has joined #openstack-kolla09:45
*** salv-orl_ has joined #openstack-kolla09:47
*** salv-orlando has quit IRC09:50
openstackgerritPaul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203609:52
openstackgerritMerged openstack/kolla: Remove /etc/kolla configuration folders  https://review.openstack.org/30542810:00
openstackgerritMerged openstack/kolla: Fix the upgrade issue when image id is the same  https://review.openstack.org/30513310:04
openstackgerritMerged openstack/kolla: Updated from global requirements  https://review.openstack.org/30489510:04
*** tfukushima has quit IRC10:06
*** tfukushima has joined #openstack-kolla10:09
*** andrewvwebber has joined #openstack-kolla10:15
*** mbound has quit IRC10:31
sbezverkpbourke ping10:32
pbourkesbezverk: hi10:32
sbezverkpbourke good morning, I have a quick question. I came accross a couple of wierdnesses in multinode neutron, wanted to know who was working on neutron in kolla..10:34
pbourkesbezverk: everyone and noone10:34
pbourkesbezverk: there's no one "neutron guy" :)10:35
pbourkesbezverk: throw out your problem maybe someone will no10:35
sbezverkpbourke I see, maybe you can help then10:35
pbourke*knpow10:35
pbourke*know10:35
sbezverkpbourke on cimpute node, by default there should not be a connection to external network10:35
sbezverkit is used only if distributed router is deployed10:36
*** mbound has joined #openstack-kolla10:36
sbezverkbut I see the installer tries to create br-ex on all compute nodes10:36
sbezverkI wanted to know if there was a specific reason for that10:36
pbourkeim not sure - is it not needed?10:38
sbezverkpbourke in classical openstack topology, compute node has only vm traffic interface and internal network10:41
*** achanda has joined #openstack-kolla10:44
pbourkesbezverk: sorry im not sure why its the case then :/10:44
sbezverkpbourke not a problem I will check with the rest of team when they join :-)10:45
pbourkesure - it would be good to know10:45
sbezverkpbourke another a bit strange thing I noticed. when you have internal interface and external interface on a controller node, keystone listens only on internel/api interface, basically this prevents anybody from outside of cloud to get authenticated.. Do you know if it was done intentionally?10:50
sbezverkcuase I am ready to file a bug for this10:51
pbourkesbezverk: this is a problem as far as I know10:52
pbourkesbezverk: I submitted a patch for it before but never completed the haproxy config10:52
sbezverkpbourke I see, so no need to file a bug then.. it looks fairly critical issue as it is usability issue, for people who uses this functionality it is a show stopper10:53
pbourkesbezverk: I'll find the original bug and we can update it10:54
pbourkesbezverk: it may be fixed though10:55
sbezverkpbourke if you need any help (I mean if you are busy) working on it, let me know I would be happy to participate10:55
sbezverkpbourke nope I hit it yesterday10:55
pbourkehttps://github.com/openstack/kolla/blob/master/ansible/roles/haproxy/templates/haproxy.cfg.j2#L74-L8110:55
sbezverkI clearly saw keystone was not listening on external interface10:56
pbourkehave you configured kolla_external_vip_address?10:57
sbezverkpbourke you see in the code you pasted it only lsitens on api interface10:57
sbezverkpbourke yes10:57
sbezverkpbourke here is my globals.yml http://paste.openstack.org/show/494038/10:58
pbourkeso its listening on eth1 rather than eth010:58
sbezverkpbourke haproxy works on external perfectly, it is keystone does not listen on this external VIP10:59
pbourkeok seems like a bug10:59
sbezverkpbourke I repro it again and file a bug then10:59
*** sajauddin has joined #openstack-kolla11:03
*** weiyu has quit IRC11:05
*** bmotz_ is now known as bmotz11:07
*** mikelk has quit IRC11:07
*** rhallisey has joined #openstack-kolla11:07
*** mikelk has joined #openstack-kolla11:10
openstackgerritMerged openstack/kolla-mesos: Updated from global requirements  https://review.openstack.org/30297011:30
openstackgerritMerged openstack/kolla: Enable unbuffered output for ansible  https://review.openstack.org/30338911:32
openstackgerritMerged openstack/kolla: Fix missing libvirt python module in ceilometer-compute  https://review.openstack.org/30065011:33
*** mbound has quit IRC11:34
*** salv-orl_ has quit IRC11:35
*** mlima has joined #openstack-kolla11:36
*** mbound has joined #openstack-kolla11:36
mlimahello guys, we need merge it https://review.openstack.org/#/c/304518/11:36
patchbotmlima: patch 304518 - kolla - cleanup-host should not remove kolla-build.conf11:36
openstackgerritMauricio Lima proposed openstack/kolla: Enable unbuffered output for ansible  https://review.openstack.org/30577011:40
openstackgerritMerged openstack/kolla: cleanup-host should not remove kolla-build.conf  https://review.openstack.org/30451811:44
openstackgerritMauricio Lima proposed openstack/kolla: cleanup-host should not remove kolla-build.conf  https://review.openstack.org/30577511:45
*** mikelk has quit IRC11:45
*** achanda has quit IRC11:46
*** vhosakot has joined #openstack-kolla11:49
*** alyson_ has joined #openstack-kolla11:57
*** dwalsh has joined #openstack-kolla12:00
*** vhosakot_ has joined #openstack-kolla12:02
*** bradjones has quit IRC12:02
*** alyson__ has joined #openstack-kolla12:05
*** achanda has joined #openstack-kolla12:05
*** larsks has quit IRC12:06
*** pbourke has quit IRC12:06
*** pbourke has joined #openstack-kolla12:06
*** intr1nsic has quit IRC12:06
*** andrewvwebber has quit IRC12:08
*** andrewvwebber has joined #openstack-kolla12:08
*** bradjones has joined #openstack-kolla12:09
*** bradjones has joined #openstack-kolla12:09
openstackgerritMauricio Lima proposed openstack/kolla: Add a CONTRIBUTING.rst for Kolla  https://review.openstack.org/30540212:09
*** intr1nsic has joined #openstack-kolla12:09
*** alyson_ has quit IRC12:10
*** vhosakot has quit IRC12:10
*** rhallisey has quit IRC12:10
*** alyson__ is now known as alyson_12:10
*** jmccarthy has joined #openstack-kolla12:11
*** larsks has joined #openstack-kolla12:15
*** salv-orlando has joined #openstack-kolla12:19
openstackgerritMerged openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30203612:21
*** rhallisey has joined #openstack-kolla12:22
coolsvappbourke, rhallisey we need some reviews on mitaka https://review.openstack.org/305775 https://review.openstack.org/305770 https://review.openstack.org/305061 https://review.openstack.org/305606 https://review.openstack.org/30558812:24
*** coolsvap has quit IRC12:25
*** vhosakot_ has quit IRC12:25
*** coolsvap has joined #openstack-kolla12:27
*** mwheckmann has joined #openstack-kolla12:27
*** reddy has joined #openstack-kolla12:32
*** sajauddin has quit IRC12:34
*** Guest36812 is now known as zeus12:45
*** zeus has quit IRC12:46
*** zeus has joined #openstack-kolla12:46
*** achanda has quit IRC12:49
*** stvnoyes has left #openstack-kolla12:51
openstackgerritMerged openstack/kolla: Make Heat bootstrapping idempotent  https://review.openstack.org/30506112:54
*** achanda has joined #openstack-kolla12:56
openstackgerritPaul Bourke proposed openstack/kolla: Fix the gate lying  https://review.openstack.org/30420512:59
*** kbyrne has quit IRC13:01
*** chopmann has quit IRC13:06
*** chopmann_ has joined #openstack-kolla13:06
*** chopmann_ is now known as chopmann13:06
openstackgerritPaul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30584013:06
*** jmccarthy has left #openstack-kolla13:08
openstackgerritPaul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30584013:10
*** ravig has joined #openstack-kolla13:10
*** kbyrne has joined #openstack-kolla13:13
*** tuvenen_ has joined #openstack-kolla13:16
*** ph03n1x_ has joined #openstack-kolla13:16
*** tuvenen has quit IRC13:17
*** Serlex has quit IRC13:17
*** lmiccini has quit IRC13:17
*** rhallisey has quit IRC13:17
*** smekel_ has quit IRC13:17
*** ph03n1x has quit IRC13:17
*** jpeeler has quit IRC13:17
*** rbergeron has quit IRC13:17
*** ph03n1x_ is now known as ph03n1x13:18
*** rhallisey has joined #openstack-kolla13:18
*** rbergeron has joined #openstack-kolla13:18
*** Serlex has joined #openstack-kolla13:19
*** jpeeler has joined #openstack-kolla13:19
*** banix has joined #openstack-kolla13:19
*** smekel_ has joined #openstack-kolla13:19
*** lmiccini has joined #openstack-kolla13:20
*** jmccarthy has joined #openstack-kolla13:22
ccesariomorning :)13:23
sbezverkccesario hey, you have multinode setup right?13:24
ccesariosbezverk, hey...13:27
ccesarioyes, with my tests I run multinode13:28
sbezverkccesario what do you use for compute to controller communication for vm traffic?13:30
*** achanda has quit IRC13:31
ccesariosbezverk, could please explain better you question!? eheheh13:33
sbezverkccesario for tenants VMs from compute node, usually they need to reach network or controller node where l3 agents run, right? some tunneling is used either vxlan or gre or vlans13:38
*** achanda has joined #openstack-kolla13:38
sbezverkI wanted to know wha you use.13:39
*** salv-orlando has quit IRC13:40
ccesariosbezverk, let me check the current configs in my last deploy13:41
ccesariosbezverk, default config... currently  ...  tenant_network_types = vxlan13:44
ccesariomechanism_drivers = linuxbridge,l2population13:44
*** achanda has quit IRC13:44
*** mlima_ has joined #openstack-kolla13:45
*** mwheckma1n has joined #openstack-kolla13:45
*** v1k0d3n_ has joined #openstack-kolla13:46
sbezverkccesario I see thank you13:47
*** daneyon_ has joined #openstack-kolla13:47
*** Serlex has quit IRC13:47
*** rmart04_ has joined #openstack-kolla13:47
ccesariosbezverk, what is your doubt ?13:47
*** rmart04 has quit IRC13:47
*** rmart04_ is now known as rmart0413:47
openstackgerritMauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment  https://review.openstack.org/30357513:48
*** tuvenen_ has quit IRC13:48
*** mwheckmann has quit IRC13:48
*** v1k0d3n has quit IRC13:48
*** tonyb has quit IRC13:48
*** banix has quit IRC13:49
*** alyson_ has quit IRC13:49
*** mlima has quit IRC13:49
*** daneyon has quit IRC13:49
*** banix has joined #openstack-kolla13:49
*** tonyb has joined #openstack-kolla13:49
*** alyson_ has joined #openstack-kolla13:50
*** tuvenen has joined #openstack-kolla13:55
*** coolsvap has quit IRC13:56
openstackgerritSerguei Bezverkhi proposed openstack/kolla: tcpdump is missing in the base  https://review.openstack.org/30586913:56
openstackgerritPaul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu  https://review.openstack.org/30584013:56
*** ayoung has quit IRC13:57
*** achanda has joined #openstack-kolla13:57
sbezverkccesario I see some strange behavior and trying to undersatnd its root cause..13:59
*** mgoddard_ has joined #openstack-kolla13:59
*** Serlex has joined #openstack-kolla14:03
*** wmiller has quit IRC14:03
*** mgoddard has quit IRC14:03
*** achanda has quit IRC14:04
*** wmiller has joined #openstack-kolla14:05
*** dougs1 has quit IRC14:06
*** alyson_ has quit IRC14:07
*** banix_ has joined #openstack-kolla14:08
*** banix has quit IRC14:10
*** banix_ is now known as banix14:10
*** dougs has joined #openstack-kolla14:10
*** ravig has quit IRC14:13
*** reddy has quit IRC14:24
*** inc0 has joined #openstack-kolla14:32
inc0good morning14:33
inc0any bug that needs immediate attention?14:33
pbourkeinc0: nothing major has come up today that im aware of14:34
*** tfukushima has quit IRC14:36
ydeshmukhHi All, how to check keystone is running and port 35357 is open?14:40
*** salv-orlando has joined #openstack-kolla14:41
ydeshmukhI'm using centos-source redistro14:41
*** achanda has joined #openstack-kolla14:42
inc0ydeshmukh, it should be visible on host14:42
inc0so netstat -nalp | grep 35357 should do the job14:43
sbezverkinc0 is it normal that keystone listens only on port 5000 on external vip and it does not listen on 35357?14:48
*** salv-orlando has quit IRC14:50
*** shardy has quit IRC14:50
ydeshmukhinc0 actually, I'm facing issue at 'Creating the Glance service and endpoint' in kolla-ansible deploy.14:51
ydeshmukhwhen I try to run same command on kolla_toolbox container,14:51
ydeshmukhansible localhost -m kolla_keystone_service -a "service_name=glance service_type=image description='Openstack Image' endpoint_region=RegionOne url='http://192.168.20.206:9292' interface='admin' region_name=RegionOne auth={{ openstack_glance_auth }}" -e "{'openstack_glance_auth':{'username': 'admin', 'project_name': 'admin', 'password': 'password', 'auth_url': 'http://192.168.20.206:35357', 'domain_name': 'default'}}"14:51
ydeshmukhit's giving error, 'Could not determine a suitable URL for the plugin'14:51
ydeshmukhany idea?14:51
inc0sbezverk, yeah14:53
inc0this is how it should be14:53
inc035357 is not secure, it's meant for admins to fix stuff so access to it should be severely limited14:53
*** ayoung has joined #openstack-kolla14:54
inc0ydeshmukh, can you show your globals.conf please?14:54
sbezverkinc0 but admin still should be able to get authenticated via 5000 port right?14:54
inc0sbezverk, but you need to create first admin account right?:14:55
inc0ayoung, correct me if I'm wrong please, but keystones 35357 doesn't really have authentication?14:55
inc0it only has admin token right?14:55
ayounginc0, wrong14:55
ayoungadmin token should be disabled14:56
inc0from Mitaka right?14:56
ayoungonly use an actual authentication request14:56
ayounginc0, always14:56
inc0but you needed it for bootstraping admin users prior to keystone-manage bootstrap14:56
ayounginc0, keystone-manage bootstrap is supposed to replace ADMIN_TOKEN14:57
ayoungit might be imperfect14:57
ayoungbut I think we did enough in Mitaka that you do not need ADMIN_TOKEN at all14:57
ayoungthink of ADMIN_TOKEN as a bad idea who's time ahs come...to DIE!14:57
*** dougs1 has joined #openstack-kolla14:57
ydeshmukhinc0 I've pasted globals.yml at http://paste.openstack.org/show/494086/14:58
inc0in mitaka we don't have admin token14:58
*** dougs has quit IRC14:59
*** mgoddard_ has quit IRC15:01
*** mgoddard has joined #openstack-kolla15:01
*** dougs1 has quit IRC15:05
inc0ydeshmukh, hmm...looks good15:05
inc0have you  tried clean and redeploy?15:05
ydeshmukhyeah I tried 3-4 times, still same issue.15:06
*** Serlex has quit IRC15:06
inc0docker ps shows all containers running?15:06
ydeshmukhyeah, all are running15:08
*** dougs has joined #openstack-kolla15:08
inc0hmm15:08
inc0sounds like a bug15:08
ydeshmukhat least till glance service creation, http://paste.openstack.org/show/494090/15:09
inc0192.168.20.206 is an unused ip in net right15:09
inc0?15:09
ydeshmukhyeah, unused IP from management network15:09
inc0can you manually ask keystone stuff?15:10
inc0so kolla-ansible post-deploy15:10
inc0will create admin.sh15:10
inc0and then keystone user-list etc15:10
sbezverkayoung which would be then the best approach to authenticate admin trying to access cloud from outside?15:11
ydeshmukhinc0, ok will ask that.15:11
inc0sbezverk, 5000;)15:12
sbezverkinc0 I do have it, it is just after I rebooted my multinode setup, all of a sudden admin could not authenticate nya more15:13
sbezverkplease check this paste: http://paste.openstack.org/show/494091/15:13
*** vhosakot has joined #openstack-kolla15:14
*** mdnadeem_ has joined #openstack-kolla15:14
sbezverkmaybe some leftovers from deployment were allowing admin to authenticate this way, but after reboot it stopped working..15:14
ayoungsbezverk, tokenless X509 ?15:15
ayoungNew feature in Mitaka15:15
ayoungwouldn';t use it for setup though15:15
*** ravig has joined #openstack-kolla15:15
*** ravig has quit IRC15:16
ayounghttps://blueprints.launchpad.net/keystone/+spec/x509-ssl-client-cert-authn15:16
ayounglet me see if I can find a more readable version.  I think we publish the specs now in a some postformatted way15:16
*** mdnadeem has quit IRC15:17
sbezverkayoung thank you.. would be good to have a document15:18
sbezverkinc0 ok I was looking at the wrong place, it was not keystone issue, after reboot mariadb container is flapping on all 3 controllers15:21
inc0yeah15:21
inc0that makes things harder15:21
inc0having working db is useful15:21
sbezverkinc0 now I know it is not just a decoration ;-)15:23
inc0exactly, wouldn't do a good job as such either, it's not particularly pretty15:25
*** chopmann has quit IRC15:26
*** mark-casey has joined #openstack-kolla15:29
sbezverkinc0 nevertheless it looks serious :-( by some reason even though IP connectivity between controller is there one mariadb fails to establish connectivity see this: http://paste.openstack.org/show/494093/15:31
inc0it does15:32
inc0maria is acting up sometimes, we need to figure this one out15:32
inc0connection timed out15:32
inc0did you clean up iptables after last deployment?15:33
inc0sometimes that's the issue15:33
inc0see if there are any leftover rules15:33
ydeshmukhinc0 that was proxy issue after setting no_proxy=192.168.20.206 it got resolved.15:33
inc0ahh there you go15:34
inc0proxy15:34
inc0I hate it15:34
ydeshmukh yeah, I was struggling for 2 days for this small issue.15:36
mlima_inc0, i updated the patchset and now the cleanup files aren't copied to /etc.15:37
mimizonehi all.15:38
mimizonecan the list of interface (ex: api_interface, neutron_interface etc...) be set for every type of node in the ansible inventory or just for the compute node?15:39
mimizonethe comment in the example multinode file is not clear on that and seems to apply only to compute15:39
*** salv-orlando has joined #openstack-kolla15:39
*** vhosakot has quit IRC15:41
inc0mlima_, thanks let me check15:42
*** vhosakot has joined #openstack-kolla15:43
inc0mimizone, it should be for every node15:44
inc0this is ansible mechanism15:44
mimizoneinc0: thanks, I didn't look at the ansible playbooks/scripts, wasn't sure it was used everywhere independently of the openstack role15:44
*** chopmann has joined #openstack-kolla15:45
inc0well, it follows usual ansible variable hierarchy, we didn't do anything new with it15:45
inc0let me know if you run into any issues15:45
*** mgoddard_ has joined #openstack-kolla15:46
*** rajathagasthya has joined #openstack-kolla15:49
*** mgoddard has quit IRC15:49
ccesariosbezverk, I gt some errors like you posted with mariadb15:52
mlima_inc0, ok15:52
*** ravig has joined #openstack-kolla15:55
sbezverkccesario glad I am not alone then ;-)15:56
inc0deploying stable, let see15:57
ccesariosbezverk, what is the exactly  steps to reproduce it ? did you get!?15:57
*** haplo37 has joined #openstack-kolla16:00
*** mbound has quit IRC16:01
*** mark-casey1 has joined #openstack-kolla16:01
inc0sooo...mariadb worked for me16:02
*** dougs has quit IRC16:03
inc0guys can you confirm that you don't have running mysql service?16:03
inc0try to run prechecks16:03
*** mark-casey has quit IRC16:04
*** sdake has joined #openstack-kolla16:05
sbezverkinc0 I see mariadb is restarting on all three controllers16:07
*** slagle_ is now known as slagle16:07
sbezverkccesario I did the initial deployment, then I rebooted all nodes and since that time maridb has gone south16:08
sdakemorning16:08
sbezverkccesario after initial deployment it was working finr16:08
inc0sbezverk, try to cleanup, rebuild all16:09
inc0run prechecks16:09
inc0and redeploy16:09
*** dougs has joined #openstack-kolla16:09
inc0also rebuild with --no-cache16:09
sbezverkinc0 maybe the goal should be not to recover asap, but better understand what is causing it. I will try to dig a bit and try to get some input from galera folks16:11
pbourkesdake: morning16:12
pbourkesdake: all oraclelinux gates working except binary deploy16:12
*** chopmann has quit IRC16:13
pbourkesdake: when are you planning to tag?16:13
ccesariosbezverk, let me try reproduce it16:13
ccesariosdake, hey!16:13
*** unicell has quit IRC16:16
sdakefriday evening16:16
andrewvwebbercoreos kolla works perfectly16:17
sdakeandrewvwebber nice which did ou fix16:17
andrewvwebberi had a bug in my cloudconfig that did not add correctly a veth interface to a bridge16:18
andrewvwebberthe udev messages were a red herring16:18
sdakethats good - that had me worried :(16:18
andrewvwebberi confirmed it with a second network interface as well as a veth added to a bridge16:18
andrewvwebberthat you so much for this great project!16:19
andrewvwebberthank16:19
mark-casey1andrewvwebber: are you just running kolla-ansible against coreos nodes or have you created unit files? or how are you launching the containers?16:19
*** chopmann has joined #openstack-kolla16:19
andrewvwebberi am currently running kolla-ansible16:19
mark-casey1sweet16:19
sdakeif we coud get docs on setup of coreos16:20
andrewvwebberhowever i will look into fleet unit soon as coreos frowns on SSHing into cattle boxes16:20
sdaketha would be fantastic ;:)16:20
andrewvwebberi am happy to write a summary16:20
inc0andrewvwebber, issue you might have is bootstrapping16:20
inc0we do stuff that needs some logic16:20
inc0on deploy16:20
inc0also make sure that things like ceph works, these might be trickier16:21
inc0as it uses udev a bit16:21
andrewvwebberi think the goal of spinning up a machine and never being able to SSH into it, is a direct odds of ansible16:21
andrewvwebberim doing ceph integration next week16:21
mark-casey1there are other connection plugins in ansible though16:21
inc0let me know if I can help you16:21
inc0yeah I think you can do ansible master16:22
inc0never tried it tho16:22
mark-casey1oh... yea pull mode16:22
andrewvwebberi am yet to really train my self on ansible, your guide was excellent enough16:22
inc0ansible is really nice when you get to know it16:23
inc0and kinda bad when you really know it;)16:23
sdakewhy does coreos frown on sshing into cattel box16:23
inc0but it gets better on 2.0 which is our N roadmap16:23
inc0sdake, because you shouldnt ahve to16:23
andrewvwebbermy only problem is i need to learn about openstack ceph integration as i would prefer to seperate the two. i dont like the idea of compute nodes talking directly to OSDs16:24
inc0you should just hit docker API and that's all16:24
inc0andrewvwebber, that's how ceph does stuff tho16:24
andrewvwebber:(16:24
inc0asks mons for crush map16:24
andrewvwebberi like the S3 gateway abstraction with CEPH gateway. no need to be on the same network as an OSD16:24
mark-casey1what's the default value of --threads for kolla-build?16:24
inc0not really :( if you consider amount of data that needs to travel over network16:24
pbourkemark-casey1: 4 i think16:25
mark-casey1ty16:25
inc0I think it's 816:25
inc0but I really suggest to add ansible.cfg16:25
inc0set it up to 100 or so and enable ssh pipelining16:25
inc0makes stuff even quicker16:25
andrewvwebbersdake: i think the coreos security/architecture concept is to not have access to individual machines and encourage cloud like thinking (unicorns vs robots)16:25
pbourkeinc0 is correct16:25
inc0andrewvwebber, but if you consider that proxy will have to transfer terabytes of data potentially16:26
inc0it quickly becomes bottlenech16:26
sdakeandrewvwebber have they stated this ssh thing anywhere?16:26
andrewvwebberinc0: im just thinking from a network security point of view16:26
inc0you really want compute ndoes to talk to osd direcrly, for speed and redudnancy16:26
inc0don't forget that users doesn't have to have access to this network16:27
inc0you can separate tenant networks from management16:27
inc0and you should16:27
andrewvwebbersdake: ssh is on by default and ships in coreos, its just common practice to disable it and use the Fleet cluster manager to deploy workloads16:27
*** achanda has quit IRC16:27
andrewvwebbersdake: if a machine is not working correctly; kill, rebuild, findout why later16:28
andrewvwebberinc0: defence in depth, with CEPH S3 gateway i only need access to the gateways from another network, typically the gateway would have more than one interface16:29
andrewvwebberinc0: but i totally understand16:29
andrewvwebberinc0: CERN performance requirements are not necessarily business security requirements16:30
inc0yeah I understand what you mean and I agree that having this edge ndoes would be more secure16:30
*** kbyrne has quit IRC16:31
inc0but you really lose a lot of nice stuff ceph gives you with it's distributed arch16:31
inc0it's really a arch decision and I'd love to include options16:31
sdakeandrewvwebber how does fleet fit in with kubernetes?16:32
andrewvwebbersdake: fleet to kubernetes is like ansible to openstack/kolla. very poor analogy16:32
andrewvwebbersdake: fleet is used to deploy kubernetes, however it can also be used as a scheduler16:33
sdakei asked how it fits in, not if it is the same16:33
andrewvwebberfleet is used to launch all the prerequisites one would need to deploy kubernetes; certificates, bonding, etcd, etc.16:34
andrewvwebberplus one would deploy kubernetes with fleet16:34
inc0oh, so it might actually be alternative to ansible16:34
andrewvwebbercorrect16:34
inc0if you can figure out bootstrapping16:35
inc0which you should16:35
inc0interesting indeed16:35
mark-casey1inc0: as far as scheduling containers go. I don't think so much on building them16:35
andrewvwebberthis is how one could deploy kolla16:35
inc0mark-casey1, we don't use ansible to build as well16:35
inc0just to schedule containers and configuration stuff16:35
mark-casey1.j2 dockerfiles?16:35
inc0jinja != ansible16:36
inc0we do use jinja, but not ansible16:36
mark-casey1right. ok. I didn't realize you were rendering the templates outside of ansible16:36
inc0https://github.com/openstack/kolla/blob/master/kolla/cmd/build.py16:36
inc0this is where it happends16:37
inc0mark-casey1, we explicitly want to decouple ansible from kolla16:37
inc0so things like coreos fleet or tripleo might use kolla containers just as much16:37
andrewvwebberinc0: that was a great decision16:37
inc0granted, ansible will probably have most logic in it16:38
inc0so other will need to follow16:38
inc0but we will keep it separate and we will keep it possible16:38
*** mgoddard has joined #openstack-kolla16:38
*** mgoddard_ has quit IRC16:38
inc0things like upgrades or reconfigure are ansible specific now16:38
*** dougs1 has joined #openstack-kolla16:38
inc0but with right amount of work it can be replicated wherever16:38
inc0it's not super hard as well, all you need to do is copy-paste-change syntax16:39
inc0logic is there16:39
inc0andrewvwebber, one thing, if you actually make it work with coreos fleet, any chances this can go upstream?16:39
inc0I'd love to have kolla-fleet repo16:39
mark-casey1inc0: would you say the goal is not to move away from ansible within Kolla but to allow non-ansible components from Kolla to work elsewhere?16:40
inc0mark-casey1, or rather non-ansible tools make use of kolla docker containers16:40
andrewvwebberinc0: of course, happy to16:40
inc0so containers will never be ansible-specific16:41
rhalliseyandrewvwebber, would you guys be interested in kolla-kubernetes?16:41
rhalliseyif it were to become a thing?16:41
inc0ansible will be perfectly good, working solution and it will provide a template for others to reproduce16:41
vhosakotmark-casey1: docker/<service name>/Dockerfile.j2 is rendered by jina python client in build.py,  and ansible/roles/<service>/templates/*.j2 files are rendered by Ansible's "template" module16:41
inc0vhosakot, that's correct16:41
inc0mark-casey1, even tho ansible is bulk of kolla code, containers are what really matters about this project16:42
mark-casey1rhallisey: for me that is my end goal. If kubernetes could run hypervisors inside pods already without crazy crazy hax I probably would even have the openstack layer16:42
mark-casey1*wouldn't16:43
*** dougs has quit IRC16:43
rhalliseymark-casey1, well if your at summit we can discuss more about it16:43
inc0mark-casey1, I think we'll have session about k8s in Austin16:43
rhalliseyI have a wip for it16:43
rhalliseyinc0, ya that'd be cool16:43
mark-casey1I've put in a request for it. and actually if I get denied I might come down for an offsite. I'm in Dallas16:44
inc0even if not, that's something we can talk over a beer16:44
sbezverkanybody knows anyone in galera domain?16:44
mark-casey1sbezverk: I'm no guru but yes16:44
andrewvwebberkolla-kubernetes would be interesting but its still not clear how that sit architecturally. launching kolla kubernetes on google container engine for me make little sense as google container engine runs in VMs as far as i know.16:44
mark-casey1I've been running it in stage for our app for a bit now16:44
inc0mark-casey1, I'm from San Antonio, so if you happen to be in area, would gladly share a liqor;)16:45
mark-casey1inc0: cool :(16:45
mark-casey1oops16:45
mark-casey1;)16:45
sbezverkmark-casey1 I see some extremelly odd behavior between three controllers running galera/mariadb, need some help to figure out what might be wrong..16:46
*** chopmann has quit IRC16:46
vhosakotmark-casey1: I dont like to use the phrase "jinja template gets rendered" as "rednering" of *.j2 is something that happens on the UI if we use *.j2 to create HTML or PHP...I think *.j2 templates get "built" by docker after variables are substituted at runtime... I think "render" is "built" in kolla :)16:46
*** kbyrne has joined #openstack-kolla16:47
mark-casey1sbezverk: sure happy to try and help16:47
vhosakotor, may be we could say render is built... I'm used to thinking rendering as something that happens on an UI on screen :)16:47
andrewvwebberrhallisey: has anyone asked for kolla-docker-swarm?16:47
mlima_we need some reviews on mitaka https://review.openstack.org/#/c/305840/ https://review.openstack.org/#/c/305775/ https://review.openstack.org/#/c/305588/ https://review.openstack.org/#/c/305770/ https://review.openstack.org/#/c/305606/16:47
patchbotmlima_: patch 305840 - kolla (stable/mitaka) - Use proper CentOS repositories for ceph and qemu16:47
patchbotmlima_: patch 305775 - kolla (stable/mitaka) - cleanup-host should not remove kolla-build.conf16:47
patchbotmlima_: patch 305588 - kolla (stable/mitaka) - Added help documentation for kolla-ansible upgrade16:47
patchbotmlima_: patch 305770 - kolla (stable/mitaka) - Enable unbuffered output for ansible16:47
patchbotmlima_: patch 305606 - kolla (stable/mitaka) - Fixes kolla_toolbox failure due to missing pytz mo...16:47
rhalliseyandrewvwebber, no16:48
vhosakotmlima_: will review16:48
mark-casey1vhosakot: yea that makes some sense16:48
rhalliseyandrewvwebber, I'd say were still expanding the kolla-* projects16:48
mark-casey1to me16:48
rhalliseysince we really only have 1 other (kolla-mesos)16:48
*** chopmann_ has joined #openstack-kolla16:48
*** chopmann_ is now known as chopmann16:48
rhalliseywe plan to split out kolla & kolla-ansible16:48
mark-casey1sbezverk: is this re: the paste earlier about reaching primary view16:49
rhalliseyto make 216:49
sbezverkmark-casey1 here is the log I collected on one of the controller http://paste.openstack.org/show/494103/ it looks like it is connectivity issue but IP unicast is working fine between them..16:49
andrewvwebberrhallisey: i think kolla-kubernetes would bring alot of attention to the project, at the very least. the 'perfect' hybrid solution16:49
mark-casey1sbezverk: this was post-reboot. is that right?16:49
sbezverkmark-casey1 yes, but that one was partial.. here is kind of complete cycle16:49
sbezverkmark-casey1 yes sir16:49
mlima_vhosakot, I Agree16:50
rhalliseyandrewvwebber, to me, I think it would do well on the controller.  I don't know how well kolla-kuberentes would do for compute nodes16:50
rhalliseykube doesn't seem to scale16:50
mark-casey1sbezverk: how clean was the shutdown? is there any chance that the first node being bootstrapped is a commit or two behind another node?16:51
mark-casey1err... db commit of course16:51
sbezverkmark-casey1 well I just rebooted nodes, exactly the same way I do with classic openstack.16:52
sdakerhallisey coreos hsa a partnership with intel working on openstack on kubernetes in containers16:52
sdakethe logical step to take ther eis to make ea kolla-kubernetes repo and get to work with them on it16:52
sbezverkmark-casey1 "160414 16:44:42 [Warning] WSREP: no nodes coming from prim view, prim not possible" this specific line means peers cannot communicate or all peers have sort of corruption?16:52
rhalliseysdake, right.  Maybe a post summit project16:52
andrewvwebberrhallisey: yes, its been a long time since i used kubernetes but my understanding is that it does not have tenant isolation16:53
inc0andrewvwebber, you work with intel folks on this?16:53
inc0care to share names?16:53
mark-casey1sbezverk: it's a slim case but if the first node dropped and the cluster still had quorum the other two nodes could have commited a change before shutting down. if you bootstrap on the member that was a commit behind you could maybe get some errors like you're getting16:54
andrewvwebbersdake: totally agree, i have asked them about the repo in IRC but they are in "watch this space" mode16:54
mark-casey1sbezverk: this is just a test cluster right?16:55
rhalliseyandrewvwebber, we'll try and talk about it at summit and maybe we can come out of it with a new repo16:55
mark-casey1http://galeracluster.com/documentation-webpages/quorumreset.html16:56
andrewvwebberrhallisey: it would be in both parties interest. i think it doesnt make sense for coreos to maintain a seperate repro from kolla16:56
sbezverkmark-casey1 hmmmm, if just normal reboot cause cause this nightmare, maybe we need a script to shutdown services gracefully before rebooting node16:56
rhalliseyandrewvwebber, are you going to be at summit?16:56
andrewvwebberrhallisey: just like to contribute back to kubernetes as their tectonic offering is managed kubernetes.16:57
rhalliseyalong with some of your colleagues16:57
mark-casey1sbezverk: it may not have this time but my understanding is it absolutely can on a busy cluster. it's my biggest problem left to solve...: when recovering my ansible stuff has to attempt bootstrap on the most advanced node16:57
mark-casey1sbezverk: http://galeracluster.com/documentation-webpages/galeraparameters.html#pc-bootstrap16:58
mark-casey1if this is the problem that particular option will probably fix it16:58
mark-casey1but, if you do it on a node that was behind, the couple commits that the other two nodes were ahead will get a full SST and those commits will be lost16:59
sdakeandrewvwebber mind i nosey around where you worik?16:59
andrewvwebberrhallisey: i would love...16:59
sbezverkmark-casey1 so if I change galera.conf and add this option it might recover right?17:00
mark-casey1sbezverk: yea thats the theory17:00
sbezverkmark-casey1 so right now all container with mariadb are flapping, I cannot change this parameter on the fly, but I can add it to json config, do you think having this parameter permanently will have any negative imapct?17:01
sdakeandrewvwebber watch which sapce17:02
andrewvwebbersake: sure, i work for a security company in munich germany doing B2B file sync and share. we're a golang shop on couchbase, ceph and baremetal coreos. im the architect trying to make VMs and containers co-exist17:02
sdakeandrewvwebber not trying doing :)17:03
*** chopmann has quit IRC17:03
*** chopmann has joined #openstack-kolla17:03
mark-casey1sbezverk: I'm actually on percona's galera; afaik I have to pass this when bootstrapping. assuming you can make this permanent I would not. my understanding is that it tells the cluster to ignore split brain and roll back to last common state17:03
mark-casey1primary component =~ the synchronous cluster's quorum17:04
andrewvwebbersake: standing on the shoulders of giants17:04
mark-casey1***I have to pass then when bootstrapping after bad things have happened and assuming I don't mind losing some stuff :D17:07
sdakedo we need percona17:08
sdakerdo repos contain galera now17:08
mark-casey1sdake: these issues seem to be wholly related to the galera plugin and afaik are common to maria and all other users17:08
sbezverkmark-casey1 I do not see a way to recover then :-( it flaps too fast to be able to update that file in between flaps17:10
mark-casey1sbezverk: have you tried stopping all nodes and bootstrapping each one, one at a time?17:11
mark-casey1if this is the issue. I would think one of them woudl work17:11
*** andrewvwebber has quit IRC17:11
mark-casey1sbezverk: can you see if you have a file called grastate.dat?17:11
sbezverkmark-casey1 let me try that17:12
mark-casey1ok. try first. then file17:12
*** athomas has quit IRC17:14
mark-casey1sorry if this is blasphemous but I think osa has dealt with this some too. https://github.com/openstack/openstack-ansible/blob/master/doc/source/install-guide/ops-galera-recovery.rst17:14
mark-casey1the multi-node failure section... as a shutdown without tiptoe-ing can more or less be a multi-node fail ;?17:15
mark-casey1*    :/17:15
sbezverkmark-casey1 with single node still flaps17:16
sbezverkmark-casey1 I see this file /var/lib/mysql/grastate.dat17:17
sdakemark-casey1 we have mariadb lights out recovery playbook17:17
sdakemark-casey1 i think that solves this problem17:17
sbezverksdake could you point me to the command to initiate it? it would be great test to see if it is working..17:18
*** vhosakot_ has joined #openstack-kolla17:18
sdakesbezverk nevr tried it let me see what i can find17:18
sbezverkmark-casey1 what that file means?17:18
sdakeneed to drop of the vpn moment17:18
mark-casey1sbezverk: that file gets removed on clean cluster shutdown. it records extra state for restarting cluster. do any nodes have a value of seqno: that is not -1?17:19
*** vhosakot has quit IRC17:20
*** mwheckma1n has quit IRC17:20
*** kbyrne has quit IRC17:21
*** ayoung has quit IRC17:21
*** mlima_ has quit IRC17:21
*** jmccarthy has quit IRC17:21
*** rbergeron has quit IRC17:21
*** larsks has quit IRC17:21
sbezverkmark-case1 node 1 http://paste.openstack.org/show/494111/17:21
*** mlima_ has joined #openstack-kolla17:21
sbezverkmark-casey1 sorry for misspel17:21
*** rmart04 has quit IRC17:21
mark-casey1no worries. also to double check you need a node with a value != -117:21
sbezverkmark-casey1 all nodes have seq -117:22
mark-casey1ok17:22
*** rbergeron has joined #openstack-kolla17:22
*** mwheckmann has joined #openstack-kolla17:22
sbezverkmark-casey1 if I remove this file on all nodes?17:23
mark-casey1I know this isn't maria specific but it should mostly apply IF the lights-out option doesn't cover this. look at Scenario 6 https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/17:23
*** sdake has quit IRC17:23
mark-casey1oh... actually is looks like an automatic solution is coming for this or is already in galera17:24
sbezverk<mark-casey1 thank you for the link if lights out does not help, will play with suggested workaround, really appreciate you sharing these very useful links.17:24
mark-casey1my pleasure17:25
*** banix has quit IRC17:25
*** sdake has joined #openstack-kolla17:26
sdakesbezverk bck17:26
sdakesbezverk kolla-ansible mariadb_recovery17:26
mark-casey1looking at that playbook I think it will work17:28
mark-casey1I think there are cases it could cause data loss though17:31
sbezverkok starting it17:31
sbezverksdake mark-casey1 so far looks good it is not flapping, need to check if I lost any data17:32
mark-casey1probably won't show in this case17:32
sdakesbezverk after running that playbook yu mean?17:32
sbezverksdake yes17:32
sdakehotness17:32
mark-casey1that method passes --wsrep-new-cluster which creates a new quorum from whatever state the bootstrapping node happens to have17:33
mark-casey1galera docs:  Note Warning: Only use the --wsrep-new-cluster argument when initializing the Primary Component.  Do not use it when you want the node to connect to an existing cluster.17:33
sbezverksdake mark-casey1 wow data is still there !! great..17:33
mark-casey1in this case even though they didn't think they had quorum the data was all probably the same17:33
*** larsks has joined #openstack-kolla17:34
mark-casey1if a node had gone down a week ago and lights out happened to bootstrap from it it'd be sad panda17:34
sbezverksdake but then there should be a procedure to gracefully shut down17:34
*** unicell has joined #openstack-kolla17:34
sdakesbezverk should be and are ae two different htings :)17:34
sbezverksdake for sure, I am just checking if I missed it or it does not exist17:35
*** ayoung has joined #openstack-kolla17:35
*** mark-casey has joined #openstack-kolla17:36
mark-casey...wifi dropped or something17:36
*** dwalsh has quit IRC17:37
*** jmccarthy has joined #openstack-kolla17:38
*** kbyrne has joined #openstack-kolla17:38
*** mark-casey1 has quit IRC17:38
*** ravig has quit IRC17:40
mark-caseysdake sbezverk imho this method needs a disclaimer that you could lose data and some steps on how to estimate how much could be lost, but a drunken guess is that it is in fact the correct thing to do 80-90% of the time17:42
openstackgerritMerged openstack/kolla: Added help documentation for kolla-ansible upgrade  https://review.openstack.org/30558817:42
sdakecan i get naother core reviewer to take a look at this review https://review.openstack.org/#/c/302434/517:43
patchbotsdake: patch 302434 - kolla - Set db connection retry to infinity17:43
*** salv-orlando has quit IRC17:44
sdakesbezverk where is the review for this https://bugs.launchpad.net/kolla/+bug/155389517:45
openstackLaunchpad bug 1553895 in kolla "Drop root for swift" [High,In progress] - Assigned to Serguei Bezverkhi (sbezverk)17:45
*** mgoddard_ has joined #openstack-kolla17:48
*** sdake_ has joined #openstack-kolla17:48
*** mark-casey1 has joined #openstack-kolla17:49
*** mkoderer__ has joined #openstack-kolla17:49
*** unicell has quit IRC17:50
*** mkoderer___ has quit IRC17:50
*** unicell has joined #openstack-kolla17:50
*** unicell has quit IRC17:50
*** unicell has joined #openstack-kolla17:50
sdake_sbezverk when you find the droproot swift patch please link it in the bug17:50
sdake_for some reason openstack infra didnt link it17:51
*** mark-casey has quit IRC17:51
*** sdake has quit IRC17:51
*** mgoddard has quit IRC17:51
mark-casey1bbl food17:51
*** mark-casey1 has quit IRC17:51
*** dwalsh has joined #openstack-kolla17:55
*** rajathagasthya has quit IRC17:58
*** salv-orlando has joined #openstack-kolla18:01
*** ravig has joined #openstack-kolla18:02
*** rajathagasthya has joined #openstack-kolla18:03
*** sdake_ is now known as sdake18:04
*** ravig has quit IRC18:08
sdakeok folks we have 5 criticcal/high bugs left18:09
sdakepease please try to get them merged by tomorrow morning18:10
sdakeinc0 can you jut ack ryan's work18:10
sdakei understand jeffrye's complaint but i think infinity is the right answer18:10
*** ravig has joined #openstack-kolla18:11
sdakeinc0 https://review.openstack.org/#/c/302434/18:11
patchbotsdake: patch 302434 - kolla - Set db connection retry to infinity18:11
*** ravig has quit IRC18:13
sdakevhosakot_ ^^18:13
vhosakot_sdake: cool, will review18:14
*** ravig has joined #openstack-kolla18:15
*** ravig has quit IRC18:17
*** sdake_ has joined #openstack-kolla18:17
*** sdake has quit IRC18:19
*** andrewvwebber has joined #openstack-kolla18:21
*** ravig has joined #openstack-kolla18:22
*** gfidente has quit IRC18:28
sdake_sbezverk drop root for swift - status?18:28
*** Marga__ has quit IRC18:30
vhosakot_sdake_: Jeffrey and mlima_ have -1 for https://review.openstack.org/#/c/302434/.. the PD LGTM18:31
patchbotvhosakot_: patch 302434 - kolla - Set db connection retry to infinity18:31
sdake_lets just get it merged18:32
*** sdake_ is now known as sdake18:32
sdakei dont want to block on figuring out the "perfect" numbeer18:32
sdakeimo the perfect number is infinity18:32
sdakewe dont have time to eresearch what the perfect number is18:32
*** achanda has joined #openstack-kolla18:33
ccesariosbezverk, could you make available your inventory file ?18:36
*** achanda has quit IRC18:37
openstackgerritAlicja Kwasniewska proposed openstack/kolla: Add Kibana quick start quide  https://review.openstack.org/30601918:38
vhosakot_sdake: yes, I agreee infinity is fine... let the services try to reconnect forever..18:42
sdakeso ack the patch then ;)18:42
*** egonzalez has joined #openstack-kolla18:45
*** egonzalez has quit IRC18:45
*** egonzalez has joined #openstack-kolla18:46
*** ravig_ has joined #openstack-kolla18:48
sdakewtb ack on https://review.openstack.org/#/c/302434/18:50
patchbotsdake: patch 302434 - kolla - Set db connection retry to infinity18:50
sbezverksdake ok will take care of swift drop root18:50
*** ravig has quit IRC18:50
sdakesbezverk friday is deadline18:51
*** banix has joined #openstack-kolla18:53
sbezverksdake it has a right bug id, it is juts missing one +218:54
sdakeink plz18:54
sdakedoe it work?18:54
mimizoneWhat is expected from this precheck "Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes"?? Can't seem to find what to do to not fail.18:54
sbezverkhttps://review.openstack.org/#/c/259374/18:54
patchbotsbezverk: patch 259374 - kolla - Drop root for swift18:54
sbezverksdake yep18:54
*** andrewvwebber has quit IRC18:55
mimizoneshould the VIP be completely on a different network that is not on any interfaces on the hosts?18:55
openstackgerritSteven Dake proposed openstack/kolla: Drop root for swift  https://review.openstack.org/30603018:56
*** rmart04 has joined #openstack-kolla18:56
mimizoneanswering my own question. putting the VIP on a different network than is not defined on any interface seems to be accepted by the precheck18:57
mimizoneoh no correcting myself now...18:58
mimizonenot working...18:58
*** Marga_ has joined #openstack-kolla18:58
*** rmart04 has quit IRC18:58
ccesarioelemoine_, https://bugs.launchpad.net/kolla/+bug/157055018:58
openstackLaunchpad bug 1570550 in kolla "elasticsearch.url param point to host address when haproxy it is enabled" [Undecided,New]18:58
mimizonedoes this error message make sense to somebody? FAILED! => {"failed": true, "msg": "The conditional check ''169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])' failed. The error was: error while evaluating conditional ('169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[318:58
mimizone])): list object has no element 3"}18:58
vhosakot_mimizone: ping18:59
vhosakot_what is your network_interface and kolla_internal_vip_address in globals.yml ?18:59
mimizonefor instance kolla_internal_vip_address: "172.30.4.100"18:59
openstackgerritMerged openstack/kolla: Drop root for swift  https://review.openstack.org/25937418:59
mimizonenetwork_interface: "eth2.2002"18:59
mimizonebut I also give a different set of interface per host cause I have different hardware19:00
vhosakot_mimizone: can you pastebin your globals.yml and output of "ip a"19:00
mimizoneip a?19:00
mimizoneon which node?19:01
openstackgerritMerged openstack/kolla: Set db connection retry to infinity  https://review.openstack.org/30243419:01
vhosakot_mimizone: globals.yml from deploy node and "ip a" from deploy node as well19:02
*** sdake has quit IRC19:02
mimizonevhosakot_: http://paste.openstack.org/show/494120/19:05
*** sdake has joined #openstack-kolla19:05
ccesariosdake, this ps https://review.openstack.org/#/c/302434/  solve it ?    http://paste.openstack.org/show/494121/19:06
patchbotccesario: patch 302434 - kolla - Set db connection retry to infinity (MERGED)19:06
ccesariovhosakot_, ^^19:06
vhosakot_ccesario: checking... thanks19:07
openstackgerritSteven Dake proposed openstack/kolla: Set db connection retry to infinity  https://review.openstack.org/30603619:07
daneyon_vhosakot_ Do you know if any patches merged that fixes this bug? https://bugs.launchpad.net/kolla/+bug/156765719:07
openstackLaunchpad bug 1567657 in kolla "Build Fails Using Trunk" [Undecided,New]19:07
sdakehttps://review.openstack.org/#/c/306036/1 needs an ack19:07
patchbotsdake: patch 306036 - kolla (stable/mitaka) - Set db connection retry to infinity19:07
rhalliseyccesario, that's a different issue19:08
vhosakot_daneyon_: no, retrying works for me on UBuntu... are you using CentOS19:08
daneyon_vhosakot_ yes, I'm using centos. My use case req's centos19:09
ccesariorhallisey, thank you.... i 'm making a double check to confirm it19:09
vhosakot_daneyon_: the error is not seen in gate.. I've been checking if centos gate sees it and it does not19:10
daneyon_vhosakot_ OK. I just started a fresh build. I'll see if I hit it again and update the bug.19:11
vhosakot_daneyon_: ok, please.. thanks!19:11
*** mark-casey has joined #openstack-kolla19:13
vhosakot_mimizone: can you paste the output of  "ip -4 -o addr show dev eth2.2002" ?19:15
vhosakot_mimizone: from deploy mode19:15
mark-caseyrhallisey inc0 my request for attending summit was approved19:15
rhalliseysweet19:15
mimizonevhosakot_: http://paste.openstack.org/show/494122/19:16
vhosakot_mimizone: 1 sec.. let me check19:16
mimizoneI think I may have found the thing. the VIP shoudl be in the IP range19:17
vhosakot_mimizone: yes :)19:17
*** ravig_ has quit IRC19:17
vhosakot_mimizone: I was gonna say that19:17
mimizonewe are still configuring things around, but we use different IP ranges per rack19:18
*** ravig has joined #openstack-kolla19:18
vhosakot_mimizone: kolla_internal_vip_address (172.30.81.100) is _not_ in the range 172.30.81.153/2519:18
mimizoneand our DHCP/PXE server for this setup is still being configured for that (Maas).19:18
sdakehttps://review.openstack.org/#/c/306037/1/reference/tags/release_cycle-trailing.rst19:18
patchbotsdake: patch 306037 - governance - add release:cycle-trailing tag19:18
mimizoneCool confirming is good19:18
*** ravig has quit IRC19:19
vhosakot_mimizone: kolla needs kolla_internal_vip_address to be in the same network range as network_interface (eth2.2002) which is 172.30.81.153/25... hence, chane kolla_internal_vip_address to a free IP in the range 172.30.81.128 - 172.30.81.25519:20
inc0mark-casey, yay!19:20
mimizoneI'll see how to play around this. I can do a quick test in the same range, but ultimately the VIP should the in the larger /21 of the same subnet. We slice a large /21 across 16 racks to make /25.19:21
*** ravig has joined #openstack-kolla19:21
mimizonethanks vhosakot_19:21
vhosakot_mimizone: cool!... it is kolla's requirements... else, services will have issues  -->kolla needs kolla_internal_vip_address to be in the same network range as network_interface19:22
vhosakot_mimizone: search for "unused" in http://docs.openstack.org/developer/kolla/quickstart.html19:23
mimizonebut our controllers may end up in different racks, therefore in a different /2519:23
*** ravig has quit IRC19:29
*** ravig has joined #openstack-kolla19:30
vhosakot_mimizone: all the nodes must be in the same network19:31
*** ravig has quit IRC19:31
mimizonewhat is the reason for that? we route at layer3 in the top of rack switch. all nodes see each other at the IP level.19:32
*** ravig has joined #openstack-kolla19:32
mimizoneis that only for the VIP the requirement? so for the API_interface?19:32
vhosakot_mimizone: mmm, in that case, will kolla_internal_vip_address will be outside the range of network_interface (outside of 172.30.81.153/25 in this case)19:33
mimizonethe vip would be in the corresponding 172.30.80.0/2119:36
vhosakot_mimizone: api_interface as well.. yes.. https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L8119:37
mimizoneI expect other surprises then :)19:39
*** mark-casey has quit IRC19:40
*** rajathagasthya has quit IRC19:40
*** rajathagasthya has joined #openstack-kolla19:45
*** chopmann has quit IRC19:46
*** sdake_ has joined #openstack-kolla19:47
*** sdake has quit IRC19:49
sbezverkinc0 ping19:53
mimizonevhosakot_: I sort of understand why the vip shouldbe in the same network as the controllers, but I don't understand why there is a check on the other roles. As long as the VIP is routed from those nodes, that should be enough. the Storage nodes don't need to care if the VIP is in a totally different network, or at least Ip range.19:54
mimizoneDon't you think?19:54
*** akscram has quit IRC19:58
*** rajathagasthya has quit IRC19:58
mimizonealso the precheck doesn't use the right interface for the check. it uses the value from the globals 'network_interface' and not the api_interface, neutron_interface etc... values I put in the inventory. should the network_interface also be overridden in the inventory?19:59
*** akscram has joined #openstack-kolla19:59
vhosakot_mimizone: api_interface is same as network_interface  https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L8120:01
mimizoneI overridden them in the inventory. is that ignored?20:01
vhosakot_mimizone: you mean in globals.yml ?20:02
mimizonenope in the ansible inventory file20:02
mlima_guys, we need to review here https://review.openstack.org/#/c/306036/ https://review.openstack.org/#/c/306030/ https://review.openstack.org/#/c/305840/ https://review.openstack.org/#/c/305775/ https://review.openstack.org/#/c/305770/20:02
patchbotmlima_: patch 306036 - kolla (stable/mitaka) - Set db connection retry to infinity20:02
patchbotmlima_: patch 306030 - kolla (stable/mitaka) - Drop root for swift20:02
patchbotmlima_: patch 305840 - kolla (stable/mitaka) - Use proper CentOS repositories for ceph and qemu20:02
patchbotmlima_: patch 305775 - kolla (stable/mitaka) - cleanup-host should not remove kolla-build.conf20:02
patchbotmlima_: patch 305770 - kolla (stable/mitaka) - Enable unbuffered output for ansible20:02
mimizonevhosakot_: for exemple "osv11syn21b ansible_user=ubuntu neutron_external_interface=eth2 api_interface=eth2.2002 storage_interface=eth2.2002 tunnel_interface=eth2.2002"20:03
vhosakot_mimizone: here  https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode#L21-L22 ?20:03
*** ayoung has quit IRC20:08
ccesariosbezverk, ping ?20:09
ccesariosbezverk, really the problem happen20:11
ccesariosbezverk,  http://sprunge.us/dHXL20:11
ccesariosbezverk, http://paste.openstack.org/show/494125/20:13
ccesariothe mariadb container stay rebooting20:14
inc0sbezverk, pong, wassup?20:14
*** egonzalez has quit IRC20:21
*** ravig has quit IRC20:22
ccesariosbezverk, steps to reproduce http://paste.openstack.org/show/494127/20:26
ccesarioinc0, did you test ?20:26
*** inc0 has quit IRC20:27
*** rmart04 has joined #openstack-kolla20:28
*** rajathagasthya has joined #openstack-kolla20:31
*** rmart04 has quit IRC20:35
*** ravig has joined #openstack-kolla20:41
ccesarioguys...if someone could test ..... I did double check and the error happen20:43
ccesariosbezverk, ^^20:43
*** banix has quit IRC20:47
*** banix has joined #openstack-kolla20:48
*** banix has quit IRC20:50
*** ccesario_ has joined #openstack-kolla20:57
*** banix has joined #openstack-kolla21:00
*** ravig has quit IRC21:07
*** ccesario_ has quit IRC21:07
*** ccesario_ has joined #openstack-kolla21:10
*** ccesario_ has quit IRC21:12
*** ccesario___ has joined #openstack-kolla21:12
*** ravig has joined #openstack-kolla21:13
*** ccesario___ has quit IRC21:17
*** salv-orl_ has joined #openstack-kolla21:17
*** ccesario___ has joined #openstack-kolla21:17
*** salv-orlando has quit IRC21:20
*** dwalsh has quit IRC21:22
*** ccesario___ has quit IRC21:23
*** ccesario___ has joined #openstack-kolla21:24
*** ccesario___ has quit IRC21:29
vhosakot_mimizone: I'm still thinking about your use case.. we need to chat more21:36
*** mark-casey has joined #openstack-kolla21:39
mark-caseydoes Kolla support centos prior to version 7?21:41
*** ccesario_ has joined #openstack-kolla21:43
*** haplo37 has quit IRC21:44
vhosakot_mark-casey: I dont think it is tested.. dont know if we get the needed repos in centOS 621:45
mark-caseyvhosakot_: ty21:45
ccesario_vhosakot_: ping ?21:47
vhosakot_mark-casey: this is how we point to 7  https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L2221:48
vhosakot_ccesario_: pong21:48
ccesario_vhosakot_: did you read my last messages there in channel ?21:48
ccesario_I think that we have problems with multinode and mariadb as previously said by sbezverk21:49
vhosakot_ccesario_: steps to repproduce mariadb reboot issue ?21:49
ccesario_yes man21:49
vhosakot_yes saw that21:49
ccesario_I repeat it 3 times  and in all tests the problem happen21:50
vhosakot_ccesario_: you reboot both control nodes 192.168.201.2 and 192.168.201.3 simultaneously, right ?21:50
ccesario_yes21:51
ccesario_and IMHO this need to be checked ... https://bugs.launchpad.net/kolla/+bug/157055021:51
openstackLaunchpad bug 1570550 in kolla "elasticsearch.url param point to host address when haproxy it is enabled" [Undecided,New]21:51
ccesario_because this no make sense when haproxy t is enabled .....21:52
*** ravig has quit IRC21:53
ccesario_in my tests  these are my reports :)  and deadline is tomorrow according sdake_21:54
vhosakot_ccesario_: that bug is not targedted for 2.0.0.. is it ?21:54
ccesario_vhosakot_: because I have checked it yesterday and discussed with elemoine_21:55
vhosakot_ah ok...cool21:56
ccesario_vhosakot_: :)21:57
ccesario_I'll back in 2 hours... need slve a customer problem now :/21:59
vhosakot_ccesario_: cool21:59
*** ravig has joined #openstack-kolla22:01
*** pbourke-home has joined #openstack-kolla22:02
*** ravig has quit IRC22:02
*** allen_gao has joined #openstack-kolla22:04
*** rajathagasthya has quit IRC22:04
mimizonevhosakot_: sure, let's do that. I have some ideas I discussed with a friend22:11
*** dwalsh has joined #openstack-kolla22:15
*** ravig has joined #openstack-kolla22:17
*** mwheckmann has quit IRC22:18
*** banix has quit IRC22:23
mimizonevhosakot_: in a nutshell, one work around to make it compatible with kolla would be to have a vip unrelated with the network interface. the IP would be advertized to the rest of the BGP fabric using a BGP client on the controllers. If we also use ECMP like we already do in the fabric, the VIP could actually be on all the controllers at the same time, and the fabric would route appropriately using ECMP algo. the other nod22:23
mimizonees, non-controllers, should not care if the VIP is on the same network or not.22:23
mimizonethe quickest work around for me right now, is to switch back to layer2 for the api network. But I don't like this...22:24
vhosakot_mimizone: so, you control, network, storage nodes are all on different networks routable thru layer-3 thru TOR ?22:24
mimizonevhosakot_: yes22:25
mimizonemore like all racks are on different networks22:26
mimizoneand the tor is a router acting as the default gateway for the rack22:26
vhosakot_mimizone: yes, that is very much valid cloud topology... separating control, network, storage nodes..22:26
mimizonebgp is used to advertise the routes across racks22:26
vhosakot_mimizone: so, the TOR device that sees all the networks.. is that used for kolla at all ?22:27
*** ph03n1x has quit IRC22:27
mimizonevhosakot_: not sure I get the question.22:27
vhosakot_mimizone: is the device that routes packets from controller's network to storage network used in the kolla inventory ?22:28
mimizonevhosakot_: no. kolla doesn't configure anything in the network/tor/router22:29
vhosakot_mimizone: there must a layer-3 point in the topology that sees/knows all the networks and can route packets from one network to other... does this point get routes advertised frm BGP clinet running on controller ?22:29
mimizoneoh ok, yes the ro sees all the networks in the rack22:30
mimizoneand if it is outside of the rack, it goes up to one of the spines22:30
mimizonebased on ECMP/BGP22:30
vhosakot_mimizone: yep...22:30
mimizoneif it's outside of the fabric, it goes to our main router for the room22:30
vhosakot_mimizone: right22:30
mimizoneI am thinking of removing the checks for the non-controller nodes for now, and try a deployment with all the controllers in the same rack for now.22:32
vhosakot_mimizone: sounds like this precheck does not make sense in your topology,, can you comment it out and check if kolla works end-to-end and you can successfully create nova VM... https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L459-L464.. if so, yes, it is a bug in prechecks and I will fix it...22:32
vhosakot_mimizone: could you do this test as you have a great topology! :)22:32
mimizonecool, we are aligned.22:33
mimizoneI'll keep you posted22:33
mimizoneand then we can discuss how to integrate maybe some bgp client in the controller to advertize the VIP to the fabric22:33
vhosakot_mimizone: thanks.. looks forward to hear back!22:33
mimizonecan you point me to the file where the check is?22:33
vhosakot_mimizone: https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L459-L46422:34
vhosakot_mimizone: yes, having BGP client on control nodes and using this BGP client to advertise the VIP to the fabric north of (beyond) the TOR is a new and very valid use case, and needs to be addressed22:35
vhosakot_mimizone: thanks for checking this!22:35
mimizoneI am not planning to be in Austin btw, but would welcome being part of the discussions on that22:36
mark-casey322:37
mark-caseylol, sorry22:37
vhosakot_mimizone: cool, I will work with you to fix and address this use case... as the first step, it would be nice to know if kolla works end-to-end and if nova VM can be booted successfully wihout that percheck in https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L459-L46422:38
vhosakot_I'm in this channel most of the time22:38
*** ayoung has joined #openstack-kolla22:39
mimizonevhosakot_: running the prechecks now.22:41
vhosakot_mimizone: cool22:41
mimizonevhosakot_: I expect a full deployment may require additional work, and not only a kolla-deploy, but let's see22:42
*** pbourke-home has quit IRC22:42
mimizoneI am confused, I commented the entire tasks in port_checks related to this point, and it is still ran22:44
mimizoneis there some cache in ansible??22:46
mimizoneI must be modifying the wrong file. I am editing the file in ~/kolla/ansible/roles....  that I checked out of git. is there another one? I commented out the entire port_checks in the main.yml too, still running...22:52
mimizoneit gets it from /usr/local/share....22:54
*** asalkeld has quit IRC23:07
*** ayoung has quit IRC23:23
*** vhosakot_ has quit IRC23:24
*** mark-casey has quit IRC23:27
*** asalkeld has joined #openstack-kolla23:29
*** inc0 has joined #openstack-kolla23:35
*** ravig has quit IRC23:39
sbezverkinc0 ping23:46
*** dwalsh has quit IRC23:50
*** ravig has joined #openstack-kolla23:53
*** inc0 has quit IRC23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!