vhosakot | o/ | 00:00 |
---|---|---|
tonyb | sdake: for reference https://www.openstack.org/summit/austin-2016/summit-schedule/events/9474 | 00:01 |
rhallisey | anyone else see the kubernetes presentation?> | 00:01 |
tonyb | is where we'll talk about how long each stable branch lives for | 00:01 |
rhallisey | keynote at summit | 00:01 |
sdake | ok folks i 've got a meeting now | 00:01 |
sdake | tonyb ack | 00:01 |
sdake | inc0 can you sort this out s you are already doing :) | 00:01 |
sdake | tonyb would it be possible to add a stable follows policy exception for 1.0.0 release | 00:02 |
inc0 | sdake, tonyb can you discuss tags later on? I don't know much about that stuff | 00:02 |
sdake | to allow fresh projects to et in the grove | 00:02 |
sdake | i mean would you be open to that? | 00:02 |
inc0 | and be vigilant about policy afterwards | 00:03 |
tonyb | sdake: I'd need to think abotu what that means | 00:03 |
inc0 | only reason we do this is to prevent people from really hurting themselves | 00:03 |
sdake | tonyb o i'll bounce something off the governance queue and open it to discussion, sound good? | 00:03 |
inc0 | and again, we're deployment tool so 1. hurting themselves means losing OpenStack... | 00:03 |
sdake | I've really gota jet - lawyers yelling at me :) | 00:03 |
tonyb | sdake: Sure but expect it to get but in the "talk about in Austin" pile :) | 00:04 |
inc0 | and 2. we don't have running service, but we want to be coupled with a release for a reason | 00:04 |
sdake | tonyb ack | 00:04 |
sdake | tonyb that is fully what i expect | 00:04 |
sdake | but it will be written down | 00:04 |
tonyb | sdake: cool | 00:04 |
tonyb | sdake: go to the meeting ;p | 00:04 |
sdake | ive already got about 6 gov repo changes that are in that pie :) | 00:04 |
inc0 | tonyb, as for technical side to it | 00:04 |
inc0 | is git co, git reset, git push possible with out infra? | 00:05 |
tonyb | inc0: as to the technical side "go nuts" | 00:05 |
tonyb | inc0: you're the project team and experts and if this is the *only* way you can keep liberty going *and* | 00:05 |
inc0 | yes, that wasn't decision we've taken lightly | 00:06 |
tonyb | ther si some kind of going from 1.0.0 -> 1.1.0 will have these extra steps documented etc then it's your call | 00:06 |
inc0 | tonyb, nobody runs 1.0.0 from what we know | 00:06 |
inc0 | right now | 00:06 |
inc0 | ops deploys 1 release back | 00:07 |
inc0 | that's why we want to make sure that if they deploy, they won't be hurt | 00:07 |
*** absubram has quit IRC | 00:07 | |
inc0 | but yes, we have few ideas how to provide migration | 00:07 |
inc0 | tonyb, how about we discuss it more tomorrow or next week? | 00:09 |
inc0 | so you'll have some time to digest it | 00:09 |
tonyb | inc0: cool. | 00:09 |
inc0 | let's do that then | 00:09 |
*** vhosakot has quit IRC | 00:10 | |
inc0 | I'll prep stable/mitaka with patch | 00:10 |
inc0 | and reach out to you and then we'll decide how to backport it in a way that nobody gets hurt | 00:10 |
inc0 | would be too much to ask for you to stick around #openstack-kolla? | 00:11 |
inc0 | or which channel you normally are on? | 00:11 |
*** achanda has joined #openstack-kolla | 00:12 | |
inc0 | well, g2g as well | 00:16 |
inc0 | cya fols | 00:16 |
*** inc0 has quit IRC | 00:16 | |
*** achanda has quit IRC | 00:17 | |
*** vhosakot has joined #openstack-kolla | 00:22 | |
sbezverk | vhosakot ping | 00:26 |
vhosakot | pong | 00:27 |
sbezverk | vhosakot have you played with multinode config? | 00:27 |
vhosakot | nope.. I use AIO.. ccesario is your man! | 00:27 |
vhosakot | mlima__ too | 00:27 |
vhosakot | sdake too | 00:27 |
vhosakot | dont know who else | 00:27 |
sbezverk | thanks I was always using AIO, so need to check with folks who knows multinode.. | 00:28 |
vhosakot | cool | 00:29 |
*** Jeffrey4l__ has quit IRC | 00:30 | |
ccesario_ | hello | 00:38 |
*** rhallisey has quit IRC | 00:41 | |
*** tonyb has quit IRC | 00:44 | |
*** salv-orlando has joined #openstack-kolla | 00:44 | |
*** tonyb has joined #openstack-kolla | 00:44 | |
*** banix has joined #openstack-kolla | 00:48 | |
*** salv-orlando has quit IRC | 00:49 | |
openstackgerrit | Hui Kang proposed openstack/kolla: Remove node_templates_directory in ansbile variable file https://review.openstack.org/305543 | 00:56 |
*** brad[]` is now known as brad[] | 00:57 | |
*** vhosakot has quit IRC | 01:07 | |
*** vhosakot has joined #openstack-kolla | 01:10 | |
*** achanda has joined #openstack-kolla | 01:13 | |
*** achanda has quit IRC | 01:19 | |
*** Marga__ has quit IRC | 01:22 | |
*** ccesario_ has quit IRC | 01:34 | |
*** jasonsb has joined #openstack-kolla | 01:45 | |
*** sdake has quit IRC | 01:55 | |
*** sdake has joined #openstack-kolla | 01:56 | |
openstackgerrit | Kuo-tung Kao proposed openstack/kolla: add unit tests for build script https://review.openstack.org/290423 | 02:03 |
*** ravig has joined #openstack-kolla | 02:06 | |
*** sdake has quit IRC | 02:12 | |
*** ravig has quit IRC | 02:12 | |
*** weiyu_ has joined #openstack-kolla | 02:12 | |
*** weiyu_ has quit IRC | 02:15 | |
*** achanda has joined #openstack-kolla | 02:15 | |
*** weiyu_ has joined #openstack-kolla | 02:16 | |
*** weiyu_ has quit IRC | 02:18 | |
*** unicell1 has quit IRC | 02:20 | |
*** weiyu has joined #openstack-kolla | 02:20 | |
*** achanda has quit IRC | 02:21 | |
*** weiyu has quit IRC | 02:23 | |
*** weiyu_ has joined #openstack-kolla | 02:27 | |
*** weiyu_ has quit IRC | 02:30 | |
*** coolsvap has joined #openstack-kolla | 02:33 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/304895 | 02:33 |
sbezverk | coolsvap ping | 02:37 |
coolsvap | sbezverk: pong | 02:37 |
sbezverk | coolsvap qq do you know by any chance why keystone only listens for authentication on api interface? | 02:37 |
sbezverk | or if separate api is not configured, then on internal interface | 02:38 |
*** salv-orlando has joined #openstack-kolla | 02:40 | |
coolsvap | sbezverk: i think its due to the fact that keystone is the entry point for all openstack services | 02:41 |
sbezverk | coolsvap agree, but if a person needs to authenticate to opensatck from outside, then this internal network is not visible. | 02:42 |
coolsvap | yes its always the case, the api interface (which is public) needs to be mapped to the internal interface | 02:43 |
coolsvap | the internal interfaces should not be visible to outside world | 02:44 |
*** salv-orlando has quit IRC | 02:44 | |
*** weiyu_ has joined #openstack-kolla | 02:46 | |
sbezverk | coolsvap but then it is problem, nobody from outside where only public VIP is visible will be able to authenticate against keystone.. | 02:47 |
sbezverk | anyway I will check with fellow operators before filing a new bug | 02:48 |
sbezverk | thank you for your help.. | 02:48 |
coolsvap | sbezverk: its always the case, the internal network, IPs are never visible/reachable to outside world | 02:49 |
coolsvap | what is the issue you are facing which got you to this question? | 02:49 |
*** ravig has joined #openstack-kolla | 02:49 | |
*** ravig has quit IRC | 02:50 | |
*** ravig has joined #openstack-kolla | 02:51 | |
*** weiyu_ has quit IRC | 02:52 | |
coolsvap | vhosakot: around | 02:54 |
vhosakot | coolsvap: ping | 02:54 |
*** weiyu_ has joined #openstack-kolla | 02:55 | |
*** ravig has quit IRC | 02:58 | |
*** ravig has joined #openstack-kolla | 02:58 | |
*** coolsvap has quit IRC | 03:00 | |
*** coolsvap has joined #openstack-kolla | 03:04 | |
coolsvap | vhosakot: did you get my messages? | 03:12 |
openstackgerrit | Kuo-tung Kao proposed openstack/kolla: add unit tests for build script https://review.openstack.org/290423 | 03:13 |
vhosakot | coolsvap: no.. I did not get your message.. | 03:13 |
vhosakot | coolsvap: can you send again | 03:13 |
coolsvap | vhosakot: some reviews https://review.openstack.org/#/c/304518/ https://review.openstack.org/304763 https://review.openstack.org/#/c/304783/ https://review.openstack.org/#/c/305073/ https://review.openstack.org/#/c/305588/ | 03:13 |
patchbot | coolsvap: patch 304518 - kolla - cleanup-host should not remove kolla-build.conf | 03:13 |
patchbot | coolsvap: patch 304783 - kolla - Update base dockerfile for formatting | 03:13 |
patchbot | coolsvap: patch 305073 - kolla - Update haproxy dockerfile for formatting | 03:13 |
patchbot | coolsvap: patch 305588 - kolla (stable/mitaka) - Added help documentation for kolla-ansible upgrade | 03:13 |
coolsvap | and i wanted to know your views on https://review.openstack.org/#/c/305251/ | 03:13 |
patchbot | coolsvap: patch 305251 - kolla (stable/mitaka) - check /e/s/s/docker.service in ubuntu | 03:13 |
vhosakot | cool, will review them.. | 03:14 |
coolsvap | vhosakot: thanks! | 03:15 |
*** achanda has joined #openstack-kolla | 03:17 | |
vhosakot | coolsvap: cool, np | 03:17 |
*** Marga_ has joined #openstack-kolla | 03:20 | |
*** achanda has quit IRC | 03:22 | |
*** Marga_ has quit IRC | 03:23 | |
*** Marga_ has joined #openstack-kolla | 03:24 | |
*** gbraad has joined #openstack-kolla | 03:25 | |
*** weiyu_ has quit IRC | 03:31 | |
*** coolsvap has quit IRC | 03:45 | |
*** jasonsb has quit IRC | 03:47 | |
*** coolsvap has joined #openstack-kolla | 03:48 | |
*** weiyu_ has joined #openstack-kolla | 03:55 | |
*** weiyu_ has quit IRC | 03:57 | |
*** weiyu_ has joined #openstack-kolla | 04:01 | |
*** achanda has joined #openstack-kolla | 04:02 | |
*** weiyu_ has quit IRC | 04:05 | |
*** achanda_ has joined #openstack-kolla | 04:05 | |
*** achanda has quit IRC | 04:06 | |
*** ph03n1x has quit IRC | 04:15 | |
*** achanda_ has quit IRC | 04:22 | |
*** alyson_ has quit IRC | 04:22 | |
*** alyson_ has joined #openstack-kolla | 04:23 | |
*** ph03n1x has joined #openstack-kolla | 04:27 | |
*** ruhe2 has joined #openstack-kolla | 04:30 | |
*** ByPasS has quit IRC | 04:30 | |
*** ruhe2 is now known as ruhe | 04:30 | |
*** ByPasS has joined #openstack-kolla | 04:31 | |
*** banix has quit IRC | 04:32 | |
*** cfarquhar has quit IRC | 04:32 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Fixes kolla_toolbox failure due to missing pytz module https://review.openstack.org/305606 | 04:32 |
*** cfarquhar has joined #openstack-kolla | 04:34 | |
*** cfarquhar has joined #openstack-kolla | 04:34 | |
coolsvap | asalkeld: are there any steps listed out to get started with kolla-mesos? | 04:35 |
*** ravig has quit IRC | 04:36 | |
*** banix has joined #openstack-kolla | 04:36 | |
*** ph03n1x has quit IRC | 04:37 | |
*** banix has quit IRC | 04:39 | |
*** weiyu has joined #openstack-kolla | 04:40 | |
*** ph03n1x has joined #openstack-kolla | 04:43 | |
*** weiyu has quit IRC | 04:44 | |
*** ph03n1x has quit IRC | 04:44 | |
*** ph03n1x has joined #openstack-kolla | 04:47 | |
*** achanda has joined #openstack-kolla | 04:53 | |
*** sbezverk has quit IRC | 04:54 | |
*** sbezverk has joined #openstack-kolla | 04:54 | |
*** salv-orlando has joined #openstack-kolla | 05:11 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: check /e/s/s/docker.service.d/kolla.conf in redhat family https://review.openstack.org/305611 | 05:17 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: check /e/s/s/docker.service in ubuntu https://review.openstack.org/305251 | 05:20 |
*** jidar has quit IRC | 05:24 | |
*** jidar has joined #openstack-kolla | 05:28 | |
*** Marga_ has quit IRC | 05:30 | |
*** achanda has quit IRC | 05:31 | |
*** coolsvap has quit IRC | 05:35 | |
*** Marga_ has joined #openstack-kolla | 05:36 | |
*** tfukushima has joined #openstack-kolla | 05:56 | |
*** dims_ has joined #openstack-kolla | 06:00 | |
*** v1k0d3n_ has joined #openstack-kolla | 06:00 | |
*** mdnadeem has joined #openstack-kolla | 06:01 | |
*** mikelk has joined #openstack-kolla | 06:02 | |
*** gbraad_ has joined #openstack-kolla | 06:03 | |
*** harmw_ has joined #openstack-kolla | 06:04 | |
*** achanda has joined #openstack-kolla | 06:04 | |
*** mlima_ has joined #openstack-kolla | 06:04 | |
*** elmiko_ has joined #openstack-kolla | 06:05 | |
*** Marga__ has joined #openstack-kolla | 06:06 | |
*** zeus- has joined #openstack-kolla | 06:07 | |
*** Marga_ has quit IRC | 06:08 | |
*** sbezverk has quit IRC | 06:08 | |
*** gbraad has quit IRC | 06:08 | |
*** vhosakot has quit IRC | 06:08 | |
*** tonyb has quit IRC | 06:08 | |
*** mlima__ has quit IRC | 06:08 | |
*** nihilifer has quit IRC | 06:08 | |
*** d_code has quit IRC | 06:08 | |
*** v1k0d3n has quit IRC | 06:08 | |
*** dims has quit IRC | 06:08 | |
*** daneyon has quit IRC | 06:08 | |
*** zeus has quit IRC | 06:08 | |
*** mdnadeem_ has quit IRC | 06:08 | |
*** smekel_ has quit IRC | 06:08 | |
*** cloudnull has quit IRC | 06:08 | |
*** elmiko has quit IRC | 06:08 | |
*** nihilifer has joined #openstack-kolla | 06:09 | |
*** zeus- is now known as zeus | 06:09 | |
*** zeus is now known as Guest36812 | 06:09 | |
*** intr1nsic has joined #openstack-kolla | 06:10 | |
*** sbezverk has joined #openstack-kolla | 06:11 | |
*** tonyb has joined #openstack-kolla | 06:14 | |
mimizone | kolla-build --base ubuntu --type binary --registry 127.0.0.1:4000 --push | 06:16 |
mimizone | throws me the following error "ImportError: No module named debtcollector" | 06:16 |
mimizone | am I missing a library? | 06:17 |
mimizone | actually just calling kolla-build throws the same error | 06:19 |
*** weiyu has joined #openstack-kolla | 06:19 | |
*** coolsvap has joined #openstack-kolla | 06:19 | |
kjelly | mimizone: post traceback in http://paste.openstack.org/ | 06:21 |
mimizone | I installed debtcollector with pip, now it's complaining about netaddr | 06:21 |
mimizone | http://paste.openstack.org/show/494018/ | 06:22 |
mimizone | I am following this documentation https://github.com/openstack/kolla/blob/master/doc/quickstart.rst | 06:22 |
*** cloudnull has joined #openstack-kolla | 06:23 | |
mimizone | adding netaddr with pip seems to be enough | 06:23 |
kjelly | mimizone: I install kolla in new virtualenv. All thing works fine. Maybe you have old oslo_config in your env. | 06:25 |
mimizone | I think I have done all the other steps with sudo, so most likely that comes from that | 06:26 |
mimizone | sudo kolla-build is not complaining of anything | 06:26 |
kjelly | mimizone: Ok. Have fun with kolla. | 06:27 |
*** jmccarthy has left #openstack-kolla | 06:31 | |
*** weiyu has quit IRC | 06:31 | |
*** coolsvap is now known as coolsvap_zzz | 06:32 | |
*** d_code has joined #openstack-kolla | 06:32 | |
openstackgerrit | MD NADEEM proposed openstack/kolla: Remove redundant value https://review.openstack.org/305625 | 06:34 |
*** achanda has quit IRC | 06:36 | |
*** weiyu_ has joined #openstack-kolla | 06:36 | |
*** jmccarthy has joined #openstack-kolla | 06:38 | |
*** weiyu_ has quit IRC | 06:38 | |
*** jmccarthy has left #openstack-kolla | 06:39 | |
*** vivek__ has quit IRC | 06:39 | |
mimizone | normal that images for magmum, designate, gnocchi, mistral, zaqar failed be build? | 06:41 |
openstackgerrit | MD NADEEM proposed openstack/kolla: WIP: Added Dockerfile for congress https://review.openstack.org/305626 | 06:43 |
openstackgerrit | MD NADEEM proposed openstack/kolla: WIP: Added Dockerfile for congress https://review.openstack.org/305626 | 06:48 |
*** vhosakot has joined #openstack-kolla | 06:49 | |
*** daneyon has joined #openstack-kolla | 06:50 | |
*** Serlex has joined #openstack-kolla | 06:56 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Enable unbuffered output for ansible https://review.openstack.org/303389 | 07:00 |
*** vhosakot has quit IRC | 07:03 | |
*** unicell has joined #openstack-kolla | 07:04 | |
openstackgerrit | Merged openstack/kolla: Identify container through PS1 shell variable https://review.openstack.org/305393 | 07:06 |
*** smekel_ has joined #openstack-kolla | 07:06 | |
*** weiyu_ has joined #openstack-kolla | 07:09 | |
*** mlima_ has quit IRC | 07:11 | |
*** sdake has joined #openstack-kolla | 07:12 | |
*** alyson_ has quit IRC | 07:12 | |
*** achanda has joined #openstack-kolla | 07:19 | |
*** unicell has quit IRC | 07:19 | |
*** unicell has joined #openstack-kolla | 07:20 | |
*** weiyu_ has quit IRC | 07:21 | |
*** weiyu has joined #openstack-kolla | 07:21 | |
*** weiyu has quit IRC | 07:24 | |
coolsvap_zzz | sdake how was the webinar? | 07:24 |
*** mbound has joined #openstack-kolla | 07:24 | |
*** mbound has quit IRC | 07:25 | |
*** athomas has joined #openstack-kolla | 07:26 | |
*** chopmann has joined #openstack-kolla | 07:27 | |
*** weiyu_ has joined #openstack-kolla | 07:28 | |
*** weiyu_ has quit IRC | 07:31 | |
*** sdake has quit IRC | 07:31 | |
*** sbezverk has quit IRC | 07:38 | |
*** sbezverk has joined #openstack-kolla | 07:40 | |
*** weiyu_ has joined #openstack-kolla | 07:46 | |
*** weiyu_ has quit IRC | 08:00 | |
*** achanda has quit IRC | 08:01 | |
*** achanda has joined #openstack-kolla | 08:10 | |
*** achanda has quit IRC | 08:11 | |
*** mbound has joined #openstack-kolla | 08:12 | |
*** jmccarthy has joined #openstack-kolla | 08:16 | |
*** jmccarthy has left #openstack-kolla | 08:16 | |
*** chopmann has quit IRC | 08:20 | |
*** daneyon has quit IRC | 08:20 | |
*** daneyon has joined #openstack-kolla | 08:21 | |
*** chopmann has joined #openstack-kolla | 08:23 | |
*** ajafo has left #openstack-kolla | 08:24 | |
*** openstackstatus has joined #openstack-kolla | 08:25 | |
*** ChanServ sets mode: +v openstackstatus | 08:25 | |
*** rmart04 has joined #openstack-kolla | 08:26 | |
*** coolsvap_zzz has quit IRC | 08:30 | |
*** weiyu has joined #openstack-kolla | 08:31 | |
*** gfidente has joined #openstack-kolla | 08:52 | |
*** v1k0d3n_ has quit IRC | 08:52 | |
*** v1k0d3n has joined #openstack-kolla | 08:52 | |
*** chopmann has quit IRC | 09:03 | |
*** ydeshmukh has joined #openstack-kolla | 09:10 | |
*** akscram has quit IRC | 09:30 | |
*** smekel_ has quit IRC | 09:30 | |
*** chopmann has joined #openstack-kolla | 09:33 | |
*** bradjones has quit IRC | 09:33 | |
*** coolsvap has joined #openstack-kolla | 09:33 | |
*** stvnoyes1 has quit IRC | 09:34 | |
*** akscram has joined #openstack-kolla | 09:35 | |
openstackgerrit | jackning proposed openstack/kolla: Added webroot config to dashboard" https://review.openstack.org/305704 | 09:36 |
*** stvnoyes has joined #openstack-kolla | 09:36 | |
*** chopmann has quit IRC | 09:38 | |
*** pbourke_ has quit IRC | 09:38 | |
*** pbourke has joined #openstack-kolla | 09:39 | |
*** bradjones has joined #openstack-kolla | 09:43 | |
*** bradjones has joined #openstack-kolla | 09:43 | |
*** smekel_ has joined #openstack-kolla | 09:45 | |
*** chopmann has joined #openstack-kolla | 09:45 | |
*** salv-orl_ has joined #openstack-kolla | 09:47 | |
*** salv-orlando has quit IRC | 09:50 | |
openstackgerrit | Paul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 09:52 |
openstackgerrit | Merged openstack/kolla: Remove /etc/kolla configuration folders https://review.openstack.org/305428 | 10:00 |
openstackgerrit | Merged openstack/kolla: Fix the upgrade issue when image id is the same https://review.openstack.org/305133 | 10:04 |
openstackgerrit | Merged openstack/kolla: Updated from global requirements https://review.openstack.org/304895 | 10:04 |
*** tfukushima has quit IRC | 10:06 | |
*** tfukushima has joined #openstack-kolla | 10:09 | |
*** andrewvwebber has joined #openstack-kolla | 10:15 | |
*** mbound has quit IRC | 10:31 | |
sbezverk | pbourke ping | 10:32 |
pbourke | sbezverk: hi | 10:32 |
sbezverk | pbourke good morning, I have a quick question. I came accross a couple of wierdnesses in multinode neutron, wanted to know who was working on neutron in kolla.. | 10:34 |
pbourke | sbezverk: everyone and noone | 10:34 |
pbourke | sbezverk: there's no one "neutron guy" :) | 10:35 |
pbourke | sbezverk: throw out your problem maybe someone will no | 10:35 |
sbezverk | pbourke I see, maybe you can help then | 10:35 |
pbourke | *knpow | 10:35 |
pbourke | *know | 10:35 |
sbezverk | pbourke on cimpute node, by default there should not be a connection to external network | 10:35 |
sbezverk | it is used only if distributed router is deployed | 10:36 |
*** mbound has joined #openstack-kolla | 10:36 | |
sbezverk | but I see the installer tries to create br-ex on all compute nodes | 10:36 |
sbezverk | I wanted to know if there was a specific reason for that | 10:36 |
pbourke | im not sure - is it not needed? | 10:38 |
sbezverk | pbourke in classical openstack topology, compute node has only vm traffic interface and internal network | 10:41 |
*** achanda has joined #openstack-kolla | 10:44 | |
pbourke | sbezverk: sorry im not sure why its the case then :/ | 10:44 |
sbezverk | pbourke not a problem I will check with the rest of team when they join :-) | 10:45 |
pbourke | sure - it would be good to know | 10:45 |
sbezverk | pbourke another a bit strange thing I noticed. when you have internal interface and external interface on a controller node, keystone listens only on internel/api interface, basically this prevents anybody from outside of cloud to get authenticated.. Do you know if it was done intentionally? | 10:50 |
sbezverk | cuase I am ready to file a bug for this | 10:51 |
pbourke | sbezverk: this is a problem as far as I know | 10:52 |
pbourke | sbezverk: I submitted a patch for it before but never completed the haproxy config | 10:52 |
sbezverk | pbourke I see, so no need to file a bug then.. it looks fairly critical issue as it is usability issue, for people who uses this functionality it is a show stopper | 10:53 |
pbourke | sbezverk: I'll find the original bug and we can update it | 10:54 |
pbourke | sbezverk: it may be fixed though | 10:55 |
sbezverk | pbourke if you need any help (I mean if you are busy) working on it, let me know I would be happy to participate | 10:55 |
sbezverk | pbourke nope I hit it yesterday | 10:55 |
pbourke | https://github.com/openstack/kolla/blob/master/ansible/roles/haproxy/templates/haproxy.cfg.j2#L74-L81 | 10:55 |
sbezverk | I clearly saw keystone was not listening on external interface | 10:56 |
pbourke | have you configured kolla_external_vip_address? | 10:57 |
sbezverk | pbourke you see in the code you pasted it only lsitens on api interface | 10:57 |
sbezverk | pbourke yes | 10:57 |
sbezverk | pbourke here is my globals.yml http://paste.openstack.org/show/494038/ | 10:58 |
pbourke | so its listening on eth1 rather than eth0 | 10:58 |
sbezverk | pbourke haproxy works on external perfectly, it is keystone does not listen on this external VIP | 10:59 |
pbourke | ok seems like a bug | 10:59 |
sbezverk | pbourke I repro it again and file a bug then | 10:59 |
*** sajauddin has joined #openstack-kolla | 11:03 | |
*** weiyu has quit IRC | 11:05 | |
*** bmotz_ is now known as bmotz | 11:07 | |
*** mikelk has quit IRC | 11:07 | |
*** rhallisey has joined #openstack-kolla | 11:07 | |
*** mikelk has joined #openstack-kolla | 11:10 | |
openstackgerrit | Merged openstack/kolla-mesos: Updated from global requirements https://review.openstack.org/302970 | 11:30 |
openstackgerrit | Merged openstack/kolla: Enable unbuffered output for ansible https://review.openstack.org/303389 | 11:32 |
openstackgerrit | Merged openstack/kolla: Fix missing libvirt python module in ceilometer-compute https://review.openstack.org/300650 | 11:33 |
*** mbound has quit IRC | 11:34 | |
*** salv-orl_ has quit IRC | 11:35 | |
*** mlima has joined #openstack-kolla | 11:36 | |
*** mbound has joined #openstack-kolla | 11:36 | |
mlima | hello guys, we need merge it https://review.openstack.org/#/c/304518/ | 11:36 |
patchbot | mlima: patch 304518 - kolla - cleanup-host should not remove kolla-build.conf | 11:36 |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Enable unbuffered output for ansible https://review.openstack.org/305770 | 11:40 |
openstackgerrit | Merged openstack/kolla: cleanup-host should not remove kolla-build.conf https://review.openstack.org/304518 | 11:44 |
openstackgerrit | Mauricio Lima proposed openstack/kolla: cleanup-host should not remove kolla-build.conf https://review.openstack.org/305775 | 11:45 |
*** mikelk has quit IRC | 11:45 | |
*** achanda has quit IRC | 11:46 | |
*** vhosakot has joined #openstack-kolla | 11:49 | |
*** alyson_ has joined #openstack-kolla | 11:57 | |
*** dwalsh has joined #openstack-kolla | 12:00 | |
*** vhosakot_ has joined #openstack-kolla | 12:02 | |
*** bradjones has quit IRC | 12:02 | |
*** alyson__ has joined #openstack-kolla | 12:05 | |
*** achanda has joined #openstack-kolla | 12:05 | |
*** larsks has quit IRC | 12:06 | |
*** pbourke has quit IRC | 12:06 | |
*** pbourke has joined #openstack-kolla | 12:06 | |
*** intr1nsic has quit IRC | 12:06 | |
*** andrewvwebber has quit IRC | 12:08 | |
*** andrewvwebber has joined #openstack-kolla | 12:08 | |
*** bradjones has joined #openstack-kolla | 12:09 | |
*** bradjones has joined #openstack-kolla | 12:09 | |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a CONTRIBUTING.rst for Kolla https://review.openstack.org/305402 | 12:09 |
*** intr1nsic has joined #openstack-kolla | 12:09 | |
*** alyson_ has quit IRC | 12:10 | |
*** vhosakot has quit IRC | 12:10 | |
*** rhallisey has quit IRC | 12:10 | |
*** alyson__ is now known as alyson_ | 12:10 | |
*** jmccarthy has joined #openstack-kolla | 12:11 | |
*** larsks has joined #openstack-kolla | 12:15 | |
*** salv-orlando has joined #openstack-kolla | 12:19 | |
openstackgerrit | Merged openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/302036 | 12:21 |
*** rhallisey has joined #openstack-kolla | 12:22 | |
coolsvap | pbourke, rhallisey we need some reviews on mitaka https://review.openstack.org/305775 https://review.openstack.org/305770 https://review.openstack.org/305061 https://review.openstack.org/305606 https://review.openstack.org/305588 | 12:24 |
*** coolsvap has quit IRC | 12:25 | |
*** vhosakot_ has quit IRC | 12:25 | |
*** coolsvap has joined #openstack-kolla | 12:27 | |
*** mwheckmann has joined #openstack-kolla | 12:27 | |
*** reddy has joined #openstack-kolla | 12:32 | |
*** sajauddin has quit IRC | 12:34 | |
*** Guest36812 is now known as zeus | 12:45 | |
*** zeus has quit IRC | 12:46 | |
*** zeus has joined #openstack-kolla | 12:46 | |
*** achanda has quit IRC | 12:49 | |
*** stvnoyes has left #openstack-kolla | 12:51 | |
openstackgerrit | Merged openstack/kolla: Make Heat bootstrapping idempotent https://review.openstack.org/305061 | 12:54 |
*** achanda has joined #openstack-kolla | 12:56 | |
openstackgerrit | Paul Bourke proposed openstack/kolla: Fix the gate lying https://review.openstack.org/304205 | 12:59 |
*** kbyrne has quit IRC | 13:01 | |
*** chopmann has quit IRC | 13:06 | |
*** chopmann_ has joined #openstack-kolla | 13:06 | |
*** chopmann_ is now known as chopmann | 13:06 | |
openstackgerrit | Paul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/305840 | 13:06 |
*** jmccarthy has left #openstack-kolla | 13:08 | |
openstackgerrit | Paul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/305840 | 13:10 |
*** ravig has joined #openstack-kolla | 13:10 | |
*** kbyrne has joined #openstack-kolla | 13:13 | |
*** tuvenen_ has joined #openstack-kolla | 13:16 | |
*** ph03n1x_ has joined #openstack-kolla | 13:16 | |
*** tuvenen has quit IRC | 13:17 | |
*** Serlex has quit IRC | 13:17 | |
*** lmiccini has quit IRC | 13:17 | |
*** rhallisey has quit IRC | 13:17 | |
*** smekel_ has quit IRC | 13:17 | |
*** ph03n1x has quit IRC | 13:17 | |
*** jpeeler has quit IRC | 13:17 | |
*** rbergeron has quit IRC | 13:17 | |
*** ph03n1x_ is now known as ph03n1x | 13:18 | |
*** rhallisey has joined #openstack-kolla | 13:18 | |
*** rbergeron has joined #openstack-kolla | 13:18 | |
*** Serlex has joined #openstack-kolla | 13:19 | |
*** jpeeler has joined #openstack-kolla | 13:19 | |
*** banix has joined #openstack-kolla | 13:19 | |
*** smekel_ has joined #openstack-kolla | 13:19 | |
*** lmiccini has joined #openstack-kolla | 13:20 | |
*** jmccarthy has joined #openstack-kolla | 13:22 | |
ccesario | morning :) | 13:23 |
sbezverk | ccesario hey, you have multinode setup right? | 13:24 |
ccesario | sbezverk, hey... | 13:27 |
ccesario | yes, with my tests I run multinode | 13:28 |
sbezverk | ccesario what do you use for compute to controller communication for vm traffic? | 13:30 |
*** achanda has quit IRC | 13:31 | |
ccesario | sbezverk, could please explain better you question!? eheheh | 13:33 |
sbezverk | ccesario for tenants VMs from compute node, usually they need to reach network or controller node where l3 agents run, right? some tunneling is used either vxlan or gre or vlans | 13:38 |
*** achanda has joined #openstack-kolla | 13:38 | |
sbezverk | I wanted to know wha you use. | 13:39 |
*** salv-orlando has quit IRC | 13:40 | |
ccesario | sbezverk, let me check the current configs in my last deploy | 13:41 |
ccesario | sbezverk, default config... currently ... tenant_network_types = vxlan | 13:44 |
ccesario | mechanism_drivers = linuxbridge,l2population | 13:44 |
*** achanda has quit IRC | 13:44 | |
*** mlima_ has joined #openstack-kolla | 13:45 | |
*** mwheckma1n has joined #openstack-kolla | 13:45 | |
*** v1k0d3n_ has joined #openstack-kolla | 13:46 | |
sbezverk | ccesario I see thank you | 13:47 |
*** daneyon_ has joined #openstack-kolla | 13:47 | |
*** Serlex has quit IRC | 13:47 | |
*** rmart04_ has joined #openstack-kolla | 13:47 | |
ccesario | sbezverk, what is your doubt ? | 13:47 |
*** rmart04 has quit IRC | 13:47 | |
*** rmart04_ is now known as rmart04 | 13:47 | |
openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a role to cleanup containers after deployment https://review.openstack.org/303575 | 13:48 |
*** tuvenen_ has quit IRC | 13:48 | |
*** mwheckmann has quit IRC | 13:48 | |
*** v1k0d3n has quit IRC | 13:48 | |
*** tonyb has quit IRC | 13:48 | |
*** banix has quit IRC | 13:49 | |
*** alyson_ has quit IRC | 13:49 | |
*** mlima has quit IRC | 13:49 | |
*** daneyon has quit IRC | 13:49 | |
*** banix has joined #openstack-kolla | 13:49 | |
*** tonyb has joined #openstack-kolla | 13:49 | |
*** alyson_ has joined #openstack-kolla | 13:50 | |
*** tuvenen has joined #openstack-kolla | 13:55 | |
*** coolsvap has quit IRC | 13:56 | |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: tcpdump is missing in the base https://review.openstack.org/305869 | 13:56 |
openstackgerrit | Paul Bourke proposed openstack/kolla: Use proper CentOS repositories for ceph and qemu https://review.openstack.org/305840 | 13:56 |
*** ayoung has quit IRC | 13:57 | |
*** achanda has joined #openstack-kolla | 13:57 | |
sbezverk | ccesario I see some strange behavior and trying to undersatnd its root cause.. | 13:59 |
*** mgoddard_ has joined #openstack-kolla | 13:59 | |
*** Serlex has joined #openstack-kolla | 14:03 | |
*** wmiller has quit IRC | 14:03 | |
*** mgoddard has quit IRC | 14:03 | |
*** achanda has quit IRC | 14:04 | |
*** wmiller has joined #openstack-kolla | 14:05 | |
*** dougs1 has quit IRC | 14:06 | |
*** alyson_ has quit IRC | 14:07 | |
*** banix_ has joined #openstack-kolla | 14:08 | |
*** banix has quit IRC | 14:10 | |
*** banix_ is now known as banix | 14:10 | |
*** dougs has joined #openstack-kolla | 14:10 | |
*** ravig has quit IRC | 14:13 | |
*** reddy has quit IRC | 14:24 | |
*** inc0 has joined #openstack-kolla | 14:32 | |
inc0 | good morning | 14:33 |
inc0 | any bug that needs immediate attention? | 14:33 |
pbourke | inc0: nothing major has come up today that im aware of | 14:34 |
*** tfukushima has quit IRC | 14:36 | |
ydeshmukh | Hi All, how to check keystone is running and port 35357 is open? | 14:40 |
*** salv-orlando has joined #openstack-kolla | 14:41 | |
ydeshmukh | I'm using centos-source redistro | 14:41 |
*** achanda has joined #openstack-kolla | 14:42 | |
inc0 | ydeshmukh, it should be visible on host | 14:42 |
inc0 | so netstat -nalp | grep 35357 should do the job | 14:43 |
sbezverk | inc0 is it normal that keystone listens only on port 5000 on external vip and it does not listen on 35357? | 14:48 |
*** salv-orlando has quit IRC | 14:50 | |
*** shardy has quit IRC | 14:50 | |
ydeshmukh | inc0 actually, I'm facing issue at 'Creating the Glance service and endpoint' in kolla-ansible deploy. | 14:51 |
ydeshmukh | when I try to run same command on kolla_toolbox container, | 14:51 |
ydeshmukh | ansible localhost -m kolla_keystone_service -a "service_name=glance service_type=image description='Openstack Image' endpoint_region=RegionOne url='http://192.168.20.206:9292' interface='admin' region_name=RegionOne auth={{ openstack_glance_auth }}" -e "{'openstack_glance_auth':{'username': 'admin', 'project_name': 'admin', 'password': 'password', 'auth_url': 'http://192.168.20.206:35357', 'domain_name': 'default'}}" | 14:51 |
ydeshmukh | it's giving error, 'Could not determine a suitable URL for the plugin' | 14:51 |
ydeshmukh | any idea? | 14:51 |
inc0 | sbezverk, yeah | 14:53 |
inc0 | this is how it should be | 14:53 |
inc0 | 35357 is not secure, it's meant for admins to fix stuff so access to it should be severely limited | 14:53 |
*** ayoung has joined #openstack-kolla | 14:54 | |
inc0 | ydeshmukh, can you show your globals.conf please? | 14:54 |
sbezverk | inc0 but admin still should be able to get authenticated via 5000 port right? | 14:54 |
inc0 | sbezverk, but you need to create first admin account right?: | 14:55 |
inc0 | ayoung, correct me if I'm wrong please, but keystones 35357 doesn't really have authentication? | 14:55 |
inc0 | it only has admin token right? | 14:55 |
ayoung | inc0, wrong | 14:55 |
ayoung | admin token should be disabled | 14:56 |
inc0 | from Mitaka right? | 14:56 |
ayoung | only use an actual authentication request | 14:56 |
ayoung | inc0, always | 14:56 |
inc0 | but you needed it for bootstraping admin users prior to keystone-manage bootstrap | 14:56 |
ayoung | inc0, keystone-manage bootstrap is supposed to replace ADMIN_TOKEN | 14:57 |
ayoung | it might be imperfect | 14:57 |
ayoung | but I think we did enough in Mitaka that you do not need ADMIN_TOKEN at all | 14:57 |
ayoung | think of ADMIN_TOKEN as a bad idea who's time ahs come...to DIE! | 14:57 |
*** dougs1 has joined #openstack-kolla | 14:57 | |
ydeshmukh | inc0 I've pasted globals.yml at http://paste.openstack.org/show/494086/ | 14:58 |
inc0 | in mitaka we don't have admin token | 14:58 |
*** dougs has quit IRC | 14:59 | |
*** mgoddard_ has quit IRC | 15:01 | |
*** mgoddard has joined #openstack-kolla | 15:01 | |
*** dougs1 has quit IRC | 15:05 | |
inc0 | ydeshmukh, hmm...looks good | 15:05 |
inc0 | have you tried clean and redeploy? | 15:05 |
ydeshmukh | yeah I tried 3-4 times, still same issue. | 15:06 |
*** Serlex has quit IRC | 15:06 | |
inc0 | docker ps shows all containers running? | 15:06 |
ydeshmukh | yeah, all are running | 15:08 |
*** dougs has joined #openstack-kolla | 15:08 | |
inc0 | hmm | 15:08 |
inc0 | sounds like a bug | 15:08 |
ydeshmukh | at least till glance service creation, http://paste.openstack.org/show/494090/ | 15:09 |
inc0 | 192.168.20.206 is an unused ip in net right | 15:09 |
inc0 | ? | 15:09 |
ydeshmukh | yeah, unused IP from management network | 15:09 |
inc0 | can you manually ask keystone stuff? | 15:10 |
inc0 | so kolla-ansible post-deploy | 15:10 |
inc0 | will create admin.sh | 15:10 |
inc0 | and then keystone user-list etc | 15:10 |
sbezverk | ayoung which would be then the best approach to authenticate admin trying to access cloud from outside? | 15:11 |
ydeshmukh | inc0, ok will ask that. | 15:11 |
inc0 | sbezverk, 5000;) | 15:12 |
sbezverk | inc0 I do have it, it is just after I rebooted my multinode setup, all of a sudden admin could not authenticate nya more | 15:13 |
sbezverk | please check this paste: http://paste.openstack.org/show/494091/ | 15:13 |
*** vhosakot has joined #openstack-kolla | 15:14 | |
*** mdnadeem_ has joined #openstack-kolla | 15:14 | |
sbezverk | maybe some leftovers from deployment were allowing admin to authenticate this way, but after reboot it stopped working.. | 15:14 |
ayoung | sbezverk, tokenless X509 ? | 15:15 |
ayoung | New feature in Mitaka | 15:15 |
ayoung | wouldn';t use it for setup though | 15:15 |
*** ravig has joined #openstack-kolla | 15:15 | |
*** ravig has quit IRC | 15:16 | |
ayoung | https://blueprints.launchpad.net/keystone/+spec/x509-ssl-client-cert-authn | 15:16 |
ayoung | let me see if I can find a more readable version. I think we publish the specs now in a some postformatted way | 15:16 |
*** mdnadeem has quit IRC | 15:17 | |
sbezverk | ayoung thank you.. would be good to have a document | 15:18 |
sbezverk | inc0 ok I was looking at the wrong place, it was not keystone issue, after reboot mariadb container is flapping on all 3 controllers | 15:21 |
inc0 | yeah | 15:21 |
inc0 | that makes things harder | 15:21 |
inc0 | having working db is useful | 15:21 |
sbezverk | inc0 now I know it is not just a decoration ;-) | 15:23 |
inc0 | exactly, wouldn't do a good job as such either, it's not particularly pretty | 15:25 |
*** chopmann has quit IRC | 15:26 | |
*** mark-casey has joined #openstack-kolla | 15:29 | |
sbezverk | inc0 nevertheless it looks serious :-( by some reason even though IP connectivity between controller is there one mariadb fails to establish connectivity see this: http://paste.openstack.org/show/494093/ | 15:31 |
inc0 | it does | 15:32 |
inc0 | maria is acting up sometimes, we need to figure this one out | 15:32 |
inc0 | connection timed out | 15:32 |
inc0 | did you clean up iptables after last deployment? | 15:33 |
inc0 | sometimes that's the issue | 15:33 |
inc0 | see if there are any leftover rules | 15:33 |
ydeshmukh | inc0 that was proxy issue after setting no_proxy=192.168.20.206 it got resolved. | 15:33 |
inc0 | ahh there you go | 15:34 |
inc0 | proxy | 15:34 |
inc0 | I hate it | 15:34 |
ydeshmukh | yeah, I was struggling for 2 days for this small issue. | 15:36 |
mlima_ | inc0, i updated the patchset and now the cleanup files aren't copied to /etc. | 15:37 |
mimizone | hi all. | 15:38 |
mimizone | can the list of interface (ex: api_interface, neutron_interface etc...) be set for every type of node in the ansible inventory or just for the compute node? | 15:39 |
mimizone | the comment in the example multinode file is not clear on that and seems to apply only to compute | 15:39 |
*** salv-orlando has joined #openstack-kolla | 15:39 | |
*** vhosakot has quit IRC | 15:41 | |
inc0 | mlima_, thanks let me check | 15:42 |
*** vhosakot has joined #openstack-kolla | 15:43 | |
inc0 | mimizone, it should be for every node | 15:44 |
inc0 | this is ansible mechanism | 15:44 |
mimizone | inc0: thanks, I didn't look at the ansible playbooks/scripts, wasn't sure it was used everywhere independently of the openstack role | 15:44 |
*** chopmann has joined #openstack-kolla | 15:45 | |
inc0 | well, it follows usual ansible variable hierarchy, we didn't do anything new with it | 15:45 |
inc0 | let me know if you run into any issues | 15:45 |
*** mgoddard_ has joined #openstack-kolla | 15:46 | |
*** rajathagasthya has joined #openstack-kolla | 15:49 | |
*** mgoddard has quit IRC | 15:49 | |
ccesario | sbezverk, I gt some errors like you posted with mariadb | 15:52 |
mlima_ | inc0, ok | 15:52 |
*** ravig has joined #openstack-kolla | 15:55 | |
sbezverk | ccesario glad I am not alone then ;-) | 15:56 |
inc0 | deploying stable, let see | 15:57 |
ccesario | sbezverk, what is the exactly steps to reproduce it ? did you get!? | 15:57 |
*** haplo37 has joined #openstack-kolla | 16:00 | |
*** mbound has quit IRC | 16:01 | |
*** mark-casey1 has joined #openstack-kolla | 16:01 | |
inc0 | sooo...mariadb worked for me | 16:02 |
*** dougs has quit IRC | 16:03 | |
inc0 | guys can you confirm that you don't have running mysql service? | 16:03 |
inc0 | try to run prechecks | 16:03 |
*** mark-casey has quit IRC | 16:04 | |
*** sdake has joined #openstack-kolla | 16:05 | |
sbezverk | inc0 I see mariadb is restarting on all three controllers | 16:07 |
*** slagle_ is now known as slagle | 16:07 | |
sbezverk | ccesario I did the initial deployment, then I rebooted all nodes and since that time maridb has gone south | 16:08 |
sdake | morning | 16:08 |
sbezverk | ccesario after initial deployment it was working finr | 16:08 |
inc0 | sbezverk, try to cleanup, rebuild all | 16:09 |
inc0 | run prechecks | 16:09 |
inc0 | and redeploy | 16:09 |
*** dougs has joined #openstack-kolla | 16:09 | |
inc0 | also rebuild with --no-cache | 16:09 |
sbezverk | inc0 maybe the goal should be not to recover asap, but better understand what is causing it. I will try to dig a bit and try to get some input from galera folks | 16:11 |
pbourke | sdake: morning | 16:12 |
pbourke | sdake: all oraclelinux gates working except binary deploy | 16:12 |
*** chopmann has quit IRC | 16:13 | |
pbourke | sdake: when are you planning to tag? | 16:13 |
ccesario | sbezverk, let me try reproduce it | 16:13 |
ccesario | sdake, hey! | 16:13 |
*** unicell has quit IRC | 16:16 | |
sdake | friday evening | 16:16 |
andrewvwebber | coreos kolla works perfectly | 16:17 |
sdake | andrewvwebber nice which did ou fix | 16:17 |
andrewvwebber | i had a bug in my cloudconfig that did not add correctly a veth interface to a bridge | 16:18 |
andrewvwebber | the udev messages were a red herring | 16:18 |
sdake | thats good - that had me worried :( | 16:18 |
andrewvwebber | i confirmed it with a second network interface as well as a veth added to a bridge | 16:18 |
andrewvwebber | that you so much for this great project! | 16:19 |
andrewvwebber | thank | 16:19 |
mark-casey1 | andrewvwebber: are you just running kolla-ansible against coreos nodes or have you created unit files? or how are you launching the containers? | 16:19 |
*** chopmann has joined #openstack-kolla | 16:19 | |
andrewvwebber | i am currently running kolla-ansible | 16:19 |
mark-casey1 | sweet | 16:19 |
sdake | if we coud get docs on setup of coreos | 16:20 |
andrewvwebber | however i will look into fleet unit soon as coreos frowns on SSHing into cattle boxes | 16:20 |
sdake | tha would be fantastic ;:) | 16:20 |
andrewvwebber | i am happy to write a summary | 16:20 |
inc0 | andrewvwebber, issue you might have is bootstrapping | 16:20 |
inc0 | we do stuff that needs some logic | 16:20 |
inc0 | on deploy | 16:20 |
inc0 | also make sure that things like ceph works, these might be trickier | 16:21 |
inc0 | as it uses udev a bit | 16:21 |
andrewvwebber | i think the goal of spinning up a machine and never being able to SSH into it, is a direct odds of ansible | 16:21 |
andrewvwebber | im doing ceph integration next week | 16:21 |
mark-casey1 | there are other connection plugins in ansible though | 16:21 |
inc0 | let me know if I can help you | 16:21 |
inc0 | yeah I think you can do ansible master | 16:22 |
inc0 | never tried it tho | 16:22 |
mark-casey1 | oh... yea pull mode | 16:22 |
andrewvwebber | i am yet to really train my self on ansible, your guide was excellent enough | 16:22 |
inc0 | ansible is really nice when you get to know it | 16:23 |
inc0 | and kinda bad when you really know it;) | 16:23 |
sdake | why does coreos frown on sshing into cattel box | 16:23 |
inc0 | but it gets better on 2.0 which is our N roadmap | 16:23 |
inc0 | sdake, because you shouldnt ahve to | 16:23 |
andrewvwebber | my only problem is i need to learn about openstack ceph integration as i would prefer to seperate the two. i dont like the idea of compute nodes talking directly to OSDs | 16:24 |
inc0 | you should just hit docker API and that's all | 16:24 |
inc0 | andrewvwebber, that's how ceph does stuff tho | 16:24 |
andrewvwebber | :( | 16:24 |
inc0 | asks mons for crush map | 16:24 |
andrewvwebber | i like the S3 gateway abstraction with CEPH gateway. no need to be on the same network as an OSD | 16:24 |
mark-casey1 | what's the default value of --threads for kolla-build? | 16:24 |
inc0 | not really :( if you consider amount of data that needs to travel over network | 16:24 |
pbourke | mark-casey1: 4 i think | 16:25 |
mark-casey1 | ty | 16:25 |
inc0 | I think it's 8 | 16:25 |
inc0 | but I really suggest to add ansible.cfg | 16:25 |
inc0 | set it up to 100 or so and enable ssh pipelining | 16:25 |
inc0 | makes stuff even quicker | 16:25 |
andrewvwebber | sdake: i think the coreos security/architecture concept is to not have access to individual machines and encourage cloud like thinking (unicorns vs robots) | 16:25 |
pbourke | inc0 is correct | 16:25 |
inc0 | andrewvwebber, but if you consider that proxy will have to transfer terabytes of data potentially | 16:26 |
inc0 | it quickly becomes bottlenech | 16:26 |
sdake | andrewvwebber have they stated this ssh thing anywhere? | 16:26 |
andrewvwebber | inc0: im just thinking from a network security point of view | 16:26 |
inc0 | you really want compute ndoes to talk to osd direcrly, for speed and redudnancy | 16:26 |
inc0 | don't forget that users doesn't have to have access to this network | 16:27 |
inc0 | you can separate tenant networks from management | 16:27 |
inc0 | and you should | 16:27 |
andrewvwebber | sdake: ssh is on by default and ships in coreos, its just common practice to disable it and use the Fleet cluster manager to deploy workloads | 16:27 |
*** achanda has quit IRC | 16:27 | |
andrewvwebber | sdake: if a machine is not working correctly; kill, rebuild, findout why later | 16:28 |
andrewvwebber | inc0: defence in depth, with CEPH S3 gateway i only need access to the gateways from another network, typically the gateway would have more than one interface | 16:29 |
andrewvwebber | inc0: but i totally understand | 16:29 |
andrewvwebber | inc0: CERN performance requirements are not necessarily business security requirements | 16:30 |
inc0 | yeah I understand what you mean and I agree that having this edge ndoes would be more secure | 16:30 |
*** kbyrne has quit IRC | 16:31 | |
inc0 | but you really lose a lot of nice stuff ceph gives you with it's distributed arch | 16:31 |
inc0 | it's really a arch decision and I'd love to include options | 16:31 |
sdake | andrewvwebber how does fleet fit in with kubernetes? | 16:32 |
andrewvwebber | sdake: fleet to kubernetes is like ansible to openstack/kolla. very poor analogy | 16:32 |
andrewvwebber | sdake: fleet is used to deploy kubernetes, however it can also be used as a scheduler | 16:33 |
sdake | i asked how it fits in, not if it is the same | 16:33 |
andrewvwebber | fleet is used to launch all the prerequisites one would need to deploy kubernetes; certificates, bonding, etcd, etc. | 16:34 |
andrewvwebber | plus one would deploy kubernetes with fleet | 16:34 |
inc0 | oh, so it might actually be alternative to ansible | 16:34 |
andrewvwebber | correct | 16:34 |
inc0 | if you can figure out bootstrapping | 16:35 |
inc0 | which you should | 16:35 |
inc0 | interesting indeed | 16:35 |
mark-casey1 | inc0: as far as scheduling containers go. I don't think so much on building them | 16:35 |
andrewvwebber | this is how one could deploy kolla | 16:35 |
inc0 | mark-casey1, we don't use ansible to build as well | 16:35 |
inc0 | just to schedule containers and configuration stuff | 16:35 |
mark-casey1 | .j2 dockerfiles? | 16:35 |
inc0 | jinja != ansible | 16:36 |
inc0 | we do use jinja, but not ansible | 16:36 |
mark-casey1 | right. ok. I didn't realize you were rendering the templates outside of ansible | 16:36 |
inc0 | https://github.com/openstack/kolla/blob/master/kolla/cmd/build.py | 16:36 |
inc0 | this is where it happends | 16:37 |
inc0 | mark-casey1, we explicitly want to decouple ansible from kolla | 16:37 |
inc0 | so things like coreos fleet or tripleo might use kolla containers just as much | 16:37 |
andrewvwebber | inc0: that was a great decision | 16:37 |
inc0 | granted, ansible will probably have most logic in it | 16:38 |
inc0 | so other will need to follow | 16:38 |
inc0 | but we will keep it separate and we will keep it possible | 16:38 |
*** mgoddard has joined #openstack-kolla | 16:38 | |
*** mgoddard_ has quit IRC | 16:38 | |
inc0 | things like upgrades or reconfigure are ansible specific now | 16:38 |
*** dougs1 has joined #openstack-kolla | 16:38 | |
inc0 | but with right amount of work it can be replicated wherever | 16:38 |
inc0 | it's not super hard as well, all you need to do is copy-paste-change syntax | 16:39 |
inc0 | logic is there | 16:39 |
inc0 | andrewvwebber, one thing, if you actually make it work with coreos fleet, any chances this can go upstream? | 16:39 |
inc0 | I'd love to have kolla-fleet repo | 16:39 |
mark-casey1 | inc0: would you say the goal is not to move away from ansible within Kolla but to allow non-ansible components from Kolla to work elsewhere? | 16:40 |
inc0 | mark-casey1, or rather non-ansible tools make use of kolla docker containers | 16:40 |
andrewvwebber | inc0: of course, happy to | 16:40 |
inc0 | so containers will never be ansible-specific | 16:41 |
rhallisey | andrewvwebber, would you guys be interested in kolla-kubernetes? | 16:41 |
rhallisey | if it were to become a thing? | 16:41 |
inc0 | ansible will be perfectly good, working solution and it will provide a template for others to reproduce | 16:41 |
vhosakot | mark-casey1: docker/<service name>/Dockerfile.j2 is rendered by jina python client in build.py, and ansible/roles/<service>/templates/*.j2 files are rendered by Ansible's "template" module | 16:41 |
inc0 | vhosakot, that's correct | 16:41 |
inc0 | mark-casey1, even tho ansible is bulk of kolla code, containers are what really matters about this project | 16:42 |
mark-casey1 | rhallisey: for me that is my end goal. If kubernetes could run hypervisors inside pods already without crazy crazy hax I probably would even have the openstack layer | 16:42 |
mark-casey1 | *wouldn't | 16:43 |
*** dougs has quit IRC | 16:43 | |
rhallisey | mark-casey1, well if your at summit we can discuss more about it | 16:43 |
inc0 | mark-casey1, I think we'll have session about k8s in Austin | 16:43 |
rhallisey | I have a wip for it | 16:43 |
rhallisey | inc0, ya that'd be cool | 16:43 |
mark-casey1 | I've put in a request for it. and actually if I get denied I might come down for an offsite. I'm in Dallas | 16:44 |
inc0 | even if not, that's something we can talk over a beer | 16:44 |
sbezverk | anybody knows anyone in galera domain? | 16:44 |
mark-casey1 | sbezverk: I'm no guru but yes | 16:44 |
andrewvwebber | kolla-kubernetes would be interesting but its still not clear how that sit architecturally. launching kolla kubernetes on google container engine for me make little sense as google container engine runs in VMs as far as i know. | 16:44 |
mark-casey1 | I've been running it in stage for our app for a bit now | 16:44 |
inc0 | mark-casey1, I'm from San Antonio, so if you happen to be in area, would gladly share a liqor;) | 16:45 |
mark-casey1 | inc0: cool :( | 16:45 |
mark-casey1 | oops | 16:45 |
mark-casey1 | ;) | 16:45 |
sbezverk | mark-casey1 I see some extremelly odd behavior between three controllers running galera/mariadb, need some help to figure out what might be wrong.. | 16:46 |
*** chopmann has quit IRC | 16:46 | |
vhosakot | mark-casey1: I dont like to use the phrase "jinja template gets rendered" as "rednering" of *.j2 is something that happens on the UI if we use *.j2 to create HTML or PHP...I think *.j2 templates get "built" by docker after variables are substituted at runtime... I think "render" is "built" in kolla :) | 16:46 |
*** kbyrne has joined #openstack-kolla | 16:47 | |
mark-casey1 | sbezverk: sure happy to try and help | 16:47 |
vhosakot | or, may be we could say render is built... I'm used to thinking rendering as something that happens on an UI on screen :) | 16:47 |
andrewvwebber | rhallisey: has anyone asked for kolla-docker-swarm? | 16:47 |
mlima_ | we need some reviews on mitaka https://review.openstack.org/#/c/305840/ https://review.openstack.org/#/c/305775/ https://review.openstack.org/#/c/305588/ https://review.openstack.org/#/c/305770/ https://review.openstack.org/#/c/305606/ | 16:47 |
patchbot | mlima_: patch 305840 - kolla (stable/mitaka) - Use proper CentOS repositories for ceph and qemu | 16:47 |
patchbot | mlima_: patch 305775 - kolla (stable/mitaka) - cleanup-host should not remove kolla-build.conf | 16:47 |
patchbot | mlima_: patch 305588 - kolla (stable/mitaka) - Added help documentation for kolla-ansible upgrade | 16:47 |
patchbot | mlima_: patch 305770 - kolla (stable/mitaka) - Enable unbuffered output for ansible | 16:47 |
patchbot | mlima_: patch 305606 - kolla (stable/mitaka) - Fixes kolla_toolbox failure due to missing pytz mo... | 16:47 |
rhallisey | andrewvwebber, no | 16:48 |
vhosakot | mlima_: will review | 16:48 |
mark-casey1 | vhosakot: yea that makes some sense | 16:48 |
rhallisey | andrewvwebber, I'd say were still expanding the kolla-* projects | 16:48 |
mark-casey1 | to me | 16:48 |
rhallisey | since we really only have 1 other (kolla-mesos) | 16:48 |
*** chopmann_ has joined #openstack-kolla | 16:48 | |
*** chopmann_ is now known as chopmann | 16:48 | |
rhallisey | we plan to split out kolla & kolla-ansible | 16:48 |
mark-casey1 | sbezverk: is this re: the paste earlier about reaching primary view | 16:49 |
rhallisey | to make 2 | 16:49 |
sbezverk | mark-casey1 here is the log I collected on one of the controller http://paste.openstack.org/show/494103/ it looks like it is connectivity issue but IP unicast is working fine between them.. | 16:49 |
andrewvwebber | rhallisey: i think kolla-kubernetes would bring alot of attention to the project, at the very least. the 'perfect' hybrid solution | 16:49 |
mark-casey1 | sbezverk: this was post-reboot. is that right? | 16:49 |
sbezverk | mark-casey1 yes, but that one was partial.. here is kind of complete cycle | 16:49 |
sbezverk | mark-casey1 yes sir | 16:49 |
mlima_ | vhosakot, I Agree | 16:50 |
rhallisey | andrewvwebber, to me, I think it would do well on the controller. I don't know how well kolla-kuberentes would do for compute nodes | 16:50 |
rhallisey | kube doesn't seem to scale | 16:50 |
mark-casey1 | sbezverk: how clean was the shutdown? is there any chance that the first node being bootstrapped is a commit or two behind another node? | 16:51 |
mark-casey1 | err... db commit of course | 16:51 |
sbezverk | mark-casey1 well I just rebooted nodes, exactly the same way I do with classic openstack. | 16:52 |
sdake | rhallisey coreos hsa a partnership with intel working on openstack on kubernetes in containers | 16:52 |
sdake | the logical step to take ther eis to make ea kolla-kubernetes repo and get to work with them on it | 16:52 |
sbezverk | mark-casey1 "160414 16:44:42 [Warning] WSREP: no nodes coming from prim view, prim not possible" this specific line means peers cannot communicate or all peers have sort of corruption? | 16:52 |
rhallisey | sdake, right. Maybe a post summit project | 16:52 |
andrewvwebber | rhallisey: yes, its been a long time since i used kubernetes but my understanding is that it does not have tenant isolation | 16:53 |
inc0 | andrewvwebber, you work with intel folks on this? | 16:53 |
inc0 | care to share names? | 16:53 |
mark-casey1 | sbezverk: it's a slim case but if the first node dropped and the cluster still had quorum the other two nodes could have commited a change before shutting down. if you bootstrap on the member that was a commit behind you could maybe get some errors like you're getting | 16:54 |
andrewvwebber | sdake: totally agree, i have asked them about the repo in IRC but they are in "watch this space" mode | 16:54 |
mark-casey1 | sbezverk: this is just a test cluster right? | 16:55 |
rhallisey | andrewvwebber, we'll try and talk about it at summit and maybe we can come out of it with a new repo | 16:55 |
mark-casey1 | http://galeracluster.com/documentation-webpages/quorumreset.html | 16:56 |
andrewvwebber | rhallisey: it would be in both parties interest. i think it doesnt make sense for coreos to maintain a seperate repro from kolla | 16:56 |
sbezverk | mark-casey1 hmmmm, if just normal reboot cause cause this nightmare, maybe we need a script to shutdown services gracefully before rebooting node | 16:56 |
rhallisey | andrewvwebber, are you going to be at summit? | 16:56 |
andrewvwebber | rhallisey: just like to contribute back to kubernetes as their tectonic offering is managed kubernetes. | 16:57 |
rhallisey | along with some of your colleagues | 16:57 |
mark-casey1 | sbezverk: it may not have this time but my understanding is it absolutely can on a busy cluster. it's my biggest problem left to solve...: when recovering my ansible stuff has to attempt bootstrap on the most advanced node | 16:57 |
mark-casey1 | sbezverk: http://galeracluster.com/documentation-webpages/galeraparameters.html#pc-bootstrap | 16:58 |
mark-casey1 | if this is the problem that particular option will probably fix it | 16:58 |
mark-casey1 | but, if you do it on a node that was behind, the couple commits that the other two nodes were ahead will get a full SST and those commits will be lost | 16:59 |
sdake | andrewvwebber mind i nosey around where you worik? | 16:59 |
andrewvwebber | rhallisey: i would love... | 16:59 |
sbezverk | mark-casey1 so if I change galera.conf and add this option it might recover right? | 17:00 |
mark-casey1 | sbezverk: yea thats the theory | 17:00 |
sbezverk | mark-casey1 so right now all container with mariadb are flapping, I cannot change this parameter on the fly, but I can add it to json config, do you think having this parameter permanently will have any negative imapct? | 17:01 |
sdake | andrewvwebber watch which sapce | 17:02 |
andrewvwebber | sake: sure, i work for a security company in munich germany doing B2B file sync and share. we're a golang shop on couchbase, ceph and baremetal coreos. im the architect trying to make VMs and containers co-exist | 17:02 |
sdake | andrewvwebber not trying doing :) | 17:03 |
*** chopmann has quit IRC | 17:03 | |
*** chopmann has joined #openstack-kolla | 17:03 | |
mark-casey1 | sbezverk: I'm actually on percona's galera; afaik I have to pass this when bootstrapping. assuming you can make this permanent I would not. my understanding is that it tells the cluster to ignore split brain and roll back to last common state | 17:03 |
mark-casey1 | primary component =~ the synchronous cluster's quorum | 17:04 |
andrewvwebber | sake: standing on the shoulders of giants | 17:04 |
mark-casey1 | ***I have to pass then when bootstrapping after bad things have happened and assuming I don't mind losing some stuff :D | 17:07 |
sdake | do we need percona | 17:08 |
sdake | rdo repos contain galera now | 17:08 |
mark-casey1 | sdake: these issues seem to be wholly related to the galera plugin and afaik are common to maria and all other users | 17:08 |
sbezverk | mark-casey1 I do not see a way to recover then :-( it flaps too fast to be able to update that file in between flaps | 17:10 |
mark-casey1 | sbezverk: have you tried stopping all nodes and bootstrapping each one, one at a time? | 17:11 |
mark-casey1 | if this is the issue. I would think one of them woudl work | 17:11 |
*** andrewvwebber has quit IRC | 17:11 | |
mark-casey1 | sbezverk: can you see if you have a file called grastate.dat? | 17:11 |
sbezverk | mark-casey1 let me try that | 17:12 |
mark-casey1 | ok. try first. then file | 17:12 |
*** athomas has quit IRC | 17:14 | |
mark-casey1 | sorry if this is blasphemous but I think osa has dealt with this some too. https://github.com/openstack/openstack-ansible/blob/master/doc/source/install-guide/ops-galera-recovery.rst | 17:14 |
mark-casey1 | the multi-node failure section... as a shutdown without tiptoe-ing can more or less be a multi-node fail ;? | 17:15 |
mark-casey1 | * :/ | 17:15 |
sbezverk | mark-casey1 with single node still flaps | 17:16 |
sbezverk | mark-casey1 I see this file /var/lib/mysql/grastate.dat | 17:17 |
sdake | mark-casey1 we have mariadb lights out recovery playbook | 17:17 |
sdake | mark-casey1 i think that solves this problem | 17:17 |
sbezverk | sdake could you point me to the command to initiate it? it would be great test to see if it is working.. | 17:18 |
*** vhosakot_ has joined #openstack-kolla | 17:18 | |
sdake | sbezverk nevr tried it let me see what i can find | 17:18 |
sbezverk | mark-casey1 what that file means? | 17:18 |
sdake | need to drop of the vpn moment | 17:18 |
mark-casey1 | sbezverk: that file gets removed on clean cluster shutdown. it records extra state for restarting cluster. do any nodes have a value of seqno: that is not -1? | 17:19 |
*** vhosakot has quit IRC | 17:20 | |
*** mwheckma1n has quit IRC | 17:20 | |
*** kbyrne has quit IRC | 17:21 | |
*** ayoung has quit IRC | 17:21 | |
*** mlima_ has quit IRC | 17:21 | |
*** jmccarthy has quit IRC | 17:21 | |
*** rbergeron has quit IRC | 17:21 | |
*** larsks has quit IRC | 17:21 | |
sbezverk | mark-case1 node 1 http://paste.openstack.org/show/494111/ | 17:21 |
*** mlima_ has joined #openstack-kolla | 17:21 | |
sbezverk | mark-casey1 sorry for misspel | 17:21 |
*** rmart04 has quit IRC | 17:21 | |
mark-casey1 | no worries. also to double check you need a node with a value != -1 | 17:21 |
sbezverk | mark-casey1 all nodes have seq -1 | 17:22 |
mark-casey1 | ok | 17:22 |
*** rbergeron has joined #openstack-kolla | 17:22 | |
*** mwheckmann has joined #openstack-kolla | 17:22 | |
sbezverk | mark-casey1 if I remove this file on all nodes? | 17:23 |
mark-casey1 | I know this isn't maria specific but it should mostly apply IF the lights-out option doesn't cover this. look at Scenario 6 https://www.percona.com/blog/2014/09/01/galera-replication-how-to-recover-a-pxc-cluster/ | 17:23 |
*** sdake has quit IRC | 17:23 | |
mark-casey1 | oh... actually is looks like an automatic solution is coming for this or is already in galera | 17:24 |
sbezverk | <mark-casey1 thank you for the link if lights out does not help, will play with suggested workaround, really appreciate you sharing these very useful links. | 17:24 |
mark-casey1 | my pleasure | 17:25 |
*** banix has quit IRC | 17:25 | |
*** sdake has joined #openstack-kolla | 17:26 | |
sdake | sbezverk bck | 17:26 |
sdake | sbezverk kolla-ansible mariadb_recovery | 17:26 |
mark-casey1 | looking at that playbook I think it will work | 17:28 |
mark-casey1 | I think there are cases it could cause data loss though | 17:31 |
sbezverk | ok starting it | 17:31 |
sbezverk | sdake mark-casey1 so far looks good it is not flapping, need to check if I lost any data | 17:32 |
mark-casey1 | probably won't show in this case | 17:32 |
sdake | sbezverk after running that playbook yu mean? | 17:32 |
sbezverk | sdake yes | 17:32 |
sdake | hotness | 17:32 |
mark-casey1 | that method passes --wsrep-new-cluster which creates a new quorum from whatever state the bootstrapping node happens to have | 17:33 |
mark-casey1 | galera docs: Note Warning: Only use the --wsrep-new-cluster argument when initializing the Primary Component. Do not use it when you want the node to connect to an existing cluster. | 17:33 |
sbezverk | sdake mark-casey1 wow data is still there !! great.. | 17:33 |
mark-casey1 | in this case even though they didn't think they had quorum the data was all probably the same | 17:33 |
*** larsks has joined #openstack-kolla | 17:34 | |
mark-casey1 | if a node had gone down a week ago and lights out happened to bootstrap from it it'd be sad panda | 17:34 |
sbezverk | sdake but then there should be a procedure to gracefully shut down | 17:34 |
*** unicell has joined #openstack-kolla | 17:34 | |
sdake | sbezverk should be and are ae two different htings :) | 17:34 |
sbezverk | sdake for sure, I am just checking if I missed it or it does not exist | 17:35 |
*** ayoung has joined #openstack-kolla | 17:35 | |
*** mark-casey has joined #openstack-kolla | 17:36 | |
mark-casey | ...wifi dropped or something | 17:36 |
*** dwalsh has quit IRC | 17:37 | |
*** jmccarthy has joined #openstack-kolla | 17:38 | |
*** kbyrne has joined #openstack-kolla | 17:38 | |
*** mark-casey1 has quit IRC | 17:38 | |
*** ravig has quit IRC | 17:40 | |
mark-casey | sdake sbezverk imho this method needs a disclaimer that you could lose data and some steps on how to estimate how much could be lost, but a drunken guess is that it is in fact the correct thing to do 80-90% of the time | 17:42 |
openstackgerrit | Merged openstack/kolla: Added help documentation for kolla-ansible upgrade https://review.openstack.org/305588 | 17:42 |
sdake | can i get naother core reviewer to take a look at this review https://review.openstack.org/#/c/302434/5 | 17:43 |
patchbot | sdake: patch 302434 - kolla - Set db connection retry to infinity | 17:43 |
*** salv-orlando has quit IRC | 17:44 | |
sdake | sbezverk where is the review for this https://bugs.launchpad.net/kolla/+bug/1553895 | 17:45 |
openstack | Launchpad bug 1553895 in kolla "Drop root for swift" [High,In progress] - Assigned to Serguei Bezverkhi (sbezverk) | 17:45 |
*** mgoddard_ has joined #openstack-kolla | 17:48 | |
*** sdake_ has joined #openstack-kolla | 17:48 | |
*** mark-casey1 has joined #openstack-kolla | 17:49 | |
*** mkoderer__ has joined #openstack-kolla | 17:49 | |
*** unicell has quit IRC | 17:50 | |
*** mkoderer___ has quit IRC | 17:50 | |
*** unicell has joined #openstack-kolla | 17:50 | |
*** unicell has quit IRC | 17:50 | |
*** unicell has joined #openstack-kolla | 17:50 | |
sdake_ | sbezverk when you find the droproot swift patch please link it in the bug | 17:50 |
sdake_ | for some reason openstack infra didnt link it | 17:51 |
*** mark-casey has quit IRC | 17:51 | |
*** sdake has quit IRC | 17:51 | |
*** mgoddard has quit IRC | 17:51 | |
mark-casey1 | bbl food | 17:51 |
*** mark-casey1 has quit IRC | 17:51 | |
*** dwalsh has joined #openstack-kolla | 17:55 | |
*** rajathagasthya has quit IRC | 17:58 | |
*** salv-orlando has joined #openstack-kolla | 18:01 | |
*** ravig has joined #openstack-kolla | 18:02 | |
*** rajathagasthya has joined #openstack-kolla | 18:03 | |
*** sdake_ is now known as sdake | 18:04 | |
*** ravig has quit IRC | 18:08 | |
sdake | ok folks we have 5 criticcal/high bugs left | 18:09 |
sdake | pease please try to get them merged by tomorrow morning | 18:10 |
sdake | inc0 can you jut ack ryan's work | 18:10 |
sdake | i understand jeffrye's complaint but i think infinity is the right answer | 18:10 |
*** ravig has joined #openstack-kolla | 18:11 | |
sdake | inc0 https://review.openstack.org/#/c/302434/ | 18:11 |
patchbot | sdake: patch 302434 - kolla - Set db connection retry to infinity | 18:11 |
*** ravig has quit IRC | 18:13 | |
sdake | vhosakot_ ^^ | 18:13 |
vhosakot_ | sdake: cool, will review | 18:14 |
*** ravig has joined #openstack-kolla | 18:15 | |
*** ravig has quit IRC | 18:17 | |
*** sdake_ has joined #openstack-kolla | 18:17 | |
*** sdake has quit IRC | 18:19 | |
*** andrewvwebber has joined #openstack-kolla | 18:21 | |
*** ravig has joined #openstack-kolla | 18:22 | |
*** gfidente has quit IRC | 18:28 | |
sdake_ | sbezverk drop root for swift - status? | 18:28 |
*** Marga__ has quit IRC | 18:30 | |
vhosakot_ | sdake_: Jeffrey and mlima_ have -1 for https://review.openstack.org/#/c/302434/.. the PD LGTM | 18:31 |
patchbot | vhosakot_: patch 302434 - kolla - Set db connection retry to infinity | 18:31 |
sdake_ | lets just get it merged | 18:32 |
*** sdake_ is now known as sdake | 18:32 | |
sdake | i dont want to block on figuring out the "perfect" numbeer | 18:32 |
sdake | imo the perfect number is infinity | 18:32 |
sdake | we dont have time to eresearch what the perfect number is | 18:32 |
*** achanda has joined #openstack-kolla | 18:33 | |
ccesario | sbezverk, could you make available your inventory file ? | 18:36 |
*** achanda has quit IRC | 18:37 | |
openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Add Kibana quick start quide https://review.openstack.org/306019 | 18:38 |
vhosakot_ | sdake: yes, I agreee infinity is fine... let the services try to reconnect forever.. | 18:42 |
sdake | so ack the patch then ;) | 18:42 |
*** egonzalez has joined #openstack-kolla | 18:45 | |
*** egonzalez has quit IRC | 18:45 | |
*** egonzalez has joined #openstack-kolla | 18:46 | |
*** ravig_ has joined #openstack-kolla | 18:48 | |
sdake | wtb ack on https://review.openstack.org/#/c/302434/ | 18:50 |
patchbot | sdake: patch 302434 - kolla - Set db connection retry to infinity | 18:50 |
sbezverk | sdake ok will take care of swift drop root | 18:50 |
*** ravig has quit IRC | 18:50 | |
sdake | sbezverk friday is deadline | 18:51 |
*** banix has joined #openstack-kolla | 18:53 | |
sbezverk | sdake it has a right bug id, it is juts missing one +2 | 18:54 |
sdake | ink plz | 18:54 |
sdake | doe it work? | 18:54 |
mimizone | What is expected from this precheck "Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes"?? Can't seem to find what to do to not fail. | 18:54 |
sbezverk | https://review.openstack.org/#/c/259374/ | 18:54 |
patchbot | sbezverk: patch 259374 - kolla - Drop root for swift | 18:54 |
sbezverk | sdake yep | 18:54 |
*** andrewvwebber has quit IRC | 18:55 | |
mimizone | should the VIP be completely on a different network that is not on any interfaces on the hosts? | 18:55 |
openstackgerrit | Steven Dake proposed openstack/kolla: Drop root for swift https://review.openstack.org/306030 | 18:56 |
*** rmart04 has joined #openstack-kolla | 18:56 | |
mimizone | answering my own question. putting the VIP on a different network than is not defined on any interface seems to be accepted by the precheck | 18:57 |
mimizone | oh no correcting myself now... | 18:58 |
mimizone | not working... | 18:58 |
*** Marga_ has joined #openstack-kolla | 18:58 | |
*** rmart04 has quit IRC | 18:58 | |
ccesario | elemoine_, https://bugs.launchpad.net/kolla/+bug/1570550 | 18:58 |
openstack | Launchpad bug 1570550 in kolla "elasticsearch.url param point to host address when haproxy it is enabled" [Undecided,New] | 18:58 |
mimizone | does this error message make sense to somebody? FAILED! => {"failed": true, "msg": "The conditional check ''169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3])' failed. The error was: error while evaluating conditional ('169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3 | 18:58 |
mimizone | ])): list object has no element 3"} | 18:58 |
vhosakot_ | mimizone: ping | 18:59 |
vhosakot_ | what is your network_interface and kolla_internal_vip_address in globals.yml ? | 18:59 |
mimizone | for instance kolla_internal_vip_address: "172.30.4.100" | 18:59 |
openstackgerrit | Merged openstack/kolla: Drop root for swift https://review.openstack.org/259374 | 18:59 |
mimizone | network_interface: "eth2.2002" | 18:59 |
mimizone | but I also give a different set of interface per host cause I have different hardware | 19:00 |
vhosakot_ | mimizone: can you pastebin your globals.yml and output of "ip a" | 19:00 |
mimizone | ip a? | 19:00 |
mimizone | on which node? | 19:01 |
openstackgerrit | Merged openstack/kolla: Set db connection retry to infinity https://review.openstack.org/302434 | 19:01 |
vhosakot_ | mimizone: globals.yml from deploy node and "ip a" from deploy node as well | 19:02 |
*** sdake has quit IRC | 19:02 | |
mimizone | vhosakot_: http://paste.openstack.org/show/494120/ | 19:05 |
*** sdake has joined #openstack-kolla | 19:05 | |
ccesario | sdake, this ps https://review.openstack.org/#/c/302434/ solve it ? http://paste.openstack.org/show/494121/ | 19:06 |
patchbot | ccesario: patch 302434 - kolla - Set db connection retry to infinity (MERGED) | 19:06 |
ccesario | vhosakot_, ^^ | 19:06 |
vhosakot_ | ccesario: checking... thanks | 19:07 |
openstackgerrit | Steven Dake proposed openstack/kolla: Set db connection retry to infinity https://review.openstack.org/306036 | 19:07 |
daneyon_ | vhosakot_ Do you know if any patches merged that fixes this bug? https://bugs.launchpad.net/kolla/+bug/1567657 | 19:07 |
openstack | Launchpad bug 1567657 in kolla "Build Fails Using Trunk" [Undecided,New] | 19:07 |
sdake | https://review.openstack.org/#/c/306036/1 needs an ack | 19:07 |
patchbot | sdake: patch 306036 - kolla (stable/mitaka) - Set db connection retry to infinity | 19:07 |
rhallisey | ccesario, that's a different issue | 19:08 |
vhosakot_ | daneyon_: no, retrying works for me on UBuntu... are you using CentOS | 19:08 |
daneyon_ | vhosakot_ yes, I'm using centos. My use case req's centos | 19:09 |
ccesario | rhallisey, thank you.... i 'm making a double check to confirm it | 19:09 |
vhosakot_ | daneyon_: the error is not seen in gate.. I've been checking if centos gate sees it and it does not | 19:10 |
daneyon_ | vhosakot_ OK. I just started a fresh build. I'll see if I hit it again and update the bug. | 19:11 |
vhosakot_ | daneyon_: ok, please.. thanks! | 19:11 |
*** mark-casey has joined #openstack-kolla | 19:13 | |
vhosakot_ | mimizone: can you paste the output of "ip -4 -o addr show dev eth2.2002" ? | 19:15 |
vhosakot_ | mimizone: from deploy mode | 19:15 |
mark-casey | rhallisey inc0 my request for attending summit was approved | 19:15 |
rhallisey | sweet | 19:15 |
mimizone | vhosakot_: http://paste.openstack.org/show/494122/ | 19:16 |
vhosakot_ | mimizone: 1 sec.. let me check | 19:16 |
mimizone | I think I may have found the thing. the VIP shoudl be in the IP range | 19:17 |
vhosakot_ | mimizone: yes :) | 19:17 |
*** ravig_ has quit IRC | 19:17 | |
vhosakot_ | mimizone: I was gonna say that | 19:17 |
mimizone | we are still configuring things around, but we use different IP ranges per rack | 19:18 |
*** ravig has joined #openstack-kolla | 19:18 | |
vhosakot_ | mimizone: kolla_internal_vip_address (172.30.81.100) is _not_ in the range 172.30.81.153/25 | 19:18 |
mimizone | and our DHCP/PXE server for this setup is still being configured for that (Maas). | 19:18 |
sdake | https://review.openstack.org/#/c/306037/1/reference/tags/release_cycle-trailing.rst | 19:18 |
patchbot | sdake: patch 306037 - governance - add release:cycle-trailing tag | 19:18 |
mimizone | Cool confirming is good | 19:18 |
*** ravig has quit IRC | 19:19 | |
vhosakot_ | mimizone: kolla needs kolla_internal_vip_address to be in the same network range as network_interface (eth2.2002) which is 172.30.81.153/25... hence, chane kolla_internal_vip_address to a free IP in the range 172.30.81.128 - 172.30.81.255 | 19:20 |
inc0 | mark-casey, yay! | 19:20 |
mimizone | I'll see how to play around this. I can do a quick test in the same range, but ultimately the VIP should the in the larger /21 of the same subnet. We slice a large /21 across 16 racks to make /25. | 19:21 |
*** ravig has joined #openstack-kolla | 19:21 | |
mimizone | thanks vhosakot_ | 19:21 |
vhosakot_ | mimizone: cool!... it is kolla's requirements... else, services will have issues -->kolla needs kolla_internal_vip_address to be in the same network range as network_interface | 19:22 |
vhosakot_ | mimizone: search for "unused" in http://docs.openstack.org/developer/kolla/quickstart.html | 19:23 |
mimizone | but our controllers may end up in different racks, therefore in a different /25 | 19:23 |
*** ravig has quit IRC | 19:29 | |
*** ravig has joined #openstack-kolla | 19:30 | |
vhosakot_ | mimizone: all the nodes must be in the same network | 19:31 |
*** ravig has quit IRC | 19:31 | |
mimizone | what is the reason for that? we route at layer3 in the top of rack switch. all nodes see each other at the IP level. | 19:32 |
*** ravig has joined #openstack-kolla | 19:32 | |
mimizone | is that only for the VIP the requirement? so for the API_interface? | 19:32 |
vhosakot_ | mimizone: mmm, in that case, will kolla_internal_vip_address will be outside the range of network_interface (outside of 172.30.81.153/25 in this case) | 19:33 |
mimizone | the vip would be in the corresponding 172.30.80.0/21 | 19:36 |
vhosakot_ | mimizone: api_interface as well.. yes.. https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L81 | 19:37 |
mimizone | I expect other surprises then :) | 19:39 |
*** mark-casey has quit IRC | 19:40 | |
*** rajathagasthya has quit IRC | 19:40 | |
*** rajathagasthya has joined #openstack-kolla | 19:45 | |
*** chopmann has quit IRC | 19:46 | |
*** sdake_ has joined #openstack-kolla | 19:47 | |
*** sdake has quit IRC | 19:49 | |
sbezverk | inc0 ping | 19:53 |
mimizone | vhosakot_: I sort of understand why the vip shouldbe in the same network as the controllers, but I don't understand why there is a check on the other roles. As long as the VIP is routed from those nodes, that should be enough. the Storage nodes don't need to care if the VIP is in a totally different network, or at least Ip range. | 19:54 |
mimizone | Don't you think? | 19:54 |
*** akscram has quit IRC | 19:58 | |
*** rajathagasthya has quit IRC | 19:58 | |
mimizone | also the precheck doesn't use the right interface for the check. it uses the value from the globals 'network_interface' and not the api_interface, neutron_interface etc... values I put in the inventory. should the network_interface also be overridden in the inventory? | 19:59 |
*** akscram has joined #openstack-kolla | 19:59 | |
vhosakot_ | mimizone: api_interface is same as network_interface https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L81 | 20:01 |
mimizone | I overridden them in the inventory. is that ignored? | 20:01 |
vhosakot_ | mimizone: you mean in globals.yml ? | 20:02 |
mimizone | nope in the ansible inventory file | 20:02 |
mlima_ | guys, we need to review here https://review.openstack.org/#/c/306036/ https://review.openstack.org/#/c/306030/ https://review.openstack.org/#/c/305840/ https://review.openstack.org/#/c/305775/ https://review.openstack.org/#/c/305770/ | 20:02 |
patchbot | mlima_: patch 306036 - kolla (stable/mitaka) - Set db connection retry to infinity | 20:02 |
patchbot | mlima_: patch 306030 - kolla (stable/mitaka) - Drop root for swift | 20:02 |
patchbot | mlima_: patch 305840 - kolla (stable/mitaka) - Use proper CentOS repositories for ceph and qemu | 20:02 |
patchbot | mlima_: patch 305775 - kolla (stable/mitaka) - cleanup-host should not remove kolla-build.conf | 20:02 |
patchbot | mlima_: patch 305770 - kolla (stable/mitaka) - Enable unbuffered output for ansible | 20:02 |
mimizone | vhosakot_: for exemple "osv11syn21b ansible_user=ubuntu neutron_external_interface=eth2 api_interface=eth2.2002 storage_interface=eth2.2002 tunnel_interface=eth2.2002" | 20:03 |
vhosakot_ | mimizone: here https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode#L21-L22 ? | 20:03 |
*** ayoung has quit IRC | 20:08 | |
ccesario | sbezverk, ping ? | 20:09 |
ccesario | sbezverk, really the problem happen | 20:11 |
ccesario | sbezverk, http://sprunge.us/dHXL | 20:11 |
ccesario | sbezverk, http://paste.openstack.org/show/494125/ | 20:13 |
ccesario | the mariadb container stay rebooting | 20:14 |
inc0 | sbezverk, pong, wassup? | 20:14 |
*** egonzalez has quit IRC | 20:21 | |
*** ravig has quit IRC | 20:22 | |
ccesario | sbezverk, steps to reproduce http://paste.openstack.org/show/494127/ | 20:26 |
ccesario | inc0, did you test ? | 20:26 |
*** inc0 has quit IRC | 20:27 | |
*** rmart04 has joined #openstack-kolla | 20:28 | |
*** rajathagasthya has joined #openstack-kolla | 20:31 | |
*** rmart04 has quit IRC | 20:35 | |
*** ravig has joined #openstack-kolla | 20:41 | |
ccesario | guys...if someone could test ..... I did double check and the error happen | 20:43 |
ccesario | sbezverk, ^^ | 20:43 |
*** banix has quit IRC | 20:47 | |
*** banix has joined #openstack-kolla | 20:48 | |
*** banix has quit IRC | 20:50 | |
*** ccesario_ has joined #openstack-kolla | 20:57 | |
*** banix has joined #openstack-kolla | 21:00 | |
*** ravig has quit IRC | 21:07 | |
*** ccesario_ has quit IRC | 21:07 | |
*** ccesario_ has joined #openstack-kolla | 21:10 | |
*** ccesario_ has quit IRC | 21:12 | |
*** ccesario___ has joined #openstack-kolla | 21:12 | |
*** ravig has joined #openstack-kolla | 21:13 | |
*** ccesario___ has quit IRC | 21:17 | |
*** salv-orl_ has joined #openstack-kolla | 21:17 | |
*** ccesario___ has joined #openstack-kolla | 21:17 | |
*** salv-orlando has quit IRC | 21:20 | |
*** dwalsh has quit IRC | 21:22 | |
*** ccesario___ has quit IRC | 21:23 | |
*** ccesario___ has joined #openstack-kolla | 21:24 | |
*** ccesario___ has quit IRC | 21:29 | |
vhosakot_ | mimizone: I'm still thinking about your use case.. we need to chat more | 21:36 |
*** mark-casey has joined #openstack-kolla | 21:39 | |
mark-casey | does Kolla support centos prior to version 7? | 21:41 |
*** ccesario_ has joined #openstack-kolla | 21:43 | |
*** haplo37 has quit IRC | 21:44 | |
vhosakot_ | mark-casey: I dont think it is tested.. dont know if we get the needed repos in centOS 6 | 21:45 |
mark-casey | vhosakot_: ty | 21:45 |
ccesario_ | vhosakot_: ping ? | 21:47 |
vhosakot_ | mark-casey: this is how we point to 7 https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L22 | 21:48 |
vhosakot_ | ccesario_: pong | 21:48 |
ccesario_ | vhosakot_: did you read my last messages there in channel ? | 21:48 |
ccesario_ | I think that we have problems with multinode and mariadb as previously said by sbezverk | 21:49 |
vhosakot_ | ccesario_: steps to repproduce mariadb reboot issue ? | 21:49 |
ccesario_ | yes man | 21:49 |
vhosakot_ | yes saw that | 21:49 |
ccesario_ | I repeat it 3 times and in all tests the problem happen | 21:50 |
vhosakot_ | ccesario_: you reboot both control nodes 192.168.201.2 and 192.168.201.3 simultaneously, right ? | 21:50 |
ccesario_ | yes | 21:51 |
ccesario_ | and IMHO this need to be checked ... https://bugs.launchpad.net/kolla/+bug/1570550 | 21:51 |
openstack | Launchpad bug 1570550 in kolla "elasticsearch.url param point to host address when haproxy it is enabled" [Undecided,New] | 21:51 |
ccesario_ | because this no make sense when haproxy t is enabled ..... | 21:52 |
*** ravig has quit IRC | 21:53 | |
ccesario_ | in my tests these are my reports :) and deadline is tomorrow according sdake_ | 21:54 |
vhosakot_ | ccesario_: that bug is not targedted for 2.0.0.. is it ? | 21:54 |
ccesario_ | vhosakot_: because I have checked it yesterday and discussed with elemoine_ | 21:55 |
vhosakot_ | ah ok...cool | 21:56 |
ccesario_ | vhosakot_: :) | 21:57 |
ccesario_ | I'll back in 2 hours... need slve a customer problem now :/ | 21:59 |
vhosakot_ | ccesario_: cool | 21:59 |
*** ravig has joined #openstack-kolla | 22:01 | |
*** pbourke-home has joined #openstack-kolla | 22:02 | |
*** ravig has quit IRC | 22:02 | |
*** allen_gao has joined #openstack-kolla | 22:04 | |
*** rajathagasthya has quit IRC | 22:04 | |
mimizone | vhosakot_: sure, let's do that. I have some ideas I discussed with a friend | 22:11 |
*** dwalsh has joined #openstack-kolla | 22:15 | |
*** ravig has joined #openstack-kolla | 22:17 | |
*** mwheckmann has quit IRC | 22:18 | |
*** banix has quit IRC | 22:23 | |
mimizone | vhosakot_: in a nutshell, one work around to make it compatible with kolla would be to have a vip unrelated with the network interface. the IP would be advertized to the rest of the BGP fabric using a BGP client on the controllers. If we also use ECMP like we already do in the fabric, the VIP could actually be on all the controllers at the same time, and the fabric would route appropriately using ECMP algo. the other nod | 22:23 |
mimizone | es, non-controllers, should not care if the VIP is on the same network or not. | 22:23 |
mimizone | the quickest work around for me right now, is to switch back to layer2 for the api network. But I don't like this... | 22:24 |
vhosakot_ | mimizone: so, you control, network, storage nodes are all on different networks routable thru layer-3 thru TOR ? | 22:24 |
mimizone | vhosakot_: yes | 22:25 |
mimizone | more like all racks are on different networks | 22:26 |
mimizone | and the tor is a router acting as the default gateway for the rack | 22:26 |
vhosakot_ | mimizone: yes, that is very much valid cloud topology... separating control, network, storage nodes.. | 22:26 |
mimizone | bgp is used to advertise the routes across racks | 22:26 |
vhosakot_ | mimizone: so, the TOR device that sees all the networks.. is that used for kolla at all ? | 22:27 |
*** ph03n1x has quit IRC | 22:27 | |
mimizone | vhosakot_: not sure I get the question. | 22:27 |
vhosakot_ | mimizone: is the device that routes packets from controller's network to storage network used in the kolla inventory ? | 22:28 |
mimizone | vhosakot_: no. kolla doesn't configure anything in the network/tor/router | 22:29 |
vhosakot_ | mimizone: there must a layer-3 point in the topology that sees/knows all the networks and can route packets from one network to other... does this point get routes advertised frm BGP clinet running on controller ? | 22:29 |
mimizone | oh ok, yes the ro sees all the networks in the rack | 22:30 |
mimizone | and if it is outside of the rack, it goes up to one of the spines | 22:30 |
mimizone | based on ECMP/BGP | 22:30 |
vhosakot_ | mimizone: yep... | 22:30 |
mimizone | if it's outside of the fabric, it goes to our main router for the room | 22:30 |
vhosakot_ | mimizone: right | 22:30 |
mimizone | I am thinking of removing the checks for the non-controller nodes for now, and try a deployment with all the controllers in the same rack for now. | 22:32 |
vhosakot_ | mimizone: sounds like this precheck does not make sense in your topology,, can you comment it out and check if kolla works end-to-end and you can successfully create nova VM... https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L459-L464.. if so, yes, it is a bug in prechecks and I will fix it... | 22:32 |
vhosakot_ | mimizone: could you do this test as you have a great topology! :) | 22:32 |
mimizone | cool, we are aligned. | 22:33 |
mimizone | I'll keep you posted | 22:33 |
mimizone | and then we can discuss how to integrate maybe some bgp client in the controller to advertize the VIP to the fabric | 22:33 |
vhosakot_ | mimizone: thanks.. looks forward to hear back! | 22:33 |
mimizone | can you point me to the file where the check is? | 22:33 |
vhosakot_ | mimizone: https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L459-L464 | 22:34 |
vhosakot_ | mimizone: yes, having BGP client on control nodes and using this BGP client to advertise the VIP to the fabric north of (beyond) the TOR is a new and very valid use case, and needs to be addressed | 22:35 |
vhosakot_ | mimizone: thanks for checking this! | 22:35 |
mimizone | I am not planning to be in Austin btw, but would welcome being part of the discussions on that | 22:36 |
mark-casey | 3 | 22:37 |
mark-casey | lol, sorry | 22:37 |
vhosakot_ | mimizone: cool, I will work with you to fix and address this use case... as the first step, it would be nice to know if kolla works end-to-end and if nova VM can be booted successfully wihout that percheck in https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L459-L464 | 22:38 |
vhosakot_ | I'm in this channel most of the time | 22:38 |
*** ayoung has joined #openstack-kolla | 22:39 | |
mimizone | vhosakot_: running the prechecks now. | 22:41 |
vhosakot_ | mimizone: cool | 22:41 |
mimizone | vhosakot_: I expect a full deployment may require additional work, and not only a kolla-deploy, but let's see | 22:42 |
*** pbourke-home has quit IRC | 22:42 | |
mimizone | I am confused, I commented the entire tasks in port_checks related to this point, and it is still ran | 22:44 |
mimizone | is there some cache in ansible?? | 22:46 |
mimizone | I must be modifying the wrong file. I am editing the file in ~/kolla/ansible/roles.... that I checked out of git. is there another one? I commented out the entire port_checks in the main.yml too, still running... | 22:52 |
mimizone | it gets it from /usr/local/share.... | 22:54 |
*** asalkeld has quit IRC | 23:07 | |
*** ayoung has quit IRC | 23:23 | |
*** vhosakot_ has quit IRC | 23:24 | |
*** mark-casey has quit IRC | 23:27 | |
*** asalkeld has joined #openstack-kolla | 23:29 | |
*** inc0 has joined #openstack-kolla | 23:35 | |
*** ravig has quit IRC | 23:39 | |
sbezverk | inc0 ping | 23:46 |
*** dwalsh has quit IRC | 23:50 | |
*** ravig has joined #openstack-kolla | 23:53 | |
*** inc0 has quit IRC | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!