| *** openstackgerrit has quit IRC | 00:03 | |
| *** irclogbot_2 has quit IRC | 00:03 | |
| *** hamalq has quit IRC | 00:03 | |
| *** jmlowe has quit IRC | 00:03 | |
| *** rm_work has quit IRC | 00:03 | |
| *** hamalq has joined #senlin | 00:05 | |
| *** jmlowe has joined #senlin | 00:05 | |
| *** rm_work has joined #senlin | 00:05 | |
| *** irclogbot_2 has joined #senlin | 00:06 | |
| *** irclogbot_2 has quit IRC | 00:10 | |
| *** spatel has joined #senlin | 00:12 | |
| *** irclogbot_0 has joined #senlin | 00:13 | |
| *** spatel has quit IRC | 00:17 | |
| *** hamalq has quit IRC | 00:40 | |
| *** spatel has joined #senlin | 02:17 | |
| *** sapd1 has joined #senlin | 02:28 | |
| spatel | dtruong: around ? | 02:42 |
|---|---|---|
| spatel | senlin default use all public endpoint to talk to other servers like keystone/nova/neutron etc.. is there a way i can tell senlin to use internal endpoint (non SSL) | 02:44 |
| spatel | I am trying these options but still no luck - https://review.opendev.org/c/openstack/senlin/+/749874 | 02:45 |
| spatel | i have set interface = internal but still senlin trying to talk to public endpoint and failing | 02:45 |
| *** sapd1 has quit IRC | 02:50 | |
| *** sapd1 has joined #senlin | 03:06 | |
| eandersson | spatel talking about senlin to another service? e.g. keystone | 05:25 |
| spatel | Yes | 05:25 |
| spatel | senlin obtain all endpoint from keystone and use public endpoint to talk to other services | 05:26 |
| spatel | i am trying to figure out how it can use internal endpoint | 05:26 |
| eandersson | In theory that patch should work | 05:27 |
| eandersson | but I think the problem is that it is getting it from the context | 05:28 |
| spatel | In my HAproxy i have external/internal IP and my senlin on internal IP ( and its failing to because can | 05:28 |
| spatel | can't talk to public interface) | 05:28 |
| spatel | hmmm context? | 05:28 |
| eandersson | It basically means that it passes on what it got from the user | 05:29 |
| spatel | hmm | 05:29 |
| eandersson | Do you have the endpoint set under | 05:30 |
| eandersson | [authentication] | 05:30 |
| eandersson | interface=internal | 05:30 |
| eandersson | ? | 05:30 |
| spatel | yes | 05:30 |
| spatel | but its still trying to talk to external IP | 05:30 |
| eandersson | Yea - I think what is happening is basically that the interface is passed on from the user request which will always be public | 05:31 |
| spatel | can we change that behavior? | 05:31 |
| spatel | senlin should get all endpoint from keystone itself right? | 05:32 |
| eandersson | Yea | 05:32 |
| spatel | why end-user influencing that behavior ? | 05:32 |
| eandersson | I think for Senlin the idea is that you are acting on behalf of the user. | 05:34 |
| eandersson | So it's like the user talking to nova etc | 05:34 |
| spatel | yes senlin acting like end-user | 05:34 |
| spatel | if somehow we can hardcode endpoint in senlin config then may be it will use internal endpoint | 05:36 |
| spatel | just like other openstack services, they have dedicated section for neutron/heat etc.. | 05:37 |
| eandersson | Yea - the problem is that the design is very different. | 05:37 |
| eandersson | We would probably need to look at something like Heat. | 05:37 |
| spatel | totally | 05:38 |
| eandersson | btw could you try to change this to internal? | 05:38 |
| eandersson | https://github.com/openstack/senlin/blob/6b60c7a87e971dd8a89662c796b9a0f337a36a7c/senlin/policies/base.py#L239 | 05:38 |
| eandersson | Just to see if that works? | 05:38 |
| spatel | i had interface=internal in senlin.conf but that didn't help | 05:39 |
| spatel | that is what you asking? | 05:40 |
| eandersson | I was thinking maybe setting it directly in code | 05:40 |
| eandersson | Actually do you know if keystone is using the internal? | 05:40 |
| eandersson | Or is even keystone using the public interface? | 05:41 |
| eandersson | I might be able to set up an env this weekend to look at it if dtruong does not have the answer | 05:42 |
| spatel | In tcpdump i have noticed it talked to internal URL to talk to keystone and it fetch all endpoint but then i noticed it started poking public side of keystone URL and just hanging there | 05:42 |
| eandersson | Interesting | 05:43 |
| spatel | I spent whole day and then in frustration i replaced my HAproxy with F5 load-balance | 05:44 |
| spatel | to solve this issue | 05:44 |
| spatel | HAproxy has two interface and its creating routing loop when senlin try to talk to external IP | 05:44 |
| spatel | I will setup lab tomorrow or in weekend to collect data to verify behavior | 05:45 |
| spatel | I will collect some pcap tomorrow to get better understanding | 05:51 |
| eandersson | We have two keystones set up for something similar | 05:53 |
| eandersson | Two different catalogs, one internal and one for the user | 05:53 |
| eandersson | Senlin would only talk to the internal one and the user would only talk to the public one | 05:54 |
| eandersson | But we have a very specialized setup and probably wouldn't work for most. | 05:56 |
| spatel | hmm interesting | 05:57 |
| spatel | but codebase should be same right? or you have very customized senlin? | 05:58 |
| eandersson | It's the same | 05:59 |
| eandersson | But you need two different catalogs | 05:59 |
| eandersson | We use keystone templatized catalogs to achieve this | 05:59 |
| spatel | very advance level :) | 06:00 |
| spatel | i have basic setup | 06:00 |
| eandersson | So if you talk to public keystone you get different endpoints for public than you would for the internal keystone | 06:00 |
| eandersson | Yea overkill | 06:00 |
| eandersson | Let me know how it goes and I'll try to take a look as well | 06:01 |
| spatel | This is what i have http://paste.openstack.org/show/801649/ | 06:01 |
| spatel | openstack-phx-1.foo.com is running on external IP of haproxy and 10.65.0.16 is internal side of haproxy | 06:02 |
| eandersson | The same for everything, nova etc right? | 06:02 |
| spatel | Yes same for all services | 06:02 |
| spatel | everything working fine except senlin :) | 06:03 |
| eandersson | spatel are you able to modify the senlin source code? | 06:06 |
| eandersson | https://opendev.org/openstack/senlin/src/branch/master/senlin/drivers/sdk.py#L117 | 06:06 |
| eandersson | If so could you try to add | 06:06 |
| eandersson | params['interface'] = 'internal' | 06:06 |
| eandersson | there | 06:07 |
| spatel | I need to setup lab.. currently i moved my senlin to F5 to fix this issue | 06:08 |
| spatel | tomorrow i will spin up new lab and verify your suggestion | 06:08 |
| spatel | that part looks interesting to test | 06:08 |
| eandersson | Sounds good | 06:09 |
| spatel | its 1:00AM here :) | 06:09 |
| spatel | sure in morning i will give it a shot and post result here | 06:09 |
| eandersson | 10PM here | 06:13 |
| spatel | Yes West cost :) | 06:16 |
| spatel | coast | 06:16 |
| spatel | eandersson: there | 06:37 |
| spatel | As soon as i tried params['interface'] = 'internal' it works! | 06:38 |
| spatel | look like you are right! we need that in param in sdk | 06:39 |
| spatel | oh! wait.. i just removed that and it still working.. | 06:42 |
| spatel | eandersson: i don't trust this lab.. tomorrow i will give it a try with fresh mind | 06:43 |
| *** spatel has quit IRC | 07:03 | |
| *** spatel has joined #senlin | 09:16 | |
| *** spatel has quit IRC | 09:21 | |
| *** sapd1 has quit IRC | 10:04 | |
| *** sapd1 has joined #senlin | 13:26 | |
| *** spatel has joined #senlin | 13:48 | |
| *** jrosser has joined #senlin | 14:05 | |
| *** jmlowe has quit IRC | 15:27 | |
| *** sapd1 has quit IRC | 17:01 | |
| *** spatel has quit IRC | 19:03 | |
| *** spatel has joined #senlin | 22:53 | |
| *** spatel has quit IRC | 22:53 | |
| eandersson | Let me know how it goes | 22:57 |
| *** jrosser has quit IRC | 23:01 | |
| *** jrosser has joined #senlin | 23:03 | |
| *** eandersson has quit IRC | 23:20 | |
| *** eandersson has joined #senlin | 23:21 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!