| gokhan_ | hello noonedeadpunk , I have nearly completed multi region deployment with osa. but I had to do multiple things manually. there is a one problem left. We can not get quotas from region 2. there are logs with os-quota-set with http 200 status but on client side we are getting unknown http 500 error. I don't know why quota set is not working on region2 | 11:38 |
|---|---|---|
| noonedeadpunk | gokhan_: and what;s in logs? as 500 shold return nice stack trace | 11:48 |
| gokhan_ | noonedeadpunk, sorry 504 error not 500 | 11:50 |
| gokhan_ | there is no trace | 11:50 |
| noonedeadpunk | so 504 that's no backend? | 11:50 |
| noonedeadpunk | timout | 11:50 |
| noonedeadpunk | huh | 11:50 |
| noonedeadpunk | I guess that quota is eing asked from the keystone | 11:51 |
| noonedeadpunk | so I wonder if it's reachable from the internal network? | 11:51 |
| noonedeadpunk | as otherwise you might need to set to use keystone over public | 11:51 |
| gokhan_ | it is weird that ı can see region2 admin quotas from horizon which is on region1 | 11:53 |
| noonedeadpunk | you share the keystone, right? | 11:54 |
| gokhan_ | yes keystone is reachable on internal network, is it possible to connect region1 keystone internal endpoint from region2 | 11:56 |
| gokhan_ | yes 2 regions are using same keystone | 11:56 |
| gokhan_ | noonedeadpunk, but I can list neutron and volume quotas, the problem is on compute only | 12:01 |
| jrosser | gokhan_: this is horizon giving you trouble? | 12:06 |
| gokhan_ | no, I am running command on region2 utility container | 12:06 |
| jrosser | ah ok, sure | 12:07 |
| jrosser | did you try —debug? | 12:07 |
| jrosser | reason I asked about horizon is that we see severe regression in horizon performance in the admin pages for compute with caracal | 12:08 |
| gokhan_ | yes I tried it. I am sharing debug logs | 12:09 |
| gokhan_ | jrosser, https://paste.openstack.org/show/b2xIlQpM1TVT8EIFgTfF/ | 12:13 |
| jrosser | and what actually is 10.0.3.98:8774? | 12:16 |
| jrosser | that is your internal vip for one of the regions? | 12:18 |
| gokhan_ | yes is it is internal vip for region 2 | 12:18 |
| jrosser | then next thing to do is see if for some reason haproxy returned that, or the backend service | 12:20 |
| jrosser | the answer will be in the log | 12:20 |
| gokhan_ | I checked nova api os compute logs but there is nothing. it is weird that sometimes I see os-quota-set requests with http status 200 | 12:22 |
| gokhan_ | nova client commands are working except quota | 12:25 |
| jrosser | the 504 has to come from somehere though | 12:40 |
| jrosser | you should tail all the logs whilst making the request | 12:40 |
| gokhan_ | jrosser, in nova api logs quota set request http status is 200 but on haproxy http status is 504 | 12:56 |
| gokhan_ | it is weird | 12:56 |
| jrosser | haproxy can hit a timeout and 504 before the backend is done, if the request takes a long time | 13:00 |
| jrosser | the various letters in the haproxy log tell you about this iirc | 13:00 |
| gokhan_ | jrosser, I tried with using nova api container, but it takes too much time | 13:05 |
| gokhan_ | and it worked | 13:05 |
| gokhan_ | I alse see warning logs about keystone.discover tries to connect region 1 keystone with internal vip but it is not accesible | 13:06 |
| gokhan_ | maybe problem is about keytone but I am not sure | 13:07 |
| gokhan_ | jrosser, Nov 01 16:01:58 hyperconverged01-nova-api-container-77e8b260 nova-api-wsgi[6004]: 2024-11-01 16:01:58.370 6004 WARNING keystoneauth.discover [None req-2a9fa0e5-1d08-48be-9d18-e73bb24c147a 2087a7fcaba54a4aa9bee259b2dd6766 758f99377e4a4c7f9bed7cdbe122739b - - default default] Failed to contact the endpoint at http://10.0.3.98:5000 for discovery. Fallback to using that endpoint as the base url. | 13:10 |
| gokhan_ | this is not expected,it is region1 internal vip | 13:11 |
| noonedeadpunk | that's why I asked if keystone is reachable | 13:27 |
| noonedeadpunk | and I guess expected depends on where keystone is placed | 13:27 |
| noonedeadpunk | as with shared keystone services will tend to connect to it | 13:27 |
| noonedeadpunk | or you stretched the database instead, and have a keystone backend in each region connecting to the same DB? | 13:28 |
| gokhan_ | noonedeadpunk, keystone is reachable on public vips but not on internal vips | 13:28 |
| noonedeadpunk | ok, you said opposite thing an hour ago | 13:28 |
| gokhan_ | noonedeadpunk, yes "I stretched the database instead, and have a keystone backend in each region connecting to the same DB", | 13:29 |
| noonedeadpunk | ok, then you need to have overrides per region group_vars, of there keystone endpoint is I assume | 13:30 |
| noonedeadpunk | or well | 13:30 |
| noonedeadpunk | not exactly... | 13:30 |
| noonedeadpunk | what I did - is that we have internal endpoints set as fqdn | 13:30 |
| noonedeadpunk | and we resolve them differently per region | 13:31 |
| noonedeadpunk | though keystone endpoint is different from others | 13:33 |
| noonedeadpunk | ie region1.cloud.com, region2.cloud.com, identity.cloud.com | 13:33 |
| gokhan_ | so you mean identity.cloud.com is used for inernal endpoint in both region | 13:35 |
| gokhan_ | interna | 13:35 |
| noonedeadpunk | well, it's internal.identity.cloud.com, but yeah | 13:35 |
| noonedeadpunk | and then I added override for hosts record to resolve it differently | 13:36 |
| noonedeadpunk | with openstack_host_custom_hosts_records | 13:37 |
| gokhan_ | noonedeadpunk, ok thanks I get it:) it seems if we connects internal vips for keystones , it can also work | 13:39 |
| noonedeadpunk | or you can override to use public IPs instead | 13:39 |
| noonedeadpunk | though you'd need to do that everywhere, as we don't have a variable to configre that right now everywhere | 13:40 |
| noonedeadpunk | though might be good idea to add one... | 13:40 |
| gokhan_ | noonedeadpunk, thanks I have to go now, may be ask new questions later :) | 13:41 |
| Baronvaile | What can I do if my ORG is decypting every SSL connection and it is causing the bootstrap-ansible script to fail? | 20:27 |
| jrosser | Baronvaile: it would be helpful if you can share an example of what happens | 20:28 |
| jrosser | for example, do you mean they man-in-the-middle all https connections and replace the cert with their own wildcard? | 20:29 |
| Baronvaile | jrosser: During the bootstrap-ansible.sh run I get an ERROR: The certificate of 'opendev.org' is not trusted. followed by one that it 'doesn't have a known issuer' | 20:30 |
| jrosser | and does anything like curl or wget work for https things? | 20:31 |
| Baronvaile | This happenes because our Palo Alto's firewalls are setup the decrypt and injust the appliance name into the certificate. | 20:31 |
| jrosser | like I say it would be most helpful if you can share the actual error at somewhere like paste.opendev.org | 20:32 |
| jrosser | otherwise it is very hard to give you specific advice | 20:32 |
| Baronvaile | It might. I had to put a GIT_SSL_NO_VERIFY=true on the deployment host git environment just to get past that part. | 20:32 |
| Baronvaile | jrosser: Thanks. I'll try to paste the text next week and bounce back here. | 20:34 |
| jrosser | if it were me, I would ignore everything to do with openstack-ansible until you can be sure that your host is setup correctly to deal with the firewall | 20:34 |
| jrosser | usually this involves getting the CA certificate that the firewall is using and installing it into the relevant trust stores on your host | 20:34 |
| jrosser | and make sure wget/curl and python are correctly using the system trust store | 20:35 |
| Baronvaile | Thanks. I'll look getting the certificate to load into the local store. | 20:38 |
| jrosser | and just a tip that python needs specific config to use that | 20:40 |
| jrosser | the python “requests” | 20:40 |
| jrosser | library uses a bundled set of CA certs and won’t see your extra cert unless you adjust things | 20:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!