| cnilesh | hi team, i am looking for the help on the bz - https://bugs.launchpad.net/openstack-ansible/+bug/2071831 | 05:23 |
|---|---|---|
| gokhani | good morning folks, can we upgrade from antelope to caracal? Is there any known issues? | 06:50 |
| noonedeadpunk | for https://review.opendev.org/c/openstack/openstack-ansible/+/923368 I guess we'd need to apply: tags: -always or smth like that | 07:27 |
| noonedeadpunk | gokhani: there are *some* | 07:27 |
| noonedeadpunk | I'd wait at least for the next ugfix release | 07:27 |
| noonedeadpunk | *bugfix | 07:27 |
| gokhani | thanks noonedeadpunk, I have started to upgrade on a test environment, ı will share my findings. | 07:31 |
| noonedeadpunk | cnilesh: hey! let me check | 07:34 |
| cnilesh | noonedeadpunk++ thank you | 07:35 |
| noonedeadpunk | so, what's the cert are you using to cover public endpoint? Is it just smth that was issued by our PKI role? Or you've provided self-signed cert? | 07:36 |
| noonedeadpunk | as eventually if it's issued with PKI role, I'd expect cert to be trusted | 07:37 |
| cnilesh | noonedeadpunk, so as I wanted to setup SSL.TLS for public endpoints I had created self-signed certs for haproxy, and added override parameter at the time of deployment, During magnum service enablement I had not added or given any SSL options, | 07:38 |
| cnilesh | so like galera[encrption by pki internally] I am expecting the same for magnum as well, | 07:38 |
| noonedeadpunk | well, it's very different case here | 07:44 |
| noonedeadpunk | while all services reach keystone through internal endpoint, magnum has a usecase where this is performed through the public one | 07:45 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_magnum/src/branch/master/templates/magnum.conf.j2#L107 | 07:45 |
| noonedeadpunk | So magnum (and, eventualy, booted master) should trust the certificate | 07:46 |
| noonedeadpunk | and there's no option in magnum to disable tls verification | 07:48 |
| noonedeadpunk | so in case if it would be just self-generated by role certificate for public haproxy endpoint - RootCA for it is made trusted to all services. Which I guess not what has happened with the one you supplied manually | 07:49 |
| noonedeadpunk | as by default it will just install the certificate on haproxy, but to make rootCA trusted - you'd need to have another variable defined... | 07:50 |
| cnilesh | ohhh this is horrible, any workaround for it | 07:52 |
| noonedeadpunk | so ideally - you'd need an FQDN for endpoints | 07:52 |
| noonedeadpunk | and a valid certificate or at least let's encrypt | 07:52 |
| noonedeadpunk | But eventually, if you'd want to have connection encrypted between HAproxy and backends - you'd still have to use self-signed certs by internal root | 07:54 |
| noonedeadpunk | and this internal rootCA we're managing with PKI role is used anyway for configuration of libvirt live migrations | 07:54 |
| cnilesh | cool, thank you for the quick help. | 07:56 |
| cnilesh | and I see same for barbican as well. | 07:56 |
| noonedeadpunk | there should be anything like that with barbican | 07:57 |
| noonedeadpunk | *should not | 07:57 |
| cnilesh | same SSL errors | 07:57 |
| noonedeadpunk | about trusts? | 07:58 |
| noonedeadpunk | or connection to keystone insecure? | 07:58 |
| noonedeadpunk | from what I see - it should be using internal endpoints | 07:59 |
| cnilesh | yes, but still showing the ssl errors on public endpoints | 07:59 |
| cnilesh | however in the config its conection to internal endponits | 07:59 |
| cnilesh | let me quickly redeploy the env without ssl/tls, will manage encryption on f5LB at hardware layer | 08:00 |
| cnilesh | will see the functioning then | 08:00 |
| cnilesh | give me some time | 08:00 |
| noonedeadpunk | you can just rely on internal pki certs.... | 08:00 |
| cnilesh | yup | 08:00 |
| cnilesh | good idea | 08:00 |
| cnilesh | thank you. | 08:00 |
| cnilesh | noonedeadpunk++ thanks | 08:01 |
| noonedeadpunk | like no reason to issue external self-signed ones, except to test that their distribution actually works | 08:01 |
| noonedeadpunk | also, if you want to cover *all* connections with TLS - you can define these: https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables_tls.yml.j2#L17-L25 | 08:04 |
| noonedeadpunk | you can drop openstack_service_backend_ssl and openstack_service_accept_both_protocols to cover admin/internal with tls in addition to public ones. | 08:05 |
| cnilesh | ths sounds good | 08:08 |
| cnilesh | cool | 08:08 |
| cnilesh | noonedeadpunk, and I added somthing like this in the overwrite variable haproxy_ssl_all_vips: False. | 08:32 |
| noonedeadpunk | it's false by default fwiw | 08:35 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/stable/2023.1/inventory/group_vars/all/haproxy.yml#L17 | 08:35 |
| cnilesh | noonedeadpunk++ thanks | 08:37 |
| *** gaudenz_ is now known as gaudenz | 08:54 | |
| jrosser | cnilesh: noonedeadpunk: https://bugs.launchpad.net/magnum/+bug/2060194 | 09:23 |
| jrosser | fwiw we have a deployment where the internal network is not routed to the public endpoint and that reveals all sorts of mess like this | 09:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 09:29 |
| cnilesh | jrosser++ thank you so much for the info. | 09:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Split large k8s_install playbook into more specific smaller playbooks https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923172 | 09:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Pass kubeconfig path directly to sonobuoy role https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923211 | 09:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add variables and hook for high-availability k8s control plane test https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923173 | 09:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 09:50 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Split large k8s_install playbook into more specific smaller playbooks https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923172 | 10:28 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Pass kubeconfig path directly to sonobuoy role https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923211 | 10:41 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add variables and hook for high-availability k8s control plane test https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923173 | 10:43 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Split large k8s_install playbook into more specific smaller playbooks https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923172 | 12:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Pass kubeconfig path directly to sonobuoy role https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923211 | 12:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 12:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add variables and hook for high-availability k8s control plane test https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923173 | 12:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 12:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Pass kubeconfig path directly to sonobuoy role https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923211 | 14:54 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add variables and hook for high-availability k8s control plane test https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923173 | 14:54 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Add support for deploying mcapi control plane k8s on rocky linux https://review.opendev.org/c/openstack/openstack-ansible-ops/+/923447 | 14:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use openstack.osa.install_defaults role instead of vars_files https://review.opendev.org/c/openstack/openstack-ansible/+/923358 | 16:23 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove remove_container_journal common task file https://review.opendev.org/c/openstack/openstack-ansible/+/923366 | 16:24 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove dynamic-grouping common task file https://review.opendev.org/c/openstack/openstack-ansible/+/923367 | 16:24 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use haproxy_endpoint_manage role from osa collection rather than common-tasks https://review.opendev.org/c/openstack/openstack-ansible/+/923368 | 16:24 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add haproxy_endpoint_manage role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/923363 | 16:32 |
| cnilesh | hi anyone familier with this issue | 17:21 |
| cnilesh | Referer checking failed - Referer is insecure while host is secure. Cookies may be turned off. Make sure cookies are enabled and try again. | 17:21 |
| jrosser | cnilesh: is that some error from your browser? | 17:26 |
| cnilesh | jrosser, yes sir | 17:26 |
| cnilesh | http://172.29.240.199/auth/login/?csrf_failure=Referer%20checking%20failed%20-%20Referer%20is%20insecure%20while%20host%20is%20secure. | 17:26 |
| jrosser | any you've switched from ssl to non ssl? | 17:27 |
| jrosser | *and | 17:27 |
| jrosser | and you've switched from ssl to non ssl for the backends? | 17:27 |
| cnilesh | https://pasteboard.co/oRR9sQuOqjft.png | 17:27 |
| cnilesh | yes, thats true, switched from ssl to nonssl | 17:28 |
| jrosser | this is becasue for some reason HTTP_REFERER is not https | 17:31 |
| cnilesh | jrosser, humnnnnn, any workaround, I can not redeploy nw,Its already 2/3 times | 17:32 |
| jrosser | well you need to get it all completely correct | 17:32 |
| jrosser | if i understand properly you have done a "no ssl" deployment then put an f5 in front which does do ssl? | 17:33 |
| cnilesh | true, but I am yet to put f5 LB | 17:33 |
| jrosser | without going through everything i cannot say what is happening | 17:35 |
| jrosser | but that error comes from django, which is the horizon | 17:36 |
| jrosser | so there is some error there | 17:36 |
| cnilesh | ok, let me see | 17:37 |
| jrosser | i always recommend protyping your setup in an all-in-one, as you can build one in just a couple of hours | 17:37 |
| jrosser | in this case i expect there is an error in the config for horizon, or the apache config, or the django setttings | 17:38 |
| cnilesh | we have 1 testbad AIO where everything is working without issue non-ssl | 17:40 |
| cnilesh | but in prod deployment is erorring out | 17:40 |
| jrosser | then you have some difference in config between the two | 17:42 |
| jrosser | or in switching prod from ssl to non ssl, have not done this completely | 17:43 |
| cnilesh | I did it, reformated the ndoes | 17:43 |
| jrosser | and also the error is not from your browser | 17:47 |
| cnilesh | yes | 17:47 |
| jrosser | it is from horizon/django rather than your browser client | 17:48 |
| jrosser | becasue the error is rendered in the page, not the browser console | 17:48 |
| cnilesh | yeah | 17:49 |
| jrosser | so the place to look is in the horizon / apache logs | 17:51 |
| jrosser | and the config for those | 17:51 |
| jrosser | but i am concerned that the behaviour is going to change when you put the f5 on the front | 17:52 |
| cnilesh | Jul 04 17:52:41 ctrl002.ct.lan apache2[5246]: [wsgi:error] [pid 5246:tid 140210059286080] [remote 172.29.231.201:53980] Forbidden (Referer checking failed - Referer is insecure while host is secure.): /auth/login/ | 17:53 |
| jrosser | it sounds like this is because you are attempting to use http on somthing that will eventually be https | 17:58 |
| jrosser | and there is a mismatch between the referrer (original http request) and what the horizon server thinks it is talking to (something https) | 17:59 |
| cnilesh | this should be an haproxy issue, | 17:59 |
| cnilesh | I am just comapiring the data and I do see the difference in the haproxy | 17:59 |
| cnilesh | from working env [non-ssl] I am seeing this line everywhere, which is true - ssl crt /etc/haproxy/ssl/haproxy_controller1-192.168.124.99.pem | 18:01 |
| cnilesh | and not in new environemnt | 18:01 |
| jrosser | cnilesh: what do you set haproxy_ssl to? | 18:14 |
| cnilesh | false | 18:14 |
| jrosser | then it is clear why you have no certificates generated for haproxy https://github.com/openstack/openstack-ansible-haproxy_server/blob/3eccf224d66df002b091c439ba8982979a40859c/tasks/main.yml#L63C7-L63C18 | 18:15 |
| jrosser | sorry it is late here, i have to go | 18:20 |
| cnilesh | jrosser, no worries, thank you sir | 18:21 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Combine vars files for all debian derivatives https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/923166 | 20:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
