| semantic | Hi, guys! | 04:16 |
|---|---|---|
| semantic | It seems to me bug was introduced in https://opendev.org/openstack/openstack-ansible/commit/2c06a65a75f501c30376f7369f4b7eccd3f0523a | 04:16 |
| semantic | horizon_webroot: "{{ (groups['skyline_all'] | default([])) | ternary('/horizon', '/') }}" renders to http-check send hdr User-Agent "osa-haproxy-healthcheck" meth HEAD uri //auth/login/ while it should be ...HEAD uri /auth/login/. | 04:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_horizon master: DNM - test https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/926254 | 08:09 |
| jrosser | ^ lets test the horizon role haproxy healthcheck | 08:09 |
| jrosser | oh actually horizon is deployed by default https://zuul.opendev.org/t/openstack/build/499ad0b14e3b43878446c7e91b5f71da/log/logs/etc/host/haproxy/conf.d/horizon.txt#15 | 08:11 |
| semantic | Sorry, not sure if I understand what you mean. Maybe tests should be fixed also, as while it has //auth/login/ instead of /auth/login/ horizon is not available, and horizon-dashboard.service constantly logs that "//auth/login is not found". Fixing the haproxy config turns everything back to normal. | 08:44 |
| jrosser | semantic: i wanted to see it in the logs of a job as i was not sure what actually the problem is | 08:47 |
| jrosser | but its helpful to know that its causing log noise rather than breaking something | 08:48 |
| semantic | Well, you can't access horizon dashboard through haproxy, so it actually breaks stuff. :) | 08:49 |
| jrosser | hmm | 08:50 |
| jrosser | when i look on my haproxy host here, i see that there is the //auth/login in the config but we can access horizon just fine | 08:50 |
| jrosser | so i expect that there is something additional / more subtle at play as well | 08:50 |
| jrosser | it's always helpful to know which release of openstack-ansible we are talking about, and on what OS | 08:51 |
| semantic | I found it looking at 29.0.1 on Debian Bookworm. I then compared to another setup with working Horizon I have on 28.0.1 (also Bookworm), and found that difference. | 08:55 |
| semantic | Actually the reason I look into newer version, is that I still can't resolve failover problems completely. I believe I've mentioned them some time ago. After all introduced stuff like quorum queues, etc, I finally can confirm openstack services can live after shutting down any single of infra hosts with RabbitMQ. Though there's still a problem if the shutdown is unexpected (i.e. poweroff). Most services like nova-compute, neutron-openvswitch- | 09:03 |
| semantic | agent, nova-conductor, etc stuck in a loop like this: | 09:03 |
| semantic | nova-conductor[4572]: 2024-08-14 06:48:01.102 4572 ERROR oslo.messaging._drivers.impl_rabbit [-] [23741f6a-4fc8-4e24-962e-5d918553cb7e] AMQP server on 10.33.177.212:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>. Restarting all the services brings everything back to normal. | 09:03 |
| semantic | It's very easily reproducible for me, just force turn off the first infra host after the fresh setup. | 09:03 |
| semantic | Struggling to find a reason for this within oslo.messaging...looks very similar to the bug: https://bugs.launchpad.net/oslo.messaging/+bug/2019978, but it was resolved some time ago... | 09:03 |
| jrosser | semantic: can you look at what version of haproxy you have installed on 29.0.1? | 09:12 |
| semantic | HAProxy version 2.6.12-1+deb12u1 2023/12/16 - https://haproxy.org/ | 09:14 |
| semantic | Status: long-term supported branch - will stop receiving fixes around Q2 2027. | 09:14 |
| semantic | Known bugs: http://www.haproxy.org/bugs/bugs-2.6.12.html | 09:14 |
| semantic | Running on: Linux 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 | 09:14 |
| jrosser | thanks | 09:19 |
| jrosser | it's really interesting that it works in our tests | 09:19 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove extra slash character from horizon haproxy healthcheck url. https://review.opendev.org/c/openstack/openstack-ansible/+/926275 | 10:56 |
| jrosser | semantic: ^ would you be able to test this? | 10:57 |
| semantic | will try, btw the difference between our setups may be that we use horizon_use_uwsgi: true | 11:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Remove extra slash character from horizon haproxy healthcheck url. https://review.opendev.org/c/openstack/openstack-ansible/+/926275 | 12:58 |
| grauzikas | finally :) by guess method made it to work. now i have question in ovn scenario should i provide additional opinions and what supported in neutron_plugin_base. example: https://paste.openstack.org/show/bvg8SSkAXF8nPTaroDa0/ | 13:02 |
| grauzikas | ok found all info. TY :) | 13:11 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Update magnum-cluster-api and ansible-collection-kubernetes versions https://review.opendev.org/c/openstack/openstack-ansible-ops/+/926347 | 17:24 |
| platta | I’ve got two things left before I’m ready to just use my lab. 1) I want to build an Ansible playbook that will create boilerplate resources for me, default public network, upload a Debian image to Glance, etc. 2) I’d still like the Horizon dashboard run without SSL so I can deal with certificates and SSL termination on my one internet-facing | 19:51 |
| platta | host that does it for all the other services I host. | 19:51 |
| platta | That said - do you have any suggestions on how I can give back to the community? I might write a blog post describing what I did or share a repo of the playbooks I created to help me bootstrap my host. If there are other things I could do that would be helpful, please let me know. | 19:51 |
| jrosser | platta: there is a role already in OSA called “openstack_resources” which should able to create your images/networks etc | 19:53 |
| platta | Oh, I will definitely check that out then. I was planning to look at the openstack.cloud collection, but this is likely better. | 19:55 |
| jrosser | well it will use that collection under the hood | 20:02 |
| jrosser | but give you a structured way to write the data describing what you want | 20:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!