| noonedeadpunk | damiandabrowski: can you check https://review.opendev.org/c/openstack/ansible-config_template/+/948915 (and the whole chain) to unblock 2.19 patches? | 09:20 |
|---|---|---|
| damiandabrowski | sure thing, will do that in a moment | 09:21 |
| gokhani | hello Folks, By default, the MariaDB backend configuration for HAProxy in OSA defines one node as the primary writer and the remaining nodes as backup. | 09:25 |
| gokhani | when the primary node goes down, the OpenStack services lose connectivity to the database because the backup nodes are not automatically promoted to active status in HAProxy. The only way we can restore the services is by manually removing the backup keyword from the config and reloading HAProxy. | 09:25 |
| gokhani | Is there a move towards using a more intelligent SQL proxy (like ProxySQL) to handle this, or is there a way to make the current HAProxy-based default more resilient? | 09:26 |
| gokhani | How does the we handle this in production? Should we be using option allbackups or a different balance strategy (like first) to ensure that one of the backup nodes takes over automatically? | 09:27 |
| gokhani | thanks noonedeadpunk, I haven't tried yet using signed cookies. | 09:28 |
| noonedeadpunk | wait, what? | 09:29 |
| noonedeadpunk | backups are becomming immeditely in use when current active is marked as down or for maintenance | 09:29 |
| gokhani | server mgmt01 10.x.x.1:3306 check port 9200 inter 12000 | 09:30 |
| gokhani | server mgmt02 10.x.x.2:3306 check port 9200 inter 12000 backup | 09:30 |
| gokhani | haproxy config is like that: | 09:30 |
| gokhani | may be I need to retest this. | 09:31 |
| noonedeadpunk | https://docs.haproxy.org/3.2/configuration.html#5.2-backup | 09:31 |
| noonedeadpunk | so failover must be pretty much atuomatical | 09:32 |
| gokhani | In my test it is failed I will test again. may be problem is specific to my environment. | 09:34 |
| noonedeadpunk | we never had to worry about this part tbh | 09:48 |
| noonedeadpunk | it works always quite nicely | 09:48 |
| noonedeadpunk | but when you check the status of backends - do you see that they are all up? | 09:49 |
| noonedeadpunk | as it could be smth related to allowed subnets to accept conection from | 09:49 |
| noonedeadpunk | ie - galera_monitoring_allowed_source | 09:51 |
| noonedeadpunk | as if haproxy is coming from unexpected address - it might be the issue, as then it maybe be not allowed to talk to mariadb | 09:51 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/galera_all.yml#L33-L39 | 09:52 |
| noonedeadpunk | and it can be if keepalived has /24 VIP instead of /32 for isntance | 09:52 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Add example of installation for user defined extra packages https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/972865 | 09:56 |
| gokhani | noonedeadpunk: thanks for detailed explanation. before test I haven't checked backend status. I have seen this issue only one of my environment. ı thought that this issue can happen in all environment but this is not true. I have tested this in different env it is working as you said. I am trying to find my env specific problem. | 10:07 |
| opendevreview | Merged openstack/openstack-ansible-lxc_container_create master: Use full service name in task name https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/972787 | 10:37 |
| noonedeadpunk | hm, our gates look completely broken on horion setup right now | 11:22 |
| noonedeadpunk | ok, so that's reproducible failure and related to setuptools somehow... | 12:17 |
| noonedeadpunk | https://paste.openstack.org/show/bIk1UwHEazI9bjgqsCSe/ | 12:18 |
| noonedeadpunk | Not really getting why/how new setuptools getting in... | 12:44 |
| noonedeadpunk | looks like its pulled by some pyproject.toml or smth | 12:44 |
| noonedeadpunk | as in the venv and requireents it\s setuptools==80.10.2 | 12:45 |
| noonedeadpunk | ok, found it... | 13:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Cleanup environment on wheel build failure https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/976967 | 13:26 |
| noonedeadpunk | so we have 2 ways to approach this | 13:30 |
| noonedeadpunk | 1. --no-build-isolation and then ensure that pbr is installed together with pip, wheels, setuptools | 13:30 |
| noonedeadpunk | 2. --build-constraint global-requirement-pins.txt | 13:31 |
| noonedeadpunk | for the first case concern is about non-OSA deployments, as we don't have list of build requirements configurable so far | 13:32 |
| noonedeadpunk | for the second case conern is that pbr is not in the global-requirement-pins.txt, and pbr is managed in upper-constraints | 13:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Do not rely on build isolation https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/976973 | 13:58 |
| noonedeadpunk | I'm kinda on the fence with this ^ | 13:59 |
| noonedeadpunk | like it feels there's an opportunity to simplify things a lot | 13:59 |
| noonedeadpunk | by actually going a different way and using --build-constraint instead | 13:59 |
| noonedeadpunk | though I'd assume using build isolation might take more time to execute | 14:00 |
| noonedeadpunk | and I can't come up with decent enough simplification to justify that | 14:02 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible master: Migrate k8s cluster from osa ops into main tree https://review.opendev.org/c/openstack/openstack-ansible/+/975277 | 14:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-python_venv_build master: Cleanup environment on wheel build failure https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/976967 | 15:42 |
| jrosser | i think that this has blown up our horizon builds https://github.com/openstack/requirements/commit/c43ee322efb5bfb5cafc633c80d12d9c009b30da | 21:59 |
| jrosser | XStatic-Angular-Schema-Form in particular | 22:00 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!