| *** odyssey4me is now known as Guest5655 | 00:47 | |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Enable TLS for live migrations https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/815224 | 00:49 |
|---|---|---|
| *** gouthamr_ is now known as gouthamr | 06:19 | |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: kata: disable installation on debian bullseye https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/817714 | 08:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Fix repository URL in galaxy.yml https://review.opendev.org/c/openstack/ansible-config_template/+/817720 | 09:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Add ability to create templated services https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/816531 | 09:24 |
| MrClayPole_ | Hi, I'm running OSA 20.2.6. When performing maintenance on one of my three infrastructure nodes. I get very slow access and HTTP 504 errors in Horizon when one of the nodes is offline. I've traced this to when 1 of the 3 Memcache containers is offline. Is there a way to enable Horizon to better handle one of its Memcache servers being offline? | 09:56 |
| noonedeadpunk | MrClayPole_: actually it's more about interaction of keystone with memcached | 09:57 |
| noonedeadpunk | But yes, you can check https://docs.openstack.org/openstack-ansible-memcached_server/latest/configure-ha.html#configuring-memcached-through-haproxy | 09:58 |
| noonedeadpunk | if you don't want to balance memcached through haproxy, you also can improve things dramatically by ensuring that keystone uses proper driver | 09:59 |
| MrClayPole_ | Which method would you recommend? | 09:59 |
| noonedeadpunk | you should check that [cache] backend in keystone.conf is `oslo_cache.memcache_pool` and not dogpile | 10:00 |
| noonedeadpunk | well actually I believe that in train backend should be already set to memcache_pool | 10:01 |
| MrClayPole_ | Looks like I'm still using dogpile | 10:02 |
| MrClayPole_ | [cache] | 10:02 |
| MrClayPole_ | backend = dogpile.cache.memcached | 10:02 |
| noonedeadpunk | so switching backend is low hanging fruit which will improve things dramatically | 10:03 |
| MrClayPole_ | Ah sorry my bad. This enviroment is still on Rocky. I'm been work so much of our test enviroment which is Train. | 10:03 |
| noonedeadpunk | as you can jsut set override fast enough. And you should not see 504 anymore | 10:03 |
| noonedeadpunk | However you will still experience some latency when controller will went down | 10:04 |
| MrClayPole_ | OK, I'll test both methods but I'm currently liking the haproxy fix. | 10:04 |
| noonedeadpunk | haproxy balancing will eliminate even this latency. but it requires more effort | 10:04 |
| noonedeadpunk | as eventually you'd better switch all services to be served through same endpoint | 10:05 |
| *** chandankumar is now known as raukadah | 10:05 | |
| MrClayPole_ | Agreed, it feels a like a better solution and less visible to the customer when we are patching/rebooting our infrastructure nodes | 10:06 |
| noonedeadpunk | also, haproxy does l3 balancing, which is not perfect. For instance, mcrouter can balance on L7, but we don't have role for it | 10:06 |
| noonedeadpunk | you can also install it on haproxy nodes and it would be even better | 10:07 |
| MrClayPole_ | .. but still I can put the backend in MAINT mode before I reboots it should be pretty seemless? | 10:07 |
| noonedeadpunk | the problem is a bit different with haproxy | 10:07 |
| noonedeadpunk | so memcached does not support clustering at all - by design | 10:08 |
| MrClayPole_ | So when we failover then services get an empty memcache? | 10:08 |
| noonedeadpunk | so haproxy uses source ip balancing, which means that it makes an assumption of destination where cache is placed | 10:08 |
| noonedeadpunk | yeah, some of them | 10:09 |
| noonedeadpunk | that's why mcrouter is more proper solution | 10:09 |
| noonedeadpunk | because it's aware about where exacly cache is stored | 10:09 |
| noonedeadpunk | and can manage failovers more efficiently | 10:09 |
| noonedeadpunk | But actually you will get empty cache in case of any failover | 10:10 |
| MrClayPole_ | OK, but how does L7 load balancing help as based on my limited understanding the requested item is either in the memcache or its not and it gets requested from source? | 10:10 |
| noonedeadpunk | it more matter if you might have corner cases without failovers or not... | 10:10 |
| noonedeadpunk | in case of failover it doesn't matter much | 10:11 |
| noonedeadpunk | but, during runtime, haproxy makes bunch of assumptions, that this ip should have stored on this backend. But in fact it has small idea where it is in reality | 10:11 |
| MrClayPole_ | So if I'm following if you had an active/active service using the memcache then haproxy could route then to different memcache instances? | 10:14 |
| noonedeadpunk | yes and it will | 10:18 |
| noonedeadpunk | likely | 10:18 |
| MrClayPole_ | Can we make haproxy route all requests to one memcache and other failover when that one is down? | 10:20 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Fix inconsistency in haproxy_frontend_raw naming https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/817727 | 10:30 |
| noonedeadpunk | MrClayPole_: sure you can | 10:40 |
| noonedeadpunk | it's all about how you describe haproxy_extra_services | 10:40 |
| noonedeadpunk | so you can define one backend in haproxy_backend_nodes and others in haproxy_backup_nodes I believe | 10:42 |
| noonedeadpunk | but well, I'm not sure this was available on rocky tbh | 10:43 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-haproxy_server master: Remove httplog directive from backends as ignored https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/817731 | 10:48 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/victoria: Add Virtual GPU Config to nova.conf template https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/817537 | 10:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Allow to provide mdev addresses as list https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/817738 | 11:02 |
| mgariepy | MrClayPole_, if it worth it with Rocky when i did upgrade the control plane i did remove the memcached that was down from the keystone.conf file which made the responses fast. | 13:09 |
| mgariepy | what i did was: ```ansible keystone_all -m lineinfile -a "path=/etc/keystone/keystone.conf regexp='^memcache_servers' line='memcache_servers = 172.16.20.220:11211,172.16.20.240:11211'``` | 13:11 |
| mgariepy | and restart the service. | 13:11 |
| noonedeadpunk | well, I'd just set backend driver to oslo_cache.memcache_pool tbh - it partially solves issue and makes it at least not that bad | 13:11 |
| MrClayPole_ | Thanks mgariepy worth considering if I know one of them is going to be down for a while. Just testing noonedeadpunk's suggestions at the moment | 13:12 |
| noonedeadpunk | because oslo comparing to dogpile somehow "marks" memcached backend as down and do not retry failed backend each time | 13:12 |
| noonedeadpunk | not perfect but super easy to use | 13:13 |
| MrClayPole_ | So are no 504 timeouts with the oslo_cache.memcache_pool but as you said there is still the lag | 13:13 |
| noonedeadpunk | and actually if you're loocking for upgrade - I'd plan to switch to haproxy/mcrouter during upgrade | 13:14 |
| MrClayPole_ | The haproxy link you sent looks straight forward enough. Just deciding if we implement mcrouter. | 13:20 |
| noonedeadpunk | well, mcrouter is a facebook child, so while it's great, it's also a bit opionated and has issues with operating systems support - they make it work for their specific usecase only afaik | 13:22 |
| noonedeadpunk | like it was with flashcache kernel module for instance... | 13:23 |
| noonedeadpunk | and thus can drop it was well. But we used it in some deployments pretty successfully | 13:23 |
| MrClayPole_ | I'll see how far I get with testing just haproxy first as I'm reluctant to install mcrouter if there isn't an osa role. | 13:37 |
| noonedeadpunk | yeah and we don't plan to ad it just in case | 13:40 |
| noonedeadpunk | *add | 13:40 |
| opendevreview | Merged openstack/openstack-ansible master: Bump ansible and collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/817375 | 14:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 14:35 |
| spatel | jamesdenton_alt morning! | 15:26 |
| *** frenzy_friday is now known as frenzyfriday|PTO | 15:59 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 17:25 |
| prometheanfire | looking for multi-region deployment docs for OSA, any pointers? | 18:21 |
| *** Guest5508 is now known as melwitt | 19:26 | |
| *** melwitt is now known as Guest5716 | 19:27 | |
| *** Guest5716 is now known as melwitt | 19:32 | |
| *** melwitt is now known as jgwentworth | 19:34 | |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Fix apache ProxyPass arguments https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/811402 | 21:06 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi stable/wallaby: Fix gnocchi multi-node when storage_driver is file https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/817428 | 22:34 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi stable/victoria: Fix gnocchi multi-node when storage_driver is file https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/817429 | 22:41 |
| opendevreview | Merged openstack/openstack-ansible master: Add playbook to generate any user defined certificates https://review.opendev.org/c/openstack/openstack-ansible/+/816522 | 22:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!