Wednesday, 2025-06-04

*** bbezak is now known as Guest1726900:00
opendevreviewVerification of a change to openstack/kayobe stable/2025.1 failed: Make custom Gather facts run with default python for kayobe-target-venv  https://review.opendev.org/c/openstack/kayobe/+/95172002:46
opendevreviewDai Dang Van proposed openstack/kolla-ansible master: Enable unified alerting for Grafana  https://review.opendev.org/c/openstack/kolla-ansible/+/94911403:24
opendevreviewDai Dang Van proposed openstack/kolla-ansible master: Add grafana native metrics target  https://review.opendev.org/c/openstack/kolla-ansible/+/94902403:25
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10  https://review.opendev.org/c/openstack/kolla/+/95039205:45
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26  https://review.opendev.org/c/openstack/kolla-ansible/+/93724205:54
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26  https://review.opendev.org/c/openstack/kolla-ansible/+/93724205:54
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ansible-lint: Make roles passable with shared profile  https://review.opendev.org/c/openstack/kolla-ansible/+/92015905:54
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26  https://review.opendev.org/c/openstack/kolla-ansible/+/93724205:55
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26  https://review.opendev.org/c/openstack/kolla-ansible/+/93724205:55
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ironic: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/94601405:57
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/94601006:02
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add ceilometer scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/94427606:03
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add ceilometer scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/94427606:03
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/94010106:04
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/94010106:04
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/94010106:05
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add ceilometer scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/94427606:06
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: gnocchi: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/94010106:08
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: cli: Allow for missing globals.yml when globals.d exists  https://review.opendev.org/c/openstack/kolla-ansible/+/94038806:12
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simply globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:12
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simply globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simply globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:13
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:14
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:14
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:16
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Simplify globals-default by moving to globals.d  https://review.opendev.org/c/openstack/kolla-ansible/+/94032006:16
mnasiadkaoh boy06:18
kevko:D06:25
kevkomnasiadka: Testing backports script or what :D ? 06:26
mnasiadkanaah, tried to be lazy and click my way in Gerrit06:27
opendevreviewPiotr Milewski proposed openstack/kolla-ansible master: masakari: Add support for using uWSGI  https://review.opendev.org/c/openstack/kolla-ansible/+/95111506:34
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26  https://review.opendev.org/c/openstack/kolla-ansible/+/93724206:38
kevko:D06:38
opendevreviewPierre Riteau proposed openstack/kayobe master: CI: Use latest cirros image  https://review.opendev.org/c/openstack/kayobe/+/93168507:28
opendevreviewMichal Arbet proposed openstack/kolla master: Fix sanity check in proxysql  https://review.opendev.org/c/openstack/kolla/+/95174107:43
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard  https://review.opendev.org/c/openstack/kolla-ansible/+/95159907:47
osmanlicilegimorning07:55
opendevreviewDai Dang Van proposed openstack/kolla-ansible master: Add grafana native metrics target  https://review.opendev.org/c/openstack/kolla-ansible/+/94902408:17
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard  https://review.opendev.org/c/openstack/kolla-ansible/+/95159908:17
kevkofrickler:  I've addressed your comments ...please can u check if you are OK now ? 08:17
opendevreviewMichal Arbet proposed openstack/kolla master: Fix sanity check in proxysql  https://review.opendev.org/c/openstack/kolla/+/95174108:21
opendevreviewMichal Arbet proposed openstack/kolla master: Fix ProxySQL rule sanity check to support username-based routing  https://review.opendev.org/c/openstack/kolla/+/95174108:26
fricklerkevko: thx, I'll have another look later, I'd have given you a +2 this morning already if it hadn't been about your comment that you wanted to do further improvements08:28
opendevreviewDai Dang Van proposed openstack/kolla-ansible master: Enable unified alerting for Grafana  https://review.opendev.org/c/openstack/kolla-ansible/+/94911408:31
kevkofrickler: thank you, well, another improvent was add another rule for username based routing (not only default_hostgroup for mysql_users, but also based on user, schemaname in mysql_query_rules ) 08:44
kevkofrickler: thank you, well, another improvent was add another rule for username based routing (not only default_hostgroup for mysql_users, but also based on user, schemaname in mysql_query_rules )  ...paste here https://paste.openstack.org/show/bbwhMsoetPpH92zCWfl7/08:50
fricklerkevko: yeah, thx, this sure does look reasonable, one day I need to dig into how proxysql really works, for now I'll just trust your testing08:59
mnasiadkafrickler: wondering - we stopped using kolla-ansible-core Gerrit group - but it still has members - should we clean it up?08:59
opendevreviewVerification of a change to openstack/kolla-ansible master failed: Replace outdated core team list by Gerrit links  https://review.opendev.org/c/openstack/kolla-ansible/+/95101509:00
fricklermnasiadka: good point, iirc there was also a way to retire these by either renaming or hiding them in gerrit, need to check with fungi for details09:04
mnasiadkafrickler: thanks :)09:23
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: Introduce docker_image_url variable  https://review.opendev.org/c/openstack/kolla-ansible/+/95174709:32
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ansible-lint: Fix yaml[octal-values]  https://review.opendev.org/c/openstack/kolla-ansible/+/95174809:34
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Bump ansible-lint to <26  https://review.opendev.org/c/openstack/kolla-ansible/+/93724209:34
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: ansible-lint: Fix yaml[octal-values]  https://review.opendev.org/c/openstack/kolla-ansible/+/95174809:35
*** mikal4 is now known as mikal09:42
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Add centos10 job  https://review.opendev.org/c/openstack/kolla-ansible/+/95175109:48
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10  https://review.opendev.org/c/openstack/kolla/+/95039209:49
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard  https://review.opendev.org/c/openstack/kolla-ansible/+/95159909:51
kevkomnasiadka frickler:  As I'm digging through the Kolla-Ansible code and working on stabilizing the upgrade process for RabbitMQ and MariaDB, and also messing around with ProxySQL while reading up a lot on these topics... I think we have a misconfigured oslo.db connection pooling.  believe this because, historically, there was an issue with skipping09:59
kevkothe VIP, which left pre-established connection pools dead... and as a result, pooling was effectively disabled in Kolla and a recycle of 10 seconds was configured. But this is fundamentally wrong — OpenStack makes a ton of small queries like SELECT * FROM something LIMIT 1; and similar... In a large stack, every single connection is quite09:59
kevkoexpensive, especially since each one initiates a new TCP connection unnecessarily.09:59
mnasiadkait was some oslo.db bug most probably, check the historical commits? they should have some information09:59
kevkomnasiadka frickler: problem with connections I solved historically with https://review.opendev.org/c/openstack/kolla-ansible/+/777772 << 10:00
kevkomnasiadka frickler: https://bugs.launchpad.net/kolla-ansible/+bug/1917068 10:00
kevkomnasiadka frickler: you thought it was ...but actually It was just workarounded by  https://review.opendev.org/q/Ib6a62d4428db9b95569314084090472870417f3d 10:01
mnasiadkawell, in mariadb directly via haproxy probably it makes sense10:02
mnasiadkawondering if we should not think about getting rid of not using proxysql10:02
kevkomnasiadka: First of all, I think this was just a small band-aid on a bigger wound — because what used to be a major freeze turned into a smaller one that now takes up to 10 seconds to resolve... And secondly, we had to set, for example, max_overflow, which creates connection pools beyond the limits configured in oslo.db...10:02
kevkomnasiadka: proxysql IS already default :) 10:03
mnasiadkais default, but we still sort of support the old mode10:03
kevkomnasiadka: but this is not about proxysql ...it was about VIP switch ...10:03
mnasiadkaif somebody is persistent10:03
mnasiadkaalthough we don't test it10:03
kevkomnasiadka: as I said ... I've seen this while haproxy was used 10:04
kevkomnasiadka: i also have a video :D ..wait a minute 10:04
kevkomaybe ...10:04
kevkomnasiadka: check https://bugs.launchpad.net/kolla-ansible/+bug/1917068 << my comments and also confirmation for example from chalansonnet (schalans) wrote on 2021-03-01: << 10:08
kevkomnasiadka: and I am 100% pretty sure this can be set again to use pooling ...for both proxysql/haproxy usage 10:11
opendevreviewVerification of a change to openstack/kayobe stable/2025.1 failed: Make custom Gather facts run with default python for kayobe-target-venv  https://review.opendev.org/c/openstack/kayobe/+/95172010:31
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Add centos10 job  https://review.opendev.org/c/openstack/kolla-ansible/+/95175110:51
kevkohaha, that 8gb node for AIO set on primary is not enough i would say :D now OOM killed VM :D 11:42
kevko2025-06-04T11:39:04.159421+00:00 debian kernel: [ 7108.969825] Out of memory: Killed process 160427 (qemu-system-x86) total-vm:3542952kB, anon-rss:664600kB, file-rss:0kB, shmem-rss:0kB, UID:42436 pgtables:1772kB oom_score_adj:011:42
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10  https://review.opendev.org/c/openstack/kolla/+/95039211:44
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Use smaller flavor for CI  https://review.opendev.org/c/openstack/kolla-ansible/+/95175811:58
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard  https://review.opendev.org/c/openstack/kolla-ansible/+/95159911:59
fungifrickler: osuosl got sufficient funding, but can always use more: https://lists.osuosl.org/pipermail/hosting/2025-May/000644.html12:29
fungioops, sorry, that was meant for another channel12:30
fungifrickler: mnasiadka: yeah, a gerrit admin can mark groups manually as hidden from th group list12:32
fungii usually also empty them and rename them just to be thorough12:32
mnasiadkafungi: can you or frickler just do that so people don't get confused? :)12:47
fungisure13:41
fungimnasiadka: frickler: done13:45
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard  https://review.opendev.org/c/openstack/kolla-ansible/+/95159913:53
*** Guest17269 is now known as bbezak13:54
kevkoHi bbezak, review this ? https://review.opendev.org/c/openstack/kolla/+/951741 ? 13:55
kevkocan u please ? 13:55
mnasiadkafungi: thank you13:59
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Fix routing of queries without schema to service's DB shard  https://review.opendev.org/c/openstack/kolla-ansible/+/95159914:04
bbezakkevko: you've got +8 now14:05
kevkobbezak thanks :) 14:14
kevkomnasiadka also thanks 👍14:16
mnasiadkaNo problem ;)14:16
kevkomnasiadka: btw, i will have set of important patches for mariadb and rabbitmq ...especially while upgrading ....14:29
kevkomnasiadka: https://review.opendev.org/c/openstack/kolla-ansible/+/95113314:29
kevkomnasiadka: and https://review.opendev.org/c/openstack/kolla-ansible/+/95147414:29
kevkoand depends on 14:30
kevkoit's working already ..but i need to amend scripts and make it bulletproof 14:30
kevkoespecially rabbitmq is needed as community drain just turn on node to maintenance ...and don't wait for leader move :D :D :D 14:30
kevkoand in a sec we are maintenance second node and then third :D 14:31
kevkowhich is bad of course :D 14:31
opendevreviewMerged openstack/kolla master: Fix ProxySQL rule sanity check to support username-based routing  https://review.opendev.org/c/openstack/kolla/+/95174115:50
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10  https://review.opendev.org/c/openstack/kolla/+/95039216:20
opendevreviewPierre Riteau proposed openstack/kayobe master: ansible-lint: Fix jinja[spacing]  https://review.opendev.org/c/openstack/kayobe/+/95179016:38
kevkomnasiadka: frickler: we need to bump AIO jobs to something more than 8cpu 8gb ram I would say ..it's unstable ..sometimes it works sometimes it's not :/ 17:18
opendevreviewVerification of a change to openstack/kayobe stable/2025.1 failed: Make custom Gather facts run with default python for kayobe-target-venv  https://review.opendev.org/c/openstack/kayobe/+/95172017:18
mnasiadkakevko: there is nothing bigger than 8gb17:19
fricklerwell there is, but not in big quantities. also things will get better in the future with the nodepool-in-zuul feature, but no reliable timeframe for that yet17:26
fricklerso for now, better look into reducing job size or making them multinode if needed?17:27
mnasiadkaI still would like to understand what is the problem and why it doesn’t happen on all distros17:37
kevkomnasiadka: well, ubuntu, rocky can have different kernel tunning,  differnet malloc, glibc... whatever ...20:41
kevkomnasiadka: i will try to run shell script with docker stats + top + some  ps aux --sort=-rss or whatever in background and log into /tmp/logs/whatever/ram_consuption ...every 10 seconds for example ...or what ...20:42
kevkobtw ...if some kolla change is merged ...but kolla-ansible job has still depends-on ...but kolla change is merged in master ...is image built before run or not ...because it looks like it's not :) 20:43
*** mikal8 is now known as mikal21:10
opendevreviewMichal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10  https://review.opendev.org/c/openstack/kolla/+/95039222:49

Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!