08:01:11 <dalees> #startmeeting magnum
08:01:11 <opendevmeet> Meeting started Tue Sep  2 08:01:11 2025 UTC and is due to finish in 60 minutes.  The chair is dalees. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:01:11 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:01:11 <opendevmeet> The meeting name has been set to 'magnum'
08:01:21 <dalees> #topic Roll call
08:01:27 <dalees> Hi everyone o/
08:02:16 <dalees> not much on the agenda document today, but feature free is upon us and there are several reviews pending we might discuss.
08:02:21 <dalees> *freeze
08:06:25 <Max_> Hi all, this is first time attending. Looking forward to contribute where I can!
08:06:54 <dalees> Hi Max_ , welcome!
08:08:46 <jakeyip> o/
08:10:08 <Max_> Hi jakeyip
08:10:23 <dalees> mnasiadka: about?
08:11:13 <mnasiadka> Sort of, waiting for my doctor’s appointment in a place with poor cell signal :-)
08:12:27 <dalees> well, we won't expect much chat from you then.
08:12:54 <dalees> #topic topic?
08:13:18 <jakeyip> anything that urgently needs to go in ?
08:13:23 <dalees> what do we want to discuss? If there's a few I can add to agenda, else we can just discuss
08:14:06 <dalees> jakeyip: matt has landed his api addition for credential rotation, which comes with the client update. So they are good for now.
08:14:31 <dalees> the ui and tempest will be needing reviews shortly, once the client version is released.
08:15:53 <dalees> there are one or two pending for Magnum, and a few capi-helm driver
08:16:29 <dalees> the Magnum one that needs considering is 959069 - the sqlachemy change. I've been reading about that today
08:16:57 <jakeyip> yeah I'm looking thru that and John is right, we can't edit like that
08:18:16 <dalees> it's been altered to just add the `mysql_DEFAULT_CHARSET` instead. But I'm curious if we can just add another migration that will go through the tables and set them all. I need to look at the bug report again.
08:20:14 <dalees> ah it's actually one of the migrations that is failing. So we have to edit a old migration in some way to affect that, we can't just add a new one.
08:24:00 <jakeyip> yeah there was a couple of reasons that can cause this, one of the mistakes was identified in the bug report where a charset was specified
08:25:44 <jakeyip> we ran into this a few times IIRC, O cam
08:26:00 <jakeyip> we ran into this a few times IIRC, I can't remember exactly now but I think there was a change in mysql too
08:29:21 <jakeyip> existing deployment just need to fix it manually IIRC, a migration won't help, I think it will get stuck at foreign keys
08:30:15 <dalees> it looks like we could modify the UPDATE query[1] to specify the collation in the impacted migration. Then adding another migration to set all tables' collations might be reasonable? (most db sizes shouldn't be too large, even if it is a re-create operation)
08:30:23 <dalees> [1] https://opendev.org/openstack/magnum/src/commit/c85b9554d3d3570cc6baa2d69fd1b058682df104/magnum/db/sqlalchemy/alembic/versions/c04e925e65c2_nodegroups_v2.py#L45
08:31:01 <dalees> there are a few approaches, anyway. I'm not sure which is most consistent with existing dbs.
08:32:17 <dalees> `utf8` vs `utf8mb4` is another can of worms we've had to deal with ;)
08:32:35 <mnasiadka> I’d say if we can define new migration file that fixes it - it would be best not to touch existing migration files
08:33:24 <dalees> mnasiadka: that linked migration file is where it's breaking in the bug report. So we *have* to touch it. But it might be able to be minimized?
08:33:32 <jakeyip> I'm saying we can't really, it's hard to know what's the status of existing DBs. depending on when they are created or what the admin as set the default to
08:39:02 <seunghunlee> I'm not entirely sure how alembic handles this but this bug will only appear when new magnum deployment was done with MariaDB 11.5 or later. So, wouldn't it be okay to touch the old alembic operations? As old deployments already has magnum tables in the DB with consistent utf8mb3_general_ci collation set.
08:42:03 <jakeyip> I think it will affect any deployment if the default for mysql cluster at that time isn't utf8mb3_general_ci?
08:47:13 <mnasiadka> I remember changing existing migrations is not the best jdea - but don’t have time this week to investigate.
08:47:37 <mnasiadka> Maybe let’s ask on openstack ML? Surely Nova or others had something similar in the past.
08:47:52 <seunghunlee> Perhaps we need to tackle this separately. New deployments and old deployments. My current bugfix will work for the new deployments. For old deployments, some different alembic op or direct sql script to check the inconsistency and alter the table?
08:48:31 <dalees> I will have an experiment tomorrow. I think we can alter the raw query SQL in the offending migration so it works even with inconsistent tables. Then we can decide if adding another migration to set all tables to a consistent collation/charset.
08:48:35 <jakeyip> yes seunghunlee that will be good
08:49:50 <jakeyip> dalees: you will need to `set foreign_key_checks = 0;` if the table has a foreign key, can alembic do that? (haven't tried)
08:50:58 <seunghunlee> Yes FK columns are also obstacles
08:51:41 <dalees> jakeyip: ah, noted. I was reading Cinders (as a randomly selected example) which does that, and updates char sets: https://github.com/openstack/cinder/blob/master/cinder/db/migrations/versions/921e1a36b076_initial.py#L921
08:52:37 <dalees> (that's all in a `if connection.engine.name == "mysql":`, however. Not sure if that matches mariadb)
08:58:29 <dalees> seunghunlee's setting of default charsets seems fairly reasonable and ends up better than adding COLLATE to the query. Then we just worry about the existing dbs
09:01:33 <jakeyip> erm qn: how are deployment tools creating databases?
09:04:38 <jakeyip> I found https://docs.openstack.org/install-guide/environment-sql-database-ubuntu.html , if followed by deployment tools, will make this a non-issue for new deployments. not taking away anything from seunghunlee's patch
09:05:02 <dalees> not the most common approach anymore, but the easiest to find: https://github.com/openstack/puppet-magnum/blob/master/manifests/db/mysql.pp#L47
09:06:51 <jakeyip> well we still use puppet :)
09:07:26 <jakeyip> so puppet defaults to that too
09:07:34 <jakeyip> so puppet defaults to utf8 too
09:08:31 <dalees> magnum ansible calls `openstack.osa.db_setup`: https://github.com/openstack/openstack-ansible-os_magnum/blob/master/tasks/main.yml#L31
09:08:58 <dalees> but i'm not sure where that role lives. if only we had someone here who knew OSA ;)
09:10:07 <jakeyip> seunghunlee: what does 'show create database magnum' on your db show?
09:11:31 <seunghunlee> One sec please
09:14:59 <mnasiadka> Not that I'm picky -  should we have some CI test/job that uses new mariadb?
09:15:10 <dalees> osa doesn't specify collation or encoding, and the ansible module doesn't either: https://github.com/openstack/openstack-ansible-plugins/blob/master/roles/db_setup/tasks/main.yml#L24
09:15:19 <mnasiadka> I'd assume just running the centos10 devstack job should uncover that problem
09:16:26 <jakeyip> mnasiadka: that'll test devstack which won't show an issue if someone is using say kolla-ansible for deployment?
09:16:49 <mnasiadka> I doubt kolla-ansible or OSA are doing something really wrong
09:19:37 <opendevreview> Michal Nasiadka proposed openstack/magnum-tempest-plugin master: CI: Introduce CentOS 10 Stream job  https://review.opendev.org/c/openstack/magnum-tempest-plugin/+/959113
09:19:41 <mnasiadka> Let's see
09:19:58 <jakeyip> not to say they are wrong, rather are they specifying any charset / collation when CREATING the DB cluster, or when creating the DBs for the service
09:20:39 <seunghunlee> jakeyip: show create database magnum shows CREATE DATABASE `magnum` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_general_ci */
09:21:36 <mnasiadka> Let's look at Kolla-Ansible - https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/magnum/tasks/bootstrap.yml#L19
09:21:38 <mnasiadka> nothing crazy here
09:21:52 <mnasiadka> https://github.com/openstack/kolla-ansible/blob/0e6bab76fe1b3e47e21295c5a23a8008e395b8de/ansible/roles/mariadb/templates/galera.cnf.j2#L5
09:21:55 <seunghunlee> But this was deployed with K-A and there was a modification to mariadb config. Such as init-connect='SET NAMES utf8mb3 COLLATE utf8mb3_general_ci'
09:21:58 <mnasiadka> this is the only place where we force utf8
09:22:14 <seunghunlee> But this did not stop the bug
09:24:45 <dalees> OSA galera has this commit setting the collation-server config. Some relevant info in the commit message: https://github.com/openstack/openstack-ansible-galera_server/commit/ac3942da3daf7c8580b9a728644909b5c52f2c13
09:26:31 <jakeyip> seunghunlee: in taht case, creating a table without charset specified should have taken the charet of the db. you can create a temp table to verify like - create table `temp` (`created_at` datetime DEFAULT NULL);
09:26:59 <mnasiadka> dalees: Kolla-Ansible sets the same, and I assume seunghunlee hasn't changed that setting - so it doesn't help
09:29:56 <jakeyip> strange , do you have a new deployment with K-A you can verify?
09:31:41 <seunghunlee> No I did not change that. I only changed init-connect to see if forcing collation for clients can fix the issue.
09:31:50 <seunghunlee> It did not
09:32:40 <jakeyip> mnasiadka:  do you have a new deployment with K-A you can verify? what ends up being the charsets of those tables and db?
09:32:46 <jakeyip> riadb docs here https://mariadb.com/docs/server/reference/data-types/string-data-types/character-sets/setting-character-sets-and-collations
09:33:05 <seunghunlee> I can try new deployment after destroying current deployment now.
09:33:50 <jakeyip> 1. if nothing is set, you end up with mariadb default of utf8mb4 and utf8mb4_uca1400_ai_ci
09:34:14 <noonedeadpunk> fwiw, I got down to this yesterday
09:35:00 <noonedeadpunk> and eventually with >=11.5 if you supply charset, then new variable character-set-collations has precedence over all existing
09:36:11 <noonedeadpunk> smth like that I come up with for OSA: https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/959075
09:36:45 <noonedeadpunk> but the fact that collation-server/character-set-server is simply ignored is extremely confusing
09:37:33 <noonedeadpunk> and with current patchset I get `utf8mb4_uca1400_ai_ci` for everything on new deployments
09:38:11 <noonedeadpunk> so kolla patch will be likely needed if you go over 11.5 to keep current behavior
09:39:16 <noonedeadpunk> but if you don't - then upgrades might start failing in some time, as newly introduced tables to services will have different collation then old ones
09:39:24 <dalees> yeah ignoring those server values is confusing. I'm glad the new default is utf8mb4 though
09:40:10 <noonedeadpunk> what is more confusing is that they are not ignored, if CHARSET is not supplied with query
09:40:58 <seunghunlee> Yeah I found this also very confusing. I don't like MariaDB's design choice here.
09:41:13 <noonedeadpunk> So if you have `collation-server = utf8mb4_general_ci` and will do first patchset (https://review.opendev.org/c/openstack/magnum/+/959069/1) - then it will be utf8mb4_general_ci
09:41:56 * noonedeadpunk going to leave bug report in place for them to evaluate the choice
09:45:12 <noonedeadpunk> but it's good that at least one service was incosistent in charsets, as otherwise it could be a bigger tragedy tbh
09:45:17 <dalees> I will end the meeting officially, thanks for participating everyone.  Continue the discussion, however!
09:45:44 <dalees> #endmeeting magnum