| opendevreview | Matthew Northcott proposed openstack/magnum-ui master: Support credential API https://review.opendev.org/c/openstack/magnum-ui/+/956850 | 00:52 |
|---|---|---|
| mnasiadka | dalees: can you have a look in 959069? | 05:59 |
| *** Max is now known as Guest25629 | 07:59 | |
| dalees | #startmeeting magnum | 08:01 |
| opendevmeet | Meeting started Tue Sep 2 08:01:11 2025 UTC and is due to finish in 60 minutes. The chair is dalees. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:01 |
| opendevmeet | The meeting name has been set to 'magnum' | 08:01 |
| dalees | #topic Roll call | 08:01 |
| dalees | Hi everyone o/ | 08:01 |
| dalees | not much on the agenda document today, but feature free is upon us and there are several reviews pending we might discuss. | 08:02 |
| dalees | *freeze | 08:02 |
| *** Guest25629 is now known as Max_ | 08:04 | |
| Max_ | Hi all, this is first time attending. Looking forward to contribute where I can! | 08:06 |
| dalees | Hi Max_ , welcome! | 08:06 |
| jakeyip | o/ | 08:08 |
| Max_ | Hi jakeyip | 08:10 |
| dalees | mnasiadka: about? | 08:10 |
| mnasiadka | Sort of, waiting for my doctor’s appointment in a place with poor cell signal :-) | 08:11 |
| dalees | well, we won't expect much chat from you then. | 08:12 |
| dalees | #topic topic? | 08:12 |
| jakeyip | anything that urgently needs to go in ? | 08:13 |
| dalees | what do we want to discuss? If there's a few I can add to agenda, else we can just discuss | 08:13 |
| dalees | jakeyip: matt has landed his api addition for credential rotation, which comes with the client update. So they are good for now. | 08:14 |
| dalees | the ui and tempest will be needing reviews shortly, once the client version is released. | 08:14 |
| dalees | there are one or two pending for Magnum, and a few capi-helm driver | 08:15 |
| dalees | the Magnum one that needs considering is 959069 - the sqlachemy change. I've been reading about that today | 08:16 |
| jakeyip | yeah I'm looking thru that and John is right, we can't edit like that | 08:16 |
| dalees | it's been altered to just add the `mysql_DEFAULT_CHARSET` instead. But I'm curious if we can just add another migration that will go through the tables and set them all. I need to look at the bug report again. | 08:18 |
| dalees | ah it's actually one of the migrations that is failing. So we have to edit a old migration in some way to affect that, we can't just add a new one. | 08:20 |
| jakeyip | yeah there was a couple of reasons that can cause this, one of the mistakes was identified in the bug report where a charset was specified | 08:24 |
| jakeyip | we ran into this a few times IIRC, O cam | 08:25 |
| jakeyip | we ran into this a few times IIRC, I can't remember exactly now but I think there was a change in mysql too | 08:26 |
| jakeyip | existing deployment just need to fix it manually IIRC, a migration won't help, I think it will get stuck at foreign keys | 08:29 |
| dalees | it looks like we could modify the UPDATE query[1] to specify the collation in the impacted migration. Then adding another migration to set all tables' collations might be reasonable? (most db sizes shouldn't be too large, even if it is a re-create operation) | 08:30 |
| dalees | [1] https://opendev.org/openstack/magnum/src/commit/c85b9554d3d3570cc6baa2d69fd1b058682df104/magnum/db/sqlalchemy/alembic/versions/c04e925e65c2_nodegroups_v2.py#L45 | 08:30 |
| dalees | there are a few approaches, anyway. I'm not sure which is most consistent with existing dbs. | 08:31 |
| dalees | `utf8` vs `utf8mb4` is another can of worms we've had to deal with ;) | 08:32 |
| mnasiadka | I’d say if we can define new migration file that fixes it - it would be best not to touch existing migration files | 08:32 |
| dalees | mnasiadka: that linked migration file is where it's breaking in the bug report. So we *have* to touch it. But it might be able to be minimized? | 08:33 |
| jakeyip | I'm saying we can't really, it's hard to know what's the status of existing DBs. depending on when they are created or what the admin as set the default to | 08:33 |
| seunghunlee | I'm not entirely sure how alembic handles this but this bug will only appear when new magnum deployment was done with MariaDB 11.5 or later. So, wouldn't it be okay to touch the old alembic operations? As old deployments already has magnum tables in the DB with consistent utf8mb3_general_ci collation set. | 08:39 |
| jakeyip | I think it will affect any deployment if the default for mysql cluster at that time isn't utf8mb3_general_ci? | 08:42 |
| mnasiadka | I remember changing existing migrations is not the best jdea - but don’t have time this week to investigate. | 08:47 |
| mnasiadka | Maybe let’s ask on openstack ML? Surely Nova or others had something similar in the past. | 08:47 |
| seunghunlee | Perhaps we need to tackle this separately. New deployments and old deployments. My current bugfix will work for the new deployments. For old deployments, some different alembic op or direct sql script to check the inconsistency and alter the table? | 08:47 |
| dalees | I will have an experiment tomorrow. I think we can alter the raw query SQL in the offending migration so it works even with inconsistent tables. Then we can decide if adding another migration to set all tables to a consistent collation/charset. | 08:48 |
| jakeyip | yes seunghunlee that will be good | 08:48 |
| jakeyip | dalees: you will need to `set foreign_key_checks = 0;` if the table has a foreign key, can alembic do that? (haven't tried) | 08:49 |
| seunghunlee | Yes FK columns are also obstacles | 08:50 |
| dalees | jakeyip: ah, noted. I was reading Cinders (as a randomly selected example) which does that, and updates char sets: https://github.com/openstack/cinder/blob/master/cinder/db/migrations/versions/921e1a36b076_initial.py#L921 | 08:51 |
| dalees | (that's all in a `if connection.engine.name == "mysql":`, however. Not sure if that matches mariadb) | 08:52 |
| dalees | seunghunlee's setting of default charsets seems fairly reasonable and ends up better than adding COLLATE to the query. Then we just worry about the existing dbs | 08:58 |
| jakeyip | erm qn: how are deployment tools creating databases? | 09:01 |
| jakeyip | I found https://docs.openstack.org/install-guide/environment-sql-database-ubuntu.html , if followed by deployment tools, will make this a non-issue for new deployments. not taking away anything from seunghunlee's patch | 09:04 |
| dalees | not the most common approach anymore, but the easiest to find: https://github.com/openstack/puppet-magnum/blob/master/manifests/db/mysql.pp#L47 | 09:05 |
| jakeyip | well we still use puppet :) | 09:06 |
| jakeyip | so puppet defaults to that too | 09:07 |
| jakeyip | so puppet defaults to utf8 too | 09:07 |
| dalees | magnum ansible calls `openstack.osa.db_setup`: https://github.com/openstack/openstack-ansible-os_magnum/blob/master/tasks/main.yml#L31 | 09:08 |
| dalees | but i'm not sure where that role lives. if only we had someone here who knew OSA ;) | 09:08 |
| jakeyip | seunghunlee: what does 'show create database magnum' on your db show? | 09:10 |
| seunghunlee | One sec please | 09:11 |
| mnasiadka | Not that I'm picky - should we have some CI test/job that uses new mariadb? | 09:14 |
| dalees | osa doesn't specify collation or encoding, and the ansible module doesn't either: https://github.com/openstack/openstack-ansible-plugins/blob/master/roles/db_setup/tasks/main.yml#L24 | 09:15 |
| mnasiadka | I'd assume just running the centos10 devstack job should uncover that problem | 09:15 |
| jakeyip | mnasiadka: that'll test devstack which won't show an issue if someone is using say kolla-ansible for deployment? | 09:16 |
| mnasiadka | I doubt kolla-ansible or OSA are doing something really wrong | 09:16 |
| opendevreview | Michal Nasiadka proposed openstack/magnum-tempest-plugin master: CI: Introduce CentOS 10 Stream job https://review.opendev.org/c/openstack/magnum-tempest-plugin/+/959113 | 09:19 |
| mnasiadka | Let's see | 09:19 |
| jakeyip | not to say they are wrong, rather are they specifying any charset / collation when CREATING the DB cluster, or when creating the DBs for the service | 09:19 |
| seunghunlee | jakeyip: show create database magnum shows CREATE DATABASE `magnum` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_general_ci */ | 09:20 |
| mnasiadka | Let's look at Kolla-Ansible - https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/magnum/tasks/bootstrap.yml#L19 | 09:21 |
| mnasiadka | nothing crazy here | 09:21 |
| mnasiadka | https://github.com/openstack/kolla-ansible/blob/0e6bab76fe1b3e47e21295c5a23a8008e395b8de/ansible/roles/mariadb/templates/galera.cnf.j2#L5 | 09:21 |
| seunghunlee | But this was deployed with K-A and there was a modification to mariadb config. Such as init-connect='SET NAMES utf8mb3 COLLATE utf8mb3_general_ci' | 09:21 |
| mnasiadka | this is the only place where we force utf8 | 09:21 |
| seunghunlee | But this did not stop the bug | 09:22 |
| dalees | OSA galera has this commit setting the collation-server config. Some relevant info in the commit message: https://github.com/openstack/openstack-ansible-galera_server/commit/ac3942da3daf7c8580b9a728644909b5c52f2c13 | 09:24 |
| jakeyip | seunghunlee: in taht case, creating a table without charset specified should have taken the charet of the db. you can create a temp table to verify like - create table `temp` (`created_at` datetime DEFAULT NULL); | 09:26 |
| mnasiadka | dalees: Kolla-Ansible sets the same, and I assume seunghunlee hasn't changed that setting - so it doesn't help | 09:26 |
| jakeyip | strange , do you have a new deployment with K-A you can verify? | 09:29 |
| seunghunlee | No I did not change that. I only changed init-connect to see if forcing collation for clients can fix the issue. | 09:31 |
| seunghunlee | It did not | 09:31 |
| jakeyip | mnasiadka: do you have a new deployment with K-A you can verify? what ends up being the charsets of those tables and db? | 09:32 |
| jakeyip | riadb docs here https://mariadb.com/docs/server/reference/data-types/string-data-types/character-sets/setting-character-sets-and-collations | 09:32 |
| seunghunlee | I can try new deployment after destroying current deployment now. | 09:33 |
| jakeyip | 1. if nothing is set, you end up with mariadb default of utf8mb4 and utf8mb4_uca1400_ai_ci | 09:33 |
| noonedeadpunk | fwiw, I got down to this yesterday | 09:34 |
| noonedeadpunk | and eventually with >=11.5 if you supply charset, then new variable character-set-collations has precedence over all existing | 09:35 |
| noonedeadpunk | smth like that I come up with for OSA: https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/959075 | 09:36 |
| noonedeadpunk | but the fact that collation-server/character-set-server is simply ignored is extremely confusing | 09:36 |
| noonedeadpunk | and with current patchset I get `utf8mb4_uca1400_ai_ci` for everything on new deployments | 09:37 |
| noonedeadpunk | so kolla patch will be likely needed if you go over 11.5 to keep current behavior | 09:38 |
| noonedeadpunk | but if you don't - then upgrades might start failing in some time, as newly introduced tables to services will have different collation then old ones | 09:39 |
| dalees | yeah ignoring those server values is confusing. I'm glad the new default is utf8mb4 though | 09:39 |
| noonedeadpunk | what is more confusing is that they are not ignored, if CHARSET is not supplied with query | 09:40 |
| seunghunlee | Yeah I found this also very confusing. I don't like MariaDB's design choice here. | 09:40 |
| noonedeadpunk | So if you have `collation-server = utf8mb4_general_ci` and will do first patchset (https://review.opendev.org/c/openstack/magnum/+/959069/1) - then it will be utf8mb4_general_ci | 09:41 |
| * noonedeadpunk going to leave bug report in place for them to evaluate the choice | 09:41 | |
| noonedeadpunk | but it's good that at least one service was incosistent in charsets, as otherwise it could be a bigger tragedy tbh | 09:45 |
| dalees | I will end the meeting officially, thanks for participating everyone. Continue the discussion, however! | 09:45 |
| dalees | #endmeeting magnum | 09:45 |
| opendevmeet | Meeting ended Tue Sep 2 09:45:44 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:45 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/magnum/2025/magnum.2025-09-02-08.01.html | 09:45 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/magnum/2025/magnum.2025-09-02-08.01.txt | 09:45 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/magnum/2025/magnum.2025-09-02-08.01.log.html | 09:45 |
| jakeyip | noonedeadpunk: do you have a link to the >=11.5 character-set-collations defaults? | 09:46 |
| dalees | the docs mention it here: https://mariadb.com/docs/server/reference/data-types/string-data-types/character-sets/setting-character-sets-and-collations | 09:48 |
| noonedeadpunk | there was a better one | 09:48 |
| dalees | ah, no thats the other defaults anyway. | 09:49 |
| dalees | https://jira.mariadb.org/browse/MDEV-25829 | 09:49 |
| noonedeadpunk | https://mariadb.com/docs/server/server-management/variables-and-modes/server-system-variables#character_set_collations | 09:49 |
| noonedeadpunk | and also there was also https://mariadb.com/docs/server/reference/data-types/string-data-types/character-sets/supported-character-sets-and-collations | 09:51 |
| noonedeadpunk | which mentions default changes between versions | 09:51 |
| jakeyip | seunghunlee: just to confirm you are running mariadb >=11.5 ? | 09:54 |
| seunghunlee | I'm testing with MaraiDB 11.8 | 09:55 |
| mnasiadka | oh lol, devstack is installing mariadb 10.11 | 09:59 |
| mnasiadka | https://zuul.opendev.org/t/openstack/build/e0e939b565ee48c3a5cd2a2c52a8e872/log/job-output.txt#6466 | 10:00 |
| jakeyip | noonedeadpunk: just to confirm, because utf8 , utf8_general_ci is NOT in the default character-set-collations (for >=11.5), those settings in mariadb server conf are simply ignored? | 10:00 |
| seunghunlee | I don't think those are ignored when set in conf. utf8 is still pointing at utf8mb3 not utf8mb4. | 10:02 |
| jakeyip | seunghunlee: can you test, by setting only collation-server to utf8_general_ci and character-set-server to utf8 (like how K-A does it), what are the charset and collation for both magnum DB and the tables ? | 10:06 |
| jakeyip | noonedeadpunk says it ends up being utf8mb4_uca1400_ai_ci , we need to confirm this | 10:07 |
| seunghunlee | jakeyip: I'm currently deploying MariaDB 11.8 and magnum with K-A default. I'll let you know | 10:09 |
| seunghunlee | K-A default overrides charset and server collation set: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/mariadb/templates/galera.cnf.j2 | 10:11 |
| noonedeadpunk | jakeyip: utf8 is an alias to utf8mb3 fwiw | 10:12 |
| noonedeadpunk | also it's complicated | 10:13 |
| noonedeadpunk | they are not fully ignored. But when you supply `CHARSET` in SQL query, instead of relying on a default collation, it seems to be picking from the character_set_collations | 10:14 |
| noonedeadpunk | so there's thin unobvious precedence that exists | 10:14 |
| seunghunlee | That's correct and that's the starting point of the bug | 10:14 |
| noonedeadpunk | even despite the fact, that all prior existing pointers say it should be a different one | 10:15 |
| noonedeadpunk | so I'm not sure it's bug per say, but it's really very unobvious behavior | 10:16 |
| noonedeadpunk | anyway | 10:18 |
| jakeyip | I noticed noonedeadpunk testing defaults to utf8mb4_uca1400_ai_ci but seunghunlee bug report got utf8mb3_uca1400_ai_ci, is that important? | 10:20 |
| noonedeadpunk | I aim for utf8mb3_general_ci at all :D | 10:21 |
| seunghunlee | I don't think so, I also experienced same bug with all table's charset is utf8mb3. Magnum bug is caused by collation set mismatch. | 10:21 |
| noonedeadpunk | as I don't think we can just go and change charset and collation | 10:22 |
| noonedeadpunk | ++ | 10:22 |
| noonedeadpunk | yeah, it's just that some tables are going with new default as charset is supplied while others go with old behavior due to not adding charset | 10:23 |
| seunghunlee | Exactly | 10:24 |
| opendevreview | Stephen Finucane proposed openstack/magnum master: api: Use application_url, not host_url https://review.opendev.org/c/openstack/magnum/+/952762 | 10:25 |
| opendevreview | Stephen Finucane proposed openstack/magnum master: trivial: Rewrap controller https://review.opendev.org/c/openstack/magnum/+/959115 | 10:25 |
| noonedeadpunk | and tbh should be backported as far as possible as well.... | 10:27 |
| jakeyip | does OSA specify both charset and collation when creating the DB (not table) | 10:28 |
| noonedeadpunk | no, we don't speccify anything when creating DB | 10:29 |
| noonedeadpunk | we expect it to use defaults, which it does | 10:29 |
| noonedeadpunk | as in bug https://jira.mariadb.org/browse/MDEV-37544 yo ucan see that db collation and charset are as expected | 10:30 |
| noonedeadpunk | have no effect on tables anymore though.... | 10:30 |
| noonedeadpunk | but if I do `CREATE TABLE .....` without `DEFAULT CHARSET=utf8mb3` - table collation will be utf8mb3_general_ci as for the database default | 10:32 |
| jakeyip | sounds like this is going to be a bigger issue as time goes by | 10:36 |
| jakeyip | thanks for this, it is a great find | 10:36 |
| noonedeadpunk | frankly - it would be way bigger if magnum as well was consistent in how to create charsets | 10:37 |
| noonedeadpunk | as then CI won't fail and it's unlikely us to realize that charset is different now by default | 10:37 |
| noonedeadpunk | until it was too late... | 10:37 |
| noonedeadpunk | *how to create tables | 10:38 |
| noonedeadpunk | so thanks magnum for us finding this issue and staying on top of it! | 10:38 |
| jakeyip | well if magnum was consistent and everyone was, it is a non-problem too :P | 10:39 |
| noonedeadpunk | it is problem... | 10:39 |
| noonedeadpunk | as think of an upgrade - all tables and databases have default of utf8mb3_general_ci for instance. Now they upgrade mariadb to 11.8 or smth, and upgrade openstack | 10:40 |
| noonedeadpunk | and now, all new tables are being created with utf8mb3_uca1400_ai_ci | 10:40 |
| noonedeadpunk | because defaults are no longer respected the same way as they were for last 2 decades | 10:40 |
| jakeyip | oh yah true, upgrading mariadb | 10:42 |
| noonedeadpunk | and now we can do smth on kolla/osa side to safeguard users as well | 10:42 |
| noonedeadpunk | so it's a good thing :) | 10:42 |
| jakeyip | going forward, should we specify charset in a table? | 10:43 |
| noonedeadpunk | and that's why I'm kinda thinking about backport of this patch down the road - as there totally could be a new deployment on 11.8 and Epoxy or anything actually | 10:43 |
| noonedeadpunk | charset is specified already | 10:43 |
| noonedeadpunk | collation is not | 10:43 |
| noonedeadpunk | and it's a good question if collation should be specified... | 10:44 |
| noonedeadpunk | I think, it should, tbh | 10:44 |
| jakeyip | I am thinking, if there is no need to, maybe we can remove specify charset and let the deployment tools handle the default | 10:45 |
| jakeyip | remove specifying charset in the table create | 10:45 |
| noonedeadpunk | frankly I'm thinking more in direction to be even more specific and define collation as well | 10:46 |
| noonedeadpunk | as the only way to change that down the road without breaking things - would be some kind of migration | 10:46 |
| noonedeadpunk | So it's not really on plate of deployment tools to handle charset/collation changes | 10:46 |
| jakeyip | if the deployment tools create the database with both charset and collation, then the projects don't need to | 10:46 |
| noonedeadpunk | we jsut have `create database XXX` more or less | 10:47 |
| noonedeadpunk | but imo - that's up to project to say - well, we can't work with utf8mb3 - we need to have utf8mb4 to store X | 10:47 |
| noonedeadpunk | and then create a migration from one charset to another | 10:48 |
| jakeyip | you already set that for the server, create database just takes that and sets it as the database default | 10:48 |
| noonedeadpunk | which is not helpful as we've just found :D | 10:49 |
| seunghunlee | K-A MariaDB 11.8 and magnum deployment is done and show create database magnum gives: CREATE DATABASE `magnum` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_general_ci */ | 10:49 |
| jakeyip | if a project has specified needs, yes, they can specify (both). else default to not specifying either | 10:49 |
| noonedeadpunk | also you probably can't do that at this point. as there could be deployments with default of latin1 for instance. now, removing charset from migrations such deployments will be broken | 10:50 |
| noonedeadpunk | also I'd say that deployment tools are covering like what? 60% of all deployments? | 10:51 |
| jakeyip | there's install guide here also - https://docs.openstack.org/install-guide/environment-sql-database-ubuntu.html | 10:52 |
| noonedeadpunk | eh, ok, then dunno :) | 10:53 |
| jakeyip | (install guide prob needs update too) :) | 10:54 |
| noonedeadpunk | I frankly now thinking of the way for us to do migration to better charset and collation as a community | 10:54 |
| noonedeadpunk | and if it's worth discussing... | 10:55 |
| jakeyip | maybe it's a TC thing? some good defaults? | 10:56 |
| noonedeadpunk | ++ | 10:56 |
| jakeyip | existing deployments and projects can figure out how to get there (if they don't have need for something different) | 10:57 |
| noonedeadpunk | Like seemingly, the easiest way could be an alembic migration.... | 10:57 |
| noonedeadpunk | but then defining no charset now would make such migration really weird | 10:58 |
| jakeyip | like if TC says default to utf8mb3 utf8mb3_general_ci, install guide and deployment tools can then follow. project will remove specifying something if the default works for them | 10:58 |
| jakeyip | existing deployments will always be a problem, but if I know the default it's easy to fix it all up manually or with a alembic | 11:00 |
| jakeyip | oh hmm... scratch that, it's not easy figuring out if the 'default' has been overridden by an operator. | 11:01 |
| jakeyip | I've summarised the convo in the bug report, https://bugs.launchpad.net/magnum/+bug/2121797/comments/6 | 12:14 |
| seunghunlee | jakeyip: That's great. Thank you | 12:30 |
| *** Max is now known as Guest25655 | 15:08 | |
| *** Max is now known as Guest25664 | 16:20 | |
| opendevreview | Jason Rayne proposed openstack/magnum-capi-helm master: Feat: Add cluster template labels for custom pod/service CIDRs https://review.opendev.org/c/openstack/magnum-capi-helm/+/958720 | 17:24 |
| *** Max is now known as Guest25671 | 18:08 | |
| *** Max is now known as Guest25674 | 19:09 | |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!