13:02:37 <mnasiadka> #startmeeting kolla 13:02:37 <opendevmeet> Meeting started Wed Sep 3 13:02:37 2025 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:02:37 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:02:37 <opendevmeet> The meeting name has been set to 'kolla' 13:02:40 <mnasiadka> #topic rollcall 13:02:41 <yuval__> R0 13:02:41 <mnasiadka> o/ 13:02:43 <mnasiadka> (now) 13:02:49 <frickler> \o 13:02:52 <amir58118> o/ 13:02:56 <bbezak_> O/ 13:03:56 <seunghunlee> o/ 13:04:15 <bbezak_> (Real bbezak is on broken matrix.org) here is the imposter :) 13:04:43 <mnasiadka> #topic agenda 13:04:43 <mnasiadka> * CI status 13:04:43 <mnasiadka> * Release tasks 13:04:43 <mnasiadka> * Regular stable releases (first meeting in a month) 13:04:43 <mnasiadka> * Current cycle planning 13:04:45 <mnasiadka> * Additional agenda (from whiteboard) 13:04:45 <mnasiadka> * Open discussion 13:04:47 <mnasiadka> #topic CI status 13:05:38 <mnasiadka> so, I've noticed Venus job is failing - check-logs.sh is not finding any fluentd data for it - so I proposed https://review.opendev.org/c/openstack/kolla-ansible/+/959298 13:06:04 <mnasiadka> zun is also failing due to changes in pyroute2 - proposing to drop jobs for now - https://review.opendev.org/c/openstack/kolla-ansible/+/959307 13:06:57 <mnasiadka> We have that dns_assignment intermittent issue in ovn jobs - we merged some fix to loop in checking that - it seems it's helping a bit more - but frickler agreed to look into a follow up testing it more and finding out if it's a Neutron bug 13:07:13 <mnasiadka> If there are any other jobs persistently failing - please let us know 13:08:03 <frickler> venus is ptl-less, might be retired soon. does anyone care for it? 13:08:25 <mnasiadka> I don't, let's wait for TC hunt for a PTL - but I doubt anyone serious is using that 13:08:47 <mnasiadka> #topic Release tasks 13:09:25 <mnasiadka> We're not doing cycle highlights, because it's hard to predict - otherwise than that we're good - we dropped RDO, switched UCA, Debian OpenStack is not releasing Bookworm 2025.2 13:10:08 <mnasiadka> By dropping RDO we might have broken skyline (not that I care a lot for this project) - it seems it's using MySQLDb pip package 13:10:14 <bbezak_> So what they are releasing? Trixie 2025.2? 13:10:19 <mnasiadka> Trixie 2025.2 13:10:47 <frickler> trixie has 2025.1 natively already, iiuc. so yes 13:10:54 <mnasiadka> Anyway, let's move on 13:11:01 <mnasiadka> #topic Current cycle planning 13:11:13 <mnasiadka> I think seunghunlee's MariaDB discussion fits more here 13:11:27 <seunghunlee> hello 13:11:28 <mnasiadka> I'm not going to paste the full drama here 13:11:30 <mnasiadka> #link https://etherpad.opendev.org/p/KollaWhiteBoard#L68 13:11:58 <mnasiadka> TLDR; MariaDB changed default collation set of charset utf8mb3 from utf8mb3_general_ci to utf8mb3_uca1400_ai_ci from 11.5 https://jira.mariadb.org/browse/MDEV-25829 13:12:15 <mnasiadka> And it breaks even deployments 13:13:21 <mnasiadka> There has been a hint of this problem by noonedeadpunk in yesterday's TC meeting - because OSA is on the same boat - but they already moved 13:13:52 <mnasiadka> From my perspective - it would be good to send a mail to openstack-discuss ML and be part of the discussion in the TC weekly meeting next week 13:14:21 <mnasiadka> I'll wait some minutes for bbezak_, frickler, seunghunlee and others to chime in 13:14:22 <seunghunlee> I also wanted to suggest that. As this is inter-service problem 13:14:38 <bbezak_> I agree, this is potentially problem for whole openstack in general 13:14:43 <mnasiadka> We don't HAVE TO migrate - mariadb 10.11 is LTS and has two more years of support 13:14:47 <frickler> I don't think we need to rush this, sticking to 11.4 or even 10.11 for another cycle or two should be fine 13:15:11 <mnasiadka> Well, maybe the solution is to jump to 11.4 this cycle 13:15:20 <mnasiadka> Because that sounds like a safe-ish solution 13:15:34 <seunghunlee> That can work too 13:15:36 <mnasiadka> buy some popcorn and wait for the evolution 13:15:39 <frickler> https://mariadb.org/11-8-is-lts/ has a nice table 13:15:47 <yuval__> :) 13:16:15 <mnasiadka> seunghunlee: so from my perspective - send out a mail with summary to openstack-discuss ML - and attend the next weekly TC meeting 13:16:21 <mnasiadka> and let's see how it evolves 13:16:41 <mnasiadka> and in parallel - we can migrate to 11.4 which will give us some more years of support 13:16:42 <seunghunlee> Sounds good 13:16:51 <bbezak_> 11.4 and popcorn 13:16:53 <mnasiadka> and should not introduce any other problems ;-) 13:16:53 <bbezak_> good 13:17:01 <seunghunlee> hopefully 13:17:21 <mnasiadka> ok then 13:17:23 <seunghunlee> 11.4 still needs adjusted way of recovery iirc 13:17:38 <seunghunlee> which is https://review.opendev.org/c/openstack/kolla-ansible/+/958281 13:17:46 <mnasiadka> if you can prepare a Gerrit topic with patches needing upgrade to 11.4 - we're happy to review it 13:18:04 <bbezak_> And 11.4 has even longer regular support than 11.8 and future 12.3 13:18:18 <bbezak_> (community) 13:18:36 <seunghunlee> Will do. Although I think this one is in the topic 13:18:57 <seunghunlee> Yeah. As in kolla_mariadb_11 13:19:01 <opendevreview> Pierre Riteau proposed openstack/kayobe master: Support CentOS Stream 10 and Rocky Linux 10 images https://review.opendev.org/c/openstack/kayobe/+/959306 13:19:24 <frickler> seunghunlee: it that new method compatible with older mariadb versions? 13:19:33 <frickler> s/it/is/ 13:20:53 <seunghunlee> It passed CI with both 10.11 and 11.8 13:21:04 <seunghunlee> and CI tests recovery 13:21:29 <frickler> ok, that should be good enough I think, thx, will put it on my review list 13:21:36 <seunghunlee> Thanks 13:21:46 <mnasiadka> ok then, let's move on 13:21:59 <mnasiadka> Ah, skipped one 13:22:01 <mnasiadka> #topic Regular stable releases (first meeting in a month) 13:22:14 <mnasiadka> bbezak_: have we merged the mariadb unpin in 2025.1? 13:22:37 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla/+/956538 13:22:38 <mnasiadka> nope 13:22:43 <mnasiadka> so no stable releases yet 13:23:29 <mnasiadka> let's try to prepare for it correctly 13:23:41 <bbezak_> We didn’t have those in a while now 13:24:04 <mnasiadka> #topic Open discussion 13:24:10 <mnasiadka> Anybody anything? 13:24:13 <bbezak_> @fungi 13:24:13 <yuval__> yea 13:24:20 <bbezak_> #link https://review.opendev.org/c/openstack/kolla/+/958763 13:24:38 <bbezak_> As promised in some IRC meeting, here is the doc update for contributors 13:24:50 <dougszu> Group IDs below 1000 13:25:00 <Vii> hey I think this is important before releasing a new version https://review.opendev.org/c/openstack/kolla-ansible/+/958888 13:25:15 <Vii> ssl hardening 13:25:45 <mnasiadka> Vii: it's not getting in stable as a backport - we we're discussing stable branch point releases 13:26:14 <Vii> It would be nice if this worked too :) it's a great help for test environments https://review.opendev.org/c/openstack/kolla-ansible/+/958861 13:26:19 <mnasiadka> bbezak_: thanks, let me and frickler review this (if he has spare cycles) and maybe fungi can chime in 13:26:29 <yuval__> I am running with ubuntu jammy, this commit in nova breaks the deployment: https://opendev.org/openstack/nova/commit/e4340cd8e54fe3b447192e7982693793c60569cd#diff-93e955ea3f3f2784989c87ebc799c16b175cbb1b 13:26:53 <mnasiadka> yuval__: which branch? master? 13:27:22 <mnasiadka> We don't see any problem with that in our master CI testing, that's for sure 13:27:35 <mnasiadka> So if you do - it's rather a topic for Nova team 13:28:03 <yuval__> the nova docker is missing a module to import oslo_service.backend 13:28:11 <yuval__> the commit was merged 3 weeks ago 13:28:22 <yuval__> so docker older than that probably see the same issue 13:28:36 <yuval__> https://quay.io/repository/openstack.kolla/nova-libvirt?tab=tags 13:28:58 <mnasiadka> we're not running any nova component in nova-libvirt 13:29:02 <mnasiadka> it's pure libvirt there 13:29:39 <mnasiadka> Basically - if you're seeing issues that we're not seeing in master branch CI - you're either using old images or old kolla-ansible code 13:29:46 <yuval__> I can check exactly where its failing 13:29:53 <yuval__> but yes I agree with you 13:30:10 <bbezak_> Please create a bug if you feel this is a kolla problem 13:30:11 <priteau> This patch for cyborg dev mode has been open for 2 years, I think it's good to go - I tested it back in April: https://review.opendev.org/c/openstack/kolla-ansible/+/890883 13:30:18 <mnasiadka> Yeah, please do - and come back with some outputs (use paste.openstack.org) 13:30:55 <mnasiadka> priteau: I'll have a look later 13:31:04 <mnasiadka> Ok then, anything else? 13:31:16 <dougszu> This change: https://review.opendev.org/c/openstack/kolla/+/930931 create a side effect of the group IDs in the container getting out of sync with the host 13:31:49 <seunghunlee> I left comment on https://review.opendev.org/c/openstack/kolla-ansible/+/928487 and https://review.opendev.org/c/openstack/kolla-ansible/+/927096 to try unblocking. They are also in kolla_mariadb_11 topic 13:32:20 <dougszu> Perhaps it is easier to manage groups IDs below 1000 explicitly as for Kolla groups? I have amended this patch to do it that way: https://review.opendev.org/c/openstack/kolla/+/955388 13:33:41 <mnasiadka> Well, today we use 424XX for Kolla users/groups 13:33:53 <mnasiadka> So what is getting 999 now? 13:34:10 <dougszu> On Noble 999 is now the systemd-journal 13:34:38 <dougszu> But in the Noble container, GID 999 gets taken by something random from udev. Since we have removed the config for systemd-journal in the container 13:36:12 <dougszu> The current approach I have taken is to reserve 999 in advance of udev taking it, which seems to be good enough for now. 13:36:19 <mnasiadka> Other option would be probably to set some MIN uid/guid range for users/groups that get added 13:36:35 <mnasiadka> But I guess that's fine for now 13:36:50 <mnasiadka> But I think we need a test in kolla-ansible to make sure we don't break it next time 13:36:55 <dougszu> This is all happening in post-install scripts from debian packages 13:37:02 <frickler> is this only for noble? or do other distros have the same issue with gid allocation? 13:37:25 <Vii> One more thing. What do you think? Is this worth continuing? So that not all users are in every image? https://review.opendev.org/c/openstack/kolla/+/951944 13:37:25 <Vii> users.py file - keystone user - example 13:37:29 <dougszu> It is only Noble where I see the issue (specifically for systemd) - I think the issue was triggered by this: https://review.opendev.org/c/openstack/kolla/+/930931 13:37:41 <mnasiadka> We're setting LAST_SYSTEM_UID in https://github.com/openstack/kolla/blob/cc328dc8656399ec4a68156573e91e6930fa1948/docker/base/Dockerfile.j2#L236 13:37:52 <mnasiadka> Maybe there's a variable for FIRST_SYSTEM_UID? 13:38:55 <dougszu> Ideally we would bind mount in the contents of /usr/lib/sysusers.d from the host and then everything would match in the container 13:39:28 <mnasiadka> Maybe that's an option 13:39:41 <mnasiadka> But still I think if you care for this functionality - we need to have it tested in CI 13:40:39 <dougszu> So we could turn on forwarding of the journal in CI perhaps 13:41:47 <dougszu> I can do that 13:42:05 <dougszu> I will grep for the registry pwd :D 13:42:25 <mnasiadka> Anything ;-) 13:42:42 <mnasiadka> Ok then, let's continue discussion in Gerrit 13:42:51 <dougszu> thanks 13:43:39 <mnasiadka> Vii: Probably it makes sense, but I think we have more priority things - we can discuss that another time - if you refresh that patch 13:43:51 <mnasiadka> Ok then, thanks for coming - see you next week :-) 13:43:53 <mnasiadka> #endmeeting