14:01:26 <mnasiadka> #startmeeting kolla
14:01:27 <opendevmeet> Meeting started Wed Nov 19 14:01:26 2025 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:27 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:27 <opendevmeet> The meeting name has been set to 'kolla'
14:01:29 <mnasiadka> #topic rollcall
14:01:30 <mnasiadka> o/
14:01:34 <bbezak> o/
14:01:44 <BertrandLanson[m]> hello !
14:01:46 <ravlew> o/
14:01:47 <mattcrees> o/
14:01:57 <darmach> \o
14:02:01 <AlexWelsh[m]> o/
14:02:12 <opendevreview> Michal Nasiadka proposed openstack/ansible-collection-kolla master: docker: Configure path in containerd config.toml  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/967676
14:02:50 <mnasiadka> #topic agenda
14:03:24 <mnasiadka> * Announcements
14:03:24 <mnasiadka> * CI status
14:03:24 <mnasiadka> * Current cycle planning
14:03:24 <mnasiadka> * Additional agenda (from whiteboard)
14:03:24 <mnasiadka> * Open discussion
14:03:33 <mnasiadka> #topic Announcements
14:03:56 <mnasiadka> Kolla/Kolla-Ansible Flamingo 2025.2 has been released - thank you for all that contributed to that
14:04:10 <mnasiadka> #topic CI status
14:04:18 <mnasiadka> So… the CI is well… red
14:04:39 <mnasiadka> RabbitMQ removed the PPA repo we used, and now we need to switch to their deb repo - that’s in progress
14:04:47 <mnasiadka> Master should be better now, but stable back ports are still in progress
14:05:11 <mnasiadka> #link https://review.opendev.org/c/openstack/kolla/+/967475
14:05:45 <mnasiadka> 2024.1 is problematic, because the tox jobs check out requirements branch stable/2024.1 (which doesn’t exist anymore, since it’s moved to unmaintained/2024.1)
14:05:59 <mnasiadka> bbezak: did you raise a patch for moving 2024.1 to unmaintained?
14:06:18 <bbezak> I will, too many meetings today ;)
14:06:27 <mnasiadka> Ok then
14:06:36 <mnasiadka> I’m trying to land that RMQ repo patch also in unmaintained/2023.1
14:06:45 <mnasiadka> But there are some other things failing
14:06:50 <BertrandLanson[m]> I'm assuming the rabbitmq stuff prevents us from building older releases until backported ?
14:07:11 <mnasiadka> BertrandLanson[m]: yup - the last good known images are on quay.io - but nothing newer ;)
14:07:29 <mnasiadka> And people locally can’t build the images as well
14:07:43 <mnasiadka> (Well, they can’t without local changes)
14:08:08 <mnasiadka> The other thing is - Docker 29 decided to switch default storage engine to containerd
14:08:16 <BertrandLanson[m]> and anything self-built aswell then, does this warrant new tags for older releases with the fix ? Or do we just install from HEAD of whatever release branch we use ?
14:08:21 <mnasiadka> That means images are now stored in /var/lib/containerd instead of /var/lib/docker
14:08:48 <mnasiadka> BertrandLanson[m]: we recommend users to use stable branches directly from git - but surely we should also tag point releases once we merge the fixes
14:09:12 <mnasiadka> And that means our build jobs run out of storage space, because images get written outside of /var/lib/docker :)
14:09:30 <mnasiadka> Trying to get that fixed in a-c-k
14:09:32 <mnasiadka> #link https://review.opendev.org/c/openstack/ansible-collection-kolla/+/967676
14:09:49 <mnasiadka> No other breakages I know of, unless anybody has something to add to the burning pile :)
14:10:05 <bbezak> Some reasoning is here #link https://www.docker.com/blog/docker-engine-version-29/
14:10:33 <mnasiadka> Ok then, let’s move on
14:10:38 <mnasiadka> #topic Current cycle planning
14:11:09 <mnasiadka> Using that RMQ is fixed for master - and 2025.2 final is released - we can now accept/review new features patches
14:11:26 <mnasiadka> So if anybody has something that needs attention - please add it in the additional agenda on the whiteboard - or post here
14:12:17 <BertrandLanson[m]> there is the old inventory bug I linked a few days ago for cyborg
14:12:46 <BertrandLanson[m]> #link https://review.opendev.org/c/openstack/kolla-ansible/+/934429
14:13:07 <BertrandLanson[m]> which deploys everything everywhere for no reason since it was introduced
14:13:25 <mnasiadka> well, we have no cyborg jobs, so nobody would notice
14:13:35 <mnasiadka> Out of interest - what do you use Cyborg for?
14:13:50 <mnasiadka> bbezak: it already has my +2 - can you have a look?
14:14:00 <bbezak> ok
14:14:13 <BertrandLanson[m]> we used to use it rather than nova for passing NVME directly to VMs
14:14:46 <mnasiadka> Ok, does it clean up these NVMes?
14:14:50 <BertrandLanson[m]> now we can do it with nova but... legacy
14:15:05 <mnasiadka> (Like after removing the instance that has it passed-through)
14:15:18 <BertrandLanson[m]> cleanup as in, wipe them after delete ?
14:15:31 <BertrandLanson[m]> I think so
14:15:46 <BertrandLanson[m]> need to recheck if we have some custom stuff for it but I don't think so
14:16:30 <mnasiadka> That would be great, sometimes I have such questions from customers ;)
14:16:49 <BertrandLanson[m]> there are critical bugs inside cyborg itself tho
14:16:55 <BertrandLanson[m]> past Antelope I believe ?
14:17:17 <BertrandLanson[m]> with oslo db not updated and the nautocommit behavior that changed
14:17:28 <mnasiadka> I’d probably would like some reviews on https://review.opendev.org/q/kolla-ansible-lint starting with https://review.opendev.org/c/openstack/kolla-ansible/+/966790
14:17:41 <mnasiadka> Bbezak: if you have some free time during next coffee break ;)
14:17:53 <mnasiadka> BertrandLanson[m]: ah, another maintained project ;)
14:18:19 <bbezak> ;)
14:19:22 <BertrandLanson[m]> quick question idk if that's the place but what are all these ansible-lint changes for ? I saw like dozens of these and didn't really understand why you were making them
14:19:42 <BertrandLanson[m]> (other the the truthy ones these are pretty obvious)
14:19:47 <mnasiadka> Basically we have exceptions for like 80% of the linting rules now
14:19:55 <mnasiadka> I’d like to stop being the linter :)
14:19:56 <BertrandLanson[m]> oh,lol
14:20:07 <BertrandLanson[m]> yh that makes sense
14:20:10 <BertrandLanson[m]> ok thanks
14:20:49 <mnasiadka> Ok then, no entries in additional agenda
14:20:52 <mnasiadka> Let’s move to open discussion
14:21:03 <mnasiadka> #topic Open discussion
14:21:07 <mnasiadka> Anybody anything?
14:22:47 <BertrandLanson[m]> podman jobs in ci ?
14:23:06 <BertrandLanson[m]> do we have any or do we want any ?
14:23:39 <bbezak> 2024.1 transition to unmaintained #link https://review.opendev.org/c/openstack/releases/+/967694
14:23:47 <mnasiadka> Well, we have a variant of each AIO job using podman
14:24:43 <mnasiadka> Do we need more?
14:24:59 <mnasiadka> bbezak: thank you very much
14:24:59 <BertrandLanson[m]> hm, cause we keep finding weird behavior in the PodmanWorker, I fixed I think most of it but it passed without any issue before
14:25:34 <mnasiadka> I think we need to ask ourselves the question why we don’t see that in CI
14:25:38 <BertrandLanson[m]> there are patches open for it cause currently it's impossible to not restart the entire cluster on reconfigure
14:25:48 <BertrandLanson[m]> cause of idempotence issues
14:25:54 <BertrandLanson[m]> do we reconfigure ?
14:26:48 <mnasiadka> There is a reconfigure step
14:26:51 <mnasiadka> It doesn’t fail
14:27:02 <mnasiadka> But maybe we don’t trigger a real reconfigure
14:27:20 <mnasiadka> Or maybe we don’t check that something that should not be restarted got restarted
14:27:21 <opendevreview> Merged openstack/kolla-ansible master: Fix inventory file for cyborg control services  https://review.opendev.org/c/openstack/kolla-ansible/+/934429
14:27:24 <mnasiadka> That’s probably the case
14:28:14 <BertrandLanson[m]> hm, I'm very bad with zuul still trying to wrap my head around how it all works. But yh podmanworker keeps finding diffs between current and desired state because of defaults imposed on podman's side that end-up "polluting" the api response
14:29:29 <BertrandLanson[m]> I've made unit tests for it in the patches I openend that should see if it breaks cause the old ones were assuming the api response was similar to the docker one which it isn't. But it will only work for as long as the api doesn't change
14:29:56 <mnasiadka> Yeah, so we would need a check after the reconfigure step - to check if any containers that shouldn’t restart have been restarted
14:30:17 <mnasiadka> #link https://opendev.org/openstack/kolla-ansible/src/branch/master/roles/kolla-ansible-reconfigure/tasks
14:30:36 <mnasiadka> Currently we only try to break OVN containers
14:30:42 <mnasiadka> So all the rest should not be restarted
14:30:56 <mnasiadka> You would need probably to dump all containers info and check their restart/start time
14:31:04 <mnasiadka> Or systemd units
14:31:50 <BertrandLanson[m]> Idk if there's a proper start time in the podman api, I can try to see what would work.
14:31:57 <BertrandLanson[m]> that would also force me to leanr it
14:33:33 <mnasiadka> haha
14:33:40 <mnasiadka> Happy to help if you’ll have questions
14:33:56 <mnasiadka> Ok then, let’s finish for today - no other questions and this chit-chat can continue outside the meeting
14:33:59 <mnasiadka> #endmeeting