| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 04:56 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 05:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 05:59 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 05:59 |
| opendevreview | Bhavana S proposed openstack/kolla-ansible master: cinder: disable backup default without ceph/driver https://review.opendev.org/c/openstack/kolla-ansible/+/994593 | 06:12 |
| *** spellbind3r is now known as darkkilla | 06:54 | |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 07:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 07:15 |
| opendevreview | Michael Still proposed openstack/kolla-ansible master: Precheck that the nova_libvirt image ships kolla_nova_libvirt_start. https://review.opendev.org/c/openstack/kolla-ansible/+/994602 | 07:59 |
| opendevreview | Michael Still proposed openstack/kolla-ansible stable/2026.1: Start virtlogd in nova-libvirt and route console logs to it. https://review.opendev.org/c/openstack/kolla-ansible/+/993458 | 08:04 |
| opendevreview | Michael Still proposed openstack/kolla-ansible stable/2025.2: Start virtlogd in nova-libvirt and route console logs to it. https://review.opendev.org/c/openstack/kolla-ansible/+/994603 | 08:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 08:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 08:10 |
| opendevreview | Michael Still proposed openstack/kolla-ansible stable/2025.1: Start virtlogd in nova-libvirt and route console logs to it. https://review.opendev.org/c/openstack/kolla-ansible/+/994604 | 08:10 |
| opendevreview | Bhavana S proposed openstack/kolla-ansible master: cinder: disable backup default without ceph/driver https://review.opendev.org/c/openstack/kolla-ansible/+/994593 | 08:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Use multi-stage builds to reduce runtime image size https://review.opendev.org/c/openstack/kolla/+/969968 | 08:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 08:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 08:57 |
| opendevreview | Merged openstack/kolla stable/2025.2: Update Prometheus to version 3.5.4 https://review.opendev.org/c/openstack/kolla/+/994144 | 09:10 |
| *** jhorstmann is now known as Guest11889 | 09:17 | |
| opendevreview | Bertrand Lanson proposed openstack/kolla-ansible master: keystone: Setup fernet credentials encryption keys https://review.opendev.org/c/openstack/kolla-ansible/+/970861 | 09:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Add support to override base repos https://review.opendev.org/c/openstack/kolla/+/993400 | 09:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: Remove npm proxy configuration https://review.opendev.org/c/openstack/kolla/+/993671 | 09:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Move PIP_* env variables to ARG and config https://review.opendev.org/c/openstack/kolla/+/993675 | 09:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Add cleanup_repos to revert build_only overrides https://review.opendev.org/c/openstack/kolla/+/993712 | 09:28 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Add cleanup_repos to revert build_only overrides https://review.opendev.org/c/openstack/kolla/+/993712 | 09:29 |
| mikal | Huh, I didn't realise that "cherry picked from" bit of the commit message actually did something in gerrit. | 09:34 |
| mikal | Can I add two commits there? The backports really merge to commits to master. | 09:35 |
| opendevreview | Michael Still proposed openstack/kolla-ansible stable/2026.1: Start virtlogd in nova-libvirt and route console logs to it. https://review.opendev.org/c/openstack/kolla-ansible/+/993458 | 09:38 |
| opendevreview | Michael Still proposed openstack/kolla-ansible stable/2025.2: Start virtlogd in nova-libvirt and route console logs to it. https://review.opendev.org/c/openstack/kolla-ansible/+/994603 | 09:39 |
| opendevreview | Michael Still proposed openstack/kolla-ansible stable/2025.1: Start virtlogd in nova-libvirt and route console logs to it. https://review.opendev.org/c/openstack/kolla-ansible/+/994604 | 09:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 09:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 09:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 09:55 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 09:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.1: Add logrotate configuration for OpenSearch Dashboards https://review.opendev.org/c/openstack/kolla-ansible/+/994614 | 10:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.1: Add logrotate configuration for OpenSearch Dashboards https://review.opendev.org/c/openstack/kolla-ansible/+/994614 | 10:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible stable/2025.1: Add logrotate configuration for OpenSearch Dashboards https://review.opendev.org/c/openstack/kolla-ansible/+/994614 | 10:12 |
| mnasiadka | frickler, blanson[m]: If possible - I’d like to get some reviews on https://review.opendev.org/c/openstack/kolla/+/969968 :-) | 10:31 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Add purity_fb to python deps for manila-base https://review.opendev.org/c/openstack/kolla/+/973417 | 10:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 10:46 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 10:47 |
| frickler | reviewing without finished CI is difficult, now I have this tab open and need to return to it later :-/ or maybe I can start reading anyway, big patch ... | 10:57 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Use multi-stage builds to reduce runtime image size https://review.opendev.org/c/openstack/kolla/+/969968 | 10:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 11:07 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 11:08 |
| blanson[m] | will take a look | 11:08 |
| blanson[m] | https://review.opendev.org/c/openstack/kolla-ansible/+/970861 updated this one with y'all recommendations, should be close to done now | 11:09 |
| blanson[m] | I will probbly abandon the patch that's on top of it to redo it properly, cause I added a bunch of stuff in the meantime | 11:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 11:13 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 11:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Use multi-stage builds to reduce runtime image size https://review.opendev.org/c/openstack/kolla/+/969968 | 11:24 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Use multi-stage builds to reduce runtime image size https://review.opendev.org/c/openstack/kolla/+/969968 | 11:26 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Use multi-stage builds to reduce runtime image size https://review.opendev.org/c/openstack/kolla/+/969968 | 11:33 |
| mnasiadka | frickler: I can split it to smaller ones, but I don’t know if it makes sense | 12:52 |
| mnasiadka | mnasiadka bbezak frickler kevko mmalchuk gkoper jovial mattcrees dougszu darmach pabloclsn ravlew amir58118 r-krcek blanson[m] - meeting in 7 minutes | 12:53 |
| mnasiadka | https://review.opendev.org/c/openstack/kolla/+/993961 needs a second review | 12:56 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: kolla-build: fix build-date LABEL invalidating Docker layer cache https://review.opendev.org/c/openstack/kolla/+/993960 | 12:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: kolla-build: fix build-date LABEL invalidating Docker layer cache https://review.opendev.org/c/openstack/kolla/+/993960 | 13:00 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Jun 24 13:00:37 2026 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:00 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: kolla-build: normalize mtime in archive creation https://review.opendev.org/c/openstack/kolla/+/993961 | 13:00 |
| enick_604 | o/ | 13:00 |
| bbezak | o/ | 13:00 |
| jovial | o/ | 13:01 |
| jph | o/ | 13:01 |
| blanson[m] | hello | 13:01 |
| frickler | \o | 13:02 |
| mnasiadka | #topic agenda | 13:05 |
| mnasiadka | #topic * CI status | 13:06 |
| mnasiadka | argh | 13:06 |
| mnasiadka | #topic agenda | 13:06 |
| mnasiadka | * CI status | 13:06 |
| mnasiadka | * Release tasks | 13:06 |
| mnasiadka | * Current cycle planning | 13:06 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:06 |
| mnasiadka | * Open discussion | 13:06 |
| mnasiadka | #topic CI status | 13:06 |
| mnasiadka | So, CI is not that bad, but Ironic is busted due to openstacksdk vs openstack.cloud collection woes | 13:06 |
| jovial | ^ What failures do you see for that? | 13:07 |
| mnasiadka | openstacksdk.version is missing | 13:08 |
| mnasiadka | Which was dropped from newest openstacksdk | 13:08 |
| mnasiadka | And then I released openstack.cloud 2.6.0 to fix that | 13:08 |
| mnasiadka | So we need to bump the collection | 13:08 |
| blanson[m] | i was gonna say I think 2.6.0 fixes it | 13:08 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 13:09 |
| mnasiadka | Ok then, let’s go on. | 13:10 |
| mnasiadka | #topic Release tasks | 13:10 |
| mnasiadka | It’s R-14 week | 13:10 |
| mnasiadka | In R-17 we switch images to current release | 13:10 |
| mnasiadka | #link https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-17-switch-source-images-to-current-release | 13:10 |
| mnasiadka | But I think that’s done already | 13:10 |
| mnasiadka | yup | 13:11 |
| mnasiadka | #topic Current cycle planning | 13:11 |
| mnasiadka | So - did anybody have any opinions of doing Kolla PTG somewhere after the cycle-trailing projects release date instead of in the regular vPTG week? | 13:12 |
| mnasiadka | (Except the positive opinions we had on the meeting last week) | 13:12 |
| frickler | just make sure we stay clear from xmas/year end | 13:13 |
| blanson[m] | kolla christmas meeting would be funny | 13:13 |
| mnasiadka | I was rather thinking of doing that mid January/beginning of Feb - just like the midcycles for other projects | 13:14 |
| blanson[m] | i think later in january people will most likely be rather busy for a while after xmas break | 13:15 |
| mnasiadka | And that would be probably limited to a 2 hour session with populated planned content (plus another one for Kayobe if they wish to cooperate in that manner) | 13:15 |
| mnasiadka | blanson[m], frickler: Should we have a poll which week is the best? | 13:15 |
| frickler | do you want to plan one for this cycle still or next one? | 13:16 |
| blanson[m] | good idea | 13:16 |
| jovial | I think it sounds like a good idea | 13:16 |
| bbezak | Maybe some people would be confused that there is PTG but where are the kolla guys (not that there were many of external contributors) - but if it is well communicated I think this is good idea | 13:16 |
| mnasiadka | frickler: Right, I don’t think we need one for this cycle (which would mean July) - we already had one in the regular slot | 13:17 |
| mnasiadka | So I’m thinking about 2027.1 PTG | 13:18 |
| mnasiadka | (Because Hibiscus is 2026.2) | 13:18 |
| frickler | agreed. but next cycle is still too far away IMO, better ask in like 3 months? | 13:18 |
| mnasiadka | Sure, let’s discuss that around oct | 13:18 |
| mnasiadka | But I assume the decision is we’re not submitting Kolla for the PTG - and I’ll send an email around pre-PTG week that Kolla is not meeting on the regular PTG | 13:19 |
| frickler | we could still make an 1h operator feedback session at the PTG, then? | 13:19 |
| mnasiadka | Oh, that probably makes sense | 13:19 |
| mnasiadka | Thanks frickler, I’ll make that happen | 13:19 |
| mnasiadka | Ok, so - I’ll submit Kolla as a project on PTG, but we’ll only have operator feedback session - and then for the regular developer session we will meet around R-10 or later | 13:20 |
| mnasiadka | Let’s move on to non-organisational things :) | 13:21 |
| mnasiadka | Anybody has any feature that they would like to discuss? | 13:21 |
| mnasiadka | Like patches that need reviewing? | 13:22 |
| mnasiadka | Or maybe I’ll start | 13:22 |
| mnasiadka | I did a bit of checking why our cache in Kolla-build is so bad - so I ended up with these two patches: | 13:22 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/993960 | 13:22 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/993961 | 13:22 |
| mnasiadka | I’m unsure if we want to backport them though | 13:23 |
| frickler | likely not I'd think | 13:24 |
| mnasiadka | Especially that 993960 touches all Dockerfiles | 13:24 |
| mnasiadka | Then I won’t even try ;-) | 13:25 |
| jovial | Very nice. How much time do we save for a build of all images? | 13:25 |
| blanson[m] | these look nice. no backport tho imo | 13:25 |
| mnasiadka | jovial: I haven’t compared —nocache vs —cache, but building like single image that has it’s all earlier layers/parents cached is super fast with these | 13:26 |
| mnasiadka | Well, 993960 only fixes the ,,next calendar day’’ cache invalidation | 13:26 |
| mnasiadka | While 993961 fixes all the plugins and additional dirs | 13:27 |
| frickler | also don't say "docker" when it can be podman, too? ;) | 13:27 |
| mnasiadka | It can, I just never test podman, so I’m biased ;-) | 13:27 |
| mnasiadka | I’ll reword it after the meeting | 13:27 |
| jovial | I see. Sounds useful for iteration time for development :) | 13:27 |
| blanson[m] | speaking of | 13:27 |
| blanson[m] | #link https://review.opendev.org/c/openstack/kolla-ansible/+/993423 | 13:28 |
| mnasiadka | Is Gerrit slow just for me? | 13:28 |
| blanson[m] | getting 500s aswell | 13:28 |
| mnasiadka | Oh boy | 13:28 |
| mnasiadka | blanson[m]: so what is this patch about if you remember? :D | 13:29 |
| blanson[m] | essentially for people who can't load it, there seems to be a bug in the podman docker-compat api | 13:29 |
| mnasiadka | Another one? Seriously? ;-) | 13:30 |
| mnasiadka | Now it opens | 13:30 |
| blanson[m] | we try to convert some result to utf8, but sometimes stdout capture from podman returns raw bytes | 13:30 |
| blanson[m] | seems to affect containers that crashed and stuff like that | 13:30 |
| blanson[m] | we probbly don't catch it in CI cause all the containers are brand new | 13:31 |
| mnasiadka | blanson[m]: not that I’m picky, but maybe a unit test would help to make sure it works? | 13:31 |
| blanson[m] | true | 13:31 |
| blanson[m] | I'll update my review to as kfor unit tests | 13:31 |
| blanson[m] | or write them if the person doesn't respond | 13:31 |
| mnasiadka | Great, thanks | 13:32 |
| mnasiadka | That reminds me I need to go back to the podman rslave thing | 13:32 |
| blanson[m] | but yh podman is infinite fun | 13:32 |
| blanson[m] | every day you get a new fun bug | 13:32 |
| mnasiadka | #link https://review.opendev.org/c/openstack/ansible-collection-kolla/+/993903 | 13:32 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: UnicodeDecodeError in PodmanWorker.compare_config https://review.opendev.org/c/openstack/kolla-ansible/+/993423 | 13:33 |
| mnasiadka | Ok then, podman front is interesting, that I can’t deny - but I don’t know if I’m comfortable enough to run it :) | 13:33 |
| mnasiadka | So, there’s the multi stage building effort that I dusted off a bit | 13:34 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla/+/969968 | 13:34 |
| mnasiadka | The zuul reports should be soon in - so we can continue the review process in Gerrit | 13:34 |
| blanson[m] | I have this on my list | 13:35 |
| mnasiadka | great | 13:35 |
| blanson[m] | will try to review by the end of week | 13:35 |
| blanson[m] | (friday that is, not sunday) | 13:35 |
| mnasiadka | I didn’t think you want to review anything on Sunday :) | 13:35 |
| blanson[m] | well, sometimes it's reviews or dishes | 13:36 |
| blanson[m] | so I review | 13:36 |
| mnasiadka | Ok, I assume that’s a lot to review and move forwards | 13:36 |
| mnasiadka | Ah, forgot to mention - I’m off next week (bbezak is off for next two) | 13:36 |
| mnasiadka | I’m going to cancel the next weekly meeting | 13:36 |
| mnasiadka | Unless anybody feels like running it | 13:36 |
| frickler | cancel good | 13:37 |
| mnasiadka | If anybody does - reach out to me until tomorrow morning, if I don’t see anybody so persistent - I’ll cancel :) | 13:37 |
| mnasiadka | Let’s go to the whiteboard agenda | 13:38 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:38 |
| mnasiadka | (jovial): Using pacemaker for nova-compute-ironic failover | 13:39 |
| mnasiadka | https://docs.openstack.org/nova/2026.1/admin/configuration/hypervisor-ironic.html#scaling-and-performance-issues | 13:39 |
| mnasiadka | The nova-compute process using the Ironic driver can be moved between different physical servers using active/passive failover. But when doing this failover, you must ensure host is the same no matter where the nova-compute process is running. Similarly you must ensure there are at most one nova-compute processes running for each conductor group. | 13:39 |
| mnasiadka | IDEA: Use pacemaker cluster from kolla-ansible: https://github.com/openstack/kolla-ansible/tree/9847a8b75cf7cc4d18cab7adb72b2f9ae87a28c8/ansible/roles/hacluster | 13:39 |
| mnasiadka | Opt in, disabled by default | 13:39 |
| mnasiadka | Register nova-compute-ironic as systemd pacemaker resource . Do not start or enable via systemd manually. | 13:39 |
| mnasiadka | Each conductor group should map to a resource group in pacemaker | 13:40 |
| mnasiadka | Question: | 13:40 |
| mnasiadka | Are there any other better options? | 13:40 |
| mnasiadka | If it was contributed would it be accepted? | 13:40 |
| mnasiadka | So, I’d like to gather opinions from other cores, before I chime in, because I might be biased... | 13:40 |
| * frickler doesn't know or care about nova-compute-ironic so far | 13:41 | |
| jovial | Just sounding out how sensible of an idea this is really :) | 13:41 |
| mnasiadka | Ok, let’s start with some background - Kolla is using hacluster only for heartbeating hypervisors for Masakari, and I don’t believe it works well | 13:41 |
| mnasiadka | We never configured any service on the hosts that are not running containers | 13:42 |
| mnasiadka | I understand the proposal, is to get in the clustering software business and configure that, so that we can overcome problems in nova-compute-ironic - so basically coming to the same discussion we had multiple times in the past? | 13:42 |
| mnasiadka | * and by service running containers I mean systemd service doing docker run or podman run | 13:43 |
| mnasiadka | Or even docker start/podman start | 13:43 |
| bbezak | My experience with pacemaker is a bit peculiar - i.e. it needed proper configuration of tie breaking/stonith, and it was picky - maybe nowadays it is better. - that said for somewhat “simple” service like nova compute ironic it may work. However it would need to support multiple nova compute ironies - as kolla does, and it may be more complicated | 13:44 |
| mnasiadka | https://review.opendev.org/c/openstack/ansible-collection-kolla/+/993903 might be the first exception, but that’s a different story | 13:44 |
| jovial | Its definitely been brought up before. But I wasn't sure if we reached a solid conclusion. | 13:44 |
| mnasiadka | Well, I remember proposals running that in k3s on the controllers and maybe a pacemaker one | 13:45 |
| jovial | Alternatively, could we introduce a flag so that kolla won't manage the lifecycle of nova-compute-ironic? This would allow an operator to develop their own solution e.g configure pacemaker outside of kolla | 13:46 |
| mnasiadka | As in enable_nova_compute_ironic: false ? | 13:46 |
| jovial | Well, I was thinking it would still template out the systemd file and pull the container, but you could just use pacemaker to start it on one host | 13:47 |
| jovial | so everything but systemctl --start --enable | 13:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 13:48 |
| blanson[m] | I understand why this would be nice, I'm not sure I like the pacemaker approach tho | 13:48 |
| jovial | blanson[m], also open to other ideas if there is a better way | 13:49 |
| blanson[m] | yh I'm trying to think about it cause I frankly don't have one right now, but this seems like using a bazooka to kill a bug to me | 13:50 |
| blanson[m] | not sure what else could be done but this add imo a lot of complexity (the whole ha resource management + clustering engine) just to not start 2 systemd services at the same time | 13:50 |
| mnasiadka | Maybe there’s a way to just have the systemd service running on the host where keepalived VIP is MASTER | 13:51 |
| blanson[m] | I'll give it a thought, this is an interesting problem | 13:51 |
| bbezak | Keepalived has support for external scripts | 13:51 |
| mnasiadka | I was rather thinking about systemd pre script and a watchdog | 13:52 |
| mnasiadka | But triggering that from keepalived is not bad | 13:52 |
| jovial | Interesting idea... | 13:52 |
| blanson[m] | there could be a mix of pre-script and the statechange script thingy in keepalived ? | 13:52 |
| mnasiadka | Yeah, and a watchdog that would stop it from starting (e.g. on a controller that has crashed and rebooted or something like that) | 13:53 |
| mnasiadka | Well, that is the pre script | 13:53 |
| jovial | I guess the keepalived might not work if you have controllers where keepalived does run i.e network node split? | 13:53 |
| jovial | does not* | 13:53 |
| bbezak | Or dedicated ironic controller nodes etc. | 13:54 |
| mnasiadka | I never understood why we run keepalived and haproxy on network nodes | 13:54 |
| bbezak | Because it seems like network :) | 13:54 |
| blanson[m] | we changed the inventory and did that anyways bbezak for what it's worth | 13:54 |
| blanson[m] | dedicated ironic stuff for running more conductors | 13:54 |
| blanson[m] | wonder if this could help | 13:55 |
| mnasiadka | Well, we could introduce some small raft-based clustering solution for that, like serf/etcd/sqlite | 13:55 |
| blanson[m] | (for this nova-compute-ironic thing) | 13:55 |
| mnasiadka | Well, maybe not etcd, since that’s not that small | 13:55 |
| mnasiadka | But it would be independent from keepalived locality | 13:55 |
| blanson[m] | there was an interesting project I saw a couple of months ago | 13:56 |
| blanson[m] | about small raft stuff | 13:56 |
| mnasiadka | rqlite is sqlite with raft, that’s like 20MB | 13:56 |
| mnasiadka | And MIT license | 13:57 |
| blanson[m] | I don't know what that is butI shall look it up | 13:57 |
| mnasiadka | And it has an API where you can look up if local node is the leader | 13:57 |
| bbezak | Aka clustering is hard | 13:58 |
| mnasiadka | And then some systemd pre script magic (and a watchdog that would stop if the current node stops being the leader) | 13:58 |
| mnasiadka | And that’s sort of pacemaker | 13:58 |
| bbezak | yeap | 13:58 |
| blanson[m] | next year on openinfra summit: "we built pacemaker but it's sqlite" | 13:59 |
| mnasiadka | Without all the crmd/stonith/corosync | 13:59 |
| jovial | Would we basically need a rqlite cluster per conductor group? | 13:59 |
| mnasiadka | I have no clue, but probably it’s easy to find out, and with it’s size I don’t see it being problematic to run multiple of them? | 14:00 |
| mnasiadka | I’d say worthy of a PoC | 14:00 |
| jovial | One other interesting thing I became aware of. Is this spec: https://blueprints.launchpad.net/nova/+spec/nova-services-graceful-shutdown | 14:01 |
| jovial | But essentially I think that could mean that we need to wait for all builds in progress to finish i.e nova-compute-ironic fully drained | 14:02 |
| jovial | before starting the new one | 14:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 14:02 |
| mnasiadka | jovial: that problem needs to be solved irrespective of what you use for ,,clustering'' | 14:03 |
| jovial | Apparently it gets a bit unhappy with operations in flight at the moment and will mark your instance as in the ERROR state | 14:03 |
| jovial | I was more thinking that we need track when nova-compute-ironic has fully shutdown before starting a new one (potentially) | 14:04 |
| mnasiadka | It seems something has been implemented | 14:04 |
| mnasiadka | #link https://review.opendev.org/q/topic:%22bp/nova-services-graceful-shutdown-part1%22 | 14:05 |
| mnasiadka | Anyway, we’re crossing a boundary of a short discussion | 14:05 |
| mnasiadka | jovial: are you going to work on this? | 14:05 |
| mnasiadka | Oh hell, we’re over time | 14:06 |
| mnasiadka | Let’s continue outside of the meeting | 14:06 |
| mnasiadka | Thank you all for coming! | 14:06 |
| mnasiadka | #endmeeting | 14:06 |
| opendevmeet | Meeting ended Wed Jun 24 14:06:31 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:06 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-06-24-13.00.html | 14:06 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-06-24-13.00.txt | 14:06 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-06-24-13.00.log.html | 14:06 |
| jovial | No plans myself, but there is an interested third party and was hoping to guide them in the right direction. | 14:06 |
| jovial | Thanks all - that was really useful feedback | 14:07 |
| Vii | hi Any thoughts on patch 993788? When do you think we could start switching to it? I'd like to begin resolving the conflicts in kolla-ansible. | 14:07 |
| Vii | config.json to YAML ;) | 14:07 |
| enick_604 | thanks, it was an interesting discussion | 14:07 |
| frickler | enick_604: nick issues? ;) | 14:09 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Modernize HAProxy health check syntax for HAProxy > 2.2 https://review.opendev.org/c/openstack/kolla-ansible/+/994748 | 14:09 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Modernize HAProxy health check syntax for HAProxy > 2.2 https://review.opendev.org/c/openstack/kolla-ansible/+/994748 | 14:10 |
| *** enick_604 is now known as ravlew | 14:15 | |
| ravlew | frickler: forgot to identify ;) | 14:15 |
| frickler | oh, I didn't know that you need to do that as a matrix client, too. interesting | 14:17 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: prometheus-mtail: Switch to a new fork https://review.opendev.org/c/openstack/kolla/+/994750 | 14:21 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Modernize HAProxy health check syntax for HAProxy > 2.2 https://review.opendev.org/c/openstack/kolla-ansible/+/994748 | 14:26 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Modernize HAProxy health check syntax for HAProxy > 2.2 https://review.opendev.org/c/openstack/kolla-ansible/+/994748 | 14:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Pin openstacksdk for ironic jobs to work https://review.opendev.org/c/openstack/kolla-ansible/+/994759 | 15:02 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 15:13 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Use HTTP HEAD for curl-based healthchecks https://review.opendev.org/c/openstack/kolla/+/994765 | 15:15 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Silence health checks and OPTIONS requests in uWSGI logs https://review.opendev.org/c/openstack/kolla-ansible/+/989557 | 15:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Pin openstacksdk for ironic jobs to work https://review.opendev.org/c/openstack/kolla-ansible/+/994759 | 15:45 |
| opendevreview | Ettore Simone proposed openstack/kolla-ansible master: UnicodeDecodeError in PodmanWorker.compare_config https://review.opendev.org/c/openstack/kolla-ansible/+/993423 | 16:14 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Modernize HAProxy health check syntax for HAProxy > 2.2 https://review.opendev.org/c/openstack/kolla-ansible/+/994748 | 16:33 |
| mnasiadka | frickler: seems 994759 works, but that stops is from using u-c - unless we prefer to replace the openstacksdk pin in u-c | 17:03 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Modernize HAProxy health check syntax for HAProxy > 2.2 https://review.opendev.org/c/openstack/kolla-ansible/+/994748 | 17:34 |
| frickler | mnasiadka: we use a similar pin downstream and we want to replace u-c anyway, so I'm fine with that solution if jobs are passing | 18:55 |
| mnasiadka | frickler: I don’t think using u-c in that place is going to help us with anything, especially that we would more mimick what end users are doing with this | 19:36 |
| mnasiadka | frickler: and long term such situations will get better if I’m able to add collections support to existing roles support in Zuul | 19:36 |
| opendevreview | Merged openstack/kolla-ansible master: UnicodeDecodeError in PodmanWorker.compare_config https://review.opendev.org/c/openstack/kolla-ansible/+/993423 | 19:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Install compatible openstack.cloud collection https://review.opendev.org/c/openstack/kolla-ansible/+/994243 | 19:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: ironic: Switch conductor to non-privileged https://review.opendev.org/c/openstack/kolla-ansible/+/994585 | 19:51 |
| blanson[m] | ah crap the unicode decode error patch merged with a typo | 19:58 |
| blanson[m] | I didn't refresh to see you +2'd it | 19:58 |
| blanson[m] | hm | 19:58 |
| opendevreview | Merged openstack/kolla master: prometheus-mtail: Switch to a new fork https://review.opendev.org/c/openstack/kolla/+/994750 | 21:50 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!