| opendevreview | Merged openstack/kolla master: Pin Lego to v4 https://review.opendev.org/c/openstack/kolla/+/1005192 | 00:32 |
|---|---|---|
| opendevreview | OpenStack Proposal Bot proposed openstack/kolla master: Updated from check-sources https://review.opendev.org/c/openstack/kolla/+/1005479 | 02:54 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.2: CI: Bump kayobe-tox-ansible to one hour https://review.opendev.org/c/openstack/kayobe/+/1005830 | 06:11 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Enable EPEL only when required https://review.opendev.org/c/openstack/kayobe/+/1005831 | 06:59 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Enable EPEL only when required https://review.opendev.org/c/openstack/kayobe/+/1005831 | 07:23 |
| opendevreview | Merged openstack/kolla master: Updated from check-sources https://review.opendev.org/c/openstack/kolla/+/1005479 | 07:48 |
| opendevreview | Merged openstack/kayobe stable/2025.2: CI: Bump kayobe-tox-ansible to one hour https://review.opendev.org/c/openstack/kayobe/+/1005830 | 07:48 |
| opendevreview | Rowan Johns proposed openstack/kayobe master: Move package installs into a single script https://review.opendev.org/c/openstack/kayobe/+/1005194 | 08:06 |
| opendevreview | Rowan Johns proposed openstack/kayobe master: Move package installs into a single script https://review.opendev.org/c/openstack/kayobe/+/1005194 | 08:08 |
| opendevreview | Rowan Johns proposed openstack/kayobe master: Move package installs into a single script https://review.opendev.org/c/openstack/kayobe/+/1005194 | 08:11 |
| opendevreview | Freerk-Ole Zakfeld proposed openstack/kolla-ansible master: Support proxy environment for grafana https://review.opendev.org/c/openstack/kolla-ansible/+/1005848 | 09:08 |
| opendevreview | Freerk-Ole Zakfeld proposed openstack/kolla-ansible master: Support proxy environment for grafana https://review.opendev.org/c/openstack/kolla-ansible/+/1005848 | 09:09 |
| opendevreview | Freerk-Ole Zakfeld proposed openstack/kolla-ansible master: Support proxy environment for grafana https://review.opendev.org/c/openstack/kolla-ansible/+/1005848 | 09:34 |
| opendevreview | Freerk-Ole Zakfeld proposed openstack/kolla-ansible master: Add `letsencrypt_lego_force_active` option https://review.opendev.org/c/openstack/kolla-ansible/+/1002739 | 09:42 |
| opendevreview | Claudia Watson proposed openstack/kolla stable/2026.1: Fix: Zstd missing from ironic conductor container. https://review.opendev.org/c/openstack/kolla/+/1005850 | 09:42 |
| mnasiadka | blanson[m]: it is always fun :) | 11:09 |
| mnasiadka | blanson[m]: willing to have a look at https://review.opendev.org/c/openstack/kolla/+/994460 ? | 11:50 |
| blanson[m] | mnasiadka: sure, there's more comment than code I like these patches :) | 11:51 |
| blanson[m] | LP is absurdly slow for me ? is that a me-problem ? | 11:52 |
| bbezak | (Works OK here - LP) | 11:54 |
| blanson[m] | hmm | 11:57 |
| mnasiadka | Internet is dying :) | 12:00 |
| blanson[m] | I hope not lol, that's the only thing I'm good at | 12:02 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Add support for configuring Dell SONiC switches https://review.opendev.org/c/openstack/kayobe/+/1004574 | 12:05 |
| opendevreview | Claudia Watson proposed openstack/kolla master: Fix: Zstd missing from ironic conductor container. https://review.opendev.org/c/openstack/kolla/+/1005869 | 12:16 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Ensure git cloned dirs are writable for the executing user https://review.opendev.org/c/openstack/kolla/+/994460 | 12:19 |
| opendevreview | Alex Welsh proposed openstack/kolla stable/2026.1: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005871 | 12:36 |
| opendevreview | Alex Welsh proposed openstack/kolla stable/2026.1: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005871 | 12:40 |
| dcapone2004 | I posted the other day about a letsencrypt issue I am facing with KA and certificates not generating. I initially thought the issue was because the LE webserver was bound to the internal API interface, but after spending time combing through documentation I see this is by design and that the requests all filter through and are proxied through horizon / HAProxy which makes sense | 12:41 |
| opendevreview | Alex Welsh proposed openstack/kolla stable/2025.2: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005872 | 12:42 |
| opendevreview | Alex Welsh proposed openstack/kolla stable/2025.1: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005873 | 12:42 |
| dcapone2004 | that caused me to dig into the HAProxy logs and see what is occuring. When manually running the letsencrypt-lego-run.sh script inside of the container, I get an unauthorized 403 unauthorized. Cross checking with the HAProxy logs shows that it is getting a 404 file not found for the URL requested by the LE servers | 12:43 |
| dcapone2004 | is this a bug, or what might be incorrectly configured to cause this? The settings for LE in globals.yml are quite minimal and seemingly self explanatory, so if it is a configuration issue, I am guessing it has to do with some HAProxy settings that needs to match or someting | 12:44 |
| bbezak | dcapone2004: https://review.opendev.org/c/openstack/kolla-ansible/+/1002207 maybe because of that? | 12:47 |
| bbezak | Try setting enable_letsencrypt | 12:47 |
| bbezak | == yes :) | 12:48 |
| dcapone2004 | enable_letsencrypt is set to true and I have a valid email, looking at the report URL now | 12:48 |
| bbezak | Exactly, please set it to “yes” :) | 12:49 |
| bbezak | Current check is only checking statement to == yes | 12:49 |
| bbezak | And not | bool | 12:49 |
| bbezak | So only “yes” will work | 12:50 |
| mnasiadka | bbezak frickler kevko mmalchuk gkoper jovial mattcrees dougszu darmach pabloclsn ravlew salmankh amir58118 r-krcek blanson[m] fprzewozn - meeting in 9 minutes | 12:51 |
| Guest17740 | o/ | 12:52 |
| mnasiadka | In 9 minutes, not now Guest17740 | 12:52 |
| Guest17740 | just checking in :D | 12:53 |
| opendevreview | Jack Hodgkiss proposed openstack/kolla-ansible master: fix: ironic `tftp_server` tracks `ironic_tftp_listen_address` https://review.opendev.org/c/openstack/kolla-ansible/+/1005874 | 12:53 |
| mnasiadka | #startmeeting kolla | 13:01 |
| opendevmeet | Meeting started Wed Sep 16 13:01:00 2026 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:01 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:01 |
| mnasiadka | #topic rollcall | 13:01 |
| mnasiadka | O/ | 13:01 |
| Guest17740 | \o | 13:01 |
| kkaptanoglu1 | o/ | 13:01 |
| bbezak | \o | 13:01 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | #link https://docs.openstack.org/kolla/latest/contributor/meeting.html#agenda | 13:03 |
| *** Guest17740 is now known as ravlew | 13:03 | |
| mnasiadka | #topic Announcements | 13:03 |
| mnasiadka | Just a reminder, that Kolla is not having a regular session during the upcoming PTG - just the operator hour slot will happen at that time | 13:03 |
| mnasiadka | #topic CI status | 13:04 |
| mnasiadka | After fixing the Debian OpenDev mirror - I think we’re fine again | 13:04 |
| mnasiadka | Not counting the jobs that fail every time and we don’t care for them | 13:04 |
| mnasiadka | #topic Release tasks | 13:05 |
| mnasiadka | We should be switching to use stable/2026.2 soon, although there’s no requirements stable/2026.2 yet | 13:05 |
| mnasiadka | #topic Current cycle planning | 13:05 |
| mnasiadka | Anybody anything? | 13:06 |
| KurtB | o/ | 13:06 |
| mnasiadka | I assume we’ve merged Ansible bump, RMQ and other components are in good shape - so we should be good to release? | 13:07 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/2026.1: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005871 | 13:07 |
| ravlew | ansible-bump is merged | 13:07 |
| mnasiadka | Ok then, seems nobody disagrees :) | 13:07 |
| mnasiadka | There are two additional topics on the whiteboard, but people who proposed them are not on the meeting | 13:08 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/2025.2: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005872 | 13:09 |
| fprzewozn | o/ | 13:09 |
| fprzewozn | sorry for being late | 13:09 |
| fprzewozn | just in time I see | 13:09 |
| mnasiadka | Oh yes | 13:09 |
| fprzewozn | so | 13:09 |
| fprzewozn | biggest one: Aetos | 13:09 |
| fprzewozn | got full kolla + kolla-ansible code ready for some time now | 13:10 |
| fprzewozn | https://review.opendev.org/c/openstack/kolla/+/997478 https://review.opendev.org/c/openstack/kolla-ansible/+/997522 | 13:10 |
| opendevreview | Pierre Riteau proposed openstack/kolla stable/2025.1: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005873 | 13:10 |
| fprzewozn | Got it also running on one of customers envs, no issues so far there | 13:10 |
| fprzewozn | then there's a topic you closed mnasiadka before meeting :) https://review.opendev.org/c/openstack/kolla-ansible/+/1003377 | 13:11 |
| fprzewozn | so got both precheck fixes in place, now time for backporting | 13:11 |
| fprzewozn | and third one | 13:11 |
| fprzewozn | ProxySQL, got change of the group mode: https://review.opendev.org/c/openstack/kolla-ansible/+/1005138 | 13:11 |
| fprzewozn | I've got it implemented partially and looks fine for me, no issues just benefits :) | 13:12 |
| fprzewozn | but it would be cool if anyone could test it as well | 13:13 |
| blanson[m] | the proxysql thing is cool thanks ! | 13:13 |
| fprzewozn | and there's an older, but still active topic of mtls control in proxysql https://review.opendev.org/c/openstack/kolla-ansible/+/964299 | 13:13 |
| fprzewozn | and that is where proxysql connects to DB | 13:13 |
| mnasiadka | Proxysql writer is also reader is merging | 13:13 |
| fprzewozn | thanks! | 13:14 |
| fprzewozn | mtls thingy is a bit more complex one | 13:14 |
| fprzewozn | I'm running this patch on multiple production clusters, where internal certs are managed outside K-A | 13:15 |
| tafkamax | hi | 13:15 |
| fprzewozn | and clientAuth for different reasons are not possible for my certs | 13:15 |
| tafkamax | sorry i am late | 13:15 |
| fprzewozn | o/ | 13:15 |
| mnasiadka | mtls reviewed | 13:15 |
| fprzewozn | that was quick | 13:15 |
| fprzewozn | :P | 13:16 |
| mnasiadka | Ok then, tafkamax is next | 13:16 |
| mnasiadka | nova-cell upgrade task with nova_safety_upgrade=true | 13:16 |
| blanson[m] | Aetos is the thing that's supposed to allow replacing gnocchi right ? | 13:16 |
| tafkamax | Yes I have a bug report aswell | 13:16 |
| fprzewozn | blanson[m] yes! and also watcher has deprecated prometheus as data source in 2026.1, aetos is the required one (it's just tenant-aware proxy for metrics that expose it via API) | 13:17 |
| mnasiadka | tafkamax: attach the bug report to 1002530 | 13:17 |
| tafkamax | https://bugs.launchpad.net/kolla-ansible/+bug/2164982 | 13:18 |
| tafkamax | will do | 13:18 |
| mnasiadka | And let’s continue the discussion in Gerrit :) | 13:18 |
| kkaptanoglu | hi, https://review.opendev.org/c/openstack/kolla-ansible/+/1000300 I originally opened this as a bugfix because opensearch_address was removed during the 2025.1 to 2026.1 upgrade. the discussion has now expanded into multiple external opensearch support, which is more of a feature request. I’d prefer to keep this change focused on the original bugfix and handle multiple external opensearch support separately. | 13:18 |
| opendevreview | Taavi Ansper proposed openstack/kolla-ansible master: Fix upgrade with nova_safety_upgrade https://review.opendev.org/c/openstack/kolla-ansible/+/1002530 | 13:19 |
| tafkamax | Linked it | 13:19 |
| mnasiadka | I don’t think we really want to support multiple external opensearch | 13:19 |
| mnasiadka | The existing broken external opensearch support (and external mariadb support) are not tested at all - and nobody really cares about them | 13:20 |
| mnasiadka | So let’s just fix what is broken and leave it be | 13:20 |
| kkaptanoglu | I don't want to either :) | 13:20 |
| kkaptanoglu | I have already fixed broken variable | 13:21 |
| Vii | Hi, I have a couple of really simple fixes - Board | 13:22 |
| mnasiadka | kkaptanoglu: I’ll have a look later this week in your patch | 13:22 |
| Vii | They’re mostly cleanup and optimization changes. | 13:22 |
| mnasiadka | Vii: we can talk about them next week - please add such things before the meeting. | 13:22 |
| kkaptanoglu | mnasiadka: thanks | 13:22 |
| mnasiadka | I don’t accept changing the meeting agenda during the meeting itself. | 13:22 |
| Vii | ok, ty | 13:23 |
| mnasiadka | (I’ll try to review them this or next week anyway - thanks for the list) | 13:23 |
| mnasiadka | Ok then, open discussion | 13:23 |
| mnasiadka | #topic Open discussion | 13:24 |
| mnasiadka | Anybody anything? | 13:24 |
| mnasiadka | Seems not | 13:26 |
| mnasiadka | Thanks for coming and see you next week :) | 13:26 |
| AlmaMC[m] | Nop | 13:26 |
| mnasiadka | #endmeeting | 13:26 |
| opendevmeet | Meeting ended Wed Sep 16 13:26:43 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:26 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-09-16-13.01.html | 13:26 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-09-16-13.01.txt | 13:26 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-09-16-13.01.log.html | 13:26 |
| ravlew | thanks | 13:26 |
| Vii | FYI: VPNaaS is now working. A fix has been implemented in Neutron. | 13:27 |
| dcapone2004 | bbezak: set enable_letencrypt to yes instead of true, ran a reconfigure, still getting a 404. Any more thoughts? | 13:28 |
| opendevreview | Taavi Ansper proposed openstack/kolla-ansible master: Add safe-upgrade scenario to CI https://review.opendev.org/c/openstack/kolla-ansible/+/1002449 | 13:29 |
| dcapone2004 | I did notice that despite running reconfigure, none of my containers appear to have restarted. lego container still shows as up 12 hours (from when it last restarted when I was playing with things yesterday) | 13:29 |
| dcapone2004 | could it be this? https://review.opendev.org/c/openstack/kolla/+/1005192 | 13:30 |
| tafkamax | If you are running from master and not 2026.1? | 13:32 |
| dcapone2004 | I am running from 2026.1 | 13:38 |
| dcapone2004 | stable | 13:38 |
| tafkamax | in the container what is the lego version? | 13:43 |
| dcapone2004 | hmmm, in the deployed letsencrypt_lego container I have lego is not an accepted command. The container uses a script letsencrypt-lego-run.sh that calls /usr/bin/letsencrypt-certificates | 13:51 |
| dcapone2004 | I assumed this was some type of lego wrapper, but I cannot find a way to get a version number from it | 13:52 |
| opendevreview | Merged openstack/kayobe master: CI: Enable EPEL only when required https://review.opendev.org/c/openstack/kayobe/+/1005831 | 13:52 |
| dcapone2004 | I am using ubuntu 24.04 in case OS matters | 13:53 |
| kkaptanoglu | Vii: yes, VPNaaS is running, but i think there's an issue with the neutron-rpc-server's rabbitmq connection. I'll look this in detail on the neutron side | 14:02 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2026.1: CI: Enable EPEL only when required https://review.opendev.org/c/openstack/kayobe/+/1005885 | 14:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.2: CI: Enable EPEL only when required https://review.opendev.org/c/openstack/kayobe/+/1005886 | 14:06 |
| Vii | kkaptanoglu: It’s working now. The issue was fixed by this patch in neutron-vpnaas: | 14:07 |
| Vii | https://github.com/openstack/neutron-vpnaas/commit/2b5a4a674147830d204c505b99b325a2105d5edb | 14:07 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.1: CI: Enable EPEL only when required https://review.opendev.org/c/openstack/kayobe/+/1005887 | 14:08 |
| Vii | I’m currently testing the upgrade to 2026.1, and VPNaaS works without any issues there. | 14:08 |
| Vii | The fix hasn’t been backported to 2025.2, so VPNaaS may still not work there. It works fine on 2025.1, which I’m using continuously. | 14:08 |
| dcapone2004 | I just updated to latest KA in 2026.1@stable and reran reconfigure, no restart to letsencrypt or HAproxy containers, same errors when running /usr/local/bin/letsencrypt-lego-run.sh | 14:22 |
| dcapone2004 | 2026/09/16 09:21:49 [INFO] [sub.domain.com] invalid authorization: acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: 1.xxx.xxxx.xxx2: Invalid response from https://sub.domain.com/.well-known/acme-challenge/tpsSQzCBSEZKUV3wqeqA-d8VKt4nleV4ONurMp18Llo: 404 cron: can't lock /var/run/crond.pid, otherpid may be 114: Resource temporarily unavailable | 14:24 |
| dcapone2004 | urls and IPs are obviously substituted and scrubbed for posting, the actual output matches what is expected | 14:24 |
| dcapone2004 | if anyone has any thoughts, they would be appreciated :-) ... I will likely add a launchpad report if I cannot resolve. Next step is likely going to be trying to set enable_letsencrypt to no, run reconfigure, rm the letsencrypt containers, and then set back to yes and see what happens. However, slightly nervous about doing so as I know KA does not exactly cleanly handle disabling services | 14:28 |
| opendevreview | minwoo seo proposed openstack/kolla master: mariadb: Remove unused replica backup script https://review.opendev.org/c/openstack/kolla/+/1000928 | 14:36 |
| opendevreview | Merged openstack/kolla-ansible master: Drop unset values from kolla_toolbox module_args https://review.opendev.org/c/openstack/kolla-ansible/+/998499 | 14:39 |
| opendevreview | Merged openstack/kolla-ansible master: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1003377 | 14:39 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.1: CI: Bump kayobe-tox-ansible to one hour https://review.opendev.org/c/openstack/kayobe/+/1005906 | 14:58 |
| dcapone2004 | ok, problem resolved. The yes must be explicitly within quotes "yes" for it to properly validate and configure properly right now it seems. Gave this a shot after reading the report more in depth | 15:15 |
| opendevreview | Merged openstack/kolla-ansible master: Fix ProxySQL not using backup writer node for reads https://review.opendev.org/c/openstack/kolla-ansible/+/1005138 | 15:40 |
| opendevreview | Merged openstack/kolla-ansible master: Update enable_letsencrypt to boolean validation https://review.opendev.org/c/openstack/kolla-ansible/+/1002207 | 16:47 |
| opendevreview | Merged openstack/kayobe stable/2025.1: CI: Bump kayobe-tox-ansible to one hour https://review.opendev.org/c/openstack/kayobe/+/1005906 | 16:47 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible stable/2026.1: Update enable_letsencrypt to boolean validation https://review.opendev.org/c/openstack/kolla-ansible/+/1005937 | 18:57 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible stable/2026.1: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1005941 | 19:08 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible stable/2025.2: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1005942 | 19:09 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible stable/2025.1: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1005943 | 19:10 |
| opendevreview | Mohsen Sepandar proposed openstack/kolla-ansible master: Fix gnocchi incoming storage override when redis is enabled https://review.opendev.org/c/openstack/kolla-ansible/+/1005955 | 20:11 |
| opendevreview | Rowan Johns proposed openstack/kayobe master: Move package installs into a single script https://review.opendev.org/c/openstack/kayobe/+/1005194 | 20:52 |
| opendevreview | Verification of a change to openstack/kolla stable/2026.1 failed: grafana: install Prometheus datasource plugin https://review.opendev.org/c/openstack/kolla/+/1005871 | 22:16 |
| opendevreview | Merged openstack/kayobe stable/2026.1: CI: Download cirros images instead of using cache https://review.opendev.org/c/openstack/kayobe/+/1004926 | 23:01 |
| opendevreview | Merged openstack/kolla-ansible stable/2026.1: Update enable_letsencrypt to boolean validation https://review.opendev.org/c/openstack/kolla-ansible/+/1005937 | 23:15 |
| opendevreview | Merged openstack/kolla-ansible stable/2026.1: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1005941 | 23:16 |
| opendevreview | Merged openstack/kolla-ansible stable/2025.2: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1005942 | 23:16 |
| opendevreview | Merged openstack/kolla-ansible stable/2025.1: prometheus: accept vault-encrypted values in precheck https://review.opendev.org/c/openstack/kolla-ansible/+/1005943 | 23:32 |
| blanson[m] | for the ealry risers tomorrow :) https://review.opendev.org/c/openstack/kolla-ansible/+/995006 I took over this one to fix the little things that were missing it should be good now | 23:41 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!