| opendevreview | jaehanbyun proposed openstack/kolla-ansible master: Support OCI digest image references https://review.opendev.org/c/openstack/kolla-ansible/+/997319 | 02:01 |
|---|---|---|
| opendevreview | KWAK JOONGKEE proposed openstack/kolla-ansible master: glance: add enable_cinder_backend_privileged for cinder store https://review.opendev.org/c/openstack/kolla-ansible/+/961235 | 02:40 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: neutron: Add kolla-ansible ovn-migration subcommand https://review.opendev.org/c/openstack/kolla-ansible/+/996096 | 07:02 |
| opendevreview | Owen Jones proposed openstack/kolla-ansible stable/2025.2: Fix nova-compute startup after reprovisioning https://review.opendev.org/c/openstack/kolla-ansible/+/997206 | 08:23 |
| opendevreview | Owen Jones proposed openstack/kolla-ansible stable/2025.1: Fix nova-compute startup after reprovisioning https://review.opendev.org/c/openstack/kolla-ansible/+/997208 | 08:24 |
| opendevreview | Bertrand Lanson proposed openstack/kolla-ansible master: feat: Nova aggregates for compute hosts https://review.opendev.org/c/openstack/kolla-ansible/+/988882 | 08:32 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Centralize service log directory management https://review.opendev.org/c/openstack/kolla-ansible/+/989850 | 08:38 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe-config-dev master: CI: support aarch64 Nova guests https://review.opendev.org/c/openstack/kayobe-config-dev/+/992285 | 08:45 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe master: CI: add Rocky 10 aarch64 jobs https://review.opendev.org/c/openstack/kayobe/+/992288 | 08:49 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla stable/2025.1: Add genisoimage package to nova-compute-ironic https://review.opendev.org/c/openstack/kolla/+/970345 | 08:50 |
| *** jhorstmann is now known as Guest13452 | 08:54 | |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] test aarch64 https://review.opendev.org/c/openstack/kolla-ansible/+/997346 | 08:55 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe master: Add support for QoS egress/ingress settings in systemd-networkd https://review.opendev.org/c/openstack/kayobe/+/997201 | 08:58 |
| opendevreview | Bartosz Bezak proposed openstack/kolla-ansible master: [DNM] test aarch64 https://review.opendev.org/c/openstack/kolla-ansible/+/997346 | 09:11 |
| opendevreview | Will Szumski proposed openstack/kayobe master: CI: Workaround regression in python-openstackclient https://review.opendev.org/c/openstack/kayobe/+/997253 | 09:13 |
| opendevreview | Bartosz Bezak proposed openstack/kayobe-config-dev master: CI: support aarch64 Nova guests https://review.opendev.org/c/openstack/kayobe-config-dev/+/992285 | 09:19 |
| opendevreview | Will Szumski proposed openstack/kayobe master: CI: Workaround regression in python-openstackclient https://review.opendev.org/c/openstack/kayobe/+/997253 | 09:33 |
| opendevreview | Merged openstack/kolla-ansible stable/2026.1: Fix nova-compute startup after reprovisioning https://review.opendev.org/c/openstack/kolla-ansible/+/997205 | 09:58 |
| opendevreview | Rafal Lewandowski proposed openstack/kayobe master: Add support for QoS egress/ingress settings in systemd-networkd https://review.opendev.org/c/openstack/kayobe/+/997201 | 10:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: libvirt: Add role for modular libvirt daemons https://review.opendev.org/c/openstack/kolla-ansible/+/997367 | 10:57 |
| blanson[m] | might not be able to make it to the meeting, lights off in a dc so fun times ahead | 12:10 |
| mnasiadka | blanson[m]: dark alleys in the DC? Isn’t that cool? ;-) | 12:13 |
| blanson[m] | yh logging in to compute nodes with 8 minutes of uptime | 12:13 |
| blanson[m] | loads of fun | 12:13 |
| blanson[m] | and of course rabbitmq partitioned itself | 12:14 |
| blanson[m] | cause he wasn't gonna miss the fun | 12:14 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2026.1: kolla-toolbox: remove leftover kolla-toolbox.json.j2 https://review.opendev.org/c/openstack/kolla-ansible/+/997381 | 12:19 |
| opendevreview | Lukasz Chrustek proposed openstack/kolla-ansible master: Fix mount leak with NFS https://review.opendev.org/c/openstack/kolla-ansible/+/975771 | 12:37 |
| opendevreview | Alex Welsh proposed openstack/kolla-ansible master: Add openstack cacert to uwsgi env for Glance https://review.opendev.org/c/openstack/kolla-ansible/+/996647 | 12:38 |
| mnasiadka | bbezak frickler kevko mmalchuk gkoper jovial mattcrees dougszu darmach pabloclsn ravlew salmankh amir58118 r-krcek blanson[m] - meeting in 6 minutes | 12:54 |
| *** damian___ is now known as LukaszCh | 13:00 | |
| fungi | i'm around for my topic as well | 13:01 |
| mnasiadka | #startmeeting kolla | 13:02 |
| opendevmeet | Meeting started Wed Jul 15 13:02:31 2026 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:02 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:02 |
| mnasiadka | #topic rollcall | 13:02 |
| ravlew | o/ | 13:02 |
| mnasiadka | o/ | 13:02 |
| butjar[mit] | 0/ | 13:02 |
| fungi | ahoy! | 13:02 |
| fprzewozn | o/ | 13:02 |
| bbezak | O/ | 13:02 |
| KurtB | o/ | 13:02 |
| owenjones | o/ | 13:03 |
| salmankh | o/ | 13:04 |
| mnasiadka | #topic agenda | 13:04 |
| mnasiadka | * CI status | 13:05 |
| mnasiadka | * Current cycle planning | 13:05 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:05 |
| mnasiadka | * Open discussion | 13:05 |
| mnasiadka | #topic CI status | 13:05 |
| mnasiadka | No obvious major breakages, apart the python-openstackclient 10.2.0 fallout with requiring cinder | 13:05 |
| mnasiadka | But 10.2.1 is out and in u-c - so should be fine | 13:06 |
| mnasiadka | #topic Current cycle planning | 13:06 |
| mnasiadka | Anybody wants to fill in on any features needing definition of direction? A thorough review? Anything like that? | 13:06 |
| mnasiadka | Ok then, let’s move to additional agenda then | 13:08 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:08 |
| mnasiadka | Let’s start with fungi | 13:08 |
| mnasiadka | (fungi 2026-07-15): Bridging the Gap Gazpacho Cycle Survey and Metrics Analysis | 13:08 |
| mnasiadka | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/3POML6PYPDWQNYM5IFB54O4ASXEPK4Z5/ Bridging the Gap Gazpacho Cycle Retrospective Survey Results | 13:08 |
| mnasiadka | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/CVLQIFEVNNJJ7BJSFZJ4RNZZA7ELQKDS/ Bridging the Gap Gazpacho Cycle Retrospective Metrics Analysis | 13:08 |
| fungi | ildiko posted openstack-wide 2026.1 (gazpacho) cycle retrospective contributor/maintainer survey results and metrics to openstack-discuss recently: | 13:09 |
| fungi | [links above] | 13:09 |
| fungi | i and the other community managers on the openinfra foundation staff have also been digging into team-specific details and i'm doing a round of outreach similar to past cycles, to go over how things may have changed | 13:09 |
| fungi | kolla had 2 maintainers and 0 contributors fill out surveys; additional participants for the next round would, as always, help us draw accurate conclusions | 13:09 |
| fungi | average ratings on the maintainer survey questions were medium-high, with the respondents averaging 3 out of 5 on timeliness of maintainer reviews and the contributor documentation being up to date, and 4 out of 5 on everything else | 13:09 |
| fungi | the common contribution challenge selected by both was getting shallow reviews from other maintainers leading to further revisions, though they also mentioned shifting consensus, inattention, nit-picking, scope creep, and external blockers | 13:09 |
| fungi | when it came to reviewing, both respondents indicated they had trouble getting change owners to follow up on feedback, but additionally cited incomplete changes missing docs/tests and mismatches with team priorities as further challenges | 13:09 |
| fungi | the maintainers also commented that many contributors seem to not be following the team's documented recommendations when it comes to the changes they submit | 13:10 |
| fungi | the new survey questions about ai tools indicate one respondent was using codex both to assist with their review workflow and write documentation as well as speed up repetitive development, the other said they don't use ai for these tasks | 13:10 |
| fungi | as for metrics, active maintainer count held steady in gazpacho, though the number of non-maintainer reviewers fell by 9%; changes opened rose by 16% in gazpacho compared to flamingo and the team closed 91% as many as were opened | 13:10 |
| fungi | we saw the median time to review more than double while the average fell by a third, suggesting maintainers may have become more disciplined in reviewing which is slowing most response times while not letting as many changes linger | 13:10 |
| fungi | an important note about metrics this time: since flamingo we've switched from bitergia to lfx insights as our data source, and while the numbers between them are similar there are some slight differences in how activities are counted | 13:10 |
| fungi | because of this, we re-ran metrics analyses for previous cycles against the new system in order to make sure we weren't trending numbers from different backends and drawing misleading conclusions due to changing measurement methods | 13:11 |
| fungi | anyway, that was a quick dump, i know it's a lot to take in but i didn't want to eat up too much of your meeting, so i'll put this on the agenda again for next week to give everyone time to digest and come up with questions or ideas | 13:11 |
| fungi | one thing we're interested in finding out is whether the team implemented any new contributor or reviewer practices over the course of the last cycle, for example techniques that were discussed in the previous round of analyses | 13:11 |
| fungi | though i'm happy to answer any immediate feedback now too if there's time | 13:11 |
| * fungi wonders if he broke everyone's irc clients | 13:13 | |
| bbezak | :) | 13:14 |
| bbezak | this is useful, thx fungi , will try to digest it | 13:15 |
| fungi | glad i could help! | 13:15 |
| blanson[m] | hello ! can be here in the end sorry to be late | 13:15 |
| mnasiadka | fungi: I’ve been looking at lfx insights - I was thinking of requesting a ‘kolla’ group that would map to all our repos - is that something I should do - or is there a plan to maybe request that based on the governance repo contents? | 13:15 |
| fungi | it's already in the works. they're supposed to be auto-building repository groups for every team based on our governance data | 13:16 |
| fungi | oh, on a related note, gouthamr posted a call for participants in the new contributor experience working group, if anyone's interested in that side of things: | 13:17 |
| fungi | #link https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/R5SW5H2LWSRKT4QFNI4RABMNGNUXHR25/ Contributor Experience Working Group is looking for members | 13:17 |
| mnasiadka | fungi: thanks, so I’ll wait ;-) | 13:17 |
| mnasiadka | I’ll wait two more minutes for questions and move on with other topics | 13:18 |
| fungi | i'm hoping there will be more questions at the next meeting once everyone has time to mull this stuff over | 13:19 |
| mnasiadka | Makes sense :) | 13:19 |
| mnasiadka | I’ll leave that on the whiteboard for next meeting | 13:19 |
| fungi | i know it's a lot to take in | 13:19 |
| opendevreview | Doug Szumski proposed openstack/kolla-ansible master: VAST Manila driver https://review.opendev.org/c/openstack/kolla-ansible/+/959440 | 13:19 |
| mnasiadka | Next topic | 13:20 |
| mnasiadka | blanson (2026/07/13) | 13:20 |
| mnasiadka | increasing neutron rpc workers fixed most of the port creation timeouts we had in CI (if not all). How do we scale that properly for users ? | 13:20 |
| mnasiadka | We have encountered this ourselves in production with ovs, once there is sufficient pressure on neutron, workers start "missing" commands and some ports never get created because the creation message is never put into rabbitmq. | 13:20 |
| mnasiadka | Official neutron recommendations are 1 rpc worker per 2 cpu cores (https://docs.openstack.org/neutron/latest/configuration/samples/neutron.html , search "rpc_workers") | 13:20 |
| mnasiadka | Initial idea was: add floor and ceiling values (like 6 and 32, idk), scale to 1 per 2 cores between these values, cap the bottom portion so that small controllers might still have a decent amount, cap the ceiling so that we don't scale this to infinity in case people have absurdly large controllers | 13:20 |
| mnasiadka | use custom values in CI because it doesn't need to be long-term stable ? biggest problems wqith too many workers is memory starvation long term. | 13:20 |
| mnasiadka | documenting this is probbly a good idea aswell. | 13:20 |
| mnasiadka | blanson[m]: not that I’m trying to divert you from that topic, but did you try raising a bug in Neutron? | 13:20 |
| blanson[m] | we haven't, we assumed it was a scaling issue, but it might be worth doing that | 13:21 |
| blanson[m] | or at least ask | 13:21 |
| mnasiadka | yeah | 13:21 |
| mnasiadka | Because in theory, whatever the pressure is, with even 1 worker it should be slow | 13:21 |
| blanson[m] | I wonder how much pressur tempest puts on the cluster | 13:22 |
| mnasiadka | But not failing | 13:22 |
| blanson[m] | for it to take 5+ minutes to create a port | 13:22 |
| blanson[m] | cause past the "missing mesages", if it takes long enough, everyone just cancels the action and the thing never get created anyway | 13:23 |
| mnasiadka | True, but I don’t think these failures showed up in the past, even after moving to dedicated rpc worker process | 13:23 |
| mnasiadka | Anyway, I think the values we set probably need some revisit | 13:24 |
| mnasiadka | Now we choose a minimum value from the set [ number of cpus, 3] | 13:25 |
| blanson[m] | yh, which turns out to hurt even more the cluster that need it ? | 13:25 |
| mnasiadka | Which is probably fine for the CI, but nowhere near fine for production | 13:25 |
| blanson[m] | cause usually one would assume the more core you put on the control plane the more need you'll have for increased rpc_workers count ? | 13:26 |
| mnasiadka | And we recently see it’s not fine for CI ;) | 13:26 |
| AlmaMC[m] | Does neutron recommend something about that ? | 13:26 |
| mnasiadka | Number of RPC worker processes for service. If not specified, the default is equal to half the number of API workers. | 13:27 |
| mnasiadka | Oh no, that’s some old comment | 13:27 |
| mnasiadka | #link https://docs.openstack.org/neutron/latest/configuration/neutron.html#DEFAULT.rpc_workers | 13:27 |
| mnasiadka | Number of RPC worker processes for service. If not specified, the default is equal to half the CPU count, never using more than half of the system memory. If set to 0, no RPC worker is launched. | 13:27 |
| mnasiadka | blanson[m]: maybe we need to stop setting this? | 13:27 |
| blanson[m] | that could also work ? | 13:28 |
| blanson[m] | I wrote the initial idea I had in the comment above | 13:28 |
| mnasiadka | So default to an empty string and check | length > 0 in the config file | 13:28 |
| mnasiadka | And that should work better | 13:28 |
| mnasiadka | And we allow users to override it easily | 13:28 |
| AlmaMC[m] | #link https://docs.openstack.org/neutron/latest/admin/config-wsgi.html#neutron-worker-processes | 13:28 |
| mnasiadka | AlmaMC[m]: that’s for API | 13:29 |
| mnasiadka | oh, RPC also | 13:29 |
| mnasiadka | sorry | 13:29 |
| mnasiadka | ;-) | 13:29 |
| AlmaMC[m] | :) | 13:29 |
| AlmaMC[m] | "For rpc_workers, there needs to be enough to keep up with incoming events from the various neutron agents. Signs that there are too few can be agent heartbeats arriving late, nova vif bindings timing out on the hypervisors, or rpc message timeout exceptions in agent logs (for example, “broken pipe” errors)." | 13:29 |
| mnasiadka | If OVN ML2 plugin is used without any additional agents, neutron requires no worker for RPC message processing. Set both rpc_workers and rpc_state_report_workers to 0, to disable RPC workers. | 13:29 |
| mnasiadka | That sounds we need to do some magic calculation if we run additional agents | 13:30 |
| mnasiadka | I’d say we should do both, default to empty rpc_workers when ML2/OVS or OVN with additional agents or set to 0 if pure OVN | 13:31 |
| mnasiadka | bbezak: any opinions? | 13:31 |
| blanson[m] | empty or 0 yields the same no ? | 13:33 |
| KurtB | From the neutron.conf doc for rpc_workers: Number of RPC worker processes for service. If not specified, the default is equal to half the CPU count, never using more than half of the system memory. If set to 0, no RPC worker is launched. | 13:34 |
| mnasiadka | blanson[m]: as in empty Ansible variable which results in unset - which results in half the CPU count | 13:34 |
| blanson[m] | hm right | 13:35 |
| bbezak | Seems sane, ovn-agent is not using rpc I assume | 13:35 |
| blanson[m] | cause this one option has a proper 0 handling and doesn't do 0 = default value | 13:35 |
| mnasiadka | Right, I posted patches to switch from neutron-ovn-metadata-agent to neutron-ovn-agent | 13:35 |
| mnasiadka | bbezak: https://review.opendev.org/c/openstack/kolla-ansible/+/996078 | 13:36 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/996078 | 13:36 |
| mnasiadka | blanson[m]: want to work on this further? | 13:38 |
| blanson[m] | your patch ? or rpc_workers ? :D | 13:38 |
| mnasiadka | rpc_workers :D | 13:38 |
| blanson[m] | I can do a proper patch for rpc_workers with what has been discussed here | 13:38 |
| mnasiadka | Leave my patch alone for now :) | 13:38 |
| blanson[m] | unset if ovs, 0 if ovn | 13:40 |
| blanson[m] | bassically ? | 13:40 |
| blanson[m] | basically* | 13:40 |
| mnasiadka | yes | 13:42 |
| blanson[m] | I'll abandon my DNM test and submit a proper chasnge this week maybe weekend depending on how bad our outage turns out to be | 13:42 |
| mnasiadka | Sounds pragmatically simple and stupid | 13:42 |
| mnasiadka | ;) | 13:42 |
| mnasiadka | Wonder what can possibly go wrong | 13:42 |
| mnasiadka | blanson[m]: thanks | 13:43 |
| KurtB | https://docs.openstack.org/neutron/2026.1/admin/config-wsgi.html#neutron-worker-processes Might the relation to api_workers setting have an impact? | 13:44 |
| mnasiadka | Ok, Vii is not on the meeting - so I’ll leave his topics for next week | 13:44 |
| mnasiadka | #topic Open discussion | 13:47 |
| mnasiadka | Anybody anything? | 13:47 |
| fprzewozn | Yeah | 13:47 |
| fprzewozn | https://review.opendev.org/c/openstack/kolla-ansible/+/975771 | 13:47 |
| fprzewozn | It was open some time ago | 13:48 |
| fprzewozn | we've hit the same issue recently | 13:48 |
| fprzewozn | and found out that the fix was almost working | 13:48 |
| fprzewozn | LukaszCh have uploaded patchset and added description there | 13:49 |
| LukaszCh | Yes, Robert was almost there with solution :) | 13:49 |
| fprzewozn | review would be appreciated | 13:50 |
| mnasiadka | Well, that makes sense, nested mounts are some drama | 13:51 |
| blanson[m] | yh this seems like a reasonable patch | 13:52 |
| mnasiadka | Added myself to reviewers | 13:52 |
| blanson[m] | I can take a look but at a glance it looks okay | 13:52 |
| mnasiadka | And RP+1 | 13:52 |
| fprzewozn | thanks | 13:52 |
| mnasiadka | Ok, anybody anything else? | 13:55 |
| mnasiadka | Ok then, thank you all for coming - see you next week! | 13:58 |
| mnasiadka | #endmeeting | 13:58 |
| opendevmeet | Meeting ended Wed Jul 15 13:58:46 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-07-15-13.02.html | 13:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-07-15-13.02.txt | 13:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-07-15-13.02.log.html | 13:58 |
| mnasiadka | blanson[m]: the nova-mnt thing actually reminds me I wanted to write a volume spec validator for nested mounts for kolla_container | 14:07 |
| blanson[m] | yh I'm suprised this even worked | 14:08 |
| blanson[m] | well it didn't | 14:08 |
| blanson[m] | but still | 14:08 |
| blanson[m] | I don't remember having a single successful attempt at doing anything nested mout related | 14:08 |
| blanson[m] | mnasiadka: like something that'd take every single mount and check they don't nest ? | 14:09 |
| mnasiadka | yeah | 14:10 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: kolla_container: Validate nested mounts in volumes https://review.opendev.org/c/openstack/kolla-ansible/+/997400 | 14:19 |
| mnasiadka | blanson[m]: let’s see if it fails on a deployment ^^ | 14:19 |
| opendevreview | Paweł Koniszewski proposed openstack/kolla-ansible master: Avoid docker volume in case of shared /var/lib/nova https://review.opendev.org/c/openstack/kolla-ansible/+/997401 | 14:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: libvirt: Add role for modular libvirt daemons https://review.opendev.org/c/openstack/kolla-ansible/+/997367 | 14:31 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Remove creating logdirs in docker images https://review.opendev.org/c/openstack/kolla/+/989849 | 15:02 |
| opendevreview | Doug Szumski proposed openstack/kolla-ansible master: Support deploying multiple Ironic conductor instances https://review.opendev.org/c/openstack/kolla-ansible/+/995977 | 16:52 |
| opendevreview | Doug Szumski proposed openstack/kolla-ansible master: Support deploying multiple Ironic conductor instances https://review.opendev.org/c/openstack/kolla-ansible/+/995977 | 16:56 |
| opendevreview | Paweł Koniszewski proposed openstack/kolla-ansible master: Avoid docker volume in case of shared /var/lib/nova https://review.opendev.org/c/openstack/kolla-ansible/+/997401 | 18:07 |
| opendevreview | Piotr Milewski proposed openstack/kolla master: Upgrade Prometheus Server to LTS version 3.13.1 https://review.opendev.org/c/openstack/kolla/+/996108 | 18:39 |
| opendevreview | Merged openstack/kayobe master: Adds seed IP to default no-proxy list https://review.opendev.org/c/openstack/kayobe/+/992204 | 21:09 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!