| opendevreview | Merged openstack/kolla master: rabbitmq: Update to 4.3 https://review.opendev.org/c/openstack/kolla/+/988425 | 06:04 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: copr - stringify part 2 https://review.opendev.org/c/openstack/kolla/+/998266 | 06:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: copr - replace secrets without newlines https://review.opendev.org/c/openstack/kolla/+/998266 | 06:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: CI: copr - replace secrets without newlines https://review.opendev.org/c/openstack/kolla/+/998266 | 06:52 |
| opendevreview | Merged openstack/kolla master: CI: copr - replace secrets without newlines https://review.opendev.org/c/openstack/kolla/+/998266 | 07:24 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2026.1: Add missing reconfigure.yml for manage-containers https://review.opendev.org/c/openstack/kayobe/+/998271 | 07:34 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2026.1: CI: Install kernel-modules-extra https://review.opendev.org/c/openstack/kayobe/+/998273 | 07:41 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.2: CI: Install kernel-modules-extra https://review.opendev.org/c/openstack/kayobe/+/998274 | 07:42 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2025.1: CI: Install kernel-modules-extra https://review.opendev.org/c/openstack/kayobe/+/998275 | 07:42 |
| mnasiadka | bbezak, blanson[m], frickler: I noticed a lot of our CI jobs are having RMQ message timeouts in logs, I think the qos_prefetch of 1 is not really helping anybody - posted https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 08:15 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: rabbitmq: Bump default rabbit_qos_prefetch_count from 1 to 10 https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 08:16 |
| blanson[m] | interesting | 08:17 |
| blanson[m] | we're not using quorum queues cause we use ancient (actual fossiles) versions of openstack and rmq, but the docs seems clear enough that this would help | 08:18 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: rabbitmq: Bump default rabbit_qos_prefetch_count from 1 to 10 https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 08:22 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: Add container state check after restart https://review.opendev.org/c/openstack/kolla-ansible/+/969053 | 08:24 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: Add container state check after restart https://review.opendev.org/c/openstack/kolla-ansible/+/969053 | 08:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: rabbitmq: Bump default rabbit_qos_prefetch_count from 1 to 10 https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 08:26 |
| *** jhorstmann is now known as Guest13924 | 08:47 | |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: rabbitmq: Bump default rabbit_qos_prefetch_count from 1 to 50 https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 08:49 |
| mnasiadka | blanson[m]: actually I think a good real-world production env value should be 50, we can lower that in CI if we see issues (but I doubt) | 08:49 |
| mnasiadka | RMQ recommends it between 50 and 200 | 08:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.20/2.21 https://review.opendev.org/c/openstack/kolla-ansible/+/996745 | 09:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.20/2.21 https://review.opendev.org/c/openstack/kolla-ansible/+/996745 | 09:07 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.20/2.21 https://review.opendev.org/c/openstack/kolla-ansible/+/996745 | 09:08 |
| opendevreview | Merged openstack/kayobe master: Remove leftover device mapper references from the code https://review.opendev.org/c/openstack/kayobe/+/997714 | 09:10 |
| opendevreview | Merged openstack/kolla master: CI: run copr periodic job only on master https://review.opendev.org/c/openstack/kolla/+/997985 | 09:11 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: Bump ansible-core to 2.20/2.21 https://review.opendev.org/c/openstack/kolla-ansible/+/996745 | 09:12 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2026.1: Add missing reconfigure.yml for manage-containers https://review.opendev.org/c/openstack/kayobe/+/998271 | 09:50 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2026.1: Remove leftover device mapper references from the code https://review.opendev.org/c/openstack/kayobe/+/998287 | 09:52 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [WIP] Bump ansible version https://review.opendev.org/c/openstack/kayobe/+/995021 | 09:55 |
| opendevreview | Bartosz Bezak proposed openstack/kolla stable/2026.1: CI: run copr periodic job only on master https://review.opendev.org/c/openstack/kolla/+/998288 | 09:58 |
| opendevreview | Bartosz Bezak proposed openstack/kolla stable/2025.2: CI: run copr periodic job only on master https://review.opendev.org/c/openstack/kolla/+/998289 | 09:58 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [WIP] Bump ansible version https://review.opendev.org/c/openstack/kayobe/+/995021 | 10:13 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2026.1: Remove leftover device mapper references from the code https://review.opendev.org/c/openstack/kayobe/+/998287 | 10:14 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: Add container state check after restart https://review.opendev.org/c/openstack/kolla-ansible/+/969053 | 10:47 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: rabbitmq: Bump default rabbit_qos_prefetch_count from 1 to 50 https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 11:17 |
| opendevreview | Merged openstack/kayobe stable/2026.1: CI: Install kernel-modules-extra https://review.opendev.org/c/openstack/kayobe/+/998273 | 12:04 |
| opendevreview | Verification of a change to openstack/kayobe stable/2025.2 failed: CI: Install kernel-modules-extra https://review.opendev.org/c/openstack/kayobe/+/998274 | 12:04 |
| opendevreview | Kari Häkkinen proposed openstack/kolla-ansible master: cinder: Define backend variables in group_vars/all https://review.opendev.org/c/openstack/kolla-ansible/+/998296 | 12:31 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: neutron: Split shared network agents into role https://review.opendev.org/c/openstack/kolla-ansible/+/975614 | 12:50 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: neutron: Split shared compute agents into role https://review.opendev.org/c/openstack/kolla-ansible/+/997600 | 12:50 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: neutron: Split ML2/OVS agents into role https://review.opendev.org/c/openstack/kolla-ansible/+/997601 | 12:50 |
| opendevreview | Grzegorz Koper proposed openstack/kolla-ansible master: neutron: Split ML2/OVN agents into role https://review.opendev.org/c/openstack/kolla-ansible/+/997602 | 12:50 |
| opendevreview | Kari Häkkinen proposed openstack/kolla-ansible master: cinder: Define backend variables in group_vars/all https://review.opendev.org/c/openstack/kolla-ansible/+/998296 | 12:54 |
| mnasiadka | bbezak frickler kevko mmalchuk gkoper jovial mattcrees dougszu darmach pabloclsn ravlew salmankh amir58118 r-krcek blanson[m] fprzewozn - meeting in 1 minute | 12:59 |
| * fungi is around again for his topic too | 13:00 | |
| Guest13507 | o/ | 13:00 |
| mnasiadka | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Jul 22 13:00:58 2026 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| mnasiadka | #topic rollcall | 13:01 |
| mnasiadka | Guest13507: I assume you’d like to change your nick ;) | 13:01 |
| mnasiadka | o/ | 13:01 |
| frickler | \o | 13:01 |
| bbezak | \o | 13:01 |
| blanson[m] | hello ! | 13:01 |
| fungi | ahoy! | 13:01 |
| owenjones | o/ | 13:02 |
| AlmaMC[m] | Hi | 13:02 |
| mnasiadka | #topic agenda | 13:03 |
| mnasiadka | * Current cycle planning | 13:03 |
| mnasiadka | * Additional agenda (from whiteboard) | 13:03 |
| mnasiadka | * Open discussion | 13:03 |
| mnasiadka | #topic Current cycle planning | 13:03 |
| mnasiadka | Anybody wants to discuss any patches for new features? | 13:04 |
| *** Guest13507 is now known as ravlew | 13:04 | |
| ravlew | \o/ | 13:05 |
| mnasiadka | Seems not | 13:05 |
| mnasiadka | So let’s go to additional agenda | 13:05 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 13:06 |
| mnasiadka | fungi: you first | 13:06 |
| mnasiadka | Bridging the Gap Gazpacho Cycle Follow-Up Discussion | 13:06 |
| fungi | briefly revisiting this topic from last week now that folks have hopefully had time to digest it better, i wanted to see if anyone had additional thoughts | 13:06 |
| fungi | we had takeaway observations noted... contributor survey responses reflected a more negative perception of review activity than are borne out by metrics collected from gerrit | 13:06 |
| fungi | the median time to merge for patches was twice as fast in gazpacho, it could help to understand if there was a shift in review activity that drove this like focusing on smaller fixes or clearer prioritization | 13:07 |
| blanson[m] | median time for kolla projects only or across the board ? I assume kolla ? | 13:07 |
| fungi | er, i had that backwards | 13:07 |
| opendevreview | Seunghun Lee proposed openstack/kolla-ansible stable/2026.1: [stable-only] Split CA of ProxySQL internal TLS and backend TLS https://review.opendev.org/c/openstack/kolla-ansible/+/994258 | 13:07 |
| fungi | blanson[m]: for kolla | 13:07 |
| fungi | i was looking at the wrong tab just now, the *median* time doubled (half as fast) but the mean/average was twice as fast | 13:08 |
| fungi | oh, and that was time to review not time to merge | 13:08 |
| fungi | we saw there was reasonable alignment in priority tracking methods in the survey responses | 13:09 |
| frickler | "Gazpacho Cycle" is the general openstack timeframe, not the shifted kolla one, right? | 13:09 |
| AlmaMC[m] | Is there some metrics on the time to merge ? | 13:09 |
| fungi | frickler: yes, rc1 thursday in flamingo to rc1 thursday in gazpacho, specifically | 13:10 |
| fungi | for the coordinated release | 13:10 |
| fungi | but the statistics were focused on the kolla deliverable repositories | 13:10 |
| fungi | AlmaMC[m]: no that's where i was confused, i forgot we didn't manage to get the mean and median times to merge this round because of the switch from bitergia to lfx insights, we're still working with the insights developers to try to surface those numbers for the next round | 13:11 |
| fungi | we did get review response times, just not merge times yet | 13:11 |
| opendevreview | Merged openstack/kayobe stable/2026.1: Add missing reconfigure.yml for manage-containers https://review.opendev.org/c/openstack/kayobe/+/998271 | 13:11 |
| mnasiadka | fungi: would be useful to get the metrics on cycle-trailing cycle, which is a bit different, and actually majority of the code lands between coordinated release and the cycle trailing release (or at least I think it lands then) | 13:12 |
| fungi | something else slightly anomalous is that the median number of patchsets per review increased from 1 to 2, and the average climbed from 4.5 to 6 | 13:12 |
| fungi | mnasiadka: that's an excellent point, and something i hadn't considered until frickler asked just a minute ago | 13:12 |
| fungi | it's an interesting challenge since the cycle-trailing projects aren't aligned to a common timeframe | 13:13 |
| fungi | there's a deadline, but they don't all release on the deadline they could "end" their work on the cycle any time between the coordinated release and the trailing deadline | 13:13 |
| fungi | also since we're trying to gauge cycle end by when the project's deliverables create their stable branches, we'd need to know somewhat in advance what date that would be for a cycle-trailing project's deliverables | 13:14 |
| mnasiadka | Kolla usually fits within the deadline, but I agree that you would need to measure merge dates in openstack/releases | 13:15 |
| mnasiadka | Which is probably non-trivial | 13:15 |
| opendevreview | Seunghun Lee proposed openstack/kolla-ansible stable/2026.1: [stable-only] Split CA of ProxySQL internal TLS and backend TLS https://review.opendev.org/c/openstack/kolla-ansible/+/994258 | 13:15 |
| fungi | it may be doable though, i'll confer with ildiko and clark | 13:15 |
| fungi | but also that would delay our release of the openstack-wide statistics since they're built on the aggregate of all teams' deliverables too | 13:16 |
| fungi | anyway, it's an excellent callout and i'll see what our options are around that | 13:16 |
| mnasiadka | That’s sort of two sets of statistics released on different dates | 13:16 |
| mnasiadka | Thanks fungi | 13:16 |
| mnasiadka | Anybody has any questions to fungi around Bridging the Gap? | 13:17 |
| fungi | i think the main observation we had is the declining maintainer count, you had 12 active maintainers up through epoxy, it fell to 9 in flamingo and hasn't recovered | 13:18 |
| fungi | i know every team wants to find new maintainers, but obviously identifying volunteers and having time to mentor them aren't trivial | 13:19 |
| mnasiadka | Oh yes, the maintainer count is falling | 13:19 |
| fungi | still might be worthwhile to focus on as a theme for the next cycles | 13:19 |
| fungi | especially with the workload itself showing no signs of slowing | 13:20 |
| fungi | (change volume is steadily increasing over the past several cycles) | 13:20 |
| AlmaMC[m] | How is considered a maintainer active from the stats ? | 13:20 |
| fungi | the number of gerrit accounts leaving code-review -2/+2 or workflow +1 votes in that timeframe | 13:21 |
| fungi | on any of the team's deliverable repositories | 13:21 |
| fungi | obviously for teams that have split/subdivided responsibilities for core reviewer subteams that can be misleading | 13:21 |
| mnasiadka | Oh yes, Kayobe is a different thing amongst all of our repos | 13:22 |
| fungi | like different sets of core reviewers on separate repositories, you're being counted together as maintainers for the team overall | 13:22 |
| fungi | seems like there may be no other questions/suggestions for now? i'll yield the floor, but you know where to find me | 13:24 |
| blanson[m] | thanks fungi | 13:25 |
| mnasiadka | Next one blanson[m] | 13:26 |
| mnasiadka | blanson (2026/07/13) | 13:27 |
| mnasiadka | increasing neutron rpc workers fixed most of the port creation timeouts we had in CI (if not all). How do we scale that properly for users ? | 13:27 |
| mnasiadka | We have encountered this ourselves in production with ovs, once there is sufficient pressure on neutron, workers start "missing" commands and some ports never get created because the creation message is never put into rabbitmq. | 13:27 |
| mnasiadka | Official neutron recommendations are 1 rpc worker per 2 cpu cores (https://docs.openstack.org/neutron/latest/configuration/samples/neutron.html , search "rpc_workers") | 13:27 |
| mnasiadka | Initial idea was: add floor and ceiling values (like 6 and 32, idk), scale to 1 per 2 cores between these values, cap the bottom portion so that small controllers might still have a decent amount, cap the ceiling so that we don't scale this to infinity in case people have absurdly large controllers | 13:27 |
| mnasiadka | use custom values in CI because it doesn't need to be long-term stable ? biggest problems wqith too many workers is memory starvation long term. | 13:27 |
| mnasiadka | documenting this is probbly a good idea aswell. | 13:27 |
| blanson[m] | I was supposed to work on it this week but then our DC went awol so | 13:28 |
| mnasiadka | Ok then | 13:28 |
| blanson[m] | I will send something this week, I'll have to remind myself of what we agreed on, but it didn't seem too complicated to do | 13:28 |
| mnasiadka | I think the rabbit_qos_prefetch thing is sort of related | 13:29 |
| mnasiadka | #link https://review.opendev.org/c/openstack/kolla-ansible/+/998219 | 13:29 |
| mnasiadka | But we should improve our api workers stance | 13:29 |
| mnasiadka | blanson[m]: I just realised neutron has a good calculation manner if you don’t supply rpc and api workers count | 13:30 |
| mnasiadka | Maybe other services have that as well? | 13:30 |
| blanson[m] | werll the thing is that with uwsgi move, api workers aren't managed by nova anymore | 13:31 |
| blanson[m] | nor neutron | 13:31 |
| blanson[m] | nor anybody really | 13:31 |
| blanson[m] | and I'm not sure the wsgi conf has as nice of a default machenism as neutron for rpc_workers | 13:31 |
| mnasiadka | true | 13:32 |
| mnasiadka | What about RPC? | 13:32 |
| blanson[m] | rpc should be fine | 13:32 |
| blanson[m] | idk if nova has the same "smart" defaults | 13:32 |
| blanson[m] | but I can look it up | 13:32 |
| mnasiadka | We would need to analyse it, if not we could maybe do some smarter default than we currently have | 13:32 |
| mnasiadka | And encourage people in the docs to fine tune them by themselves | 13:33 |
| mnasiadka | Anyway, just thoughts | 13:33 |
| mnasiadka | Hopefully you can have more insight next week | 13:33 |
| blanson[m] | nova conductor default to cpu count, scheduler worker just says "read the docs" so idk :D | 13:34 |
| mnasiadka | That’s fantastically universal | 13:34 |
| mnasiadka | Ok, let’s move on | 13:34 |
| mnasiadka | #topic Open discussion | 13:34 |
| mnasiadka | Anybody anything? | 13:34 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: docker: Use ansible_facts for rpm subdir https://review.opendev.org/c/openstack/ansible-collection-kolla/+/998302 | 13:37 |
| blanson[m] | lots of RP+1 review are almost over the line | 13:39 |
| mnasiadka | As in they need a second reviewer? | 13:40 |
| mnasiadka | Or what is the line? | 13:40 |
| blanson[m] | maybe we push a few of them over this week huh ? :D | 13:40 |
| blanson[m] | yh I think there are like 5-8 reviews awaiting a second review | 13:40 |
| blanson[m] | I will try to get some done this queue keeps growing it scares me | 13:40 |
| mnasiadka | Ok, I can have a look | 13:42 |
| mnasiadka | If bbezak or frickler can do the same it would be nice | 13:42 |
| blanson[m] | 9/30 have a +2 already. 5/30 don't pass the gate and 7/30 have not been reviewed or have been vote-reset | 13:43 |
| blanson[m] | I'll get some work done this week hopefully I have time | 13:44 |
| mnasiadka | If something is broken beyond comprehension and there are no updates from author - we should remove the RP vote (there should be a bin icon available) | 13:44 |
| mnasiadka | Ok then, let’s finish for today | 13:46 |
| mnasiadka | Thank you all for coming! | 13:46 |
| mnasiadka | #endmeeting | 13:46 |
| opendevmeet | Meeting ended Wed Jul 22 13:46:36 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:46 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-07-22-13.00.html | 13:46 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-07-22-13.00.txt | 13:46 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2026/kolla.2026-07-22-13.00.log.html | 13:46 |
| blanson[m] | thank you | 13:48 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: docker: Store yum gpg file in the role https://review.opendev.org/c/openstack/ansible-collection-kolla/+/998306 | 13:53 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!