| opendevreview | Sergey Kraynev proposed openstack/octavia master: Validate duplicate l7rules https://review.opendev.org/c/openstack/octavia/+/995661 | 07:37 |
|---|---|---|
| skraynev | @gthiemonge hi. one practice question. which maximum size of Loadbalancer have you seen in production Octavia deployment? I mean combinations listeners and members. I practically met some LBs created by k8s and they could containes around 1k listeners. after that healthmanager could not process answer on 1 minute fails. Also such loadbalancer could not process batch update members - because it could continue more that 10 minutes and | 08:32 |
| skraynev | as result it could block one worker process totally during significant period of time | 08:32 |
| gthiemonge | skraynev: TBH I don't know in production. I remember that I make some tests with a lot of listeners and members just to check the limit of the heartbeat messages (it's basically a compressed json payload in a UDP packet, if it exceeds 64kB, the packet is dropped). | 08:38 |
| gthiemonge | skraynev: have you observed messages like https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/health/heartbeat_udp.py#L406-L411 | 08:38 |
| skraynev | @gthiemonge exactly this message | 08:43 |
| gthiemonge | skraynev: there's also this message in DEBUG: https://github.com/openstack/octavia/blob/master/octavia/amphorae/drivers/health/heartbeat_udp.py#L247-L248 | 08:44 |
| gthiemonge | skraynev: I think we need to investigate and reproduce, can you open a launchpad to track it? it would be great to know the average number of pools/hms/members per listener too | 08:46 |
| skraynev | sure. I actually wondering about processing batch update members - it took really much time. May be it's improved in last release, I use pretty old 2023.1 | 08:48 |
| gthiemonge | skraynev: same, open a 2nd launchpad for the batch update, please provide numbers (how many members, how many different subnets in the members,etc...) we already improved it but it was before 2023.1 https://review.opendev.org/c/openstack/octavia/+/854680 | 08:51 |
| skraynev | ok. I will try to provide as much as possible info. thx | 08:52 |
| eduardomorais[m] | gthiemonge and rcruise-redhat can u guys review this PR https://review.opendev.org/c/openstack/octavia/+/1011494, appreciate that. | 12:11 |
| rcruise-redhat | eduardomorais[m] - will do! | 12:29 |
| eduardomorais[m] | thx | 12:31 |
| opendevreview | Richard Cruise proposed openstack/octavia master: Validate L7 policy in L7 rule API requests https://review.opendev.org/c/openstack/octavia/+/1011675 | 15:19 |
| gthiemonge | #startmeeting Octavia | 16:00 |
| opendevmeet | Meeting started Wed Oct 21 16:00:39 2026 UTC and is due to finish in 60 minutes. The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'octavia' | 16:00 |
| gthiemonge | o/ | 16:00 |
| opendevreview | Takashi Kajinami proposed openstack/octavia master: devstack: Do not clean up nova directories https://review.opendev.org/c/openstack/octavia/+/1011687 | 16:03 |
| rcruise-mob | o/ | 16:03 |
| viniciusr | o/ | 16:04 |
| gthiemonge | #topic Announcements | 16:05 |
| gthiemonge | * Indri PTG | 16:05 |
| gthiemonge | Great PTG session last week! | 16:05 |
| gthiemonge | Thanks for the discussions there | 16:06 |
| gthiemonge | The meeting notes are available at | 16:06 |
| gthiemonge | https://etherpad.opendev.org/p/oct2026-ptg-octavia | 16:06 |
| gthiemonge | with the PTG, we officially kicked off the Indri reelase cycle | 16:07 |
| gthiemonge | you'll find the release schedule at | 16:07 |
| gthiemonge | https://releases.openstack.org/indri/schedule.html | 16:07 |
| gthiemonge | we have major releases every 6 months, however if you pay attention to the deadlines, the feature freeze is in only 4 months | 16:08 |
| gthiemonge | we need to keep that in mind | 16:08 |
| gthiemonge | that's all I have for the announcements this week | 16:08 |
| gthiemonge | anything else? | 16:08 |
| rcruise-mob | None from me | 16:09 |
| viniciusr | none from me too | 16:09 |
| gthiemonge | #topic Security bugs/vulnerabilities | 16:10 |
| gthiemonge | just a quick topic on that | 16:10 |
| gthiemonge | we've had a few new security bugs, that were not classified as CVEs/vulnerabilities, but it's important to mention them | 16:11 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2170914 | 16:11 |
| gthiemonge | > L7Policy path-based rules can be bypassed via unnormalized URI (Parser Differential) | 16:11 |
| gthiemonge | so it's related to how haproxy works, it applies the ACL on the raw URLs, so it means that you can bypass the L7Rules by encoding strings | 16:12 |
| gthiemonge | we can't fix it, however we need to update the doc to explain how it works | 16:12 |
| gthiemonge | I also reported a few other bugs (Opus 5.5 found them for me) | 16:14 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2171382 | 16:14 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2171384 | 16:14 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2171385 | 16:14 |
| gthiemonge | they were made public today | 16:14 |
| gthiemonge | we're working on fixes, then backports | 16:14 |
| rcruise-mob | Do we backport to unmaintained branches, or just maintained ones? | 16:15 |
| gthiemonge | IMHO only maintained branches. folks can backport to unmaintained branches if they need, however I cannot approve them (I'm not core on unmaintained) | 16:16 |
| rcruise-mob | Okay, makes sense | 16:17 |
| gthiemonge | that's it for me on this topic, questions? | 16:17 |
| rcruise-mob | No more from me | 16:18 |
| viniciusr | nope | 16:18 |
| gthiemonge | (I don't maintain what is unmaintained ;-) | 16:18 |
| gthiemonge | (or we need to find a new name) | 16:18 |
| gthiemonge | #topic Brief progress reports / bugs needing review | 16:18 |
| gthiemonge | a lot of reviews for me, and these security issues that I mentioned ^^ | 16:19 |
| rcruise-mob | Not much to update from me, mostly helping out with the security issue mentioned above | 16:19 |
| gthiemonge | I also tried to rebase a few interesting changes that were in merge_conflicts, i don't have the list here, but I may request some help for reviewing them | 16:19 |
| shogoyoshioka | Hi. I want to discuss following bug. | 16:20 |
| shogoyoshioka | https://bugs.launchpad.net/octavia/+bug/2171606 | 16:20 |
| shogoyoshioka | Recently, a bug was reported concerning an outage during "loadbalancer failover": https://bugs.launchpad.net/octavia/+bug/2168988 | 16:21 |
| shogoyoshioka | On the other hand, this bug describes an issue in which traffic is dropped in an ACTIVE-STANDBY load balancer when an “Amphora failover” is triggered for the MASTER Amphora holding the VIP address. | 16:21 |
| rcruise-mob | Yep, I filed the second bug, 2268988. From the looks of it, it's the same mechanism at work | 16:22 |
| rcruise-mob | That is, keepalived isn't being cleanly stopped during failover of the Master amphora | 16:23 |
| gthiemonge | I think we identified the same proposed fix: stop keepalived gracefully through the amphora agent API | 16:23 |
| rcruise-mob | So I think they're a duplicate of the same issue, but I'm open to corrections | 16:23 |
| shogoyoshioka | So, they are similar problems, but I believe the issues that need to be solved are different. | 16:24 |
| rcruise-mob | Ah yeah, I'm re-reading your bug and they do seem a bit different | 16:26 |
| shogoyoshioka | Regarding https://bugs.launchpad.net/octavia/+bug/2171606, | 16:26 |
| gthiemonge | (argh sorry network outage) | 16:26 |
| gthiemonge | (i may have lost a bit of the discussion) | 16:27 |
| shogoyoshioka | Regarding https://bugs.launchpad.net/octavia/+bug/2171606, I believe that the outage occurring during an Amphora failover requires implementing logic to hand over the VIP to the BACKUP when the MASTER is deleted, as described in the bug report. | 16:27 |
| rcruise-mob | I feel like there's 2 different problems being described in that bug | 16:28 |
| rcruise-mob | 1. Issues with automatic failover result in the amphora in Error | 16:29 |
| rcruise-mob | 2. Failover of the Master results in a traffic drop while the Backup waits to take over | 16:29 |
| rcruise-mob | I'm not quite following how they're related to each other | 16:30 |
| rcruise-mob | I do agree that a clean stop of keepalived before shutting down the amphora should help with the traffic drop | 16:31 |
| gthiemonge | IMHO they are the same, maybe we need to clarify the description of the first bug, and remove the noise | 16:37 |
| shogoyoshioka | (Looks like my network was down..) | 16:38 |
| rcruise-mob | shogoyoshioka: no problem, I don't think you missed anything. They look to be symptoms of the same issue | 16:39 |
| shogoyoshioka | Let me post my comments again? | 16:39 |
| gthiemonge | we can investigate if the root cause is different, but the LB failover and amphora failover flows are almost the same | 16:39 |
| shogoyoshioka | > gthiemonge Yes! However, I believe the challenges that need to be addressed are different. | 16:40 |
| shogoyoshioka | Regarding https://bugs.launchpad.net/octavia/+bug/2171606, I believe that the outage occurring during an Amphora failover requires implementing logic to hand over the VIP to the BACKUP when the MASTER is deleted, as described in the bug report. | 16:41 |
| shogoyoshioka | Regarding https://bugs.launchpad.net/octavia/+bug/2168988, I believe that the outage during a load balancer failover occurs because the newly rebuilt BACKUP cannot communicate to old MASTER via VRRP and is therefore unable to take over the VIP when the old MASTER is deleted. | 16:41 |
| gthiemonge | ack, i see | 16:41 |
| rcruise-mob | Ah okay, I see the difference | 16:43 |
| gthiemonge | for us, 2168988 was created to hand over the VIP to the BACKUP properly, so what you describe in 2171606 | 16:43 |
| gthiemonge | we will check what happens in 2168988 | 16:44 |
| gthiemonge | if it's the same problem or not, then we will add a comment in the launchpad | 16:44 |
| gthiemonge | let me just copy your words in the launchpad | 16:45 |
| gthiemonge | done | 16:46 |
| shogoyoshioka | Thank you. To add a bit more context, YushoYamaguchi—who submitted the bug report—is a colleague of mine, and we are currently working on the issue together. | 16:46 |
| gthiemonge | ack | 16:47 |
| gthiemonge | shogoyoshioka: do you plan to propose a fix? | 16:47 |
| gthiemonge | 2168988 is assigned to rcruise-mob | 16:47 |
| rcruise-mob | I can test 2171606 to see if it's fixed as well | 16:48 |
| gthiemonge | rcruise-mob: "is fixed"? is there a patch? | 16:49 |
| rcruise-mob | I haven't proposed one, but I'm testing a fix | 16:49 |
| rcruise-mob | Essentially it's a clean shutdown of keepalived on amphora before it is shut down. It should signal the peer to take over the VIP | 16:50 |
| gthiemonge | right | 16:50 |
| shogoyoshioka | Regarding https://bugs.launchpad.net/octavia/+bug/2168988, while no concrete implementation has been reached yet, this bug was filed with the intention of drafting a specification that proposes a solution. | 16:50 |
| gthiemonge | shogoyoshioka: ok let's discuss in the launchpad, I don't want to duplicate the efforts if you're working on a similar solution | 16:51 |
| rcruise-mob | I need to double check the Loadbalancer failover flow to see what happens between the old Master amphora and newly created Backup | 16:51 |
| rcruise-mob | shogoyoshioka: yes, if you have ideas for a solution it would be good to hear them. Let's discuss on Launchpad in the bug | 16:53 |
| gthiemonge | shogoyoshioka: BTW if the time of the meeting is outside your working hours, feel free to ping us during your workday | 16:53 |
| shogoyoshioka | gthiemonge: rcruise-mob: The two issues are similar, yet complex and difficult to explain. I appreciate your cooperation. Thank you so much. | 16:54 |
| rcruise-mob | No problem at all :) | 16:54 |
| gthiemonge | #topic Open Discussion | 16:55 |
| *** raineszm0 is now known as raineszm | 16:55 | |
| gthiemonge | we're almost at the top of the hour, anything else for this meeting? | 16:56 |
| viniciusr | I would like to give a quick overview of my ongoing efforts on BGP-AA feat. I've been working on validating the API (stuff like the lifecycle of peers, speakers, RBAC, flavors), apart from that I still need to validate dedicated distributor in my lab. The complete feature is massive (around 14k insertions, and 3k deletions on my current branch). | 16:56 |
| rcruise-redhat | Wow, that's going to need a lot of reviewing :D | 16:57 |
| gthiemonge | 4 months until feature freeze ;-) | 16:57 |
| gthiemonge | thanks for your efforts viniciusr! | 16:57 |
| viniciusr | hehe, I know you folks are quite busy, from my side I'll try to land it in the next couple weeks :) | 16:58 |
| viniciusr | this is all for me, just a quick sync. thanks for the support! | 16:58 |
| gthiemonge | thanks! | 16:58 |
| gthiemonge | I guess that's it for this week | 16:59 |
| gthiemonge | thank you folks! have a good one! | 16:59 |
| gthiemonge | #endmeeting | 16:59 |
| opendevmeet | Meeting ended Wed Oct 21 16:59:15 2026 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/octavia/2026/octavia.2026-10-21-16.00.html | 16:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/octavia/2026/octavia.2026-10-21-16.00.txt | 16:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/octavia/2026/octavia.2026-10-21-16.00.log.html | 16:59 |
| rcruise-redhat | Thanks everyone! | 16:59 |
| viniciusr | Cheers! | 16:59 |
| shogoyoshioka | Thank you all!! | 17:00 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!