16:00:03 <johnsom> #startmeeting Octavia
16:00:03 <opendevmeet> Meeting started Wed Jul 23 16:00:03 2025 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:03 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:03 <opendevmeet> The meeting name has been set to 'octavia'
16:00:08 <gthiemon1e> o/
16:00:14 <johnsom> Hello everyone!
16:00:48 <johnsom> #topic Announcements
16:01:13 <johnsom> I don't have any announcements this week. Any others?
16:01:35 <gthiemon1e> nothing here
16:01:51 <johnsom> #topic Brief progress reports / bugs needing review
16:02:31 <johnsom> I have been focused on the latest gate breakage. This is another issue related to log offloading and rsyslog in Ubuntu
16:03:04 <johnsom> They are making changes to apparmor for rsyslog and it is breaking us in various ways.
16:03:58 <johnsom> I fixed the issues in the amphora, but now the nodepool instance running devstack is seeing issues with permission denied when rsyslog attempts to create some log files. It's impacting the haproxy package in addition to our logs from what I see.
16:04:29 <johnsom> It's been hard to track down as locally I have not yet reproduced it. Anyway, that has been a good chunk of what I am working on this week.
16:04:38 <johnsom> Aside from downstream work
16:05:27 <gthiemon1e> I proposed a patch that fixes random failures in the unit test
16:05:32 <gthiemon1e> https://review.opendev.org/c/openstack/octavia/+/955265
16:06:09 <fungi> note that there are no more nodepool test nodes in opendev, all test nodes are booted by zuul from images built in zuul jobs, as of the past week
16:06:35 <gthiemon1e> I also opened a launchpad for the client, when a user wants to see their quotas, using the project name doesn't work, they have to pass the ID of the project
16:06:47 <gthiemon1e> (someone reported it here in the channel)
16:06:48 <johnsom> Ah, interesting. I missed that memo. I wonder if that is related given the timing of the issue
16:06:53 <fungi> (this has been a gradual migration off nodepool, and the openstack tenant was left for last but has been underway for about a month-ish)
16:06:59 <gthiemon1e> https://bugs.launchpad.net/octavia/+bug/2117394
16:07:41 <fungi> zuul image build jobs still rely on the same diskimage-builder elements nodepool-builder did, so the images should be essentially the same
16:08:24 <johnsom> Ok, Yeah, this started on the 17th, but like I said earlier, I see in the changelog that the Ubuntu packages are changing the apparmor rules
16:08:48 <fungi> right, changes to ubuntu's packages are a far more likely suspect for something like this
16:09:05 <johnsom> I will continue to track this down. I don't want to just do something in our plugin to fix it and not understand the "Why" part
16:09:06 <fungi> mainly just pointing out that "nodepool instance" is a bit of a misnomer at this point
16:09:27 <fungi> there is no nodepool, only zuul
16:09:28 <johnsom> Cool, thank you!
16:09:33 <johnsom> lol
16:09:49 <fungi> ;)
16:10:15 <gthiemon1e> can we disable apparmor?
16:11:27 <johnsom> We could.... It might be a bit tricky to do on the "test node" instance. Though I'm not 100% sure the current issue is apparmor as I pushed a test that should have moved rsyslog to complain only and it still failed.
16:11:59 <johnsom> In general I don't like just disabling it as I would expect most deployments are going to have it on and run the defaults.
16:13:00 <johnsom> More investigating to do....
16:14:16 <johnsom> My plan is to build a fresh VM from the ground up again today and look at a few things
16:14:36 <fungi> keep in mind that if you disable apparmor in testing, then you may be deviating from how users are actually deploying your software so just hiding the problems they'll trip over
16:14:51 <johnsom> +1
16:15:04 <fungi> probably useful for figuring out the problem, not so much a great long-term strategy
16:16:07 <johnsom> This is one of those "butterfly flapped it's wings in the rain forest and now our tests break" situations
16:17:04 <johnsom> Any other progress updates today?
16:17:22 <gthiemon1e> no
16:17:22 <johnsom> #topic Open Discussion
16:17:31 <johnsom> Any other topics this week?
16:17:41 <gthiemon1e> nothing on my side
16:18:06 <johnsom> Thanks Greg for finding that octavia-status has an oslo config issue. I will poke at that as well.
16:18:34 <gthiemon1e> oh yeah octavia-status is broken, I forgot it
16:18:43 <johnsom> Ok, thanks for another week of finding and smashing bugs! Have a good one.
16:18:50 <gthiemon1e> o/
16:19:02 <johnsom> #endmeeting