Wednesday, 2024-09-25

JHHi all, I was really hoping someone can help me with my Octavia deployment. I have this deployed on openstack using kolla-ansible. I have the lb currently deployed on my internal subnet (10.0.0.144) and it’s in an ACTIVE provisioning state however the status shows ERROR. The health checks are failing with a simple TCP for port 80 healthcheck to the members which are on the same subnet (10.0.0.206)05:07
JHI see the amphora instance is up. It’s on the lb-mgmt-network (10.1.0.144). I noticed if I ssh into one of the members and tried to ping the LB VIP(10.0.0.144) I won’t get a response. Not sure what else I can check. 05:10
JHWould like to mention that the default security group is used so everything g is pretty much opened.05:14
opendevreviewSeunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates  https://review.opendev.org/c/openstack/octavia/+/93021908:20
opendevreviewSeunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates  https://review.opendev.org/c/openstack/octavia/+/93021909:46
opendevreviewOleksandr Kozachenko proposed openstack/octavia stable/2023.1: Use cryptography to load PKCS12 certificates  https://review.opendev.org/c/openstack/octavia/+/93034110:46
opendevreviewTakashi Kajinami proposed openstack/octavia stable/2023.1: Use cryptography to load PKCS12 certificates  https://review.opendev.org/c/openstack/octavia/+/93034113:34
opendevreviewTakashi Kajinami proposed openstack/octavia master: Add auth options to [neutron] section  https://review.opendev.org/c/openstack/octavia/+/93047013:55
opendevreviewTakashi Kajinami proposed openstack/octavia master: Add auth options to [neutron] section  https://review.opendev.org/c/openstack/octavia/+/93047014:30
opendevreviewMerged openstack/octavia master: Fix pep8 with pylint 3.3.0  https://review.opendev.org/c/openstack/octavia/+/93027015:52
opendevreviewSeunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates  https://review.opendev.org/c/openstack/octavia/+/93021915:56
gthiemonge#startmeeting Octavia16:00
opendevmeetMeeting started Wed Sep 25 16:00:12 2024 UTC and is due to finish in 60 minutes.  The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'octavia'16:00
gthiemongeo/16:00
johnsomo/16:00
gthiemonge#topic Announcements16:00
gthiemonge* 2024.2 Dalmatian Release Schedule: R-116:00
gthiemongeOne week before the Dalmatian release!!16:00
gthiemongeAs mentioned in the previous meetings, Octavia is ready!16:01
johnsomWhoot16:01
gthiemonge* 2025.1 Epoxy Release Schedule16:01
gthiemongeReminder that the PTG will be held October 21-2516:02
gthiemongeI created an etherpad for the Octavia session16:02
gthiemongehttps://etherpad.opendev.org/p/oct2024-ptg-octavia16:02
gthiemongeif you have topics, don't hesitate to add them there!16:02
gthiemongeany other topics?16:04
gthiemongefor the announcements?16:04
johnsomNothing I can think of16:04
gthiemongeack16:04
gthiemonge#topic CI Status16:04
gthiemongeA few patches for CI stability (gates are blocked because of new pylint release)16:04
gthiemonge"Fix pep8 with pylint 3.3.0"16:04
gthiemongehttps://review.opendev.org/c/openstack/octavia-lib/+/93027116:05
gthiemongehttps://review.opendev.org/c/openstack/octavia/+/93027016:05
gthiemongethat's an annoying new checker16:05
gthiemongeI had to disable it, it reports 91 errors in the octavia code base16:05
gthiemongeand not very relevant16:06
johnsomYeah, that one is... questionable16:07
gthiemonge#topic Brief progress reports / bugs needing review16:08
gthiemongeI updated an old patch for octavia-tempest-plugin16:08
gthiemongehttps://review.opendev.org/c/openstack/octavia-tempest-plugin/+/77369616:08
gthiemonge"Add a test for custom compute flavors for amphorae"16:08
opendevreviewMerged openstack/octavia-lib master: Fix pep8 with pylint 3.3.0  https://review.opendev.org/c/openstack/octavia-lib/+/93027116:08
gthiemongethat was quick16:08
johnsomI am working on the RBAC changes to the tempest plugin. Going slow as API test run times can be long and I managed to get myself into a retry loop yesterday that made the test runtime over 3 hours. sigh16:08
johnsomI have also had downstream distractions this week.16:09
gthiemongeI also fixed a small annoying bug in octaviaclient16:11
gthiemongehttps://review.opendev.org/c/openstack/python-octaviaclient/+/92999416:11
gthiemonge#topic Open Discussion16:12
gthiemongeanything else for this week?16:12
johnsomI don't have anything16:13
gthiemongeack, then we can close this meeting16:13
gthiemongethank you johnsom 16:13
gthiemonge#endmeeting16:13
opendevmeetMeeting ended Wed Sep 25 16:13:47 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:13
opendevmeetMinutes:        https://meetings.opendev.org/meetings/octavia/2024/octavia.2024-09-25-16.00.html16:13
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/octavia/2024/octavia.2024-09-25-16.00.txt16:13
opendevmeetLog:            https://meetings.opendev.org/meetings/octavia/2024/octavia.2024-09-25-16.00.log.html16:13
noonedeadpunkfolks, have you heard anything about issues with jobboard and zookeeper after 2024.1?16:14
noonedeadpunkjust checking that it's smth new and unheard before putting time into figuring out wtf is happening16:14
noonedeadpunkas start getting plenty of https://paste.openstack.org/show/blmiBLQWlmIfKMLdpx7Q/16:16
gthiemongenoonedeadpunk: no, sorry16:16
gthiemongewe don't even have a job with zookeeper16:17
noonedeadpunkand indeed - I hardly see active connections in zookeeper. though if only I drop `jobboard_backend_namespace` - it get's instantly created and couple of connections stay for a while...16:17
noonedeadpunkyeah, I see. And it's actually taskflow as well, which I'm kinda /o\ about16:18
gthiemongetaskflow/zookeeper hasn't been updated for a long time16:19
gthiemongemaybe some changes in kazoo?16:19
gthiemongeor zookeeper too16:19
noonedeadpunkZookepeer is not smth that is _that_ active. Also see cinder operating normally in there...16:20
noonedeadpunkbut it's using tooz directly16:20
noonedeadpunkand here it somehow feels some threading/eventlet thing to be frank. As obvsiouly it connects, does "thinkgs" and then smth happens16:21
gthiemongemaybe you can check whether it's the client or the server that triggers ConnectionLoss16:24
gthiemongedo you see any logs displayed by this function: https://github.com/openstack/taskflow/blob/master/taskflow/jobs/backends/impl_zookeeper.py#L741 ?16:25
noonedeadpunkit's kinda unclear what closes session...16:29
noonedeadpunkhttps://paste.openstack.org/show/bYhHUT3098tgsoZRwiy1/16:29
noonedeadpunkclients are octavia IPs16:29
* noonedeadpunk googles now about RequestThrottler16:30
gthiemongehmm16:31
noonedeadpunknope, I don't see that16:31
gthiemongehttps://github.com/python-zk/kazoo/blob/c5ab98819b3a797e12a0315e97e51851525da70f/kazoo/client.py#L59216:32
gthiemongeprobably comes from ^16:32
gthiemonge_session_callback receives a state, I'm wondering if there could be a new state that is not handled by kazoo16:33
gthiemongenoonedeadpunk: which zookeeper version are you using?16:34
gthiemongeI need to update https://review.opendev.org/c/openstack/octavia/+/86267116:35
noonedeadpunk3.8.316:36
gthiemongeack, I'll test it in the patch tomorrow16:36
noonedeadpunkwell, I can get into the code and add some logging there...16:36
gthiemongeit woudl be great to have a zk gate16:36
noonedeadpunk++16:38
noonedeadpunkand I think I can rebase the patch as well :)16:38
opendevreviewDmitriy Rabotyagov proposed openstack/octavia master: Add zookeeper backend for jobboard in devstack  https://review.opendev.org/c/openstack/octavia/+/86267116:44
noonedeadpunkfwiw, in https://github.com/python-zk/kazoo/blob/c5ab98819b3a797e12a0315e97e51851525da70f/kazoo/client.py#L592 state  == CLOSED16:53
* noonedeadpunk returning back to RequestThrottler16:53
gthiemongehttps://review.opendev.org/c/openstack/octavia/+/862671/10/devstack/settings we can change the zookeeper version if needed17:34
opendevreviewSeunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates  https://review.opendev.org/c/openstack/octavia/+/93021920:49

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!