JH | Hi all, I was really hoping someone can help me with my Octavia deployment. I have this deployed on openstack using kolla-ansible. I have the lb currently deployed on my internal subnet (10.0.0.144) and it’s in an ACTIVE provisioning state however the status shows ERROR. The health checks are failing with a simple TCP for port 80 healthcheck to the members which are on the same subnet (10.0.0.206) | 05:07 |
---|---|---|
JH | I see the amphora instance is up. It’s on the lb-mgmt-network (10.1.0.144). I noticed if I ssh into one of the members and tried to ping the LB VIP(10.0.0.144) I won’t get a response. Not sure what else I can check. | 05:10 |
JH | Would like to mention that the default security group is used so everything g is pretty much opened. | 05:14 |
opendevreview | Seunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates https://review.opendev.org/c/openstack/octavia/+/930219 | 08:20 |
opendevreview | Seunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates https://review.opendev.org/c/openstack/octavia/+/930219 | 09:46 |
opendevreview | Oleksandr Kozachenko proposed openstack/octavia stable/2023.1: Use cryptography to load PKCS12 certificates https://review.opendev.org/c/openstack/octavia/+/930341 | 10:46 |
opendevreview | Takashi Kajinami proposed openstack/octavia stable/2023.1: Use cryptography to load PKCS12 certificates https://review.opendev.org/c/openstack/octavia/+/930341 | 13:34 |
opendevreview | Takashi Kajinami proposed openstack/octavia master: Add auth options to [neutron] section https://review.opendev.org/c/openstack/octavia/+/930470 | 13:55 |
opendevreview | Takashi Kajinami proposed openstack/octavia master: Add auth options to [neutron] section https://review.opendev.org/c/openstack/octavia/+/930470 | 14:30 |
opendevreview | Merged openstack/octavia master: Fix pep8 with pylint 3.3.0 https://review.opendev.org/c/openstack/octavia/+/930270 | 15:52 |
opendevreview | Seunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates https://review.opendev.org/c/openstack/octavia/+/930219 | 15:56 |
gthiemonge | #startmeeting Octavia | 16:00 |
opendevmeet | Meeting started Wed Sep 25 16:00:12 2024 UTC and is due to finish in 60 minutes. The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
opendevmeet | The meeting name has been set to 'octavia' | 16:00 |
gthiemonge | o/ | 16:00 |
johnsom | o/ | 16:00 |
gthiemonge | #topic Announcements | 16:00 |
gthiemonge | * 2024.2 Dalmatian Release Schedule: R-1 | 16:00 |
gthiemonge | One week before the Dalmatian release!! | 16:00 |
gthiemonge | As mentioned in the previous meetings, Octavia is ready! | 16:01 |
johnsom | Whoot | 16:01 |
gthiemonge | * 2025.1 Epoxy Release Schedule | 16:01 |
gthiemonge | Reminder that the PTG will be held October 21-25 | 16:02 |
gthiemonge | I created an etherpad for the Octavia session | 16:02 |
gthiemonge | https://etherpad.opendev.org/p/oct2024-ptg-octavia | 16:02 |
gthiemonge | if you have topics, don't hesitate to add them there! | 16:02 |
gthiemonge | any other topics? | 16:04 |
gthiemonge | for the announcements? | 16:04 |
johnsom | Nothing I can think of | 16:04 |
gthiemonge | ack | 16:04 |
gthiemonge | #topic CI Status | 16:04 |
gthiemonge | A few patches for CI stability (gates are blocked because of new pylint release) | 16:04 |
gthiemonge | "Fix pep8 with pylint 3.3.0" | 16:04 |
gthiemonge | https://review.opendev.org/c/openstack/octavia-lib/+/930271 | 16:05 |
gthiemonge | https://review.opendev.org/c/openstack/octavia/+/930270 | 16:05 |
gthiemonge | that's an annoying new checker | 16:05 |
gthiemonge | I had to disable it, it reports 91 errors in the octavia code base | 16:05 |
gthiemonge | and not very relevant | 16:06 |
johnsom | Yeah, that one is... questionable | 16:07 |
gthiemonge | #topic Brief progress reports / bugs needing review | 16:08 |
gthiemonge | I updated an old patch for octavia-tempest-plugin | 16:08 |
gthiemonge | https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/773696 | 16:08 |
gthiemonge | "Add a test for custom compute flavors for amphorae" | 16:08 |
opendevreview | Merged openstack/octavia-lib master: Fix pep8 with pylint 3.3.0 https://review.opendev.org/c/openstack/octavia-lib/+/930271 | 16:08 |
gthiemonge | that was quick | 16:08 |
johnsom | I am working on the RBAC changes to the tempest plugin. Going slow as API test run times can be long and I managed to get myself into a retry loop yesterday that made the test runtime over 3 hours. sigh | 16:08 |
johnsom | I have also had downstream distractions this week. | 16:09 |
gthiemonge | I also fixed a small annoying bug in octaviaclient | 16:11 |
gthiemonge | https://review.opendev.org/c/openstack/python-octaviaclient/+/929994 | 16:11 |
gthiemonge | #topic Open Discussion | 16:12 |
gthiemonge | anything else for this week? | 16:12 |
johnsom | I don't have anything | 16:13 |
gthiemonge | ack, then we can close this meeting | 16:13 |
gthiemonge | thank you johnsom | 16:13 |
gthiemonge | #endmeeting | 16:13 |
opendevmeet | Meeting ended Wed Sep 25 16:13:47 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:13 |
opendevmeet | Minutes: https://meetings.opendev.org/meetings/octavia/2024/octavia.2024-09-25-16.00.html | 16:13 |
opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/octavia/2024/octavia.2024-09-25-16.00.txt | 16:13 |
opendevmeet | Log: https://meetings.opendev.org/meetings/octavia/2024/octavia.2024-09-25-16.00.log.html | 16:13 |
noonedeadpunk | folks, have you heard anything about issues with jobboard and zookeeper after 2024.1? | 16:14 |
noonedeadpunk | just checking that it's smth new and unheard before putting time into figuring out wtf is happening | 16:14 |
noonedeadpunk | as start getting plenty of https://paste.openstack.org/show/blmiBLQWlmIfKMLdpx7Q/ | 16:16 |
gthiemonge | noonedeadpunk: no, sorry | 16:16 |
gthiemonge | we don't even have a job with zookeeper | 16:17 |
noonedeadpunk | and indeed - I hardly see active connections in zookeeper. though if only I drop `jobboard_backend_namespace` - it get's instantly created and couple of connections stay for a while... | 16:17 |
noonedeadpunk | yeah, I see. And it's actually taskflow as well, which I'm kinda /o\ about | 16:18 |
gthiemonge | taskflow/zookeeper hasn't been updated for a long time | 16:19 |
gthiemonge | maybe some changes in kazoo? | 16:19 |
gthiemonge | or zookeeper too | 16:19 |
noonedeadpunk | Zookepeer is not smth that is _that_ active. Also see cinder operating normally in there... | 16:20 |
noonedeadpunk | but it's using tooz directly | 16:20 |
noonedeadpunk | and here it somehow feels some threading/eventlet thing to be frank. As obvsiouly it connects, does "thinkgs" and then smth happens | 16:21 |
gthiemonge | maybe you can check whether it's the client or the server that triggers ConnectionLoss | 16:24 |
gthiemonge | do you see any logs displayed by this function: https://github.com/openstack/taskflow/blob/master/taskflow/jobs/backends/impl_zookeeper.py#L741 ? | 16:25 |
noonedeadpunk | it's kinda unclear what closes session... | 16:29 |
noonedeadpunk | https://paste.openstack.org/show/bYhHUT3098tgsoZRwiy1/ | 16:29 |
noonedeadpunk | clients are octavia IPs | 16:29 |
* noonedeadpunk googles now about RequestThrottler | 16:30 | |
gthiemonge | hmm | 16:31 |
noonedeadpunk | nope, I don't see that | 16:31 |
gthiemonge | https://github.com/python-zk/kazoo/blob/c5ab98819b3a797e12a0315e97e51851525da70f/kazoo/client.py#L592 | 16:32 |
gthiemonge | probably comes from ^ | 16:32 |
gthiemonge | _session_callback receives a state, I'm wondering if there could be a new state that is not handled by kazoo | 16:33 |
gthiemonge | noonedeadpunk: which zookeeper version are you using? | 16:34 |
gthiemonge | I need to update https://review.opendev.org/c/openstack/octavia/+/862671 | 16:35 |
noonedeadpunk | 3.8.3 | 16:36 |
gthiemonge | ack, I'll test it in the patch tomorrow | 16:36 |
noonedeadpunk | well, I can get into the code and add some logging there... | 16:36 |
gthiemonge | it woudl be great to have a zk gate | 16:36 |
noonedeadpunk | ++ | 16:38 |
noonedeadpunk | and I think I can rebase the patch as well :) | 16:38 |
opendevreview | Dmitriy Rabotyagov proposed openstack/octavia master: Add zookeeper backend for jobboard in devstack https://review.opendev.org/c/openstack/octavia/+/862671 | 16:44 |
noonedeadpunk | fwiw, in https://github.com/python-zk/kazoo/blob/c5ab98819b3a797e12a0315e97e51851525da70f/kazoo/client.py#L592 state == CLOSED | 16:53 |
* noonedeadpunk returning back to RequestThrottler | 16:53 | |
gthiemonge | https://review.opendev.org/c/openstack/octavia/+/862671/10/devstack/settings we can change the zookeeper version if needed | 17:34 |
opendevreview | Seunghun Lee proposed openstack/octavia master: Fix verification of private CA signed certificates https://review.opendev.org/c/openstack/octavia/+/930219 | 20:49 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!