16:00:21 <gthiemonge> #startmeeting Octavia 16:00:21 <opendevmeet> Meeting started Wed Nov 15 16:00:21 2023 UTC and is due to finish in 60 minutes. The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:21 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:21 <opendevmeet> The meeting name has been set to 'octavia' 16:00:28 <gthiemonge> o/ 16:01:51 <oschwart> o/ 16:03:20 <tweining> o/ 16:04:05 <gthiemonge> #topic Announcements 16:04:12 <gthiemonge> * 2024.1 Caracal Release Schedule: Caracal-1 16:04:16 <gthiemonge> this week is Caracal-1 16:04:32 <gthiemonge> patches for new releases of octavia-lib and python-octaviaclient have been proposed 16:04:38 <gthiemonge> https://review.opendev.org/c/openstack/releases/+/900760 16:04:42 <opendevmeet> gthiemonge: Error: Can't start another meeting, one is in progress. Use #endmeeting first. 16:04:45 <gthiemonge> oops 16:04:52 <gthiemonge> https://review.opendev.org/c/openstack/releases/+/900777 16:05:32 <gthiemonge> note: the fix in python-octaviaclient was already included in a bugfix release 16:06:30 <tweining> you mean the hsts "fix" 16:06:44 <gthiemonge> yep 16:08:26 <gthiemonge> #topic CI Status 16:08:37 <gthiemonge> FYI we are facing 2 recurring errors in the CI 16:08:52 <gthiemonge> a DB deadlock (due to the rework for sqlalchemy 2) 16:08:57 <gthiemonge> "Potential deadlock in check_quota_met during the first call of a tenant" 16:09:01 <gthiemonge> https://bugs.launchpad.net/octavia/+bug/2038798 16:09:12 <johnsom> o/ 16:09:14 <gthiemonge> there are 2 patches that should fix it 16:09:20 <gthiemonge> https://review.opendev.org/c/openstack/octavia/+/899663 16:09:32 <gthiemonge> johnsom: \o 16:09:52 <gthiemonge> the 2nd error is: "greenlet.error: cannot switch to a different thread" 16:09:57 <gthiemonge> https://bugs.launchpad.net/octavia/+bug/2039346 16:10:16 <gthiemonge> there's a thread on the ML about eventlet 16:10:22 <gthiemonge> [all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries 16:10:43 <gthiemonge> if we drop eventlet, it will fix our issue :D 16:11:21 <johnsom> But that will be probably D or E release I would expect. 16:11:34 <johnsom> That is a hard turn for a big ship.... 16:12:25 <gthiemonge> johnsom: so maybe we need to report this issue to oslo.messaging 16:12:54 <johnsom> Yeah, I think we might want to pursue fixing oslo messaging 16:13:00 <gthiemonge> johnsom: based on some queries I did in opensearch, it also impacts other projects 16:13:19 <johnsom> Yeah, there are definitely reports from other projects 16:13:28 <gthiemonge> recent reports? 16:15:15 <johnsom> I sent you some links, they start in 2021 and are still open on the eventlet side, but a recent change in oslo messaging seems to have increased it 16:15:24 <johnsom> #link https://github.com/eventlet/eventlet/issues/662 16:15:27 <johnsom> for example 16:15:42 <gthiemonge> oh yeah I saw some old reports but that specific issue was addressed 16:15:43 <johnsom> #link https://github.com/eventlet/eventlet/issues/432 16:16:18 <johnsom> Let me find the oslo messaging patch, give me a minute or two on that 16:16:28 <gthiemonge> becasue that issue reappeared in october 2023 16:17:48 <johnsom> There is this one too: 16:17:51 <johnsom> #link https://github.com/ChameleonCloud/zun/issues/15 16:19:10 <gthiemonge> ack 16:19:15 <johnsom> There is some chatter that this change make the occurrence more often: 16:19:17 <johnsom> #link https://github.com/openstack/oslo.messaging/commit/fd2381c723fe805b17aca1f80bfff4738fbe9628 16:20:42 <johnsom> I think the heartbeat_in_pthread setting should block all import and reference to eventlet in oslo messaging, which is not the case today. 16:20:54 <johnsom> #link https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_utils.py#L25 16:22:00 <gthiemonge> hmm ok 16:22:29 <gthiemonge> unfortunately I don't reproduce it in my env, I ran the tests during 12h, not a single occurrence of the bug 16:23:24 <johnsom> I think it triggers when rabbit has an issue, like a network interruption, but I don't have a reproducer either 16:24:57 <gthiemonge> ok, I will report it to the oslo.messaging launchpad, I'll share the link here 16:25:10 <gthiemonge> maybe my env is not really up-to-date 16:26:06 <johnsom> I saw it in the check gates 16:26:26 <gthiemonge> yeah many times 16:29:07 <gthiemonge> #topic Brief progress reports / bugs needing review 16:29:19 <gthiemonge> there's a new bug reported by skraynev: 16:29:20 <opendevreview> Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 16:29:26 <gthiemonge> https://bugs.launchpad.net/octavia/+bug/2043582 16:29:43 <gthiemonge> basically Octavia rejects/triggers an error when the user creates a listener with a SSL certificate that has an empty CN field 16:30:05 <johnsom> Yeah, we should go back to the standards and see if having no CN at all is valid. I think not. 16:30:38 <johnsom> Well, it's not CN really, it is the subject field. As there are alternatives to CN 16:31:49 <gthiemonge> https://security.stackexchange.com/questions/55414/is-the-common-name-mandatory-for-digital-certificates 16:32:16 <opendevreview> Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 16:33:08 <johnsom> It looks like there are just two RFCs to parse through 16:33:28 <johnsom> I can do some digging on this to confirm/deny 16:33:40 <gthiemonge> johnsom: thanks 16:33:46 <johnsom> That said, I have mentioned before, that cert_parser code is not good 16:35:01 <gthiemonge> yeah I remember that 16:35:43 <tweining> I've also been working on setting up certificates and https://review.opendev.org/c/openstack/octavia/+/901017 is the result of some of the trouble I had 16:36:11 <tweining> one mistake was probably that I looked at octavia's config options around certs instead of refering to the guide 16:36:46 <johnsom> Yeah, focus on the guide, it is the only source of truth. lol 16:37:19 <tweining> anyway, gthiemonge and I had a discussion about the agent_server_* options in https://review.opendev.org/c/openstack/octavia/+/901017/7/octavia/common/config.py 16:38:01 <tweining> it seems the path is already hardcoded in https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/tasks/compute_tasks.py#L195-L198 so changing the config option would break Octavia?! 16:38:42 <gthiemonge> IMHO it's purely internal, admins should not change the setting 16:39:47 <gthiemonge> we should even not mention it in the doc 16:39:50 <johnsom> Correct, the ONLY reason those are even in the config definition is we don't have a separate code base for the agent, so it shares the config infrastructure with the controller side. Those are not configurable by admins 16:40:12 <johnsom> We didn't until the docs were changed and the /etc/octavia.conf was removed. 16:40:26 <gthiemonge> :D 16:40:30 <gthiemonge> https://opendev.org/openstack/octavia/src/branch/master/octavia/opts.py#L30 16:40:33 <tweining> if it really cannot be changed without breaking octavia, we could simply remove it IMO 16:41:02 <gthiemonge> if we remove this line, they will not be included in the doc, but I don't know the other impacts of this change 16:41:45 <tweining> I guess we don't have to decide it now, but we can discuss it in my patch 16:41:58 <gthiemonge> yep 16:42:09 <tweining> the other question I have is about the guide https://docs.openstack.org/octavia/latest/admin/guides/certificates.html 16:42:10 <johnsom> We can break the agent out into it's own repository.... 16:42:42 <gthiemonge> or split config.py into 2 files 16:43:01 <johnsom> oslo cfg is a global 16:43:09 <gthiemonge> argh 16:43:33 <tweining> there are the setting [certificates].ca_certificate and [haproxy_amphora].server_ca that get set to the same path. If it's the same setting, why are there two config options? 16:44:04 <johnsom> See, the conf that get created automatically and dropped in the config drive sets those. the agent has to share the cfg with the controllers, so it's listed here. 16:44:34 <johnsom> What line in the guide tweining? 16:44:53 <gthiemonge> tweining: in devstack it's not the same path 16:44:58 <tweining> Under "Configuring Octavia" 16:45:14 <tweining> point 2 and 4 16:45:30 <gthiemonge> https://opendev.org/openstack/octavia/src/branch/master/devstack/plugin.sh#L432-L437 16:46:10 <tweining> ah, ok. thanks. 16:47:01 <johnsom> Yeah, in dual CA (don't look at deployment tools like tripleo that do it completely wrong) those are different files 16:47:58 <johnsom> Oh, no, those are the same file, just used in different places 16:48:11 <johnsom> Ugh, I haven't looked at this in a bit. 16:48:24 <johnsom> The guide is right 16:48:58 <tweining> I guess it is fine. It's just something I noticed an wondered why it is the same path in the guide. 16:50:16 <johnsom> Right, I remember now, the haproxy_amphora server_ca may need to include intermediate certs, where the usage in [certificates] would not 16:50:34 <tweining> one other thing that surprised me a bit was that the defaults for the cert and key path is in https://opendev.org/openstack/octavia/src/branch/master/octavia/certificates/common/local.py /etc/ssl/certs and not /etc/octavia/certs 16:51:21 <tweining> but in my case I intend to override the default using the environment variable 16:52:19 <tweining> anyway, that is all from me for today I think. 16:52:21 <johnsom> I think those are only used (if they are) in tests 16:52:23 <gthiemonge> no idea on this 16:52:33 <QG> Hi ! i have a question about a patch i did : https://review.opendev.org/c/openstack/octavia/+/898803, 16:52:33 <QG> when I run the unit test alone, I don't reproduce the prb ( octavia.tests.functional.api.v2.test_load_balancer.TestLoadBalancer.test_create_with_vip_subnet_fills_network ) 16:53:02 <johnsom> snakeoil certs are created by the OS at install time. They are bogus self-signed (but valid if you ignore the self signed part) 16:53:30 <johnsom> My wonder is if those globals are even used anywhere 16:53:31 <gthiemonge> QG: you're talking about this error? https://zuul.opendev.org/t/openstack/build/e7e27ff8071140e5a142a5eadd4eee6f 16:54:02 <gthiemonge> QG: I'll take a look 16:54:09 <QG> gthiemonge : Yes ! 16:54:10 <gthiemonge> there are some other weird errors 16:54:22 <gthiemonge> {"faultcode": "Client", "faultstring": "Invalid input for field/attribute vip_network_id. Value: \'<MagicMock name=\'get_network().id\' id=\'139946822340080\'>\'. Value should be UUID format", "debuginfo": null}' 16:54:44 <gthiemonge> I'm surprised that you don't have it in your nev 16:54:48 <gthiemonge> env 16:55:42 <gthiemonge> anyway, I'll run the tests tomorrow morning 16:55:54 <QG> gthiemonge: ok thanks ! 16:56:08 <gthiemonge> folks, do you have any other topics for today? 16:56:11 <oschwart> I wanted to ask quick question about 16:56:13 <oschwart> https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/893066 16:56:23 <oschwart> if it is too late we can talk about it tomorrow 16:56:34 <gthiemonge> you have 4 min oschwart! 16:56:40 <oschwart> haha 16:56:42 <oschwart> :o 16:56:43 <oschwart> After the latest review, I removed the additional config option here, and after the check pipepline failed, I remembered why we had them in the first place :) 16:56:51 <oschwart> as stable branches don't have the noop certificate manager 16:57:08 <gthiemonge> right 16:57:29 <gthiemonge> I remember that 16:57:34 <oschwart> should I revert the patch to have the noop cert option too? 16:57:53 <johnsom> So either backport the noop cert manager or gate the tests on the API version 16:58:36 <gthiemonge> johnsom: you don't agree with this new config option? 16:59:01 <johnsom> As I commented, it's really duplicate 16:59:35 <gthiemonge> FYI I have another meeting in 1min 16:59:47 <oschwart> let's talk about it tomorrow then 16:59:49 <johnsom> It seems we could backport it 16:59:56 <gthiemonge> let's close the meeting, I'll check wiht oschwart 17:00:02 <tweining> +1 17:00:08 <gthiemonge> thank you! 17:00:08 <oschwart> thanks for the input all 17:00:12 <gthiemonge> #endmeeting