| opendevreview | Gregory Thiemonge proposed openstack/octavia stable/wallaby: Fix incorrect removal of IP rules in the amphora https://review.opendev.org/c/openstack/octavia/+/900983 | 08:13 |
|---|---|---|
| skraynev | gthiemon1e: hello. I am wondering, why it's not possible to use SSL cert with empty CN (common name) for HTTPs Terminated listener? | 10:02 |
| skraynev | looks like ha-proxy allows it, but octavia parse CN and fails if it's empty | 10:02 |
| skraynev | I talk about this part of code: https://github.com/openstack/octavia/blob/7310986de9bf68ed86de90de0501a1bc46945526/octavia/common/tls_utils/cert_parser.py#L262-L265 | 10:03 |
| gthiemon1e | skraynev: hmmm no idea | 10:25 |
| gthiemon1e | skraynev: what's the issue? an exception when getting the attributes? | 10:25 |
| skraynev | yeap. I try to use SSL certificate with empty CN field on creation listener and it fails with error: Unreadable Certificate. it happens exactly in the place which I mentioned above, because: `cert.subject.get_attributes_for_oid(x509.OID_COMMON_NAME)` returns empty list | 10:31 |
| skraynev | so I try to figure out the reason, why does code expect cn will be not empty | 10:32 |
| gthiemon1e | the CN is set in the TLSContainer object: https://github.com/openstack/octavia/blob/7310986de9bf68ed86de90de0501a1bc46945526/octavia/common/tls_utils/cert_parser.py#L407 | 10:35 |
| gthiemon1e | but the primary_cn attribute is never used | 10:35 |
| skraynev | yes. I understand, that this case is should be defined, but it also looks like it could be None. for example in old code it had soft validation: https://review.opendev.org/c/openstack/octavia/+/184868/4/octavia/common/tls_utils/cert_parser.py#b106 | 10:42 |
| skraynev | maybe someone from authors of this review could shed a light on the using CN attribute? however I don't see them online (or don't know right nickname :) ) | 10:44 |
| gthiemon1e | wow that's old | 10:58 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify the client_ca and agent_server_ca options https://review.opendev.org/c/openstack/octavia/+/901017 | 11:25 |
| opendevreview | Merged openstack/octavia stable/zed: Fix incorrect removal of IP rules in the amphora https://review.opendev.org/c/openstack/octavia/+/893941 | 11:35 |
| opendevreview | Merged openstack/octavia stable/yoga: Fix incorrect removal of IP rules in the amphora https://review.opendev.org/c/openstack/octavia/+/893942 | 11:39 |
| opendevreview | Merged openstack/octavia stable/xena: Fix incorrect removal of IP rules in the amphora https://review.opendev.org/c/openstack/octavia/+/893943 | 11:39 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 | 11:39 |
| skraynev | @gthiemon1e indeed :) | 11:50 |
| skraynev | gthiemon1e: I put all information to the bug description: https://bugs.launchpad.net/octavia/+bug/2043582. it will be great if someone else take a look on it. | 12:06 |
| gthiemon1e | skraynev: ack, we have our weekly meeting today, we can discuss it | 12:12 |
| skraynev | gthiemon1e: sounds great. thank you. | 12:15 |
| gthiemon1e | skraynev: "that certificate without SSL is not allowed" you mean without CN, right? | 12:20 |
| skraynev | yes | 12:20 |
| skraynev | thank you for catching this misspell | 12:21 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 | 13:13 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 | 13:51 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 | 13:52 |
| opendevreview | Merged openstack/octavia stable/2023.1: Fix incorrect removal of IP rules in the amphora https://review.opendev.org/c/openstack/octavia/+/893780 | 14:56 |
| *** gthiemon1e is now known as gthiemonge | 15:55 | |
| gthiemonge | #startmeeting Octavia | 16:00 |
| opendevmeet | Meeting started Wed Nov 15 16:00:21 2023 UTC and is due to finish in 60 minutes. The chair is gthiemonge. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'octavia' | 16:00 |
| gthiemonge | o/ | 16:00 |
| oschwart | o/ | 16:01 |
| tweining | o/ | 16:03 |
| gthiemonge | #topic Announcements | 16:04 |
| gthiemonge | * 2024.1 Caracal Release Schedule: Caracal-1 | 16:04 |
| gthiemonge | this week is Caracal-1 | 16:04 |
| gthiemonge | patches for new releases of octavia-lib and python-octaviaclient have been proposed | 16:04 |
| gthiemonge | https://review.opendev.org/c/openstack/releases/+/900760 | 16:04 |
| gthiemonge | #startmeeting Octaviahttps://review.opendev.org/c/openstack/releases/+/900777 | 16:04 |
| opendevmeet | gthiemonge: Error: Can't start another meeting, one is in progress. Use #endmeeting first. | 16:04 |
| gthiemonge | oops | 16:04 |
| gthiemonge | https://review.opendev.org/c/openstack/releases/+/900777 | 16:04 |
| gthiemonge | note: the fix in python-octaviaclient was already included in a bugfix release | 16:05 |
| tweining | you mean the hsts "fix" | 16:06 |
| gthiemonge | yep | 16:06 |
| gthiemonge | #topic CI Status | 16:08 |
| gthiemonge | FYI we are facing 2 recurring errors in the CI | 16:08 |
| gthiemonge | a DB deadlock (due to the rework for sqlalchemy 2) | 16:08 |
| gthiemonge | "Potential deadlock in check_quota_met during the first call of a tenant" | 16:08 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2038798 | 16:09 |
| johnsom | o/ | 16:09 |
| gthiemonge | there are 2 patches that should fix it | 16:09 |
| gthiemonge | https://review.opendev.org/c/openstack/octavia/+/899663 | 16:09 |
| gthiemonge | johnsom: \o | 16:09 |
| gthiemonge | the 2nd error is: "greenlet.error: cannot switch to a different thread" | 16:09 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2039346 | 16:09 |
| gthiemonge | there's a thread on the ML about eventlet | 16:10 |
| gthiemonge | [all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries | 16:10 |
| gthiemonge | if we drop eventlet, it will fix our issue :D | 16:10 |
| johnsom | But that will be probably D or E release I would expect. | 16:11 |
| johnsom | That is a hard turn for a big ship.... | 16:11 |
| gthiemonge | johnsom: so maybe we need to report this issue to oslo.messaging | 16:12 |
| johnsom | Yeah, I think we might want to pursue fixing oslo messaging | 16:12 |
| gthiemonge | johnsom: based on some queries I did in opensearch, it also impacts other projects | 16:13 |
| johnsom | Yeah, there are definitely reports from other projects | 16:13 |
| gthiemonge | recent reports? | 16:13 |
| johnsom | I sent you some links, they start in 2021 and are still open on the eventlet side, but a recent change in oslo messaging seems to have increased it | 16:15 |
| johnsom | #link https://github.com/eventlet/eventlet/issues/662 | 16:15 |
| johnsom | for example | 16:15 |
| gthiemonge | oh yeah I saw some old reports but that specific issue was addressed | 16:15 |
| johnsom | #link https://github.com/eventlet/eventlet/issues/432 | 16:15 |
| johnsom | Let me find the oslo messaging patch, give me a minute or two on that | 16:16 |
| gthiemonge | becasue that issue reappeared in october 2023 | 16:16 |
| johnsom | There is this one too: | 16:17 |
| johnsom | #link https://github.com/ChameleonCloud/zun/issues/15 | 16:17 |
| gthiemonge | ack | 16:19 |
| johnsom | There is some chatter that this change make the occurrence more often: | 16:19 |
| johnsom | #link https://github.com/openstack/oslo.messaging/commit/fd2381c723fe805b17aca1f80bfff4738fbe9628 | 16:19 |
| johnsom | I think the heartbeat_in_pthread setting should block all import and reference to eventlet in oslo messaging, which is not the case today. | 16:20 |
| johnsom | #link https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_utils.py#L25 | 16:20 |
| gthiemonge | hmm ok | 16:22 |
| gthiemonge | unfortunately I don't reproduce it in my env, I ran the tests during 12h, not a single occurrence of the bug | 16:22 |
| johnsom | I think it triggers when rabbit has an issue, like a network interruption, but I don't have a reproducer either | 16:23 |
| gthiemonge | ok, I will report it to the oslo.messaging launchpad, I'll share the link here | 16:24 |
| gthiemonge | maybe my env is not really up-to-date | 16:25 |
| johnsom | I saw it in the check gates | 16:26 |
| gthiemonge | yeah many times | 16:26 |
| gthiemonge | #topic Brief progress reports / bugs needing review | 16:29 |
| gthiemonge | there's a new bug reported by skraynev: | 16:29 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 | 16:29 |
| gthiemonge | https://bugs.launchpad.net/octavia/+bug/2043582 | 16:29 |
| gthiemonge | basically Octavia rejects/triggers an error when the user creates a listener with a SSL certificate that has an empty CN field | 16:29 |
| johnsom | Yeah, we should go back to the standards and see if having no CN at all is valid. I think not. | 16:30 |
| johnsom | Well, it's not CN really, it is the subject field. As there are alternatives to CN | 16:30 |
| gthiemonge | https://security.stackexchange.com/questions/55414/is-the-common-name-mandatory-for-digital-certificates | 16:31 |
| opendevreview | Tom Weininger proposed openstack/octavia master: Clarify certificate related config options https://review.opendev.org/c/openstack/octavia/+/901017 | 16:32 |
| johnsom | It looks like there are just two RFCs to parse through | 16:33 |
| johnsom | I can do some digging on this to confirm/deny | 16:33 |
| gthiemonge | johnsom: thanks | 16:33 |
| johnsom | That said, I have mentioned before, that cert_parser code is not good | 16:33 |
| gthiemonge | yeah I remember that | 16:35 |
| tweining | I've also been working on setting up certificates and https://review.opendev.org/c/openstack/octavia/+/901017 is the result of some of the trouble I had | 16:35 |
| tweining | one mistake was probably that I looked at octavia's config options around certs instead of refering to the guide | 16:36 |
| johnsom | Yeah, focus on the guide, it is the only source of truth. lol | 16:36 |
| tweining | anyway, gthiemonge and I had a discussion about the agent_server_* options in https://review.opendev.org/c/openstack/octavia/+/901017/7/octavia/common/config.py | 16:37 |
| tweining | it seems the path is already hardcoded in https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/tasks/compute_tasks.py#L195-L198 so changing the config option would break Octavia?! | 16:38 |
| gthiemonge | IMHO it's purely internal, admins should not change the setting | 16:38 |
| gthiemonge | we should even not mention it in the doc | 16:39 |
| johnsom | Correct, the ONLY reason those are even in the config definition is we don't have a separate code base for the agent, so it shares the config infrastructure with the controller side. Those are not configurable by admins | 16:39 |
| johnsom | We didn't until the docs were changed and the /etc/octavia.conf was removed. | 16:40 |
| gthiemonge | :D | 16:40 |
| gthiemonge | https://opendev.org/openstack/octavia/src/branch/master/octavia/opts.py#L30 | 16:40 |
| tweining | if it really cannot be changed without breaking octavia, we could simply remove it IMO | 16:40 |
| gthiemonge | if we remove this line, they will not be included in the doc, but I don't know the other impacts of this change | 16:41 |
| tweining | I guess we don't have to decide it now, but we can discuss it in my patch | 16:41 |
| gthiemonge | yep | 16:41 |
| tweining | the other question I have is about the guide https://docs.openstack.org/octavia/latest/admin/guides/certificates.html | 16:42 |
| johnsom | We can break the agent out into it's own repository.... | 16:42 |
| gthiemonge | or split config.py into 2 files | 16:42 |
| johnsom | oslo cfg is a global | 16:43 |
| gthiemonge | argh | 16:43 |
| tweining | there are the setting [certificates].ca_certificate and [haproxy_amphora].server_ca that get set to the same path. If it's the same setting, why are there two config options? | 16:43 |
| johnsom | See, the conf that get created automatically and dropped in the config drive sets those. the agent has to share the cfg with the controllers, so it's listed here. | 16:44 |
| johnsom | What line in the guide tweining? | 16:44 |
| gthiemonge | tweining: in devstack it's not the same path | 16:44 |
| tweining | Under "Configuring Octavia" | 16:44 |
| tweining | point 2 and 4 | 16:45 |
| gthiemonge | https://opendev.org/openstack/octavia/src/branch/master/devstack/plugin.sh#L432-L437 | 16:45 |
| tweining | ah, ok. thanks. | 16:46 |
| johnsom | Yeah, in dual CA (don't look at deployment tools like tripleo that do it completely wrong) those are different files | 16:47 |
| johnsom | Oh, no, those are the same file, just used in different places | 16:47 |
| johnsom | Ugh, I haven't looked at this in a bit. | 16:48 |
| johnsom | The guide is right | 16:48 |
| tweining | I guess it is fine. It's just something I noticed an wondered why it is the same path in the guide. | 16:48 |
| johnsom | Right, I remember now, the haproxy_amphora server_ca may need to include intermediate certs, where the usage in [certificates] would not | 16:50 |
| tweining | one other thing that surprised me a bit was that the defaults for the cert and key path is in https://opendev.org/openstack/octavia/src/branch/master/octavia/certificates/common/local.py /etc/ssl/certs and not /etc/octavia/certs | 16:50 |
| tweining | but in my case I intend to override the default using the environment variable | 16:51 |
| tweining | anyway, that is all from me for today I think. | 16:52 |
| johnsom | I think those are only used (if they are) in tests | 16:52 |
| gthiemonge | no idea on this | 16:52 |
| QG | Hi ! i have a question about a patch i did : https://review.opendev.org/c/openstack/octavia/+/898803, | 16:52 |
| QG | when I run the unit test alone, I don't reproduce the prb ( octavia.tests.functional.api.v2.test_load_balancer.TestLoadBalancer.test_create_with_vip_subnet_fills_network ) | 16:52 |
| johnsom | snakeoil certs are created by the OS at install time. They are bogus self-signed (but valid if you ignore the self signed part) | 16:53 |
| johnsom | My wonder is if those globals are even used anywhere | 16:53 |
| gthiemonge | QG: you're talking about this error? https://zuul.opendev.org/t/openstack/build/e7e27ff8071140e5a142a5eadd4eee6f | 16:53 |
| gthiemonge | QG: I'll take a look | 16:54 |
| QG | gthiemonge : Yes ! | 16:54 |
| gthiemonge | there are some other weird errors | 16:54 |
| gthiemonge | {"faultcode": "Client", "faultstring": "Invalid input for field/attribute vip_network_id. Value: \'<MagicMock name=\'get_network().id\' id=\'139946822340080\'>\'. Value should be UUID format", "debuginfo": null}' | 16:54 |
| gthiemonge | I'm surprised that you don't have it in your nev | 16:54 |
| gthiemonge | env | 16:54 |
| gthiemonge | anyway, I'll run the tests tomorrow morning | 16:55 |
| QG | gthiemonge: ok thanks ! | 16:55 |
| gthiemonge | folks, do you have any other topics for today? | 16:56 |
| oschwart | I wanted to ask quick question about | 16:56 |
| oschwart | https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/893066 | 16:56 |
| oschwart | if it is too late we can talk about it tomorrow | 16:56 |
| gthiemonge | you have 4 min oschwart! | 16:56 |
| oschwart | haha | 16:56 |
| oschwart | :o | 16:56 |
| oschwart | After the latest review, I removed the additional config option here, and after the check pipepline failed, I remembered why we had them in the first place :) | 16:56 |
| oschwart | as stable branches don't have the noop certificate manager | 16:56 |
| gthiemonge | right | 16:57 |
| gthiemonge | I remember that | 16:57 |
| oschwart | should I revert the patch to have the noop cert option too? | 16:57 |
| johnsom | So either backport the noop cert manager or gate the tests on the API version | 16:57 |
| gthiemonge | johnsom: you don't agree with this new config option? | 16:58 |
| johnsom | As I commented, it's really duplicate | 16:59 |
| gthiemonge | FYI I have another meeting in 1min | 16:59 |
| oschwart | let's talk about it tomorrow then | 16:59 |
| johnsom | It seems we could backport it | 16:59 |
| gthiemonge | let's close the meeting, I'll check wiht oschwart | 16:59 |
| tweining | +1 | 17:00 |
| gthiemonge | thank you! | 17:00 |
| oschwart | thanks for the input all | 17:00 |
| gthiemonge | #endmeeting | 17:00 |
| opendevmeet | Meeting ended Wed Nov 15 17:00:12 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/octavia/2023/octavia.2023-11-15-16.00.html | 17:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/octavia/2023/octavia.2023-11-15-16.00.txt | 17:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/octavia/2023/octavia.2023-11-15-16.00.log.html | 17:00 |
| opendevreview | Michael Johnson proposed openstack/octavia stable/2023.1: Add Noop Certificate Manager https://review.opendev.org/c/openstack/octavia/+/900997 | 17:01 |
| opendevreview | Michael Johnson proposed openstack/octavia stable/zed: Add Noop Certificate Manager https://review.opendev.org/c/openstack/octavia/+/900998 | 17:01 |
| opendevreview | Michael Johnson proposed openstack/octavia stable/yoga: Add Noop Certificate Manager https://review.opendev.org/c/openstack/octavia/+/900999 | 17:01 |
| johnsom | Clean backports even | 17:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!