| *** oklhost_ is now known as oklhost | 07:16 | |
| noonedeadpunk | Hey! Any thought about application credential/keystone issue when trying to use ssl termination? | 12:10 |
|---|---|---|
| noonedeadpunk | Just will place link to log not to repeat myself a lot https://meetings.opendev.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2022-01-28.log.html :) | 12:12 |
| noonedeadpunk | johnsom: maybe you have some insight if building session based on the token is really expected thing to work when we're talking about application credentials? I tried to find how nova/cinder handles communication, but they leverage castellan, so not sure | 12:17 |
| gthiemonge | noonedeadpunk: nop sorry, first time I see this error | 12:37 |
| noonedeadpunk | super easily reproducable btw | 12:37 |
| noonedeadpunk | just matter of trying to create https terminated listener when authenticated with application credentials | 12:38 |
| noonedeadpunk | As keystone requires application_credential id to be passed with session | 12:38 |
| noonedeadpunk | So things fail even before reaching barbican for request | 12:38 |
| noonedeadpunk | The main question I'm kind of bothered with - if it's keystone issue or octavia one... | 12:39 |
| noonedeadpunk | from keystone side it's super easy to fix by not requiring application id in context. But I'm kind of afraid of doing that as I can imagine some intention in such requirement. Also nova/cinder somehow succeed with current code in keystone | 12:47 |
| gthiemonge | I'm not familiar with this code in Octavia, maybe johnsom will help us on it | 12:47 |
| noonedeadpunk | (as they also pass user session somehow to barbican when creating encrypted volume or booting vm | 12:47 |
| johnsom | noonedeadpunk I can take a quick look at this today. Can you pastebin me how you are issuing the token? | 15:19 |
| noonedeadpunk | I'm not sure how originally it was done, but I reproduced just with openstackclient and defining application credentials in clouds.yaml | 15:22 |
| noonedeadpunk | so we `openstack application credential create`, create clouds.yaml, then openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=https://tky1.citycloud.com:9311/v1/containers/09db5046-f9d6-4717-8691-8cd45b2b7334 40e06916-e573-4641-b1ca-5780134a914b fails with `OctaviaClientException: Could not retrieve certificate` | 15:23 |
| johnsom | Ok. Just wanted to make sure I had the required reproducer steps. I will poke at this today. | 15:24 |
| noonedeadpunk | was trying with current master of everything | 15:24 |
| noonedeadpunk | So we did bunch of prints and found that basically this stanza was provided to keystoneauth https://paste.opendev.org/show/812468/ | 15:25 |
| noonedeadpunk | and well, if you try just to create a token when authenticated using application credentials and pass it to keystone it won't like it either and jsut fail | 15:27 |
| noonedeadpunk | as it expects application_credential['id'] to be provided with token | 15:27 |
| noonedeadpunk | johnsom: ah, yes, I clean forgot to mention that we filled in a bug in keystone as were not sure where to land that | 15:53 |
| johnsom | Ha, yeah, just saw that. | 15:53 |
| noonedeadpunk | Keystone fix sounds easy solution, not sure if it's correct one though | 15:53 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Add support for Centos 9 Stream https://review.opendev.org/c/openstack/octavia/+/816370 | 16:35 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add centos-9-stream job https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/816369 | 16:35 |
| johnsom | Yeah, I can reproduce this | 17:20 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Add support for Centos 9 Stream https://review.opendev.org/c/openstack/octavia/+/816370 | 17:30 |
| opendevreview | Gregory Thiemonge proposed openstack/octavia master: Add centos-9-stream jobs https://review.opendev.org/c/openstack/octavia/+/827330 | 17:30 |
| opendevreview | Don Kehn proposed openstack/octavia stable/wallaby: Add keepalive for redis-based taskflow boards https://review.opendev.org/c/openstack/octavia/+/826028 | 18:48 |
| johnsom | dkehn ^^^^ I think that was an opps | 18:51 |
| johnsom | noonedeadpunk I have dug into this issue, I think it's a problem in keystone as you have mentioned. Comments here: https://bugs.launchpad.net/keystone/+bug/1959674 | 20:22 |
| noonedeadpunk | Yeah that's pretty close to what we saw | 20:25 |
| noonedeadpunk | What kind of bugs me - how in the world nova/cinder handle interaction with barbican | 20:25 |
| noonedeadpunk | as it's working nicely for them | 20:25 |
| noonedeadpunk | and eventually I'd assume that with such issue it would uncover for other usecases as well. | 20:26 |
| johnsom | They are probably not trying to take action on behalf of the user like we are. | 20:26 |
| noonedeadpunk | well, they do) | 20:26 |
| noonedeadpunk | as if you want encrypted volume - you need to have secret in barbican and it's stored with user scope | 20:27 |
| johnsom | Though it is odd, as I read the application credential docs, you are always supposed to create a token from it, so ... yeah | 20:27 |
| noonedeadpunk | And same comes to VM boot or volume attachment | 20:27 |
| noonedeadpunk | as without secret that stored for volume, compute won't be able to boot vm, so it reads secret data un decrypt LUKS | 20:28 |
| noonedeadpunk | and there's no ACL created for this, so it's just plain user scope | 20:29 |
| johnsom | I think to go forward we need someone with more keystone expertise than I have. I'm 85% confident it's not an Octavia issue at this point. | 20:29 |
| noonedeadpunk | agree | 20:29 |
| noonedeadpunk | feels old keystone cores are not around nowadays though :( | 20:30 |
| johnsom | Yeah, sadly true on many projects | 20:30 |
| noonedeadpunk | indeed. thatnk for watching into it:) | 20:31 |
| johnsom | NP | 20:31 |
| noonedeadpunk | as I was kind of more inclined to blame octavia :P as I haven't seen anything like that in cinder code for instance, and they all use castellan instead of direct barbican interaction | 20:32 |
| johnsom | We also support castellan in Octavia | 20:32 |
| noonedeadpunk | oh, rly? is it default or...? | 20:34 |
| noonedeadpunk | as our sandbox just led us to imports of barbicanclient directly | 20:35 |
| johnsom | https://github.com/openstack/octavia/blob/master/setup.cfg#L97 | 20:35 |
| johnsom | There have been/are a few folks using Octavia with Vault via castellan. Seems like a common pattern | 20:35 |
| johnsom | https://github.com/openstack/octavia/blob/master/etc/octavia.conf#L163 | 20:36 |
| noonedeadpunk | mmm.... and how then to provide what backend castellan should use? | 20:38 |
| noonedeadpunk | as it can be both barbican or vault | 20:38 |
| johnsom | I think the href for the secret informs castellan. It's been a while since I messed with this code. | 20:39 |
| noonedeadpunk | or it should be smth like castellan_cert_manager.key_manager.barbican_key_manager.BarbicanKeyManager ? | 20:40 |
| johnsom | Maybe there are config settings for castellan that go in the octavia.conf. Ugh, it's been a while | 20:41 |
| noonedeadpunk | I actually tried to find smth regarding that one day :) it interesting if issue is the same when using catellan instead... | 20:41 |
| johnsom | Yeah, looks like there are castellan settings that go in the conf. | 20:43 |
| noonedeadpunk | I mean - if there's way to define to work through castellan, then I guess route through barbican_acl_auth won't be applicable? | 20:44 |
| noonedeadpunk | so would be interesting to test this out | 20:45 |
| noonedeadpunk | I don;t see any config here https://opendev.org/openstack/octavia/src/branch/master/octavia/certificates/manager/castellan_mgr.py | 20:47 |
| noonedeadpunk | So hardly understand how it really works... So like you can jsut define [key_manager] section and that's it? | 20:49 |
| noonedeadpunk | hm. interesting | 20:49 |
| johnsom | https://opendev.org/openstack/castellan/src/branch/master/castellan/key_manager/__init__.py#L25 | 20:49 |
| johnsom | Yeah, you will set our setting to the castellan manager. Then you configure castellan with it's settings | 20:49 |
| noonedeadpunk | ok, thanks! will try that! | 20:50 |
| noonedeadpunk | would be interesting if that just works... | 21:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!