Tuesday, 2022-02-01

*** oklhost_ is now known as oklhost07:16
noonedeadpunkHey! Any thought about application credential/keystone issue when trying to use ssl termination?12:10
noonedeadpunkJust will place link to log not to repeat myself a lot https://meetings.opendev.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2022-01-28.log.html :)12:12
noonedeadpunkjohnsom: maybe you have some insight if building session based on the token is really expected thing to work when we're talking about application credentials? I tried to find how nova/cinder handles communication, but they leverage castellan, so not sure12:17
gthiemongenoonedeadpunk: nop sorry, first time I see this error12:37
noonedeadpunksuper easily reproducable btw12:37
noonedeadpunkjust matter of trying to create https terminated listener when authenticated with application credentials12:38
noonedeadpunkAs keystone requires application_credential id to be passed with session12:38
noonedeadpunkSo things fail even before reaching barbican for request12:38
noonedeadpunkThe main question I'm kind of bothered with - if it's keystone issue or octavia one...12:39
noonedeadpunkfrom keystone side it's super easy to fix by not requiring application id in context. But I'm kind of afraid of doing that as I can imagine some intention in such requirement. Also nova/cinder somehow succeed with current code in keystone12:47
gthiemongeI'm not familiar with this code in Octavia, maybe johnsom will help us on it12:47
noonedeadpunk(as they also pass user session somehow to barbican when creating encrypted volume or booting vm12:47
johnsomnoonedeadpunk I can take a quick look at this today. Can you pastebin me how you are issuing the token?15:19
noonedeadpunkI'm not sure how originally it was done, but I reproduced just with openstackclient and defining application credentials in clouds.yaml15:22
noonedeadpunkso we `openstack application credential create`, create clouds.yaml, then openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=https://tky1.citycloud.com:9311/v1/containers/09db5046-f9d6-4717-8691-8cd45b2b7334 40e06916-e573-4641-b1ca-5780134a914b fails with `OctaviaClientException: Could not retrieve certificate`15:23
johnsomOk. Just wanted to make sure I had the required reproducer steps. I will poke at this today.15:24
noonedeadpunkwas trying with current master of everything15:24
noonedeadpunkSo we did bunch of prints and found that basically this stanza was provided to keystoneauth https://paste.opendev.org/show/812468/15:25
noonedeadpunkand well, if you try just to create a token when authenticated using application credentials and pass it to keystone it won't like it either and jsut fail15:27
noonedeadpunkas it expects application_credential['id'] to be provided with token15:27
noonedeadpunkjohnsom: ah, yes, I clean forgot to mention that we filled in a bug in keystone as were not sure where to land that15:53
johnsomHa, yeah, just saw that.15:53
noonedeadpunkKeystone fix sounds easy solution, not sure if it's correct one though15:53
opendevreviewGregory Thiemonge proposed openstack/octavia master: Add support for Centos 9 Stream  https://review.opendev.org/c/openstack/octavia/+/81637016:35
opendevreviewGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add centos-9-stream job  https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/81636916:35
johnsomYeah, I can reproduce this17:20
opendevreviewGregory Thiemonge proposed openstack/octavia master: Add support for Centos 9 Stream  https://review.opendev.org/c/openstack/octavia/+/81637017:30
opendevreviewGregory Thiemonge proposed openstack/octavia master: Add centos-9-stream jobs  https://review.opendev.org/c/openstack/octavia/+/82733017:30
opendevreviewDon Kehn proposed openstack/octavia stable/wallaby: Add keepalive for redis-based taskflow boards  https://review.opendev.org/c/openstack/octavia/+/82602818:48
johnsomdkehn ^^^^ I think that was an opps18:51
johnsomnoonedeadpunk I have dug into this issue, I think it's a problem in keystone as you have mentioned. Comments here: https://bugs.launchpad.net/keystone/+bug/195967420:22
noonedeadpunkYeah that's pretty close to what we saw20:25
noonedeadpunkWhat kind of bugs me - how in the world nova/cinder handle interaction with barbican20:25
noonedeadpunkas it's working nicely for them20:25
noonedeadpunkand eventually I'd assume that with such issue it would uncover for other usecases as well.20:26
johnsomThey are probably not trying to take action on behalf of the user like we are.20:26
noonedeadpunkwell, they do)20:26
noonedeadpunkas if you want encrypted volume - you need to have secret in barbican and it's stored with user scope20:27
johnsomThough it is odd, as I read the application credential docs, you are always supposed to create a token from it, so ... yeah20:27
noonedeadpunkAnd same comes to VM boot or volume attachment20:27
noonedeadpunkas without secret that stored for volume, compute won't be able to boot vm, so it reads secret data un decrypt LUKS20:28
noonedeadpunkand there's no ACL created for this, so it's just plain user scope20:29
johnsomI think to go forward we need someone with more keystone expertise than I have. I'm 85% confident it's not an Octavia issue at this point.20:29
noonedeadpunkfeels old keystone cores are not around nowadays though :(20:30
johnsomYeah, sadly true on many projects20:30
noonedeadpunkindeed. thatnk for watching into it:)20:31
noonedeadpunkas I was kind of more inclined to blame octavia :P as I haven't seen anything like that in cinder code for instance, and they all use castellan instead of direct barbican interaction20:32
johnsomWe also support castellan in Octavia20:32
noonedeadpunkoh, rly? is it default or...?20:34
noonedeadpunkas our sandbox just led us to imports of barbicanclient directly20:35
johnsomThere have been/are a few folks using Octavia with Vault via castellan. Seems like a common pattern20:35
noonedeadpunkmmm.... and how then to provide what backend castellan should use?20:38
noonedeadpunkas it can be both barbican or vault20:38
johnsomI think the href for the secret informs castellan. It's been a while since I messed with this code.20:39
noonedeadpunkor it should be smth like castellan_cert_manager.key_manager.barbican_key_manager.BarbicanKeyManager ?20:40
johnsomMaybe there are config settings for castellan that go in the octavia.conf.  Ugh, it's been a while20:41
noonedeadpunkI actually tried to find smth regarding that one day :) it interesting if issue is the same when using catellan instead...20:41
johnsomYeah, looks like there are castellan settings that go in the conf.20:43
noonedeadpunkI mean - if there's way to define to work through castellan, then I guess route through barbican_acl_auth won't be applicable?20:44
noonedeadpunkso would be interesting to test this out20:45
noonedeadpunkI don;t see any config here https://opendev.org/openstack/octavia/src/branch/master/octavia/certificates/manager/castellan_mgr.py20:47
noonedeadpunkSo hardly understand how it really works... So like you can jsut define [key_manager] section and that's it?20:49
noonedeadpunkhm. interesting20:49
johnsomYeah, you will set our setting to the castellan manager. Then you configure castellan with it's settings20:49
noonedeadpunkok, thanks! will try that!20:50
noonedeadpunkwould be interesting if that just works...21:01

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!