Thursday, 2020-11-05

*** tosky has quit IRC00:14
*** tosky has joined #openstack-barbican08:13
*** JohnnyRainbow has joined #openstack-barbican10:15
*** noonedeadpunk has quit IRC10:32
*** noonedeadpunk has joined #openstack-barbican10:32
*** noonedeadpunk has quit IRC11:21
*** noonedeadpunk has joined #openstack-barbican11:25
*** raildo has joined #openstack-barbican12:23
*** moguimar has quit IRC14:17
*** moguimar has joined #openstack-barbican14:18
JohnnyRainbowIs a somewhere a special role which needs to be added between octavia and barbican to make it operational? I created SNI container and default tls container from certs without password, but octaviaclient can't get certificates somewhere, which is strange, especially that I'm able to get it from my project. I added even ACL for octavia user, but it is the same all the time. Here is14:32
JohnnyRainbow"full" listing: https://paste.ofcode.org/PRdnQNacjmpuWyMjWfXnrG I know that we were trying to solve similar issue already, but it was with certs with pass, which was pretty OK that it was failing, but in that story it shouldn't take a place... :)14:33
*** moguimar has quit IRC14:56
*** moguimar has joined #openstack-barbican14:58
openstackgerritMerged openstack/barbican stable/train: Use serial number or label for PKCS#11 tokens  https://review.opendev.org/76015416:37
johnsomJohnnyRainbow What version of the python-barbicanclient do you have installed there?16:51
johnsomIt would be on the API controllers BTW17:00
JohnnyRainbow@johnsom do you mean barbican api controllers or octavia?17:30
johnsomJohnnyRainbow the Octavia API17:31
johnsompython-barbicanclient will be installed there, what I am looking for is the version there17:31
JohnnyRainbowjust to be sure about my openstack architecture, I have separate VM for every single component, like barbican, octavia, cinder etc17:31
johnsomThat is ok17:32
JohnnyRainbowDo I need to have python-barbicanclient installed on my octavia VM, isn't enough openstackclient?17:32
JohnnyRainbowthe question is, because I see in api logs that all commands are going via curl in api...but I can be wrong :)17:33
johnsomYes, it has to be there as python-barbicanclient is the python language binding for barbican. Octavia will use the libraries in python-barbicanclient to access barbican resouces17:33
JohnnyRainbowI have version 4.6.0-0ubuntu1.117:34
JohnnyRainbowof python-barbicanclient17:34
johnsomOk, that is the problem. This patch is missing: https://github.com/openstack/python-barbicanclient/commit/4eec7121b39de3849b469c56d85b95520aab7bad17:35
johnsomI would try updating that package to a newer version, restarting the Octavia processes and trying again.17:36
JohnnyRainbowwow, you're fast...which version it should be ok?17:36
JohnnyRainbowI have barbican 8.0.1-0ubuntu1 installed on my barbican VMs17:37
johnsomWhat version of Octavia are you running?17:37
JohnnyRainbow4.1.017:37
johnsomAccording to this: https://releases.openstack.org/stein/index.html#python-barbicanclient17:38
johnsomYou want 4.8.117:38
johnsomof python-barbicanclient17:38
johnsomThat will include the fix17:39
johnsomIt's a bummer the ubuntu package hasn't been updated....17:41
JohnnyRainbowhmm...should I compile by myself?17:41
johnsomHmm, actually, as I look at the ubuntu changelog, they came the backported it....17:42
johnsomhttp://changelogs.ubuntu.com/changelogs/pool/main/p/python-barbicanclient/python-barbicanclient_4.6.0-0ubuntu1.1/changelog17:42
JohnnyRainbow  * d/p/0001-Allow-fetching-by-UUID-and-respect-interface.patch,17:43
JohnnyRainbowis it that one?17:43
johnsomd/p/0002-Secret-payload-should-also-be-fetched-by-UUID.patch17:43
johnsomSo it must be some other issue.17:44
JohnnyRainbowcan I somehow verify in the code of that python?17:44
JohnnyRainbowif that is included?17:44
johnsomCan you look in the Octavia API process logs and paste me the messages it logs when you attempt to use the certificate?17:44
johnsomYes, you could look for the secrets.py file and check for the change included here: https://github.com/openstack/python-barbicanclient/commit/4eec7121b39de3849b469c56d85b95520aab7bad17:45
johnsomBut I trust the packaging, I suspect it is some other issue.17:45
JohnnyRainbowok, let me check secrets.py first and if it's there I will collect octavia api logs17:45
johnsomIt is this change: https://github.com/openstack/python-barbicanclient/commit/4eec7121b39de3849b469c56d85b95520aab7bad#diff-317e364199973933c481d6a92b785455c02740e4b2d305d3c3f2acff13e0c8f717:46
JohnnyRainbowok, fix is there as I see17:51
JohnnyRainbowthat part of code can be found :)17:52
JohnnyRainbowbut when I execute my command for create listener I do not see new prompts in /var/log/octavia for such listener creation...but I do not have debug flag set to true for octavia...should I set it to true?17:55
johnsomHmm, well, with an error occurring I would expect there to be a message even without debug enabled, but turning debug on for the API would not hurt.17:57
JohnnyRainbowOK, here is a print from api.log: https://paste.ofcode.org/qcZ3PZFDxSqTT4magYLRJj18:08
JohnnyRainbowI see format_exception again18:09
johnsomYep18:09
JohnnyRainbowhmm...I'm using p12 files and certificates are self generated by openssl, but without any passwords18:09
johnsomI need to open a bug to raise that from debug to info18:09
JohnnyRainbowis there any additional requirement? I was following some cookbooks available in openstack docs and also redhat docs and suse docs :)18:11
johnsomThese are the best docs: https://docs.openstack.org/octavia/latest/user/guides/basic-cookbook.html#deploy-a-tls-terminated-https-load-balancer18:12
JohnnyRainbowyes, that is what I was using...my certs are generally created that way, so I don't know why it is format issue :)18:13
johnsomYeah, me either. Give me a minute to look at a few things and think on this18:14
JohnnyRainbowthanks a lot!18:15
openstackgerritMerged openstack/barbican stable/train: Use barbican.conf in barbican-manage  https://review.opendev.org/76063118:19
johnsomOk, how do you feel about adding some logging code to the Octavia API?18:26
JohnnyRainbowI can do it, but my env is managed by puppet, so I need to turn off it for a while :)18:28
JohnnyRainbowbut no problem18:28
johnsomhttps://www.irccloud.com/pastebin/uGx8OVjc/18:28
JohnnyRainbowdo I need to restart octavia?18:34
JohnnyRainbowafter change?18:34
johnsomJust the API process18:34
JohnnyRainbowI do not see something more in api.log18:36
JohnnyRainbowshould I paste it to you?18:36
johnsomSure18:36
JohnnyRainbowhttps://paste.ofcode.org/mGKdv5cLsaPpkpEyp369pA18:37
johnsomOk, I see one change that isn't present in stein. Are your endpoints HTTPS?18:37
JohnnyRainbowwhich one? Most of them are http18:38
johnsomYeah, the new message is in there. It's a 404 though.18:38
johnsomCan you paste the output of "openstack endpoint list"?18:40
JohnnyRainbowsure, just a second18:40
johnsomhttps://controller.tc.eco.atman.pl:35357/v2.0/tokens Seems to be the URL returning the 40418:40
JohnnyRainbowhttps://paste.ofcode.org/VWTpvrcbBpnnW8T626UFqs18:41
johnsomIn the octavia.conf file for the Octavia API, in the keystone_authtoken section, is auth_url set ?18:47
JohnnyRainbowit is18:48
JohnnyRainbowto https://controller.tc.eco.atman.pl:3535718:48
johnsomOk, this might be the problem. It seems to be trying to connect to "https://controller.tc.eco.atman.pl:35357/v2.0/tokens" but I suspect the v2.0/tokens path doesn't exist18:50
JohnnyRainbowbut auth_version is set to 3 in keystone_authtoken18:50
johnsomYeah,  I don't know why it would be trying v2.018:51
JohnnyRainbowhmm...but why only for barbican? :)18:52
johnsomCan you check the service_auth section is also version 3 and doesn't have the v2.0?18:52
JohnnyRainbowin octavia.conf?18:52
johnsomYes18:52
johnsomI doubt it is wrong as you are able to create a load balancer.18:53
JohnnyRainbowit is 3 as well18:53
johnsomOk, I have a long shot thing we can try18:54
JohnnyRainbowsure18:54
johnsomhttps://www.irccloud.com/pastebin/Bs0jrVOV/18:55
johnsomRestart API , try again. I see this change in master, but not stein. Maybe somehow a CA mismatch is being reported as a 404 by keystone client.18:56
JohnnyRainbowhttps://paste.ofcode.org/UCzDYGb5mz7Y9b3VEvCuF19:03
JohnnyRainbowhmm...similar error19:03
johnsomYeah, that was a longshot19:03
johnsomThat doesn't seem like the right conf path really. Do you have "cafile" under the keystone_auth section in the octavia.conf?19:09
JohnnyRainbowdo you mean under keystone_authtoken?19:10
johnsomyes19:10
JohnnyRainbowno, it's not there19:10
JohnnyRainbowis that a fault?19:12
johnsomI'm not sure really. I am still trying to figure out the v2.0 thing.19:14
JohnnyRainbowno worries, at least I see direction of a fault, but it is strange to me either :)19:15
JohnnyRainbowI'll be AFK for 1h19:19
johnsomOk, I'm pretty sure the issue is with the "[service_auth]" section of the octavia.conf file. Specifically I think the "auth_url" in that section may be set incorrectly19:20
johnsomOr maybe the "interface" setting19:22
JohnnyRainbowin service auth I have cafile provided, and auth_url is the same, I mean: https://controller.tc.eco.atman.pl:3535719:22
johnsomIs the "interface" setting admin?19:23
JohnnyRainbowwhich setting is that?19:25
johnsom[service_auth] interface = admin19:25
JohnnyRainbowI do not have this in my conf19:25
johnsomHmm, ok, one thing to check. You can compare your [keystone_authtoken] and [service_auth] sections with those in the neutron.conf. They should be the same19:26
JohnnyRainbowhmm...service_auth is not in neutron.conf :)19:30
JohnnyRainbowbut generally config is the same19:30
johnsomYeah, I think that got removed19:30
JohnnyRainbowbut other things are same, except www_authenticate_uri which is https://controller.tc.eco.atman.pl:35357/v3 for octavia and https://controller.tc.eco.atman.pl:35357  for neutron19:32
JohnnyRainbowbut I believe it's not a case19:32
johnsomWell, I'm running low on ideas and you need to step away. I'm pretty sure the issue is a configuration issue in the keystone_authtoken and/or service_auth sections.19:32
JohnnyRainbowok, anyway, thanks a lot for your help, I need to take some break, think and most probably find a solution which is just a step ahead :)19:33
JohnnyRainbowthanks a lot again19:33
johnsomYou could try setting the auth_url to https://controller.tc.eco.atman.pl:35357/v3 to see if it stops trying v2.019:33
johnsomOk, good luck. Let us know what you find19:34
JohnnyRainbowdefinitely!19:34
JohnnyRainbowthanks19:34
*** dwilde has quit IRC20:46
*** d34dh0r53 has joined #openstack-barbican20:46
*** JohnnyRainbow has quit IRC23:54
*** tosky has quit IRC23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!