Tuesday, 2020-11-17

*** tosky has quit IRC00:06
*** iurygregory has quit IRC01:32
*** johnsom has quit IRC04:46
*** johnsom has joined #openstack-barbican04:49
*** johnsom has quit IRC05:18
*** johnsom has joined #openstack-barbican05:20
*** johnsom has quit IRC06:27
*** johnsom has joined #openstack-barbican06:27
*** rm_work has quit IRC07:26
*** rm_work has joined #openstack-barbican07:28
*** gagehugo has quit IRC07:51
*** gagehugo has joined #openstack-barbican07:51
*** iurygregory has joined #openstack-barbican08:52
*** tosky has joined #openstack-barbican08:56
*** JohnnyRainbow has joined #openstack-barbican09:00
*** xek has joined #openstack-barbican09:36
*** JohnnyRainbow has quit IRC11:11
*** JohnnyRainbow has joined #openstack-barbican11:14
*** JohnnyRainbow has joined #openstack-barbican11:15
JohnnyRainbowHi Guys, @ade_lee just a question for you, based on this error message: http://paste.openstack.org/show/800058/ -> Is barbican able to cooperate with the newest freeipa release, as I see it requires nss_db and certificate, but freeipa doesn't serve kra_admin_cert anymore, as per that document: https://www.freeipa.org/page/V4/Replace_NSS_with_OpenSSL they changed NSS with OpenSSL. I'm11:33
JohnnyRainbowjust wondering if I'm able to integrate barbican with dogtag from freeipa somehow, seems that integration is a bit different than in this tutorial: https://vakwetu.wordpress.com/2015/11/30/barbican-and-dogtagipa/ Is anyone having experience with this and can share some useful hints? Thanks in advance11:33
*** raildo_ has joined #openstack-barbican12:14
*** raildo has quit IRC12:16
*** Luzi has joined #openstack-barbican12:36
redrobot#startmeeting barbican13:00
openstackMeeting started Tue Nov 17 13:00:27 2020 UTC and is due to finish in 60 minutes.  The chair is redrobot. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
*** openstack changes topic to " (Meeting topic: barbican)"13:00
openstackThe meeting name has been set to 'barbican'13:00
redrobot#topic Roll Call13:00
*** openstack changes topic to "Roll Call (Meeting topic: barbican)"13:00
redrobotCourtesy ping for ade_lee dave-mccowan hrybacki jamespage Luzi lxkong mhen moguimar raildo rm_work xek nearyo13:00
Luzio/13:00
rm_worko/13:01
redrobotHi y'all!13:01
rm_workAnything interesting going on? :)13:03
redrobotrm_work, good to see you! :D13:03
redrobotSame 'ol same 'ol13:03
redrobotWe'll eventually add that pkcs12 secret type13:03
rm_workSweet, LMK13:03
rm_workI'm about to write a castellan driver for our custom in-house secret storage thing13:04
rm_workSo that should be fun13:04
redrobotHehe13:04
redrobotOkay, let's see ...13:04
redrobot#topic Review Past Action Items13:05
*** openstack changes topic to "Review Past Action Items (Meeting topic: barbican)"13:05
redrobot#link http://eavesdrop.openstack.org/meetings/barbican/2020/barbican.2020-11-10-13.00.html13:05
redrobotI did not do these :(13:05
rm_workThat makes me feel better about all the stuff I'm probably supposed to be doing but am not :D13:06
redrobot#action redrobot to add a kanban card for Cinder -> Barbican + Vault issue13:06
redrobotrm_work, lol13:06
redrobot#action redrobot to update the kanban board13:06
redrobot^^ is update to include action items from PTG13:06
redrobotOK, moving on13:07
redrobot#topic Liaison Updates13:07
*** openstack changes topic to "Liaison Updates (Meeting topic: barbican)"13:07
redrobotmoguimar? tosky?13:07
toskyno news from me13:07
redrobotcool, thanks tosky13:08
redrobotnot sure if moguimar is around13:08
redrobot#topic Kanban Review13:09
*** openstack changes topic to "Kanban Review (Meeting topic: barbican)"13:09
redrobot#link https://tree.taiga.io/project/dmend-openstack-barbican/kanban13:09
redrobotnot much movement happened there this week13:09
redrobotI did add a card to track the 1.1 Microversion13:09
redrobot#topic Bug Review13:10
*** openstack changes topic to "Bug Review (Meeting topic: barbican)"13:10
redrobot#link https://storyboard.openstack.org/#!/project_group/barbican13:10
redrobotLooks like the only new bug was for the Cinder + Barbican->Vault issue13:11
redrobot#link https://bugs.launchpad.net/castellan/+bugs?orderby=-id&start=013:13
redrobotLook like no new bugs in Castellan either13:13
redrobotWe'd usually do the Wayward Reviews right now, but there does not seem to be any other cores logged on right now13:13
redrobotso,13:13
redrobot#topic Open Discussion13:13
*** openstack changes topic to "Open Discussion (Meeting topic: barbican)"13:14
redrobotanything else y'all want to talk about?13:14
rm_workDid y'all ever enact the plan to split out the cert request stuff entirely and make that a separate service from secret management?13:17
redrobotwe did deprecate it13:19
redrobotand then had a patch to remove it, which I need to rebase13:19
redrobotWe don't have any plans for a new cert service13:20
redrobotI'd suggest deploying an ACME server13:20
redrobotlike Boulder13:20
rm_workHmm ok13:21
redrobotIf ade_lee was here he'd suggest IPA for certs :)13:22
rm_workWe have our own internal certificate system so I think we don't technically need stuff like that, but was just curious :D13:22
redrobotRight on13:26
redrobotOK, it sounds like we're done for the day13:26
redrobotthanks for joining everyone!13:27
redrobot#endmeeting13:27
*** openstack changes topic to "OpenStack Barbican Development - Weekly Meeting Agenda: https://etherpad.openstack.org/p/barbican-weekly-meeting"13:27
openstackMeeting ended Tue Nov 17 13:27:24 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:27
openstackMinutes:        http://eavesdrop.openstack.org/meetings/barbican/2020/barbican.2020-11-17-13.00.html13:27
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/barbican/2020/barbican.2020-11-17-13.00.txt13:27
openstackLog:            http://eavesdrop.openstack.org/meetings/barbican/2020/barbican.2020-11-17-13.00.log.html13:27
*** xek has quit IRC14:18
*** Luzi has quit IRC14:18
*** xek has joined #openstack-barbican14:18
*** d34dh0r53 has quit IRC14:22
*** d34dh0r53 has joined #openstack-barbican14:30
JohnnyRainbowHi @ade_lee, I just have seen your message at #freeipa channel, but just wanted to tell you that issue still exist. It looks like nss_db parameters are mandatory for dogtag plugin and new format of KRA cert is poiting to my error which I observed. So the question is how to disable nss or my understanding is incorrect?15:23
ade_leeJohnnyRainbow, sorry - in a meeting right now - but I'll respond shortly15:24
JohnnyRainbowsure, thanks a lot15:24
*** iurygregory has quit IRC16:29
*** xek has quit IRC16:32
*** iurygregory has joined #openstack-barbican17:19
*** xek has joined #openstack-barbican18:15
*** d34dh0r53 has quit IRC18:43
*** d34dh0r53 has joined #openstack-barbican18:46
ade_leeJohnnyRainbow, ok - back - sorry, meetings and catching up from my day off18:54
ade_leeJohnnyRainbow, so, looking at the plugin code ..18:55
ade_leeJohnnyRainbow, the code that is failing is likely here -- https://opendev.org/openstack/barbican/src/branch/master/barbican/plugin/dogtag.py#L64-L7618:57
ade_leeJohnnyRainbow, which is pointing to cryptoutil.NSSCryptoProvider.setup_database()18:58
ade_leewhich is in pki.Crypto18:59
ade_leepki.crypto,NSSCryptoProvider.setup_database()18:59
ade_leeJohnnyRainbow, where are you getting the dogtag client packages?18:59
ade_leeJohnnyRainbow, my guess is that whatever you have is calling this -- https://github.com/dogtagpki/pki/blob/master/base/common/python/pki/crypto.py#L14519:02
ade_leeand that the error you have is due to a change in the format of the nss database (ie. old vs. new_19:03
ade_leeJohnnyRainbow, what params do you have for your barbican dogtag config?19:07
ade_leeJohnnyRainbow, I'm wondering actually if you just need to have created the directory for the nss db19:11
*** xek has quit IRC19:44
*** xek has joined #openstack-barbican19:44
JohnnyRainbow@ade_lee sorry, I was AFK. But anwering to your questions, as a dogtag client packages I'm using pki-tools pki-base-java and pki-base packages for ubuntu. I can deliever exact versions if needed, but it was part of our previous discussion at #freeipa and #dogtag-pki channels20:35
JohnnyRainbowabout dogtag config, everything is here: http://paste.openstack.org/show/800113/ -> so looks like normal config :) At least connection with dogtag host is working but then issue is with NSS, as it's part of Freeipa 4.8.4 and it seems it's not there. Additionally I'm not fully sure if my kra_admin_cert.pem is correct as I concatenated it from ra-agent.pem and ra-agent.key from20:39
JohnnyRainbow/var/lib/ipa directory as that was suggested in this docs: https://www.freeipa.org/page/V4/Replace_NSS_with_OpenSSL20:39
ade_leeJohnnyRainbow, does the directory /etc/barbican/alias exist?21:03
JohnnyRainbowI created it manually, but I believe it should be created automatically, right?21:10
JohnnyRainbowwhen I deleted it,, then I got an error in logs21:10
ade_leeJohnnyRainbow, yeah - it should be, but I recall times in the past where it was easiest just to create it21:11
ade_leeJohnnyRainbow, so yeah - if you create it manually first (empty), does the deployment succeed?21:11
JohnnyRainbowyeah, but when I created it, then in logs I see this: The nss_db_path provid21:13
JohnnyRainbowed already exists, so the database is assumed to be already set up.21:13
JohnnyRainbowy21:13
ade_leeah ok21:13
JohnnyRainbowand later on I have an error with this invalid cert21:13
JohnnyRainbowI mean The certificate/key database is in an old, unsupported format.: nss.error.NSPRError:21:13
ade_leeright - and what do you see in that directory?21:14
JohnnyRainbowit's empty: http://paste.openstack.org/show/800115/21:15
JohnnyRainbowmaybe something wrong is with my kra cert? It looks like that: http://paste.openstack.org/show/800116/21:16
ade_leeno thats ok21:16
ade_leewhat's supposed to go into the /etc/barbican/alias is an nss db with the transport cert inside21:17
ade_leeJohnnyRainbow, we can try to create it manually21:17
ade_leeJohnnyRainbow, and see where its failing21:17
JohnnyRainbowhmm...how to do this?21:17
ade_leeso -- cd /etc/barbican/alias21:18
JohnnyRainbowI'm there21:18
ade_leeJohnnyRainbow, let me pastebin somehting .. just a sec21:19
JohnnyRainbowsure, no rush21:19
ade_leeJohnnyRainbow, who owns /etc/barbican/alias?21:19
ade_lee(just wondering if the db is not being created because of a perms issue21:20
JohnnyRainbowdrwxr-xr-x 2 root     barbican 4.0K Nov 17 16:58 alias21:21
ade_leehttp://paste.openstack.org/show/800117/21:23
ade_leeJohnnyRainbow, what do you see when that happens?21:23
JohnnyRainbowhttp://paste.openstack.org/show/800118/21:24
JohnnyRainbowhere you can find a result21:24
ade_leeok some files should have been created, right?21:25
JohnnyRainbowbasically it's empty database, I have only few secrets in my barbican so far21:26
JohnnyRainbowhttp://paste.openstack.org/show/800119/21:26
JohnnyRainbowhere is a content with files21:26
ade_leeJohnnyRainbow, ok - thats interesting -- its a db with the old legacy nss format21:27
ade_leewhich will be empty right now21:27
ade_leenow we need to populate it with the kra transport cert21:27
JohnnyRainbowso...should I copy my kra_admin_cert.pem there?21:28
ade_leeno thats not the same thing21:28
ade_leethe admin cert is the cert used for authenticating the admin user to be able to store secrets in dogtag21:29
ade_leethe transport cert is used to encrypt the data between barbican and the kra21:29
JohnnyRainbowok, got it21:29
ade_leewe just need to get the transport cert and put it in this new nss db21:30
JohnnyRainbowso, the transport cert should be somehow loaded automatically, but it's not, right?21:30
ade_leewell -- sorta -- we're doing this -- https://opendev.org/openstack/barbican/src/branch/master/barbican/plugin/dogtag.py#L101-L10421:31
ade_leeJohnnyRainbow, so the initial db create -- which is just what we did manually -- failed21:32
ade_leefor some reason21:32
JohnnyRainbowok, I see21:32
ade_leeJohnnyRainbow, if you can modify the code to remove the conditional on line 103 -- then it should pull in the certs automatically21:33
ade_leeotherwise we need to do whats in line 104 manually21:33
JohnnyRainbowok, let me find it21:34
JohnnyRainbowjust to be sure, it should be like that: http://paste.openstack.org/show/800120/21:40
ade_leeJohnnyRainbow, yeah line up the # with the previous line too though21:41
JohnnyRainbowhmm...I haven't done it yet, but a bit new error: http://paste.openstack.org/show/800121/21:42
ade_leehmm .. well its trying to get the cert in ..21:44
ade_leeok -- lets do it manually //21:44
JohnnyRainbowhmm...looks like connectivity?21:44
ade_leenot sure21:44
JohnnyRainbowwhat port it should be hitted?21:44
ade_leewhat happens when you try to go to /curl -> https://<dogtag_url>:9443/rest/config/cert/transport ?21:45
JohnnyRainbow9443 or 8443?21:47
ade_leenot sure - its been awhile -- I thought it was 9443 by default ..21:47
ade_leewhatever is correct in your enviornment21:48
JohnnyRainbowhttp://paste.openstack.org/show/800122/21:50
JohnnyRainbowI have 404 not found21:50
ade_leeJohnnyRainbow, I'm looking here -- https://github.com/dogtagpki/pki/blob/master/base/common/python/pki/systemcert.py#L6121:51
ade_leeoh -- maybe .. https://<dogtag_url>:8443/kra/rest/config/cert/transport21:53
JohnnyRainbowthat actually works http://paste.openstack.org/show/800123/21:54
ade_leeJohnnyRainbow, ok -- copy / paste the cert part of that into a file21:55
JohnnyRainbowshould it be saved under specific filename?21:56
ade_leenah21:56
ade_leeJohnnyRainbow, now you need to import that cert into the db21:58
ade_leecertutil -A -d . -n "KRA transport cert" -t ",," -i foo.txt22:00
ade_leewhere foo.txt is the cert file you just created22:00
ade_leeJohnnyRainbow, if that doesn't work - you might need to strip off the header and footer -- I don't remember what works or not ..22:01
ade_leemake sure there are no extra chars22:01
ade_leeonce done, you can do a certutil -L -d .  to see the cert imported22:02
JohnnyRainbowyeah, I just played a bit with certificate retrieved via curl as it contains some special characters22:02
JohnnyRainbowhttp://paste.openstack.org/show/800124/22:03
JohnnyRainbowlooks like it's added22:03
ade_leecool -- what about certutil -L -d . -n "KRA transport cert"22:03
ade_leethat should show you the cert22:04
JohnnyRainbowyeah, it is22:04
JohnnyRainbowlike openssl x509, right?22:04
ade_leeJohnnyRainbow, yup22:04
ade_leeyou should now be set -- revert that code change22:04
JohnnyRainbowok, I have it22:04
ade_leeand again22:04
JohnnyRainbowok, let me do this22:05
JohnnyRainbowhmm...looks like it was not taken into account: http://paste.openstack.org/show/800126/22:07
JohnnyRainbowagain it's failing with certificate format22:07
ade_leeJohnnyRainbow, wish there were a stack trace to see where that error is being thrown22:08
JohnnyRainbowcan I somehow get it?22:09
ade_leeis there anything further up in the log?22:09
JohnnyRainbowhere is a full print: http://paste.openstack.org/show/800127/22:10
JohnnyRainbowand my testing command is: barbican secret store --payload-content-type='text/plain' --name='private_key_barbican_scenario1' --payload="$(cat server2.key)"22:10
JohnnyRainbowI was using it easily with simple_crypto22:11
ade_leethe other thing we can try is making sure the nssdb is using the new format, and not the old one22:11
ade_leeso to do that , you have to delete the nssdb you created22:12
JohnnyRainbowhmm...somehow by certutil tool?22:13
JohnnyRainbowor just /etc/barbican/alias directory?22:13
ade_leethat is delete the cert8* file and the key* file22:13
ade_leekeep the rest -- you'll need them again22:13
JohnnyRainbowthose two files from here: http://paste.openstack.org/show/800128/ ?22:14
ade_leeyup and the secmod one22:14
ade_leethen ..22:15
ade_lee export NSS_DEFAULT_DB_TYPE="sql"22:15
ade_leeand try create the db again ..22:15
JohnnyRainbowthat export globally?22:15
ade_leejust for this session22:16
JohnnyRainbowok, done22:16
JohnnyRainbowand execute my barbican command again?22:16
ade_leeso recreeate the db as before ..22:16
ade_leecertutil -N -d .  (etc)22:16
ade_leewhat does it look like now?22:17
JohnnyRainbowif like that then I hit an error: http://paste.openstack.org/show/800129/22:18
ade_leehuh?  you did ...  http://paste.openstack.org/show/800117/ ?22:20
JohnnyRainbownope, just certutil...ok, I can repeat all commands22:20
*** xek has quit IRC22:22
JohnnyRainbowhttp://paste.openstack.org/show/800130/22:23
JohnnyRainbowseems it is recreated22:23
ade_leeJohnnyRainbow, cool - and notice the different file names22:23
ade_leethats the new format db22:23
ade_leenow import the kra cert as before22:24
ade_lee(and make sure its there)22:24
JohnnyRainbowok, let me do this22:24
JohnnyRainbowok, done22:25
ade_leeJohnnyRainbow, ok - lets try again :)22:26
JohnnyRainbowa step ahead, but it is failing because of secretstore: http://paste.openstack.org/show/800131/22:27
ade_leeok progress22:28
JohnnyRainbowyeah22:29
JohnnyRainbowhttp://paste.openstack.org/show/800132/ secretstore config is really simple. do not know what is wrong for barbican there :)22:30
ade_leedid you restart barbican?22:30
JohnnyRainbownope, but I didn't change anything in config at all22:31
JohnnyRainbowwe just played with certificates manually :)22:31
ade_leelets try restart it22:31
JohnnyRainbowok, all components?22:31
JohnnyRainbowor just api?22:31
ade_leejust api22:32
JohnnyRainbowsame story...but a bit strange, seems for stein it changed, but for ubuntu I don't have barbican-api service, I have: barbican-keystone-listener.service  barbican-worker.service22:36
JohnnyRainbowso I restarted both22:36
ade_leeyeah -- there must be a barbican-api22:37
ade_leeor something similar -- thats the important one22:37
JohnnyRainbowhmm...apache2?22:37
ade_leehmm .. maybe?  I don't know much about how ubuntu delivers this ..22:38
JohnnyRainbowok, I see it's wsgi and apache2 responsible for barbican-api22:39
ade_leeok22:39
JohnnyRainbowhttp://paste.openstack.org/show/800133/22:39
ade_leeso year restart apache22:40
JohnnyRainbowafter restart of apache, I'm back to issue with invalid cert: http://paste.openstack.org/show/800134/22:41
JohnnyRainbowI mean unsupported format...that what we fixed before :)22:42
ade_leeum .. the nssdb is still there, right?22:43
ade_leeand still has the cert in it?22:43
JohnnyRainbowit is http://paste.openstack.org/show/800135/22:44
JohnnyRainbowif I can believe this command :)22:44
ade_leeI wonder if this is a perms thing -- maybe apache can't read the directory/certdb?22:45
JohnnyRainbowdo  you mean /etc/barbican/alias?22:46
ade_leeyup22:46
JohnnyRainbowhmm...I added user www-data to barbican group22:51
JohnnyRainbowbut still issue exist22:51
ade_leeJohnnyRainbow, weird22:53
ade_leefirst off I would have expected to see the message that the db was already set up22:53
JohnnyRainbowI do not want to change owner of that directory, I believe barbican is ok and adding www-data to it, should be ok22:53
ade_leeJohnnyRainbow, yeah22:54
JohnnyRainbowhmm...actually there is a message like that, I just created error codes22:54
JohnnyRainbow2020-11-17 23:52:22.577 30011 INFO barbican.plugin.dogtag [req-7ca0c8dd-e3fb-4ee3-b1f9-8e03c2c5bc68 0d63c8861a124f4fbebe4170a9d59e61 175e079b3aef47a38da16d125863fd9d - default default] The nss_db_path provided already exists, so the database is assumed to be already set up.22:54
JohnnyRainbowthat is before, but that is INFO log :)22:54
ade_leeok - thats good -22:55
JohnnyRainbowsorry for not pasting everything22:55
ade_leeit would be useful to figure out where the exception is coming from ..22:55
JohnnyRainbownot really easy to inject something to dogtag.py code :)22:56
ade_leeJohnnyRainbow, well you can always add debug statements ..22:57
ade_leeJohnnyRainbow, basically in here -- https://opendev.org/openstack/barbican/src/branch/master/barbican/plugin/dogtag.py#L18422:58
JohnnyRainbowadd some log.INFO?22:59
ade_leeso statements of "I got here" before line 190 and 191 etc.22:59
ade_leeyup22:59
JohnnyRainbowlet me add23:00
ade_leeJohnnyRainbow, Ihave to head out - but if you can figure out where the exception is being thrown , we can figure out why23:00
JohnnyRainbowhmm...ok, thanks a lot for your help, seems my logs are not printed to api.log, even if I configured it like that: http://paste.openstack.org/show/800136/23:05
ade_leeJohnnyRainbow, hmm .. feel free to ping me tommorow once you figure out where the logs are --23:07
ade_leemaybe apache logs?23:07
JohnnyRainbowlet me see23:09
JohnnyRainbowok, I got the point, it seems function DogtagKRAPlugin is not even executed yet, as i.e. that log is printed LOG.info("The nss_db_path provided already exists, so the "23:17
JohnnyRainbowok, I need to think about it easily23:17
JohnnyRainbowthanks a lot for your help today, I will let you know about progress tomorrow23:17
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!