Monday, 2016-12-05

*** jovon has joined #craton01:41
*** VW has joined #craton01:58
*** Mudpuppy has quit IRC02:44
*** Mudpuppy has joined #craton02:44
*** Mudpuppy has quit IRC02:49
*** VW has quit IRC02:53
*** VW has joined #craton02:57
*** VW has quit IRC03:01
*** VW has joined #craton03:03
*** VW has quit IRC03:05
*** VW has joined #craton03:06
*** Mudpuppy has joined #craton03:15
*** VW has quit IRC03:22
*** tojuvone has quit IRC03:28
*** jovon has quit IRC03:48
*** VW has joined #craton05:22
*** VW has quit IRC05:27
*** tojuvone has joined #craton05:51
*** Mudpuppy has quit IRC06:15
*** Mudpuppy has joined #craton06:16
*** Mudpuppy has quit IRC06:20
*** Mudpuppy has joined #craton10:19
*** Mudpuppy has quit IRC10:24
*** tojuvone has quit IRC10:25
*** tojuvone has joined #craton11:05
*** tojuvone has quit IRC11:28
*** tojuvone has joined #craton12:01
*** tojuvone has quit IRC12:01
*** tojuvone has joined #craton12:01
*** VW has joined #craton12:23
*** VW has quit IRC12:27
sulo@all : i have updated my workflow bp with what i am doing12:39
suloplease review https://blueprints.launchpad.net/craton/+spec/craton-workflow-engine12:39
*** VW has joined #craton13:40
*** VW has quit IRC13:41
*** VW has joined #craton13:50
*** VW has quit IRC13:57
*** VW has joined #craton13:57
*** valw has quit IRC14:04
*** VW has quit IRC14:08
*** Mudpuppy has joined #craton14:21
*** Mudpuppy has quit IRC14:27
sigmavirusjimbaker: sulo git-harry chrisspencer meeting in #openstack-meeting-4 ?15:01
*** VW has joined #craton15:04
*** Syed__ has joined #craton15:07
*** valw has joined #craton15:07
tojuvonesulo: +1 https://blueprints.launchpad.net/craton/+spec/craton-workflow-engine15:25
*** Mudpuppy has joined #craton15:28
*** jovon has joined #craton15:40
sulotwitter has gone crazy, just sayin16:01
sulobrb16:03
jovonhey jimbaker16:12
jovonregarding https://bugs.launchpad.net/craton/+bug/162309516:12
openstackLaunchpad bug 1623095 in craton "Refactor argument parsing, validation, error reporting in REST API " [High,New] - Assigned to Jim Baker (jimbaker)16:12
jovoni agree with splitting the bug for easier review and I'm willing to help you tackle that if you don't mind16:12
jimbakerjovon, that's definitely a megabug - a bug of indeterminate scope, and possibly never ending :)16:18
jimbakerhaving said that, when i reported the bug, we were seeing lots of problems due to inconsistency. i think it's better now, but we need to revisit and determine where it's happening16:20
jimbakerin the rest api itself? in the implementation of the db api? or in the calling client?16:20
jimbakerjovon, help in any part there will be very much appreciated16:21
*** valw has quit IRC16:24
jovon i think better exception handling in the db api will help us monitor and limit erros16:26
jimbakerjovon, definitely16:27
jimbakerone thing that comes to mind - we are not using the causation support provided by python 3.x16:28
jovonmore verbose handling to serve as "reminders" to why this exception may have been raised and maybe point to future concerns16:28
jimbakerhttp://stackoverflow.com/a/16414892/42300616:29
jimbakergives a nice example of exception chaining16:30
jimbakerjovon, let's take a look at https://github.com/openstack/craton/blob/master/craton/db/sqlalchemy/api.py#L11916:33
jimbakerwhich is a common example of the code pattern we can improve upon16:34
jovonim following16:34
jimbakerget_user_info has a try statement with the following exception handler horns16:34
jimbakerexcept sa_exc.NoResultFound: will result in being raised insteasd as exceptions.NotFound()16:35
jimbakerthis is fairly reasonable for diagnosis, because it's 1-to-116:35
jimbakerhowever the catchall horn loses a lot of diagnostic information16:35
*** valw has joined #craton16:35
jimbakerall other exceptions are converted to UnknownException (status code 500)16:36
jimbakerall info is lost16:36
jimbakernow this is a good thing for the client - we don't want to expose all exceptions there16:36
jimbakerbut we do want this in our log16:36
jimbaker(good thing = such things as, not exposing internal details that could leak security considerations)16:37
jimbakerso if instead we changed by raise exceptions to this pattern16:38
jimbakerraise exceptions.NotFound() from err16:39
jimbakerthat would take advantage of the exception chaining. pretty simple16:39
jimbakeralso i overstated it in this particular case16:39
jimbakerbecause we do wrap using the msg param in the second horn16:39
jimbakerraise exceptions.UnknownException(message=err)16:39
*** valw has quit IRC16:40
jimbakerthen again, that could leak :)16:40
jimbakeranyway, it's almost certainly the case that we want to do exception chaining in such re-raise scenarios, especially at boundaries like dbapi16:41
jimbakerjovon, so that could be a good starting point in your investigation16:41
jovonmost certainly16:43
jovonyou bring a very valid point16:43
*** valw has joined #craton16:51
*** valw has quit IRC17:39
*** valw has joined #craton17:48
*** harlowja has joined #craton17:50
*** valw has quit IRC17:53
jimbakersulo, so secrets...18:09
suloyeah ?18:10
jimbakerwe don't need them for pure inventory. but we do need for access to remote systems (virtualized variables) and workflows, unless the workers have all the necessary ssh keys installed on them18:10
jimbakeri don't know if i have really any good ideas here. but i will try this out as a strawman18:11
jimbakerwe store encrypted secrets in our variables; https://bugs.launchpad.net/craton/+bug/160688518:12
openstackLaunchpad bug 1606885 in craton "Variables should support secrets" [Undecided,New] - Assigned to Jim Baker (jimbaker)18:12
jimbakerlet's assume the variable uses a well-known key. then we get the usual scoping behavior18:13
jimbakerso we get an encrypted secret that corresponds to a given resource as desired18:13
jimbakerso the question becomes, what's the process of decrypting that encrypted secret on demand18:14
jimbakerin the referenced bug, there's some handwaving that is roughly "barbican solves"18:15
suloso you want to keep a salted version in the db .. and unsalt it on response if user requesting it has permission for it18:15
jimbakerit's not clear to me this is the case, other than barbican allows us to work with this - or possibly related secrets - on a HSM18:16
jimbakersimilar capabilities in https://aws.amazon.com/cloudhsm/18:16
jimbakersulo, so yeah, we have some scheme to recover the original secret as necessary. the alternative is that we have delegation and corresponding trusts fully implemented. sure, that's going to be the case ;)18:17
suloalso i guess you are thinking of the access keys as the same18:18
jimbakersulo, may just want to nuke the specific support for access keys18:18
jimbakersince the many-to-many we get with that is more or less the same as using scoping18:18
jimbakerand 1-to-1 is obviously the same18:18
jimbakerok, back to part 1 of my proposal18:19
jimbakerso in the virtualized variables case, we have the user provide their unlocking secret in the request18:20
jimbakerthis gets passed through, and when the corresponding secrets are accessed, this unlocks18:20
suloohhh .. i see what you are saying ... i thought of that slightly differntly18:21
jimbakerof course this means we have one master key per some set of credentials18:21
jimbakeri suppose we could do some variants18:21
sulomore in terms fo rbac .. so we keep the secret .. if person requesting has permission to the variable18:22
sulowe query barbican (or db) with the said secret18:22
suloand send it back to the user18:22
jimbakerso i'm thinking we are not a barbican secret store - it should be one way for the user18:23
jimbakerthey can put in secrets, but they do not need to retrieve. only use to do other work18:23
suloyeah i guess thats a valid point .. we dont want to keep any secrets18:23
jimbakeralso i would prefer if we did not keep any secrets that craton could use directly18:24
suloi guess keeping even one opens a differnt set of problems18:24
sulojimbaker: yeah that makes sense18:24
jimbakerso if the user always has to unlock, we are minimizing the attack surface18:24
sulojimbaker: +1 i agree, so then it makes sense for user to pass it during the all18:25
jimbakerhow the user keeps track of its secrets; or how say multiple users can coordinate...18:25
suloi think that works for other stores also .. like pwsafe for example .. or somethign else .. each access to secret is decided by that system rather than us18:25
jimbakerwell the first could be barbican/cloudhsm/vault18:25
jimbakeryeah18:25
jimbakermaybe we provide coordination support. i did briefly look at stuff like http://www.unix.com/man-page/debian/7/gfshare/18:26
jimbakeralternatively https://cryptography.io/en/latest/fernet/#cryptography.fernet.MultiFernet18:28
jimbakerok, so that works for user mediated interactions18:28
jimbakerwith some possible subtleties (which we may or may not care about, because we let the user figure out)18:29
jimbakerand so long as we do things like lots of iterations for deriving the actual salt, we are good, and craton is not introducing new vulnerabilities :)18:30
jimbakersulo, #2. what to do about workflows?18:30
jimbakerin this case, we have a long running workflow that needs access to the unlocking keys during the lifetime of the workflow18:31
jimbakerin particular, let's assume the api server bounced during the workflow18:31
suloyeah thats getting tricky .. if workflow has separate identity to user .. iirc thats what the rbac assumes right18:37
jimbakersulo, yeah18:42
jimbakeri mean we could derive workflow from a base class of User18:42
jimbakerinstead of principal. but that doesn't really change things18:43
jimbakerit's the long running aspect that matters here18:43
sulois that because decryption secret is lost in case of restart18:44
jimbakerso maybe at that point we simply (reversible) salt with a system secret, which is controlled by a craton admin18:44
jimbakerduring the lifetime of the process18:45
sulobut wont that have to be stored somewhere ?18:45
jimbakersulo, yes18:45
jimbakerbut the vulnerability is limited to the workflow lifetime. maybe we could break up in some way too18:46
sulonot sure i am followig this tbh, how does per wf secret generated18:47
sulojimbaker: ^18:48
jimbakerlet's try this scheme out. 1. for running workflow, generate usual high entropy symmetric key18:48
jimbaker2. encrypt all needed secrets for the workflow in a temporary location18:49
jimbakerusing said workflow key18:49
jimbakeradmins could restart this process as necessary, by beginning at #1 again18:50
jimbakerso just a question of where to store the workflow key, etc18:50
*** Mudpuppy_ has joined #craton18:52
*** openstackstatus has joined #craton18:53
jimbakersulo, so maybe we narrow the usual key distribution problem; and we still have to know what variables we actually need access to (so workflow metadata)18:54
jimbakerthe first part feels handwavey to me. for the second: we can readily support the necessary metadata in the workflow def. and workflows should know the devices they are working with, so that's an easy join18:55
*** Mudpuppy has quit IRC18:55
*** VW_ has joined #craton18:56
*** Mudpuppy_ has quit IRC18:56
*** VW_ has quit IRC18:56
*** VW_ has joined #craton18:57
jimbakersulo, so let's assume we have something like the following. key name is "secret:id_rsa"18:58
jimbakerso the secret:* prefix is an important aspect here in terms of mgmt. and we can do RBAC etc against it18:58
*** VW_ has quit IRC18:59
*** VW has quit IRC18:59
*** VW__ has joined #craton18:59
jimbakerfor each device, we can resolve its secret:id_rsa, decrypt with the user supplied secret; then temporarily re-encrypt with the workflow-specific secret19:00
*** VW__ has quit IRC19:00
*** VW has joined #craton19:00
jimbakerthe workflow worker pools have access to the workflow-specfic secret19:01
sulojimbaker: sure19:01
sulothat should work i think19:01
sulowell lets try it atleast19:02
jimbakerand as necessary, we can rekey if we have to restart the pools. maybe do multifernet: admin + workflow secret19:02
jimbakeryeah19:02
jimbakeri don't think we are introducing new security holes here19:02
jimbakerclearly if we have to work with a secret for some length of time, we have to work with that secret for the length of time in question19:02
jimbakertautological19:03
jimbakerso in the user case, it's request to response that defines19:03
jimbakerin the workflow case, it's workflow duration19:03
jimbakerso just a question of getting the seq diagram right19:04
suloso what are secrets here ?19:05
sulorsa keys19:05
jimbakersulo, or AWS IAM supplied keys19:05
sulopasswords or user secrets to run playbooks19:05
jimbaker(which presumably are scope limited by proper usage of IAM)19:06
jimbakeror a nova credential19:06
jimbakerto get access about the hypervisor <=> VM mapping19:06
jimbakeretc19:06
jimbakeralso anything ansible would store in a vault file19:07
jimbakerno idea if OSA uses ansible vault19:07
jimbakerpalendae, any thoughts here?19:07
palendaeWe don't use it by default, iirc there's no issues with doing so though19:08
jimbakeryeah, sort of orthogonal i would think19:08
palendaeThere may be something with the wrapper script, but I haven't tried it myself and don't remember anyone talking about it otherwise19:09
jimbakerpalendae, i would assume most secret usage is just with respect to ssh keys, right?19:10
palendaeService passwords19:10
sulothere is a ton of user secrets19:11
palendaeA lot of them are generated though19:11
palendaehttps://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/user_secrets.yml19:11
jimbakerright, which are just generated, but still have to be managed somewhere19:11
jimbakerafter the fact19:11
palendaePlus https://github.com/openstack/openstack-ansible/blob/master/scripts/pw-token-gen.py19:11
palendaeThe generated file is just managed with file permissions by default19:12
palendaeSo, right now, no extra requirements from us on that front19:13
jimbakerpalendae, yeah, so these secrets would be ones we want to slurp in and present as necessary19:13
jimbakerso let's just say we assumed the earlier namespace, secret:ironic_rabbitmq_password would be available for some level in the hierarchy19:15
jimbakeri don't know ironic to say definitively, but presumably having different passwords for say each cell would be fine19:16
jimbakerthen the OSA inventory script would read from craton as necessary this password. if it's a user, they would get back the encrypted value (which they could decrypt with their user key). or if they are applying a role in a workflow, it would be decrypted on demand (through that rekeying procedure i mentioned earlier)19:18
*** chrisspencer_ has joined #craton19:20
jimbakeranyway, good discussion. i'm sure there's more to think about!19:20
*** VW has quit IRC19:23
*** VW has joined #craton19:24
*** valw has joined #craton19:24
*** odyssey4me_ has joined #craton19:26
*** odyssey4me has quit IRC19:28
*** chrisspencer has quit IRC19:28
*** chrisspencer_ is now known as chrisspencer19:29
*** VW has quit IRC19:35
*** valw has quit IRC19:37
*** VW has joined #craton19:37
*** valw has joined #craton19:41
*** Mudpuppy has joined #craton19:54
*** valw has quit IRC19:56
*** VW has quit IRC19:58
*** VW has joined #craton20:02
*** valw has joined #craton20:10
*** valw has quit IRC20:15
*** jovon has quit IRC20:18
Syed__Need help with Craton routes test ...20:29
Syed__sulo: let me know when you get sometime20:29
*** VW has quit IRC20:38
*** VW has joined #craton20:39
*** valw has joined #craton20:44
*** VW has quit IRC21:03
*** VW has joined #craton21:10
*** valw has quit IRC21:18
*** VW has quit IRC21:31
*** VW has joined #craton21:32
*** VW has quit IRC22:01
*** VW has joined #craton22:04
*** VW has quit IRC22:04
*** VW has joined #craton22:05
*** VW_ has joined #craton22:26
*** VW has quit IRC22:30
*** VW_ has quit IRC22:31
*** valw has joined #craton22:35
*** VW has joined #craton22:37
*** valw_ has joined #craton22:39
*** valw has quit IRC22:40
*** VW has quit IRC22:41
*** VW has joined #craton23:10
*** VW has quit IRC23:15
jimbakerharlowja, we are trying to figure out our secrets storage. for inventory itself we have a good solution, namely any secrets will be decrypted on the fly by a user provided master key (and provided presumably down to sqlalchemy-based layers through the session context... smop)23:31
harlowjacools23:31
jimbakerharlowja, but we are trying to come up with a scheme whereby any unlocked secrets can be provided to a workflow23:32
jimbakereg ssh keys23:32
jimbakerof target hosts23:32
jimbakerso our thought is: 1) invoking the workflow will mean getting the user provided master key (just like inventory)23:33
jimbaker2) a workflow specific key is generated, and any necessary secrets are rekeyed with it. so we can safely store the secrets in the db with minimum scope23:34
jimbaker3) somehow we share the workflow specific key and ensure it has a limited lifetime23:34
jimbakerthis may be complete junk btw...23:35
jimbakerso review on just that welcome23:35
jimbakerharlowja, here's the taskflow specific piece. what can we do to share this chunk of data with say the worker-based model ?23:36
jimbakerchunk of data = workflow-specific master key23:37
harlowjahmmmmm23:38
jimbakeri was thinking of leveraging zk in some way, namel the use of ephemeral nodes. that might provide enough robustness to share23:40
jimbakeranyway, it doesn't seem all that different from key distribution problems in general23:40
harlowjayup yup23:41
jimbakerbut ideally we can just hook into how workflows operate23:41
jimbakeranyway, i just wanted to plant that seed :)23:42
harlowjakk23:42
harlowjawill think abouts it :-P23:42
jimbakerbtw, sulo has started to work on workflows for real, based on the work i did prior to summit, based on some code sulo implemented earlier23:42
harlowjawoot23:42
harlowjafor real!23:42
harlowjaha23:42
jimbakerwe might call it a "hot potato"23:42
jimbakerharlowja, it had to happen at some time23:42
harlowja:)23:49
*** VW has joined #craton23:52
*** VW has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!