| *** jovon has joined #craton | 01:41 | |
| *** VW has joined #craton | 01:58 | |
| *** Mudpuppy has quit IRC | 02:44 | |
| *** Mudpuppy has joined #craton | 02:44 | |
| *** Mudpuppy has quit IRC | 02:49 | |
| *** VW has quit IRC | 02:53 | |
| *** VW has joined #craton | 02:57 | |
| *** VW has quit IRC | 03:01 | |
| *** VW has joined #craton | 03:03 | |
| *** VW has quit IRC | 03:05 | |
| *** VW has joined #craton | 03:06 | |
| *** Mudpuppy has joined #craton | 03:15 | |
| *** VW has quit IRC | 03:22 | |
| *** tojuvone has quit IRC | 03:28 | |
| *** jovon has quit IRC | 03:48 | |
| *** VW has joined #craton | 05:22 | |
| *** VW has quit IRC | 05:27 | |
| *** tojuvone has joined #craton | 05:51 | |
| *** Mudpuppy has quit IRC | 06:15 | |
| *** Mudpuppy has joined #craton | 06:16 | |
| *** Mudpuppy has quit IRC | 06:20 | |
| *** Mudpuppy has joined #craton | 10:19 | |
| *** Mudpuppy has quit IRC | 10:24 | |
| *** tojuvone has quit IRC | 10:25 | |
| *** tojuvone has joined #craton | 11:05 | |
| *** tojuvone has quit IRC | 11:28 | |
| *** tojuvone has joined #craton | 12:01 | |
| *** tojuvone has quit IRC | 12:01 | |
| *** tojuvone has joined #craton | 12:01 | |
| *** VW has joined #craton | 12:23 | |
| *** VW has quit IRC | 12:27 | |
| sulo | @all : i have updated my workflow bp with what i am doing | 12:39 |
|---|---|---|
| sulo | please review https://blueprints.launchpad.net/craton/+spec/craton-workflow-engine | 12:39 |
| *** VW has joined #craton | 13:40 | |
| *** VW has quit IRC | 13:41 | |
| *** VW has joined #craton | 13:50 | |
| *** VW has quit IRC | 13:57 | |
| *** VW has joined #craton | 13:57 | |
| *** valw has quit IRC | 14:04 | |
| *** VW has quit IRC | 14:08 | |
| *** Mudpuppy has joined #craton | 14:21 | |
| *** Mudpuppy has quit IRC | 14:27 | |
| sigmavirus | jimbaker: sulo git-harry chrisspencer meeting in #openstack-meeting-4 ? | 15:01 |
| *** VW has joined #craton | 15:04 | |
| *** Syed__ has joined #craton | 15:07 | |
| *** valw has joined #craton | 15:07 | |
| tojuvone | sulo: +1 https://blueprints.launchpad.net/craton/+spec/craton-workflow-engine | 15:25 |
| *** Mudpuppy has joined #craton | 15:28 | |
| *** jovon has joined #craton | 15:40 | |
| sulo | twitter has gone crazy, just sayin | 16:01 |
| sulo | brb | 16:03 |
| jovon | hey jimbaker | 16:12 |
| jovon | regarding https://bugs.launchpad.net/craton/+bug/1623095 | 16:12 |
| openstack | Launchpad bug 1623095 in craton "Refactor argument parsing, validation, error reporting in REST API " [High,New] - Assigned to Jim Baker (jimbaker) | 16:12 |
| jovon | i agree with splitting the bug for easier review and I'm willing to help you tackle that if you don't mind | 16:12 |
| jimbaker | jovon, that's definitely a megabug - a bug of indeterminate scope, and possibly never ending :) | 16:18 |
| jimbaker | having said that, when i reported the bug, we were seeing lots of problems due to inconsistency. i think it's better now, but we need to revisit and determine where it's happening | 16:20 |
| jimbaker | in the rest api itself? in the implementation of the db api? or in the calling client? | 16:20 |
| jimbaker | jovon, help in any part there will be very much appreciated | 16:21 |
| *** valw has quit IRC | 16:24 | |
| jovon | i think better exception handling in the db api will help us monitor and limit erros | 16:26 |
| jimbaker | jovon, definitely | 16:27 |
| jimbaker | one thing that comes to mind - we are not using the causation support provided by python 3.x | 16:28 |
| jovon | more verbose handling to serve as "reminders" to why this exception may have been raised and maybe point to future concerns | 16:28 |
| jimbaker | http://stackoverflow.com/a/16414892/423006 | 16:29 |
| jimbaker | gives a nice example of exception chaining | 16:30 |
| jimbaker | jovon, let's take a look at https://github.com/openstack/craton/blob/master/craton/db/sqlalchemy/api.py#L119 | 16:33 |
| jimbaker | which is a common example of the code pattern we can improve upon | 16:34 |
| jovon | im following | 16:34 |
| jimbaker | get_user_info has a try statement with the following exception handler horns | 16:34 |
| jimbaker | except sa_exc.NoResultFound: will result in being raised insteasd as exceptions.NotFound() | 16:35 |
| jimbaker | this is fairly reasonable for diagnosis, because it's 1-to-1 | 16:35 |
| jimbaker | however the catchall horn loses a lot of diagnostic information | 16:35 |
| *** valw has joined #craton | 16:35 | |
| jimbaker | all other exceptions are converted to UnknownException (status code 500) | 16:36 |
| jimbaker | all info is lost | 16:36 |
| jimbaker | now this is a good thing for the client - we don't want to expose all exceptions there | 16:36 |
| jimbaker | but we do want this in our log | 16:36 |
| jimbaker | (good thing = such things as, not exposing internal details that could leak security considerations) | 16:37 |
| jimbaker | so if instead we changed by raise exceptions to this pattern | 16:38 |
| jimbaker | raise exceptions.NotFound() from err | 16:39 |
| jimbaker | that would take advantage of the exception chaining. pretty simple | 16:39 |
| jimbaker | also i overstated it in this particular case | 16:39 |
| jimbaker | because we do wrap using the msg param in the second horn | 16:39 |
| jimbaker | raise exceptions.UnknownException(message=err) | 16:39 |
| *** valw has quit IRC | 16:40 | |
| jimbaker | then again, that could leak :) | 16:40 |
| jimbaker | anyway, it's almost certainly the case that we want to do exception chaining in such re-raise scenarios, especially at boundaries like dbapi | 16:41 |
| jimbaker | jovon, so that could be a good starting point in your investigation | 16:41 |
| jovon | most certainly | 16:43 |
| jovon | you bring a very valid point | 16:43 |
| *** valw has joined #craton | 16:51 | |
| *** valw has quit IRC | 17:39 | |
| *** valw has joined #craton | 17:48 | |
| *** harlowja has joined #craton | 17:50 | |
| *** valw has quit IRC | 17:53 | |
| jimbaker | sulo, so secrets... | 18:09 |
| sulo | yeah ? | 18:10 |
| jimbaker | we don't need them for pure inventory. but we do need for access to remote systems (virtualized variables) and workflows, unless the workers have all the necessary ssh keys installed on them | 18:10 |
| jimbaker | i don't know if i have really any good ideas here. but i will try this out as a strawman | 18:11 |
| jimbaker | we store encrypted secrets in our variables; https://bugs.launchpad.net/craton/+bug/1606885 | 18:12 |
| openstack | Launchpad bug 1606885 in craton "Variables should support secrets" [Undecided,New] - Assigned to Jim Baker (jimbaker) | 18:12 |
| jimbaker | let's assume the variable uses a well-known key. then we get the usual scoping behavior | 18:13 |
| jimbaker | so we get an encrypted secret that corresponds to a given resource as desired | 18:13 |
| jimbaker | so the question becomes, what's the process of decrypting that encrypted secret on demand | 18:14 |
| jimbaker | in the referenced bug, there's some handwaving that is roughly "barbican solves" | 18:15 |
| sulo | so you want to keep a salted version in the db .. and unsalt it on response if user requesting it has permission for it | 18:15 |
| jimbaker | it's not clear to me this is the case, other than barbican allows us to work with this - or possibly related secrets - on a HSM | 18:16 |
| jimbaker | similar capabilities in https://aws.amazon.com/cloudhsm/ | 18:16 |
| jimbaker | sulo, so yeah, we have some scheme to recover the original secret as necessary. the alternative is that we have delegation and corresponding trusts fully implemented. sure, that's going to be the case ;) | 18:17 |
| sulo | also i guess you are thinking of the access keys as the same | 18:18 |
| jimbaker | sulo, may just want to nuke the specific support for access keys | 18:18 |
| jimbaker | since the many-to-many we get with that is more or less the same as using scoping | 18:18 |
| jimbaker | and 1-to-1 is obviously the same | 18:18 |
| jimbaker | ok, back to part 1 of my proposal | 18:19 |
| jimbaker | so in the virtualized variables case, we have the user provide their unlocking secret in the request | 18:20 |
| jimbaker | this gets passed through, and when the corresponding secrets are accessed, this unlocks | 18:20 |
| sulo | ohhh .. i see what you are saying ... i thought of that slightly differntly | 18:21 |
| jimbaker | of course this means we have one master key per some set of credentials | 18:21 |
| jimbaker | i suppose we could do some variants | 18:21 |
| sulo | more in terms fo rbac .. so we keep the secret .. if person requesting has permission to the variable | 18:22 |
| sulo | we query barbican (or db) with the said secret | 18:22 |
| sulo | and send it back to the user | 18:22 |
| jimbaker | so i'm thinking we are not a barbican secret store - it should be one way for the user | 18:23 |
| jimbaker | they can put in secrets, but they do not need to retrieve. only use to do other work | 18:23 |
| sulo | yeah i guess thats a valid point .. we dont want to keep any secrets | 18:23 |
| jimbaker | also i would prefer if we did not keep any secrets that craton could use directly | 18:24 |
| sulo | i guess keeping even one opens a differnt set of problems | 18:24 |
| sulo | jimbaker: yeah that makes sense | 18:24 |
| jimbaker | so if the user always has to unlock, we are minimizing the attack surface | 18:24 |
| sulo | jimbaker: +1 i agree, so then it makes sense for user to pass it during the all | 18:25 |
| jimbaker | how the user keeps track of its secrets; or how say multiple users can coordinate... | 18:25 |
| sulo | i think that works for other stores also .. like pwsafe for example .. or somethign else .. each access to secret is decided by that system rather than us | 18:25 |
| jimbaker | well the first could be barbican/cloudhsm/vault | 18:25 |
| jimbaker | yeah | 18:25 |
| jimbaker | maybe we provide coordination support. i did briefly look at stuff like http://www.unix.com/man-page/debian/7/gfshare/ | 18:26 |
| jimbaker | alternatively https://cryptography.io/en/latest/fernet/#cryptography.fernet.MultiFernet | 18:28 |
| jimbaker | ok, so that works for user mediated interactions | 18:28 |
| jimbaker | with some possible subtleties (which we may or may not care about, because we let the user figure out) | 18:29 |
| jimbaker | and so long as we do things like lots of iterations for deriving the actual salt, we are good, and craton is not introducing new vulnerabilities :) | 18:30 |
| jimbaker | sulo, #2. what to do about workflows? | 18:30 |
| jimbaker | in this case, we have a long running workflow that needs access to the unlocking keys during the lifetime of the workflow | 18:31 |
| jimbaker | in particular, let's assume the api server bounced during the workflow | 18:31 |
| sulo | yeah thats getting tricky .. if workflow has separate identity to user .. iirc thats what the rbac assumes right | 18:37 |
| jimbaker | sulo, yeah | 18:42 |
| jimbaker | i mean we could derive workflow from a base class of User | 18:42 |
| jimbaker | instead of principal. but that doesn't really change things | 18:43 |
| jimbaker | it's the long running aspect that matters here | 18:43 |
| sulo | is that because decryption secret is lost in case of restart | 18:44 |
| jimbaker | so maybe at that point we simply (reversible) salt with a system secret, which is controlled by a craton admin | 18:44 |
| jimbaker | during the lifetime of the process | 18:45 |
| sulo | but wont that have to be stored somewhere ? | 18:45 |
| jimbaker | sulo, yes | 18:45 |
| jimbaker | but the vulnerability is limited to the workflow lifetime. maybe we could break up in some way too | 18:46 |
| sulo | not sure i am followig this tbh, how does per wf secret generated | 18:47 |
| sulo | jimbaker: ^ | 18:48 |
| jimbaker | let's try this scheme out. 1. for running workflow, generate usual high entropy symmetric key | 18:48 |
| jimbaker | 2. encrypt all needed secrets for the workflow in a temporary location | 18:49 |
| jimbaker | using said workflow key | 18:49 |
| jimbaker | admins could restart this process as necessary, by beginning at #1 again | 18:50 |
| jimbaker | so just a question of where to store the workflow key, etc | 18:50 |
| *** Mudpuppy_ has joined #craton | 18:52 | |
| *** openstackstatus has joined #craton | 18:53 | |
| jimbaker | sulo, so maybe we narrow the usual key distribution problem; and we still have to know what variables we actually need access to (so workflow metadata) | 18:54 |
| jimbaker | the first part feels handwavey to me. for the second: we can readily support the necessary metadata in the workflow def. and workflows should know the devices they are working with, so that's an easy join | 18:55 |
| *** Mudpuppy has quit IRC | 18:55 | |
| *** VW_ has joined #craton | 18:56 | |
| *** Mudpuppy_ has quit IRC | 18:56 | |
| *** VW_ has quit IRC | 18:56 | |
| *** VW_ has joined #craton | 18:57 | |
| jimbaker | sulo, so let's assume we have something like the following. key name is "secret:id_rsa" | 18:58 |
| jimbaker | so the secret:* prefix is an important aspect here in terms of mgmt. and we can do RBAC etc against it | 18:58 |
| *** VW_ has quit IRC | 18:59 | |
| *** VW has quit IRC | 18:59 | |
| *** VW__ has joined #craton | 18:59 | |
| jimbaker | for each device, we can resolve its secret:id_rsa, decrypt with the user supplied secret; then temporarily re-encrypt with the workflow-specific secret | 19:00 |
| *** VW__ has quit IRC | 19:00 | |
| *** VW has joined #craton | 19:00 | |
| jimbaker | the workflow worker pools have access to the workflow-specfic secret | 19:01 |
| sulo | jimbaker: sure | 19:01 |
| sulo | that should work i think | 19:01 |
| sulo | well lets try it atleast | 19:02 |
| jimbaker | and as necessary, we can rekey if we have to restart the pools. maybe do multifernet: admin + workflow secret | 19:02 |
| jimbaker | yeah | 19:02 |
| jimbaker | i don't think we are introducing new security holes here | 19:02 |
| jimbaker | clearly if we have to work with a secret for some length of time, we have to work with that secret for the length of time in question | 19:02 |
| jimbaker | tautological | 19:03 |
| jimbaker | so in the user case, it's request to response that defines | 19:03 |
| jimbaker | in the workflow case, it's workflow duration | 19:03 |
| jimbaker | so just a question of getting the seq diagram right | 19:04 |
| sulo | so what are secrets here ? | 19:05 |
| sulo | rsa keys | 19:05 |
| jimbaker | sulo, or AWS IAM supplied keys | 19:05 |
| sulo | passwords or user secrets to run playbooks | 19:05 |
| jimbaker | (which presumably are scope limited by proper usage of IAM) | 19:06 |
| jimbaker | or a nova credential | 19:06 |
| jimbaker | to get access about the hypervisor <=> VM mapping | 19:06 |
| jimbaker | etc | 19:06 |
| jimbaker | also anything ansible would store in a vault file | 19:07 |
| jimbaker | no idea if OSA uses ansible vault | 19:07 |
| jimbaker | palendae, any thoughts here? | 19:07 |
| palendae | We don't use it by default, iirc there's no issues with doing so though | 19:08 |
| jimbaker | yeah, sort of orthogonal i would think | 19:08 |
| palendae | There may be something with the wrapper script, but I haven't tried it myself and don't remember anyone talking about it otherwise | 19:09 |
| jimbaker | palendae, i would assume most secret usage is just with respect to ssh keys, right? | 19:10 |
| palendae | Service passwords | 19:10 |
| sulo | there is a ton of user secrets | 19:11 |
| palendae | A lot of them are generated though | 19:11 |
| palendae | https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/user_secrets.yml | 19:11 |
| jimbaker | right, which are just generated, but still have to be managed somewhere | 19:11 |
| jimbaker | after the fact | 19:11 |
| palendae | Plus https://github.com/openstack/openstack-ansible/blob/master/scripts/pw-token-gen.py | 19:11 |
| palendae | The generated file is just managed with file permissions by default | 19:12 |
| palendae | So, right now, no extra requirements from us on that front | 19:13 |
| jimbaker | palendae, yeah, so these secrets would be ones we want to slurp in and present as necessary | 19:13 |
| jimbaker | so let's just say we assumed the earlier namespace, secret:ironic_rabbitmq_password would be available for some level in the hierarchy | 19:15 |
| jimbaker | i don't know ironic to say definitively, but presumably having different passwords for say each cell would be fine | 19:16 |
| jimbaker | then the OSA inventory script would read from craton as necessary this password. if it's a user, they would get back the encrypted value (which they could decrypt with their user key). or if they are applying a role in a workflow, it would be decrypted on demand (through that rekeying procedure i mentioned earlier) | 19:18 |
| *** chrisspencer_ has joined #craton | 19:20 | |
| jimbaker | anyway, good discussion. i'm sure there's more to think about! | 19:20 |
| *** VW has quit IRC | 19:23 | |
| *** VW has joined #craton | 19:24 | |
| *** valw has joined #craton | 19:24 | |
| *** odyssey4me_ has joined #craton | 19:26 | |
| *** odyssey4me has quit IRC | 19:28 | |
| *** chrisspencer has quit IRC | 19:28 | |
| *** chrisspencer_ is now known as chrisspencer | 19:29 | |
| *** VW has quit IRC | 19:35 | |
| *** valw has quit IRC | 19:37 | |
| *** VW has joined #craton | 19:37 | |
| *** valw has joined #craton | 19:41 | |
| *** Mudpuppy has joined #craton | 19:54 | |
| *** valw has quit IRC | 19:56 | |
| *** VW has quit IRC | 19:58 | |
| *** VW has joined #craton | 20:02 | |
| *** valw has joined #craton | 20:10 | |
| *** valw has quit IRC | 20:15 | |
| *** jovon has quit IRC | 20:18 | |
| Syed__ | Need help with Craton routes test ... | 20:29 |
| Syed__ | sulo: let me know when you get sometime | 20:29 |
| *** VW has quit IRC | 20:38 | |
| *** VW has joined #craton | 20:39 | |
| *** valw has joined #craton | 20:44 | |
| *** VW has quit IRC | 21:03 | |
| *** VW has joined #craton | 21:10 | |
| *** valw has quit IRC | 21:18 | |
| *** VW has quit IRC | 21:31 | |
| *** VW has joined #craton | 21:32 | |
| *** VW has quit IRC | 22:01 | |
| *** VW has joined #craton | 22:04 | |
| *** VW has quit IRC | 22:04 | |
| *** VW has joined #craton | 22:05 | |
| *** VW_ has joined #craton | 22:26 | |
| *** VW has quit IRC | 22:30 | |
| *** VW_ has quit IRC | 22:31 | |
| *** valw has joined #craton | 22:35 | |
| *** VW has joined #craton | 22:37 | |
| *** valw_ has joined #craton | 22:39 | |
| *** valw has quit IRC | 22:40 | |
| *** VW has quit IRC | 22:41 | |
| *** VW has joined #craton | 23:10 | |
| *** VW has quit IRC | 23:15 | |
| jimbaker | harlowja, we are trying to figure out our secrets storage. for inventory itself we have a good solution, namely any secrets will be decrypted on the fly by a user provided master key (and provided presumably down to sqlalchemy-based layers through the session context... smop) | 23:31 |
| harlowja | cools | 23:31 |
| jimbaker | harlowja, but we are trying to come up with a scheme whereby any unlocked secrets can be provided to a workflow | 23:32 |
| jimbaker | eg ssh keys | 23:32 |
| jimbaker | of target hosts | 23:32 |
| jimbaker | so our thought is: 1) invoking the workflow will mean getting the user provided master key (just like inventory) | 23:33 |
| jimbaker | 2) a workflow specific key is generated, and any necessary secrets are rekeyed with it. so we can safely store the secrets in the db with minimum scope | 23:34 |
| jimbaker | 3) somehow we share the workflow specific key and ensure it has a limited lifetime | 23:34 |
| jimbaker | this may be complete junk btw... | 23:35 |
| jimbaker | so review on just that welcome | 23:35 |
| jimbaker | harlowja, here's the taskflow specific piece. what can we do to share this chunk of data with say the worker-based model ? | 23:36 |
| jimbaker | chunk of data = workflow-specific master key | 23:37 |
| harlowja | hmmmmm | 23:38 |
| jimbaker | i was thinking of leveraging zk in some way, namel the use of ephemeral nodes. that might provide enough robustness to share | 23:40 |
| jimbaker | anyway, it doesn't seem all that different from key distribution problems in general | 23:40 |
| harlowja | yup yup | 23:41 |
| jimbaker | but ideally we can just hook into how workflows operate | 23:41 |
| jimbaker | anyway, i just wanted to plant that seed :) | 23:42 |
| harlowja | kk | 23:42 |
| harlowja | will think abouts it :-P | 23:42 |
| jimbaker | btw, sulo has started to work on workflows for real, based on the work i did prior to summit, based on some code sulo implemented earlier | 23:42 |
| harlowja | woot | 23:42 |
| harlowja | for real! | 23:42 |
| harlowja | ha | 23:42 |
| jimbaker | we might call it a "hot potato" | 23:42 |
| jimbaker | harlowja, it had to happen at some time | 23:42 |
| harlowja | :) | 23:49 |
| *** VW has joined #craton | 23:52 | |
| *** VW has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!