18:02:26 <heckj> #startmeeting 18:02:27 <openstack> Meeting started Tue May 15 18:02:26 2012 UTC. The chair is heckj. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:28 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:56 <heckj> Big topic for today: rafaduran discussing https://bugs.launchpad.net/keystone/+bug/963098 and related blueprint 18:02:58 <uvirtbot> Launchpad bug 963098 in keystone "Keystone isn't acting on consecutive failed logins" [High,Triaged] 18:03:04 <heckj> #topic https://bugs.launchpad.net/keystone/+bug/963098 18:03:18 <heckj> why don't we start there - rafaduran, you good with that? 18:03:30 <rafaduran> no problem 18:04:03 <rafaduran> as bug https://bugs.launchpad.net/keystone/+bug/963098 reported keystone is not acting on consecutive login fails 18:04:04 <uvirtbot> Launchpad bug 963098 in keystone "Keystone isn't acting on consecutive failed logins" [High,Triaged] 18:04:29 <rafaduran> however I think the real problem is that keyhstone is no acting on any suspicious activity 18:04:42 <ayoung> \O/ 18:04:54 <rafaduran> thus I've registered a blueprint for that https://blueprints.launchpad.net/keystone/+spec/improve-keystone-security 18:05:13 <rafaduran> that show how I think keyston should manage this situation 18:05:33 <heckj> #link https://blueprints.launchpad.net/keystone/+spec/improve-keystone-security 18:05:43 <rafaduran> there is also a working draft for this https://review.openstack.org/#/c/7239/ 18:06:12 <ayoung> rafaduran, think that link is wrong 18:06:32 <rafaduran> ayoung: which one? 18:06:41 <ayoung> https://review.openstack.org/#/c/7239/ 18:06:50 <ayoung> I'm getting a 404 18:06:51 <joesavak> 404 on that 18:06:51 <heckj> https://review.openstack.org/#/c/7239/ is returning a "NotFound" 18:06:57 <joesavak> lol 18:07:01 <rafaduran> not for me 18:07:09 <dolphm> same 18:07:15 <dolphm> (404) 18:07:19 <ayoung> Its too secure 18:07:25 <liemmn> lol 18:07:30 <ayoung> rafaduran, is it draft only? 18:07:38 <rafaduran> ayoung: yes 18:07:48 <ayoung> I think you need to publish or perish 18:08:04 <rafaduran> published 18:08:16 <rafaduran> is working now? 18:08:28 <dolphm> yep 18:08:28 <joesavak> yup 18:08:38 <joesavak> so experation, tolerance, etc are global configs? 18:08:45 <ayoung> #link https://review.openstack.org/#/c/7239/ 18:08:53 <clarkb> you can also add specific people to drafts (if you still want to keep it mostly secret) 18:09:28 <rafaduran> joesavak: no they are specific for a middleware that send emails on 401 errors 18:10:04 <rafaduran> joesavak: that middleware is just an example but it also solves original bug 18:10:11 <joesavak> ok 18:10:51 <dolphm> i assume this all should be disabled out of the box? 18:11:25 <rafaduran> doesn't matter 18:12:09 <ayoung> dolphm, I think that the middle ware must be explicitly added to the pipeline in order to work 18:12:18 <ayoung> but by default it would be left out? 18:12:53 <rafaduran> If you think if should be disabled by default i can disable in etc/keystone.conf.sample 18:13:33 <rafaduran> if everything looks good I will add some tests and updated docs how to enable it 18:13:44 <ayoung> rafaduran, so is it "all or nothing"? 18:13:58 <ayoung> Or can you turn on a limited subset of Functionaliry 18:14:00 <ayoung> ity 18:14:51 <rafaduran> the backends are not activate unless a middleware use them 18:14:55 <dolphm> should be removed from the pipeline out of the box; doc'd as to how to enable it; most config should be commented out with assumed defaults in case it remains commented out; 'admin_emails' needs to be renamed, as 'admin' is very overloaded and the existing usage doesn't apply here 18:15:33 <rafaduran> dolphm: ok 18:16:24 <dolphm> not clear on what the 'secinfo' table is storing -- is it a log in sql? 18:17:31 <rafaduran> not it stores information about requests/responses, so a middleware can query later and thus track suspicious activity 18:17:33 <joesavak> id, response code, request mothod, path, date, response body - look like it's an audit trail 18:17:50 <joesavak> not sure what "extra" is. Just a col to store notes? 18:17:56 <dolphm> rafaduran: it doens't appear to store statistics, are they computed per request then? 18:19:03 <rafaduran> dolphm: yes, e.g: the MailConsecutive401 middleware store information if response is 401 and then query the 401 in a certain period 18:19:56 <rafaduran> dolphm: then if 401 errors are greater than a given number (tolerance) it will send an email 18:21:30 <joesavak> +1 - i like it 18:22:16 <dolphm> so the 'security' driver implements a queryable request/response log; the middleware provides email-based alerts (but doesn't actually throttle, right?) 18:22:43 <joesavak> right - not acting as repose or a rate-limit. Just notificaiton. Back-end could be atom-hopper 18:22:53 <rafaduran> yes 18:23:07 <rafaduran> the rate limit would be the second part 18:23:18 <zykes-> uhm, vishy still getting: /usr/bin/qemu-system-x86_64 -S -M pc-1.0 -no-kvm -m 4096 -smp 18:23:23 <zykes-> whyyyyyy 18:23:37 <rafaduran> but as I mentioned at bp ayoung at last meeting said that it probably conflict http work 18:23:41 <vishy> zykes-: this is the meeting room 18:23:43 <heckj> zykes-: wrong channel… :-) 18:23:44 <dolphm> zykes-: wrong channel? 18:23:50 <joesavak> raf - you may want to look at http://openrepose.org/ for rate-limiting 18:23:50 <rafaduran> so I don't work on that after httpd work is done 18:23:57 <zykes-> ah, wrong chan :) 18:24:26 <ayoung> rafaduran, maybe, but also seems like you could use either. 18:24:38 <ayoung> It might be possible to Rate limit this way 18:25:05 <rafaduran> ayoung: you mean using middlewares? 18:25:05 <ayoung> But... 18:25:31 <ayoung> if each request is handled by a separate process...it would need to be dealt with at the HTTPD level 18:26:06 <ayoung> so I think, while this might function OK, it wouldn't actually protect against a DOS or cracking attempt... 18:26:17 <ayoung> unless it hits a DB table each time? 18:26:51 <rafaduran> yes it queries DB for each request if response is 401 18:27:14 <ayoung> I like the idea of using middlewares for it. I think it will work ok then against the cracking attempt, just not against a DOS 18:27:25 <ayoung> but that is still better than nothing 18:27:29 <ayoung> by a long shot 18:27:34 <dolphm> rafaduran: if the response is 401, isn't it too late to rate limit? 18:27:45 <dolphm> the dos has already succeeded at that point 18:28:04 <rafaduran> dolphm: as I said right now I'm not doing rate limiting, just reporting 18:29:07 <joesavak> rate limiting should be split outside of keystone, imo 18:29:14 <dolphm> 'security' is probably too broad of a driver name, then... it's really just very specific monitoring 18:29:22 <liemmn> I think this will increase the network chattiness for the middleware quite a bit... When I think of rate limiting, it is better done at the server, like a rest proxy. Why can't we configure the server to optionally send notification, rather than involving the middleware? 18:29:28 <dolphm> i.e. there's no additional security 18:29:34 <rafaduran> dolphm: I like monitoring 18:29:46 <dolphm> rafaduran: very specific monitoring 18:30:36 <dolphm> liemmn: +1 18:31:41 <rafaduran> liemmm: I think the idea is keystone itself can handle it, but of course you can provide same thing using others approachs 18:32:06 <dolphm> heckj: thoughts 18:32:07 <dolphm> ? 18:32:38 <liemmn> I am just concerned that we need to keep the middleware "lean and mean"... A lot of token (and later signature) validation is going through it already... 18:32:43 <heckj> I think having an optional middleware that does ratelimiting is just fine 18:33:12 <heckj> If we also want to enable some simple audit&report mechanism, then I think that might be best as a separate middleware 18:33:41 <joesavak> fyi - rackspace developed this and made rate-limting configurable through a couple API calls. I can share some doc on that if it would help. 18:33:49 <heckj> per liemmn comment's we want to be very careful about keeping the /token API piece performance 18:33:57 <dolphm> heckj: that's what this is - audit & reporting via email 18:34:05 <liemmn> Audt trails have to be secure... It's easier to secure it on the keystone server than on a bunch of service nodes... (IMO) 18:34:30 <joesavak> liemn - +1 - especially since you're putting in the path which could include tokens 18:34:56 <dolphm> liemmn: referring to the 'secinfo' table as the audit trail? 18:36:03 <ayoung> heckj, that is why the PKI Blueprint...the fastest code is that which doesn't have to run.... 18:36:12 <liemmn> no... I think that is not the audit trail right? That's just for checking the rate of failure.... If we want to implement audit trails, I am just saying it should be done on the server side. 18:37:11 <heckj> ayoung: word 18:37:30 <dolphm> ayoung: i'm a fan :) 18:37:47 <liemmn> NOOP is the fastest :) 18:38:13 <liemmn> and bug-free 18:38:15 <ayoung> I updated the PKI Blueprint to make it clearer. I can talk through it if anyone wants 18:38:16 <dolphm> liemmn: noop takes a cycle :P 18:38:35 <ayoung> if we are done with rafaduran 's code? 18:38:47 <dolphm> ayoung: yeah, we can carry that into code review 18:38:50 <liemmn> dolphm: I am talking about code ;) 18:38:51 <heckj> rafaduran: did you get the feedback you were after? 18:39:14 <rafaduran> I think so, the reporting seems is not going to work 18:39:29 <rafaduran> but then we are not addressing the original bug 18:40:03 <dolphm> rafaduran: cite the bug in your commit msg, and clean up the Change-Id's plz 18:40:20 <dolphm> i didn't even know this was for a bug 18:40:24 <rafaduran> #link https://bugs.launchpad.net/keystone/+bug/963098 18:40:25 <uvirtbot> Launchpad bug 963098 in keystone "Keystone isn't acting on consecutive failed logins" [High,Triaged] 18:41:00 <rafaduran> I've explained it at the start 18:41:28 <dolphm> rafaduran: sorry, i jumped straight to the review 18:41:36 <rafaduran> np 18:42:37 <ayoung> Am I up? 18:42:37 <rafaduran> I think I will re-think this and try something not using middlewares 18:42:52 <rafaduran> ayoung: yes 18:42:57 <ayoung> OK updated the write up here: http://wiki.openstack.org/PKI specifically the section "Delegation and Scaling" 18:43:09 <heckj> #topic: PKI ness 18:43:15 <dolphm> lol 18:43:16 <heckj> #link http://wiki.openstack.org/PKI 18:43:33 <ayoung> I had to find out from some people smarter than me what the right API to use was 18:43:38 <ayoung> turns out it is called CMS 18:43:44 <ayoung> Crypt Message Syntax 18:43:50 <ayoung> the short of it is this 18:44:09 <ayoung> when you get a token, there is a part of the response that lists the tenant and roles 18:44:26 <ayoung> so the data is something like 18:44:58 <ayoung> {user: ayoung, tenant: coop-city, role: hallmonitor, groundskeeper} 18:45:20 <ayoung> you then take that date and sign it by encrypting it with a private key 18:45:47 <ayoung> someone else with the corresponding public key can decrypt it and know that it was encrypted by the private key 18:46:07 <ayoung> now this key-pair is going to be from Keystone, thus validating the name Keystone 18:46:36 <ayoung> and so when you get a token signed like this, something like Nova won't have to go back to Keystone in order to authenticate 18:47:05 <ayoung> now, the expiration date needs to be in there, and there are details about key management, but that the TL:DR version 18:47:10 <dolphm> (this is the exciting part) 18:47:38 <ayoung> dolphm, sorry, that was the exciting part. 18:47:51 <ayoung> the rest is boring details 18:47:52 <dolphm> ayoung: i meant nova not having to go back to keystone :) 18:48:04 <dolphm> ayoung: it's like magic scalability 18:48:39 <ayoung> it will put a little more CPU load on the services, but I suspect that what is saved in terms of network and SSL back to Keystone will more than make up for it 18:48:45 <dolphm> agree 18:48:49 <dolphm> ayoung: cpu time is cheap 18:49:04 <ayoung> If we really are concerned about CPU time for this, we've won... 18:49:37 <dolphm> ayoung: so, this all fits with the existing X-Subject-Token header, it'll just be encrypted? 18:49:41 <ayoung> My next task is to get a proof of concept working using the command line tools 18:49:44 <dolphm> err X-Auth-Token** 18:49:44 <ayoung> dolphm, yes 18:49:52 <heckj> ayoung: with the API pieces you've found, is it possible to have multiple signers 18:49:56 <ayoung> I think the X-Auth-Token will just be huge 18:50:10 <ayoung> heckj, yes 18:50:21 <dolphm> ayoung: how much bigger than the data it contains? 18:50:32 <ayoung> heckj, so say there are 3 keystone servers, each has a different key. That needs to be part of the token 18:50:34 <ayoung> dolphm, 1K 18:50:55 <ayoung> I think that is the nature of using the encryption 18:50:56 <dolphm> ayoung: is there a maximum length on header values? 18:51:06 <ayoung> I think we are OK. I'll confirm 18:51:16 <ayoung> but I've seen some that are huge 18:51:16 <dolphm> (first google: http://stackoverflow.com/questions/686217/maximum-on-http-header-values ) 18:51:50 <heckj> ayoung: I believe you're OK, and the token as it stands today is already a string (albiet much smaller). It makes a nice add-on extra-fit 18:52:04 <ayoung> dolphm, to answer your question "the HTTP spec does not define a limit, however many servers do by default. This means, practically speaking, the lower limit is 8K." 18:52:55 <ayoung> heckj, yes. The idea is that we should be able to switch out Keystone first, and then the rest. However, I don't know if /tokens/{id} will allow the super long ones either.... 18:53:09 <ayoung> but if we go with the scheme that token auth is in the page as opposed the URL we should be OK 18:53:24 <ayoung> and having tokens in the URL is problematic for other reasons already stated 18:53:41 <heckj> ayoung: yp, with ya 18:54:20 <dolphm> heckj: does this have potential to be core for v3? 18:54:32 <heckj> dolphm: absolutely 18:54:45 <dolphm> heckj: that would be awesome if we can do it 18:55:23 <ayoung> dolphm, I'll try to have a demo for next week, totally hand jammed, but should show the flow 18:55:36 <heckj> ayoung: that would be excellent 18:56:02 <heckj> We've got 5 minutes left. Any more questions or feedback for ayoung? 18:56:20 <ayoung> I'd like to use python-nss as the Crypto library, and it will need to have the CMS API added to it. CMS is in the native. Fortunatly, I've talked with the maintainer and he is more than willing to add it. 18:57:16 <heckj> #topic open discussion 18:57:47 <rafaduran> I've have some patches that need review again... 18:57:47 <heckj> Administrative bits - there's quite a number of reviews pending for Keystone, with lots of new/good work. Please go take a look and put in your thoughts on the code and such 18:57:57 <rafaduran> #link https://review.openstack.org/#/c/6425/ 18:58:06 <rafaduran> #link https://review.openstack.org/#/c/7127/ 18:58:07 <heckj> #link https://review.openstack.org/#/q/status:open+keystone,n,z 18:58:29 <heckj> I'm a bit behind on bug triage, but will be going through it this week as well 19:00:09 <heckj> And… that's it for this week! 19:00:13 <heckj> #endmeeting