17:00:17 #startmeeting keystone-office-hours 17:00:19 Meeting started Tue Aug 7 17:00:17 2018 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:00:20 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:00:22 The meeting name has been set to 'keystone_office_hours' 17:00:44 cmurphy: i suppose this is an inconvenient time for you 17:01:05 not at all 17:01:26 * lbragstad is all ears 17:01:30 so this is the bug that captures it i think https://bugs.launchpad.net/keystone/+bug/1470205 17:01:30 Launchpad bug 1470205 in OpenStack Identity (keystone) "Keystone IdP SAML metadata insufficient for websso flow" [Wishlist,Triaged] 17:01:44 ++ 17:02:09 though we're not in the business of being a webapp 17:02:24 when keystone is set up as a SAML IdP it isn't capable of doing the WebSSO profile, all it does is this very weird IdP-initiated ECP flow which is not a real profile 17:02:26 and the websso flow kinda requires that. 17:02:47 yeah 17:02:57 ahh 17:03:05 we'd need a form to login, and session cookies. 17:03:13 so we have gaps in our federated IdP implementation specifically 17:03:25 so, fwiw, with flask, i'll be moving us towards having an actual non-json page 17:03:31 if possible 17:03:44 which leads towards possibly being able to render websso things more cleanly 17:03:47 kmalloc: that would be really helpful 17:03:55 flask opens doors in ways webob was a REAL pain 17:04:13 i already have some logic for "accept header" variations and rendering different content 17:04:22 so, we can head down that path 17:04:30 which i think we needed for json/home 17:04:36 yes 17:04:47 in flask that was a lot easier to do than in webob 17:05:04 and once we're 100% on flask-restful, i'll be making a push to go to restplus 17:05:12 so we get swagger documentation 17:05:33 if we already have that, we can line up some views for web-rendering that can be webapp/sso flow compatible 17:05:51 so, think in S release we might be able to head down that path. 17:05:58 sounds reasonable 17:06:09 depending on how the rest of flask conversion goes 17:06:15 cmurphy: you have colleagues asking for this specifically? 17:06:24 or customers? 17:06:41 lbragstad: change of plans, use https://review.openstack.org/#/c/589546/ [if it passes check] as the base for /policy API 17:06:48 kmalloc: ack 17:06:54 lbragstad: i changed the way full_url and base_url work 17:07:07 so you can pass in a path to them and get /v3 added automatically 17:07:08 etc 17:07:12 lbragstad: colleages/project managers 17:07:37 ignoring the SAML part, the other part I have a hard time wrapping my head around is how a not-openstack application would know how to use keystone fernet tokens - i guess it would have to use keystonemiddleware or have some other kind of middleware that knows to redirect users to keystone for auth and requires the x-auth-token header for all requests 17:08:02 yeah... 17:08:15 and then what does it do with information about projects? what would not-openstack do with projects and role assignments 17:08:24 right 17:08:40 if we implemented openid connect, we would already be compatible with most apps 17:08:53 we also have a derivative of RBAC that's isn't exaclty an industry standard 17:09:09 the problem is the RBAC, more or less. 17:09:12 yeah 17:09:18 i agree 17:09:45 i think this same kinda problem came up when we talked about oauth2 support 17:11:30 i guess it depends on what's using keystone and what they are trying to protect (without sounds overly vague) 17:11:41 how are the resources grouped? 17:12:02 yeah 17:12:26 depending on the answers, could you generalize keystone enough to work for each? 17:14:02 oauth2.0 has the concept of scopes (think roles). but nothing really project based 17:15:29 i'm also not familiar with other implementations of scoped rbac like in openstack. 17:15:46 i'm not aware of any other implementations of scoped-rbac 17:16:06 i want to say scoped-rbac is openstack-specific 17:16:17 in that case, does the application care about the project at all? 17:16:29 which application? 17:16:40 external application that want to integrate with keystone 17:16:53 i think the answer to that depends on what the application is responsible for 17:16:55 if all they care is users and roles (and have no concept of scoped rbac), but just plain rbac 17:17:22 that should be doable in a generalizable way 17:17:30 knikolla: hmm so maybe there'd be a dummy project and the application only pays attention to the role name? 17:17:51 if the application in question is something like nova that provides APIs for managing all sorts of things across different authorization levels, then i think keystone needs to care about scoped-rbac 17:18:16 Hey guys. Had a quick question about Keystone + LDAP. I have 3 controllers configured and Keystone seems to be sending an authentication request from Horizon off to LDAP 3 times, assumingly one per controller. This is causing an issue because our lockout policy in LDAP is 3 bad passwords... so if a user enters 1 bad password we get an instant locko 17:18:16 ut. Is this expected behaviour? Why would keystone be sending the authentication request 3 times for the same request? 17:18:40 cmurphy: yeah, something like that 17:19:26 that's essentially what we started with, i think 17:19:59 deploy keystone with a single "level" of authorization and only use that layer 17:20:02 orange_julius: your haproxy should be configured to forward requests to just one keystone, three keystones making the same request at once would indeed all go to ldap 17:20:08 depends on how they want to do the integration. if they care about users, use a dummpy project. if they care about projects (as in the webapp provides services to extend an existing openstack deployemnt therefore project isolation matters), we can send the project_id as the username in the assertion. 17:20:23 <_ix> Maybe this is actually a quick question... if I'm using passwords, is there a reason that I'd specify `v3password` instead of `password`? 17:20:40 lbragstad: heh yeah i guess this sort of takes us back to pre-multitenancy 17:20:47 <_ix> I don't know why, but some of the services I'm working on have a v3password specified in the conf files. 17:21:50 _ix: in theory they should both be the same, `password` i think either looks at the url or does a discovery step to figure out what version to use 17:22:10 cmurphy: so the question is, in the webapp, do you want to act on behalf of the user (username in assertion = user_id, webapp will have no concept of projects). do you want to act on behalf of a project (username in assertion is project_id, webapp will have no concept of individual users) 17:22:16 <_ix> Ah. So if I'm explicit with the endpoint, I'll be in good shape. Thanks! 17:23:16 knikolla: that's a good way to look at it 17:23:18 knikolla: in the first example, i'm inclined to say we already support that pretty easily 17:24:08 * lbragstad feels like we're coming up with various deployment models 17:25:20 ++ 17:25:38 protecting apps the have no concept of scope or individual projects is a pretty low bar since most of that can be done between keystone and ksm 17:26:08 if an app starts caring about tenancy, then it becomes a question of where that logic lives, imo 17:26:56 lbragstad: in the case that the app cares about tenancy, that would probably be tied to an attribute/claim of the assertion. 17:27:12 we could make that configurable 17:28:04 Keycloak for example allows you to map user attributes -> claims for each specific client which is using it for auth. 17:28:44 but the app being protected needs to understand those claims, right? 17:29:11 assuming it's already using a standard openid connect / saml idp. it probably already does 17:31:59 we just need to document the mapping between keystone terminology -> openid connect / saml, and deployment strategies. 17:32:16 like the two cases i mentioned above. 17:32:29 and the resources managed by the webapp? 17:33:06 it's up to the webapp, based on the roles keystone says u have 17:33:11 not any different than what it is now 17:33:37 today - that's done by policy 17:33:42 or our specific policy engine 17:34:06 because that's where we say "instances are project resources, endpoints are system resources" 17:34:53 you can flatten that into the webapp checking instances require the project_member role, endpoint require the system_member role. 17:35:32 yeah - using a policy file is an implementation detail 17:35:37 yup 17:36:02 i guess what i'm seeing is that the resource can make a difference in that mapping depending on the app 17:37:29 lbragstad: you could also say that scope is an implementation detail 17:37:37 user has access to project which owns resource 17:38:03 user acts on behalf of project which owns resource. 17:38:43 like i don't think most openstack project care about users 17:38:54 they store resources with a project_id in their database 17:40:11 therefore given (project_id, role) a webapp has enough information to enforce policy 17:41:00 sure 17:41:01 i also remember the early discussions about making system a part of the project tree 17:43:59 true 17:44:29 we decided against that because we thought there was a difference between the resources in openstack's case 17:45:08 Could keystone be picking up the authentication requests from RabbitMQ? It doesn't seem like HAProxy is forwarding the requests to all 3 keystone servers 17:47:04 orange_julius: keystone doesn't support authentication via the message bus 17:47:34 hmmm ok. Didn't think so but am running out of ideas. 17:48:16 does the problem go away if you only route requests so a single keystone node? 17:51:43 Not sure yet. We have to schedule an outage for that sort of test. 17:57:40 i suppose you could try and recreate it by making the same request to a single keystone node, too 17:57:50 as opposed to taking nodes out of the pool 17:57:57 * lbragstad grabs lunch quick 17:59:18 lbragstad: our tests are still spewing the "scope check" fail message 17:59:25 lbragstad: do we have a fix for that? 17:59:49 this is a LOT of extra logging that seems like it is not actionable atm. 18:00:31 oslo_context. 18:01:40 because this makes me want to push a change to squash all logging from oslo_context 18:01:49 by default 18:16:24 Morgan Fainberg proposed openstack/keystone master: Allow wrap_member and wrap_collection to specify target https://review.openstack.org/589288 18:58:45 kmalloc: i had a wip series of patches for that 18:59:05 https://review.openstack.org/#/c/551337/ 18:59:11 and https://review.openstack.org/#/c/551411/1 19:28:35 kmalloc: i don't need to depend on https://review.openstack.org/#/c/589288/2 to convert the policy API? just https://review.openstack.org/#/c/589546/1 ? 19:35:48 So it doesn't look like HAProxy is forwarding the keystone authentication requests to all keystone controllers. I tested with a similiar configuration and saw an HTTP request hit just one HTTP server sitting behind haproxy with similiar configs 19:40:31 but the user is still getting locked out? 19:49:58 Lance Bragstad proposed openstack/oslo.policy master: Move _capture_stdout to a common place https://review.openstack.org/534440 19:50:47 Yes the user is still getting locked out 19:52:20 Lance Bragstad proposed openstack/keystone master: Update release notes for the rocky release https://review.openstack.org/589578 19:54:03 Yeah 19:54:32 lbragstad: just convert the api 19:54:51 ok - working on that now 19:55:02 lbragstad: and you can base on the patch preceeding that one you linked 19:55:36 The one that changes base_url and full_url 19:56:28 orange_julius: are your users in sql or ldap? 19:56:45 orange_julius: you're using the [security_compliance] section of keystone.conf, yeah? 19:57:24 Users are in active directory. Not using security_compliance. The only lockout policy is defined in active directory 19:57:37 ahh... 19:58:44 I see two possibilites. Let me know if I am wrong. Either a single keystone instance is sending multiple requests to AD, or all 3 keystone instances are seeing the login attempt and then asking AD for authorization 19:58:46 are you able to pin point which user is getting locked out? is it the user that's authenticating or the bind user being used to query ldap? 19:59:10 Its the user that is authenticating and its actually any user. We were able to replicate with my account (that sucked) 19:59:26 do you have a different value for that? 19:59:35 er - does your account have a different lockout setting? 20:00:04 No, its a "global" policy. 3 bad attempts 20:00:07 and just to confirm, this is what users are doing POST /v3/auth/tokens ? 20:00:20 s/what/when/ 20:00:50 Yes. well its through horizon and a POST /v3/auth/tokens is seen in the logs when they attempt. I believe we were seeing that come through the keystone.log on all three controllers 20:01:13 ok - that makes sense 20:01:24 horizon is building a request to keystone to get a token for the user trying to log in 20:01:45 and they are using usernames and passwords? 20:02:03 correct 20:02:35 orange_julius: which release are you using? 20:02:41 master, queens, pike? 20:03:21 newton actually =( 20:03:29 oh ok 20:03:47 luckily, i don't think the ldap bits of the identity driver have changed drastically since then 20:04:44 We will be turning on verbose logging on keystone + ldap which will hopefully help identify the problem. For now we don't have a whole lot unfortunately. Its a "production" system so we can't bounce things without an approval process 20:04:59 That won't happen until Friday though 20:05:24 right now does the user just get an 'Invalid user / password' error? 20:06:02 Correct. They log into their windows workstation, go to horizon, fatfinger the horizon password, and then their AD account is locked out 20:06:13 and receive bad password on horiozn 20:06:16 horizon* 20:07:30 horizon might be making multiple attempts 20:07:32 fwiw 20:08:48 at a high level - we start getting into that logic here https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/core.py#L907 20:09:19 which calls into the ldap specific implementation https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/core.py#L60-L78 20:09:56 and there are some common utilities for getting connections to ldap 20:09:58 https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/common.py#L1214-L1238 20:10:12 but afaict, i just see one connection getting initiatied? 20:11:27 Sweet that is going to be very helpful. Horizon access logs only show up on the controller with the VIP which is only showing one POST to /dashboard/auth/login. 20:13:18 orange_julius: do you know if you're setting this in your deployment? 20:13:19 https://github.com/openstack/keystone/blob/master/keystone/conf/ldap.py#L409-L417 20:13:41 No we are not. At least not explicitly 20:13:54 i wonder if that would have an impact 20:14:03 it attempts to reconnect 3 times before failing 20:14:17 if using connection pooling (which appears to be enabled by default) 20:14:40 so - if that's the case, that could be where your 3 attempts are getting used up 20:14:52 Would the fact that the POST to auth/tokens showing up on all controllers keystone.log indicate that keystone is probably making the request 3 times? one per controller node? 20:14:59 Wouldn't it only attempt to reconnect if it failed? 20:15:47 right - that's the part i'm curious about 20:16:07 if the password is wrong, are we attempting to reconnect? 20:16:22 and burning password attempts accidentally? 20:16:29 Ahh I see 20:17:00 So we could set that to... 50. and have our max lockouts be also set at 50 and receive the same result 20:17:02 orange_julius: do you have the ability to stand up a test node and point it to your adfs? 20:17:09 orange_julius: or just disable it.. 20:17:17 yes 20:18:04 I'll start standing up a test infra to validate. Won't be able to test disabling until Friday at the earliest 20:18:14 sure - that makes sense 20:18:26 just to be clear, i have no idea if this is actually the problem :) 20:18:31 but it looks suspicious 20:18:45 and might just be an unfortunate coincidence 20:19:07 that our default for that particular setting happens to match your lockout setting in AD 20:19:57 i guess if you have read access to your AD deployment, a single keystone node would be able to recreate this 20:20:19 by keeping those default set, and trying to authenticate using POST /v3/auth/tokens with the wrong password 20:20:51 No worries. Its worth investigating for sure. Is the retry logic around those links you sent earlier? I see it is supposed to raise an invalid credentials error, but not sure how that is handled 20:20:53 https://developer.openstack.org/api-ref/identity/v3/index.html#password-authentication-with-unscoped-authorization 20:21:01 Doug Hellmann proposed openstack/oslo.limit master: add python 3.6 unit test job https://review.openstack.org/589599 20:21:18 Doug Hellmann proposed openstack/oslo.policy master: add python 3.6 unit test job https://review.openstack.org/589603 20:21:57 we use another library for ldappool functionality, but i think if retries fail, that exception bubbles up to the user 20:22:51 https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/common.py#L25 20:23:35 https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/common.py#L708-L716 20:24:05 and we pass that configuration option into the constructor for that object https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/common.py#L711 20:27:52 Ok thanks. I have some things to investigate =D 20:27:59 hopefully it helps 20:29:08 i'm curious to know if that's actually the problem 20:30:01 I will report back once I have the test case. 20:30:30 ++ 20:31:24 except ldap.LDAPError as error: I wonder if an ldap authentication error is of type LDAPError 20:31:27 HmmmmMMmMM 20:36:00 https://github.com/python-ldap/python-ldap/blob/e8cc39db0990bfb56e95c6ae1bd2f8be10e88683/Doc/reference/ldap.rst#exceptions 20:36:02 It is! 20:37:18 https://github.com/openstack/ldappool/blob/master/ldappool/__init__.py#L248-L265 20:37:47 248-https://git.openstack.org/cgit/openstack/ldappool/tree/ldappool/__init__.py#n248-265 20:37:50 damnit bragstad 20:38:01 =P 20:38:47 Yea so that is totally what is happening right? LDAPError is the base class, so that exception will catch all errors including auths. ldappool will retry 3 times in .3 seconds and lock a user out instantly 20:39:14 it's looking to be more and more the case 20:39:37 totally and unfortunate combination of configuration options between two systems lol 20:39:46 haha 20:39:51 I'm just surprised this hasn't come up before 20:39:59 yeah... me, to 20:40:03 too* 20:40:17 but i guess it depends on what people are setting their locks out to on AD 20:40:47 if they set it to 5, then the likelihood of hitting this drops significantly (i would assume) 20:42:17 Yea seems like it. I'm going to set up a VM and play with the connection pooling settings to see if I can replicate. If I can... what do you want me to submit for this? 20:42:47 i mean - we could document this as a bug and just write down the coincidence and mark it as invalid 20:43:12 that would make it discoverable in case other people are randomly hitting this 20:43:29 otherwise, i'm not sure we can adjust the defaults in keystone, just because what would you choose? 20:44:06 it becomes a guessing game as to what you'd expect a reasonable retry to be and assuming that's what people are doing 20:44:18 but the net would be 20:44:29 Well why would you ever retry on a bad password? 20:44:40 I guess that would be something for ldappool 20:44:45 right 20:44:57 you could teach ldappool to try and distinguish the difference 20:45:32 so - a bug to ldappool is probably more appropriate 20:45:41 I think getting it documented would be a great start. You are right, you can't increase the number of retries or you might break more people. 3 is fairly low already 20:46:01 we can just document the workaround in the ldappool bug 20:46:18 Sounds good to me 20:47:06 "if you're relying ldap to enforce user lock outs, just be sure to set retry_max to 0 until ldappool knows how to distinguish different authentication failure types" 20:47:33 (y) 20:47:38 otherwise you risk locking users out prematurely 20:48:36 You also could introduce strange behaviors which might have been masked by the retries, but its better than an instant lock out.. especially when people have to call to get their account unlocked =p 20:49:38 right - i can see that being totally frustrating 20:49:58 want me to open a bug or have you already started? 20:50:40 I can open it if you are busy, but I havn't started yet. 20:51:20 kmalloc: we're putting ldappool bugs here - yeah? https://bugs.launchpad.net/ldappool 20:52:09 kmalloc: i think so 20:52:17 kmalloc: i mean, that is where i'd put them ;) 20:52:19 me too, we just haven't had one opened yet ;) 20:52:35 oh man nice! A first! 20:53:26 * lbragstad cringes watching ldappool paint suffer it's first scratch 20:54:21 i'll leave the honors to orange_julius heh 20:58:35 Maybe I'm just dumb, but I don't see a button to report bugs. Its usually on the far rightyea? 20:58:50 https://bugs.launchpad.net/ldappool/+filebug 20:59:05 w00t thanks 20:59:19 you should see a Report Bug hiding in the upper right corning of the page 21:09:07 Thanks for helping troubleshoot this lbragstad. Much appreciated 21:10:14 Submitted: https://bugs.launchpad.net/ldappool/+bug/1785898 21:10:14 Launchpad bug 1785898 in ldappool "Connection Pooling Retries Failed Passwords" [Undecided,New] 21:10:36 I had to reallllly restrain myself from putting "first" somewhere in there =P 21:15:07 Morgan Fainberg proposed openstack/keystone master: Convert regions API to flask native dispatching https://review.openstack.org/589640 21:15:08 Morgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching https://review.openstack.org/589641 21:15:08 Morgan Fainberg proposed openstack/keystone master: Convert endpoints api to flask native dispatching https://review.openstack.org/589642 21:15:24 lbragstad: ^ annnnnnnnd there is another subsystem done. 21:15:35 lbragstad: i.. i think we're down to the "really hard"(tm) ones 21:24:01 sweet 21:24:32 orange_julius: anytime - i'll wait to confirm the bug once we know it's recreateable with your system 21:29:36 Sounds good. I'll probably get around to reproducing tomorrow in a test environment and then hopefully we can our change request approved for Friday. Once I confirm it there I'll update the bug 21:30:02 hopefully we can get our change request approved** 21:32:48 ++ 23:16:29 Merged openstack/keystone master: Convert OS-AUTH1 paths to flask dispatching https://review.openstack.org/588064 23:53:20 Morgan Fainberg proposed openstack/keystone master: Convert regions API to flask native dispatching https://review.openstack.org/589640 23:54:18 Morgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching https://review.openstack.org/589641 23:54:40 Morgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching https://review.openstack.org/589641 00:37:58 Morgan Fainberg proposed openstack/keystone master: Convert endpoints api to flask native dispatching https://review.openstack.org/589642 00:38:39 Morgan Fainberg proposed openstack/keystone master: Convert regions API to flask native dispatching https://review.openstack.org/589640 00:38:50 Morgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching https://review.openstack.org/589641 00:38:55 Morgan Fainberg proposed openstack/keystone master: Convert endpoints api to flask native dispatching https://review.openstack.org/589642 02:36:04 Boxiang Zhu proposed openstack/python-keystoneclient master: refactor the getid method in keystoneclient/base.py https://review.openstack.org/589689 09:35:39 OpenStack Release Bot proposed openstack/python-keystoneclient master: Update reno for stable/rocky https://review.openstack.org/589791 10:07:06 Vishakha Agarwal proposed openstack/keystone master: Implement Trust Flush via keystone-manage. https://review.openstack.org/589378 10:25:58 Vishakha Agarwal proposed openstack/keystone master: Implement Trust Flush via keystone-manage. https://review.openstack.org/589378 13:40:03 knikolla: have you been following http://lists.openstack.org/pipermail/edge-computing/2018-August/000379.html at all? 13:40:06 just curious 13:41:08 that call is happening now fyi 13:42:28 or no it's the other edge group i guess 13:44:58 there hasn't been a time set for the keystone-specific one has there? 13:45:28 i don't know 13:46:50 i'll assume not since no one has replied to the thread yet 13:48:01 ildikov: ^ 13:48:04 lbragstad: cmurphy: today's call is an OPNFV one 13:48:14 ack 13:48:18 yeah i got them confused 13:48:29 lbragstad: cmurphy: I've just wanted to ping you and knikolla to ask whether anyone would be available tomorrow :) 13:49:00 we would like to start evaluating options that we collected regarding how to use Keystone in edge scenarios 13:49:21 and in addition we can plan for the PTG and follow up testing and demo activities 13:49:25 i think i have another meeting 13:49:34 i can sit in tomorrow, but i'm going to be traveling all day friday 13:49:49 and as no one besides the keystone team is an expert of Keystone at this moment we need someone from this team ideally 13:50:05 lbragstad: the most votes are for 1300 UTC tomorrow 13:50:31 lbragstad: would be a Zoom call as that's what people seem to prefer these days, but we usually try to take notes on the edge IRC channel as well 13:50:31 oh i should be able to do 1300 13:50:48 cmurphy: basically the same time as the OPNFV call today 13:51:01 it's probably a first of a series 13:51:12 i can make that work for tomorrow i think 13:51:52 ildikov: i've added a slot for edge related discussions to the federation section of our PTG etherpad https://etherpad.openstack.org/p/keystone-stein-ptg 13:52:15 (line 52 - 53) 13:53:02 lbragstad: awesome, I wanted to ask you about that too :) 13:53:25 according to the PTG schedule, we have monday set aside for x-project related things 13:53:33 thursday and friday are supposed to be project specific 13:53:42 lbragstad: I will try to get the StarlingX people and the Edge Computing Group people in sync about this 13:53:57 but if there is a time that works better for other, let me know or just jot it down in the etherpad 13:54:10 (i don't plan on formatting it into an actual schedule until the end of this month) 13:54:28 lbragstad: those groups meet one on Tuesday the other one on Wednesday so if we cannot make Monday work we still have one room each following day to figure it out 13:55:03 lbragstad: sounds great, I will let both groups know about this so that we can plan 13:55:22 without getting ahead of myself, i was going to try and keep tuesday open to deal with fallout of monday cross-project work 13:55:36 but maybe i can tag along in the edge discussions with folks on wednesday 13:55:48 and if keystone comes up, it comes up 13:56:45 we can try to position this for Monday and then figure it out when to follow up if we need to 13:56:53 would that sound good? 13:59:25 sure 14:01:00 cool, I will write up a mail about it to the ML's and will keep you posted about how things go 14:01:35 did you have any time estimation for the topic or we have some flexibility on that? 14:05:20 i expect we'll have some flexibility - i imagine the major x-project discussions to be unified limits and policy stuff 14:06:44 ildikov: If it wasn't a zoom call, I still can't get dialed into those at all, I'd be able to join 14:07:17 Basically, zoom doesn't work at all for me. :( 14:07:38 I've tried everything short of running a windows VM at this point. 14:08:00 I guess I need to do that if I were to want to join. 14:08:25 it works surprisingly well for me 14:09:07 It plain doesn't run/work on my 18.04 laptop, can't install on my RHEL one, fedora it crashes on. 14:09:08 i think i just had to download a plugin when i clicked on the meeting link 14:09:18 The plug-in is terrible. 14:09:28 And that is my problem. 14:10:50 They really need to ditch the plug-in and go with a browser-pluginless solution. 14:12:15 it made me download a full on desktop client 14:12:49 which annoyed me but it works so ¯\_(ツ)_/¯ 14:13:00 huh - maybe that's what i did (version 2.0.115900.1201) 14:13:19 Apparently they have a browser only option as of August 2018 14:13:26 heh 14:13:28 I'll try again tomorrow. 14:13:42 What time was it? 14:13:57 1300 utc i think 14:14:16 Oh, nope, I won't make it 14:14:30 6am calls are not in the plan for me 14:15:18 Hazards of West coast living when apparently everyone else in the community is not in a compatible time zone. 14:17:03 Oh well, I can weigh in on ML and/or after the fact as needed :) 14:17:55 ildikov: sorry, I can't make 6am video calls, but trust lbragstad, cmurphy, and knikolla to be good representation for keystone :) 14:21:09 kmalloc: we have people from China, which kills using Hangouts and I only have Zoom account besides that 14:22:13 kmalloc: I'm open to any suggestion for any other time when the time slot is more convenient for you as Zoom works pretty well for those who can access it and I didn't heard other complaints about client issues, etc 14:22:40 kmalloc: so I'm happy to adopt, but I haven't tested too many open tools to be able to come up with alternative that I trust to be stable enough 14:22:56 s/adopt/adapt/ 14:25:18 i used mumble for a while 14:25:23 but that's audio only 14:39:26 cool, I'll keep that in mind just in case 14:45:48 So I can't seem to replicate the keystone+ldap issue with a single keystone instance configured. I am starting to wonder if the issue is with the HA or with how the dashboard passing the authentication to keystone. I am going to try to install the dashboard and see if I can get it to replicate in the test environment 14:53:25 The ldappool connector is created using the designated bind user in the domain configuration is it not? Not the user information that would be passed via the Horizon dashboard 14:54:22 when i was reading the code, it looked like it was the user that was attempting to authenticate 14:54:27 but i could be wrong 15:02:46 ildikov: zoom is less of an issue ATM, I'll run a VM if needed, but timing is a bigger issue, I really can't do before about 1530 utc 15:03:34 kmalloc: fair enough, as we have people from APAC that wasn't really an option 15:03:59 kmalloc: I will try to look into alternating slots and also making sure that the info makes its way out to the ML's 15:04:27 kmalloc: we just don't have a big group of people so I'm trying my best to keep them on one call to quick things off at least 15:23:56 ildikov: yep, and you have knikolla lbragstad and cmurphy, so changing for.me.isnt important 15:25:03 kmalloc: I don't think you're less important than others, would be better if you could make it too! 15:25:25 kmalloc: I blame time zones :) 15:54:08 Lance Bragstad proposed openstack/keystone master: Convert policy API to flask https://review.openstack.org/589950 15:54:27 kmalloc: ^ that's going to fail 18 tests, but wanted to make sure i was doing things right 15:54:41 that just moves the /policy api and doesn't touch the endpoint policy API, yet 17:21:03 Is there anything I should be aware of with Keystone + RabbitMQ that might cause wierdness? How does keystone use rabbitmq? 17:49:46 orange_julius: keystone only uses rabbit for outgoing notifications 17:50:10 Hmm ok thanks 17:50:20 for example, when certain events happen in keystone, it can place a notification on the message bus for other things to consume 17:50:29 (e.g. you want to know when project resources are updated) 17:51:57 otherwise - keystone doesn't consume anything off the message bus.. 17:52:33 in fact, message queues are entirely optional for keystone from a deployment perspective 17:55:59 Ok just making sure. Thanks 17:56:50 yep - are you able to recreate the bug using your test env? 17:57:46 Not yet. I'm trying to find out what is different. A simple test of vanilla keystone with LDAP doesn't appear to show the same symptoms.. Although actually... I havn't tried it from the cmdline on the prod env yet.. 17:58:42 so - you've backed your test keystone to the same AD but you're not locked out when you try and login to horizon using a bad password? 17:59:25 I don't have horizon set up yet. I've been testing with the openstack cmdline tools. 'source rc_file; openstack project list' 17:59:37 but yes. it is not locking me out 17:59:48 interesting 18:00:14 Also interesting tidbit. On the test environment I am receiving a "bad password" while doing the cmdline tool test. In prod I receive a "The request you have made requires authentication" message. 18:01:02 That may be what it is returning when keystone finds a locked user in AD though 18:06:48 * lbragstad spins up a new instances to test with openldap 18:08:13 After looking more a tldappool and ldap/common.py it really seems like this should be the issue.... 18:12:53 i'm curious if i can recreate this using openldap 18:13:03 http://www.openldap.org/doc/admin24/overlays.html#Password%20Policies 18:14:40 Should work in theory 18:37:02 Cannot replicate with x3 keystones + haproxy either 18:38:06 Have not tried Galera replication yet. Would that effect it in some way? I wouldn't think so.. 18:51:31 lbragstad: commented, minor nits mostly 18:51:55 lbragstad: one thing that warrants a -1, and it's about needing to refactor the deprecated decorator 18:58:05 Morgan Fainberg proposed openstack/keystone master: Convert endpoints api to flask native dispatching https://review.openstack.org/589642 19:01:31 orange_julius: no - i wouldn't either 19:03:25 kmalloc: ack - thanks for the review 19:14:27 lbragstad: i found a minor data leak bug in our role API 19:14:46 lbragstad: i think. 19:14:53 * kmalloc reads this one more time 19:15:42 it is possible to determine if a role exists [by id] unauthenticated or even if you're authenticated but not allowed to get roles. 19:16:45