Wednesday, 2016-08-17

*** code-R has joined #openstack-keystone00:01
*** spzala has joined #openstack-keystone00:02
*** code-R_ has joined #openstack-keystone00:03
*** su_zhang has quit IRC00:04
*** tqtran has quit IRC00:05
*** code-R has quit IRC00:06
*** tqtran has joined #openstack-keystone00:06
*** spzala has quit IRC00:07
openstackgerritMerged openstack/keystone: Trust controller refactoring
openstackgerritTin Lam proposed openstack/keystone: api-ref: Document domain specific roles
*** roxanaghe has quit IRC00:20
*** hockeynut has joined #openstack-keystone00:22
*** GB21 has joined #openstack-keystone00:31
*** adrian_otto has joined #openstack-keystone00:39
*** thumpba_ has quit IRC00:39
*** thumpba has joined #openstack-keystone00:39
*** adrian_otto has quit IRC00:43
*** thumpba has quit IRC00:44
*** gyee has quit IRC00:46
*** tqtran has quit IRC00:51
*** chlong has joined #openstack-keystone00:58
*** hockeynut has quit IRC01:02
*** hockeynut has joined #openstack-keystone01:02
*** spzala has joined #openstack-keystone01:04
*** woodster_ has quit IRC01:09
*** spzala has quit IRC01:09
openstackgerritJamie Lennox proposed openstack/keystone: Move audit initiator creation to request
*** adrian_otto has joined #openstack-keystone01:34
*** GB21 has quit IRC01:35
*** adrian_otto has quit IRC01:39
*** code-R_ has quit IRC01:42
*** EinstCrazy has joined #openstack-keystone01:42
*** BjoernT has joined #openstack-keystone01:44
*** julim has joined #openstack-keystone01:46
*** BjoernT has quit IRC01:49
*** tqtran has joined #openstack-keystone01:53
*** esp has quit IRC01:57
*** tqtran has quit IRC01:57
*** davechen has joined #openstack-keystone02:00
*** EinstCrazy has quit IRC02:01
*** EinstCrazy has joined #openstack-keystone02:03
*** asettle has joined #openstack-keystone02:10
*** sdake has quit IRC02:13
*** sdake has joined #openstack-keystone02:13
*** EinstCra_ has joined #openstack-keystone02:16
*** GB21 has joined #openstack-keystone02:16
*** julim has quit IRC02:17
*** asettle has quit IRC02:17
*** EinstCrazy has quit IRC02:18
*** adrian_otto has joined #openstack-keystone02:27
*** spzala has joined #openstack-keystone02:29
*** spzala has quit IRC02:30
*** spzala has joined #openstack-keystone02:30
*** adrian_otto has quit IRC02:33
*** haplo37_ has quit IRC02:43
*** spzala has quit IRC02:44
*** spzala has joined #openstack-keystone02:44
*** GB21 has quit IRC02:44
*** spzala has quit IRC02:48
*** EinstCrazy has joined #openstack-keystone02:56
*** su_zhang has joined #openstack-keystone02:59
*** EinstCra_ has quit IRC03:00
*** thumpba has joined #openstack-keystone03:19
*** su_zhang has quit IRC03:20
*** adrian_otto has joined #openstack-keystone03:23
*** su_zhang has joined #openstack-keystone03:23
*** esp has joined #openstack-keystone03:25
*** adrian_otto has quit IRC03:28
*** su_zhang has quit IRC03:31
*** sdake has quit IRC03:34
*** wangqun has joined #openstack-keystone03:42
stevemaranyone want to take a look at ?03:44
patchbotstevemar: patch 350704 - keystone - Make all token provider behave the same with trusts03:44
openstackgerritAnh Tran proposed openstack/keystone: api-ref: Fix parameters attributes
*** links has joined #openstack-keystone03:54
*** tqtran has joined #openstack-keystone03:55
*** dikonoor has joined #openstack-keystone03:57
*** tqtran has quit IRC03:59
*** hockeynut has quit IRC04:03
*** GB21 has joined #openstack-keystone04:06
*** su_zhang has joined #openstack-keystone04:16
*** GB21 has quit IRC04:23
*** code-R has joined #openstack-keystone04:44
*** code-R_ has joined #openstack-keystone04:49
*** code-R has quit IRC04:52
*** GB21 has joined #openstack-keystone04:54
*** adrian_otto has joined #openstack-keystone05:10
*** jaosorior has joined #openstack-keystone05:13
*** adrian_otto has quit IRC05:14
*** itisha has quit IRC05:20
*** code-R_ has quit IRC05:24
*** code-R has joined #openstack-keystone05:24
*** code-R has quit IRC05:30
*** su_zhang has quit IRC05:39
*** thumpba has quit IRC05:40
*** rcernin has joined #openstack-keystone05:46
*** thumpba has joined #openstack-keystone05:51
*** thumpba has quit IRC05:56
*** asettle has joined #openstack-keystone06:00
*** asettle has quit IRC06:05
*** roxanaghe has joined #openstack-keystone06:16
*** roxanagh_ has joined #openstack-keystone06:16
*** roxanaghe has quit IRC06:20
*** roxanagh_ has quit IRC06:22
*** adriant has quit IRC06:37
*** code-R has joined #openstack-keystone06:38
*** code-R has quit IRC06:43
*** code-R has joined #openstack-keystone06:51
*** belmoreira has joined #openstack-keystone06:53
*** adrian_otto has joined #openstack-keystone06:58
*** markvoelker has quit IRC07:00
*** tesseract- has joined #openstack-keystone07:00
*** adrian_otto has quit IRC07:03
*** code-R has quit IRC07:07
*** GB21 has quit IRC07:10
*** amoralej|off is now known as amoralej07:26
*** GB21 has joined #openstack-keystone07:44
*** jamielennox is now known as jamielennox|away07:45
*** pnavarro has joined #openstack-keystone07:48
*** thumpba has joined #openstack-keystone07:52
*** thumpba has quit IRC07:57
*** zzzeek has quit IRC08:00
*** markvoelker has joined #openstack-keystone08:00
*** thumpba has joined #openstack-keystone08:01
*** jed56 has joined #openstack-keystone08:01
*** zzzeek has joined #openstack-keystone08:01
*** markvoelker has quit IRC08:05
*** thumpba has quit IRC08:06
openstackgerritDavanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c
*** belmoreira has quit IRC08:18
*** asettle has joined #openstack-keystone08:22
*** rdopiera has joined #openstack-keystone08:31
*** dkehn_ has quit IRC08:32
bretonmorning, keystone08:34
*** Jaison has joined #openstack-keystone08:35
*** links has quit IRC08:37
openstackgerrithenry-nash proposed openstack/keystone: Add expand, data migration and contract logic to keystone-manage
*** adrian_otto has joined #openstack-keystone08:47
*** jamielennox|away is now known as jamielennox08:50
*** adrian_otto has quit IRC08:51
*** dkehn_ has joined #openstack-keystone08:52
*** GB21 has quit IRC08:52
*** asettle has quit IRC08:59
*** markvoelker has joined #openstack-keystone09:01
*** asettle has joined #openstack-keystone09:01
*** asettle has quit IRC09:02
*** thumpba has joined #openstack-keystone09:02
*** asettle has joined #openstack-keystone09:03
*** markvoelker has quit IRC09:06
*** GB21 has joined #openstack-keystone09:06
*** thumpba has quit IRC09:08
*** tqtran has joined #openstack-keystone09:19
*** asettle has quit IRC09:20
*** gus has quit IRC09:22
alogastevemar: are you around for a short discussion about oidc?09:23
*** tqtran has quit IRC09:23
*** maestropandy has joined #openstack-keystone09:29
*** maestropandy has left #openstack-keystone09:30
*** code-R has joined #openstack-keystone09:30
*** mvk has quit IRC09:35
*** asettle has joined #openstack-keystone09:36
*** asettle has quit IRC09:38
*** jamielennox is now known as jamielennox|away09:43
*** spzala has joined #openstack-keystone09:45
*** spzala has quit IRC09:49
*** amakarov_away is now known as amakarov09:49
*** dikonoor has quit IRC09:56
*** openstack has joined #openstack-keystone10:15
*** sdake_ has joined #openstack-keystone10:16
*** sdake_ has quit IRC10:18
*** sdake_ has joined #openstack-keystone10:19
*** sdake has quit IRC10:19
*** dkehn_ has joined #openstack-keystone10:22
*** adrian_otto has joined #openstack-keystone10:35
*** adrian_otto has quit IRC10:40
*** EinstCrazy has quit IRC10:40
stevemarmorning breton10:43
stevemaraloga: i am semi-here now10:43
alogastevemar: Hi there o/10:44
stevemarhidy ho10:44
alogastevemar: I have been testing and working on the OpenID Connect and Keystone integration10:46
alogastevemar: so far, this works perfectly when using the dashboard, the user browser and mod_auth_openidc10:46
stevemargood to hear...10:46
alogastevemar: since the OpenID Connect client is Apache, and it gets the access token and id token from the oidc server, and then it gets the user information from the user information endpoint10:47
alogastevemar: so we can map any additional claims (for example email)10:48
alogastevemar: this works flawless10:48
alogastevemar: the problem here comes with the CLI...10:48
alogai.e. openstackclient + keystoneauth110:48
alogain this case, in the CLI we obtain an access token from the oidc server10:49
alogathen we exchange this token with keystone, to obtain a keystone token10:49
alogathis exchange is done in a OAuth 2.0 protected url (/v3/OS-FEDERATION/identity_providers/<idp>/protocols/<protocol>/auth), since this is not oidc but oauth10:50
alogathis works ok as well10:50
alogaas long as you only rely in the access token claims to do the mapping10:50
alogasince the URL is proceted using oauth20 (i.e AuthType oauth2 from mod_auth_openidc) there is no way to fetch the user information from the user information endpoint, since this is somethig specific of openid connect10:53
*** asettle has joined #openstack-keystone10:55
stevemarlet me re-read that10:56
*** mvk has quit IRC10:58
alogastevemar: the problem here is who is the OpenID Connect client11:01
alogain the horizon case, it is the apache server where keystone is running, therefore the openid connect client gets the user information and it is able to pass it down to the keystone app11:01
alogain the CLI case, the oidc client is keystoneauth1, therefore it is not able to fetch this information11:02
alogawell, it is able to fetch it, but it is useless as it cannot be passed to keystone11:02
stevemaraloga: i thought you could setup mod_auth_oidc to do introspection to get user details?11:02
alogastevemar: yes, indeed11:02
alogastevemar: but only introspection on the access_token11:02
alogai.e. you won't be able to get any additional claims (email, preferred_username, etc.)11:03
*** marekd2 has joined #openstack-keystone11:09
stevemarbut you'll still get the base claims?11:14
alogayes, so to be clear, if the mapping rules only rely on the access token claims (iss, sub, etc.) this works11:14
alogahowever, if somebody wants to create some additional mapping rules based on the OpenID Connect user info claims, this won't work11:14
stevemar(via the CLI)11:16
stevemarthats frustrating11:16
alogastevemar: yes11:16
stevemaraloga: anyway to make additional calls at the keystoneauth level?11:16
stevemarto get additional claims11:16
alogastevemar: yes, we could get the user claims since the user info endpoint should be protected using oauth 2.0, and we can use the access token to access it11:17
alogai.e. we only need to make an additional call to fetch that information11:18
alogahowever, how can we pass this to keystone?11:18
*** sdake_ is now known as sdake11:24
alogastevemar: one possibility is that keystone is the one fetching the user information11:25
alogastevemar: before the mapping11:25
openstackgerrithenry-nash proposed openstack/keystone: Add expand, data migration and contract logic to keystone-manage
stevemarhenrynash: i PMed you, not sure if you're around, or your laptop is busy processing 100 notifications from all your open apps11:30
*** jaosorior has quit IRC11:35
*** jaosorior has joined #openstack-keystone11:36
*** su_zhang has joined #openstack-keystone11:39
*** amoralej is now known as amoralej|lunch11:40
stevemaraloga: not sure if we can add that to keystone, it's pretty specific to the protocol right11:44
alogastevemar: for the record, if I am not mistaken, google is adding the user information to the response obtained from the oauth 2.0 introspection endpoint11:46
alogastevemar: however that's not the case for all the oauth providers, as the specification does not force it
stevemarwe (as keystone) could just tell folks to talk to whoever implements their OIDC :P11:48
stevemarbut they may even be in the same organization or workplace11:48
alogastevemar: regarding keystone doing the call to the userinfo endpoint that's oidc specific, but we could implement an auth plugin for that11:51
alogainstead of using the mapped plugin directly11:51
aloga(i.e. oidc = keystone.auth.plugins.mapped.Mapped)11:51
stevemaraloga: so, a slightly different topic, but let me know if it makes sense11:52
alogago ahead11:52
stevemaraloga: dstanek wants to add code to keystone to handle all the SAML requests, so he doesn't have to use mod_auth_saml in apache11:52
stevemarhe wants to use the pysaml, offer up an API, and handle everything with pysaml instead of mod_auth_saml11:53
stevemarthe reasoning behind this is that it'll scale better with automation11:53
stevemarmanaging the apache configuration is not fun11:54
stevemarwould a similar idea be useful for oidc?11:54
alogastevemar: yes, indeed11:54
stevemaryou could put your additional user info call there :P11:55
alogastevemar: the problem with Apache, as I see it, is that you need to configure things at two sides: keystone (i.e. the mapping) and Apache (i.e. the OpenID connect configuration)11:56
alogaif you need to manage several oidc providers, that's a bit messy11:56
stevemaraloga: yeah, thats where dstanek and rax/intel are having their issues11:57
alogamoreover, if you need to add the proctected urls with oauth20 for the introspection...11:57
stevemarthey can't do that easily with automation11:57
*** julim has joined #openstack-keystone12:01
*** sdake_ has joined #openstack-keystone12:01
*** code-R_ has joined #openstack-keystone12:03
*** sdake has quit IRC12:05
*** sdake_ is now known as sdake12:05
*** code-R has quit IRC12:06
*** su_zhang has quit IRC12:07
*** su_zhang has joined #openstack-keystone12:07
stevemaraloga: i don't have an easy answer for you :)12:08
alogastevemar: none of the solutions are perfect12:08
alogastevemar: some require hard work, some are not too clean12:09
*** markvoelker has joined #openstack-keystone12:09
stevemaraloga: i'm guessing this impacts you highly?12:09
stevemaraloga: you have a mapping that depends on additional claims now in the default claim12:10
alogastevemar: well, i need to check, maybe we can survive for the moment12:10
alogabut this is part of a larger project, were we would need for sure more complex mappings12:10
alogalike mappings based on groups available in the additional claims12:11
*** jamielennox|away is now known as jamielennox12:11
stevemaraloga: i'm guessing when i did this it wasn't the same :(12:12
*** su_zhang has quit IRC12:12
alogastevemar: that's more or less the configuration that we have12:14
*** rodrigods has quit IRC12:14
*** rodrigods has joined #openstack-keystone12:14
alogathe problem is that the server is not returning the additional claims in the instrospect response12:14
*** jpena is now known as jpena|lunch12:15
alogaunfortunately that depends on the server :(12:15
bretonlol @ bug 161406912:19
openstackbug 1614069 in OpenStack Identity (keystone) "API v2.0 responds with HTTP 200 when trying to add a non-existent user to a project" [Undecided,New]
*** marekd2 has quit IRC12:20
*** marekd2 has joined #openstack-keystone12:21
stevemarbreton: unfotuntately i think we removed the validation that checks for users and projects when creating a role assignment12:23
stevemarand we never put them back12:23
stevemarbreton: abandon: ? :)12:24
patchbotstevemar: patch 347876 - keystone - test12:24
bretonstevemar: done, thanks12:25
*** marekd2 has quit IRC12:25
*** GB21 has quit IRC12:28
*** raildo has joined #openstack-keystone12:29
stevemarnow i can't find that patch12:31
*** marekd2 has joined #openstack-keystone12:31
*** spzala_ has joined #openstack-keystone13:01
*** EinstCrazy has joined #openstack-keystone13:03
*** gordc has joined #openstack-keystone13:05
*** mvk has joined #openstack-keystone13:06
*** amoralej|lunch is now known as amoralej13:12
*** su_zhang has joined #openstack-keystone13:13
*** jpena|lunch is now known as jpena13:18
*** tqtran has joined #openstack-keystone13:20
*** gagehugo has joined #openstack-keystone13:21
*** EinstCrazy has quit IRC13:22
*** tqtran has quit IRC13:25
*** eandersson_ has joined #openstack-keystone13:30
*** woodster_ has joined #openstack-keystone13:34
*** edmondsw has joined #openstack-keystone13:37
henrynashanyone have an understanding of oslo_db test_base DBFixtures? We use them for test_sql_upgrade and am getting weird interaction between test classes (like a schema of one database type seemingly still in place when test class for a different db comes around)13:39
*** ametts has joined #openstack-keystone13:40
*** Jaison has quit IRC13:46
*** sdake has quit IRC13:50
*** thumpba has joined #openstack-keystone13:52
stevemarhenrynash: maybe bknudson ?14:00
openstackgerritGage Hugo proposed openstack/keystone: [api] add relationship links to v3-ext
*** nishaYadav has joined #openstack-keystone14:00
stevemargagehugo: change the commit message to "Closes-Bug" :)14:00
henrynashstevemar: yep was going to ask him....he is our liason after all :-)14:00
bknudsonhenrynash: not sure I follow -- database type? Like mysql vs postgresql?14:00
*** sdake_ has joined #openstack-keystone14:01
openstackgerritGage Hugo proposed openstack/keystone: [api] add relationship links to v3-ext
gagehugostevemar: done14:01
stevemargagehugo: ty14:01
henrynashbknudson: yep...I have 3 classes (each which runs a set of tests), one for sqlite, mysql and postgres14:01
henrynashbknudson: each work fine if I run them on their own14:02
gagehugostevemar: some of the pages already had the links brought over just fyi14:02
bknudsonhenrynash: can you post the code in progress?14:02
stevemargagehugo: yes, i recall someone did it when they brought over endpoint policy or endpoint filtering14:03
stevemargagehugo: i question if those are correct14:03
henrynashbknudson: it's already up for review:
patchbothenrynash: patch 349939 - keystone - Add expand, data migration and contract logic to k...14:03
gagehugoThey seemed alright, I can take another look14:03
lbragstadhenrynash you're familiar with the driver_hints stuff right?14:04
henrynashlbragstad: yep14:04
lbragstadhenrynash have time for a real quick (probably trivial) question?14:04
henrynashlbragstad: sure14:05
lbragstadhenrynash so I just migrated a column in the database14:05
lbragstadhenrynash which is pretty straight forward -
patchbotlbragstad: patch 355618 - keystone - Add key_hash column to credential table14:05
lbragstadhenrynash and I want to ask the credential api for any credentials that have 'key_hash' set to None (meaning they haven't been encrypted yet)14:06
lbragstadhenrynash when I do this - it blows up in the sql code
lbragstadin ^ that specific case, I have two filters that I want applied14:06
lbragstadbut it doesn't seem to like that fact that the second one has a value of None14:07
lbragstadbut it looks like we use that pattern in other places14:07
henrynashlbragstad: indeed14:08
lbragstadhenrynash am I just using this wrong?14:08
*** SamYaple has quit IRC14:09
henrynashlbragstad: looking14:09
*** SamYaple has joined #openstack-keystone14:09
henrynashlbragstad: and where's teh code where you actually create teh hints filter?14:10
*** esp has quit IRC14:10
lbragstadhenrynash let me grab a paste14:10
*** tonytan4ever has joined #openstack-keystone14:10
gagehugostevemar: yes they are good14:11
*** SamYaple has quit IRC14:11
*** SamYaple has joined #openstack-keystone14:11
*** adrian_otto has joined #openstack-keystone14:12
*** SamYaple has quit IRC14:13
*** SamYaple has joined #openstack-keystone14:13
*** spzala_ has quit IRC14:14
openstackgerritMikhail Nikolaenko proposed openstack/keystone: [WIP] Move fernet utils to backend
*** spzala has joined #openstack-keystone14:14
*** ezpz has joined #openstack-keystone14:15
henrynashlbragstad: hmm, odd...the check() method was added after I wrote the stuff...but we do seem you use it with None values....14:15
lbragstadhenrynash apparently using '' instead of None works!14:15
lbragstadthanks dstanek ;)14:15
henrynashlbragstad: I've got to go offline for a bit (flight to catch), but will look at it more14:15
lbragstadhenrynash sounds good - thanks14:15
henrynashlbragstad: sounds like a bug in teh check() code14:16
lbragstadhenrynash yeah - either way we should probably address that case?14:16
*** SamYaple has quit IRC14:17
*** spzala has quit IRC14:17
*** adrian_otto has quit IRC14:17
*** spzala has joined #openstack-keystone14:17
*** SamYaple has joined #openstack-keystone14:17
*** ravelar has joined #openstack-keystone14:18
*** lamt has joined #openstack-keystone14:19
stevemargagehugo: coolio14:19
*** hockeynut has joined #openstack-keystone14:20
*** haplo37_ has joined #openstack-keystone14:22
*** esp has joined #openstack-keystone14:25
*** itisha has joined #openstack-keystone14:30
rdopierahi guys, anywbody has a moment to chat about bug ?14:35
openstackLaunchpad bug 1415588 in OpenStack Dashboard (Horizon) "Cannot list users and groups with Keystone v3" [Undecided,New]14:35
*** edtubill has joined #openstack-keystone14:35
rdopieraI'm trying to fix it by providing a more informative error message, but I have a hard time telling this case from an expired session or a similar error14:35
rdopieraI wonder if "unauthorized" is really the best error to return in this case14:36
*** ezpz has quit IRC14:37
*** michauds has joined #openstack-keystone14:40
*** marekd2 has quit IRC14:46
*** erhudy has joined #openstack-keystone14:51
*** spedione|AWAY is now known as spedione14:53
*** rvba has quit IRC14:59
*** nishaYadav has quit IRC14:59
*** openstackgerrit has quit IRC15:03
*** openstackgerrit has joined #openstack-keystone15:04
ayoungrdopiera, looking15:11
ayoungrdopiera, so, yes, the problem is that the user is trying to do an operation that they are not authorized to perform.  In this use case, Horizon devs have enough knowledge of how to work around it, and you could put that info into the message returned to the user, no?15:13
*** marekd2 has joined #openstack-keystone15:14
*** BjoernT has joined #openstack-keystone15:15
*** nishaYadav has joined #openstack-keystone15:16
openstackgerritNisha Yadav proposed openstack/python-keystoneclient: Add ec2 functional tests
nishaYadavsamueldmq, rodrigods stevemar please have a look ^15:18
rdopieraayoung: how can I distinguish that "unauthorized" and an expired session just happening in the same place?15:20
ayoungrdopiera, I don't know.  If there is any question, Horizon can always try to validate the users token, but, in general, it seems more like something you should not request if you don't have reason to think the user can do it.15:21
ayoungrdopiera, geting Horizon sufficient info to do auth driven UI has been a long standing Windmill that I've tilted at15:22
ayoungwe need the rest of the Keystone devs to accept that it is OK to tell a user "here is the role you need to perform that action"15:22
rdopieraayoung: I mean, how can I tell the difference between "your session expired, please re-login" and "you don't have the right to perform this operation as this user"?15:23
rdopieraayoung: if both return "Unauthorized: The request you have made requires authentication."15:23
ayoungrdopiera, is this the only place you have that problem?15:24
stevemarnishaYadav: yes ma'am! </salute>15:25
rdopierait's the first place I encountered this problem, but I just took over horizon three weeks ago...15:25
rdopieraI suppose this will happen everywhere where keyston decides to return "requires authorization" to an already logged-in user15:25
ayoungrdopiera, yep.15:26
ayoungrdopiera, it would be nice if we distinguished.  An expired token should give a different error than a forbidden operation15:26
rdopieraseems to be it should rather be something like "forbidden"15:26
rdopierais there any chance for this?15:26
ayoungexpired token should be 401.  UNauthorized operation should be 40315:27
rdopieraI suppose that's a change in behavior and would mean an api version bump?15:28
*** hatTip has joined #openstack-keystone15:29
*** ravelar1 has joined #openstack-keystone15:29
*** ravelar has quit IRC15:29
ayoungrdopiera, indubitably15:33
rdopieraallright, thank you for the information, that helps15:36
openstackgerritNisha Yadav proposed openstack/python-keystoneclient: Add auth functional tests
*** su_zhang has quit IRC15:41
mfischcrinkle: setting up to test your fix15:42
mfischstill need to build a container etc but getting close15:42
*** thumpba has quit IRC15:44
*** code-R_ has quit IRC15:49
tsufievstevemar, o/15:49
stevemartsufiev: ahoy15:49
tsufievstevemar, ducttape in horizon channel told me you're the right person to ask about domain quotas ;)15:50
stevemartsufiev: eww, but okay. also, you can ask the channel in general :)15:50
tsufievwell, my initial question is quite broad: do you know anything about them?15:50
*** ravelar1 has quit IRC15:51
tsufievsince quoting his earlier reply 'I think part of domain quotas is to distribute management of quotas, really15:51
tsufiev and this has been a long standing background issue with domains too15:51
tsufiev stevemar might have more background15:51
tsufiev but it might be that domains are kind of a dead target too, and hierarchical projects really are the next thing'15:51
AndyWojoHi, I'm trying to use Ceilometer, the instructions say to use v2, but I'm getting 401's with V2. but when I switch to v3 i get 404's15:51
AndyWojoAnyone have any pointers?15:51
*** jaosorior has quit IRC15:51
tsufievI am really curious if HMT is a way to go for quotas spanning several tenants instead of domain quotas?15:52
AndyWojo <-- that's my service stanza15:52
AndyWojoI get  The request you have made requires authentication. (HTTP 401) (Request-ID: req-01d9427d-6679-459c-aebb-03217c3052d0) with v215:52
stevemartsufiev: ugh, i don't even know the right answer any more15:56
*** Ephur has joined #openstack-keystone15:56
stevemartsufiev: i know we re-worked domains to just be projects... under the covers when you create a domain you are in fact creating a project with a special "is_domain=true" attribute15:57
stevemartsufiev: this was done for HMT purposes, everythings a project15:57
tsufievmy initial guess was that keystone has almost nothing to do with quotas, even domain ones (despite the name)15:58
tsufievperhaps ducttape gave me false lead :)15:58
*** rcernin has quit IRC15:58
*** nisha_ has joined #openstack-keystone15:59
*** su_zhang has joined #openstack-keystone15:59
*** su_zhang has quit IRC15:59
*** lamt has quit IRC15:59
tsufievalmost = just get a list of tenants inside a domain to sum up their quotas and usages15:59
tsufievbut... well, that doesn't seem like a scalable solution, on the other hand16:00
*** sdake_ is now known as sdake16:00
*** sdake is now known as sdake_16:00
*** nishaYadav has quit IRC16:02
stevemartsufiev: sorry, distracted. i want to give you more information16:04
tsufievI'm all attention16:04
stevemartsufiev: i know we don't manage or handle quotas directly in keystone16:04
stevemartheres no code for that16:05
*** ravelar has joined #openstack-keystone16:05
stevemari want to say that each project should handle things consistently, but this is openstack we're talking about16:05
*** dmellado is now known as dmellado|off16:05
tsufievyep, that's reasonable16:06
*** amoralej is now known as amoralej|off16:06
stevemartsufiev: i would assume that a domain can be configured to have a quota that is spread out to all it's projects, or not have one at all16:07
tsufievstevemar, do you mean 'parent gives the same amount to its children, which is equal to the one that parent has' by 'spread out'?16:09
lbragstaddstanek henrynash i opened a bug for now so that I can get back to the credential encryption work16:10
openstackLaunchpad bug 1614154 in OpenStack Identity (keystone) "DriverHints with values of None seem to be broken" [Undecided,New]16:10
stevemartsufiev: i was thinking a domain would have a quota of 100 jellybeans and project A could have 90 and project B could have 1016:10
tsufievyes, that was the same thing I was thinking about, stevemar16:10
*** ezpz has joined #openstack-keystone16:11
*** code-R has joined #openstack-keystone16:13
*** nisha_ has quit IRC16:20
*** roxanaghe has joined #openstack-keystone16:20
*** asettle has quit IRC16:30
amakarovstevemar, hi!16:34
dstaneklbragstad: thoughts on ?16:38
stevemaramakarov: ahoy16:39
stevemaroh hey there16:39
lbragstaddstanek sure - that works16:39
*** gyee has joined #openstack-keystone16:39
lbragstaddstanek that just makes it so that when we run that test we always advance the clock (instead of only in the fernet case), right?16:40
dstaneklbragstad: yes16:40
dstanekhmmm....actually i have another test for Forbidden, but it looks like i lost that commit in my rebasing16:41
lbragstaddstanek sweet - does that get rid of the transient issue you were seeing?16:41
*** spzala has quit IRC16:41
dstanekthe issue was mostly because we were mocking time in 2 different ways16:41
dstaneknot it only uses the fixture16:41
*** spzala has joined #openstack-keystone16:42
amakarovstevemar, that's the benchmark you asked for. It has some prerequisites I haven't stated: devstack and openrc environment16:43
amakarovIt won't work another way16:43
*** edtubill has quit IRC16:46
*** spzala has quit IRC16:46
lbragstaddstanek I think I have most of the credential migration and rotation logic written - i'll push another patchset soon16:46
lbragstaddstanek I need to figure out how to test the rotation/migration stuff16:47
stevemaramakarov: that s cool though16:47
dstaneklbragstad: the bad tests that caused the Fernet Forbidden error actually showed an interesting corner case16:49
lbragstaddstanek really?16:50
amakarovstevemar, have some problems though... please wait16:50
openstackgerritDavid Stanek proposed openstack/keystone: Add test for revocation corner case in Fernet
dstaneklbragstad: ^ see that one16:53
lbragstaddstanek huh - interesting16:56
lbragstaddstanek so that makes sure we're actually hitting the disabled user case.16:56
lbragstadand not the time issue16:56
*** Gorian|work has joined #openstack-keystone16:56
dstanekon a single node or maybe mutiple nodes that have exactly the same time Fernet will return unauthorized because of the revocation list.16:57
dstanekit is possible, however, to have a token generated in the future (which this test tests) or maybe even not having an updated revocation list (which would have the exact same behavior)16:58
*** tesseract- has quit IRC17:00
dstaneklunch time!17:01
rderoseme too!17:10
openstackgerritLance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest
lbragstaddstanek ^17:12
lbragstadi need to find a way to test that :)17:13
*** su_zhang has joined #openstack-keystone17:20
stevemarlbragstad: enjoy!17:20
*** roxanaghe has quit IRC17:21
*** tqtran has joined #openstack-keystone17:22
*** marekd2 has quit IRC17:23
*** asettle has joined #openstack-keystone17:23
*** marekd2 has joined #openstack-keystone17:23
*** tonytan4ever has quit IRC17:25
*** debu__ has joined #openstack-keystone17:26
*** rcernin has joined #openstack-keystone17:26 there any way to find to which region a compute host belongs17:26
*** tqtran has quit IRC17:27
*** marekd2 has quit IRC17:28
*** asettle has quit IRC17:28
*** mvk has quit IRC17:31
*** tqtran has joined #openstack-keystone17:35
amakarovstevemar, I've updated my benchmark - now it works :)17:37
*** su_zhang has quit IRC17:43
*** stevemar_ has joined #openstack-keystone17:43
*** ChanServ sets mode: +o stevemar_17:43
*** stevemar_ has quit IRC17:43
*** hockeynut has quit IRC17:48
*** roxanaghe has joined #openstack-keystone17:51
*** spzala has joined #openstack-keystone17:53
*** amakarov is now known as amakarov_away17:55
*** spzala has quit IRC17:58
*** hockeynut has joined #openstack-keystone17:59
*** code-R has quit IRC18:00
*** spzala has joined #openstack-keystone18:00
*** pauloewerton has joined #openstack-keystone18:04
*** mvk has joined #openstack-keystone18:07
*** dkehn_ has quit IRC18:08
*** tqtran has quit IRC18:09
*** tqtran has joined #openstack-keystone18:19
*** dkehn_ has joined #openstack-keystone18:20
*** su_zhang has joined #openstack-keystone18:21
*** su_zhang has quit IRC18:25
*** tonytan4ever has joined #openstack-keystone18:26
*** tonytan4ever has quit IRC18:31
*** slberger has joined #openstack-keystone18:32
*** tqtran has quit IRC18:40
*** tqtran has joined #openstack-keystone18:43
*** ametts has quit IRC18:45
*** woodster_ has quit IRC18:49
*** hoonetorg has quit IRC18:53
*** ametts has joined #openstack-keystone18:56
*** debu__ has quit IRC18:56
*** rdopiera has left #openstack-keystone19:02
*** thumpba has joined #openstack-keystone19:05
*** jamielennox has quit IRC19:06
*** edtubill has joined #openstack-keystone19:06
*** hoonetorg has joined #openstack-keystone19:06
*** fifieldt has quit IRC19:07
*** jamielennox has joined #openstack-keystone19:09
*** ChanServ sets mode: +v jamielennox19:09
*** gyee has quit IRC19:11
samueldmqjamielennox:  stevemar: I need help to figure out what's going on in patch
patchbotsamueldmq: patch 350245 - python-keystoneclient - Add ec2 functional tests19:14
*** hatTip has quit IRC19:14
samueldmqI have been trying to understand what is causing the failure, but no success19:14
samueldmqmyself and nisha would appreciate some help :)19:14
*** fifieldt has joined #openstack-keystone19:18
*** spzala has quit IRC19:18
*** spzala has joined #openstack-keystone19:19
*** spzala_ has joined #openstack-keystone19:22
stevemarsamueldmq: pfft, if you can't figure it out, what makes you think i can :|19:23
*** spzala has quit IRC19:23
samueldmqstevemar: not idea, you and jamielennox are configured as my gateway for the client19:25
dolphmlbragstad: is the performance bot running with memcache enabled now? regarding
patchbotdolphm: patch 309146 - keystone - Pre-cache new tokens19:25
samueldmqstevemar: no*19:25
stevemarsamueldmq: hehe19:25
dolphmlbragstad: i.e. did you confirm memcache is being used with my patch?19:25
*** spzala_ has quit IRC19:26
lbragstaddolphm I did not19:27
lbragstadI talked to odyssey4me about it for a little bit a couple weeks ago (i believe stevemar and I were asking about it)19:27
*** ayoung has quit IRC19:27
dolphmlbragstad: and?19:27
*** edmondsw has quit IRC19:28
lbragstadodyssey4me made it sound like there were a couple other things that needed to be done19:29
samueldmqlbragstad: dolphm: on change 354495, if we consider encryption to be enabled, what do we do just after upgrade when the creds are not ecrypted yet19:29
lbragstaddolphm I'd have to do dig in the irc logs to find it19:29
samueldmqpatch 35449519:29
patchbotsamueldmq: - keystone - Add conf to support credential encryption19:29
dolphmsamueldmq: what DO we do, or what SHOULD  we do? ;)19:29
samueldmqdolphm: well, we are supposed to keep things running after an upgrade19:31
lbragstadIf encryption is enabled we should check the key_hash value when getting a credential19:31
lbragstadif a credential has a key_hash then we know for a fact it was encrypted at some point, either through the credential CRUD or through the migration19:31
lbragstadif the key_hash is None then we know the credential is still in it's plaintext form - and we should return that19:32
samueldmqwe can still retrieve old credentials that are not encrypted19:32
samueldmqbut we can't create new ones without encrypting19:32
samueldmqdolphm: lbragstad: is this the idea ? ^19:32
lbragstadnewly created credentials should be encrypted if encryption is enabled - but should be able to retrieve old credentials that haven't been encrypted yet19:32
lbragstadI think that is the only way we don't incur downtime encrypting credentials19:33
lbragstadotherwise you'll have to take the whole cluster down, run the encryption migration, and stand the whole thing back up19:33
samueldmqlbragstad: nice, dolph's idea is to have it alwyas enabled19:33
lbragstadand for deployments that have millions of credentials, that's a lot of downtime19:34
samueldmqand I like it19:34
samueldmqlbragstad: ++, but that doesn't stop us from being able to always have it enabled19:35
lbragstadsamueldmq yeah - i still need to work through that case19:35
lbragstadsamueldmq the problem is that if you have 3 keystone nodes, two of them are running mitaka, and one of them is upgraded to newton19:36
*** eandersson_ has quit IRC19:36
lbragstadand you migrate your credentials to be encrypted19:36
lbragstadnow the mitaka nodes are going to be exposes ciphertext via the api19:36
lbragstadalong with the key_hash attribute19:37
samueldmqreleasenote: "As of Newton, all new credentials are encrypted. Trying to create new credentials without properly setting up the key repository will result in a server exception. Old credentials should be encrypted using keystone manage, however they still can be retrieved from the API."19:37
lbragstadsamueldmq where is that?19:37
samueldmqlbragstad: just here :) I am just summarizing the idea19:38
lbragstadsamueldmq yeah - i agree.. but I don't know how you would enforce that on the mitaka nodes without backporting something to make mitaka aware of the things we added in newton19:38
samueldmqwhy do we need to care about mitaka?19:39
samueldmqnew things, do migrations19:39
lbragstadsamueldmq if you're doing a rolling upgrade19:39
lbragstadunless we make deploying newton first a requirement of the upgrade19:40
lbragstadso you would be required to deploy newton to *all* nodes in the deployment before turning on credential encryption19:40
lbragstadand before migrating any plaintext credentials19:41
samueldmqlbragstad: commented on that patch19:42
lbragstadsamueldmq which patch?19:42
patchbotsamueldmq: patch 354495 - keystone - Add conf to support credential encryption19:42
samueldmqlbragstad: sorry I can't see the issue with having encryption=true by default and rolling upgrades19:43
lbragstadsamueldmq if you have encryption=true and you do a rolling upgrade you're going to have nodes that are mitaka and nodes that are newton deployed at the same time19:43
lbragstadsamueldmq so what happens if you ask a mitaka node for a credential?19:44
samueldmqfor an encrypted credential?19:44
lbragstadany credential really19:44
samueldmqit fails ? mitaka doesn't understand encrypted credentials19:45
-openstackstatus- NOTICE: The volume for filled up rather suddenly, causing a number of jobs to fail with a POST_FAILURE result and no logs; we're manually expiring some logs now to buy breathing room, but any changes which hit that in the past few minutes will need to be rechecked and/or approved again19:45
*** spzala has joined #openstack-keystone19:45
lbragstadsamueldmq should it though?19:45
*** code-R has joined #openstack-keystone19:45
lbragstadhow would it fail?19:46
lbragstadit would get a credential by ID19:46
lbragstadand return the reference19:46
lbragstadwhich would contain ciphertext in the blob if the credential is already migrated19:46
lbragstadif not - it would contain an extra attribute exposed through the API ('key_hash') because mitaka wouldn't know to pop that out of the reference before returning to the user19:47
samueldmqlbragstad: if the credential was created in mitaka, mitaka can still retrieve, right?19:48
samueldmqlbragstad: the issue is newton credentials (encrypted) that can't be retrieved with mitaka code ?19:48
*** code-R_ has joined #openstack-keystone19:49
lbragstadsamueldmq I don't think the problem is that it can't... i think the problem is that depending on how the rolling upgrade is done can expose sensitive data through the API19:49
*** spzala has quit IRC19:49
*** spzala has joined #openstack-keystone19:50
lbragstadif mitaka code attempts to retrieve a credential after it has been migrated - data will be exposed.19:50
*** gyee has joined #openstack-keystone19:50
samueldmqlbragstad: what data?19:50
samueldmqthat wasn't exposed before..19:50
lbragstadsamueldmq the key_hash and the blob of the credential19:51
lbragstadthe blob of the credential will be ciphertext19:51
*** su_zhang has joined #openstack-keystone19:51
lbragstadwhich isn't useful to the user19:51
lbragstadand the key_hash is a hash of they key that was used to encrypt the credential19:51
samueldmqlbragstad: does mitaka understand key_hash?19:51
lbragstadthat would be backporting a migration19:52
samueldmqlbragstad: if not, it will just expose the ciphertext19:52
samueldmqwhich is not an issue?19:52
*** code-R has quit IRC19:52
lbragstadsamueldmq if i'm a user and i ask keystone for a credential, i expect some secret blob that I created to be returned19:52
lbragstadwhen what actually gets returned is ciphertext19:53
lbragstadas a user, i can't do anything with ciphertext19:53
samueldmqyes, because your provider is in the middle of an upgrade and is encrypting things in the meantime19:53
lbragstadso if i have any automated tooling around credentials that relies on pulling the blob out of the credential response, that is broken19:53
samueldmqonly do the old creds migration after upgrading all nodes then19:54
samueldmqonly do the old creds migration after upgrading all nodes then19:54
samueldmqoops,  sorry19:54
dstaneklbragstad: i love credential encryption!19:54
samueldmq1. upgrade a node to mitaka, it still can read not-encrypted creds, but only creates encrypted creds19:55
samueldmq2. an old node can still read old creds, but can't the new ones (encrypted)19:55
lbragstadupgrade to mitaka?19:55
samueldmq3. after migrating all nodes, encrypt old credentials19:55
lbragstador newton?19:55
*** su_zhang has quit IRC19:56
lbragstadso - after step 1 you have one node running newton and two running mitaka, what happens if you hit a mitaka node asking for a credential you just created on a newton node?19:56
samueldmqonly the case in 2. would have mitaka nodes returning the cyphertext, because it's trying to read credentials created in the middle of the upgrade process with an old node19:57
lbragstadsamueldmq what if we did this19:57
*** slberger has quit IRC19:57
lbragstad1.) upgrade each node to newton and make sure `[credential] encrypt`=False19:57
samueldmqthat works19:58
lbragstad2.) migrate all credentials using `keystone-manage credential_migrate`19:58
samueldmqbut we can't address dolphm's suggestion to make it enabled by default19:58
lbragstad2.) gracefully roll each node to enable encryption19:58
lbragstad3.) forcibly migrate remaining credentials19:59
samueldmqlbragstad: what happens if...19:59
lbragstadsamueldmq true - i don't know how to get around that without exposing sensitive data through the API19:59
samueldmqwhen enabling encryption in a node, a credential get created in that node (then it's encrypted)19:59
lbragstador providing inconsistent user-experience19:59
samueldmqand another node (newton) without it enabled yet tried to read it ? gmm... it works20:00
samueldmqlbragstad: how long does a rolling upgrade take ?20:00
lbragstaddepends on how many credentials you have20:00
samueldmqlbragstad: I know it depends on the deployment, but usually...20:00
samueldmqa couple of hours ? days ?20:00
lbragstadi have no idea20:00
samueldmqlbragstad: what happens if we add a new attribute to an entity20:01
samueldmqand in a rolling upgrade the old code ignores it ? but the new code doesn't ?20:02
samueldmqyou would get incosistent responses depending on the node you hit, right?20:02
lbragstadsamueldmq yes - i believe so20:02
samueldmqso I think this is something that has to be understood in a rolling upgrade20:03
samueldmqthere are things being added/removed, the code won't break, but the server responses may vary depending on the version of the node it hits20:03
samueldmqbecause new things are returned/handled differently in new code20:04
samueldmqand this is just for the time the rolling upgrade takes20:04
lbragstadthe thing that gets me is that the inconsistency returned is ciphertext20:04
samueldmqold data always work, new data makes sense in the new server version20:04
samueldmqso it's a UX issue now20:05
lbragstadwould it be considered a backwards incompatible change?20:05
samueldmqwe are not modifying anything in a backwards way20:06
samueldmqthe "backward" data is still there as it was (old credentials)20:06
lbragstadsamueldmq the api contract stats the blob returned is the blob you passed in20:06
*** su_zhang has joined #openstack-keystone20:07
samueldmqif you're using the same server version, yes20:07
lbragstadso - depending on which node i hit, i might get the blob i'm expecting or not20:07
lbragstadi might get something entirely different20:07
openstackgerritMerged openstack/ldappool: Updated from global requirements
*** slberger has joined #openstack-keystone20:09
samueldmqlbragstad: I understand your point, it's fair, I am mulling it a bit more20:11
samueldmqlooks like this is another level of rolling upgrades, it's not just about the database being able to be used by different versions20:12
samueldmqit's about keeping the API contracts during the upgrade20:13
samueldmqthe API contract is kept before the upgrade20:13
samueldmqand after the upgrade20:13
samueldmqbut it can't really be kept *during* the upgrade in that case20:13
samueldmqlbragstad: the solution would be to store the encrypted credential in a new column20:16
samueldmqand drop the old column in N+120:16
samueldmqnewton code stores it in both, mitaka reads from the old20:16
lbragstadthat might work20:17
samueldmqdoes that add a lot of complexity for implementing it ?20:17
*** ayoung has joined #openstack-keystone20:18
*** ChanServ sets mode: +v ayoung20:18
lbragstadso the newton code would store an encrypted version of the credential in a separate column and update the old column to have the new plaintext20:18
samueldmqlbragstad: ++20:19
lbragstadI think this would absolutely require a contract phase to remove the old column though20:19
lbragstadotherwise we would still have plaintext secrets in the database20:19
samueldmqor just make it disabled by default20:20
samueldmqand change the default in +1 release20:20
lbragstadmake what disabled?20:20
openstackgerritMerged openstack/keystone: api-ref: Fix parameters attributes
lbragstadwe could address dolphm's concern with rolling database upgrades20:21
lbragstadbut that would have to land before the rest of the credential encryption work20:21
lbragstadthen i could see not having an encrypt configuration option20:22
samueldmqanother solution is: encryption is disabled, upgrades nodes to newton. after upgrading all of them, kesytone manage old creds and enable the config option20:22
lbragstadsamueldmq yeah - that was my original upgrade path20:22
samueldmqin the N+1 we change the encryption=true to be the fdefault20:22
lbragstaddolphm thoughts?20:23
lbragstaddstanek thoughts?20:23
samueldmqexactly, keep that path, and we can go to a sane default (encryption=true) in N+120:23
dstaneklbragstad: i'll have to read up. been working on some things.20:23
samueldmqwithout breaking anything because we committed to N+1 upgrades and not N+220:23
mfischEmilienM: le ping20:26
mfischsorry wrong room!20:26
openstackgerritSteve Martinelli proposed openstack/python-keystoneclient: Add ec2 functional tests
* stevemar glares at mfisch -_-20:34
*** rcernin has quit IRC20:40
stevemardolphm: got a minute for ?20:41
patchbotstevemar: patch 350704 - keystone - Make all token provider behave the same with trusts20:41
stevemarits the beginning of the chain for fixing cache20:41
dstaneklbragstad: samueldmq: haven't read through it all yet, but had an interesting thought. make a new column `cipher_text` and do a rename type flow for the upgrade20:42
lbragstaddstanek I just started thinking about that20:42
lbragstad1.) upgrade a single node to Newton20:42
lbragstad2.) run db_sync --expand to create new 'key_hash' and 'encrypted_blob' columns20:43
lbragstadbut if a user creates a credential using a newton node20:43
lbragstadand updates it using a mitaka node,20:43
lbragstadthe trigger won't update the 'encrypted_blob' column20:44
*** roxanaghe has quit IRC20:44
dstaneklbragstad: there should be no reason when triggers can't do that20:45
dstanekthe hard (impossible?) part is doing the encryption in a trigger. seems like it would have to be an incremental migration in Python20:45
lbragstaddstanek the mitaka node just passes the plaintext update to the database20:45
lbragstaddstanek right20:46
lbragstaddstanek but the hard part is making mitaka aware of what it needs to do for newton schemas20:46
lbragstadin order for Mitaka code to update properly - it would need to understand the new schema and how to encrypt (right?)20:46
dstaneklbragstad: or have the new code keep writing to the plaintext column20:47
dstanekbut even if you do that new records from mitaka will not be encrypted20:47
lbragstadbut what happens if someone updates a credential using the old code?20:47
dstanekso there is a short amount of time for some number of records to be out of sync20:47
lbragstaddstanek what if we require all nodes to be upgrades to newton before enabling `[credential] encrypt`?20:48
lbragstadonce all nodes are on Newton, you can enable encryption and batch process credentials20:49
dstaneklbragstad: that would be ideal. that was the original plan, right? before we started talking about the ability to turn it off20:49
lbragstaddstanek yeah - but dolphm brought up a good point that we should really allow the storage of unencrypted credentials to be an option20:50
stevemarsamueldmq: want to take a quick look at ?20:50
patchbotstevemar: patch 356485 - keystone - [api] add relationship links to v3-ext20:50
stevemari just verified all the entries, should be a quick one :)20:50
dstaneklbragstad: i thought he didn't want that ability?20:50
lbragstaddstanek i'm still thinking of this as a one way migration20:50
lbragstaddstanek see his first comment here -
patchbotlbragstad: patch 354495 - keystone - Add conf to support credential encryption20:51
dstaneklbragstad: he is saying to have no way to turn it off20:52
lbragstaddstanek he's saying don't have a `[credential] encrypt` option20:52
dstanekwhich i agree with, but it makes the upgrade hard20:52
lbragstaddstanek right - it's a trade off20:52
lbragstadwe can use the option as a way to make the migration easier20:52
dstaneki think right now it's probably ok since they are not used that much. afaict20:53
lbragstadotherwise we risk upgrading and having some amount of data out of sync during the upgrade20:53
lbragstaddstanek what's ok?20:53
lbragstaddstanek allowing the deployer to specify encrypt=False?20:53
lbragstaddstanek or allowing some amount of error in the upgrade?20:54
dstanekallowing for some errors during the upgrade20:54
dstanekwe already don't have a guaranteed uptime yet20:54
lbragstaddstanek doesn't rolling upgrades guarantee that?20:56
dstanekwe don't have a guarantee for M->N just yet. we've talked about doing best effort right now20:56
lbragstaddstanek so - you're saying we should remove the encrypt option?20:57
*** raildo has quit IRC20:57
lbragstadthat would require all nodes to be upgraded to Newton *and* all credentials to be migrated before starting any of the nodes20:57
lbragstadso downtime would be dependent on how many credentials your storing in the backend.20:58
*** edmondsw has joined #openstack-keystone20:58
dstaneki'm ok with it it other people are. deployers have the option of leaving it up and having some inconsistent api responses.20:59
dstanekwe can also mitigate that quite a bit this an incremental migration20:59
dstanekleave mitaka running, migrate, turn off mitaka, migrate again and start up newton - is one possible flow21:00
lbragstadthat would be a tough one21:01
lbragstadbecause if a credential is created using a newton node21:02
lbragstadthen upgraded using a mitaka node21:02
*** edtubill has quit IRC21:02
lbragstadthe credential migration command would have to somehow make sure each credential is encrypted with the right key using the key_hash21:03
lbragstadand if not - it would have to re-encrypt to update the ciphertext and the key_hash of the credential21:03
dstanekin that scheme mitaka and newton are not running in parallel21:03
lbragstadwe wouldn't be able to assume a populated key_hash value means the credential is encrypted21:04
dstanekyou can also track that with 'updated_date' and an incremental migration can re-migrate anything what was updated after it started21:04
lbragstadthe upgrade path should just disable credential create and update through policy21:05
*** gordc has quit IRC21:08
dstanekthat's a good idea. a deployer can totally do that without any keystone support21:08
lbragstadso in that case you would:21:09
lbragstad1.) disable credential create and update in policy21:09
lbragstadactually - i'm not sure how that would work21:11
*** edtubill has joined #openstack-keystone21:11
lbragstad2.) roll nodes to newton code21:12
*** fifieldt has quit IRC21:13
lbragstadnewton code can attempt to decrypt values and if it fails, return the original value21:13
lbragstad.. actually that doesn't work either21:13
dstanek1 - upgrade a single node and expand the schema (included a new cipher_text column)21:15
dstanek2 - roll out policy to disable create/update for credentials21:15
dstanek3 - migrate21:15
*** michauds has quit IRC21:15
dstanek4 - rolling upgrade to newton21:15
*** gyee_ has joined #openstack-keystone21:15
dstanek5 - revert policy change21:15
dstanek6 - collapse (will remove the old blob column)21:16
lbragstaddstanek so this would be using the rolling upgrade idea but without triggers21:16
dstanekyeah, the policy changes would limit the need for triggers21:17
*** spzala has quit IRC21:17
dstanekdolphm: what do you think? if it's good it might be worth calling out in the spec21:17
*** spzala has joined #openstack-keystone21:18
lbragstaddstanek which spec?21:18
lbragstadthe credential encryption one?21:18
dolphmreading back...21:18
lbragstaddolphm you're here!21:18
dolphmsorry wrong channel21:19
*** gyee has quit IRC21:19
*** pnavarro has quit IRC21:19
dstaneklbragstad: no the rolling upgrade one21:20
dolphmlbragstad: with rolling upgrades, you could do this more smoothly by introducing a new encrypted blob column21:20
dstaneki'm sure there will be other cases where data can't up transformed in triggers21:20
dstanekdolphm: that was my step 1 :-)21:20
lbragstaddstanek ++21:20
dolphmlbragstad: old code reads from plaintext column, new code reads from ciphertext column, data migration does all the hard work21:20
dolphmdstanek: oh, i missed that bit, then ++!21:21
lbragstaddolphm but what happens if a credential is updated?21:21
lbragstadon a mitaka node?21:21
dolphmdstanek: why the policy bit, though?21:21
dolphmlbragstad: oh, triggers can't do the encryption21:21
* samueldmq 's back21:21
samueldmqlbragstad: so we keep that idea of adding a new column?21:21
samueldmqwhat is this policy thing you're talking about21:22
lbragstadsamueldmq still working through the issues21:22
dstaneki think that's close enought to 0 downtime since a low use feature would be readonly for a short period of time21:22
* samueldmq is not really up to date with the current proposal of rolling upgrades21:22
*** spzala has quit IRC21:22
lbragstadwe would require that migrating to newton means making credentials read only21:22
dstaneklbragstad: we'd probably also need a 'updated_date' column that gets created in --expand and remove in --contract21:24
samueldmqdolphm: nice21:24
dstanekthat way you can deal with incremental migration21:24
*** fifieldt has joined #openstack-keystone21:24
samueldmqdolphm: the triggers remove the need of the code updating both columns21:24
samueldmqsince the db deals with that itself21:24
dolphmsamueldmq: correct21:24
lbragstadbut in this cases it's a little strange since triggers would have to encrypt/decrypt21:25
stevemarsee you all in a few hours!21:25
dolphmlbragstad: which they can't21:25
stevemarwell, whoever is still online :)21:25
dolphmstevemar: the trigger could reject the write, though21:25
lbragstaddolphm right - that would be insane21:25
dolphmlbragstad: err ^21:25
stevemarsamueldmq: ^21:25
dolphmdstanek: ^21:26
stevemarhe meant you21:26
dstanekstevemar: you don't get a break21:26
lbragstaddstanek ^21:26
samueldmqdolphm: ^21:26
stevemardstanek: :(21:26
dstanekdolphm: if you rejected the write in the trigger you'd get a 500 error right?21:26
dolphmdstanek: how about instead of a policy change, we setup triggers to abort writes to the old column?21:26
dolphmdstanek: yes21:26
lbragstadso if a credential is created on a Newton node and updated on a Mitaka one, what happens?21:27
dstaneki wish it could be a 50321:27
dolphmdstanek: yeah21:27
dolphmlbragstad: i don't know if we could support that21:27
dolphmlbragstad: leave the plaintext column blank?21:28
dolphmalthough, the blob is probably required21:28
lbragstadit is21:28
dstaneki'd be OK with either approach21:28
*** vivek has joined #openstack-keystone21:28
dstaneklbragstad: we can't create/update during the migration or we'd be giving out bad data21:28
*** vivek is now known as Guest8152921:29
lbragstadah - ok21:29
lbragstadi think i getit21:29
Guest81529hi, I am trying to run devstack and using RestAPIs of Keystone, I am continuously facing [cors] issues21:29
lbragstadso the expand phase would create a trigger that wouldn't allow writes to the old column21:29
Guest81529Anything that I am missing. I edited the keystone.conf for cors21:30
dolphmlbragstad: in mysql, for example, you could raise a custom signal in the trigger
dolphmlbragstad: same sort of thing in sqlite afaik
lbragstadso the migration would look like21:30
lbragstad1.) take one node out of the mesh and update it to newton21:31
dolphmlbragstad: you could go so far as to disable any creates or updates during the migration process, in either direction21:31
dstanekok, going to bail for a little bit to get dinner - i'll catch up when i get back21:31
lbragstad2.) run keystone-manage db_sync --expand21:31
dolphm(which lays down triggers to reject all writes)21:31
dolphm(except deletes)21:31
lbragstad3.) migrate all credentials/21:31
dolphm(in --migrate, which would require the newton node to have a populated credentials repo)21:32
lbragstadso that would be the equivalent to my `keystone-manage credential_migrate` command21:32
dolphmlbragstad: yeah, i guess you wouldn't need that anymore?21:33
lbragstaddolphm I don't think so21:33
lbragstad4.) rolling upgrade the rest of the keystone nodes?21:33
dolphmthey'll be able to read all credentials as they come online21:34
lbragstad5.) keystone-manage db_sync --contract (removes triggers and old unencrypted column)21:34
dolphmand old nodes will be able to read all credentials while they're still around21:34
dolphmlbragstad: ++21:34
lbragstadmy brain hurts!21:34
dolphmrderose: henrynash: interesting edge case for trigger-based migrations ^^ when triggers can't possibly know what to write based on the database alone21:34
lbragstadok - so the implementation of credential encryption will be dependent on rolling upgrades with triggers21:35
samueldmqlbragstad: so if we can't create/update during the upgrade21:35
dstaneklbragstad: dolphm: what don't you need anymore? the credential_migrate command?21:35
*** roxanaghe has joined #openstack-keystone21:35
samueldmqlbragstad: so it'd be fine to set ecryption=true as the default21:35
dolphmdstanek: yes, do we need it?21:35
dstaneksamueldmq: or like dolphm says not even have the option21:35
dstanekdolphm: yes, to rotate keys21:35
samueldmqdstanek: exactly21:35
lbragstadwe also wouldn't need the encrypt config option21:35
dstanekunless that is a different command21:35
rderosedolphm: yep, especially where the default comes from python21:36
dolphmGuest81529: cors isn't really part of keystone itself (it's an oslo project)21:36
*** marekd2 has joined #openstack-keystone21:36
dolphmGuest81529: but you'll need to describe the error you're seeing, your cors configuration, etc, if you expect anyone to be able to help21:36
dstanekcors stuff shouldn't come into play unless there is a browser involved, right?21:36
Guest81529from FireFox with RestAPI plugin everything is working fine21:37
dstanekdolphm: lbragstad: i do think you still need that command - but now i'm really leaving to get dinner21:37
dolphmGuest81529: sure, CORS is designed to protect against cross-origin requests ... if you're hitting keystone directly, CORS won't stop you21:37
Guest81529but when I wrote sample js app which is running on a simple node server...the response says "Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource"21:38
lbragstaddstanek ok - i gotta run too21:38
lbragstaddstanek dolphm samueldmq  I'm going to noodle on this for an hour or two and then I'll be back21:38
rderosedolphm henrynash: like if the default is based on config file and business logic in source21:38
dolphmGuest81529: CORS headers need to be provided by both your javascript server and the keystone endpoint, IIRC21:39
rderosedolphm henrynash: we probably could find ways to solve this in our current model21:39
rderosedolphm henrynash: perhaps another POC, just to be sure21:39
*** pauloewerton has quit IRC21:39
dolphmrderose: we can always lay down config options into triggers; in this case it's encryption that the db cannot handle by itself21:39
Guest81529I did this in keystone.conf on my devstack [cors] allowed_origin = * allow_methods = GET21:39
dolphmrderose: nevermind the keys to do the encryption21:39
Guest81529infact allo_methods = *21:40
lbragstadrderose taking an unencrypted plaintext field and encrypting it into something keystone can understand21:40
lbragstad(i.e. using fernet)21:40
* lbragstad steps away 21:40
Guest81529and I am putting headers: {"Access-Control-Allow-Origin":"*"} in header when making request21:41
*** marekd2 has quit IRC21:41
samueldmqlbragstad: kk21:41
rderosedolphm lbragstad: that is an interesting edge case, hmm...21:42
dolphmGuest81529: are you running the cors middleware in your keystone pipeline? is your client seeing the CORS headers in responses?21:42
Guest81529I am not getting response from keystone server21:43
Guest81529I see error log in keystone that IP not permitted21:43
Guest81529I only changed the conf I need to do middleware chnages too with oslo?21:44
dolphmGuest81529: yes,
Guest81529ok..thanks let me try that and I will get back on this...I just followed
*** su_zhang has quit IRC21:46
rderosedolphm lbragstad: we could temporarily add an unencrypted column and let the new code deal with both; remove the new column in the next release. Not ideal, but...21:46
dolphmGuest81529: sounds like you missed this step?
*** su_zhang has joined #openstack-keystone21:47
dolphmGuest81529: well, i guess i can't link to a direct section21:47
rderosedolphm lbragstad: we may at times need these kind of workarounds21:47
dolphmGuest81529: "Enabling CORS with PasteDeploy"21:47
dolphmrderose: liiike today21:47
rderosedolphm lbragstad: that being said, I willing to bet that 99% of our typical db changes will be covered by the new strategy21:49
*** su_zhang has quit IRC21:49
Guest81529@dolphm 'Enabling CORS with configuration' i used the wild card configuration21:49
dolphmrderose: right21:49
*** spedione is now known as spedione|AWAY21:49
dolphmGuest81529: okay, well you should also ensure cors is in your paste pipeline21:50
*** su_zhang has joined #openstack-keystone21:51
*** edtubill has quit IRC21:51
*** Gorian|work has quit IRC21:52
*** su_zhang has quit IRC21:53
*** adriant has joined #openstack-keystone21:54
bknudsonhenrynash: making some progress by setting breakpoints in the tests -- check this out:
*** slberger has left #openstack-keystone21:59
*** ezpz has quit IRC21:59
bknudsonfor some reason when running the test again with diff db, the call to self.initialize_repo(repo_name=EXPAND_REPO) winds up running contract_repo migrations.22:00
*** adu has joined #openstack-keystone22:02
*** su_zhang has joined #openstack-keystone22:02
*** ametts has quit IRC22:04
*** su_zhang has quit IRC22:05
*** sdake_ has quit IRC22:06
*** adu has quit IRC22:07
*** ravelar has quit IRC22:07
*** adu has joined #openstack-keystone22:07
*** gyee_ has quit IRC22:09
*** ayoung has quit IRC22:14
*** gagehugo has quit IRC22:17
*** spzala has joined #openstack-keystone22:17
*** spzala has quit IRC22:17
*** spzala has joined #openstack-keystone22:17
*** sdake has joined #openstack-keystone22:21
*** Gorian|work has joined #openstack-keystone22:22
*** hockeynut has quit IRC22:22
*** amoralej|off has quit IRC22:24
*** jaugustine has quit IRC22:24
*** gagehugo_ has quit IRC22:24
*** amoralej has joined #openstack-keystone22:30
dolphmbknudson: is find_migrate_repo() returning the wrong path or something?22:32
*** gyee_ has joined #openstack-keystone22:35
*** sdake has quit IRC22:38
*** adu_ has joined #openstack-keystone22:40
*** adu has quit IRC22:41
*** adu_ is now known as adu22:41
bknudsondolphm: find_migrate_repo is returning the expected path22:48
*** BjoernT has quit IRC22:48
*** gagehugo has joined #openstack-keystone22:49
*** jaugustine has joined #openstack-keystone22:50
*** esp has quit IRC22:51
*** spzala has quit IRC22:58
*** spzala has joined #openstack-keystone22:58
dolphmbknudson: is it operating on the correct database?23:01
dolphmbknudson: or is it running against sqlite or something twice?23:01
bknudsonthis is the engine: Engine(postgresql://openstack_citest:***@localhost/ivqwxvlodp)23:02
*** erhudy has quit IRC23:02
bknudsonI run the tests like this:23:02
bknudson.tox/py27/bin/python -m unittest keystone.tests.unit.test_sql_upgrade.SqlContractSchemaUpgradeTests.test_001_created_at_made_non_nullable keystone.tests.unit.test_sql_upgrade.PostgreSQLOpportunisticContractSchemaUpgradeTestCase.test_001_created_at_made_non_nullable23:02
bknudsonso first it's sqlite and 2nd is postgresql23:03
*** spzala has quit IRC23:03
bknudsonif I swap the order, still fails (except with sqlite this time)23:03
bknudsonleaning towards some cached global in sqlalchemy-migrate23:04
*** asettle has joined #openstack-keystone23:05
*** asettle has quit IRC23:09
bknudson -- first time through, after self.initialize_repo, the python module is as expected.23:11
bknudsonbut the next time through, it calls self.initialize_repo(repo_name=EXPAND_REPO) , but the schema has contract_repo23:11
dolphmbknudson: is it ever realistic for us to test against multiple databases in the gate?23:12
bknudsonwe do now.23:12
bknudsonand it works23:12
dolphmbknudson: so we can drop sqlite? :D23:12
bknudsonwell, we only test db migrations with other dbs.23:13
bknudsonof course, we should fix our unit tests to not require a db.23:13
bknudsonor at least minimize db testing23:14
dolphmbknudson: well, our other tests don't work with real db's (which dstanek is working on)23:14
bknudsonshoot. I know I had it working at one point in order to test db2.23:15
dolphmbknudson: yeah - but the "root" domain thing is what's broken last i spoke to dstanek about it23:16
dolphmbknudson: the tests don't bother to actually create the "<<< root magic domain entry >>>!>" thing23:17
dolphmbknudson: so mysql & postgres just balk with foreign key errors23:17
bknudsonthat's pretty new23:17
dolphmbknudson: still time to remove it then23:17
bknudsondoesn't root magic domain entry get created during migrations?23:18
*** ayoung has joined #openstack-keystone23:18
*** ChanServ sets mode: +v ayoung23:18
bknudsonI'm wondering if this could be worked around by not having migration repos with names the same.23:29
*** Gorian|work has quit IRC23:39
openstackgerritMerged openstack/python-keystoneclient: Improve docs for v3 auth
bknudsonthis is crazy.23:46
*** adu has quit IRC23:50
*** adu has joined #openstack-keystone23:54

Generated by 2.14.0 by Marius Gedminas - find it at!