Thursday, 2015-08-13

morgan_404gyee: ok done with travel00:02
morgan_404gyee: so the only concern i have with that bike is that the wheels are 700x30c instead of 700x25c00:03
morgan_404it'll be a bit wider wheels so more rolling resistance00:03
morgan_404but probably more comfortable00:03
morgan_404i also don't like disc brakes00:03
morgan_404but i ride for different reasons00:04
morgan_404gyee: i also wouldn't go for a bike that has less than ultegra 105 on it00:04
morgan_404gyee: again personal preference00:05
jamielennoxdevstack run in v3 only gate complete!00:07
openstackjamielennox: Error: "!!!" is not a valid command.00:07
jamielennoxopenstack: you are not a valid command00:07
richmjamielennox: +25600:07
jamielennoxstill 108 tempest failures but at least we can start using the gate to test this stuff00:08
morgan_404openstack: shush00:08
gyeemorgan_404, yeah, I am concern about the disc break too00:09
morgan_404you probably wont ever see an issue with them00:09
gyeehigher maintenance cost too, unless I do it myself00:09
morgan_404but I don't like them. it also prevents me from competing atm00:09
morgan_404so i wont convert to them until it doesn't prevent anything00:10
gyeethey are getting better, from what I've reading00:11
*** jecarey has joined #openstack-keystone00:13
gyeeits more of a hybrid than pure road bike00:13
* morgan_404 sticks to pure roadbike00:15
*** ankita_wagh has quit IRC00:15
* morgan_404 sighs and didn't get a ride today ...00:15
morgan_404stupid expense report submission00:15
*** markvoelker has joined #openstack-keystone00:15
gyeeyeah, EEM was finally back online, I had to get creative a couple weeks back for the boston midcycle expenses00:16
openstackgerritSam Leong proposed openstack/keystone: Tokenless authz with X.509 SSL client certificate
*** markvoelker has quit IRC00:21
mordredmorgan_404, jamielennox
*** shadower has quit IRC00:23
*** shadower has joined #openstack-keystone00:23
mordredmorgan_404: I thnik my comment about it being better in ksa is correct, but if it's not it shoudl be :)00:23
morgan_404it should be better in KSA00:23
morgan_404thats for sure00:23
jamielennoxwhy are you manually testing that?00:24
jamielennoxoh, because token_endpoint doesn't have a setuptools entrypoint in ksc00:25
jamielennoxyea, OSC camped on that name so it loads their plugin, in ksa we take it back00:25
morgan_404jamielennox: exactly00:26
morgan_404keystoneauth is *way* better00:26
morgan_404jamielennox: so... session loading.00:26
morgan_404since you're here00:26
morgan_404not sure how to make that one better-ish00:26
jamielennoxmorgan_404: neither00:26
morgan_404do we need to just camp it as is?00:27
morgan_404i mean... i'm ok with that00:27
jamielennoxpart of the reason to split plugin loading is so you're not stuck with just one way to create a plugin00:27
morgan_404i just get the feeling it could be better00:27
jamielennoxlike there are valid reasons particularly in federation to have different options for the same plugin00:27
jamielennoxin session...00:27
jamielennoxi don't know how many ways i want people to load a session, and it's useful having it on the class because i know for example OSC overrides those class methods00:27
morgan_404so like i said, i'll just press "goooooo" if that is really the fix00:28
jamielennoxbut if we're removing oslo.config then sessoin loading needs to be in the same place to get the opts...00:28
*** sigmavirus24 is now known as sigmavirus24_awa00:32
jamielennoxmorgan_404, mordred: commented, looks ok to me00:33
mordredjamielennox: ossum, thanks!00:33
rodrigodseasy review:
rodrigodssome fixes for the API spec after the merge of the Project Tree Deletion sepc00:35
*** thedodd has quit IRC00:39
*** gyee has quit IRC00:40
*** jasonsb has quit IRC00:47
*** lhcheng_ has joined #openstack-keystone00:49
*** lhcheng has quit IRC00:52
*** richm has quit IRC00:53
*** roxanaghe has quit IRC00:54
*** josecastroleon has quit IRC00:56
*** boris-42 has quit IRC00:58
*** flwang has quit IRC00:59
*** flwang has joined #openstack-keystone01:00
*** boris-42 has joined #openstack-keystone01:00
*** richm has joined #openstack-keystone01:03
*** chlong has joined #openstack-keystone01:03
*** drjones has quit IRC01:11
*** lhcheng_ has quit IRC01:25
*** piyanai has joined #openstack-keystone01:27
*** ayoung has quit IRC01:27
jamielennoxmorgan_404: are we sure that auth_token still does an in-memory cache if memcache isn't configured?01:41
morgan_404at least afaik that hasn't changed01:41
morgan_404sec.. le tme find it01:41
jamielennoxmorgan_404: oh, right found it01:42
jamielennoxit's an oslo thing01:42
jamielennoxi guess it made sense with eventlet01:47
morgan_404even there not really01:53
morgan_404once worker eventlet maybe.01:53
openstackgerritDave Chen proposed openstack/keystone: Fix the misspelling
*** fangzhou has quit IRC02:02
*** markvoelker has joined #openstack-keystone02:05
*** woodster_ has quit IRC02:10
*** piyanai has quit IRC02:12
openstackgerritOpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/keystoneauth: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/keystoneauth-saml2: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/keystonemiddleware: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/oslo.policy: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/pycadf: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/python-keystoneclient: Updated from global requirements
*** piyanai has joined #openstack-keystone02:22
*** jecarey has quit IRC02:26
*** zzzeek has quit IRC02:27
*** jecarey has joined #openstack-keystone02:28
*** mylu has quit IRC02:36
*** stevemar has joined #openstack-keystone02:46
*** ChanServ sets mode: +v stevemar02:46
*** stevemar has quit IRC02:49
jamielennoxmorgan_404: damnit, now i know way too much about how auth_token handles memcache02:49
*** stevemar_ has joined #openstack-keystone02:49
*** ChanServ sets mode: +v stevemar_02:49
jamielennoxit's horrible02:49
*** jasonsb has joined #openstack-keystone02:50
*** hakimo_ has joined #openstack-keystone02:52
*** elmiko has quit IRC02:53
*** hakimo has quit IRC02:54
*** ankita_wagh has joined #openstack-keystone02:55
jamielennoxstevemar_: have you seen it may be something worth handling in OSC in the mean time02:56
openstackLaunchpad bug 1482772 in Keystone "Region filtering for endpoints does not work" [Medium,Confirmed] - Assigned to Lin Hua Cheng (lin-hua-cheng)02:56
jamielennoxpython-memcached 1.56 release 17 days ago has python 3 support and is in global requirements03:06
morgan_404jamielennox: why do you think I hate that code so much :(03:23
jamielennoxi never knew...03:24
morgan_404I tried to save you.03:24
morgan_404You went for it anyway :P03:24
morgan_404The whole crypto thing is insane too03:24
morgan_404I should just rip that out and go to dogpile and make the crypto thing a proxy (dogpile lets you later proxies in seamlessly)03:25
jamielennoxright, it's a disaster - i wonder if anyone uses it?03:27
jamielennoxwhy would you crypt your memcache?03:27
jamielennoxis that a thing?03:28
morgan_404It is a swift thing03:31
jamielennoxmorgan_404: so most of our caching tests are designed around memory cache)03:32
morgan_404It has always been unfun/unfriendly and just icky03:33
jamielennoxso yesterday, when you were saying just go remove the memory cache instead and implying it was easy - that was a lie03:34
*** ankita_wagh has quit IRC03:37
*** piyanai has quit IRC03:43
*** mylu has joined #openstack-keystone03:53
openstackgerritMerged openstack/keystone: Give some message when an invalid token is in use
*** mylu has quit IRC03:58
*** lhcheng has joined #openstack-keystone03:59
*** ChanServ sets mode: +v lhcheng03:59
*** richm has quit IRC04:04
*** stevemar_ has quit IRC04:07
openstackgerritMerged openstack/keystonemiddleware: Updated from global requirements
*** hrou has joined #openstack-keystone04:13
*** hrou has quit IRC04:19
*** hrou has joined #openstack-keystone04:19
openstackgerritMerged openstack/python-keystoneclient: Updated from global requirements
*** Ephur has quit IRC04:36
openstackgerritMerged openstack/pycadf: Updated from global requirements
openstackgerritMerged openstack/keystone: Updated from global requirements
*** hrou has quit IRC04:48
anteayaso cryptography 1.0 was released yesterday, the 12th05:01
anteayahas anyone in here fired up a keystone with it yet?05:01
anteayamorgan_404: ^^ if you are still around05:02
anteayajamielennox: too05:02
anteayayou might be up05:02
morgan_404anteaya: eating füds05:02
anteayaa must do thing05:02
morgan_404But no haven't tried with new crypto lib05:02
jamielennoxanteaya: i saw the notice it had been released05:02
anteayaokay thanks05:02
anteayajamielennox: do you have time to try it with keystone?05:03
morgan_404Gate didn't explode afaict05:03
anteayawell we are running on images05:03
morgan_404That is a good thing? Or is it capped?05:03
jamielennoxanteaya: i don't know what we're using it for05:03
anteayasome third party cis that don't have all gone bust05:03
jamielennoxoh fernet05:03
anteayayeah fernet05:03
morgan_404Oh hah fernets05:03
anteayamight be nothing05:03
*** davechen has joined #openstack-keystone05:03
anteayabut might splode the gate when images refresh05:04
morgan_404I can try when I get home05:04
jamielennoxi can give it a run but fernet is really high level stuff that hasn't changed in a while, i don't expect any difference05:04
morgan_404But it'll be a couple hours05:04
anteayawe have about 9 hours before images refresh05:04
anteayawould be nice to at least say keystone works with 1.005:04
morgan_404Fernet shouldn't cause issues here though. Unless they totally screwed their contract up05:04
anteayalike I said, might be nothing05:05
morgan_404We don't dig in under to the primatives.05:05
morgan_404Where I'd expect breakage05:05
anteayabut something is causing some third party ci images to go boom05:05
anteayaand so far cryptography is new05:05
morgan_404Got an example of the error?05:05
anteayathat is as much as I have05:05
morgan_404I can look at the trace back and probably tell you pretty quickly05:05
anteayain -infra, jgriffith and I were talking05:05
jamielennoxanteaya: i don't think fernet is the default anyway so it shouldn't have an affect on third party ci05:05
morgan_404Fernet is not on in devstack05:06
jamielennoxright, was going to spin up a devstack but wasn't sure how to even enable it05:06
anteayait is a devstack issue05:06
anteayajamielennox: can you spin up a devstack?05:06
anteayahe is failing on stack.sh05:06
anteayawhen keystone is enabled05:06
anteayacan you see what you experience?05:06
anteayaI know I should be doing this myself05:07
jamielennoxanteaya: ok, i'll start from scratch05:07
anteayabeen so long since I spun up anything05:07
*** stevemar has joined #openstack-keystone05:08
*** ChanServ sets mode: +v stevemar05:08
morgan_404Ok i've seen this before its usually an associated thing not keystone directly. It could also be some odd devstack + syntax change in somethig else.05:08
morgan_404I'll need to poke the logs a bit more. See if I can do it from teh mobile05:08
morgan_404Oh they dont publish the screens :(05:09
morgan_404That doesn't help.05:09
jamielennox2015-08-13 03:21:42.119 | + export OS_TOKEN=11122233344405:10
jamielennox2015-08-13 03:21:42.119 | + OS_TOKEN=11122233344405:10
jamielennox2015-08-13 03:21:42.119 | + export OS_URL=
jamielennox2015-08-13 03:21:42.119 | + OS_URL=
jamielennox2015-08-13 03:21:42.119 | + create_keystone_accounts05:10
jamielennox2015-08-13 03:21:42.120 | ++ get_or_create_project admin default05:10
jamielennox2015-08-13 03:21:42.120 | ++ local project_id05:10
jamielennox2015-08-13 03:21:42.121 | +++ openstack --os-url= --os-identity-api-version=3 project create admin --domain=default --or-show -f value -c id05:10
jamielennox2015-08-13 03:21:42.779 | ERROR: openstack Missing parameter(s):05:10
jamielennox2015-08-13 03:21:42.779 | Set a service URL, with --os-url, OS_URL or auth.url05:10
jamielennoxso that looks like the first errror - and that's just weird05:10
anteayajamielennox: you get the same error05:10
jamielennoxanteaya: no, looking at your log05:10
morgan_404jamielennox: Set a service URL, with --os-url, OS_URL or auth.url yeah05:10
jamielennoxthat'll take a while05:10
anteayaand yeah, totally weird error05:11
morgan_404Wonder if something in keystonebootatrap changed05:11
*** stevemar has quit IRC05:11
anteayafirst time it happens is when enabling keystone, so I'm in here05:11
jamielennoxinternal cloud instance says "unable to schedule new instance" - so that cloud's out05:11
morgan_404And they just refreshed their devstack itself05:11
*** davechen1 has joined #openstack-keystone05:13
*** vivekd has joined #openstack-keystone05:13
jamielennoxanteaya: actually OSC just had a release05:14
anteayajohn said they didn't so I didn't look05:14
jamielennoxstevemar was planning it for yesterday at least05:14
*** davechen has quit IRC05:15
anteaya38 hours ago05:15
jamielennoxso yesterday-ish depending on timezone05:15
anteayathis error cropped up in the last 3 hours05:15
anteayalooks like the timestamps in the log is utc time05:16
jamielennoxanteaya: it wouldn't have popped up in devstack until the requirements change merged to allow it:
anteayait was passing earlier:
* jamielennox learnt this the hard way yesterday05:17
anteayajamielennox: oh05:17
jamielennoxso that's like still 19 hours05:18
anteayathat still merged about 18 hours ago05:18
anteayaround there05:18
anteayabut thanks I didn't know that about the requirements change05:18
jamielennoxdon't trust my maths05:18
anteayait is more than 5 hours05:19
anteayawhich takes us out of the window of change05:19
jamielennoxsure, but if this is third party i assume there's some pip caching05:19
anteayajgriffith is offline right now, so I don't know what caching he has going on05:20
anteayaat least if you have a successful run we know it isn't crypto 1.005:20
anteayaI think05:20
jamielennoxanteaya: i'm doubting it's crypto05:21
*** davechen has joined #openstack-keystone05:25
*** davechen1 has quit IRC05:28
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Handle memcache pool arguments collectively
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Create Environment cache pool
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Import _memcache_pool normally
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Seperate standalone cache tests
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Disable memory caching of tokens
jamielennoxmorgan_404: ^05:30
jamielennoxoo,   File "/usr/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/", line 11, in <module>05:31
jamielennox    from cryptography.hazmat.bindings._openssl import ffi, lib05:31
jamielennoxAttributeError: 'module' object has no attribute '_init_cffi_1_0_external_module'05:31
jamielennoxthat's a crypotgraphy issue05:31
anteayathat is from your devstack run?05:32
*** hafe has quit IRC05:32
*** hafe has joined #openstack-keystone05:32
jamielennoxit seems to be coming from a horizon manage command though05:32
jamielennoxno, i can't import OpenSSL from python05:33
anteayais this an openssl issue?05:33
anteayasorry I should just wait and let you share when you have something05:34
jamielennoxanteaya: i didn't log the devstack run, so is as much trace as i have05:35
jamielennoxwhich is at least plenty to see this issue05:35
anteayawell if it is an issue, it is reproducable, yeah?05:35
anteayawhat are you thinking?05:35
jamielennoxno idea atm, i've never really dived into how cffi works05:37
anteayais this something we can ask about in cryptography-dev?05:37
anteayabecause _I_ have no idea05:38
jamielennoxanteaya: that's my next guess05:38
anteayawe == you in this case05:38
anteayaI'm in that channel if you are willing to post a question05:39
*** markvoelker has quit IRC05:41
anteaya0.9.3 was the cryptography version just prior to 1.005:44
anteayaI'm going to offer a patch to requirements to pin it at 0.9.305:44
anteayajamielennox: thoughts?05:44
*** fifieldt has joined #openstack-keystone05:49
morgan_404Icky :(05:49
morgan_404Safe is to cap at
*** med_ has quit IRC05:50
morgan_404Of the req is proposed it can be abandoned if needed but it gets the gates running on it.05:50
morgan_404If* not of05:50
morgan_404jamielennox: can you try with 0.9.3 of crypto?05:51
morgan_404If it doesnt break that way we know crypto did a bad bad thing05:51
jamielennoxmorgan_404, anteaya: so if i downgrade cryptography to <1.0 then i can at least import OpenSSL05:51
jamielennoxhowever something crazy is happening with pip caching05:51
morgan_404Ok sounds like we need a pin.05:51
morgan_404Erm cap05:51
jamielennoxi had to install a specific cffi to get it to work05:52
jamielennoxand now i can't upgrade again05:52
morgan_404Cliff released on monday05:53
anteayaworking on patches, I posted in infra asking about the order05:53
anteayacap is a better word then pin?05:53
*** Nirupama has joined #openstack-keystone05:53
* anteaya edits the commit message05:53
morgan_404Pin i think is == thisexactversion05:53
morgan_404Cap is "less than x"05:53
morgan_404In my mind05:54
morgan_404Cliff shouldnt affect this though.05:54
jamielennoxi've no idea what's happening there05:54
jamielennoxsomewhere there is a precompiled object with a different version of cffi and they are having a bad interaction05:55
jamielennoxbut at this point i'm thinking i just kill the vm05:55
morgan_404Ugh :(05:55
openstackgerritAnita Kuno proposed openstack/keystone: Cap cryptography to less than 1.0
*** ParsectiX has joined #openstack-keystone06:04
openstackgerritOpenStack Proposal Bot proposed openstack/keystone: Imported Translations from Transifex
*** hafe has quit IRC06:18
*** vivekd has quit IRC06:18
*** med_ has joined #openstack-keystone06:23
*** med_ is now known as Guest47106:23
*** afazekas has joined #openstack-keystone06:40
*** markvoelker has joined #openstack-keystone06:41
*** rdo has quit IRC06:42
openstackgerritEric Brown proposed openstack/keystone: Utilize min and max of oslo.config
*** rdo has joined #openstack-keystone06:43
*** Guest471 has quit IRC06:44
openstackgerritMerged openstack/keystone-specs: Fix nits from Project Tree Deletion spec
*** markvoelker has quit IRC06:46
*** yottatsa has joined #openstack-keystone06:49
*** lhcheng has quit IRC06:58
*** med_ has joined #openstack-keystone07:15
*** med_ is now known as Guest6139407:15
*** fifieldt has quit IRC07:17
*** e0ne has joined #openstack-keystone07:17
*** browne has quit IRC07:20
*** charz has quit IRC07:22
*** mtreinish has quit IRC07:24
*** therve has quit IRC07:25
*** therve has joined #openstack-keystone07:25
*** dguerri` is now known as dguerri07:26
*** henrynash has joined #openstack-keystone07:26
*** ChanServ sets mode: +v henrynash07:26
*** charz has joined #openstack-keystone07:27
*** mtreinish has joined #openstack-keystone07:27
*** henrynash has quit IRC07:28
*** dguerri is now known as dguerri`07:31
*** e0ne has quit IRC07:41
*** shadower has quit IRC07:41
*** haneef_ has quit IRC07:41
*** tellesnobrega_af has quit IRC07:41
*** alex_xu has quit IRC07:41
*** openstackgerrit has quit IRC07:41
*** akscram has quit IRC07:41
*** nonameentername has quit IRC07:41
*** hogepodge has quit IRC07:41
*** csd has quit IRC07:41
*** crinkle has quit IRC07:41
*** _fortis has quit IRC07:41
*** Kiall has quit IRC07:41
*** timburke has quit IRC07:41
*** devananda has quit IRC07:41
*** rmstar has quit IRC07:41
*** ParsectiX has quit IRC07:41
*** henrynash has joined #openstack-keystone07:42
*** ChanServ sets mode: +v henrynash07:42
*** hughsaunders has quit IRC07:42
*** shadower has joined #openstack-keystone07:43
*** haneef_ has joined #openstack-keystone07:43
*** tellesnobrega_af has joined #openstack-keystone07:43
*** alex_xu has joined #openstack-keystone07:43
*** openstackgerrit has joined #openstack-keystone07:43
*** akscram has joined #openstack-keystone07:43
*** nonameentername has joined #openstack-keystone07:43
*** hogepodge has joined #openstack-keystone07:43
*** csd has joined #openstack-keystone07:43
*** crinkle has joined #openstack-keystone07:43
*** _fortis has joined #openstack-keystone07:43
*** Kiall has joined #openstack-keystone07:43
*** timburke has joined #openstack-keystone07:43
*** devananda has joined #openstack-keystone07:43
*** rmstar has joined #openstack-keystone07:43
*** _fortis has quit IRC07:44
*** crinkle_ has joined #openstack-keystone07:44
*** csd has quit IRC07:45
*** shadower has quit IRC07:45
*** alex_xu has quit IRC07:45
*** openstackgerrit has quit IRC07:45
*** akscram has quit IRC07:45
*** nonameentername has quit IRC07:45
*** devananda has quit IRC07:45
*** rmstar has quit IRC07:45
*** hafe has joined #openstack-keystone07:45
*** haneef_ has quit IRC07:46
*** tellesnobrega_af has quit IRC07:46
*** hogepodge has quit IRC07:46
*** crinkle has quit IRC07:46
*** Kiall has quit IRC07:46
*** timburke has quit IRC07:46
*** hughsaunders has joined #openstack-keystone07:47
*** devananda has joined #openstack-keystone07:47
*** rmstar has joined #openstack-keystone07:47
*** yottatsa has quit IRC07:47
*** Kiall has joined #openstack-keystone07:48
*** tellesnobrega has joined #openstack-keystone07:50
*** alex_xu has joined #openstack-keystone07:51
*** csd has joined #openstack-keystone07:52
*** nonameentername has joined #openstack-keystone07:52
*** timburke has joined #openstack-keystone07:55
*** rm_work is now known as rm_work|away07:56
*** rm_work|away is now known as rm_work07:58
*** akscram has joined #openstack-keystone07:58
*** haneef_ has joined #openstack-keystone07:58
*** openstackgerrit has joined #openstack-keystone08:00
*** rm_work is now known as rm_work|away08:00
*** _fortis has joined #openstack-keystone08:00
*** shadower has joined #openstack-keystone08:00
*** rm_work|away is now known as rm_work08:03
*** ParsectiX has joined #openstack-keystone08:08
*** fhubik has joined #openstack-keystone08:11
*** chlong has quit IRC08:13
*** Kennan has quit IRC08:14
*** fhubik is now known as fhubik_brb08:14
*** fhubik_brb is now known as fhubik08:19
*** Kennan has joined #openstack-keystone08:21
*** katkapilatova has joined #openstack-keystone08:23
*** katkapilatova has left #openstack-keystone08:24
*** katkapilatova has joined #openstack-keystone08:24
*** rdo has quit IRC08:24
*** rm_work is now known as rm_work|away08:25
*** rm_work|away is now known as rm_work08:25
*** jistr has joined #openstack-keystone08:26
*** rm_work is now known as rm_work|away08:28
*** rm_work|away is now known as rm_work08:29
*** knl has joined #openstack-keystone08:29
*** yottatsa has joined #openstack-keystone08:35
*** rdo has joined #openstack-keystone08:38
*** crinkle_ is now known as crinkle08:41
*** ankita_wagh has joined #openstack-keystone08:42
*** markvoelker has joined #openstack-keystone08:43
*** hogepodge has joined #openstack-keystone08:43
knlhi all, I think that bug resurfaced again, and is not a duplicate of the reported bug08:47
openstackLaunchpad bug 1057436 in Keystone "duplicate for #1054362 Delete role does not delete roles assignment in tenants" [High,Fix released] - Assigned to Jose Castro Leon (jose-castro-leon)08:47
*** markvoelker has quit IRC08:47
knlwe recently upgraded to kilo and started seeing this behavirour, which didn't exist in icehouse08:48
knlcould anyone confirm?08:48
*** yottatsa has quit IRC08:49
openstackgerritDave Chen proposed openstack/keystone: Show helpful message when request body is not provided
*** davechen has left #openstack-keystone08:53
rodrigodshenrynash, can you check the replies in
*** stevemar has joined #openstack-keystone09:09
*** ChanServ sets mode: +v stevemar09:09
*** stevemar has quit IRC09:12
*** rodrigods has quit IRC09:13
*** rodrigods has joined #openstack-keystone09:13
*** navid__ has joined #openstack-keystone09:18
*** navid___ has joined #openstack-keystone09:22
*** navid__ has quit IRC09:27
*** navid___ has quit IRC09:27
henrynashrodigods: will be doing so in an hour or os09:27
*** navid__ has joined #openstack-keystone09:27
*** navid__ has quit IRC09:31
*** navid__ has joined #openstack-keystone09:32
*** ankita_wagh has quit IRC09:41
*** ali_ has joined #openstack-keystone09:41
*** ali_ has quit IRC09:42
*** henrynash has quit IRC09:44
*** navid__ has quit IRC09:45
*** navid__ has joined #openstack-keystone09:46
*** navid__ has quit IRC09:47
*** Navid__ has joined #openstack-keystone09:48
knlI see in etc/keystone.conf.sample that a lot of options in [ldap] section is deprecated. Is there any document that explains why, and what will be the replacements?09:48
*** fhubik is now known as fhubik_brb09:58
*** boris-42 has quit IRC10:00
*** Navid__ has quit IRC10:01
*** Navidp has joined #openstack-keystone10:01
*** fhubik_brb is now known as fhubik10:05
*** navid_ has quit IRC10:23
*** Navidp is now known as navid_10:23
navid_ /nick navid10:25
bretonfolks, I am concerned about
bretonit broke heat and I wonder what impact it had on multiple backends for domains10:35
*** yottatsa has joined #openstack-keystone10:37
bretonwhen we use multiple backends for domain, service users are in default domain, human users in non-default domain. Human user gets a token and it will not be validated by any component who still cannot use keystone v310:37
*** markvoelker has joined #openstack-keystone10:43
*** markvoelker has quit IRC10:48
*** claudiub has joined #openstack-keystone10:51
*** chlong has joined #openstack-keystone11:04
*** stevemar has joined #openstack-keystone11:10
*** ChanServ sets mode: +v stevemar11:10
*** stevemar has quit IRC11:14
*** fhubik is now known as fhubik_brb11:25
*** dikonoor has joined #openstack-keystone11:30
*** dikonoo has joined #openstack-keystone11:30
*** shunliz has joined #openstack-keystone11:31
*** dikonoor has quit IRC11:33
*** ParsectiX has quit IRC11:33
*** annasort has quit IRC11:42
*** markvoelker has joined #openstack-keystone11:44
*** fhubik_brb is now known as fhubik11:45
*** markvoelker has quit IRC11:49
openstackgerritSamuel de Medeiros Queiroz proposed openstack/keystone: Create unit tests for endpoint policy SQL driver
*** marzif has joined #openstack-keystone11:51
openstackgerritSamuel de Medeiros Queiroz proposed openstack/keystone: Fixes return usage in endpoint-policy
*** markvoelker has joined #openstack-keystone11:53
openstackgerritPaweł Pamuła proposed openstack/keystone: IdP deletion triggers token revocation
openstackgerritSamuel de Medeiros Queiroz proposed openstack/keystone: Enable Cache-Control HTTP values in responses
openstackgerritSamuel de Medeiros Queiroz proposed openstack/keystone: Create Cached Policy Table
openstackgerritSamuel de Medeiros Queiroz proposed openstack/keystone: Centralized Policies Distribution Mechanism
*** mylu has joined #openstack-keystone12:00
*** knl has quit IRC12:02
*** piyanai has joined #openstack-keystone12:03
*** mylu has quit IRC12:09
*** gordc has joined #openstack-keystone12:13
*** annasort has joined #openstack-keystone12:30
*** annasort_ has joined #openstack-keystone12:31
*** topol has joined #openstack-keystone12:32
*** ChanServ sets mode: +v topol12:32
*** annasort has quit IRC12:35
*** annasort_ is now known as annasort12:35
*** yottatsa has quit IRC12:50
*** jsavak has joined #openstack-keystone12:50
*** gordc has quit IRC12:51
*** piyanai has quit IRC12:52
*** fhubik is now known as fhubik_brb12:53
*** richm has joined #openstack-keystone12:54
*** doug-fish has joined #openstack-keystone12:54
*** Nirupama has quit IRC12:55
*** bapalm has joined #openstack-keystone12:55
*** richm has quit IRC12:56
*** richm has joined #openstack-keystone12:56
*** bapalm__ has joined #openstack-keystone12:57
samueldmqbreton, I think the response is in your question12:57
samueldmqbreton, if one wants to use multiple identity backends, this implies in multiples domains12:58
samueldmqbreton, if multiple domains is required, v3 must be used12:58
samueldmqbreton, :)12:58
*** bapalm___ has joined #openstack-keystone12:58
*** jecarey has quit IRC12:58
*** bapalm has quit IRC12:59
*** openstackgerrit has quit IRC13:01
*** bapalm__ has quit IRC13:01
samueldmqdolphm, I will likely need to add unit tests to /policy as well, besides /endpoint_policy13:02
*** fhubik_brb is now known as fhubik13:02
samueldmqdolphm, should I create a bp for them?13:02
*** openstackgerrit has joined #openstack-keystone13:02
bretonsamueldmq: it is not possible to use v3 everywhere now.13:03
*** mestery has quit IRC13:03
dolphmsamueldmq: a bp to write tests? absolutely not13:03
samueldmqdolphm, just a bug then? it's not likely a bug as well13:04
samueldmqdolphm, I am just worried on how to target them13:04
samueldmqdolphm, maybe nothing at all? just proposing them?13:04
dolphmsamueldmq: things that have zero impact on end users do not need to be tracked in bugs or blueprints (tests, refactors, docstr changes, etc)13:04
*** edmondsw has joined #openstack-keystone13:05
*** topol has quit IRC13:05
dolphmsamueldmq: if the audience for your patch is only developers, then gerrit provides all the tracking we need13:05
samueldmqdolphm, makes sense, except in special cases where the change is complex and we need a spec (and then a bp?)13:05
*** bapalm___ has quit IRC13:05
samueldmqdolphm, but that isn't the case13:05
dolphmsamueldmq: when the change is complex, break it down into digestible patches so that it can be easily reviewed13:06
samueldmqdolphm, ++13:06
samueldmqdolphm, all the tracking on lp is used is to generate the release notes?13:07
dolphmsamueldmq: we land complicated multi-step refactors all the time... as long as they're broken down into multiple digestible patches13:07
dolphmsamueldmq: yes13:07
dolphmsamueldmq: Wishlist bugs and blueprints feed directly into release notes13:07
samueldmqdolphm, nice, so what you said makes sense to me13:07
samueldmqdolphm, makes sense (yes, I worked on the release notes with you last cycle)13:08
samueldmqdolphm, that's why that question :)13:08
openstackgerritMarianne Linhares Monteiro proposed openstack/keystone: List credentials by type
samueldmqdolphm, thanks for the clarifications13:09
*** stevemar has joined #openstack-keystone13:11
*** ChanServ sets mode: +v stevemar13:11
*** mestery has joined #openstack-keystone13:12
*** henrynash has joined #openstack-keystone13:12
*** ChanServ sets mode: +v henrynash13:12
*** atiwari1 has joined #openstack-keystone13:13
*** atiwari has quit IRC13:14
*** stevemar has quit IRC13:15
dstanekdolphm: i have a question about that EC2 crud when you have  sec13:19
*** bebech has joined #openstack-keystone13:24
bretonI guess yesterdays's change broke a lot of things.13:24
dstanekbreton: ?13:26
dstaneksamueldmq: if your cache policy table for auditing?13:26
*** bebech has quit IRC13:27
samueldmqdstanek, auditing isn't the initial intention13:28
dstanekwhat is the intention then?13:28
samueldmqdstanek, since I need to deliver a version of the policy for the next {policy_timeout} seconds13:28
samueldmqdstanek, I store a copy at the start time, and deliver the copy in the policy_cache table13:28
samueldmqdstanek, so endpoint processes coming at different times always get the same policy13:28
samueldmqdstanek, even if updates in the policy occur in the meantime13:29
*** browne has joined #openstack-keystone13:30
dstanekok, we are super paranoid here :-)13:31
bretondstanek: . After the patch mentioned there I suddenly see a lot of discussion of EC213:31
openstackLaunchpad bug 1484086 in Keystone "ec2tokens authentication is failing during Heat tests" [Undecided,New]13:31
*** jsavak has quit IRC13:31
breton(maybe it's just a coincident)13:31
samueldmqdstanek, hehe, remember the meantime can be long (5 minutes), and it isn't hard to occur updates in the meantime13:32
samueldmqdstanek, otherwise, if we havent that table, one would accept endpoints to be inconsistent for the next {policy_timeout} seconds13:33
samueldmqdstanek, if that's acceptable, we don't need that table :)13:33
samueldmqdstanek, but I don't think it is13:34
*** zzzeek has joined #openstack-keystone13:35
*** henrynash has quit IRC13:35
dstanekmaybe 5 mins is too long?13:35
samueldmqdstanek, so making the default like 1 min ?13:37
dstanekbreton: i don't know about that bug. i was the ML and was wondering how version of the client they were using. i don't know how to install the "kilo" version13:37
*** gordc has joined #openstack-keystone13:41
*** hrou has joined #openstack-keystone13:42
samueldmqdstanek, maybe that's acceptable, however reducing the timeout implies on lots of unnecessary  requests, since the policy does not change that much all the time13:42
samueldmqdstanek, and we don't do If-Modified-Since requests13:42
*** henrynash has joined #openstack-keystone13:43
*** ChanServ sets mode: +v henrynash13:43
samueldmqdstanek, (I think we should implements IMS requests, to avoid unnecessary network traffic)13:43
rodrigodshenrynash, hi... should we chat about the domain_id = None issue here?13:43
henrynashrodigods: sure13:43
*** jsavak has joined #openstack-keystone13:43
henrynashrodigods: I don’t see how we can break that without a deprecation period13:44
rodrigodshenrynash, yeah =(13:44
rodrigodsso we should go with the bad UX than, right?13:44
rodrigodsprojects acting as domains and regular projects act different in that case13:44
rodrigodshtruta, ^13:45
henrynashrodigods: I think you are right, maybe we do deprecate the whole idea of using a domin token (especially as we are going to kill domain tokens if I get my way)13:45
dstaneksamueldmq: i'm not worried about the network traffic as it would be minimal - i am more worries about the application13:45
rodrigodshenrynash, ++13:45
dstaneksamueldmq: that's why my poc for caching roughed out if-none-match support13:45
henrynashrodigods: I’m not sure we have much choice. but to go with what we have (I agree with the 6 cases you laid out)13:46
htrutahenrynash: ++13:46
samueldmqdstanek, if-none-match support ?13:46
htrutahenrynash, rodrigods, but we'll still have the case of project scope tokens, as I said in the comment in the patch13:46
henrynashrodigods: I’ll have to work out how we exaplin that clearly in the API doc?!!13:46
dstaneksamueldmq: that's for etags13:47
rodrigodshenrynash, htruta, project scoped tokens falls back to the behavior it is implemented right now / the one henrynash is fixing13:47
rodrigodshenrynash, +1, maybe add a follow up patch here ?13:48
henrynashhruta: so I think that works the same as it does now, if you use a project token then we haev to honor this undocumented functionality of trying the default domin until the deprecation time is up13:48
*** alex_xu has quit IRC13:48
henrynashrodigods: yes, I’m writing one already to follow that to state that teh fall back to default domain is deprecated13:49
htrutahenrynash, rodrigods, I see. that's some really bad UX. But I guess we can work this out for a while13:49
rodrigodshenrynash, great!13:49
henrynashhruta, rodigods: yep, the sins of our fathers….oh, that was us!!13:50
htrutaso, as long as it is well documented, I'm fine with it. someone will need to read something before using reseller, anyway13:50
htrutahenrynash: heh13:50
*** alex_xu has joined #openstack-keystone13:51
samueldmqdstanek, so you think that it would be acceptable13:54
*** petertr7_away is now known as petertr713:54
samueldmqdstanek, the model would be simpler though with this approach13:54
samueldmqdstanek, and policy 'instability' would occur only when updates occur13:55
samueldmqmorgan_404, you around ?13:56
samueldmqmorgan_404, what do you think about it ? ^13:56
dstaneksamueldmq: i really don't know. since you are saying that you think policy has very strict caching requirements you are building CMS feature into it, which seems too much13:56
samueldmqdstanek, I am being as much restrict as I can be in policy caching, to deliver the closer to CMS as it could be13:57
samueldmqdstanek, if I am being to paranoid on how much strict it should be, I really want to listen to others and get some feedback :)13:59
*** doug-fish has quit IRC13:59
henrynashanyone know what the        .[ldap] line is meant to do in the dependences in tox.ini?13:59
*** diazjf has joined #openstack-keystone13:59
henrynash(line 12 of current trunk version)14:00
dstanekhenrynash: it adds the optional dependency14:03
dstanekhenrynash: some of the testing deps where moved into extras14:03
*** afazekas has quit IRC14:03
*** annasort has quit IRC14:03
henrynashdstanek: do you knwo what I have to do to make it satified, runing tox just start fainling on my machine14:03
*** marzif has quit IRC14:04
henrynashsaying it couldn;t satisfy that dependency14:04
dstanekhenrynash: you may have to update your pdr14:04
henrynashdtsanek: oh….right14:04
*** marzif has joined #openstack-keystone14:04
henrynashdstanek: let me give that a go…thanks14:05
dstaneksamueldmq: i'm a little worried about the herd problem again here. how many different policies will nova typically use? 1 endpint right?14:07
samueldmqdstanek, yes 1 policy14:07
samueldmqdstanek, if we reduce the timeout, we increase the prob of occurring the herd problem14:08
dstaneksamueldmq: so ever 5 minutes 10s of 1000s of nova instances could hit keystone at nearly the same time?14:08
*** henrynash has quit IRC14:08
*** hafe has quit IRC14:08
*** fifieldt has joined #openstack-keystone14:08
samueldmqdstanek, only the endpoints sharing the same policy id hits the server at the same time14:08
*** sigmavirus24_awa is now known as sigmavirus2414:10
*** stevemar has joined #openstack-keystone14:12
*** ChanServ sets mode: +v stevemar14:12
samueldmqdstanek, so if we define a policy for nova service, all nova endpoints will get at the same time14:12
dstaneksamueldmq: yeah, that doesn't scale14:13
samueldmqdstanek, that's because the timeout expires based on a hash of the policy_id14:13
samueldmqdstanek, I can make that dependent of the endpoint_id14:13
samueldmqdstanek, so only endpoints with the same id will hit keystone at the same time14:14
dstanekwon't that basically be the same for all nova anyway?14:14
samueldmqdstanek, maybe makes more sense14:14
samueldmqdstanek, the same endpoint id ?14:14
*** narengan has joined #openstack-keystone14:14
dstaneksamueldmq: yeah14:14
samueldmqdstanek, I don't no, since there are different regions, etc14:15
samueldmqdstanek, but I don't know how that is used in practice14:15
samueldmqdstanek, need to listen from gyee on that14:15
*** stevemar has quit IRC14:15
openstackgerritRodrigo Duarte proposed openstack/keystone: Add is_domain field in Project Table
samueldmqdstanek, that's an interesting point though14:16
samueldmqdstanek, I am glad you understand the proposed solution deeply too14:16
samueldmqdstanek, I don't think so actually, otherwise we didn't need to implement endpoint filter14:17
dstaneksamueldmq: that's why i worry about having something expire at exactly X time for everyone14:17
samueldmqdstanek, if we always used a single endpoint/service, and wouldn't need to make endpoints different of services14:18
*** fhubik has quit IRC14:19
dstaneksamueldmq: if you have 10k nova instances in a region will they use a different endpoint_id or the same one?14:19
*** jecarey has joined #openstack-keystone14:19
*** fhubik has joined #openstack-keystone14:19
*** fhubik has quit IRC14:20
dolphmlbragstad: not that you have a use for this, but here's a list of every git SHA that has touched fernet in master (in order by date)
samueldmqdstanek, it really depends on the deployment, but I think it is expected to be more than one14:22
samueldmqdstanek, maybe we need to listen to operators on that front, (operators midcycle next week?)14:23
samueldmqdstanek, maybe gyee has a good idea on this as well .. so we can be sure on how we distribute the endpoint processes to avoid the herd14:23
*** devlaps has quit IRC14:24
*** phalmos has joined #openstack-keystone14:24
dstaneksamueldmq: their midcycle is a good place to bring this up14:25
dstaneki worry that is we don't have some amount of variance then we can't scale this14:25
samueldmqdstanek, are you going to attend it ?14:25
dstaneksamueldmq: no14:25
samueldmqdstanek, what if we do ims calls ?14:26
samueldmqdstanek, anyway they would be calls hitting keystone14:26
*** stevemar has joined #openstack-keystone14:27
*** ChanServ sets mode: +v stevemar14:27
dstaneksamueldmq: let me paint my picture for i have no idea how buys these servers are so i may be way off14:27
dstaneksamueldmq: 1 nova in 1 region that does 10 request/sec will hit use up to 10 times the second it needs to fetch policy14:28
dstaneksamueldmq: with 10 novas that's up to 10014:28
dstaneksamueldmq: with 1000 we just locked up keystone for 5 seconds and API requests fail14:29
samueldmqdstanek, are you supposing we fetch policy at every nova api call?14:29
*** e0ne has joined #openstack-keystone14:29
*** doug-fish has joined #openstack-keystone14:29
dstaneksamueldmq: no, i'm assuming they use it at every API call and then every 5 minutes need to fetch it14:30
samueldmqdstanek, sure, misunderstood14:31
samueldmqdstanek, so with 1000 novas, we have 1k policy requests on keystone14:32
*** topol has joined #openstack-keystone14:32
*** ChanServ sets mode: +v topol14:32
samueldmqdstanek, and 10k nova requests waiting for that to then proceed14:33
samueldmqdstanek, right?14:33
samueldmqdstanek, would 1k request lock up keystone?14:33
dstanekmaybe? not sure. plus there is other traffic like token validations happening on keystone14:34
samueldmqdstanek, keystone is also expected to be high-available to serve 1k novas receiving a total amount of 10k request/seconds altogether14:35
samueldmqdstanek, maybe we would be expecting too much of a single keystone instance...14:35
dstaneksamueldmq: the point is that this makes too many request simultaneously14:40
samueldmqdstanek, hmm, actually this doesn't make _simultaneously_ ... it's when the first request arrives in the middleware, after the policy is obsolete14:41
*** vmbrasseur has quit IRC14:42
samueldmqdstanek, maybe makes things slightly better, or not ?14:42
dstanekassuming my numbers for 1000 novas is correct... the operator scales keystone to handle peak traffic let say (avg * 1.45); now they have to scale much higher14:42
dstaneksamueldmq: if you assume each node is getting 10 rps then those 10 would simultaneously hit keystone14:43
lbragstaddolphm: nice, thanks14:43
samueldmqdstanek, I think the number of rps nova gets doesn't matter14:47
*** vmbrasseur has joined #openstack-keystone14:47
samueldmqdstanek, just the number of nova endpoints asking for the policy, right?14:48
*** dikonoo has quit IRC14:48
dstaneksamueldmq: how will each thread know another thread is already getting the policy?14:50
dstaneksamueldmq: i'm not trying to make this overly complicated, but caching is very hard14:51
dstanekthere's an old saying that "if your app is too slow add caching; if your app has too many bug remove caching" or something like that14:51
*** petertr7 is now known as petertr7_away14:52
lbragstadjamielennox: do you want me to add myself as an "Other Contributor" -
samueldmqdstanek, yes I completely understand your point, and I think it is valid14:54
samueldmqdstanek, that an intersting saying :p14:54
morgan_404dstanek: your app is slow, so you add caching. Now you have two problems14:54
*** marzif has quit IRC14:54
samueldmqdstanek, I think like this: the first request arrives in middleware, makes it request the policy, and let the requests pass14:54
samueldmqdstanek, don't block requests when getting the policy14:55
samueldmqdstanek, if that makes sense14:55
*** morgan_404 is now known as morgan_30214:55
samueldmqmorgan_404, hey o/14:55
dstanekmorgan_404: how critical do you think it is that policies can't be out of sync at all?14:55
dstaneksamueldmq: so you are going to add some thread coordination logic to middleware?14:55
samueldmqdstanek, actually it's 302 now14:55
morgan_302dstanek: at a given endpoint it would be bad afaict.14:55
morgan_302But between endpoint not as bad14:56
samueldmqdstanek, just a var set to fetch_mode=True, don't need overly complicated coordination14:56
morgan_302This is the classic thundering herd issue if i'm reading correctly14:56
samueldmqmorgan_302, this is exactly what we are talking about14:57
*** jistr is now known as jistr|mtg14:57
* morgan_302 wants a distributed kvs so keystone can push policy out once and all endpoints can get it14:58
morgan_302So if you use dogpile cacher, you can use an async runner to grab the policy and keep it alive for longer in memory. But lets be fair keystone distributing policy is kindof rife with issues.14:59
dstanekmorgan_302: yeah, well maybe that. but the more we talk about strange requirements the more i think HTTP isn't right for the job14:59
dstaneksamueldmq: what var?14:59
morgan_302dstanek: honestly i think policy is a config management issue still.14:59
morgan_302dstanek: part of the reason ive stayed out of this discussion for the most part15:00
dstanekmorgan_302: so the way i want thinking abut it is that right now if they roll out policy they don't reboot everything at once to reread - so they already have a period where things are out of sync15:00
morgan_302Policy is refreshed from disk automatically iirc15:01
dstaneksamueldmq: somehow thread 1 will need to know that thread 0 is getting a policy and not get one15:01
samueldmqdstanek, each request gets it thread?15:01
dstaneksamueldmq: to make matters worse what about multiple processes?15:01
dstaneksamueldmq: eventlet :-)15:01
samueldmqdstanek, well .. there is env_var that needs to be set to true, and then the request continue15:02
*** Ephur has joined #openstack-keystone15:02
dstanekso yes, basically15:02
samueldmqdstanek, doesn't matter how many times it was set to true15:02
samueldmqdstanek, dunno if that works actually :)15:02
dstaneksamueldmq: is that across threads?15:02
morgan_302So it is doable to use a single async runner15:02
dstaneki think you'll have to use a lock15:02
morgan_302And just have that lock-enforced runner (non-blocking but so one only runs) drops the file on disk15:03
dstanekmorgan_302: ++ but i think someone rules that out because it was yet another thing to have operators run15:03
*** jsavak has quit IRC15:03
morgan_302Nah. Async runner can be spawned by kystone. Its how dogpile works15:03
morgan_302If you build your tool to do thst15:03
morgan_302But... Otherwise it is a cron/other service.15:04
morgan_302I didnt say async runner was the best idea either. Just that it could be used15:04
samueldmqmorgan_302, dstanek policy in its own service?15:04
dstaneksamueldmq: no15:04
morgan_302If it drops the policy on disk all processes can load from that single endpoint. So thundering herd is limited to number of real servers without shared disk for a given service type no services * processes15:05
*** phalmos has quit IRC15:05
* morgan_302 15:05
samueldmqok so service * processes can be solved15:06
* morgan_302 dislikes having to design these types of systems. They always have tons of edge cases15:06
samueldmqmorgan_302, so a previous discussion included 'how many novas can be in the same endpoint_id?'15:06
*** Ephur has quit IRC15:06
morgan_302It is still a thundering herd15:06
samueldmqmorgan_302, if it can be 10k, it would be 10k requests at the same time15:06
morgan_302samueldmq: behind a loadbalancer? I dont know how many?15:07
*** jsavak has joined #openstack-keystone15:07
morgan_30210k would be silly15:07
morgan_302But not impossible15:07
*** phalmos has joined #openstack-keystone15:07
morgan_302This is the "what level of scale are we talking about"? And we dont have numbers15:08
samueldmqmorgan_302, a single nova region is expected to contain multiple endpoints registered in keystone?15:08
morgan_302It could15:08
samueldmqmorgan_302, or could one register a single one and use it for all its 10k novas15:08
morgan_302We dont prevent it15:08
dstanekthat's why i think we should find out what the tolerance is for differences. even as little as 10 seconds dramatically helps. this is why people often use a splay to get updates in a distributed system15:08
morgan_302All of these are possible15:08
samueldmqmorgan_302, dstanek yes, that's all possible, but is that acceptable to have a concrete and very well documented solution for this cycle15:08
samueldmqmorgan_302, dstanek as it is experimental15:09
morgan_302dstanek: part of the issue is you break requests going through the system when they hit different endpoints in a LB15:09
samueldmqso we look further starting next cycle, as this policy work will open the door for other things ..15:09
dstaneksamueldmq: since it is experimental i think using the simpler solution of having the differences would be better anyway15:09
dstaneksamueldmq: why over engineer now when we can do it later!15:09
samueldmqand we don't accept it as stable if we don't solve all that, or at least find good explanations15:10
samueldmqdstanek, if morgan's ok, I have no objection15:10
morgan_302dstanek: honestly i really think we need something like consul where keystone could just push the policy file to the dkvs.15:10
morgan_302Then the endpoints just read it15:10
morgan_302Or keystone pushes to some other similar shared *fast* storage15:10
dstaneki have no issue with that. if the past i've uses zookeeper for similar configuration15:11
morgan_302Zk is the same concept ;)15:11
dstaneki just don't want to develop a CMS inside of keystone15:11
morgan_302Hope we get that next cycle. See DLM mailinglist topic15:11
dstanekcurrent code is trending toward a publish-based cms15:11
*** roxanaghe has joined #openstack-keystone15:12
morgan_302Im also leaning towards policy doesnt belong in keystone at all15:12
morgan_302It almost needs to be a separate process/endpoint15:12
morgan_302For distribution / management15:13
* morgan_302 greatly dislikes the "be everything for everyone" mentality we keep pushing into openstack projects15:13
morgan_302There is a reason i've mostly stayed out of this convo15:14
samueldmqmorgan_302, and that is the reason ? ^15:14
samueldmqmorgan_302, I still think we can enable this as experimental, maybe simpler as dstanek proposes15:15
samueldmqthen we can talk at the summit about a policy service? cms policy? next steps, etc15:15
samueldmqand so on..15:15
samueldmqif that makes sense15:15
* morgan_302 waves at roxanaghe15:18
morgan_302roxanaghe: if guang is in the office today... Throw things at him for me :P. Mostly cause im not there to do so today ;)15:19
morgan_302roxanaghe: throw things at a person as a service ;) :P15:19
samueldmqThrowing As A Service15:20
dstanekroxanaghe: please have someone video it if you do :-) that would be great in a keystone core team trailer15:20
* lbragstad feels a spin off of coming on... 15:20
*** arunkant_ has joined #openstack-keystone15:22
roxanaghemorgan_302, hehe, I would be glad to do that only if you approve it with a +2 :)15:23
*** geoffarnold has joined #openstack-keystone15:23
roxanaghebut I would have to delegate it for today since I'm stuck with fever at home :(15:23
roxanagheSan Francisco summer...15:24
samueldmqmorgan_302, you ok with endpoints having 'different' policies for {policy_timeout} seconds, only when policy updates occur?15:27
samueldmqmorgan_302, that'd make the implementation/model slightly simpler ..15:27
dstaneksamueldmq: we should find out who will actually use this feature and then we can know how much not to build15:28
*** shunliz has quit IRC15:29
*** arunkant_ has quit IRC15:31
*** e0ne has quit IRC15:32
samueldmqdstanek, ok so the questions are: i) how much of thundering it can actually happen ? (this depends on how deployments are and how we can distribute the policy timeout)15:32
samueldmqdstanek, ii) what level of inconsistency is acceptable? some minutes when updates occur?15:33
dstanekor seconds15:33
*** arunkant_ has joined #openstack-keystone15:33
dstanekiii) what is the current level of inconsistency15:33
samueldmqdstanek, ++15:33
samueldmqdstanek, so yes, we can get the answers from ops-summit next week15:34
*** jistr|mtg is now known as jistr15:34
samueldmqdstanek, let's leave our current implementation as paranoid as it is for now15:34
dstaneksamueldmq: i think we need to wait on merging implementation until then15:34
samueldmqdstanek, and we adjust depending on the feedback15:34
samueldmqdstanek, so we can 'relax' the implementation (if needed) once we get feedback15:37
samueldmqdstanek, probably yes, were you planning to have the code merged before the ending of next week?15:37
*** henrynash has joined #openstack-keystone15:37
*** ChanServ sets mode: +v henrynash15:37
samueldmqdstanek, I am finishing some tests on the server side, so I will prepare a demo with multiple glance servers behind a proxy next week15:38
dolphmlbragstad: remember when you suggested that we put all the fernet backports into a single review sequence to avoid merge conflicts as they're approved?15:38
samueldmqdstanek, all the code will be ready, and we do the adjustments according to fedback from their midcyle15:39
lbragstaddolphm: I suggested that?15:39
dolphmlbragstad: with a couple specific patches, yes15:39
dolphmlbragstad: and also breaking out those patches i squashed so brant will review them...15:39
*** yottatsa has joined #openstack-keystone15:39
samueldmqdstanek, sounds a good plan?15:39
lbragstaddolphm: yep, I saw that15:40
dstaneksamueldmq: sure....we (gyee) need to make it clear that we are talking about policy over HTTP15:41
dolphmdid the bot that links to newly posted reviews / patchset vanish?15:41
samueldmqdstanek, what would be alternative for HTTP within openstck services?15:42
samueldmqdstanek, if any .15:42
dolphmsamueldmq: ooh, HTTPS!15:42
*** phalmos has quit IRC15:42
samueldmqdolphm, my ' ' ometime fail when I type it15:43
dstaneksamueldmq: i guess the real question is "how complicated do we want Keystone to become?"15:43
dolphmlbragstad: anywaym, i'm backporting all the fernet things into a single review sequence right now15:43
dstanekdolphm: re: the ML thread. are they actually running the Kilo client?15:43
lbragstaddolphm: thank you15:44
dolphmdstanek: *shrug*15:44
samueldmqdstanek, yes, that's the concern from morgan_30215:44
*** petertr7_away is now known as petertr715:44
openstackgerritTerry Howe proposed openstack/keystoneauth: Keep a consistent logger name for keystoneauth
samueldmqdstanek, and that's why I asked if that should be a separate service15:44
dolphmlbragstad: review sequence will start here
samueldmqdstanek, all that can be discussed at the summit I think15:44
dstanekdolphm: i'm just wondering if there is actually a problem in the server in kilo and they are confusing client side changes with a server commit15:45
samueldmqdstanek, I mean, the next steps.. if policy shouldn't  be under keystone, let's create a separate service to manage/distribute it15:45
dstaneksamueldmq: yeah, but you are pushing for L, which is why it's a problem to solve now15:45
*** piyanai has joined #openstack-keystone15:45
dolphmdstanek: i believe if you install the client from debian packaging for example, you'll get the stable/kilo client15:46
samueldmqdstanek, putting it as experimental? experimenting it is another reason to see how good/bad it can be15:46
dolphmdstanek: and their complaint seems to line up with jamie's fix, and i think jamie's fix is backportable15:46
samueldmqdstanek, another way*15:46
dolphmdstanek: sooo, i was happy to oblige :)15:46
dstaneksamueldmq: see, that is where i disagree. i don't believe that 'experimental' should be used for science experiments. it's for features we are sure we want and are just not production ready yet15:47
samueldmqdstanek, we want that, I am not saying we don't want15:47
samueldmqdstanek, but I agree that may require changes in the future15:47
*** jistr has quit IRC15:48
samueldmqdstanek, like migrating to its own service? consul in openstack?15:48
samueldmqdstanek, I am for the feature, and some operators also told us that they agree that it's useful (our email in the ops list)15:48
dstaneksamueldmq: that's what scares me...the main implementor isn't sure if it belongs in keystone :-)15:48
dstanekwhen i say we i mean keystone not openstack15:49
samueldmqdstanek, policy crud association is already in place15:49
samueldmqdstanek, we're just allowing to effectively use it by fetching15:50
samueldmqdstanek, I suppose the person who introduced the policy CRUD + endpoint policy has the answers for why they're in keystone15:50
dstanek..or how many people use it15:51
*** afazekas has joined #openstack-keystone15:51
dolphmdstanek: ++ experimental means it's intended for production, it just hasn't been proven as stable yet15:51
dolphmit's like debian's testing branch :) it's not unstable -- but there's no guarantees and limited support15:52
dstanekwhy is everyone so allergic to the phrase "out of tree"?15:53
samueldmqdstanek, today I don't know if anyone does, since we don't allow the fetch15:53
samueldmqdstanek, also we want to allow that so we can work on the other bits it opens the door for15:53
*** petertr7 is now known as petertr7_away15:53
stevemarmorgan_302: wishlist or won't fix?15:54
openstackLaunchpad bug 1484366 in Keystone "No way to specify password strength in keystone." [Undecided,New]15:54
dstaneksamueldmq: so does this current work make up an mvp? or is it still useless without more work15:54
samueldmqdstanek, mvp?15:55
samueldmqdstanek, I think it makes the existing code for policy crud + association useful15:55
samueldmqdstanek, so it's useful15:56
morgan_302stevemar: uhh wut?15:56
*** woodster_ has joined #openstack-keystone15:57
morgan_302Today i think is wont fix. But it is more wishlist and "not this cycle"15:57
dstaneksamueldmq: minimum viable product15:57
morgan_302dstanek: most valuable player! :p15:58
morgan_302More Vibrant Plant15:58
*** yottatsa has quit IRC15:58
dstanek#vote morgan_30215:58
morgan_302Maximum Vital... No wait lets stop there15:59
dstanek#vote ficus15:59
*** _cjones_ has joined #openstack-keystone15:59
*** _cjones_ has quit IRC15:59
*** _cjones_ has joined #openstack-keystone15:59
morgan_302#startvote mvp means? MostValuablePlayer, MoreVibrantPlants,ficus16:00
samueldmqdstanek, is that ^ enough to make you believe it's a mvp?16:01
stevemarmorgan_302: i was asking if should be marked as wishlist or won't fix?16:01
openstackLaunchpad bug 1484366 in Keystone "No way to specify password strength in keystone." [Undecided,New]16:01
samueldmqdstanek, (minimum viable product in this case) :)16:01
*** yottatsa has joined #openstack-keystone16:01
morgan_302Id say wont fix. But its probably wishlist and "not this cycle". I wont devote time to fixing the sql backend16:02
*** Ephur has joined #openstack-keystone16:02
morgan_302In that regard. Ldap is better and does that for us16:02
dstaneksamueldmq: no idea, it depends on who will use this in L16:02
*** jasonsb has quit IRC16:05
*** jasonsb has joined #openstack-keystone16:05
*** edmondsw has quit IRC16:06
samueldmqdstanek, sure, but again, having this in L allow us to work on the other bits next cycle, otherwise we will only get this in M, I believe16:06
dstaneksamueldmq: if this doesn't land in L you can still work on it in M. that is not a prereq16:07
*** jasonsb has quit IRC16:10
stevemarmorgan_302: i'll mark it as wishlist then16:10
*** tsubic has quit IRC16:10
*** katkapilatova has left #openstack-keystone16:10
openstackgerritMerged openstack/keystoneauth: Updated from global requirements
stevemarpoke all keystone folks presently chatting: dstanek morgan_302 dolphm samueldmq the tokenless auth patch could use some eyes/love
dstaneksamueldmq: really i guess the question for gyee is what does he need to use it and what release does he use...16:12
bretonyes, please review x.50916:13
samueldmqstevemar, looking :)16:14
*** bknudson has joined #openstack-keystone16:15
*** ChanServ sets mode: +v bknudson16:15
dolphmstevemar: sweeet, i didn't realize that had ever been published16:15
samueldmqdstanek, why asking the level of inconsistency acceptable for deployers + providing it in L isn't acceptable?16:16
samueldmqdstanek, because possibly all the policy model is bad and we shouldn't build on the top of it?16:16
samueldmqdstanek, and then fix it (new service? consul?) etc before going further?16:17
dstaneksamueldmq: i'm not sure what you are asking. but my view on this has always been that we need to find a sponsor on the ops side to help guide direction16:18
samueldmqdstanek, what I am arguing is that we already have the implementation (I'd say almost ready)16:20
*** diazjf has quit IRC16:20
samueldmqdstanek, and the message in the ops-list revealed operators interested in it16:20
dstaneksamueldmq: is it the right implementation?16:20
samueldmqdstanek, it's only missing some details (the help to guide the implementation)16:21
samueldmqdstanek, I don't know if it's the right implementation, but it's as precise as it could be (from an operator side)16:21
samueldmqdstanek, what we need now is to listen to them next week16:21
dstaneksamueldmq: and find one to use it16:22
samueldmqdstanek, and see if we are being paranoid with the cache mechanism16:22
*** gyee has joined #openstack-keystone16:22
*** ChanServ sets mode: +v gyee16:22
*** alejandrito has joined #openstack-keystone16:22
samueldmqdstanek, I suppose people who responded saying that 'synchronizing policy in endpoints behind a proxy isn't an easy task' and so will use it16:23
samueldmqdstanek, basically people saying, yes we want it should be using it16:23
*** tsubic has joined #openstack-keystone16:24
samueldmqdstanek, did you see the operators feedback on the ml?
samueldmqdstanek, I think we are ok with stackholders now, it's more a matter of adjusting the details on the implementation16:25
samueldmqif that makes sense16:25
dstaneksamueldmq: which one signed up to help with the design?16:27
samueldmqdstanek, we didn't ask for that, and that's what I am proposing we look for in the ops-summit16:28
dstaneksamueldmq: iirc the ML thread people were agreeing that the global admin thing isn't good. was there talk about fetching the policy from keystone?16:28
dstaneksamueldmq: a stakeholder should be involved in the design to some extent16:28
samueldmqdstanek, yes we also talk about fetching it from keystone, but lots of conversation are about the admin16:28
dstanekat least inform the design16:28
dstaneksamueldmq: the original email talked about fetching from keystone, but did the operators?16:29
samueldmqdstanek, sure, so we have people interested on it, and we need someone to validate/refine the design, right?16:29
*** diazjf has joined #openstack-keystone16:30
*** edmondsw has joined #openstack-keystone16:30
dstaneksamueldmq: so in a normal software development process this would be the step where you would work with the stakeholder to see how to solve the problem in a tech agnostic way. then you come up with a tech design to present and explain it's constraints. then iterate and finally build.16:31
*** openstackgerrit has quit IRC16:31
samueldmqdstanek, talking about distributing the polciies : "Not the easiest task to either get right, or make sure that the files16:31
samueldmqare distributed around in an HA setting.  But absolutely necessary.16:31
*** openstackgerrit has joined #openstack-keystone16:32
samueldmqdstanek, yeah, makes sense, but I suspect some guys here have some background in deployment, so we aren't totally lost16:32
*** petertr7_away is now known as petertr716:32
dstaneksamueldmq: you mean the ops guys?16:33
samueldmqdstanek, and what I am trying to do, at this point, is to fix it16:33
samueldmqdstanek, no, I think some of us, like adam and gyee who have been involved on the solution discussion, so that's not the case our solution is bad16:33
samueldmqdstanek, we just need to validate with a real stackholder, at this point16:33
samueldmqdstanek, that's how I see it16:34
dstaneksamueldmq: that very well could be16:36
*** ankita_wagh has joined #openstack-keystone16:36
*** ankita_wagh has quit IRC16:37
samueldmqdstanek, if we're going to fetch from an openstack server, the only thing we need to sync up with them is "how often? for how long inconsistencies are ok?"16:38
samueldmqdstanek, the other options are: don't implement that on openstack at all (and forget the other benefits16:39
samueldmqdstanek, and then deprecate the current apis for policy16:39
samueldmqdstanek, or provide a common CMS implementation, which is not our task16:39
dstaneksamueldmq: so you have 3 designs to pitch?16:40
samueldmqdstanek, so I think that for where we want to get, we are okay, just need some validation16:40
*** petertr7 is now known as petertr7_away16:40
samueldmqdstanek, basically yes: i) do it ii) don't do and deprecate the existing apis or iii) do it with a tool outside openstack (as today)16:41
*** ankita_wagh has joined #openstack-keystone16:41
samueldmqdstanek, so we are only interested on i) and ii)16:41
samueldmqdstanek, and people already said i) so do it, we just need to agree on how to do it16:42
samueldmqdstanek, that's then the validation of the architecture with operators16:42
dstaneksamueldmq: so i didn't see were anybody cared about the HTTP solution. let's wait and discuss next week.16:43
samueldmqdstanek, http solution == managing/fetching using openstack services, right?16:43
samueldmqdstanek, there are a lot of benefits ... that's the whole dynamic policies work16:44
samueldmqdstanek, like providing better interfaces for defining policies16:44
samueldmqdstanek, distributing them transparently, and so on16:44
samueldmqdstanek, gyee 's up, want to take some conversation if he is available?16:45
dstaneksamueldmq: that's find it that's the only way to solve the global admin problem, but it looks like several operators have developed their own solution to the problems16:45
*** jamielennox is now known as jamielennox|away16:45
dstaneks/find it/fine if/16:45
*** afaranha has joined #openstack-keystone16:49
*** afaranha has left #openstack-keystone16:49
samueldmqdstanek, yes and they are building their own tools on top of openstack16:50
gyeesamueldmq, \o16:50
gyeereading back16:50
*** spandhe has joined #openstack-keystone16:50
samueldmqgyee, hey, that's a huge conversation, I can summarize it for you :)16:51
samueldmqgyee, so .. I and dstanek have been discussing about the policy solution we are proposing16:52
samueldmqdstanek, .. correct me if I am wrong :)16:52
samueldmqgyee, so basically we need to validate our proposed design with operators16:52
*** henrynash has quit IRC16:52
samueldmqgyee, for example, is it acceptable to have policies inconsistent through endpoint processes when a policy update occurs? if so, for how long?16:53
samueldmqgyee, also, we need to know a bit more how is the architecture of big deployments, so we can avoid the classical thundering herd problem16:54
*** browne has quit IRC16:54
gyeethat's all depends on the deployment16:54
samueldmqgyee, for example, if the cloud has a region and there are 10k nova nodes under that region using the same endpoint_id ?16:54
samueldmqdstanek, am I missing something else?16:55
gyeesay if you revoke a token, would the Nova instance become inaccessible of it was started by that token?16:55
gyeesecurity is a process, software is a tool :)16:55
gyeeit all depends of the security requirements for a particular deployment16:56
dstanekgyee: what does the token have to do with the policy files?16:56
gyeedstanek, same deal, if policy are inconsistent for a few seconds while the system is absorbing the new policy changes, that's customer's choice16:57
gyeewe don't dictate that16:57
dstaneki don't think we can do that in a scalable way in keystone then16:57
gyeewe just provide the flexibility to allow them to make that choice16:57
navid_nick navid16:57
*** navid_ has quit IRC16:58
dstanekwe basically need a distributed datastore accessible to all nodes that use policy16:59
dstanekgyee: the problem i see is that we cause the herd because of an exact timeout17:00
*** jsavak has quit IRC17:00
gyeedstanek, isn't Keystone a distributed store?17:02
gyeeif Keystone dies, the system is unusable17:04
dstanekgyee: not in the same way as consule, zookeeper, etc.17:04
dstanekit is not as easy for use to absorb a few thousand extra concurrent requests17:04
*** petertr7_away is now known as petertr717:05
gyeedstanek, how about 50 keystone nodes spread across? :)17:05
haneefstevemar:  What does observor mean in cadf?17:05
gyeeobservor usually means read-only17:06
dstanekgyee: is that acceptable?17:06
gyeedstanek, it depends on expectations, HA, scale and performance17:06
*** jsavak has joined #openstack-keystone17:06
haneefgyee: I don't get you.  I cadf payload what does observor contain?17:07
dstanekhaneef: no idea, but this may be a good starting point
stevemarhaneef: the service that "sees" the request17:09
dstanekgyee: my though it that is we use caching things will be out of sync, so we can embrace it in the design17:09
haneefstevemar: example please?  If I do create user ,  does that mean, it is keystone?17:10
*** petertr7 is now known as petertr7_away17:10
gyeedstanek, even with consul and zookeeper, there still opportunity for things to get out of sync right?17:10
stevemarhaneef: right17:10
dstanekgyee: yes, to some extent17:10
stevemarhaneef: if it were a request to create a VM, the observer shuld be nova17:10
gyeeso we have some risk tolerance either way17:10
dstanekgyee:  the currently impl we have is going out of it's way to cause the thundering herd problem; and the question i have to the operators is how big of a problem is that?17:11
haneefstevemar: Also  question about id. Some ids are prefixed by openstack: ... does that mean, those ids are cadf specific?17:11
dstanekgyee: how many nodes they they have for either endpoints and what the average traffic across those nodes17:11
haneefe.g:  "id": "openstack:11fc2c26-0d67-4875-90d0-09186863e502"17:11
stevemarhaneef: pycadf does that if you don't specify an ID when creating the cadf event17:11
*** boris-42 has joined #openstack-keystone17:11
samueldmqdstanek, the herd problem it can suffer is: N endpoints processes which have the same id in keystone and receive an user request at the same time after the policy becomes stale, will cause N policy requests to keystone17:13
haneefstevemar: One more thing , If observer is id of service , doesn't it have to same every time?  If I do 2 user create, will I get same id for observor17:13
samueldmqdstanek, right? not endpoint vs request, as we talked wiht morgan17:13
gyeedstanek, that all depends on the size of the deployment too. HP public cloud, for example, even support multiple regions17:14
stevemarhaneef: you should, yes17:14
dstanekN * (number of concurrent eventlet threads)17:14
gyeeso replicating stuff across region is going to be a bit slower, but should now be slower than order of miliseconds17:14
stevemarhaneef: i think you're getting at this?
openstackLaunchpad bug 1428946 in Keystone "add keystone service id to observer audit" [Low,Triaged]17:14
dstanekgyee: we're not talking replication17:15
gyeewe also have monitoring tools in place to make sure things are running at *normal speed*17:15
dstanekgyee: assume you have 1000 nodes for an endpoint with an average of 10 requests per second - that means up to 10,000 requests to keystone every 5 minutes to get policy17:16
dstanekand not over 5 minutes; *at* the 5 minute mark17:16
dstanekso my question for the operators is how many nodes they have for an endpoint and how much traffic they see17:16
haneefstevemar: yes.  Isn't it  as simple as adding an identifier for kesytone in conf file?17:17
samueldmqdstanek, at the 5 minute mark if they all get a request at that exact time17:17
dstanekgyee: so that size of a deployment would need to add 20 keystone servers to handle the extra traffic...17:17
gyeedstanek, 5 minutes is nothing compare to token validation17:17
samueldmqdstanek, since that's how middleware works17:17
samueldmqdstanek, that's a good quesiton though17:18
dstaneksamueldmq: right. assuming 10 request per second they absolutely could17:18
stevemarhaneef: you could do that, but we already store the service ID in the keystone database, so we should get it from there17:18
*** lhcheng has joined #openstack-keystone17:18
*** ChanServ sets mode: +v lhcheng17:18
gyeepolicy request should be relatively fast, either the policy has been updated or not17:18
gyeeif updated, fetch and cache17:18
dstanekgyee: a few db records to at least look at, but yes relatively fast17:19
gyeemost of time, it is nothing more than a ping17:19
dstanekgyee: no, we still have to check17:19
gyeeassume we don't update policy every few mins17:19
haneefstevemar: got it. Thanks17:20
dstanekgyee: even if we 304, we still do the server side work to make sure a 304 is the correct response17:20
gyeedstanek, there's no DB hit on policy check I hope17:20
samueldmqdstanek, but that's quick, just re-calculate the max-age to tell the endpoint to ask again17:20
dstanekgyee: sure there is!17:21
samueldmqlast time it is updated have to be stored in db17:21
dstanekgyee: and whey you guys start doing dynamic policies it'll be serveral!17:21
gyeewe're not caching it on the server side?17:21
* gyee needs to read the fantastical code17:22
dstanekgyee: because we don't want it to get out of sync at all we are using the database at a system of record for when a policy is valid17:22
dstanekotherwise the policy could change in between the 5 minute mark (because an admin updated it) and only a few nodes would get the update17:22
samueldmqand that's ^ bad behind a proxy17:23
samueldmqwhere processes should apply the same access control rules at a given time17:23
gyeehold the phone, aren't we caching it on the server side the same way?17:23
gyeejust like storing per-domain config in sql right?17:23
samueldmqgyee, not yet, there is no granular CRUD on policies now, we just store the blob so far17:25
samueldmqgyee, if that's your question ..17:25
*** petertr7_away is now known as petertr717:25
*** piyanai has quit IRC17:26
gyeesamueldmq, yeah, we can think of ways to optimize it later17:26
gyeeI suspect we won't know till we have a chance to kick the tire on this one17:26
samueldmqgyee, yes, I am planning to have it next cycle17:27
gyeehell I need to find me 1000 nodes to start with :)17:27
samueldmqgyee, to easily update the policy rules without looking at the whole policy17:27
*** pgbridge has quit IRC17:27
dstanekgyee: HP public cloud?17:27
*** petertr7 is now known as petertr7_away17:27
dstanekgyee: download the code and start kicking the tires!17:28
gyeedstanek, sure17:28
gyeeI suspect they'll revoke my parking space if I use HP public cloud as the playground17:28
dstanekgyee: what is the amount of time machines in HP public cloud have inconsistent policies now?17:28
samueldmq#vote gyee17:29
dstanekgyee: and that's what i'm worried about :-) current impl is probably fine for small private cloud17:29
stevemar#vote gyee17:29
gyeedstanek, we don't have inconsistent policy right now cause we don't use that feature in public cloud17:30
dstanekgyee: you don't use policy files?17:30
gyeewe do17:30
openstackgerritTerry Howe proposed openstack/keystoneauth: Use human readable exception messages
gyeebut we don't do dynamic policy fetch right now17:30
gyeepolicy update still done via chef/ansible17:31
dstanekso after an update how long are things inconsistent?17:31
dstanekgyee: ^17:32
gyeeit depends, usually just a push of a deploy button17:32
*** HT_sergio has joined #openstack-keystone17:32
gyeeafaik, they don't go out of sync publically17:32
gyeeupdate usually goes like this17:32
samueldmqthe time ansible/chef takes, it is the maximum inconsistency, i.e updating the first and last endpoints17:32
gyee1) remove the node from LB17:33
gyee2) update it17:33
gyee3) put it back on17:33
samueldmqso there is no inconsistency17:33
gyeenot publically I don't think17:34
dstanekgyee: what you just said creates inconsistency unless you take all nodes out at the same time!17:34
*** diazjf has left #openstack-keystone17:35
samueldmqdstanek, or half of them17:35
gyeedstanek, take x number of nodes off line, update them, then swap17:35
samueldmqdstanek, update a half, and them the other half later :)17:35
samueldmqgyee, ++17:35
dstanekgyee: exactly, so for some periiod of time some nodes are serving old policy and some are serving the new policy17:36
gyeebut the ones serving new policy are not back onto LB till the swap17:36
dstaneksamueldmq: i think x usually depends on the current traffic and other things. i've not done amy 50/50 swaps in the past; that's why i'm asking17:37
dstanekgyee: so you do a 50% swap?17:37
gyeedoesn't have to be 50/5017:37
dstanekgyee: so you only have a few seconds on inconsistency then17:38
gyeeprobably, depending on how quickly they can switch the nodes over17:39
gyeeI can double check with the ops guys17:39
samueldmqdstanek, is it possible to lock/coordinate the policy fetch in a system var? so it would be shared even across eventlet threads17:41
samueldmqdstanek, or even a file ..17:41
*** browne has joined #openstack-keystone17:42
dstaneksamueldmq: sure, it's possible - don't know if it's wise yet17:42
gyeesamueldmq, lets not trying to solve that for now17:42
gyeeif folks can't tolerate inconsistency for a short period of time, let them do it the old fashion way17:43
gyeetake down the nodes and do chef/ansible17:43
samueldmqgyee, k just making sure the herd problem will be solved as well ;)17:43
gyeeits really about choice and risk management17:44
*** ankita_w_ has joined #openstack-keystone17:44
samueldmqgyee, dstanek k let's keep on the requirements now17:44
gyeewhat requirement? zero inconsistency?17:45
dstanekgyee: yes, that's one17:45
dstanekthat's what causes the herd17:45
gyeeI don't think we'll ever going to achieve that17:45
gyeenot even with zookeeper I don't think17:46
dstanekgyee: we've got lots of code and a new DB table to try17:46
gyeebut sure, if we're gonna do it, do it big :)17:46
*** Ephur_ has joined #openstack-keystone17:46
dstanekgyee: i totally agree and what i was saying yesterday(?) is that we should say what the timeframe for inconsistency will be for the experimental feature and go with that17:46
*** ankita_wagh has quit IRC17:47
gyeedstanek, no objection here17:47
dstanekgyee: i agre with gyee[-2] not the 'go big'17:47
gyeego big or go home :)17:47
dstaneki'm very comfortable saying that this is experimental any under the default config your policy could be out of whack for 1 minute or whatever17:48
samueldmqyou guys are fun to work with :)17:48
dstanekgyee: side note. today is the first pre-season game17:48
*** Ephur has quit IRC17:48
gyeego browns!17:49
gyeeand bucks17:49
gyeedstanek, 1 minutes inconsistency may not fly, 5 seconds, sure17:50
*** Ephur has joined #openstack-keystone17:50
*** diazjf has joined #openstack-keystone17:50
dstanekgyee: you're free to reconfigure the default and overload your keystone :-)17:51
gyeeif network latency is more than a few miliseconds, all hell break loose17:51
*** Ephur_ has quit IRC17:51
*** petertr7_away is now known as petertr717:51
samueldmqdstanek, haha that'd happen if we only depend on the timeout17:51
gyeesamueldmq, yeah, we should avoid DB hit on check17:52
samueldmqgyee, we can't do that I believe, think of proxied keystone servers17:52
samueldmqgyee, proxied (don't know if this word even exists)17:53
samueldmqgyee, the only way a keystone process can check if the policy is valid is looking into db :/17:53
gyeeI wonder if we can make use of notification mechanism17:56
gyeeanyway, more investigation work to do17:56
samueldmqgyee, I think there is no guarantee the other side is going to receive the notification17:56
samueldmqgyee, I've heard something like that17:57
gyeeI mean notification among keystone nodes to update the cache17:58
*** rm_work is now known as rm_work|away17:58
*** rm_work|away is now known as rm_work18:00
dstanekgyee: now you're getting crazy18:00
*** ankita_wagh has joined #openstack-keystone18:02
*** narengan has quit IRC18:02
*** ankita_w_ has quit IRC18:04
gyeedstanek, yeah, btw, how does consul or zookeeper sync up?18:05
*** ankita_w_ has joined #openstack-keystone18:06
*** ankita_wagh has quit IRC18:09
dstanekgyee: each has their own replication ability; i think consul's is a separate server/application to run18:10
gyeenice, lemme play with it to see how it goes18:15
*** piyanai has joined #openstack-keystone18:17
*** petertr7 is now known as petertr7_away18:23
*** piyanai has quit IRC18:23
*** fangzhou has joined #openstack-keystone18:27
*** ParsectiX has joined #openstack-keystone18:31
*** ParsectiX has quit IRC18:34
*** petertr7_away is now known as petertr718:39
*** narengan has joined #openstack-keystone18:40
openstackgerritMerged openstack/python-keystoneclient: Stop using .keys() on dicts where not needed
*** gyee has quit IRC18:45
*** piyanai has joined #openstack-keystone18:47
*** HT_sergio has quit IRC18:47
*** diazjf has left #openstack-keystone18:48
*** nkinder has joined #openstack-keystone18:49
*** atiwari1 has quit IRC18:49
*** nkinder has quit IRC18:50
*** jsavak has quit IRC18:51
*** ankita_wagh has joined #openstack-keystone18:52
*** jasonsb has joined #openstack-keystone18:52
*** narengan has quit IRC18:53
*** jsavak has joined #openstack-keystone18:54
*** narengan has joined #openstack-keystone18:54
*** ankita_w_ has quit IRC18:55
*** akscram has quit IRC18:56
*** narengan has quit IRC18:57
*** narengan has joined #openstack-keystone18:57
*** akscram has joined #openstack-keystone18:58
*** narengan has quit IRC18:58
*** jsavak has quit IRC18:58
*** Navid_ has joined #openstack-keystone18:58
*** jsavak has joined #openstack-keystone18:59
*** narengan has joined #openstack-keystone18:59
samueldmqdstanek, have you heard about any python tool to check architectural constraints?19:00
samueldmqdstanek, or something like that ...19:00
*** vivekd has joined #openstack-keystone19:00
samueldmqdstanek, I just noticed that some drivers don't inherit from the Driver class defined at core.py19:00
dstaneksamueldmq: what sort of architectural constraints are you referring to?19:00
*** narengan_ has joined #openstack-keystone19:01
samueldmqdstanek, look at endpoint_policy/backends/sql for example19:01
samueldmqdstanek, to this ^19:01
dstaneki don't believe i've heard of anything that can automatically check architectural constraints19:01
*** henrynash has joined #openstack-keystone19:01
*** ChanServ sets mode: +v henrynash19:01
dstaneksamueldmq: that's not an architecture thing at all. technically it doesn't have to subclass it, but we probably want it to19:03
*** narengan has quit IRC19:03
samueldmqdstanek, hmm, I thought it was part of our architecture, like defining classes, the communication between them and so on19:04
samueldmqdstanek, although I was wondering how that works if it don't inherit from the Driver class defined there19:04
dstanekpython is duck-typed so it doesn't need to19:05
*** phalmos has joined #openstack-keystone19:05
samueldmqdstanek, facepalm19:07
*** bapalm_ is now known as bapalm19:07
samueldmqdstanek, had forget about that :/19:07
samueldmqdstanek, all this policy discussion today made me crazy I think19:07
*** pgbridge has joined #openstack-keystone19:12
*** phalmos has quit IRC19:17
samueldmqdstanek, I will adapt the implementation to variate the policy timeout based on the endpoint_id (currently it is on policy_id)19:20
*** petertr7 is now known as petertr7_away19:20
samueldmqdstanek, tomorrow we can discuss a little bit more about it with gyee19:20
samueldmqdstanek, sounds good ? or have you any idea / conclusion already?19:20
samueldmqdstanek, I learned he wants 0 inconsistency .. like we're doing now19:21
samueldmqdstanek, we need to solve the herd problem though19:21
*** yottatsa has quit IRC19:23
*** phalmos has joined #openstack-keystone19:24
*** petertr7_away is now known as petertr719:25
*** piyanai has quit IRC19:27
*** piyanai has joined #openstack-keystone19:31
*** fifieldt_ has joined #openstack-keystone19:33
*** fifieldt has quit IRC19:34
*** alejandrito has quit IRC19:44
anteayastevemar: do you feel like updating this bug?
openstackLaunchpad bug 1484448 in devstack "ERROR: openstack Missing parameter(s): Set a service URL, with --os-url, OS_URL or auth.url" [Undecided,New]19:48
anteayaor jamielennox|away?19:48
*** piyanai has quit IRC19:53
*** spandhe has quit IRC19:59
*** piyanai has joined #openstack-keystone19:59
stevemaranteaya: hmm, i thought i caught all the dupes20:10
stevemarlooks like i missed one20:11
stevemaranteaya: marked as dupe20:14
*** jsavak has quit IRC20:15
*** jsavak has joined #openstack-keystone20:21
*** petertr7 is now known as petertr7_away20:27
*** tjcocozz has joined #openstack-keystone20:27
*** alejandrito has joined #openstack-keystone20:28
*** jasonsb has quit IRC20:29
*** jasonsb has joined #openstack-keystone20:29
*** petertr7_away is now known as petertr720:32
*** jasonsb has quit IRC20:34
htrutahenrynash: are you still here?20:35
*** topol has quit IRC20:35
*** mylu has joined #openstack-keystone20:37
*** ayoung has joined #openstack-keystone20:37
*** ChanServ sets mode: +v ayoung20:37
*** phalmos has quit IRC20:37
dstanekanyone have thoughts on ?20:41
openstackLaunchpad bug 1240163 in Keystone "Can't store a PKI token with a large catalog" [High,Triaged] - Assigned to Adam Young (ayoung)20:41
ayoungdstanek, deprecate PKI tokens20:42
dstanekayoung: i'm happy with a wontfix we'll be deprecating pki tokens soon :-)20:43
ayoungdstanek, testing that the tokens starts with MI[IJK]  is probably a better short term fix20:44
*** narengan_ has quit IRC20:46
*** narengan has joined #openstack-keystone20:46
ayoung  dstanek20:46
*** rdo has quit IRC20:50
*** woodster_ has quit IRC20:50
*** narengan has quit IRC20:50
*** narengan has joined #openstack-keystone20:51
*** Guest61394 is now known as med_20:51
*** med_ has joined #openstack-keystone20:51
*** rdo has joined #openstack-keystone20:52
*** narengan has quit IRC20:52
*** narengan has joined #openstack-keystone20:52
*** woodster_ has joined #openstack-keystone20:54
*** mylu has quit IRC20:57
*** narengan has quit IRC20:57
*** mylu has joined #openstack-keystone20:59
*** raildo is now known as raildo-afk20:59
*** mylu has quit IRC21:00
*** narengan has joined #openstack-keystone21:03
*** tjcocozz has quit IRC21:04
*** petertr7 is now known as petertr7_away21:07
*** piyanai has quit IRC21:07
*** narengan has quit IRC21:16
*** narengan has joined #openstack-keystone21:16
*** jsavak has quit IRC21:18
*** narengan has quit IRC21:21
*** ayoung has quit IRC21:21
*** narengan has joined #openstack-keystone21:22
*** ankita_wagh has quit IRC21:26
*** ankita_wagh has joined #openstack-keystone21:27
openstackgerritBrant Knudson proposed openstack/keystone: Remove "tenants" from user_attribute_ignore default
*** ankita_wagh has quit IRC21:31
dstanekbknudson: any reason for me to not mark this as invalid for Keystone?
openstackLaunchpad bug 1359995 in Keystone "Tempest failed to delete user" [Undecided,Incomplete]21:32
bknudsondstanek: go ahead. I'll open a new bug next time there's a failure and the keystone logs are useless.21:33
*** narengan has quit IRC21:35
*** narengan has joined #openstack-keystone21:36
*** narengan has quit IRC21:40
*** alejandrito has quit IRC21:41
*** fangzhou has quit IRC21:41
*** gyee has joined #openstack-keystone21:42
*** ChanServ sets mode: +v gyee21:42
*** ankita_wagh has joined #openstack-keystone21:42
*** ankita_wagh has quit IRC21:43
*** ankita_wagh has joined #openstack-keystone21:43
*** sigmavirus24 is now known as sigmavirus24_awa21:46
openstackgerritBrant Knudson proposed openstack/keystone: Bandit config updates
openstackgerritBrant Knudson proposed openstack/keystone: Enable bandit check for password_config_option_not_marked_secret
*** stevemar has quit IRC21:51
*** morgan_302 has quit IRC22:04
*** morganfainberg has joined #openstack-keystone22:07
*** ChanServ sets mode: +v morganfainberg22:07
*** doug-fish has left #openstack-keystone22:11
*** gyee is now known as gyee_50022:12
*** ayoung has joined #openstack-keystone22:13
*** ChanServ sets mode: +v ayoung22:13
*** jecarey has quit IRC22:19
openstackgerritRoxana Gherle proposed openstack/keystone: Domain configs are loaded with wrong type from db
openstackgerritayoung proposed openstack/python-keystoneclient: Shorten PKI Token Identifier to MI
*** gordc has quit IRC22:25
openstackgerritVivek Dhayaal proposed openstack/keystone: Stable Keystone Driver Interfaces
*** stevemar has joined #openstack-keystone22:26
*** ChanServ sets mode: +v stevemar22:26
openstackgerritMerged openstack/keystonemiddleware: Updated from global requirements
*** jamielennox|away is now known as jamielennox22:33
*** vivekd has quit IRC22:34
ayoungroxanaghe, so...good catch on the bug.  I'd recommend making the patch title the positive:22:42
ayoungSomething like: maintain datatypes when loading configs frm DB22:43
ayoungit should be saying what you are doing, not what the bug was that you are fixing.22:44
roxanagheayoung, ok - thanks for suggestion - I'll make it positive22:48
*** edmondsw has quit IRC22:48
ayoungroxanaghe, I added a rating22:49
ayoungroxanaghe, I wonder if it makes sense to ever convert?  If so...does doing a blanket convert=False might break something else...22:49
ayoungneed some context on that change, I think22:49
openstackgerritMerged openstack/python-keystoneclient: Updated from global requirements
roxanagheayoung: so I added that optional flag in oslo.config recently22:51
ayoungroxanaghe, so there is one case where we want ot explicitly convert?22:51
ayoungor, some other project did?22:51
roxanagheif we change it with enforce_type=True always it will break a lot of other code mostly because people use None as values and they want None not the string 'None'22:52
roxanaghe*None as an override value22:53
ayoungroxanaghe, what is the default you set in upstream?22:53
roxanagheayoung, False22:53
ayoungroxanaghe, so that is no change in default behavior,right 422:54
roxanaghebecause that was the behaviour previously - it would override the config with whatever value you pass to the function, I added the enforcement part22:55
roxanagheayoung, right22:55
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Disable memory caching of tokens
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Seperate standalone cache tests
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Handle memcache pool arguments collectively
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Import _memcache_pool normally
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Create Environment cache pool
*** hrou has quit IRC22:55
*** morganfainberg is now known as morgan_40422:57
*** woodster_ has quit IRC23:00
openstackgerritRoxana Gherle proposed openstack/keystone: Maintain datatypes when loading configs from DB
*** ayoung has quit IRC23:07
*** jamielennox is now known as jamielennox|away23:18
*** stevemar has quit IRC23:20
*** stevemar has joined #openstack-keystone23:21
*** ChanServ sets mode: +v stevemar23:21
*** geoffarnold has quit IRC23:22
*** stevemar has quit IRC23:24
*** spandhe has joined #openstack-keystone23:26
*** geoffarnold has joined #openstack-keystone23:32
*** gildub has joined #openstack-keystone23:36
openstackgerritBrant Knudson proposed openstack/keystone: Fix logging in federation/
openstackgerritBrant Knudson proposed openstack/keystone: Enhance tests for saml2 signing exception logging
*** zzzeek has quit IRC23:42
openstackgerritBrant Knudson proposed openstack/keystone: Fix logging in federation/
openstackgerritBrant Knudson proposed openstack/keystone: Enhance tests for saml2 signing exception logging
*** jamielennox|away is now known as jamielennox23:47
openstackgerritBrant Knudson proposed openstack/keystone: Documentation for other services
*** markvoelker has quit IRC23:50
*** topol has joined #openstack-keystone23:57
*** ChanServ sets mode: +v topol23:57

Generated by 2.14.0 by Marius Gedminas - find it at!