Friday, 2018-01-12

*** spzala has joined #openstack-keystone00:02
*** markvoelker has joined #openstack-keystone00:04
*** bigdogstl has quit IRC00:11
*** bigdogstl has joined #openstack-keystone00:11
*** spzala has quit IRC00:12
*** nicolasbock has joined #openstack-keystone00:19
*** bigdogstl has quit IRC00:21
*** bigdogstl has joined #openstack-keystone00:27
*** bigdogstl has quit IRC00:31
*** edmondsw has joined #openstack-keystone00:37
*** bigdogstl has joined #openstack-keystone00:41
*** edmondsw has quit IRC00:42
*** bigdogstl has quit IRC00:46
*** bigdogstl has joined #openstack-keystone00:50
*** markvoelker has quit IRC00:53
*** markvoelker has joined #openstack-keystone00:54
*** threestrands has quit IRC00:58
*** markvoelker has quit IRC00:58
*** itlinux has joined #openstack-keystone01:01
*** Dinesh_Bhor has joined #openstack-keystone01:01
*** threestrands has joined #openstack-keystone01:02
*** threestrands has quit IRC01:03
*** threestrands has joined #openstack-keystone01:04
*** markvoelker has joined #openstack-keystone01:04
*** markvoelker has quit IRC01:09
*** threestrands has quit IRC01:11
*** markvoelker_ has joined #openstack-keystone01:13
*** afred312 has quit IRC01:22
*** afred312 has joined #openstack-keystone01:24
*** threestrands has joined #openstack-keystone01:24
*** threestrands has quit IRC01:24
*** threestrands has joined #openstack-keystone01:24
*** threestrands has quit IRC01:25
*** threestrands has joined #openstack-keystone01:26
*** afred312 has quit IRC01:26
*** gyee has quit IRC01:26
*** zhurong has joined #openstack-keystone01:45
*** dave-mccowan has joined #openstack-keystone01:46
*** threestrands has quit IRC01:54
openstackgerritwangqiang-bj proposed openstack/keystone master: fix wrong url link of User trusts
*** afred312 has joined #openstack-keystone01:56
*** threestrands has joined #openstack-keystone01:56
*** kukacz has quit IRC02:00
*** afred312 has quit IRC02:01
*** kukacz_ has joined #openstack-keystone02:01
lbragstadwxy: looks like is struggling with zuul :-/02:02
wxyyeah. zuul restarts everyday...02:03
lbragstadi noticed a few emails about that02:03
*** afred312 has joined #openstack-keystone02:03
lbragstadhopefully that last recheck does the trick02:03
wxyand there were almost 270+ patch wait for check yesterday.02:03
lbragstadhopefully the break over the weekend lets things catch up02:04
wxyeven it seems that zuulv3 is not stable enough these days.02:04
lbragstadsounds like they're flushing out issues, which is good02:04
lbragstadoutside of zuul issues that are out of my control - is there anything i can help with?02:06
lbragstadi plan to review the unified limit controller tomorrow02:06
lbragstadi was caught up with the manager bits and morgan reviewed it02:06
lbragstadi spent most of today on application credentials, so i should be free to look through all the controller pieces for limits02:07
wxyThanks. I've refreshed the patch locally already. I'll submit it later.02:11
wxyI have a question about system scope.
wxyhow could this patch passed py27, since it changed the scope to only system.02:12
wxylbragstad: the test shows that it only uses project scoped token.
wxyMaybe I missed something?02:15
lbragstadoh - sure02:15
lbragstadthat's because we landed this in oslo policy02:15
lbragstadso - a project has to be configured to enforce scope02:16
wxyoh, the default value is False.02:17
lbragstadyeah - at least for now for backwards compatibility02:18
lbragstada warning will be logged through02:18
*** markvoelker_ has quit IRC02:31
*** annp has joined #openstack-keystone02:39
*** Dinesh_Bhor has quit IRC02:44
*** Dinesh_Bhor has joined #openstack-keystone02:47
ayoung  cmurphy care to pull the trigger?  Time to get this stack moving.02:56
*** Dinesh_Bhor has quit IRC03:02
*** bigdogstl has quit IRC03:14
*** edmondsw has joined #openstack-keystone03:14
*** bigdogstl has joined #openstack-keystone03:18
*** edmondsw has quit IRC03:18
*** itlinux has quit IRC03:19
*** bigdogstl has quit IRC03:33
*** dave-mccowan has quit IRC03:33
*** bigdogstl has joined #openstack-keystone03:48
*** Dinesh_Bhor has joined #openstack-keystone03:56
*** bigdogstl has quit IRC03:57
*** bigdogstl has joined #openstack-keystone03:58
*** Dinesh_Bhor has quit IRC03:58
*** ayoung has quit IRC03:58
openstackgerritchenpengzi proposed openstack/keystone master: Modify users-list-response links: users list response links not contain identity
*** zhurong has quit IRC04:03
openstackgerritchenpengzi proposed openstack/keystone master: Modify users-list-response links:
*** bigdogstl has quit IRC04:05
*** bigdogstl has joined #openstack-keystone04:07
*** markvoelker has joined #openstack-keystone04:11
*** bigdogstl has quit IRC04:12
*** edmondsw has joined #openstack-keystone04:18
*** nicolasbock has quit IRC04:21
*** edmondsw has quit IRC04:22
*** links has joined #openstack-keystone04:31
*** bigdogstl has joined #openstack-keystone04:32
*** itlinux has joined #openstack-keystone04:32
*** bigdogstl has quit IRC04:37
*** namnh has joined #openstack-keystone04:51
*** bigdogstl has joined #openstack-keystone05:04
*** bigdogstl has quit IRC05:15
*** Dinesh_Bhor has joined #openstack-keystone05:39
*** zhurong has joined #openstack-keystone05:42
*** Dinesh_Bhor has quit IRC05:52
*** Dinesh_Bhor has joined #openstack-keystone05:52
*** Dinesh_Bhor has quit IRC06:07
*** Dinesh_Bhor has joined #openstack-keystone06:08
*** Dinesh_Bhor has quit IRC06:12
openstackgerritwangqiang-bj proposed openstack/keystone master: adjust response code in order of
*** bigdogstl has joined #openstack-keystone06:17
*** Dinesh_Bhor has joined #openstack-keystone06:18
*** bigdogstl has quit IRC06:22
*** bigdogstl has joined #openstack-keystone06:23
*** harlowja has quit IRC06:35
*** bigdogstl has quit IRC06:41
openstackgerritOpenStack Proposal Bot proposed openstack/keystone master: Imported Translations from Zanata
openstackgerritwangqiang-bj proposed openstack/keystone master: put response code in table of ''''
*** Dinesh_Bhor has quit IRC06:54
*** Dinesh_Bhor has joined #openstack-keystone06:55
*** Dinesh_Bhor has quit IRC06:56
*** daidv has quit IRC06:57
*** annp has quit IRC06:57
*** annp has joined #openstack-keystone06:58
*** daidv has joined #openstack-keystone06:58
*** Dinesh_Bhor has joined #openstack-keystone07:01
*** rha has quit IRC07:03
*** harlowja has joined #openstack-keystone07:10
*** harlowja has quit IRC07:10
openstackgerritwangqiang-bj proposed openstack/keystone master: adjust response code order in ''''
*** bigdogstl has joined #openstack-keystone07:16
*** itlinux has quit IRC07:20
*** bigdogstl has quit IRC07:20
*** rcernin has quit IRC07:28
*** pcaruana has joined #openstack-keystone07:29
*** Dinesh_Bhor has quit IRC07:35
*** Dinesh_Bhor has joined #openstack-keystone07:37
*** Dinesh_Bhor has quit IRC07:41
openstackgerritOpenStack Proposal Bot proposed openstack/keystonemiddleware master: Imported Translations from Zanata
*** bigdogstl has joined #openstack-keystone07:46
*** Dinesh_Bhor has joined #openstack-keystone07:46
*** threestrands has quit IRC07:56
*** bigdogstl has quit IRC07:57
*** tesseract has joined #openstack-keystone08:08
*** bigdogstl has joined #openstack-keystone08:12
*** bigdogstl has quit IRC08:16
*** AlexeyAbashkin has joined #openstack-keystone08:23
*** Dinesh_Bhor has quit IRC08:45
*** Dinesh_Bhor has joined #openstack-keystone08:47
openstackgerritwanghui proposed openstack/keystone master: adjust response code order in ''''
*** bigdogstl has joined #openstack-keystone08:58
openstackgerritThomas Bechtold proposed openstack/keystone-tempest-plugin master: Use openstackdocstheme for docs and release notes
*** bigdogstl has quit IRC09:10
*** zhurong has quit IRC09:20
*** vaishali has quit IRC09:21
*** eglute has quit IRC09:21
*** sxc731 has joined #openstack-keystone09:21
*** dikonoor has joined #openstack-keystone09:25
*** Dinesh_Bhor has quit IRC09:25
*** eglute has joined #openstack-keystone09:26
*** vaishali has joined #openstack-keystone09:26
*** bigdogstl has joined #openstack-keystone09:28
*** Dinesh_Bhor has joined #openstack-keystone09:28
*** bigdogstl has quit IRC09:33
*** Dinesh_Bhor has quit IRC09:40
openstackgerritMerged openstack/keystone master: Add db operation for unified limit
*** bigdogstl has joined #openstack-keystone10:00
*** daidv has quit IRC10:10
*** bigdogstl has quit IRC10:12
*** StefanPaetowJisc has joined #openstack-keystone10:14
*** namnh has quit IRC10:15
*** wxy has quit IRC10:16
*** StefanPaetowJisc has quit IRC10:19
*** sambetts|afk is now known as sambetts10:23
openstackgerritColleen Murphy proposed openstack/keystone master: Add Application Credentials manager
*** markvoelker has quit IRC10:44
*** mvk has quit IRC10:52
*** wxy has joined #openstack-keystone10:54
wxyping cmurphy: I downloaded your controller patch and test with RESTful request.10:55
cmurphywxy: you mean that's where you saw the db reference error?10:57
cmurphywxy: okay i'll see if i can reproduce that way, thanks!10:57
wxypart of pbr test:
*** AlexeyAbashkin has quit IRC10:57
cmurphyi thought this would work since trusts does it the same way but maybe that's why trusts doesn't use a foreign key for roles10:58
wxycmurphy: not sure if it happened only in my env. mysql version: Ver 14.14 Distrib 5.7.19, for Linux (x86_64) using  EditLine wrapper11:02
*** edmondsw has joined #openstack-keystone11:03
*** mvk has joined #openstack-keystone11:07
openstackgerritwangxiyuan proposed openstack/keystone master: Add limit provider
openstackgerritwangxiyuan proposed openstack/keystone master: Implement policies for limits
openstackgerritwangxiyuan proposed openstack/keystone master: Expose unified limit APIs
wxycmurphy: I have to go home now. leave some information to you if needed: SQLAlchemy 1.1.15, oslo.db 4.31.0. Bye.11:07
*** edmondsw has quit IRC11:08
cmurphywxy: alright, have a good night!11:09
*** AlexeyAbashkin has joined #openstack-keystone11:12
*** bigdogstl has joined #openstack-keystone11:16
*** odyssey4me has quit IRC11:19
*** bigdogstl has quit IRC11:28
*** nicolasbock has joined #openstack-keystone11:36
*** odyssey4me has joined #openstack-keystone11:42
*** odyssey4me has quit IRC11:45
*** odyssey4me has joined #openstack-keystone11:45
*** odyssey4me has quit IRC11:53
*** odyssey4me has joined #openstack-keystone11:54
*** odyssey4me has quit IRC11:58
*** odyssey4me has joined #openstack-keystone11:58
*** StefanPaetowJisc has joined #openstack-keystone11:59
*** raildo has joined #openstack-keystone12:02
*** StefanPaetowJisc has quit IRC12:04
*** annp has quit IRC12:07
*** edmondsw has joined #openstack-keystone12:11
*** odyssey4me has quit IRC12:13
*** odyssey4me has joined #openstack-keystone12:13
*** bigdogstl has joined #openstack-keystone12:16
*** bigdogstl has quit IRC12:30
*** sxc731 has quit IRC12:35
*** markvoelker has joined #openstack-keystone12:45
openstackgerritColleen Murphy proposed openstack/keystone master: Add application credentials driver
openstackgerritColleen Murphy proposed openstack/keystone master: Add Application Credentials manager
openstackgerritColleen Murphy proposed openstack/keystone master: Extract expiration validation to utils
*** dave-mccowan has joined #openstack-keystone12:54
*** bigdogstl has joined #openstack-keystone13:02
*** hrybacki has quit IRC13:10
*** hrybacki has joined #openstack-keystone13:13
*** bigdogstl has quit IRC13:19
*** bigdogstl has joined #openstack-keystone13:19
*** markvoelker has quit IRC13:20
*** sxc731 has joined #openstack-keystone13:28
*** efried is now known as fried_rice13:38
*** bigdogstl has quit IRC13:40
*** panbalag has joined #openstack-keystone13:41
*** panbalag has quit IRC13:48
*** david-lyle has quit IRC13:49
bhagyashrisiawn: Hi, are you around?13:54
bhagyashrissorry for spell mistake13:55
cmurphybhagyashris: he lives in australia so you might have to wait a few hours, you're also more likely to catch him in #openstack-infra than here13:57
cmurphybhagyashris: if there's something we can help you with go ahead and ask your question and if someone's around who can answer we will13:57
bhagyashriscmurphy: thank you for the inputs :) just request to review one of my patch
cmurphybhagyashris: #openstack-qa would be a better place to ask for that, the keystone team can't help much with devstack :)14:01
*** panbalag has joined #openstack-keystone14:01
bhagyashriscmurphy: ok thank you :) actually he was give +2 on my patch earlier but becuase of some review comments it's removed so i am trying to contact him but yeah thank you for your suggestion to contact the devstack team14:03
*** panbalag has left #openstack-keystone14:05
*** bigdogstl has joined #openstack-keystone14:10
*** bigdogstl has quit IRC14:15
*** edmondsw has quit IRC14:19
*** edmondsw has joined #openstack-keystone14:19
*** ayoung has joined #openstack-keystone14:22
*** edmondsw has quit IRC14:24
*** bigdogstl has joined #openstack-keystone14:26
*** odyssey4me has quit IRC14:30
*** odyssey4me has joined #openstack-keystone14:31
*** odyssey4me has quit IRC14:32
*** odyssey4me has joined #openstack-keystone14:33
*** edmondsw has joined #openstack-keystone14:52
*** markvoelker has joined #openstack-keystone14:52
*** bigdogstl has quit IRC15:09
*** bigdogstl has joined #openstack-keystone15:10
*** sxc731 has quit IRC15:12
*** sxc731 has joined #openstack-keystone15:13
*** abhi89 has quit IRC15:26
*** dansmith is now known as superdan15:31
*** AlexeyAbashkin has quit IRC15:34
lbragstadkmalloc: apparently stable/pike is broken
lbragstad^ that should fix it pending what zuul says15:37
lbragstadi've ask smcginnis to review and push it through if you don't have the time to look into it15:37
*** david-lyle has joined #openstack-keystone15:38
sxc731lbragstad: hello!15:45
lbragstadsxc731: o/15:46
lbragstadhow goes it?15:46
sxc731lbragstad: not too bad, thank you and yourself?15:46
lbragstadwell - it's friday :015:46
sxc731True enough ;-)15:46
sxc731Remember  you gave me that trick on enabling debugging for KS caching?  So I now know my caching isn't working but it seems the cause is that different keys are generated for the same token...15:47
sxc731Simple test: try to validate the same token twice in a row.  I get:15:48
sxc731CACHE_GET: key115:49
sxc731CACHE_GET: key115:49
sxc731CACHE_PUT: key115:49
sxc731then 2nd validation:15:49
sxc731CACHE_GET: key215:49
sxc731CACHE_GET: key215:49
sxc731CACHE_PUT: key215:49
sxc731I have compared the content of the token, which is output at CACHE_PUT time and it's exactly the same15:49
sxc731This is within a single process.  What could causing this?15:49
sxc731(the issue is that key1 != key2)15:50
sxc731(not sure why I get 2x GET but not too big a deal I gather)15:50
lbragstadis this for authenticating or only validating a token?15:51
lbragstadso you're doing this with GET /v3/auth/tokens and X-Auth-Token and X-Subject-Token in the headers15:51
sxc731Yep, this is still taken from dolph's benchmark15:52
lbragstaddo you have your config (sans sensitive data) handy?15:53
sxc731Sure (it's not a prod env anyway).  Let me paste it15:53
lbragstadcool - i want to try and recreate locally with caching debug enabled15:53
*** cburgess has quit IRC15:57
sxc731(essentially this is a standard OSA cfg, with the cache debugging stuff added)15:58
*** dikonoor has quit IRC15:58
lbragstadcool - so not much has changed since the last time i looked15:58
lbragstadlet me set this up locally and see what happens15:58
sxc731Would you like a quick look at the log excerpt; perhaps you can spot smth odd?15:59
*** cburgess has joined #openstack-keystone15:59
*** dikonoor has joined #openstack-keystone15:59
sxc731I'm just wondering: could this explain the perceived regression between our ~150ms validation time compared with Dolph's ~6ms from Feb 2015?  Surely someone would have noticed, right??16:01
sxc731Dolph's catalog is quite a bit slimmer than mine but that doesn't explain.  Do you guys have tests that validate caching?16:02
lbragstadso - i can validate that caching is on16:10
lbragstadbut i'm not seeing the debug out put16:10
sxc731Hmm, that's weird; have you looked in the main keystone.log?  Let me paste my sytemd Exec line16:11
sxc731First you need the following 4 lines to enable cache debug output (took me a while to figure this out too ;-)16:12
*** itlinux has joined #openstack-keystone16:12
sxc731debug = True16:13
sxc731default_log_levels = ... , oslo.cache=DEBUG,...16:13
sxc731debug_cache_backend = True16:13
lbragstadchecking those values16:13
lbragstadah - i'm missing the default_log_levels bit16:14
lbragstadadding that16:14
sxc731Yes, your cache seems to make a difference from ~200ms down to 5016:14
sxc731Yeah, the default_log_levels thing was a real treat to figure out16:15
sxc731(it's always obvious in hindsight ;-)16:15
lbragstadgranted - i'm running everything locally16:16
lbragstadok - so i have two different tokens16:18
lbragstadone for X-Auth-Token and one for X-Subject-Token16:19
lbragstadand i see two different cache hits16:19
sxc731Yep my test was the same and I guess what you see shows it's working ok16:19
lbragstadkey 57c045e2c35baec049d532a74edba38bd5328ae7 is x-auth-token16:20
sxc731Let me paste my output16:20
lbragstadand key 8edd04a60920836974fc8dbefc9b58badffdfda9 is the x-subject-token16:20
*** itlinux has quit IRC16:20
*** hoonetorg has quit IRC16:22
sxc731First validation round:
sxc7312nd round (about 2 secs later):
openstackgerritayoung proposed openstack/keystone master: Implement controller logic for system group assignments
sxc731(this output is obviously quite distilled as I struggled to make sense of it otherwise).  If you compare the subject tokens which are output at CACHE_SET time (I have) you'll see the token is exactly the same; yet the cache key is different16:25
sxc731I'm running an early-ish Pike baseline; I gather you're running master?16:25
lbragstadit's the exact same token?16:25
lbragstadyeah - i just installed master, but the code should be the same for each with respect to the caching stuff16:26
sxc731Yep, I've done some sed'ing and JSON pretty formatting16:26
lbragstadsxc731: are you still using the benchmark script?16:27
sxc731When we last spoke, you mentioned you thought "order of 150ms was about right" according to some past (recent) benchmarks.   Yet what you've just shown me is order of 50ms and I suppose this is on a dev box as opposed to prod-class HW?16:27
lbragstadright - i'm just running a container on my laptop16:28
sxc731I'm just sending through single curl's (actually 'ab -n1' as this is what was in the bench)16:28
lbragstadwhen i had access to production-ish hardware, i was seeing about 120 ms response times with about 120k requests a minute16:28
ayounglbragstad, first 2 patches of your current service role series are +2Aed16:29
sxc731AH!  Mine is a single request on unloaded HW16:29
lbragstadayoung: thank you, sir16:29
lbragstadsxc731: can you export a token to an env variable and run this?16:30
lbragstads/token/two tokens/16:30
lbragstadcurl -X GET -H "X-Auth-Token: $TOKEN" -H "X-Subject-Token: $TOKEN2" -w "\n\n%{time_total}\n\n"16:30
lbragstadreplacing your endpoint, too16:30
lbragstadi just want to make sure we're using the same technique, i'll take another look at the benchmark script16:31
sxc731The BM script uses 'ab' (Apache Bench).  Should be functionally equivalent but I'll double-check...16:31
*** jistr has quit IRC16:37
*** sapd_ has quit IRC16:37
*** cloudnull has quit IRC16:37
*** asettle has quit IRC16:37
*** lamt has quit IRC16:37
*** charz has quit IRC16:37
*** lamt has joined #openstack-keystone16:37
*** charz has joined #openstack-keystone16:37
*** sapd_ has joined #openstack-keystone16:37
*** asettle has joined #openstack-keystone16:37
*** cloudnull has joined #openstack-keystone16:37
*** asettle is now known as Guest3153216:37
*** lamt is now known as Guest6566216:37
sxc731OK, I still get same orders of magnitude: 0.379s; let me check the cache debugging16:38
*** links has quit IRC16:38
*** hoonetorg has joined #openstack-keystone16:39
*** itlinux has joined #openstack-keystone16:39
*** jistr has joined #openstack-keystone16:40
*** links has joined #openstack-keystone16:43
lbragstadsxc731: even with verifying the same tokens are being passed to keystone?16:45
-openstackstatus- NOTICE: Zuul has been restarted and lost queue information; changes in progress will need to be rechecked.16:45
lbragstadsxc731: let me stand up a pike environment quick16:46
sxc731lbragstad: not sure what you mean? I'm using the same curl cmdline with the env vars as you suggested so it should be the same token, right?  Beside, the CACHE_SET lines explode the content of the token and they're identical.16:46
sxc731I think the cache is behaving as expected.  What seems broken is the cache key generation part. I presume this uses some sort of hashing?  What could possibly throw this off?16:48
lbragstadhere's what i've done16:49
lbragstadi dropped my database, created a new one, installed stable/pike (d0721d7cf4dc808946a7016b0ca2830c8850d5d9), re-bootstrapped the deployment16:49
lbragstadgenerated a couple new tokens16:50
lbragstadand this is my output16:50
sxc731Looks alright!16:52
sxc731My KS baseline  6a67918f9d5f39564af8eacc57b80cba98242683 # HEAD of "stable/pike" as of 28.09.2017 (which is what OSA 16.0.1 delivers).  I'm assuming you would be aware if a significant bug like this had been reoslved?16:53
sxc731I've actually been wondering why OSA uses commits as baselines rahter than tags...16:53
sxc731Guess I need to roll fwd my Keystone and retest16:54
sxc731(sorry that's OSA 16.0.2)16:55
sxc731lbragstad: Gonna be afk for 15-20 mins; will pick up then.  Thank you for all your help so far!!16:57
lbragstadsxc731: anytime, pasting my logs here in a bit16:58
*** sxc731_ has joined #openstack-keystone16:58
*** sxc731 has quit IRC17:02
cmurphyfrom ttx
*** pcaruana has quit IRC17:03
lbragstadsxc731_: here is my entire uwsgi log
lbragstadcmurphy: mmm17:04
lbragstadcmurphy: is the tools documentation referencing our documentation?17:06
cmurphylbragstad: i think they mean their customers' tools don't have documentation indicating either way17:07
lbragstadcmurphy: ok - that's what i thought, but wasn't sure17:07
*** fried_rice is now known as fried_rolls17:07
openstackgerritGage Hugo proposed openstack/keystone master: WIP - Add functional testing gate
*** tesseract has quit IRC17:12
*** AlexeyAbashkin has joined #openstack-keystone17:12
*** sxc731_ has quit IRC17:16
*** sxc731 has joined #openstack-keystone17:16
*** AlexeyAbashkin has quit IRC17:17
*** AlexeyAbashkin has joined #openstack-keystone17:21
lbragstadsxc731: actually - what happens if you modify your keystone configuration to go to only one cache?17:21
sxc731lbragstad: let me try that for fun ;-)17:21
lbragstadi wonder if keystone it querying different memcached backends and getting misses17:22
lbragstadso - technically it would be storing the same token in three different places...17:23
sxc731Down to order of 0.04s17:23
sxc731Tha'ts quite a bit nicer for sure!17:24
*** links has quit IRC17:24
sxc731Still doesn't explain why the keys would be different...17:24
sxc731Surely if different keys are generated for the same subject token (even if different cache instances) that would qualify as a bug?17:25
sxc731I can't imagine I'm the only one running with multiple cache instances though...17:25
*** AlexeyAbashkin has quit IRC17:25
lbragstadi wouldn't think you are the only one either17:26
lbragstadi'm not really familiar with how those keys are generated17:26
sxc731Can you reproduce the degenerate behaviour with multiple caches?17:26
lbragstadI would assume it is generated from data in memcache?17:26
sxc731That key hashing would have to be done client-side because that's before you've managed to retrieve the data, right?17:27
*** spzala has joined #openstack-keystone17:28
lbragstadcc kmalloc ^17:31
sxc731lbragstad: getting some higher-level (wife!) interrupts as it's already 6:30 pm here in Europe.  Looking at the KS baselines, it seems there was only one for Pike: 12.0.0, right?17:31
sxc731I mean "tag"17:32
lbragstadoh - yeah17:32
sxc731How should I proceed if I'm to raise a clean ticket on this?  Update my baseline or is what I have ok?17:34
sxc731or perhaps if you could confirm that you can reproduce the issue with multiple caches it might be useful?17:35
lbragstadwhat commit do you have installed locally?17:35
sxc7316a67918f9d5f39564af8eacc57b80cba98242683 # HEAD of "stable/pike" as of 28.09.2017 (per OSA 16.0.2)17:35
lbragstadtechnically - the worst case scenario would be you suffer a full performance hit for each cache in the deployment17:35
lbragstadyeah - you are only three commits behind what i have17:36
lbragstadand i don't think those three commits would affect this at all17:36
lbragstadand i'm almost positive this could be recreated on master17:37
lbragstadif i stand up multiple memcached instances17:37
sxc731Interesting... so maybe there's a genuine issue here?  Been wondering about the syntax of that 'backend_argument' config line as it looks a little weird...17:37
sxc731backend_argument = url:,,
sxc731But I double-checked it somewhere else and it looked similar17:38
lbragstadfrom what i can tell, that's suppose to aggregate the caches and shard data across them17:38
*** sxc731_ has joined #openstack-keystone17:38
lbragstadso if you have 3 keystone containers and 3 memcache instances, you can share data across all of them equally17:38
*** sxc731_ has quit IRC17:39
*** sxc731_ has joined #openstack-keystone17:40
lbragstadvalidating a token on keystone-1 will result in a validation on keystone-3 pulling the token from the cache instead of having to set it in the cache like it would on the first hit17:40
*** sxc731 has quit IRC17:41
*** AlexeyAbashkin has joined #openstack-keystone17:42
*** sxc731_ has quit IRC17:43
*** sxc731_ has joined #openstack-keystone17:44
*** AlexeyAbashkin has quit IRC17:44
sxc731_My debugging really suggested the issue was key generation...17:44
lbragstadright - let me try and recreate locally and track down the key generation stuff17:45
sxc731_Not sure how that’s impacted by multiple caches but certainly seems related...17:46
lbragstadyeah - me either17:46
kmalloclbragstad: that isn't how it works17:48
kmallocit relies on python-memcache code.17:48
kmallocoh wait, yes it shoud share data17:48
kmallocit doesn't cause data replication17:48
kmallocat all17:48
kmallocand python-memcache sucks.17:49
kmalloca lot of issues around how it does the scanning etc. but there are ways to optimise for cases when a memcache is down17:49
kmallocbut in short, we're still leaning on a bad and unmaintained code base17:49
sxc731_Kmalloc:sounds bad!17:50
kmallocthe key generation should be mostly consistent, but is based upon args passed to the methods17:50
kmallocso, if an arg is slightly different for some reason17:50
kmallocthe key will be different17:51
sxc731_So could the presence of multiple cache back-ends cause different keys to be generated?  That’s certainl y what this looks like...17:51
kmalloccan you verify that  the keys are actually different (Cache-key) or if python-memcache is sending the request to a differtent memcache server17:52
sxc731_Our older Kilo/Fuel instance did behave better in this respect though so it feels like a regression...17:52
kmallocmy guess is it is the latter17:52
kmallocmy guess (zero basis) is that the issue is a dict hash issue, where we pass a dict to the backend and the order of the memcache servers is not consistent17:53
kmallocand it is possibly an issue with oslo.cache's processing vs the keystone-implemented one before...17:54
sxc731_The keys are different because the Oslo.cache logging shows them to be.  Is the chosen cache instance exposed that far into app code?17:54
kmallocnah. the keys should be consistent17:54
kmallocand shouldn't be impacted by the backend unless someone set a diffewrent key-gen algo17:54
lbragstadi'm standing up an env with master and 3 memcached instances17:54
sxc731_^^ that sounds a likely reason!!  When was that switch made roughly?17:54
kmallocif the keys are really different it's not the backend code.17:55
kmallocit is either a different keygen (somehow) or somehow data is different between requests.17:55
kmallocthe keys should never be impacted in our case, we hard set to a SHA1 hash of args17:55
*** mvk has quit IRC17:55
sxc731_The app..level payload (ks token) is the same17:56
kmallocand explicitly ban kwargs to avoid difference between dict hashing17:56
kmallocif oslo.cache is showing the cache-key is different, then it's hashing the values differently and somehow breaking memoization17:56
* kmalloc tosses original theory in the round file.17:56
kmallocok. so something wierd with how we're key-hashing.17:57
kmallocthat makes me a bit more sad. it's harder to debug.17:57
kmallocare you using memcache-pool or just memcache backend?17:57
kmalloc(this is just to eliminate one other wonky-bit of code/look at the interactions)17:58
sxc731_I think it’s memcached-backend (see a little earlier; sorry I’m on an ipad in a bus :-(17:58
kmallocno worries17:59
kmallocthat is what i expected. but figured i'd ask17:59
*** markvoelker has quit IRC18:00
kmalloclbragstad: let me know what you dig up, i'm looking at the oslo.cache code, but ... hmm. weird18:03
lbragstadjust about to try and recreate18:04
sxc731_Thanks guys!  I really appreciate your help. Going to be afk for a moment but will definitely check in later!18:11
*** david-lyle has quit IRC18:12
lbragstadkmalloc: so - i have pike installed and i have two tokens exported to env vars18:13
lbragstadand doing a constant token validate call18:13
lbragstadcurl -X GET -H "X-Auth-Token: $AUTH_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" -w "\n\n%{time_total}\n\n"18:13
lbragstadbut i set CACHE_SET everywhere in the keystone.log18:13
lbragstadtechnically - there should only be two set18:13
lbragstadtwo SETs18:13
lbragstadone for the initial auth token and one for the initial subject token, right?18:14
kmallocand you have a memcache?18:14
lbragstadyeah - i have three of them18:14
lbragstadand this is my configuration in keystone18:14
sxc731_Lbragstad: all these SETs: exactly what I saw.  That’s because every GET is a miss based on the fact that the keys are always different!18:16
kmalloclbragstad: aND If you reconfig for 1 memcache, you get one set and multiple gets?18:16
lbragstadif i reconfigure with a single memcache instance (backend_argument = url: i don't see *any* CACHE_SETs and I only see CACHE_GETs18:19
lbragstadbut token validation looks right given the configuration
lbragstad0.337 for the first call 0.022 after that and so on18:20
kmallocok so somehow the multiple backends is leaking into the cache key18:20
kmallocit's an oslo.cache detail then.18:20
kmalloclooking at the region,18:20
lbragstadbut where in the world does the first key get set?!18:20
lbragstador the second...18:21
lbragstadtechnically it should be putting both in the cache since the tokens in the call are different18:21
kmallocit might be pre-seeded18:21
lbragstadlet me recheck18:22
kmallocalso... wonky18:22
* lbragstad swiftly kicks memcache and keystone18:22
kmallocyou're not getting any GETs in the multi-setup18:22
kmallocbecause you should see GET/SET/GET/SET/GET/SET18:22
*** sxc731_ has quit IRC18:23
lbragstadok - so if i kick memcache and keystone18:24
lbragstadand redo the test18:24
lbragstadi see the sets for everything18:25
lbragstad(catalog, roles, users, etc..)18:25
lbragstadand eventually the token18:25
lbragstadso it must be pre-seeded18:25
kmallocthere are some cases we did some magic for that18:26
lbragstadi know we did for authenticate18:26
kmallocalso, the SET might be hidden right after the first GET18:26
lbragstadwe put the token in the cache after creation18:26
kmallocso, if you just kick memcache over and validate you should see a set.18:26
lbragstadyeah - i do18:27
kmallocthat aside.18:27
lbragstadconfiguring multiple backends seems hosed18:27
kmalloci don't know why this would change the key.18:27
lbragstadme either18:27
lbragstadwe don't accept kwargs on the method we wrap specifically for that reason18:28
kmallocright, even though we have a kwarg mangle now.18:28
kmallocbnut that aside18:28
lbragstaddef _validate_token(self, token_id):18:28
*** sxc731 has joined #openstack-keystone18:29
lbragstadthat's ^ the method we wrap18:29
kmallocand the keys look like SHA1 hashes right?18:31
kmallocwhen you're looking at them?18:31
kmallochold on trying to setup a VM really quickly to test this.18:32
kmallocs/really quickly/looking for an image i can import/18:32
lbragstadlet me grab a snippet of the logs18:32
*** vaishali has quit IRC18:34
*** hugokuo has quit IRC18:34
*** dstanek has quit IRC18:34
*** errr has quit IRC18:34
*** d34dh0r53 has quit IRC18:34
*** sxc731 has quit IRC18:34
*** d34dh0r53 has joined #openstack-keystone18:34
*** dstanek has joined #openstack-keystone18:34
lbragstadCACHE_GET and CACHE_SET all in ^18:34
*** errr has joined #openstack-keystone18:35
*** sxc731 has joined #openstack-keystone18:35
*** hugokuo has joined #openstack-keystone18:35
lbragstadthe cache key for the one of the tokens being validated was c9019b39555c4c860ec8c4e95a605bb328c441e118:36
lbragstadit gets set once and then it gets a successful hit every time after that (so working as expected)18:36
lbragstadwith a single cache18:36
kmallocyeah that looks fine.18:36
lbragstadgrabbing some logs with multi-cache configured18:37
kmalloc~10minutes left on the image download.18:40
kmallocor... now 30m.. damn18:41
lbragstadevery single CACHE_GET results in a dogpile.cache.api.NoValue object18:41
lbragstadso it attempts to set it18:41
kmallocthat is fine. that is normal operation if the backend is cache missing18:41
kmallocNoValue is a magic "not set" value18:42
lbragstadonly up the the total number of cache instances18:42
lbragstadtechnically your worst case would be 3 cache misses18:42
lbragstadif you hit every cache with the same key before hitting one that has it18:42
kmallocthis is sounding like a bug in python-memcache...18:43
*** vaishali has joined #openstack-keystone18:43
lbragstadeither we're generating keys weird or yeah.. this is a bug in python-memcached18:43
lbragstadwhat about oslo.cache?18:43
kmallocis the key consistent on requests?18:43
*** sxc731 has quit IRC18:43
kmallocso c9xxxxxx is the same key on the GET18:44
lbragstadlet me check quick18:44
kmalloclets elminiate oslo/dogpile issues or memcache issues18:44
*** vaishali has quit IRC18:44
*** vaishali has joined #openstack-keystone18:45
lbragstaddoes paste.o.o have a limit size?18:45
*** sxc731 has joined #openstack-keystone18:45
lbragstadlatest run with multi-cache enabled
kmalloclooks like the cache key is consistent?18:49
kmallocor am i mis-reading this18:49
kmalloci wonder if this is a squashed exception somewhere18:50
*** gyee has joined #openstack-keystone18:52
lbragstadkeys are getting generated differently18:52
lbragstadi dont' think my log showed it because i maxed out the paste limit18:52
lbragstadbut if you look elsewhere in the logs18:53
lbragstadi can see the same token getting set18:53
lbragstadbecause of the audit-ids18:53
lbragstadand the audit-id chain18:53
lbragstadbut they have different cache keys18:53
kmallocwait. ...18:53
kmallocthat shouldn't be the case18:53
kmallocif you are doing a GET on a single token with a consistent token18:53
kmallocaudit ids should be 100% the same18:53
kmallocif you keep issuing a new token, yes it will have a chain id issue18:54
lbragstadright - they are18:54
lbragstadi'm validating the same token over and over18:54
lbragstadso the audit-ids are the same18:54
kmallocok, so... uh18:54
lbragstadwhich proves the cache key is different18:54
lbragstadfor the same token18:54
kmalloccan you show me?18:54
kmalloci don't see how that is possible18:54
kmallocthe audit_id and chain_id should be the same18:54
lbragstadlet me find a paste service that works for logs18:54
kmallocemail me a log?18:55
kmallocor hangout+shard screen18:55
kmallocjust looking for quick18:55
ayounglbragstad, are System scope assignments going to honor the implied roles mechanisms?19:00
*** phalmos has joined #openstack-keystone19:04
lbragstad^ if anyone else is interested in debugging cache problems19:05
*** sxc731 has quit IRC19:07
*** phalmos_ has joined #openstack-keystone19:08
*** phalmos has quit IRC19:09
*** sambetts is now known as sambetts|afk19:09
*** gyee has quit IRC19:16
*** sxc731 has joined #openstack-keystone19:20
lbragstadkmalloc: locals: {'key': 'keystone.token.provider:_validate_token|gAAAAABaWQd9Vr5NSlgloAbyGS-A7nkx660QtRKZVMblDSqUjwT0J_N4a9k-zDodsF9vvmmeSWAeyeRjVgDQjX96_Xe2ProJvyepPKsUL45rw5_i2KYNZznim2-yyw3sA0C2SVLT4_D4nKFROgR-HFtV6pbYhtbOA7BBaIdFEKK5t1Q     i5D1zHXY:@+L\xdby(.\xb4\x03\xb6'}19:25
*** sxc731 has quit IRC19:25
*** AlexeyAbashkin has joined #openstack-keystone19:25
*** abhi89 has joined #openstack-keystone19:26
*** blake has joined #openstack-keystone19:27
*** harlowja has joined #openstack-keystone19:27
*** david-lyle has joined #openstack-keystone19:33
*** jose-phi_ has quit IRC19:38
*** jose-phillips has joined #openstack-keystone19:39
*** AlexeyAbashkin has quit IRC19:42
*** sxc731 has joined #openstack-keystone19:47
*** bigdogstl has quit IRC19:54
*** blake has quit IRC19:55
*** sxc731 has quit IRC20:02
*** harlowja has quit IRC20:04
*** sxc731 has joined #openstack-keystone20:05
*** sxc731_ has joined #openstack-keystone20:05
*** sxc731_ has quit IRC20:08
openstackgerritColleen Murphy proposed openstack/keystone master: Extract expiration validation to utils
sxc731lbragstad: so you've managed to reproduce the issue and you're looking for a fix with kmalloc, right?  Would it be useful if I raised a ticket?20:10
lbragstadayoung: eventually20:10
lbragstadsxc731: well - it's strange...20:10
kmallocyeah standing up a VM now.20:10
kmallocit's wonky20:10
sxc731In my experience the absence of cache hits due to different keys seems 100% reproducible with multiple caches.  I'm just surprised nobody else noticed because what I used was the off-the-shelf OSA deployment pattern.  And the performance impact is _very_ noticeable20:11
*** bigdogstl has joined #openstack-keystone20:13
lbragstadyeah - we're trying to figure out if it is a keystone issue or not20:14
lbragstadkeystone is passing the same key, but when dogpile or oslo.cache goes to generate a cache key, it comes up different each time20:14
lbragstadwith multiple cache backends configured20:15
lbragstadwhich is the weird part20:15
sxc731kmalloc mentioned a switch from keystone-based interface to memcache to oslo.cache in the recent-ish past...20:15
kmallocyeah a while ago20:16
kmallocwe swapped to oslo.cache20:16
kmallocat some point20:16
sxc731I didn't experience20:16
lbragstadyeah - that was pre-pike20:16
lbragstadfor sure20:16
sxc731... this issue in Kilo20:16
sxc731Just speculating, not having read the code at all but I don't suppose the cache instance being has any bearing on the key generation?20:17
*** bigdogstl has quit IRC20:19
lbragstadthat's the part i'm about to start digging into20:23
*** bigdogstl has joined #openstack-keystone20:25
sxc731I'm still baffled that nobody noticed this before...20:26
lbragstadme too20:27
lbragstadsxc731: solid find, sir... solid find20:28
*** bigdogstl has quit IRC20:30
*** bigdogstl has joined #openstack-keystone20:33
lbragstadkmalloc: i can confirm that this is what is getting called20:38
kmalloci knew that much20:38
kmallocinstalling keystone now20:38
sxc731Here's the defect for committing against:
openstackLaunchpad bug 1743036 in OpenStack Identity (keystone) "Multiple memcached back-end instances breaks caching" [Undecided,New]20:39
lbragstadso - the key that is generated there doesn't have garbage at the end...20:40
lbragstadlike we were seeing in oslo.cache logs20:40
lbragstadand that is called from
lbragstadwell - specifically20:45
*** sxc731 has quit IRC20:45
*** sxc731_ has joined #openstack-keystone20:46
*** openstackstatus has quit IRC20:46
*** openstack has joined #openstack-keystone20:47
*** ChanServ sets mode: +o openstack20:47
*** openstackstatus has joined #openstack-keystone20:48
*** ChanServ sets mode: +v openstackstatus20:48
lbragstadkmalloc: the garbarge get amended here -
kmallocyeah that might actually cause issues with multiple processes.20:52
lbragstadipdb> invalidation_manager.region_id20:52
kmalloclet me check waht region_id actually is.20:53
kmallocyeah, what is THAT supposed to be?20:53
lbragstadgood question20:53
lbragstaddstanek: might know :)20:53
lbragstadwe somehow stub this in
kmallocreturn os.urandom(10)20:55
kmallocok THAT is ...20:55
kmalloci just don't get it20:56
lbragstadwe seem to like calling it20:56
lbragstadand we use it when we invalidate a region20:56
kmallocyeah that is an issue20:56
kmalloclike... THAT SHOULD JUST BE NAMESPACE changing20:56
kmallocwhat in the ... ugh20:57
kmallocthis is icky20:57
*** panbalag has joined #openstack-keystone20:57
kmallocok i think this is a keystone bug, and it's going to need some rather serious unwinding.20:58
kmallocwe're setting random crap on the cache keys20:59
kmallocand each process is going to cache it differently afaict20:59
kmallocthis is... bad news.20:59
lbragstadso - this was the patch we wrote to handle distributed cache invalidation
*** fried_rolls is now known as fried_rice21:01
kmallocthe fact we generate random junk is a serious issue21:02
*** itlinux has quit IRC21:02
kmallocbecause we hard set this to in-memory21:02
lbragstadthe weird part is that we generate it on the key... when it looks region specific21:03
kmallocyeah it's so we can force a whole region to be invalidated at once21:03
lbragstadat least that's weird to me...21:03
kmallocit was/is a hack around bugs in dogpile21:03
kmalloci think this is no longer needed with modern dogpile21:03
kmallocbut... i need to dig into to make sure we didn't just land the same broken thing to dogpile21:03
kmalloci *think*.21:03
kmallocthis is going to take a chunk of time to unwind.21:04
lbragstadok - i want to understand how this works when a single backend is there but it doesn't for multiple21:05
lbragstadwhere is that case in our keystone code?21:05
kmallocthis might actually be ok code wise.21:06
kmallocyeah ok, this is mostly ok.21:07
lbragstadthat looks suspicious21:08
* lbragstad is suspicious of everything now21:08
kmallocthats fine21:10
kmallocok this is an oke patch, but i really dislike we use a random() function instead of a real string21:12
kmalloc... we should fix that to be something like uuid.uuid4.hex21:12
kmallocand we should replace the namespace NOT tack random crap on the end.21:13
kmallocthat is a different story21:13
kmallocthis should be using the same mechanism for setting up backends as the primary config21:13
kmallocso, it should be perfectly fine.21:13
kmallocwe have to override it21:14
lbragstadyeah looks like it uses the method from dog pile21:14
lbragstadit's blowing my mind how this works for a single backend but not three21:14
lbragstadbecause all the backend stuff is being handled by dogpile as far as i can tell21:15
*** lintonv has quit IRC21:15
*** panbalag has joined #openstack-keystone21:15
kmallocunless we're getting some inconsistency in memcache backends.21:15
kmallocin how we're passing backends.21:16
lbragstadso - do we conditionally build regions based on backends?21:16
kmallocwe do the buildconf21:17
kmallocfrom the oslo.conf data21:17
kmallocwhich turns into a dict that is passed to the backend(s)21:17
*** spzala has quit IRC21:18
lbragstadyep - that makes sense21:19
lbragstadand from what i can tell, that seems sane21:19
*** spzala has joined #openstack-keystone21:19
kmallocwait let me see your config again21:20
kmallocyour config is bad21:20
kmallocthis is incorrect: backend_argument = url:,,
kmallocyou're missing ports on the first 2 elements21:21
lbragstadthat's weird21:21
lbragstadis that format documented somewhere?21:22
kmallocso, my guess is you're hitting a case where the servers are being marked down and erroring21:22
kmalloci think so*21:22
kmallocbut the format is.........21:22
lbragstadi think that was part of the OSA template21:22
kmalloc        cfg.MultiStrOpt('backend_argument', default=[], secret=True,21:23
kmalloc                        help='Arguments supplied to the backend module. '21:23
kmalloc                             'Specify this option once per argument to be '21:23
kmalloc                             'passed to the dogpile.cache backend. Example '21:23
kmalloc                             'format: "<argname>:<value>".'),21:23
kmallocso: <key>:<value>21:23
lbragstadbackend_argument = url:,,
lbragstador ..21:23
kmallocyou could use memcached_servers opt21:23
*** spzala has quit IRC21:23
kmalloci'll be this is a case where we are hitting issues with servers being marked as dead and the ring distribution is breaking in python-memcached21:24
lbragstadbackend_argument = url:
lbragstadbackend_argument = url:
lbragstadbackend_argument = url:
kmallocyou must combine all values for a single key on a single line21:24
*** spzala has joined #openstack-keystone21:24
kmallocbackend_argument may be multiple backend arguments21:25
lbragstadthat's misleading when the option is defined with a name of MultiStrOpt21:25
kmallocit is because you might have url, expiry, password21:25
kmallocall different options that need to be passed in a backend-specific way21:25
kmallocthe multistropt is about allowing that21:25
kmallocurl is just *one* of them21:26
lbragstad{"error": {"message": "An unexpected error prevented the server from fulfilling your request: Unable to parse connection string: \",,\". (Disable insecure_debug mode to suppress these details.)", "code": 500, "title": "Internal Server Error"}}21:26
*** harlowja has joined #openstack-keystone21:26
kmallocwait.. wut.21:26
kmallocuse memcache_servers21:27
kmallocit's a list opt21:27
kmallocuse that for the moment21:27
kmallocsee if it fixes the bug21:27
kmallocwe can chase the other issues down after21:27
kmallocyeah. the backend argument bit is somewhat broken somehow21:28
kmallocit's going to need to be re-worked21:28
kmallocyeah that is going to be just broken.21:29
kmallocin short, don't use that mechanism to configure for now21:29
*** spzala has quit IRC21:30
kmalloci'll bet this has been broken for quite a while.21:30
lbragstadso - memcache_servers =,,
kmalloclbragstad: yeah.21:31
kmallocactually it might be multiple lines, for the multistropt, id' need to poke at oslo.config to see what it does.21:31
kmallocyou're example with backend_argument = url:<xxxx> multiple times is correct21:31
lbragstadbut with memcache_servers?21:32
lbragstadwith memcache_servers =,, i'm still only seeing traffic to the first server21:32
*** itlinux has joined #openstack-keystone21:32
kmallocthat is going to depend on how the ring is parsed in python-memcached21:33
kmallocis it still missing on 100% of the calls?21:33
kmallocthat's all i care about to start.21:33
lbragstadcaching is working21:33
kmalloctry with multiple lines on the multistropt21:33
lbragstadoh - wait21:33
kmallocif memcache_servers is working21:34
*** spzala has joined #openstack-keystone21:34
lbragstadso - caching works with memcache_servers =,,
lbragstadbut i only see traffic to 24321:35
lbragstadtailing syslog on each memcache instance and that is the only one reporting GET and SET traffic21:36
*** edmondsw has quit IRC21:36
kmallocthat is related to how the buckets work internal to python-memcache21:37
kmalloci don't have a lot of insight21:37
kmalloctry with the multi-line url: bit21:37
kmallocfor backend-argument21:37
kmallocmy guess is it will be the same21:37
*** spzala has quit IRC21:39
lbragstadok - this is what i have21:39
kmallocyep lets try that21:40
kmallocif that works like i think it will...21:40
kmallocbut i am not 100% sure21:40
lbragstadit only sends traffic to the first one in config21:42
lbragstadconfirmed by switching them around21:42
kmalloccan you debug what the config ends up looking like21:43
lbragstadbumped up the logging21:44
kmallocis it working as expected?21:44
lbragstadwith memcache_servers =,,
lbragstadrunning a loop so i can see traffic21:44
lbragstadon each21:44
kmallocok backend-argument is broken i think21:44
kmallocthe more i look at this21:44
kmallocit... never should ahve worked21:45
kmalloctl;dr, use memcache_servers21:45
kmallocit is the only way that consistently works atm21:45
kmallocwe're going to need to totally re-work the processing for backend-arguemnt21:46
lbragstadyep - ok cool21:46
lbragstadi can confirm21:46
kmallocbut is in oslo.cache21:46
lbragstadusing memcached servers in a list works as expected21:46
lbragstadand it sends traffic to each memcache server21:46
lbragstadand it looks like they are all responding within reasonable time21:46
kmallocand not getting endless misses21:46
lbragstadcc sxc731_^21:46
kmallocok, i know what needs to happen. but very short, backend-arguemnt is totally busted.21:47
kmallocwe have to add a bunch of hint processing for the backends to have a clue what it should be21:47
lbragstadexample response times with a cluster of 3 memcache servers21:47
openstackgerritGage Hugo proposed openstack/keystone master: WIP - Add functional testing gate
kmallocso possiby something like "list:delim🔑value"21:47
kmallocor <type>|<delim>|<key>|<value>21:48
kmallocsomething so we know what the type should be + delimiter if relevant21:48
kmallocit's going to be a lot of re-working21:48
kmallocso it probably should look like backend-argument = type:list:,|url|,
kmallocand the default behavior if the there is no "type" hint, is to process as a string like it is today21:50
lbragstadwell - that was fun21:52
*** harsha has joined #openstack-keystone21:52
*** r-daneel has joined #openstack-keystone21:52
* lbragstad heads over to osa21:52
kmallocgo ahead and assign me the bug.21:52
kmallocit's going to take a bit of work to re-do this21:52
openstackgerritGage Hugo proposed openstack/keystone master: WIP - Add functional testing gate
harshaHello, can anyone let me know what's the best configuration for the number of WSGIDaemonProcess process and threads that can be configured for keystone21:53
*** spzala has joined #openstack-keystone21:56
lbragstadkmalloc: i don't have the power to do that for oslo.cache21:57
openstackLaunchpad bug 1743036 in oslo.cache "Multiple memcached back-end instances breaks caching" [Undecided,Confirmed]21:57
lbragstadbut i marked it as invalid for keystone21:57
lbragstadharsha: that's going to totally depend on the hardware you have21:57
lbragstadharsha: that openstack-ansible commuity might be able to point you in the first direction since they do some math when automating the deployment of keystone21:58
*** panbalag has quit IRC21:58
harsha@lbragstad thanks you21:59
sxc731_Lbragstadt, kmalloc: thank you for your hard work guys.  Not only have you figured out the issue but you even have a workaround. Outstanding!22:03
*** spzala has quit IRC22:04
*** spzala has joined #openstack-keystone22:05
lbragstadsxc731_: anytime - i have the workaround proposed to osa -
lbragstadthat should hopefully get you around the issue until we get to the bottom of the python-memcache issues22:06
lbragstador oslo.cache issues22:06
kmallocit's def oslo.cache issues22:06
sxc731_lbragstad: also explains why this wasn’t widely experienced before.  If you used the old-style config, you’d be fine... Only too bad for OSA deployments...22:07
*** itlinux has quit IRC22:10
*** spzala has quit IRC22:17
*** sxc731_ has quit IRC22:21
*** phalmos_ has quit IRC22:26
openstackgerritColleen Murphy proposed openstack/keystone master: Add application credentials db migration
openstackgerritColleen Murphy proposed openstack/keystone master: Add application credentials driver
openstackgerritColleen Murphy proposed openstack/keystone master: Add Application Credentials manager
openstackgerritColleen Murphy proposed openstack/keystone master: Add Application Credentials controller
openstackgerritColleen Murphy proposed openstack/keystone master: WIP Add application credential auth plugin
*** harsha has quit IRC22:27
*** dikonoor has quit IRC22:36
*** dikonoor has joined #openstack-keystone22:36
*** spzala has joined #openstack-keystone22:37
*** dave-mccowan has quit IRC22:39
*** spzala has quit IRC22:42
*** AlexeyAbashkin has joined #openstack-keystone22:54
*** AlexeyAbashkin has quit IRC22:59
*** panbalag has joined #openstack-keystone23:00
*** harlowja has quit IRC23:01
*** spzala has joined #openstack-keystone23:01
*** raildo has quit IRC23:04
*** panbalag has quit IRC23:09
*** spzala has quit IRC23:10
*** spzala has joined #openstack-keystone23:13
*** spzala has quit IRC23:20
*** itlinux has joined #openstack-keystone23:22
*** spzala has joined #openstack-keystone23:25
*** spzala has quit IRC23:30
*** fried_rice is now known as efried23:35
*** r-daneel has quit IRC23:35
*** spzala has joined #openstack-keystone23:37
*** itlinux has quit IRC23:39
*** spzala has quit IRC23:46
*** itlinux has joined #openstack-keystone23:57
*** spzala has joined #openstack-keystone23:57

Generated by 2.15.3 by Marius Gedminas - find it at!