Friday, 2017-08-11

samueldmqlbragstad: approved https://review.openstack.org/#/c/49191600:03
samueldmqlbragstad: anything we need to land today?00:03
samueldmqanything else*00:03
*** itlinux has quit IRC00:06
*** dstepanenko has joined #openstack-keystone00:06
*** _apple_tree has quit IRC00:08
*** thorst has quit IRC00:09
*** thorst has joined #openstack-keystone00:10
*** ducttape_ has quit IRC00:10
*** thorst has quit IRC00:11
*** dstepanenko has quit IRC00:11
*** edmondsw has joined #openstack-keystone00:17
*** sbezverk has quit IRC00:21
*** edmondsw has quit IRC00:22
*** ducttape_ has joined #openstack-keystone00:25
*** aselius has quit IRC00:32
*** ducttape_ has quit IRC00:36
*** ducttape_ has joined #openstack-keystone00:37
lbragstadsamueldmq: no that should be good00:41
lbragstadsamueldmq: i should be able to cut rc1 tomorrow00:41
samueldmqlbragstad: sweet00:41
*** ducttape_ has quit IRC00:42
samueldmqlbragstad: I had to go afk and was unable to proceed with that password update debug00:42
samueldmqthe current plans are to make a bunch of logs around places that return 401 on token issue00:42
samueldmqand try to reproduce it again, and analyze the logs00:42
otleimatlbragstad: I ran pep8 on my machine locally and it passed but failed when I checked it in to Gerrit. I'll make sure to inspect/fix that tomorrow morning, sorry about that :/00:48
*** zhurong has joined #openstack-keystone00:51
*** markvoelker has joined #openstack-keystone01:03
*** itlinux has joined #openstack-keystone01:04
*** Shunli has joined #openstack-keystone01:04
*** thorst has joined #openstack-keystone01:11
*** gyee has quit IRC01:14
*** gyee has joined #openstack-keystone01:14
*** thorst has quit IRC01:17
*** mjax has joined #openstack-keystone01:25
*** mjax has quit IRC01:26
*** mjax has joined #openstack-keystone01:29
*** mjax has quit IRC01:31
*** PsionTheory has quit IRC01:35
*** mjax has joined #openstack-keystone01:48
*** mjax has quit IRC01:49
openstackgerritMerged openstack/keystone master: Unset project ids for all identity backends  https://review.openstack.org/49191601:52
*** dstepanenko has joined #openstack-keystone01:54
*** dstepanenko has quit IRC01:59
*** edmondsw has joined #openstack-keystone02:06
*** edmondsw has quit IRC02:10
*** ppiela_ has joined #openstack-keystone02:10
*** zsli_ has joined #openstack-keystone02:11
*** thorst has joined #openstack-keystone02:12
*** gongysh has joined #openstack-keystone02:17
*** thorst has quit IRC02:17
*** Shunli has quit IRC02:18
*** ppiela has quit IRC02:18
*** jistr has quit IRC02:18
*** clayton has quit IRC02:18
*** aloga has quit IRC02:18
*** clayton has joined #openstack-keystone02:19
*** openstackgerrit has quit IRC02:21
*** lifeless_ has joined #openstack-keystone02:21
*** jistr has joined #openstack-keystone02:23
*** lifeless has quit IRC02:28
*** ducttape_ has joined #openstack-keystone02:38
*** ducttape_ has quit IRC02:43
*** openstackgerrit has joined #openstack-keystone02:50
openstackgerritOpenStack Release Bot proposed openstack/keystone master: Update reno for stable/pike  https://review.openstack.org/49277402:50
openstackgerritKristi Nikolla proposed openstack/keystone master: Remove deprecation of domain_config_upload  https://review.openstack.org/49269402:58
*** spzala has joined #openstack-keystone02:59
*** dave-mccowan has quit IRC03:06
*** thorst has joined #openstack-keystone03:13
*** mjax has joined #openstack-keystone03:17
*** mjax has quit IRC03:18
*** nicolasbock has joined #openstack-keystone03:21
*** thorst has quit IRC03:21
*** sbezverk has joined #openstack-keystone03:24
*** namnh has joined #openstack-keystone03:33
*** gongysh has quit IRC03:34
itlinuxhello keystone guys..03:40
*** aojea has joined #openstack-keystone03:41
itlinuxI have a question when I do openstack user list --domain mydomain I can see the users.. but I get this when I try to login You are not authorized for any projects or domains.03:41
itlinuxany tips! TY03:41
*** dstepanenko has joined #openstack-keystone03:42
*** aojea has quit IRC03:46
*** dstepanenko has quit IRC03:47
*** edmondsw has joined #openstack-keystone03:54
*** mvk has joined #openstack-keystone03:57
*** SamYaple has quit IRC03:57
*** edmondsw has quit IRC03:58
*** prashkre_ has joined #openstack-keystone04:00
*** sbezverk has quit IRC04:07
*** SamYaple has joined #openstack-keystone04:11
*** thorst has joined #openstack-keystone04:17
*** thorst has quit IRC04:22
openstackgerritArundhati Surpur proposed openstack/keystone-tempest-plugin master: Removed the older version of python and added 3.5  https://review.openstack.org/48519104:32
*** dstepanenko has joined #openstack-keystone04:37
*** dstepanenko has quit IRC04:41
*** prashkre_ has quit IRC04:46
*** gyee has quit IRC04:50
*** thorst has joined #openstack-keystone05:17
*** thorst has quit IRC05:22
*** ducttape_ has joined #openstack-keystone05:30
*** ducttape_ has quit IRC05:34
*** tobberydberg has joined #openstack-keystone05:38
*** edmondsw has joined #openstack-keystone05:42
*** gongysh has joined #openstack-keystone05:43
*** prashkre_ has joined #openstack-keystone05:44
*** edmondsw has quit IRC05:46
*** otleimat has quit IRC05:48
*** itlinux has quit IRC05:55
*** itlinux has joined #openstack-keystone05:57
*** tesseract has joined #openstack-keystone06:16
*** thorst has joined #openstack-keystone06:18
*** rcernin has joined #openstack-keystone06:22
*** thorst has quit IRC06:23
*** dstepanenko has joined #openstack-keystone06:25
*** itlinux has quit IRC06:28
*** dstepanenko has quit IRC06:30
*** ducttape_ has joined #openstack-keystone06:31
*** ducttape_ has quit IRC06:35
*** jaosorior has quit IRC07:03
*** jaosorior has joined #openstack-keystone07:15
*** ioggstream has joined #openstack-keystone07:17
*** aloga has joined #openstack-keystone07:19
*** thorst has joined #openstack-keystone07:19
*** thorst has quit IRC07:23
*** clenimar has joined #openstack-keystone07:35
*** aloga has quit IRC07:43
*** aloga has joined #openstack-keystone07:43
*** openstackgerrit has quit IRC08:02
*** openstackgerrit has joined #openstack-keystone08:03
openstackgerritOpenStack Proposal Bot proposed openstack/keystone master: Imported Translations from Zanata  https://review.openstack.org/49284408:03
*** dstepanenko has joined #openstack-keystone08:13
*** mvk has quit IRC08:14
*** dstepanenko has quit IRC08:18
*** mvk has joined #openstack-keystone08:21
*** mvpnitesh has joined #openstack-keystone09:04
*** tobberyd_ has joined #openstack-keystone09:09
*** tobberydberg has quit IRC09:12
*** tobberyd_ has quit IRC09:15
*** markvoelker has quit IRC09:17
*** thorst has joined #openstack-keystone09:21
*** thorst has quit IRC09:25
*** zsli_ has quit IRC09:37
*** mvpnitesh has quit IRC09:47
*** dstepanenko has joined #openstack-keystone10:01
*** sapd has joined #openstack-keystone10:07
*** dstepanenko has quit IRC10:07
*** Drankis has joined #openstack-keystone10:16
*** Drankis has quit IRC10:16
*** Drankis has joined #openstack-keystone10:16
*** markvoelker has joined #openstack-keystone10:18
*** thorst has joined #openstack-keystone10:22
*** markvoelker has quit IRC10:23
*** thorst has quit IRC10:26
openstackgerritMerged openstack/keystoneauth master: Enable some off-by-default checks  https://review.openstack.org/47777510:26
*** prashkre_ has quit IRC10:29
*** namnh has quit IRC10:36
*** ducttape_ has joined #openstack-keystone10:36
*** markvoelker has joined #openstack-keystone10:39
*** ducttape_ has quit IRC10:41
*** prashkre_ has joined #openstack-keystone10:42
*** markvoelker_ has joined #openstack-keystone10:44
*** markvoelker has quit IRC10:44
*** markvoelker_ has quit IRC10:44
*** markvoelker has joined #openstack-keystone10:45
*** mvpnitesh has joined #openstack-keystone10:52
*** zhurong has quit IRC11:00
*** edmondsw has joined #openstack-keystone11:10
*** sapd has quit IRC11:14
*** gongysh has quit IRC11:17
*** thorst has joined #openstack-keystone11:22
*** aloga has quit IRC11:24
*** thorst has quit IRC11:27
*** sapd has joined #openstack-keystone11:28
openstackgerritAleksey Nakoryakov proposed openstack/python-keystoneclient master: Raise ClientError if url parameter is None. Added test for this. Closes-Bug: 1498693  https://review.openstack.org/49296011:31
openstackbug 1498693 in python-keystoneclient "unfriendly error when keystone tries to parse a URL" [Medium,Triaged] https://launchpad.net/bugs/1498693 - Assigned to Aleksey Nakoryakov (alfnak)11:31
*** aloga has joined #openstack-keystone11:34
*** ducttape_ has joined #openstack-keystone11:37
*** ducttape_ has quit IRC11:40
*** ducttape_ has joined #openstack-keystone11:40
*** ducttape_ has quit IRC11:42
*** sbezverk has joined #openstack-keystone11:42
*** ducttape_ has joined #openstack-keystone11:42
*** ducttape_ has quit IRC11:46
*** sapd has quit IRC11:48
*** dstepanenko has joined #openstack-keystone11:49
*** dikonoor has joined #openstack-keystone11:49
*** sbezverk has quit IRC11:50
*** dstepanenko has quit IRC11:54
*** raildo has joined #openstack-keystone11:59
*** sapd has joined #openstack-keystone12:01
openstackgerritAleksey Nakoryakov proposed openstack/python-keystoneclient master: Raise ClientError if url parameter is None.  https://review.openstack.org/49296012:03
*** mvpnitesh has quit IRC12:05
*** ioggstream has quit IRC12:07
*** thorst has joined #openstack-keystone12:08
*** markvoelker_ has joined #openstack-keystone12:10
*** markvoelker has quit IRC12:13
*** dikonoor has quit IRC12:15
*** dikonoor has joined #openstack-keystone12:16
*** clayton has quit IRC12:21
*** efried is now known as fried_rice12:23
*** clayton has joined #openstack-keystone12:24
*** dave-mccowan has joined #openstack-keystone12:24
knikollao/12:27
*** catintheroof has joined #openstack-keystone12:30
*** ducttape_ has joined #openstack-keystone12:43
*** dstepanenko has joined #openstack-keystone12:44
*** ayoung has quit IRC12:46
*** ducttape_ has quit IRC12:47
*** ioggstream has joined #openstack-keystone12:47
*** dstepanenko has quit IRC12:48
*** ayoung has joined #openstack-keystone12:50
*** ducttape_ has joined #openstack-keystone12:50
*** ioggstream has quit IRC12:52
*** ducttape_ has quit IRC12:54
*** prashkre_ has quit IRC12:55
*** ioggstream has joined #openstack-keystone12:57
*** dstepanenko has joined #openstack-keystone13:07
*** iogg has joined #openstack-keystone13:08
*** ioggstream has quit IRC13:08
*** sbezverk has joined #openstack-keystone13:09
*** dikonoo has joined #openstack-keystone13:11
*** josecastroleon has joined #openstack-keystone13:15
*** dikonoor has quit IRC13:15
lbragstado/13:31
openstackgerritLance Bragstad proposed openstack/keystone master: Update release notes for pike  https://review.openstack.org/49301813:35
lbragstadi didn't think about this yesterday, but ^ should have gone in before we cut RC113:36
lbragstadwhen that merges we'll do an rc213:36
*** otleimat has joined #openstack-keystone13:46
*** jmlowe_ has joined #openstack-keystone13:47
samueldmqlbragstad: approved that, no need to hold it then13:48
*** jmlowe has quit IRC13:49
lbragstadsamueldmq: there is a list of doc patches we need to land, too13:51
lbragstadknikolla: is https://review.openstack.org/#/c/448773/ a patch we want to land for pike/13:52
*** markvoelker_ has quit IRC13:52
samueldmqlbragstad: cool, is that list up somewhere? same topic?13:52
lbragstadsamueldmq: http://logs.openstack.org/67/492767/2/check/gate-releases-tox-list-changes-ubuntu-xenial/0d2ac66/console.html#_2017-08-11_02_13_59_84680613:53
lbragstaddims:  pointed me to that just a bit ago13:53
lbragstad^13:53
lbragstadit compiles a list of all open patches that have changes to docs or release notes13:53
samueldmqoO13:54
lbragstadwhich is handy, i didn't know about that13:54
samueldmqyes it really is13:54
*** lucasxu has joined #openstack-keystone13:54
samueldmqthanks dims13:54
lbragstadso - if there is anything in that list that needs to be in pike documentation, then we need to merge it soon13:54
lbragstadsome aren't relevant, but a couple are13:55
lbragstadhttps://review.openstack.org/#/c/479998/13:55
lbragstadhttps://review.openstack.org/#/c/490649/13:55
lbragstadhttps://review.openstack.org/#/c/486608/13:55
samueldmqlbragstad: nice, will look13:56
samueldmqbtw for that user update error I have a new approach, I am adding logs to here https://github.com/openstack/keystone/blob/master/keystone/common/wsgi.py#L229-L23613:57
lbragstadsamueldmq: https://etherpad.openstack.org/p/keystone-pike-rc2-patches13:57
samueldmqI will log the stacktrace and find the root of the unauthorized13:57
samueldmqlbragstad: cool, rc2 to be out today as well, correct?13:57
lbragstadsamueldmq: yes - we can cut rc2 whenever14:03
lbragstadsamueldmq: but we have to release another candidate by th 25th14:03
lbragstadthe 25th*14:03
lbragstadhttps://releases.openstack.org/pike/schedule.html14:03
lbragstadideally - the sooner we get an rc2 out, the better14:03
samueldmqlbragstad: nice. we can get that update passwd issue fixed by then14:04
lbragstadand that should give us more time in the event a reason for rc3 pops up14:04
*** mvk_ has joined #openstack-keystone14:06
dimssamueldmq : lbragstad : the kudos and foresight go to dhellmann :)14:06
*** mvk has quit IRC14:08
*** prashkre_ has joined #openstack-keystone14:10
lbragstadsamueldmq: looks like we're only waiting on 3 patches then14:12
lbragstadhttps://review.openstack.org/48660814:12
lbragstadhttps://review.openstack.org/49064914:12
lbragstadhttps://review.openstack.org/47999814:13
samueldmqlbragstad: yes, they're all gating right now14:13
cmurphy\o/14:13
lbragstadsweet14:13
lbragstadi'll babysit those and propose rc2 as soon as they merge14:14
samueldmqlbragstad: nice. I looked at each of the patches in https://etherpad.openstack.org/p/keystone-pike-rc2-patches14:16
samueldmqand I agree with your comments14:16
samueldmqcmurphy: o/14:17
lbragstadsamueldmq: good deal - thanks for double checking14:17
cmurphysamueldmq: \o14:17
lbragstadcmurphy: its about quitin' time where you're at14:17
cmurphylbragstad: nah most of my team works on the west coast now so i have to stick around a few more hours14:18
lbragstadaha14:19
*** mdavidson has quit IRC14:26
*** Drankis has quit IRC14:30
lbragstadsamueldmq: when you recreated that issue with devstack - were you just running the identity tests?14:30
*** itlinux has joined #openstack-keystone14:34
lbragstadwhew - running devstack locally is making my laptop scream14:35
*** mdavidson has joined #openstack-keystone14:35
*** SamYaple has quit IRC14:35
*** SamYaple has joined #openstack-keystone14:35
cmurphythe x1 loves devstack :)14:36
lbragstadnot bad... http://paste.openstack.org/show/618191/14:41
lbragstadi'll take it14:41
samueldmqlbragstad: I am running just that very specific test14:43
samueldmqI can share the command in a bit (rebuilding devstack right now)14:44
samueldmqlbragstad: tempest run --regex tempest.api.identity.admin.v3.test_users.UsersV3TestJSON.test_password_history_not_enforced_in_admin_reset14:53
lbragstadsamueldmq: awesome - let me see if i can recreate14:53
*** sbezverk has quit IRC14:54
samueldmqlbragstad: ++14:55
samueldmqI am doing "for i in {1..100}; do tempest.... ; done"14:55
*** sbezverk has joined #openstack-keystone14:55
samueldmqa go grab some coffee14:56
samueldmq:)14:56
*** rcernin has quit IRC14:57
lbragstadsamueldmq: http://paste.openstack.org/show/618192/14:58
lbragstad`fail tempest run --regex tempest.api.identity.admin.v3.test_users.UsersV3TestJSON.test_password_history_not_enforced_in_admin_reset`14:58
cmurphylbragstad: nice15:00
lbragstad72 attempts so far without a failure15:01
*** dikonoo has quit IRC15:02
lbragstadwoo - attempt 84 failed!15:02
kmallocfun =/15:05
gagehugo"fun"15:05
*** dstepanenko has quit IRC15:07
*** dstepanenko has joined #openstack-keystone15:07
*** dstepanenko has quit IRC15:07
kmalloclbragstad: paste the exception/error for me?15:10
lbragstadkmalloc: ack15:13
lbragstadhttp://paste.openstack.org/show/618198/15:13
lbragstadthis is the failing test - https://github.com/openstack/tempest/blob/master/tempest/api/identity/admin/v3/test_users.py#L144-L16615:14
lbragstadspecifically this assertion - https://github.com/openstack/tempest/blob/master/tempest/api/identity/admin/v3/test_users.py#L16615:14
*** Dinesh_Bhor has quit IRC15:15
*** dklyle has joined #openstack-keystone15:15
*** david-lyle has quit IRC15:15
*** ducttape_ has joined #openstack-keystone15:17
kmalloccache race15:17
gagehugothat time.sleep seems sketchy15:18
kmallocis it possible something else is touching that user at the same time?15:18
kmallocbecause...15:18
kmalloc        self.get_user.invalidate(self, old_user_ref['id'])15:18
kmalloc        self.get_user_by_name.invalidate(self, old_user_ref['name'],15:18
kmalloc                                         old_user_ref['domain_id'])15:18
kmalloc        ref = driver.update_user(entity_id, user)15:18
kmallocwe invalidate then update15:19
kmalloc... we must update before we invalidate the cache15:19
kmallocalways15:19
knikollakmalloc: sounds very plausible.15:19
kmalloci don't know if i reviewd that cache patch, but caching and invalidation is hard(tm)15:20
kmallochttps://review.openstack.org/#/c/110575/15:20
knikollakmalloc: that's a pretty old patch. why would it start causing issues now?15:21
kmallocit's *very* narrow and racy15:21
kmallocand/or something is touching the user (doing a get) at the same time now15:22
kmallocso, lbragstad: swap the update until after the update15:22
gagehugoI think that issue started up around the end of June if I remember correctly15:22
kmalloclbragstad: run again?15:22
kmallocthere are a ton of things that change across the board15:23
kmalloci don't know all of them15:23
kmallocfor all i know, tempest changed some detail of how it does an update15:23
kmallocto the user15:23
lbragstadi can try and turn off identity caching specifically15:24
*** mvk has joined #openstack-keystone15:24
kmallocthat would be worth it too15:24
kmallocbut i would invert that first15:24
kmallocand do a test15:24
*** clenimar has quit IRC15:24
openstackgerritMerged openstack/keystone-tempest-plugin master: Removed the older version of python and added 3.5  https://review.openstack.org/48519115:24
knikolla++15:24
knikollagiven the time it takes to get it to fail15:24
*** mvk_ has quit IRC15:25
*** jaosorior has quit IRC15:27
knikollakmalloc: so, let me see if i got this right. cache is invalidated first. something causes the user to get cached before password is updated, so user is cached with old password. user is updated, cached version has old password. user tries to auth, cached version is picked up and new password fails.15:27
kmallocthat is at a glance a real possibility15:29
*** prashkre_ has quit IRC15:29
*** prashkre__ has joined #openstack-keystone15:29
kmallocespecially with multiple keystone processes running15:29
*** dikonoo has joined #openstack-keystone15:29
*** mvk has quit IRC15:29
*** mvk has joined #openstack-keystone15:29
knikollaand would fit the 1/100 chance.15:30
*** gyee has joined #openstack-keystone15:30
lbragstadhmm15:31
lbragstadi wonder if the switch to uwsgi did anything with the keystone logs15:31
cmurphyit might have http://lists.openstack.org/pipermail/openstack-dev/2017-July/119877.html15:33
*** dikonoo has quit IRC15:33
lbragstadcmurphy: oh - sure15:34
*** sjain has joined #openstack-keystone15:34
lbragstadhmm15:34
lbragstadknikolla: kmalloc would that case still be possible if devstack creates a new user and project for each test15:37
lbragstad?15:37
knikollalbragstad: it makes all calls during that test as that user. so plausible15:38
lbragstadand it would be handled by separate keystone processes15:39
knikollayes.15:39
*** dstepanenko has joined #openstack-keystone15:40
openstackgerritMerged openstack/keystone master: Update docs: fernet is the default provider  https://review.openstack.org/48660815:41
*** dikonoo has joined #openstack-keystone15:42
lbragstadknikolla: cmurphy samueldmq kmalloc well - disabling caching for identity seems to get us farther15:43
lbragstadup to 110 consecutive runs without a failure15:43
knikollalbragstad: and still going?15:44
cmurphylbragstad: meaning you saw a failure at 110 or it's still going without a failure?15:44
lbragstadno failure yet - up to 123 attempts15:44
*** dstepanenko has quit IRC15:45
lbragstadit appears caching certainly has something to do with it15:45
cmurphyi'll believe it around 20015:45
knikollai think we just closed 2 bugs.15:45
lbragstaddamn15:45
lbragstadattempt 128 failed15:45
cmurphyha15:46
gagehugo:(15:46
knikollaspoke too soon.15:46
lbragstadi just *had* to say something15:46
*** itlinux has quit IRC15:46
openstackgerritMerged openstack/keystone master: Updated URLs in docs  https://review.openstack.org/49064915:49
*** aselius has joined #openstack-keystone15:51
openstackgerritLance Bragstad proposed openstack/keystone master: Update release notes for pike  https://review.openstack.org/49301815:52
lbragstadsamueldmq: ^15:52
lbragstadsamueldmq: fixed the failure15:52
kmallocit could have happened.15:53
kmallocbut without cache...15:54
kmallocthen the answer might be more subtle, let me look at the pass... oh sec15:54
kmalloci might know15:54
kmallocyep15:54
kmalloclbragstad: i think i see the bg15:55
kmallocbug15:55
* lbragstad waits anxiously15:55
kmalloc                                 order_by='Password.created_at')15:57
kmallocmysql loses resolution below 1 second15:57
kmallocin many versions15:57
kmallocso.15:57
kmallocwe are changing the password twice within a second15:58
kmallocand the old password sorts above the new one15:58
kmallocwe have had the same issue wtih the revocation table(s) for a looong time15:58
lbragstadyeah15:58
kmallocbasically... we need something more reliable than a datetime field15:58
lbragstadwe need an int with microsecond precison15:59
kmallocwe shouldn't store datetime in the db we should convert to int... yes15:59
lbragstadprecision*15:59
kmallocnow...15:59
kmallocwith multiple keystones, you *may* have the same issue15:59
lbragstadbut on a much smaller scale15:59
kmallocdepends on clock skew15:59
*** dikonoo has quit IRC15:59
kmallocthat will never happen in tempest16:01
lbragstadhttps://review.openstack.org/#/c/243742/916:02
kmallocyep16:02
kmallocsame exact thing16:02
kmallocwe should stop trying to store datetime fields in the db if we are keying off them for anything besides audit16:02
lbragstadwe've hit enough use cases to justify doing that work i think16:02
kmallocthis should be keystone wide.16:03
kmallocftr16:03
lbragstadyeah16:03
lbragstadcompletely agree16:03
lbragstadsome things will still require rounding to the second - but we can handle that in keystone16:03
lbragstadwhich is a better approach anyway because you're controlling it in the application16:03
lbragstadinstead of leaving it up to the database16:03
kmallocwe should create a custom SQL-Column-Type that does the int->datetime conversion for us16:04
lbragstadwhich can change approaches to rounding across versions16:04
kmallocand we just do the second-trim-rounding with the normal datetime object16:04
kmallocso we store datetime and it writes int, and vice versa for reading16:04
lbragstadit pulls out an integer from the backend and automatically converts it to a datetime object16:05
kmallocyep16:05
*** sbezverk has quit IRC16:05
kmallocpretty easy to do16:05
kmallocalso... that cache race is real16:05
kmallocit just wasn't the culprit here16:05
lbragstadand you should have the same conversion at the front of the application16:05
lbragstadand all business logic should center around a single datetime format16:06
kmallocso, i would (initially) write everything to just convert on the backend16:06
kmallocinternally we can look at lifting that upwards once we fix the actual bug16:07
lbragstadwhich would fix the bug at hand16:07
kmallocexactly16:07
kmallocdo you need me to spin up the change?16:07
*** lucasxu has quit IRC16:07
lbragstadbut it would all be moving towards a better story for datetime consistency16:07
lbragstadkmalloc: go for it16:07
kmallocit's going to take a pretty significant chunk of time.16:07
lbragstadkmalloc: does this need to be in pike16:07
lbragstad?16:07
kmallocso, if we want this RC2, expect that we wont RC until tomorrow/next day16:07
lbragstadit will require a db migration16:07
kmallocideally, it should be if it's critical enough to cause problems16:08
kmallocif we rate this as high/crit bug16:08
kmallocyes it goes in pike16:08
kmallocif we live with it (meh)16:08
kmallocit goes in Queens16:08
kmalloc(medium/low/wish)16:08
lbragstadkmalloc: ack - i'll update the bug report with the findings16:09
kmallocnot only does it require a migration, it requires writing to multiple columns since old keystones have to still get "datetime"16:09
kmallocfor rolling-upgrades16:09
lbragstadi assume https://bugs.launchpad.net/keystone/+bug/1703917 is caused by the same bug16:10
openstackLaunchpad bug 1703917 in OpenStack Identity (keystone) "Sometimes test_update_user_password fails with Unauthorized" [Medium,Triaged]16:10
kmallocand this is not something easily done in a trigger16:10
kmallocyeah i'm sure thats the same bug16:10
*** dstepanenko has joined #openstack-keystone16:10
kmallocthis is the same kind of problem we had with password hashing, we'll need to write to both locations for pike and in queens we can write the contract16:10
lbragstadi'm going to mark https://bugs.launchpad.net/keystone/+bug/1703917 as a dupe16:10
kmallocunless we use something like https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html#function_from-unixtime but i worry when trying to be multi-db aware16:11
* kmalloc is inclined to just make it like password hashing.16:11
lbragstadkmalloc: yeah - that seems risky depending on the db16:11
lbragstadi don't expect all dbs to support that16:12
*** dstepanenko has quit IRC16:12
kmallocbut we wont need a special option to enable the migrations we'll just write both for pike16:12
kmallocand in queens contract + drop the second write16:12
kmallocok so, fix revocation events and password16:12
lbragstadyeah - and we'll have to do the same approach for other db fields i think16:12
kmallocanything else that should be converted?16:12
kmallocshould i aim to hit everything for pike?16:12
kmallocit'd be a big big RC change16:13
kmallocbut i could.16:13
lbragstadkmalloc: i would be good with that so long as it isn't a huge change16:13
kmallocit will be a lot of change (lots of migration code)16:13
lbragstadif it's huge - then we can rescope to only fix what is needed for the bug16:13
kmallocwell, the big change will be a bit of code to any/all models16:13
lbragstadthen make the rest of the change in queens and remove old stuff in rocky16:13
kmalloci'd prefer to land it all in pike if possible16:13
kmallocit'll be a few hundred lines of code16:14
kmallocand a significant migration16:14
lbragstadkmalloc: if you're confident in it - i trust your judgement16:14
kmalloci'll aim for passwords first16:14
kmallocand expand/replicate from there16:14
lbragstadworks for me16:14
kmallocgoing to call the new fields "<Xxxxx>-int"16:14
kmallocor?16:14
lbragstadkmalloc: yeah16:15
kmallocok16:15
kmallocand i'll write a data migration to move all the data as well to the int-field16:15
lbragstadi'd perfer to keep it as explicit as possible16:15
kmalloc(i need to do the same thing for password-hashes anyway in Queens)16:15
openstackgerritOmar Tleimat proposed openstack/keystone master: Fix mapping_purge failure  https://review.openstack.org/40830416:15
* kmalloc sighs.16:15
kmallocthis has been a long time coming16:15
lbragstadyeah...16:15
kmallocalso... in rocky, you get to delete all persistent token stuff ;)16:16
kmallocbtw16:16
kmalloccollapse everything down to fernet16:16
kmallocsince uuid is deprecated and slated for removal (yay)16:16
lbragstadthat will be nice16:16
kmalloci would actually drop the token driver being configurable at the same time16:16
lbragstadso we don't want to keep uuid around in case of a security issue with fernet?16:16
kmallocnah, fix fernet16:17
*** dikonoo has joined #openstack-keystone16:17
kmallocand merge most of that code down into sane modules that build the token. the only thing the fernet driver should do is the encrypt/hash16:17
lbragstadright16:18
kmallocand unpack16:18
kmalloceverything else should be baseline code for tokens. not optional/configurable/driverable16:18
lbragstadthat means we can pull the v3 token data stuff up to the controller where it should be16:18
kmallocexactly16:18
lbragstadand use token models consistently16:18
kmalloc*yep*16:18
kmallocanyway.16:18
kmallocoff to write a ton of icky code for datetime->int16:18
lbragstadsweet - updating the bug16:19
kmallocthere is a smaller fix, now that i think of it16:19
kmallocwe could just mark the old password expired when a new one is issued...16:19
kmallocand do: AND not expired16:19
kmallocbut... that makes the update multiple updates and transactions16:19
kmallocpotentially scary16:20
lbragstadless atomic i assume16:20
kmallocit can be atomic16:20
kmallocbut more risky16:20
kmallocthe real fix is stop relying on datetime data type in mysql16:20
kmallocin either case16:20
*** aselius has quit IRC16:21
*** aselius has joined #openstack-keystone16:21
*** markvoelker has joined #openstack-keystone16:25
kmalloclbragstad: /me just told people to stop doing endpoint filtering.16:27
kmallocagain16:27
kmallocbut this time on the ML16:27
lbragstadkmalloc: updated https://bugs.launchpad.net/keystone/+bug/170221116:30
openstackLaunchpad bug 1702211 in OpenStack Identity (keystone) "test_password_history_not_enforced_in_admin_reset failed in tempest test" [Medium,Confirmed]16:30
kmallocayoung: i'm a bit sad. but this fix is a loooong time coming.16:30
kmalloclbragstad: ok give me a sec, need to check on the bird. he's making odd sounds.16:30
kmallocthen i'm on to the code.16:30
*** itlinux has joined #openstack-keystone16:34
*** jmlowe_ has quit IRC16:34
*** iogg has quit IRC16:38
*** tesseract has quit IRC16:41
*** dklyle has quit IRC16:44
*** dklyle has joined #openstack-keystone16:45
*** prashkre__ has quit IRC16:47
*** jmlowe has joined #openstack-keystone16:56
*** markvoelker has quit IRC16:59
*** mjax has joined #openstack-keystone17:03
*** dstepanenko has joined #openstack-keystone17:04
knikollalbragstad: let's sync up on monday about global roles. have some work to do today.17:06
*** dstepanenko has quit IRC17:09
lbragstadknikolla: sounds good - i was just about to ping you about that17:14
lbragstadknikolla: what time on monday?17:15
knikollalbragstad: morning would be good.17:15
lbragstad10:00 AM your time work?17:16
knikollalbragstad: sounds good.17:16
*** mvk has quit IRC17:17
lbragstadknikolla: sent17:18
*** dikonoor has joined #openstack-keystone17:21
lbragstadkmalloc: i'll watch for a patch when you have one ready - or ping me, i have the environment setup locally so i should be able to test with devstack17:21
kmallocOkie17:22
knikollalbragstad: any homework reading before that?17:23
lbragstadknikolla: https://review.openstack.org/#/c/462733/ ?17:23
lbragstadknikolla: https://review.openstack.org/#/c/460344/17:23
lbragstadhttps://review.openstack.org/#/c/464763/17:24
knikollalbragstad: ack17:24
lbragstadi think those are the big ones17:24
lbragstadotherwise i think we'll just be digging into the implementation17:24
*** dikonoo has quit IRC17:24
*** prashkre__ has joined #openstack-keystone17:27
*** simondodsley has joined #openstack-keystone17:33
*** ducttape_ has quit IRC17:39
*** ducttape_ has joined #openstack-keystone17:42
samueldmqlbragstad: kmalloc did you find anything on the passwd update bug?17:42
samueldmqnvm saw the update in the bug. nice find!17:43
*** ducttape_ has quit IRC17:46
*** ducttape_ has joined #openstack-keystone17:50
*** markvoelker has joined #openstack-keystone17:55
*** sjain has quit IRC18:11
*** markvoelker has quit IRC18:29
openstackgerritMerged openstack/keystone master: Update reno for stable/pike  https://review.openstack.org/49277418:29
*** tobberydberg has joined #openstack-keystone18:31
*** tobberydberg has quit IRC18:36
*** nicolasbock has quit IRC18:43
*** tobberydberg has joined #openstack-keystone18:58
*** tobberydberg has quit IRC19:03
*** swain has joined #openstack-keystone19:03
*** ducttape_ has quit IRC19:03
*** ducttape_ has joined #openstack-keystone19:09
lbragstadlooks like keystone-manage bootstrap is currently incompatible with devstack and ldap configuration19:10
*** ducttap__ has joined #openstack-keystone19:15
*** ducttape_ has quit IRC19:16
*** prashkre__ has quit IRC19:20
*** ducttap__ has quit IRC19:20
*** aselius has quit IRC19:21
*** markvoelker has joined #openstack-keystone19:27
*** ducttape_ has joined #openstack-keystone19:29
kmallocldap in general19:31
kmallocor ldap as default domain19:31
kmallocbecause i'd expect as much19:32
kmalloclbragstad: hmm19:39
kmalloclbragstad: do we store *any* data that is not UTC/No-TZ in the db?19:40
kmallocor should it always be TZ'd to UTC?19:40
lbragstadkmalloc: actually - it was this https://bugs.launchpad.net/keystone/+bug/164330119:41
openstackLaunchpad bug 1643301 in OpenStack Identity (keystone) "bootstrapping keystone failed when LDAP backend is in use" [Wishlist,Triaged]19:41
lbragstadkmalloc: that's a good question19:41
*** fried_rice has quit IRC19:41
openstackgerritOmar Tleimat proposed openstack/keystone master: Fix mapping_purge failure  https://review.openstack.org/40830419:46
lbragstadkmalloc: i would think they'd all be utc19:51
lbragstadkmalloc: but maybe i'm being hopeful?19:52
*** fried_rice has joined #openstack-keystone19:53
*** markvoelker has quit IRC19:59
kmalloci'm going to call normalize in the object20:04
kmalloclbragstad: so... i am leaning towards using a decimal type20:05
kmallocinstead of a float.20:06
kmallocor double20:06
*** tobberydberg has joined #openstack-keystone20:06
*** tobberydberg has quit IRC20:11
lbragstadkmalloc: ok - major differences there?20:14
lbragstadkmalloc: shouldn't any of those cover our use cases?20:14
kmallocdecimal doesn't round20:14
lbragstadohh20:14
lbragstadyes20:14
kmalloctrying to see how to build it.20:14
openstackgerritMerged openstack/keystone master: Fix typo in index documentation  https://review.openstack.org/47999820:14
lbragstadmake mysql accept what we give it :)20:14
lbragstadwithout asking questions20:14
kmalloclol20:14
lbragstadkmalloc: sanity check - configuring ldap for keystone should be done through domain specific ldap configuration, right?20:15
kmallocyeah20:15
lbragstadcc knikolla cmurphy20:15
kmallocshould be though domain specific20:15
*** catintheroof has quit IRC20:15
lbragstadis there *ever* a reason to set https://github.com/openstack/keystone/blob/99b3641fa0eb99d903488b2f04f9b69b79b3ff16/keystone/conf/identity.py#L69 to `ldap`?20:16
kmallocyes.... sadly20:16
kmallocbecause someone does it20:16
kmallocnot because it's a good idea20:16
lbragstadso we have to continue to support it?20:16
kmallocmostly historical20:16
kmallocyep20:16
lbragstadok20:16
kmallocah better solution20:21
kmallocgoing to just make sure we have microseconds and then do *100000020:22
kmallocand then cast to float() and / 1000000 on load20:22
kmallocway easier to ensure consistent data20:22
*** tobberydberg has joined #openstack-keystone20:26
*** raildo has quit IRC20:29
lbragstadthat'll work20:37
*** tobberydberg has quit IRC20:38
*** spzala has quit IRC20:42
*** jmlowe has quit IRC20:45
cmurphylbragstad: yeah if people want to have their admin user and service users come from ldap20:45
cmurphyor if they want their ldap users in the default domain and the service users in the other domain20:45
cmurphybut it's hard to do that now that the admin token isn't a thing20:46
lbragstadcmurphy: yeah - that seems to be what i'm hitting with https://bugs.launchpad.net/keystone/+bug/164330120:46
openstackLaunchpad bug 1643301 in OpenStack Identity (keystone) "bootstrapping keystone failed when LDAP backend is in use" [Wishlist,Triaged]20:46
*** itlinux has quit IRC20:46
lbragstadbut - i'm wondering if in the typical ldap case if we can just use sql as the default and then have ldap for domain specific stuff20:47
lbragstador if we want to test all those permutations (service and admin users in ldap in the default case020:47
lbragstad)20:47
*** jmlowe has joined #openstack-keystone20:47
cmurphylbragstad: ldap in the separate domain should be typical20:48
lbragstadcmurphy: this issue now is that the KEYSTONE_IDENTITY_BACKEND var is used for both the default and ldap domain config if set to `ldap`20:49
lbragstadso `keystone.conf [identity] driver = ldap` and `/etc/keystone/domains/keystone.Users.conf [identity] driver = ldap`20:50
lbragstadso - i guess we have two options20:50
lbragstadmake bootstrap bypass user writes when ldap is configured20:51
lbragstador add another case to devstack where we always setup sql as the default identity driver backend20:51
*** dikonoor has quit IRC20:51
*** thorst has quit IRC20:52
cmurphybootstrap would also have to detect an admin user in ldap and assign the admin role to it20:53
lbragstadyeah20:55
lbragstadwhich slope is less slippery20:56
*** jmlowe has quit IRC20:56
*** markvoelker has joined #openstack-keystone20:56
cmurphy¯\_(ツ)_/¯20:57
*** edmondsw has quit IRC21:01
*** jistr is now known as jistr|off21:02
*** gyee has quit IRC21:02
lbragstadPor que no los dos?21:04
lbragstadcmurphy: the manager.ldif is the one that's applied when installing ldap, right?21:04
lbragstadhttps://github.com/openstack-dev/devstack/blob/master/files/ldap/manager.ldif.in21:04
*** dave-mccowan has quit IRC21:12
*** swain has quit IRC21:23
*** gyee has joined #openstack-keystone21:29
*** markvoelker has quit IRC21:30
cmurphylbragstad: i've actually never touched our devstack plugin, maybe knikolla knows?21:40
cmurphylbragstad: looks like it though21:40
lbragstadcmurphy: yeah - it was21:41
lbragstadit looks like it bootstraps the Manager as the ldap admin21:41
knikollao/21:41
knikollagotta love train wifi21:41
lbragstadthen populates a demo user21:41
knikollashould i read back?21:41
cmurphywe should really put https://wiki.openstack.org/wiki/OpenLDAP into the contributor docs21:41
lbragstadbaha!21:42
lbragstadi didn't even know that was a thing21:42
cmurphythat's why it needs a better home :P21:42
lbragstadexactly21:42
lbragstadi just tripped over about every mistake21:43
lbragstad`The setup on Ubuntu is somewhat different. This was done on Ubuntu 11.10`21:43
lbragstadlol21:44
*** thorst has joined #openstack-keystone21:44
cmurphylol21:44
kmalloclbragstad: i might have a working patch sans tests21:44
lbragstadkmalloc: sweet21:44
kmalloclbragstad: running tests locally and then i'll write an upgrade test21:44
kmallocoh nope21:45
kmalloci screwed up21:45
kmallocsomewhere21:45
* lbragstad hands kmalloc the semi-colon he's missing21:45
knikollaplease don't make bootstrap do more smart stuff.21:48
*** thorst has quit IRC21:48
knikollaif we make it support ldap even a little bit, people will use it for ldap.21:48
lbragstadknikolla: https://bugs.launchpad.net/keystone/+bug/164330121:49
openstackLaunchpad bug 1643301 in OpenStack Identity (keystone) "bootstrapping keystone failed when LDAP backend is in use" [Wishlist,Triaged]21:49
lbragstad*just* updated that21:49
knikollalbragstad: yes! domain specific drivers!21:50
lbragstadknikolla: so - maybe what we do is add devstack to the bug - confirm the approach to always configure the default identity driver as ldap21:51
lbragstadand put up a patch to split that patch into two21:51
lbragstad(in devstack)21:51
lbragstadthen document in devstack that the preferred way to deploy ldap with devstack is in that config21:52
knikollalbragstad: and make bootstrap fail verbosely when identity driver = ldap21:52
lbragstadwe could also add a section to our own bootstrapping docs that say bootstrapping is only required when sql is used21:52
lbragstadyeah21:53
lbragstadwe can tack that onto the same report if we want to21:53
knikolla\s\verbosely\gracefully21:53
knikolla+121:53
*** spzala has joined #openstack-keystone22:00
*** ayoung has quit IRC22:01
lbragstadkmalloc: i gotta run for a couple hours - but i'll be available tonight to test a patch if you have one by then22:06
lbragstadif not we'll hit it monday22:06
*** markvoelker has joined #openstack-keystone22:10
kmallockmalloc: ok i have the tests running22:11
kmalloci need to write the upgrade test22:11
kmalloclbragstad: it is fair to require a data migrate run, right?22:11
bretoni had to implement limiting admins by ip recently22:11
kmalloclbragstad: for rolling upgrades?22:11
kmalloclbragstad: expand, migrate *required* to run the new code?22:11
bretondue to company's policy to forbid any admin actions outside of VPN22:11
bretonand one of the requirements was not to change keystone code22:11
bretonit was fun!22:11
kmallocproxy? l7 inspection?22:12
lbragstadkmalloc: require a datamigration run?22:12
kmalloclbragstad: must migrate data22:12
kmallocdb_sync migrate22:12
kmallocor whatever22:12
kmallocnot require a contract22:12
kmallocbut so, expand, migrate required22:13
bretonit turns out it is not easy to do it in keystone with auth plugins or even custom token providers22:13
kmallocha-proxy, l7 inspection22:14
lbragstadkmalloc: yeah - you aren't required to have a contract, just pass though it22:14
kmalloclbragstad: no, i mean... require data migration to be run22:14
kmallocto use pike code in this case22:14
bretonkmalloc: how would they help me?22:14
kmallocexpand is required.22:14
kmallocbreton: look at the url and just 401/404/whatever the request if it is to an admin function22:15
kmallocand not coming via VPN22:15
lbragstadkmalloc: oh - then you'd need to lock tables to make sure old nodes aren't writing to it when your doing the data migration22:15
kmallocright.22:15
kmallocit's the password table.22:16
kmalloci can handle it in code, but it's kinda icky22:16
lbragstadthat would mean no password updates or user creations during the upgrade22:16
lbragstadso a partial outage22:16
kmallocyes22:16
kmallocis running data migrate after expand optional?22:17
kmallocoh wait nvm22:17
lbragstadkmalloc: yes22:17
kmalloci'll do it in code then22:17
lbragstadmigrating data isn't a requirement22:17
kmallocsigh22:17
lbragstadbecause of the additive only case22:17
* lbragstad shuts laptop22:17
*** lbragstad has quit IRC22:17
kmallocthe no-downtime upgrade is a silly request in the way we deal with data22:17
*** ducttape_ has quit IRC22:18
*** ducttape_ has joined #openstack-keystone22:21
*** markvoelker has quit IRC22:33
*** spzala has quit IRC22:41
*** spzala has joined #openstack-keystone23:04
*** clayton has quit IRC23:18
*** clayton has joined #openstack-keystone23:18
openstackgerritMorgan Fainberg proposed openstack/keystone master: Add int storage of datetime for password created/expires  https://review.openstack.org/49325923:23
kmalloccmurphy, knikolla: ^23:24
*** zzzeek has quit IRC23:43
*** zzzeek has joined #openstack-keystone23:44

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!