| bknudson | one thing that's kind of interesting -- I changed memcache.py to create a file on import, and I only get 1 instance of the file even though I've got uwsgi configured for multiple processes | 00:04 |
|---|---|---|
| bknudson | so looks like they're all sharing the one file descriptor | 00:04 |
| bknudson | so I wonder if they couldn't also be sharing other file descriptors? | 00:04 |
| *** chlong has quit IRC | 00:16 | |
| *** sdake has joined #openstack-keystone | 00:16 | |
| *** sdake has quit IRC | 00:21 | |
| *** ddieterly has joined #openstack-keystone | 00:25 | |
| *** sdake has joined #openstack-keystone | 00:25 | |
| *** itisha has quit IRC | 00:30 | |
| *** tonytan4ever has quit IRC | 00:33 | |
| *** ravelar has joined #openstack-keystone | 00:37 | |
| *** spzala has joined #openstack-keystone | 00:39 | |
| dstanek | bknudson: that's odd. are you using uwsgi in multiprocess mode? | 00:40 |
| *** ravelar has quit IRC | 00:41 | |
| *** wangqun has joined #openstack-keystone | 00:41 | |
| lbragstad | dstanek ah - damn! | 00:42 |
| bknudson | dstanek: I changed my trace so it's associated with a single _Host | 00:42 |
| lbragstad | dstanek I was wrong about https://review.openstack.org/#/c/355618/ | 00:42 |
| lbragstad | the last change I made | 00:42 |
| lbragstad | apparently alter isn't allowed in expand | 00:42 |
| lbragstad | cc dolphm ^ | 00:42 |
| bknudson | dstanek: http://paste.openstack.org/show/564942/ and it prints out the pid | 00:42 |
| dstanek | bknudson: i need to provision a big VM now to try out those instructions | 00:42 |
| bknudson | dstanek: so this shows opened in pid 6474, then there were a couple of readlines in 6474, but then it goes to pid 6499!! | 00:43 |
| *** spzala has quit IRC | 00:43 | |
| dstanek | lbragstad: lol, can we fix that test to allow destructive things in our migrations? | 00:44 |
| dstanek | lbragstad: i can see why we wouldn't want to add a constraint in expand, but dropping one should be ok | 00:45 |
| bknudson | uwsgi's got some options for keeping fds open: http://uwsgi-docs.readthedocs.io/en/latest/Options.html#safe-fd | 00:45 |
| dstanek | bknudson: have you looked at setting close-on-exec? | 00:47 |
| bknudson | dstanek: good q. | 00:47 |
| bknudson | python-memcached does "s = socket.socket(self.family, socket.SOCK_STREAM)" | 00:48 |
| bknudson | can I set CLOEXEC on that? | 00:48 |
| dstanek | maybe. i would expect other problems with fds too | 00:49 |
| dstanek | bknudson: iirc when we moved from flub to uwsgi we had to make our apps load lazily | 00:50 |
| dstanek | and disable threading | 00:50 |
| openstackgerrit | Merged openstack/keystone: Update developer docs for new rolling upgrade repos https://review.openstack.org/359383 | 00:50 |
| bknudson | should be easy to set a breakpoint and see where the non-lazy memcache conn is coming from. maybe it oslo.cache? | 00:51 |
| bknudson | dstanek: http://paste.openstack.org/show/564956/ | 00:54 |
| dstanek | i wouldnt't think you would see it in a breakpoint. it's just there because of the fork | 00:54 |
| *** asettle has joined #openstack-keystone | 00:54 | |
| *** shaleh has quit IRC | 00:55 | |
| bknudson | dstanek: this is something that you just changed! | 00:57 |
| bknudson | http://git.openstack.org/cgit/openstack/keystone/commit/keystone/common/cache/core.py?id=42eda48c78f1153081b4c193dc13c88561409fd3 | 00:57 |
| dstanek | ? | 00:57 |
| bknudson | there's no apply_invalidation_patch anymore. | 00:58 |
| dstanek | bknudson: nope, that's all gone | 00:58 |
| dstanek | bknudson: did i fix on accident or make it worse? | 00:58 |
| bknudson | for some reason that was getting a memcache client | 00:58 |
| bknudson | good q. Need to get new keystone I guess | 00:58 |
| bknudson | I'll first try setting CLOEXEC on the socket. | 00:59 |
| *** asettle has quit IRC | 00:59 | |
| bknudson | CLOEXEC didn't do it for whatever reason | 01:08 |
| dolphm | lbragstad: is there any reason that ALTER should be in the expand instead of the contract? | 01:11 |
| *** esp has quit IRC | 01:12 | |
| dolphm | lbragstad: actually, why does the blob column need to be nullable at all? | 01:12 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 01:13 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 01:14 |
| stevemar | rderose: o/ | 01:16 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 01:17 |
| rderose | stevemar: :) | 01:17 |
| rderose | stevemar: have a couple more ideas on this; now brute force approach :) | 01:18 |
| rderose | stevemar: this one is ready https://review.openstack.org/#/c/358111/ :) | 01:19 |
| bknudson | dstanek: when I start up keystone with your patch there's no memcache sockets opened. | 01:20 |
| bknudson | unfortunately, keystone doesn't seem to work at all after I applied the patch | 01:20 |
| dstanek | really? | 01:20 |
| bknudson | well, after I upgraded everything | 01:20 |
| dstanek | what happens? | 01:20 |
| bknudson | didn't mean to upgrade everything, accidentally did a pip install -U | 01:20 |
| bknudson | 2016-08-31 01:19:01.759 1391 ERROR keystone.common.wsgi DBError: (pymysql.err.InternalError) (1054, u"Unknown column 'password.self_service' in 'field list'") [SQL: u'SELECT password.id AS password_id, password.local_user_id AS password_local_user_id, password.pa | 01:21 |
| dstanek | did you db_sync? | 01:21 |
| bknudson | nope, just did that and now I'm testing again. | 01:22 |
| bknudson | remember how during the meeting I was poo-pooing your change and saying it didn't fix anything? | 01:22 |
| *** david-lyle_ has joined #openstack-keystone | 01:23 | |
| bknudson | Usually this fails in a couple of seconds. | 01:23 |
| bknudson | it's possible that this is fixed by one of the other library changes I picked up. I'll have to try it again without all these changes. | 01:24 |
| bknudson | also, can just wait until we pick up the new keystone with your change. | 01:24 |
| bknudson | still scary that this could be broken at any time by a change that opens a memcache conn on init. | 01:25 |
| stevemar | rderose: why can't we just set server_default to the created_at column? | 01:28 |
| bknudson | browne: can you try with http://git.openstack.org/cgit/openstack/keystone/commit/keystone/common/cache/core.py?id=42eda48c78f1153081b4c193dc13c88561409fd3 to see if that fixes the memcache issues? | 01:29 |
| rderose | stevemar: because if fails for older versions of mysql | 01:30 |
| rderose | 5.5 | 01:30 |
| browne | bknudson: sure, it'll take me a while, but i'll give it a shot | 01:30 |
| stevemar | rderose: that stinks | 01:30 |
| rderose | stevemar: for mysql versions 5.5 and older, we would need to use timestamp | 01:30 |
| rderose | stevemar: tell me about | 01:30 |
| rderose | stevemar: not giving up yet, I've got a few more tricks up my sleeve ;) | 01:30 |
| bknudson | browne: what it looks like was happening is that there were some connections opened in one process that another process inherited | 01:31 |
| bknudson | and they wound up stomping on each other. | 01:31 |
| *** tonytan4ever has joined #openstack-keystone | 01:33 | |
| *** EinstCrazy has joined #openstack-keystone | 01:33 | |
| *** tonytan4ever has quit IRC | 01:38 | |
| openstackgerrit | Dolph Mathews proposed openstack/keystone: Disallow new migrations in the legacy migration repository https://review.openstack.org/363343 | 01:40 |
| *** chlong has joined #openstack-keystone | 01:41 | |
| openstackgerrit | Dolph Mathews proposed openstack/keystone: Disallow new migrations in the legacy migration repository https://review.openstack.org/363343 | 01:42 |
| *** ddieterly has quit IRC | 01:46 | |
| browne | bknudson: interesting | 01:50 |
| notmorgan | bknudson: socket weirdness? | 01:50 |
| *** davechen has joined #openstack-keystone | 01:50 | |
| notmorgan | browne: or is this something not cache related? | 01:51 |
| bknudson | notmorgan: yes, it looked like 2 processes wound up sharing the memcache socket. | 01:51 |
| notmorgan | ah | 01:51 |
| notmorgan | that should be impossible. we had(?) a mutex to prevent that. | 01:51 |
| bknudson | notmorgan: http://paste.openstack.org/show/564942/ | 01:51 |
| notmorgan | or is this some wonkyness with thread local in memcache python lib | 01:51 |
| notmorgan | we may want to just quickly roll a pymemcache backend and call it a day | 01:51 |
| bknudson | I added the pid to the "readline data" line | 01:51 |
| notmorgan | because then we aren't thread localling deep in the python-memcache lib | 01:52 |
| notmorgan | stevemar: what did I promise to do earlier today?. OH right, rev event -> dict | 01:52 |
| bknudson | I don't think it has anything to do with use of thread local | 01:52 |
| notmorgan | bknudson: well it might make things weird(er) | 01:52 |
| * notmorgan has been dealing with cache issues all day | 01:54 | |
| * notmorgan looks at shade | 01:54 | |
| bknudson | what's the issue? | 01:54 |
| notmorgan | mis used of dogpile in the case of shade | 01:54 |
| notmorgan | :P | 01:54 |
| stevemar | notmorgan: oh right, that can go into rc | 01:55 |
| notmorgan | negative caching (None/False values being cached in a way that was breaking nodepool) | 01:55 |
| *** spzala has joined #openstack-keystone | 01:55 | |
| stevemar | bknudson: since you already looked at it https://review.openstack.org/#/c/358872/ or i can get dstanek to look at it in the morning if you're heads down on something | 02:00 |
| bknudson | stevemar: I'd rather see if the bug goes away given dstanek's change | 02:01 |
| stevemar | bknudson: dstanek has something up? | 02:01 |
| bknudson | stevemar: looks like http://git.openstack.org/cgit/openstack/keystone/commit/keystone/common/cache/core.py?id=42eda48c78f1153081b4c193dc13c88561409fd3 fixed it. | 02:01 |
| dstanek | notmorgan: the mutex would prevent the forking behavior | 02:01 |
| bknudson | by changing the code so that there aren't memcache clients created before forking | 02:02 |
| stevemar | bknudson: i mentioned that it might and you called me crazy | 02:02 |
| bknudson | stevemar: I know. I'm eating crow now. | 02:02 |
| stevemar | bknudson: :) | 02:02 |
| *** EinstCra_ has joined #openstack-keystone | 02:03 | |
| stevemar | bknudson: nothing makes me happier than marking a bug as invalid | 02:03 |
| bknudson | stevemar: the bug is valid. it's fixed. | 02:03 |
| stevemar | ok ok, dupe, fix released; whatever takes it off my queue | 02:04 |
| dstanek | if i accidentally fixes 2 extra bugs with that commit what do i win? | 02:04 |
| bknudson | dstanek: you can tweet about it | 02:04 |
| dstanek | and maybe that deserves a #success | 02:05 |
| bknudson | definitely a #success. | 02:05 |
| stevemar | dstanek: whats the first extra bug? | 02:05 |
| stevemar | or other extra bug | 02:05 |
| stevemar | bknudson: dstanek want to update https://bugs.launchpad.net/keystone/+bug/1609566 with details ? | 02:05 |
| openstack | Launchpad bug 1609566 in OpenStack Identity (keystone) "500 error from revocation event deserialize" [Medium,In progress] - Assigned to Morgan Fainberg (mdrnstm) | 02:05 |
| bknudson | stevemar: let's leave https://bugs.launchpad.net/keystone/+bug/1609566 open since we think that one might be something else. | 02:06 |
| *** EinstCrazy has quit IRC | 02:06 | |
| bknudson | I haven't been able to test this very well yet. | 02:06 |
| stevemar | okay with me | 02:06 |
| dstanek | stevemar: i'm building a test environment as we speak so that i can check for myself | 02:06 |
| stevemar | cool | 02:06 |
| stevemar | i was hoping it would close https://bugs.launchpad.net/keystone/+bug/1600393 and https://bugs.launchpad.net/keystone/+bug/1600394 :( | 02:07 |
| openstack | Launchpad bug 1600393 in OpenStack Identity (keystone) "v2.0 catalog seen in v3 token" [High,Confirmed] | 02:07 |
| openstack | Launchpad bug 1600394 in OpenStack Identity (keystone) "memcache raising "too many values to unpack"" [High,Confirmed] | 02:07 |
| bknudson | those 2 should be closed by the change | 02:07 |
| bknudson | well, I don't know about 1600393 now. | 02:07 |
| stevemar | bknudson: can you comment on 394? | 02:08 |
| bknudson | I never saw that one myself. | 02:08 |
| stevemar | nevermind then | 02:09 |
| *** ddieterly has joined #openstack-keystone | 02:10 | |
| bknudson | Updated https://bugs.launchpad.net/keystone/+bug/1600394 | 02:12 |
| openstack | Launchpad bug 1600394 in OpenStack Identity (keystone) "memcache raising "too many values to unpack"" [High,Confirmed] | 02:12 |
| stevemar | thanks bknudson <3 | 02:12 |
| bknudson | It's nice to finally have some idea what the problem was. | 02:12 |
| *** sdake has quit IRC | 02:16 | |
| *** sdake has joined #openstack-keystone | 02:17 | |
| *** Michaellaneous has quit IRC | 02:19 | |
| stevemar | bknudson: the caching was broken :) | 02:22 |
| *** tqtran has quit IRC | 02:22 | |
| *** EinstCrazy has joined #openstack-keystone | 02:24 | |
| *** spzala has quit IRC | 02:26 | |
| *** EinstCra_ has quit IRC | 02:27 | |
| *** david-lyle_ has quit IRC | 02:31 | |
| *** gyee has quit IRC | 02:31 | |
| *** ddieterly has quit IRC | 02:33 | |
| openstackgerrit | Dolph Mathews proposed openstack/keystone: Fix nits in db migration dev docs https://review.openstack.org/363358 | 02:33 |
| notmorgan | bknudson: oh we had a client pre-fork being made | 02:34 |
| notmorgan | sigh | 02:34 |
| *** tonytan4ever has joined #openstack-keystone | 02:34 | |
| *** tonytan4ever has quit IRC | 02:38 | |
| *** tonytan4ever has joined #openstack-keystone | 02:38 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: Fix nits in db migration dev docs https://review.openstack.org/363358 | 02:50 |
| *** chrisshattuck has joined #openstack-keystone | 02:52 | |
| *** chrisshattuck has quit IRC | 02:57 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 03:04 |
| *** zouyapeng has quit IRC | 03:07 | |
| *** browne has quit IRC | 03:13 | |
| *** tqtran has joined #openstack-keystone | 03:21 | |
| *** dikonoor has joined #openstack-keystone | 03:26 | |
| *** ayoung has quit IRC | 03:27 | |
| *** esp has joined #openstack-keystone | 03:34 | |
| *** dikonoor has quit IRC | 03:34 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 03:38 |
| *** woodster_ has quit IRC | 03:39 | |
| *** adriant has quit IRC | 03:41 | |
| openstackgerrit | Ha Van Tu proposed openstack/keystone: Config logABug feature for Keystone api-ref https://review.openstack.org/363374 | 03:45 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 03:46 |
| *** namnh has joined #openstack-keystone | 03:47 | |
| *** esp has quit IRC | 03:49 | |
| *** esp has joined #openstack-keystone | 03:51 | |
| *** Gorian has quit IRC | 04:13 | |
| *** crinkle_ has joined #openstack-keystone | 04:16 | |
| *** crinkle has quit IRC | 04:17 | |
| *** su_zhang has joined #openstack-keystone | 04:26 | |
| *** spzala has joined #openstack-keystone | 04:26 | |
| *** spzala has quit IRC | 04:31 | |
| *** ccneill_ has joined #openstack-keystone | 04:35 | |
| *** ccneill has quit IRC | 04:35 | |
| *** links has joined #openstack-keystone | 04:36 | |
| *** chlong has quit IRC | 04:38 | |
| *** chlong has joined #openstack-keystone | 04:39 | |
| *** esp has quit IRC | 04:45 | |
| *** chlong has quit IRC | 04:45 | |
| *** su_zhang has quit IRC | 04:47 | |
| *** ravelar has joined #openstack-keystone | 04:48 | |
| *** su_zhang_ has joined #openstack-keystone | 04:50 | |
| *** su_zhang_ has quit IRC | 04:52 | |
| *** su_zhang has joined #openstack-keystone | 04:53 | |
| *** esp has joined #openstack-keystone | 04:54 | |
| *** su_zhang has quit IRC | 04:57 | |
| *** chlong has joined #openstack-keystone | 04:57 | |
| *** asettle has joined #openstack-keystone | 04:57 | |
| *** asettle has quit IRC | 05:02 | |
| *** sdake has quit IRC | 05:10 | |
| *** sdake_ has joined #openstack-keystone | 05:10 | |
| *** ccneill_ is now known as ccneill | 05:11 | |
| *** jaosorior has joined #openstack-keystone | 05:21 | |
| *** roxanaghe has joined #openstack-keystone | 05:25 | |
| *** sdake_ has quit IRC | 05:26 | |
| *** dikonoor has joined #openstack-keystone | 05:27 | |
| *** ravelar has quit IRC | 05:27 | |
| *** roxanaghe has quit IRC | 05:32 | |
| *** sdake has joined #openstack-keystone | 05:33 | |
| *** richm has quit IRC | 05:40 | |
| *** ccneill has quit IRC | 05:50 | |
| *** markvoelker has joined #openstack-keystone | 05:58 | |
| *** pcaruana has joined #openstack-keystone | 06:15 | |
| openstackgerrit | Ha Van Tu proposed openstack/keystone: Config logABug feature for Keystone api-ref https://review.openstack.org/363374 | 06:17 |
| *** rcernin has joined #openstack-keystone | 06:21 | |
| *** sdake has quit IRC | 06:22 | |
| *** sheel has joined #openstack-keystone | 06:24 | |
| *** markvoelker has quit IRC | 06:34 | |
| openstackgerrit | Ukesh Kumar proposed openstack/keystone: check for user existence, for role add to user https://review.openstack.org/362606 | 06:36 |
| *** namnh has quit IRC | 07:03 | |
| *** tesseract- has joined #openstack-keystone | 07:08 | |
| *** jpena|off is now known as jpena | 07:13 | |
| *** spzala has joined #openstack-keystone | 07:26 | |
| *** markvoelker has joined #openstack-keystone | 07:30 | |
| *** spzala has quit IRC | 07:31 | |
| *** markvoelker has quit IRC | 07:35 | |
| *** spzala has joined #openstack-keystone | 07:42 | |
| *** spzala has quit IRC | 07:47 | |
| *** spzala has joined #openstack-keystone | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** joerch has joined #openstack-keystone | 08:00 | |
| *** markvoelker has joined #openstack-keystone | 08:01 | |
| *** zzzeek has joined #openstack-keystone | 08:01 | |
| *** spzala has quit IRC | 08:03 | |
| *** chlong has quit IRC | 08:06 | |
| *** markvoelker has quit IRC | 08:07 | |
| *** zhangjl has joined #openstack-keystone | 08:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c https://review.openstack.org/318435 | 08:10 |
| *** markvoelker has joined #openstack-keystone | 08:29 | |
| *** auggy has quit IRC | 08:30 | |
| *** rkrum has quit IRC | 08:30 | |
| *** auggy has joined #openstack-keystone | 08:31 | |
| *** markvoelker has quit IRC | 08:34 | |
| *** samueldmq has quit IRC | 08:38 | |
| *** samueldmq has joined #openstack-keystone | 08:39 | |
| *** ChanServ sets mode: +v samueldmq | 08:39 | |
| *** asettle has joined #openstack-keystone | 08:58 | |
| *** markvoelker has joined #openstack-keystone | 08:59 | |
| *** andreykurilin_ has joined #openstack-keystone | 09:00 | |
| *** markvoelker has quit IRC | 09:04 | |
| *** namnh has joined #openstack-keystone | 09:06 | |
| *** andreykurilin_ has quit IRC | 09:25 | |
| *** andreykurilin has joined #openstack-keystone | 09:26 | |
| *** andreykurilin has left #openstack-keystone | 09:26 | |
| *** markvoelker has joined #openstack-keystone | 09:29 | |
| *** markvoelker has quit IRC | 09:33 | |
| *** andreykurilin__ has quit IRC | 09:36 | |
| *** andreykurilin has joined #openstack-keystone | 09:36 | |
| *** code-R has joined #openstack-keystone | 09:45 | |
| *** code-R_ has joined #openstack-keystone | 09:48 | |
| *** code-R has quit IRC | 09:51 | |
| *** markvoelker has joined #openstack-keystone | 09:57 | |
| *** markvoelker has quit IRC | 10:02 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c https://review.openstack.org/318435 | 10:10 |
| *** richm has joined #openstack-keystone | 10:12 | |
| *** wangqun has quit IRC | 10:19 | |
| *** davechen has left #openstack-keystone | 10:24 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 10:25 |
| *** markvoelker has joined #openstack-keystone | 10:26 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 10:26 |
| *** markvoelker has quit IRC | 10:30 | |
| *** chrichip has joined #openstack-keystone | 10:34 | |
| *** EinstCrazy has quit IRC | 10:35 | |
| *** zhangjl has quit IRC | 10:36 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 10:48 |
| *** sdake has joined #openstack-keystone | 10:49 | |
| *** chrichip has quit IRC | 10:50 | |
| *** chrichip has joined #openstack-keystone | 10:51 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 10:53 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 10:53 |
| *** code-R_ has quit IRC | 10:54 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 10:55 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 10:55 |
| *** chrichip has quit IRC | 10:56 | |
| *** markvoelker has joined #openstack-keystone | 10:56 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 10:56 |
| *** chrichip has joined #openstack-keystone | 10:57 | |
| *** markvoelker has quit IRC | 11:00 | |
| *** code-R has joined #openstack-keystone | 11:03 | |
| *** chrichip has quit IRC | 11:10 | |
| *** chrichip has joined #openstack-keystone | 11:11 | |
| *** chrichip has quit IRC | 11:18 | |
| *** chrichip has joined #openstack-keystone | 11:19 | |
| *** namnh has quit IRC | 11:23 | |
| *** markvoelker has joined #openstack-keystone | 11:25 | |
| *** chrichip has quit IRC | 11:26 | |
| *** chrichip has joined #openstack-keystone | 11:27 | |
| *** markvoelker has quit IRC | 11:29 | |
| *** amakarov_away is now known as amakarov | 11:40 | |
| amakarov | stevemar, today | 11:40 |
| amakarov | stevemar, starting right now | 11:41 |
| *** chrichip has quit IRC | 11:44 | |
| *** chrichip has joined #openstack-keystone | 11:45 | |
| *** asettle has quit IRC | 11:53 | |
| *** markvoelker has joined #openstack-keystone | 11:56 | |
| *** ddieterly has joined #openstack-keystone | 11:57 | |
| *** asettle has joined #openstack-keystone | 11:58 | |
| *** asettle has joined #openstack-keystone | 11:59 | |
| *** spzala has joined #openstack-keystone | 11:59 | |
| *** chrichip has quit IRC | 12:00 | |
| *** chrichip has joined #openstack-keystone | 12:00 | |
| *** markvoelker has quit IRC | 12:01 | |
| *** ddieterly has quit IRC | 12:02 | |
| *** spzala has quit IRC | 12:03 | |
| *** jaosorior has quit IRC | 12:04 | |
| *** jaosorior has joined #openstack-keystone | 12:06 | |
| *** rodrigods has quit IRC | 12:09 | |
| *** rodrigods has joined #openstack-keystone | 12:09 | |
| *** jpena is now known as jpena|lunch | 12:12 | |
| *** ddieterly has joined #openstack-keystone | 12:12 | |
| *** sheel has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-keystone | 12:24 | |
| *** pauloewerton has joined #openstack-keystone | 12:28 | |
| *** markvoelker has quit IRC | 12:29 | |
| *** ddieterly has quit IRC | 12:29 | |
| *** Michaellaneous has joined #openstack-keystone | 12:31 | |
| Michaellaneous | I am kinda majorly fucked I think. | 12:31 |
| Michaellaneous | Someone suggest to me to delete default domain to solve some issue. | 12:31 |
| Michaellaneous | Now...nothing works. | 12:31 |
| Michaellaneous | How can I reeanble token auth? | 12:31 |
| stevemar | o/ | 12:32 |
| stevemar | Michaellaneous: you can recreate the default domain by running keystone-manage bootstrap | 12:33 |
| Michaellaneous | Either --bootstrap-password argument or OS_BOOTSTRAP_PASSWORD must be set. | 12:33 |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Pre-cache new tokens https://review.openstack.org/309146 | 12:33 |
| Michaellaneous | I suppose I need a password. | 12:33 |
| stevemar | Michaellaneous: yep | 12:33 |
| stevemar | amakarov: thanks! | 12:34 |
| *** openstackgerrit has quit IRC | 12:34 | |
| *** openstackgerrit has joined #openstack-keystone | 12:34 | |
| amakarov | stevemar, let's wait for jenkins )) | 12:34 |
| Michaellaneous | ERROR keystone DomainNotFound: Could not find domain: default | 12:34 |
| Michaellaneous | :( | 12:34 |
| stevemar | Michaellaneous: let me get some breakfast in me and i can help you out | 12:35 |
| Michaellaneous | Thank you. | 12:35 |
| *** su_zhang has joined #openstack-keystone | 12:36 | |
| Michaellaneous | Also one thing I don't understand, I have a full system backup of when it was working. | 12:38 |
| Michaellaneous | Trying to copy it over with a rescue system. | 12:39 |
| Michaellaneous | And...it still doesn't work. | 12:39 |
| stevemar | Michaellaneous: you're trying to create a fresh openstack install? | 12:42 |
| Michaellaneous | No. | 12:42 |
| Michaellaneous | It was working perfectly fine. | 12:42 |
| Michaellaneous | Then I deleted the default domain because LDAP wasn't working and someone suggested it. | 12:43 |
| stevemar | Michaellaneous: i saw a few messages yesterday from you aobut adding ldap | 12:43 |
| Michaellaneous | Yeah. | 12:43 |
| Michaellaneous | Well I removed that again. | 12:43 |
| Michaellaneous | From the keystone conf | 12:43 |
| stevemar | alright, what was the output of the bootstrap command? anything *before* the error? | 12:44 |
| Michaellaneous | 2016-08-31 14:34:24.095 12879 WARNING keystone.assignment.core [req-b51cb88f-6b5f-49b3-b808-f9fdafa262b8 - - - - -] Deprecated: Use of the identity driver config to automatically configure the same assignment driver has been deprecated, in the "O" release, the assignment driver will need to be expicitly configured if different than the default (SQL). | 12:45 |
| Michaellaneous | 2016-08-31 14:34:24.247 12879 INFO keystone.cmd.cli [req-b51cb88f-6b5f-49b3-b808-f9fdafa262b8 - - - - -] Domain default already exists, skipping creation. | 12:45 |
| Michaellaneous | 2016-08-31 14:34:24.253 12879 CRITICAL keystone [req-b51cb88f-6b5f-49b3-b808-f9fdafa262b8 - - - - -] DomainNotFound: Could not find domain: default | 12:45 |
| *** woodster_ has joined #openstack-keystone | 12:48 | |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Pre-cache new tokens https://review.openstack.org/309146 | 12:50 |
| *** jefrite has quit IRC | 12:51 | |
| Michaellaneous | I have never done bootstrap before. | 12:52 |
| Michaellaneous | I followed this guide. | 12:52 |
| Michaellaneous | http://docs.openstack.org/mitaka/install-guide-ubuntu/keystone-install.html | 12:52 |
| Michaellaneous | And I mean it worked. | 12:52 |
| stevemar | Michaellaneous: ahhh "Domain default already exists, skipping creation." | 12:53 |
| Michaellaneous | Yeah and the next error is Domain not found. | 12:53 |
| stevemar | Michaellaneous: interesting | 12:53 |
| *** markvoelker has joined #openstack-keystone | 12:54 | |
| stevemar | Michaellaneous: using mysql as your database? | 12:54 |
| Michaellaneous | MariaDB | 12:54 |
| Michaellaneous | Yeah, mysql | 12:54 |
| stevemar | can you display whatever is in one of the tables? | 12:55 |
| *** jefrite has joined #openstack-keystone | 12:56 | |
| stevemar | Michaellaneous: kinda like... http://paste.openstack.org/show/565167/ | 12:56 |
| Michaellaneous | Yeo hold on. | 12:57 |
| stevemar | cool | 12:57 |
| Michaellaneous | Yep it's there. | 12:57 |
| Michaellaneous | https://i.imgur.com/oNRNPmK.png | 12:57 |
| stevemar | okay, its just got a funky ID instead of the usual "default" | 12:58 |
| stevemar | i think the rdo guide does that now | 12:58 |
| stevemar | meh, no difference | 12:58 |
| stevemar | so, what's your openrc file look like ? | 12:58 |
| stevemar | as long as that default domain is there, things are do-able | 12:58 |
| *** markvoelker has quit IRC | 12:58 | |
| Michaellaneous | http://pastebin.com/aKzSgtKM | 12:59 |
| *** EinstCrazy has joined #openstack-keystone | 13:03 | |
| stevemar | Michaellaneous: and whats the output of something like ... `openstack user list --debug` | 13:04 |
| *** markvoelker has joined #openstack-keystone | 13:04 | |
| *** spzala has joined #openstack-keystone | 13:04 | |
| Michaellaneous | http://pastebin.com/3VrnP60p | 13:05 |
| Michaellaneous | Waot. | 13:05 |
| Michaellaneous | Using auth-plugin token endpoint? | 13:05 |
| Michaellaneous | That...doesn't seem right. | 13:05 |
| *** ddieterly has joined #openstack-keystone | 13:05 | |
| *** joerch has quit IRC | 13:05 | |
| Michaellaneous | Oh wait whoops.. | 13:06 |
| Michaellaneous | Sorry hold on. | 13:06 |
| Michaellaneous | http://pastebin.com/PtYbCEh2 | 13:06 |
| Michaellaneous | Now I sourced admin-openrc | 13:06 |
| stevemar | ah okay | 13:06 |
| *** ddieterly has quit IRC | 13:09 | |
| stevemar | Michaellaneous: still failing to auth, that's not good | 13:10 |
| *** markvoelker has quit IRC | 13:10 | |
| *** raildo has joined #openstack-keystone | 13:11 | |
| Michaellaneous | Hm... | 13:14 |
| *** sdake_ has joined #openstack-keystone | 13:16 | |
| *** jpena|lunch is now known as jpena | 13:18 | |
| *** su_zhang has quit IRC | 13:18 | |
| *** sdake has quit IRC | 13:19 | |
| breton | Michaellaneous: > 'user_domain_name': 'default', 'project_domain_name': 'default' | 13:21 |
| Michaellaneous | Yes? | 13:21 |
| breton | Michaellaneous: it should probably be Default and Default (note the case) | 13:21 |
| Michaellaneous | Nope didn't do it. | 13:21 |
| Michaellaneous | It worked before with that. | 13:21 |
| Michaellaneous | If I delete the keystone DB, do all of the identify stuff again. | 13:22 |
| Michaellaneous | How much of my other services do I have to redo? | 13:22 |
| *** sdake_ is now known as sdake | 13:22 | |
| Michaellaneous | All of it I presume. | 13:22 |
| breton | i highly doubt that "default" as name will work and ever worked before | 13:24 |
| stevemar | Michaellaneous: eh, you can backup your db and try it | 13:24 |
| Michaellaneous | How do I back it up? | 13:24 |
| stevemar | if it messes up, restore the old one, you'll be in no worse state | 13:24 |
| bknudson | the dbs typically do case-insensitive comparisons | 13:24 |
| stevemar | http://www.liquidweb.com/kb/how-to-back-up-mysql-databases-from-the-command-line/ | 13:25 |
| Michaellaneous | Actually...where is the keystone db saved. | 13:27 |
| Michaellaneous | When I use apache2 for auth. | 13:27 |
| dstanek | Michaellaneous: what's the current error? | 13:29 |
| Michaellaneous | http://pastebin.com/PtYbCEh2 | 13:29 |
| stevemar | amakarov: what about the invalidations, shouldn't those be conditional as well? | 13:30 |
| *** su_zhang has joined #openstack-keystone | 13:31 | |
| dstanek | Michaellaneous: do you have the server side log? the client side won't ever say why you got rejected | 13:31 |
| Michaellaneous | 2016-08-31 15:33:02.201 12554 WARNING keystone.auth.controllers [req-b1d1eb85-cdc8-4a33-921a-c67c5b02cb76 - - - - -] Domain is disabled: 36d024eae7ea4865b87d1e29bd73cef5 | 13:33 |
| Michaellaneous | 2016-08-31 15:33:02.202 12554 WARNING keystone.common.wsgi [req-b1d1eb85-cdc8-4a33-921a-c67c5b02cb76 - - - - -] Authorization failed. The request you have made requires authentication. from 127.0.0.1 | 13:33 |
| dstanek | Michaellaneous: can you enable it? | 13:33 |
| Michaellaneous | no | 13:33 |
| dstanek | why not? | 13:34 |
| openstackgerrit | Merged openstack/keystone: Fix nits in db migration dev docs https://review.openstack.org/363358 | 13:34 |
| Michaellaneous | It can't find the command openstack domain | 13:34 |
| amakarov | stevemar, looks like last rebase wasn't good :) What for region invalidation - it just should be there or we end up with false positive token validations | 13:35 |
| dstanek | Michaellaneous: since you are in such a bad place i would say back up you DB and do the update by hand | 13:36 |
| Michaellaneous | You mean redo keystone. | 13:37 |
| dstanek | no, backup your DB and enable the domain | 13:37 |
| rderose | henrynash: regarding 362510, just curious, what do you think it would do? I mean, the column would no longer allow nulls, so it would have to apply the default value | 13:37 |
| dstanek | Michaellaneous: what release are you using? | 13:37 |
| Michaellaneous | mitaka | 13:37 |
| Michaellaneous | How do I enable the domain by hand? | 13:38 |
| rderose | henrynash: working on a test to prove this out, but I can't imagine it not working | 13:38 |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Pre-cache new tokens https://review.openstack.org/309146 | 13:38 |
| dstanek | Michaellaneous: you'll have to update the database record | 13:38 |
| dstanek | Michaellaneous: do you have any account that works? | 13:38 |
| Michaellaneous | I don't have any openstack accounts that work,no. | 13:39 |
| *** su_zhang has quit IRC | 13:39 | |
| stevemar | dstanek: probably not | 13:39 |
| *** ddieterly has joined #openstack-keystone | 13:39 | |
| *** su_zhang has joined #openstack-keystone | 13:39 | |
| dstanek | Michaellaneous: yeah, before you trash everything and spend all that time i'd try to manually fix it | 13:40 |
| *** chrichip has quit IRC | 13:40 | |
| Michaellaneous | Yeah. | 13:40 |
| *** EinstCrazy has quit IRC | 13:40 | |
| Michaellaneous | Hm... | 13:41 |
| Michaellaneous | I think I am getting closer. | 13:41 |
| Michaellaneous | When I tried to backup it says "No table "access_token". | 13:41 |
| *** chrichip has joined #openstack-keystone | 13:41 | |
| Michaellaneous | When I try to populate it I get the same error. | 13:41 |
| dstanek | what are you doing to backup? | 13:42 |
| Michaellaneous | sqldump | 13:42 |
| Michaellaneous | mysqldump: Got error: 1146: "Table 'keystone.access_token' doesn't exist" when using LOCK TABLES | 13:42 |
| dstanek | are you missing tables? | 13:42 |
| *** su_zhang has quit IRC | 13:44 | |
| Michaellaneous | Maybe? | 13:45 |
| Michaellaneous | what if I create the table and then populate the DB again | 13:46 |
| *** chrichip has quit IRC | 13:46 | |
| Michaellaneous | I don't understand why my full system backup did not work... | 13:46 |
| lbragstad | dolphm blob needs to be nullable so that triggers will prevent writes with the proper error on newton nodes during the migration? | 13:46 |
| lbragstad | s/?// | 13:46 |
| *** chrichip has joined #openstack-keystone | 13:47 | |
| dolphm | lbragstad: so, if the old column is non-nullable, what error do you get on writes in newton? | 13:47 |
| dstanek | Michaellaneous: can you paste a list of tables you have using 'show tables'? | 13:48 |
| Michaellaneous | Hold on I am trying to figure mysql out. | 13:50 |
| Michaellaneous | Sometime doesnt want e to work | 13:50 |
| Michaellaneous | dstanek, http://pastebin.com/hL0vHRDJ | 13:52 |
| *** esp has quit IRC | 13:54 | |
| dstanek | Michaellaneous: odd. that looks correct | 13:54 |
| *** ayoung has joined #openstack-keystone | 13:54 | |
| *** ChanServ sets mode: +v ayoung | 13:54 | |
| *** jamielennox|away is now known as jamielennox | 13:55 | |
| lbragstad | dolphm you end up getting a 500 from sql saying it can't write to the credential table because blob is None (since the new code doesn't know about blob) | 13:59 |
| lbragstad | dolphm so - it's the same result just caused by a different error | 13:59 |
| lbragstad | dolphm this is the original error http://cdn.pasteraw.com/lv0hubtjes1an7km3h1nifsivcqb4xh | 14:00 |
| lbragstad | when blob is nullable | 14:00 |
| *** links has quit IRC | 14:00 | |
| lbragstad | s/is/was not/ | 14:01 |
| dstanek | lbragstad: does the alter not work after all or are you trying to find another way? | 14:01 |
| *** jamielennox is now known as jamielennox|away | 14:01 | |
| lbragstad | dstanek nope - the alter was right | 14:01 |
| lbragstad | dstanek I can alter the table to make blob nullable | 14:02 |
| dstanek | bknudson: i don't know what happened, but my env was cloning openstack all night and just finished | 14:02 |
| *** chrichip has quit IRC | 14:02 | |
| lbragstad | dstanek and the result is that triggers make the credential table read only | 14:02 |
| *** jamielennox|away is now known as jamielennox | 14:02 | |
| lbragstad | dstanek it just fails unit tests because of the banned sql operation | 14:02 |
| bknudson | dstanek: it does take a long time to setup & deploy | 14:03 |
| *** chrichip has joined #openstack-keystone | 14:03 | |
| dstanek | lbragstad: i added a comment on the review that shows how to stop that. did you see it? | 14:03 |
| lbragstad | dstanek yeah - modifying the patch and pushing for review | 14:04 |
| *** sdake_ has joined #openstack-keystone | 14:06 | |
| *** sdake has quit IRC | 14:07 | |
| dstanek | Michaellaneous: if that's not working you can either just run the 'update' since it's only a single row or maybe standup a new keystone node and see where you can get to | 14:08 |
| Michaellaneous | run the update? | 14:08 |
| *** tonytan4ever has quit IRC | 14:09 | |
| dolphm | lbragstad: even with the trigger being BEFORE INSERT? | 14:09 |
| lbragstad | dolphm yep | 14:09 |
| *** tonytan4ever has joined #openstack-keystone | 14:09 | |
| dolphm | lbragstad: so, the column needs to be nullable but it will never contain a null value after a transaction? | 14:10 |
| lbragstad | dolphm once we make blob nullable the triggers are in place to make the credential table read only | 14:10 |
| dstanek | Michaellaneous: update that domain/project row in the database to enable it | 14:11 |
| dolphm | lbragstad: well, there's a tiny race there, but sure | 14:11 |
| dolphm | lbragstad: triggers are added later | 14:11 |
| *** spzala has quit IRC | 14:12 | |
| dolphm | lbragstad: you r patch has a confict btw | 14:12 |
| lbragstad | dolphm resolving it now | 14:12 |
| *** spzala has joined #openstack-keystone | 14:12 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 14:13 |
| Michaellaneous | dstanek, ERROR 1146 (42S02): Table 'keystone.domain' doesn't exist | 14:15 |
| *** spzala has quit IRC | 14:17 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Document credential encryption https://review.openstack.org/354497 | 14:18 |
| *** tonytan4ever has quit IRC | 14:18 | |
| *** esp has joined #openstack-keystone | 14:19 | |
| *** joerch has joined #openstack-keystone | 14:19 | |
| *** tonytan4ever has joined #openstack-keystone | 14:20 | |
| *** michauds has joined #openstack-keystone | 14:20 | |
| *** su_zhang has joined #openstack-keystone | 14:21 | |
| dstanek | Michaellaneous: what's that from? | 14:23 |
| Michaellaneous | hold on a moment | 14:24 |
| *** spedione|AWAY is now known as spedione | 14:24 | |
| dolphm | does anyone use the file or path query operators in gerrit? i can't get something basic like this working: project:openstack/keystone file:"^keystone/*" | 14:26 |
| lbragstad | dolphm try - file:"^keystone/common/sql/@" | 14:28 |
| *** spzala has joined #openstack-keystone | 14:28 | |
| lbragstad | dolphm or in your case project:openstack/keystone file:"^keystone/@" | 14:28 |
| dolphm | lbragstad: thank you! that works - is that documented somewhere?! | 14:29 |
| *** tonytan4ever has quit IRC | 14:30 | |
| dolphm | rderose: if you use Partial-Bug on a patch, then the last patch in the series that finally fixes the bug should still be Closes-Bug | 14:31 |
| *** tonytan4ever has joined #openstack-keystone | 14:31 | |
| rderose | dolphm: okay | 14:31 |
| dolphm | rderose: otherwise the bug will be left hanging open | 14:32 |
| lbragstad | dolphm i clicked on a link somewhere a long time ago | 14:32 |
| lbragstad | dolphm but i think it was after a bunch of random googling | 14:32 |
| rderose | dolphm: thanks, done | 14:32 |
| *** ksavich has joined #openstack-keystone | 14:33 | |
| Michaellaneous | dstanek, it's when I wanna look at what is inside the domain table | 14:33 |
| Michaellaneous | but everything is fucked | 14:33 |
| Michaellaneous | i cant even drop keystone | 14:33 |
| dolphm | rderose: is an arbitrary date for all databases really the best possible solution? | 14:34 |
| rderose | dolphm: the only possible solution :) | 14:35 |
| rderose | dolphm: unless we drop support for mysql 5.5 | 14:35 |
| dolphm | rderose: well, it's certainly not the *only* possible solution for ALL databases | 14:36 |
| *** su_zhang has quit IRC | 14:36 | |
| rderose | :) | 14:36 |
| rderose | dolphm: true | 14:36 |
| rderose | dolphm: I've tried a number of different options and this is the only one the works | 14:36 |
| *** su_zhang has joined #openstack-keystone | 14:37 | |
| *** david-lyle_ has joined #openstack-keystone | 14:37 | |
| dolphm | rderose: on 5.5 | 14:37 |
| rderose | dolphm: for 5.5, I would need to change the column type to timestamp | 14:37 |
| rderose | dolphm: however, this wouldn't work for postgres | 14:37 |
| rderose | dolphm: so would have to do something different there | 14:38 |
| *** ddieterly is now known as ddieterly[away] | 14:38 | |
| dolphm | rderose: what's wrong with timestamp in postgres? | 14:39 |
| bknudson | does openstack say what the min version of mysql is? | 14:39 |
| bknudson | I assume you can't use 1.0 | 14:40 |
| dolphm | bknudson: packagers do | 14:40 |
| *** ddieterly[away] is now known as ddieterly | 14:40 | |
| *** ravelar has joined #openstack-keystone | 14:40 | |
| dstanek | Michaellaneous: sounds like you need to start over then. almost seems like you mysql installation is somehow messed up | 14:40 |
| Michaellaneous | yeh | 14:40 |
| rderose | dolphm: I don't know zzzeek mentioned in that in yesterday's chat, "for POstgresql , sqlite others you need to stick with the DateTime + server_default=func.now()" | 14:40 |
| *** su_zhang has quit IRC | 14:41 | |
| dolphm | rderose: so, it's easy to check the engine and set a different server default | 14:41 |
| *** su_zhang has joined #openstack-keystone | 14:41 | |
| rderose | dolphm: right, but different column type | 14:41 |
| rderose | ? | 14:41 |
| dolphm | rderose: besides datetime? | 14:42 |
| rderose | dolphm: right, mysql timestamp and postgres would be datetime | 14:42 |
| dolphm | rderose: i'm just suggesting you at least vary the default across engines, not the column type | 14:43 |
| *** tonytan4ever has quit IRC | 14:43 | |
| dolphm | rderose: so all of mysql receives the terrible default, but postgres is correct | 14:43 |
| Michaellaneous | wait | 14:43 |
| Michaellaneous | I deleted keystone | 14:43 |
| Michaellaneous | started over | 14:43 |
| rderose | dolphm: but I would have to vary the column type for it to work | 14:43 |
| *** tonytan4ever has joined #openstack-keystone | 14:43 | |
| Michaellaneous | it still gives me error that I am not authorized | 14:43 |
| dolphm | rderose: the next step would be identify the server's mysql version, and then only setting the terrible default in 5.5 | 14:43 |
| rderose | dolphm: or, given the complexity, hardcoding an arbitrary date :) | 14:44 |
| rderose | dolphm: as this doesn't have any side effects, since I'm updating the column values | 14:44 |
| rderose | dolphm: I understand the issue though, but given everything, this seems like an acceptable solution | 14:45 |
| dstanek | Michaellaneous: did you setup keystone already? | 14:47 |
| rderose | dolphm: because even if I identify the mysql version, I'm still forced to change the column type for that version, right? | 14:47 |
| Michaellaneous | I started from here: http://docs.openstack.org/mitaka/install-guide-ubuntu/keystone-install.html | 14:48 |
| dolphm | rderose: i guarantee we're going to get a bug report for password created in the past - when it's something other than epoch | 14:48 |
| dstanek | rderose: are you still struggling with a default date? | 14:48 |
| dstanek | Michaellaneous: what step did you get to? | 14:48 |
| rderose | dstanek: welcome to my wordl | 14:49 |
| rderose | *world | 14:49 |
| *** tonytan_brb has joined #openstack-keystone | 14:49 | |
| dolphm | rderose: when people run this migration 15 months from now, how will that arbitrary date choice by impacted by other PCI features? | 14:49 |
| dstanek | rderose: :-) it's weird because i have done it in the past. what is the problem? it just doesn't work? | 14:49 |
| rderose | dolphm: no way, it's being set in the model and a default is defined in the model | 14:49 |
| Michaellaneous | Next site openstack service create \ | 14:49 |
| Michaellaneous | --name keystone --description "OpenStack Identity" identity | 14:49 |
| rderose | dolphm: how a about I guarantee we won't get a bug report | 14:50 |
| rderose | ? | 14:50 |
| rderose | dolphm: I'm updating the created_at value | 14:50 |
| rderose | dolphm: see the update statement after making it not nullable | 14:50 |
| *** tonytan4ever has quit IRC | 14:50 | |
| rderose | dolphm: so when you run the migration, I overwrite the arbitrary default value | 14:51 |
| dstanek | Michaellaneous: are you using the ADMIN_TOKEN or did you create a real user? | 14:51 |
| Michaellaneous | admin token | 14:51 |
| dolphm | rderose: then why choose an non-obvious arbitrary date if it's going to be discarded? | 14:52 |
| dstanek | does you OS_TOKEN environment value match what you have in the keystone.conf? | 14:52 |
| rderose | dolphm: so that I can make the column not nullable | 14:52 |
| Michaellaneous | Yep. | 14:52 |
| *** markvoelker has joined #openstack-keystone | 14:52 | |
| rderose | dolphm: otherwise, I can't | 14:52 |
| dolphm | rderose: i'm asking about the specific date | 14:53 |
| dolphm | rderose: why choose something non-obviously arbitrary? why not epoch? why not the date the migration is actually run? | 14:53 |
| rderose | dolphm: oh | 14:53 |
| bknudson | http://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html#uwsgidecorators.postfork -- "uWSGI is a preforking (or “fork-abusing”) server" | 14:53 |
| dolphm | rderose: why not anything else other than a hardcoded value that will result in bug reports? | 14:53 |
| bknudson | that's a funny thing to say | 14:53 |
| rderose | dolphm: tried to do that, passing in a date, it didn't work | 14:53 |
| dolphm | rderose: you're just passing a string | 14:54 |
| dolphm | rderose: you can pass any string | 14:54 |
| rderose | dolphm: tell me about it | 14:54 |
| rderose | "'" + now + "'" | 14:54 |
| rderose | didn't work | 14:54 |
| rderose | dolphm: tried server_default = now | 14:54 |
| Michaellaneous | dstanek, I got it. In a later config you disabled auth token | 14:55 |
| bknudson | uwsgi - "If this behavior is undesirable for some reason, use the lazy-apps option." | 14:55 |
| bknudson | I'll try this dstanek ^ | 14:55 |
| *** afred312 has joined #openstack-keystone | 14:55 | |
| bknudson | http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html | 14:55 |
| rderose | dstanek: short story, server_default doesn't work with mysql 5.5 or older | 14:55 |
| rderose | dstanek: for default datetime values | 14:56 |
| dstanek | rderose: so are you going to manually alter columns? | 14:57 |
| dstanek | bknudson: yeah, that's what i was talking about yesterday | 14:57 |
| dstanek | bknudson: that fixes all fds. | 14:57 |
| bknudson | oh, I didn't catch that. | 14:57 |
| rderose | dstanek: hardcoded in a date value as the server_default | 14:57 |
| bknudson | oh, I thought that was a change you made in your code to make things lazy-load. | 14:58 |
| bknudson | which is essentially what happened to fix the memcache issue | 14:58 |
| bknudson | http://uwsgi-docs.readthedocs.io/en/latest/Options.html#lazy-apps | 14:58 |
| bknudson | there's also options "cheap" and "cheaper" | 14:59 |
| bknudson | (uwsgi has lots of options) | 14:59 |
| *** david-lyle_ has quit IRC | 14:59 | |
| rderose | zzzeek: what do you think of this option: https://review.openstack.org/#/c/362501/14/keystone/common/sql/migrate_repo/versions/105_add_password_date_columns.py | 15:02 |
| zzzeek | rderose: that's fine | 15:03 |
| rderose | dolphm: ^ | 15:03 |
| rderose | :) | 15:03 |
| zzzeek | rderose: well, problem, if an applciation later on tries to insert data | 15:03 |
| zzzeek | rderose: now you get a garbage date in there | 15:03 |
| rderose | zzzeek: the application layer sets this value and the model has a default defined | 15:04 |
| zzzeek | rderose: you have an UPDATE right there: stmt = password_table.update().values(created_at=now) why not just do an UPDATE on this column as well then set NOT NULL afterwards? | 15:04 |
| *** dmellado is now known as dmellado|mtg | 15:04 | |
| zzzeek | rderose: then you dont even need the server_default | 15:05 |
| zzzeek | rderose: FYI, when you do the ALTER and set NOT NULL with the default, it is doing the UPDATE in any case | 15:05 |
| zzzeek | you just dont see it | 15:05 |
| *** jamielennox is now known as jamielennox|away | 15:05 | |
| zzzeek | rderose: there shouldnt be a server_default here at all if the app is relied upon to set the date in all cases | 15:05 |
| *** slberger has joined #openstack-keystone | 15:06 | |
| zzzeek | rderose: otherwise, someday something will be broken and the symptom will be, "hardcoded august 29 date in all our DB rows" | 15:06 |
| rderose | zzzeek: right, but I can't set it as nullable due to rolling upgrades issue | 15:07 |
| zzzeek | rderose: 1. create column as "nullable=True" 2. UPDATE column with date 3. ALTER column set nullable=False | 15:07 |
| zzzeek | rderose: oh youre worried about a write in between the UPDATE and the ALTER? | 15:08 |
| rderose | zzzeek: during a rolling upgrade, is old code adds a record, I could have a null value, right? | 15:08 |
| rderose | * if old code... | 15:08 |
| *** jaosorior has quit IRC | 15:08 | |
| zzzeek | rderose: yes. but also, once you set this server default hardoded to august 29, then for the next N days, old code will be putting crap in there | 15:09 |
| dstanek | bknudson: yeah, we had to lazy load applications because we had database connections and other service connection being cloned and that cause all kinds of havoc | 15:09 |
| zzzeek | rderose: you must change the datatype to TIMESTAMP for MySQL 5.5 only | 15:09 |
| rderose | zzzeek: right, but then datetime for everything else | 15:10 |
| bknudson | dstanek: I will likely propose a doc change and devstack change to set lazy loading for keystone. | 15:10 |
| rderose | zzzeek: seems more complicated to do that | 15:10 |
| zzzeek | rderose: the datatype is actually called TIMESTAMP on Postgresql in any case | 15:10 |
| dstanek | bknudson: cool. add me as a review pls | 15:10 |
| bknudson | dstanek: I wonder if the openstack-ansible does uwsgi? | 15:11 |
| *** ddieterly is now known as ddieterly[away] | 15:11 | |
| *** ddieterly[away] is now known as ddieterly | 15:11 | |
| rderose | zzzeek: what about other versions of mysql, sqlite | 15:11 |
| rderose | ? | 15:11 |
| bknudson | or any of the myriad other deployers | 15:11 |
| zzzeek | rderose: if you guys dont mind applications shoving a garbage date in there until all the old applications have been turned off, that's fine, sounds like a recipe for very embarrassing bugs IMO | 15:11 |
| *** ddieterly is now known as ddieterly[away] | 15:11 | |
| dolphm | zzzeek: ++ | 15:12 |
| rderose | zzzeek: that's what Dolph said, although, I am updating that date after creating the column, so overwriting the values | 15:12 |
| zzzeek | rderose: changing datatype here not a big deal. just call it TIMESTAMP | 15:12 |
| zzzeek | rderose: derr this is a brand new column. it will work fine | 15:12 |
| dstanek | bknudson: they are just starting to | 15:12 |
| rderose | zzzeek: what about other versions of mysql and sqlite? | 15:13 |
| rderose | will timestamp work? | 15:13 |
| zzzeek | rderose: SQLite doesn't have "real" datatypes so it doesnt matter what you name it there, SQLAlchemy's TIMESTAMP datatype will write strings to it like the DateTime type does | 15:13 |
| dstanek | bknudson: https://review.openstack.org/#/c/357539/10/templates/keystone-uwsgi.ini.j2 | 15:14 |
| zzzeek | rderose: I'd keep the func.now() default. MySQL 5.5 w/ TIMESTAMP will accept it. the only caveat is that only one column on the table can have this default for mysql 5.5 | 15:14 |
| bknudson | dstanek: "lazy-apps = true" -- these guys know what they're doing! | 15:14 |
| rderose | zzzeek: okay, let me try that. thanks :) | 15:14 |
| *** tonytan4ever has joined #openstack-keystone | 15:15 | |
| dstanek | bknudson: i think that's common in Python land since apps typically do too much at import | 15:16 |
| bknudson | "add-header = Connection: close" -- that's going to lower your efficiency considerably. | 15:17 |
| dstanek | bknudson: only if you expect pipelined requests | 15:18 |
| *** tonytan_brb has quit IRC | 15:18 | |
| bknudson | uwsgi server doesn't handle http pipelining anyways. | 15:19 |
| dstanek | i wonder what the middleware behavior is | 15:19 |
| bknudson | pipelining has only been part of the http spec for 17 years. | 15:20 |
| *** chrisshattuck has joined #openstack-keystone | 15:20 | |
| *** ddieterly[away] is now known as ddieterly | 15:20 | |
| *** ksavich has quit IRC | 15:20 | |
| dstanek | bknudson: yep, it's too new | 15:20 |
| *** hockeynut has joined #openstack-keystone | 15:21 | |
| *** pcaruana has quit IRC | 15:22 | |
| *** rcernin has quit IRC | 15:22 | |
| *** su_zhang has quit IRC | 15:25 | |
| *** su_zhang has joined #openstack-keystone | 15:26 | |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Pre-cache new tokens https://review.openstack.org/309146 | 15:28 |
| *** esp has quit IRC | 15:29 | |
| rderose | zzzeek: so like this: https://review.openstack.org/#/c/362501/15/keystone/common/sql/migrate_repo/versions/105_add_password_date_columns.py | 15:29 |
| *** ksavich_ has joined #openstack-keystone | 15:30 | |
| *** su_zhang has quit IRC | 15:30 | |
| bknudson | browne was seeing this using apache... not sure if there's a similar option for lazy application loading there. | 15:30 |
| *** browne has joined #openstack-keystone | 15:31 | |
| dstanek | not sure. i've never used mod-wsgi for anything real before | 15:33 |
| *** dmellado|mtg is now known as dmellado | 15:33 | |
| *** code-R has quit IRC | 15:35 | |
| rderose | zzzeek: getting an error: (sqlite3.OperationalError) Cannot add a column with non-constant default [SQL: u'\nALTER TABLE password ADD created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL'] | 15:35 |
| rderose | zzzeek: will try to code around sqlite | 15:36 |
| *** tonytan_brb has joined #openstack-keystone | 15:36 | |
| zzzeek | rderose: yep | 15:36 |
| bknudson | browne can switch to uwsgi | 15:36 |
| *** tesseract- has quit IRC | 15:36 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 15:36 |
| *** dikonoor has quit IRC | 15:37 | |
| browne | bknudson: so that fix won't help me if i'm using eventlet still? | 15:37 |
| *** tonytan4ever has quit IRC | 15:38 | |
| bknudson | browne: the commit should fix the problem for now. I'm worried that something will come along later that will break it again. | 15:38 |
| *** ccneill has joined #openstack-keystone | 15:39 | |
| browne | bknudson: ok cool. i'll give it a shot here soon. just need an environment again | 15:39 |
| Michaellaneous | dstanek, I got everything to work agian | 15:40 |
| Michaellaneous | Thank you. | 15:40 |
| *** sdake_ is now known as sdake | 15:40 | |
| dstanek | Michaellaneous: np. did you see when you went wrong the first time? | 15:40 |
| Michaellaneous | not really | 15:43 |
| Michaellaneous | i mean it worked before | 15:43 |
| Michaellaneous | then someone told me to delete the default domain and do it with bootstrap instead | 15:44 |
| Michaellaneous | that is when everything went downholl | 15:44 |
| Michaellaneous | because stupid LDAP won't work | 15:44 |
| Michaellaneous | Does anybody here have a working LDAP configuration for openstack? | 15:44 |
| browne | Michaellaneous: yes | 15:45 |
| browne | but ldap settings are very specific for each server | 15:45 |
| Michaellaneous | yeah i realize that | 15:45 |
| Michaellaneous | could you still give me your confs | 15:45 |
| Michaellaneous | I guess keystone | 15:45 |
| *** code-R has joined #openstack-keystone | 15:46 | |
| browne | are you using multi-domain backend? | 15:46 |
| *** EinstCrazy has joined #openstack-keystone | 15:46 | |
| *** EinstCrazy has quit IRC | 15:48 | |
| dstanek | Michaellaneous: once you already have a default domain there is no reason to recreate it. 'keystone-manage bootstrap' is the new way to do it, but for now the old way still works | 15:48 |
| *** markvoelker has quit IRC | 15:48 | |
| browne | well, there are caveats with the default domain | 15:49 |
| browne | only keystone-manage boostrap will create it with an ID of "default" | 15:49 |
| browne | otherwise you need to edit keystone.conf and add the generated UUID for that domain and restart keystone | 15:49 |
| dstanek | Michaellaneous: Michaellaneous here's how devstack configures it: https://github.com/openstack-dev/devstack/blob/master/lib/keystone#L224 | 15:50 |
| lbragstad | dolphm dstanek stevemar https://review.openstack.org/#/c/355618/ passed with switching blob to be nullable in the expand step | 15:50 |
| dstanek | Michaellaneous: here's some ldif stuff from their setup https://github.com/openstack-dev/devstack/tree/master/files/ldap | 15:50 |
| *** michauds has quit IRC | 15:51 | |
| dstanek | my successful ldap deployments are limited to having devstack set it up for me | 15:51 |
| *** rcernin has joined #openstack-keystone | 15:54 | |
| Michaellaneous | hmmmm | 15:56 |
| Michaellaneous | what does the manager ldi do? | 15:56 |
| *** diltram has joined #openstack-keystone | 15:56 | |
| diltram | hey, is there any documentantion how to implement keystone sessions in new projects? | 15:57 |
| *** sheel has joined #openstack-keystone | 15:57 | |
| diltram | I'm working on Octavia and we're trying to add that in two ways, on frontend apis and also on backend because we're using Service VM and we need to communicate with nova/neutron/barbican | 15:57 |
| diltram | I used all class specified here - http://docs.openstack.org/developer/keystoneauth/migrating.html#authentication-plugins | 15:58 |
| ayoung | stevemar, so, we are not very nice to people. The Keystone page gives no indication that you need to use the openstack CLI to do anything. http://docs.openstack.org/developer/keystone/ | 15:58 |
| diltram | and when I'm creating new Neutron/Nova client using that session I'm getting that I need to specify url endpoint when I'm not using auth | 15:59 |
| diltram | plugin | 15:59 |
| ayoung | we need a link that says " here is how you do things to keystone" that points to http://docs.openstack.org/cli-reference/openstack.html | 15:59 |
| henrynash | did ATC codes get sent out already for Barcelona? | 15:59 |
| dstanek | henrynash: a while ago | 15:59 |
| diltram | henrynash: yes | 15:59 |
| dstanek | i got mine maybe a month ago | 15:59 |
| henrynash | dstanek: that's what I thought... | 15:59 |
| ayoung | AAAAAH ! and not even internal anchorts | 15:59 |
| henrynash | dstanek: i seem to have lost my email....ok, I';; go hunting | 16:00 |
| henrynash | dstanek: do you remember who the From address was? | 16:00 |
| *** gyee has joined #openstack-keystone | 16:00 | |
| dstanek | fungi i think | 16:02 |
| henrynash | dstanek: ok, got it, thx | 16:03 |
| fungi | should show up as From: Jeremy Stanley <jeremy@openstack.org> (that's my work e-mail, i pretty much only use it for sending summit discount codes and security advisories) | 16:03 |
| *** edtubill has joined #openstack-keystone | 16:04 | |
| *** michauds has joined #openstack-keystone | 16:04 | |
| fungi | i'm not generally a fan of using mailboxes on servers i don't run ;) | 16:04 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 16:07 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable https://review.openstack.org/362501 | 16:08 |
| *** links has joined #openstack-keystone | 16:11 | |
| dstanek | fungi: that's pretty hardcore | 16:11 |
| fungi | you should see my tinfoil hat collection | 16:12 |
| *** asettle has quit IRC | 16:12 | |
| *** ianw has quit IRC | 16:13 | |
| *** pauloewerton has quit IRC | 16:13 | |
| *** asettle has joined #openstack-keystone | 16:13 | |
| *** chrichip has quit IRC | 16:16 | |
| *** asettle has quit IRC | 16:17 | |
| *** chrichip has joined #openstack-keystone | 16:17 | |
| stevemar | ayoung: :( i recall we had some osc commands there | 16:18 |
| ayoung | stevemar, is that generated from git keystone/docs? | 16:18 |
| stevemar | ayoung: we have http://docs.openstack.org/developer/keystone/configuration.html#example-usage | 16:19 |
| stevemar | and http://docs.openstack.org/developer/keystone/configuringservices.html#setting-up-services | 16:19 |
| ayoung | stevemar, we need to bump that up to its own page, I think | 16:20 |
| ayoung | too buried | 16:20 |
| ayoung | stevemar, if you are looking for help about how to do things with a remote keystone server, it is kind of frustrating to only see things about how to manage your own. Which do you think is the dominant use case? | 16:23 |
| *** eandersson has quit IRC | 16:25 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 16:27 |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Fixes migration where password created_at is nullable before 105 fix https://review.openstack.org/362510 | 16:27 |
| *** roxanaghe has joined #openstack-keystone | 16:28 | |
| *** ddieterly is now known as ddieterly[away] | 16:28 | |
| Michaellaneous | so if I included my openstack with ldap | 16:32 |
| Michaellaneous | how...do I configure roles and users with it | 16:33 |
| *** ddieterly[away] is now known as ddieterly | 16:34 | |
| *** esp has joined #openstack-keystone | 16:36 | |
| stevemar | ayoung: the whole "configuration" page needs to be more prominent | 16:38 |
| ayoung | stevemar, worth a docs discussion at the summit, with that as one topic | 16:38 |
| *** aswadr_ has joined #openstack-keystone | 16:39 | |
| stevemar | ayoung: wrote it down to remind myself in a few weeks | 16:39 |
| stevemar | amakarov: still around? | 16:42 |
| amakarov | stevemar, o/ | 16:42 |
| stevemar | amakarov: can you create a release note for the work, then i'm +2 :) | 16:42 |
| *** ddieterly is now known as ddieterly[away] | 16:42 | |
| amakarov | stevemar, ack, will do | 16:42 |
| *** ianw has joined #openstack-keystone | 16:43 | |
| *** ddieterly[away] is now known as ddieterly | 16:49 | |
| *** daemontool has joined #openstack-keystone | 16:50 | |
| *** ddieterly is now known as ddieterly[away] | 16:52 | |
| *** roxanaghe has quit IRC | 16:52 | |
| *** rreimberg has joined #openstack-keystone | 16:53 | |
| *** roxanaghe has joined #openstack-keystone | 16:54 | |
| amakarov | stevemar, are there release notes for newton? I don't see the folder in the source tree | 17:00 |
| stevemar | amakarov: they all get dumped together under releasenotes/notes | 17:01 |
| stevemar | amakarov: magic happens based on git merges, and the site knows how to display them http://docs.openstack.org/releasenotes/keystone/ | 17:02 |
| stevemar | amakarov: you can look at https://github.com/openstack/keystone/blob/master/releasenotes/notes/mapping_populate-521d92445505b8a3.yaml as an example | 17:03 |
| *** code-R_ has joined #openstack-keystone | 17:06 | |
| *** pauloewerton has joined #openstack-keystone | 17:06 | |
| *** spzala has quit IRC | 17:09 | |
| *** code-R has quit IRC | 17:09 | |
| amakarov | stevemar, should I add prelude part? | 17:09 |
| stevemar | amakarov: sure, "Tokens can now be cached when issued" | 17:10 |
| stevemar | amakarov: "prelude" and "new feature" | 17:10 |
| *** ksavich_ has quit IRC | 17:11 | |
| stevemar | breton: were you going to backport the caching fix? | 17:12 |
| stevemar | breton: i'm eager to get mfisch to test it out | 17:12 |
| *** ddieterly[away] is now known as ddieterly | 17:13 | |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Pre-cache new tokens https://review.openstack.org/309146 | 17:14 |
| amakarov | stevemar, ^ | 17:15 |
| breton | stevemar: in progress | 17:15 |
| *** su_zhang has joined #openstack-keystone | 17:15 | |
| *** markvoelker has joined #openstack-keystone | 17:15 | |
| *** jpena is now known as jpena|off | 17:17 | |
| *** code-R_ has quit IRC | 17:21 | |
| *** itisha has joined #openstack-keystone | 17:22 | |
| *** ddieterly is now known as ddieterly[away] | 17:22 | |
| *** su_zhang has quit IRC | 17:23 | |
| *** su_zhang has joined #openstack-keystone | 17:23 | |
| *** su_zhang has quit IRC | 17:24 | |
| stevemar | amakarov: thx! | 17:24 |
| stevemar | breton: thx to you too :) | 17:24 |
| *** joerch has quit IRC | 17:24 | |
| stevemar | dolphm: want to take a final look at https://review.openstack.org/#/c/309146/21 | 17:24 |
| *** su_zhang has joined #openstack-keystone | 17:25 | |
| *** ddieterly[away] is now known as ddieterly | 17:27 | |
| *** bradjones has quit IRC | 17:38 | |
| *** bradjones has joined #openstack-keystone | 17:38 | |
| *** tonytan_brb has quit IRC | 17:38 | |
| *** pcaruana has joined #openstack-keystone | 17:39 | |
| *** bradjones is now known as Guest30374 | 17:39 | |
| *** ddieterly is now known as ddieterly[away] | 17:43 | |
| *** markvoelker has quit IRC | 17:43 | |
| dolphm | stevemar: why does it need to default to off? keystone.conf is supposed to represent our production-friendly defaults, and disabling caching is not production friendly at all | 17:45 |
| dolphm | i don't even understand why it should be optional | 17:45 |
| stevemar | dolphm: both dstanek and bknudson seem a bit skeptical about the invalidation it can do, i'm okay with flipping it to default=True in one release | 17:50 |
| lbragstad | i'm breaking for lunch quick but when I get back I'm going to finish up some documentation on the rotation process for credential keys | 17:51 |
| dolphm | stevemar: what is the benefit of keeping it as false in the mean time? what do we expect to gain? | 17:51 |
| stevemar | dolphm: stability, it'll continue to work as expected | 17:52 |
| dolphm | stevemar: i expect it to work faster | 17:52 |
| stevemar | bknudson: dstanek want to chime in here? i may not be speaking for you both correctly enough | 17:54 |
| rderose | stevemar: anything you want me to do for: https://review.openstack.org/#/c/358111/ | 17:56 |
| rderose | stevemar: or, are you still digging into this one? | 17:56 |
| stevemar | rderose: still digging | 18:00 |
| stevemar | got distracted | 18:00 |
| rderose | stevemar: okay, cool | 18:00 |
| stevemar | rderose: should be good though | 18:00 |
| rderose | stevemar: alright | 18:00 |
| breton | stevemar: dstanek: why does the patch combine both key_mangler and invalidation strategy? | 18:00 |
| bknudson | stevemar: dolphm I haven't been able to do any perf testing on it and without that I wouldn't want it enabled. | 18:01 |
| breton | stevemar: dstanek: oh i got it, nevermind | 18:02 |
| breton | stevemar: dstanek: so... do we expect dogpile.cache to be 0.6.2 in stable/mitaka? | 18:02 |
| stevemar | breton: oh... right | 18:03 |
| breton | stevemar: dstanek: i thought relying on a new release of a lib is no-no | 18:03 |
| bknudson | we can't rely on a new lib being available on stable | 18:03 |
| breton | *for backporting | 18:03 |
| stevemar | breton: for mitaka we're capped at 0.5.7 | 18:03 |
| stevemar | https://github.com/openstack/requirements/blob/stable/mitaka/upper-constraints.txt | 18:03 |
| stevemar | dng | 18:03 |
| * breton sighs | 18:04 | |
| breton | maybe we could take an old version of dstanek's patch, the one that doesn't use strategy | 18:04 |
| *** spzala has joined #openstack-keystone | 18:05 | |
| *** chrisshattuck has quit IRC | 18:05 | |
| *** thumpba has joined #openstack-keystone | 18:05 | |
| stevemar | breton: well that stinks | 18:06 |
| *** su_zhang has quit IRC | 18:07 | |
| *** xenogear has joined #openstack-keystone | 18:12 | |
| *** gagehugo_ has quit IRC | 18:13 | |
| *** nk2527 has quit IRC | 18:13 | |
| *** chrisshattuck has joined #openstack-keystone | 18:14 | |
| *** xenogear has quit IRC | 18:17 | |
| *** ccneill has quit IRC | 18:17 | |
| *** tonytan4ever has joined #openstack-keystone | 18:18 | |
| *** spedione is now known as chris_hultin | 18:22 | |
| breton | dstanek: are any of your old patches, that monkeypatched Region, good enough? | 18:22 |
| *** ddieterly[away] is now known as ddieterly | 18:24 | |
| dstanek | breton: maybe? i think it would be pretty easy to convert that to be a subclass of Region again without going back | 18:24 |
| dstanek | breton: the monkey patching shouldn't be necessary since i changed everthing in keystone to use the new create_region | 18:25 |
| breton | dstanek: i don't understand about create_region. How does it prevent us from monkeypatching? | 18:26 |
| *** ccneill has joined #openstack-keystone | 18:26 | |
| dstanek | breton: it doesn't prevent it. i just think we can get by without it since we can return our own region | 18:27 |
| dstanek | breton: but if it's easier to monkey patch, i'm fine with it. | 18:27 |
| breton | dstanek: oh, ok, i got it. | 18:28 |
| *** jaugustine has joined #openstack-keystone | 18:32 | |
| Michaellaneous | do I have to enable LDAP before I do all that user/domain/roles shit? | 18:35 |
| *** xenogear has joined #openstack-keystone | 18:40 | |
| *** ddieterly is now known as ddieterly[away] | 18:41 | |
| *** gagehugo has joined #openstack-keystone | 18:44 | |
| *** ddieterly[away] is now known as ddieterly | 18:45 | |
| *** ddieterly is now known as ddieterly[away] | 18:46 | |
| *** esp has quit IRC | 18:46 | |
| stevemar | Michaellaneous: enabling ldap can be your last step | 18:46 |
| Michaellaneous | okay | 18:46 |
| Michaellaneous | that makes little sense to me | 18:46 |
| Michaellaneous | how am I gonna create users/roles then? | 18:46 |
| stevemar | Michaellaneous: you create users/roles and all that for your default domain, when that's all setup and working you can create another domain (foo), and associate it with an ldap connection | 18:48 |
| Michaellaneous | ohhhhhh | 18:48 |
| Michaellaneous | and I use the admin of the default domain | 18:48 |
| Michaellaneous | to add users to the ldap domain | 18:48 |
| *** thumpba has quit IRC | 18:49 | |
| *** thumpba has joined #openstack-keystone | 18:49 | |
| stevemar | Michaellaneous: slightly... you use the admin of the default domain, think of him as the overall cloud admin, to create projects that the ldap users can use. | 18:49 |
| Michaellaneous | okay | 18:49 |
| stevemar | Michaellaneous: the ldap users should be able to just log in | 18:49 |
| stevemar | no need to "create" them, we strictly read from ldap | 18:49 |
| Michaellaneous | but I still have to create roles for the ldap stuff. | 18:49 |
| stevemar | right | 18:49 |
| Michaellaneous | yeah and there is why I fall falt | 18:49 |
| Michaellaneous | not sure how to associate an existing ldap user | 18:50 |
| Michaellaneous | with...rights, and projects. | 18:50 |
| stevemar | Michaellaneous: i've got a blog post written up on this stuff, let me shoot it over to you | 18:50 |
| Michaellaneous | that would be amazing | 18:50 |
| stevemar | Michaellaneous: https://developer.ibm.com/opentech/2015/08/14/configuring-keystone-with-ibms-bluepages-ldap/ -- except i don't setup the default bits | 18:50 |
| stevemar | Michaellaneous: maybe try whipping this all up with a devstack instance first? that would save you some of the pain you had earlier in trying to salvage stuff | 18:51 |
| Michaellaneous | oh it works all now again | 18:51 |
| *** amakarov is now known as amakarov_away | 18:51 | |
| Michaellaneous | i use a 6 server setup | 18:51 |
| Michaellaneous | and this has to be running for in a few days | 18:51 |
| Michaellaneous | so if I start again with DevStack and something goes wrong? | 18:51 |
| Michaellaneous | can devstack even do multiple serverS? | 18:51 |
| stevemar | Michaellaneous: not easily, i think it's possible but i've never tried | 18:52 |
| Michaellaneous | yeah see | 18:52 |
| Michaellaneous | I have like 5 high powered servers | 18:52 |
| stevemar | if you've got the default stuff working then stick with that | 18:52 |
| Michaellaneous | LDAP is a project requirement | 18:52 |
| Michaellaneous | my chef wants it | 18:52 |
| Michaellaneous | because this is used with students | 18:52 |
| Michaellaneous | and creating multiple accounts for each service we have running is a pain | 18:52 |
| *** david-lyle_ has joined #openstack-keystone | 18:52 | |
| *** david-lyle_ has quit IRC | 18:52 | |
| stevemar | Michaellaneous: most folks create the service accounts, like admin/nova/etc, in sql, its just a one time setup | 18:53 |
| *** nk2527 has joined #openstack-keystone | 18:53 | |
| Michaellaneous | Yeah I have done that | 18:53 |
| Michaellaneous | I just want user accounts on ldap | 18:53 |
| Michaellaneous | The service can stay like that. | 18:53 |
| Michaellaneous | So step one is to create a new domain | 18:53 |
| stevemar | Michaellaneous: yep | 18:55 |
| Michaellaneous | Do I need a domain specific config for my default doman? | 18:55 |
| stevemar | i updated the page again | 18:56 |
| stevemar | Michaellaneous: no, just for non-default ones | 18:56 |
| *** sheel has quit IRC | 18:56 | |
| Michaellaneous | do you have phpldap? | 18:57 |
| *** ddieterly[away] is now known as ddieterly | 18:59 | |
| Michaellaneous | stevemar, just so I can compare the values with your ldap strcutre | 19:00 |
| breton | dstanek: why did you redefine get, set, get_multi etc in https://review.openstack.org/#/c/349704/10/keystone/common/cache/core.py ? | 19:00 |
| *** esp has joined #openstack-keystone | 19:01 | |
| stevemar | Michaellaneous: i do not | 19:04 |
| Michaellaneous | damn | 19:04 |
| stevemar | i used jxplorer | 19:04 |
| Michaellaneous | lemme try to figure this out and I'll come back to you in a minute | 19:05 |
| Michaellaneous | I don't have ibmPerson | 19:06 |
| Michaellaneous | obviously. what else could I use | 19:06 |
| Michaellaneous | inetOrgPerson? | 19:06 |
| stevemar | inetOrgPerson ? | 19:07 |
| Michaellaneous | https://i.imgur.com/Lb5t7JW.png | 19:07 |
| stevemar | looks like it | 19:08 |
| Michaellaneous | damn I don't have a uid tho | 19:08 |
| Michaellaneous | ill just use the date created as ID | 19:10 |
| dstanek | breton: you mean in my older reviews? | 19:11 |
| *** links has quit IRC | 19:13 | |
| Michaellaneous | stevemar, http://pastebin.com/8BGyJruK | 19:14 |
| Michaellaneous | https://i.imgur.com/FmSRH4J.png | 19:14 |
| Michaellaneous | https://i.imgur.com/37mR4VP.png | 19:14 |
| dstanek | breton: i think all you need to do is make RegionInvalidationManager as subclass of CacheRegion and maybe rename to CacheRegion like i had in older reviews | 19:15 |
| dstanek | breton: then rename the invalidate_method to invalidate | 19:15 |
| dstanek | breton: and for bonus point rename the vars used to hold a reference to the region (currently name *invalidation_manager) | 19:16 |
| *** ddieterly is now known as ddieterly[away] | 19:16 | |
| Michaellaneous | stevemar, I followed your guide, and it seems...everything is working | 19:20 |
| *** ddieterly[away] is now known as ddieterly | 19:20 | |
| Michaellaneous | Except I can't find the user | 19:20 |
| *** daemontool has quit IRC | 19:23 | |
| *** gyee has quit IRC | 19:29 | |
| stevemar | Michaellaneous: hmm, maybe one of the ldap query/scope options is bad | 19:30 |
| Michaellaneous | in the apache2 keystone log | 19:30 |
| Michaellaneous | I always get a warning that domaind efalt cold not be found | 19:30 |
| Michaellaneous | but...I can still run commands | 19:30 |
| *** aswadr_ has quit IRC | 19:31 | |
| Michaellaneous | also, looking at that log it still seems to try and get the user from keystone | 19:32 |
| Michaellaneous | as opposed to ldap | 19:32 |
| *** sdake_ has joined #openstack-keystone | 19:32 | |
| *** sdake has quit IRC | 19:33 | |
| *** sdake has joined #openstack-keystone | 19:35 | |
| *** sdake_ has quit IRC | 19:36 | |
| stevemar | Michaellaneous: sorry, was in a meeting - how are things now? | 19:42 |
| stevemar | lbragstad: will you merge dolphm's changes from https://review.openstack.org/#/c/360723/ into your base implementation? | 19:43 |
| Michaellaneous | give me a second | 19:43 |
| Michaellaneous | ah | 19:45 |
| Michaellaneous | ahahhhhh | 19:45 |
| Michaellaneous | it works | 19:45 |
| *** ddieterly has quit IRC | 19:45 | |
| Michaellaneous | root@wildcat:~# openstack user show testuser --domain ldapdomain | 19:46 |
| Michaellaneous | +-----------+------------------------------------------------------------------+ | 19:46 |
| Michaellaneous | | Field | Value | | 19:46 |
| Michaellaneous | +-----------+------------------------------------------------------------------+ | 19:46 |
| Michaellaneous | | domain_id | c8a0996b5f4b4d299a1aa8698aff68cc | | 19:46 |
| Michaellaneous | | id | 06e5d6e104789f805f5161609f99a982e39b63c5b885e6c5366c7e96b595e0d9 | | 19:46 |
| Michaellaneous | | name | testuser | | 19:46 |
| Michaellaneous | +-----------+------------------------------------------------------------------+ | 19:46 |
| Michaellaneous | yeah boiii | 19:46 |
| stevemar | Michaellaneous: so you did that as your cloud admin correct? | 19:47 |
| Michaellaneous | yes | 19:47 |
| stevemar | Michaellaneous: nice | 19:47 |
| Michaellaneous | normal admin openrc file | 19:47 |
| stevemar | now you can try to authenticate as that user :) | 19:47 |
| Michaellaneous | hold on, gotta fix groups first | 19:47 |
| stevemar | Michaellaneous: a minimal set of options should be ... $ openstack token issue --os-username testuser --os-password MYPASS --os-user-domain-name ldapdomain --os-auth-url YOUR_URL --os-identity-api-version 3 | 19:48 |
| stevemar | this is assuming you are using a newish openstackclient | 19:48 |
| stevemar | ~2.6.0 | 19:48 |
| Michaellaneous | well, I still have issues with groups | 19:48 |
| stevemar | womp womp :( | 19:49 |
| Michaellaneous | nope | 19:49 |
| Michaellaneous | it was my stupidity | 19:49 |
| Michaellaneous | all fixed | 19:49 |
| stevemar | dolphm: does the config option in precaching make it non +2 able? | 19:49 |
| stevemar | Michaellaneous: nice | 19:49 |
| stevemar | dolphm: cause i'm either upsetting you or bknudson :) | 19:50 |
| stevemar | can't please everyone :( | 19:50 |
| bknudson | they could also post performance results | 19:50 |
| dolphm | bknudson: steve did, i believe amakarov_away did, and i was hoping to see lbragstad's results myself | 19:51 |
| bknudson | also, my complaint isn't about the config option. I asked that the code be refactored so that it works correctly first. | 19:51 |
| Michaellaneous | stevemar, one thing is strange tho | 19:52 |
| Michaellaneous | when I say "what groups is this user in" | 19:52 |
| Michaellaneous | it correctly tells me "students" | 19:52 |
| Michaellaneous | but when I do it the other way around | 19:52 |
| Michaellaneous | i dont get any answer | 19:52 |
| Michaellaneous | I think it has to do with memberattribute | 19:52 |
| stevemar | 18:38:55 <bknudson> what if there was a config option to enable pre-caching of tokens? | 19:52 |
| bknudson | if people are seeing better performance then I would be fine with this. | 19:52 |
| bknudson | I'm fine with there being no config option | 19:53 |
| stevemar | *tosses up hands in frustration* | 19:53 |
| bknudson | I'm not fine with the complexity of the implementation | 19:53 |
| stevemar | Michaellaneous: sounds like it, if it's working for most other things it's probably a setting you're missing | 19:53 |
| stevemar | or have set incorrectly | 19:53 |
| bknudson | stevemar: I made that comment because I didn't think there was any proof that the performance was improved | 19:53 |
| lbragstad | i believe the performance bot tested it but we had to make changes since osa didn't enable caching | 19:54 |
| stevemar | bknudson: what is it about the implementation you do not like? | 19:54 |
| stevemar | scrap this, i'm going to approve it | 19:55 |
| stevemar | dolph, make you already smart ansible even smarter | 19:55 |
| stevemar | at this point we're being douches to amakarov_away | 19:55 |
| dstanek | breton: stevemar: looking at https://review.openstack.org/362785 - this check only happens if fernet is comfigured right? | 19:56 |
| *** code-R has joined #openstack-keystone | 19:57 | |
| bknudson | it's having to set for both self._validate_v3_token, self._validate_token, and self.validate_non_persistent_token | 19:57 |
| bknudson | there should only be one path to get a token so that they can all share the cache | 19:57 |
| bknudson | there's also a self._validate_v2_token | 19:58 |
| *** su_zhang has joined #openstack-keystone | 19:58 | |
| Michaellaneous | so | 19:59 |
| Michaellaneous | this is my group thingie | 20:00 |
| Michaellaneous | http://pastebin.com/0JgBZDBD | 20:00 |
| Michaellaneous | and as I said. I can find groups via users, but the other way around I get no output | 20:00 |
| Michaellaneous | the fact that it works one way but no the other | 20:00 |
| Michaellaneous | bothers me | 20:00 |
| Michaellaneous | and directly looking for the group works too | 20:01 |
| dstanek | Michaellaneous: have you debugged the queries to see what they are doing? | 20:02 |
| Michaellaneous | where can I do that? | 20:02 |
| Michaellaneous | i havent foudn the log for that | 20:02 |
| stevemar | yeah, the keystone server logs should have that info | 20:03 |
| dstanek | setting the debug setting in the config like we talked about yesterday | 20:03 |
| *** gyee has joined #openstack-keystone | 20:03 | |
| stevemar | Michaellaneous: depends on where, but /var/logs/apache2/keystone.log ? | 20:03 |
| *** tonytan4ever has quit IRC | 20:03 | |
| stevemar | also you'd need to be in debug mode | 20:03 |
| stevemar | Michaellaneous: but you'd be able to see the exact query it's trying to perform using ldapsearch | 20:03 |
| dstanek | [DEFAULT]/debug and [ldap]/debug_level i think | 20:03 |
| stevemar | Michaellaneous: with that, you can take that query and run it from a terminal, see where it's going wrong | 20:04 |
| *** chrisshattuck has quit IRC | 20:04 | |
| openstackgerrit | Chris Spencer proposed openstack/keystone: Add documentation on how to set a user's tenant. https://review.openstack.org/363292 | 20:04 |
| dstanek | bknudson: one path means there is no choice | 20:07 |
| *** chrisshattuck has joined #openstack-keystone | 20:07 | |
| *** sigmavirus is now known as sigmavirus|awa | 20:07 | |
| bknudson | dstanek: that is very zen | 20:08 |
| Michaellaneous | i cant make a lot of this debug output | 20:08 |
| bknudson | if v2 doesn't work, I can use v3 or validate_non_persistent_token instead! | 20:08 |
| Michaellaneous | http://pastebin.com/JE53gFvd | 20:09 |
| dstanek | you may just feel like it's a v2 kinda day | 20:09 |
| *** esp has quit IRC | 20:09 | |
| *** code-R_ has joined #openstack-keystone | 20:09 | |
| stevemar | Michaellaneous: whats the output of `openstack group list --user testuser --user-domain ldapdomain | 20:10 |
| *** chrissha_ has joined #openstack-keystone | 20:10 | |
| Michaellaneous | +------------------------------------------------------------------+-----------+ | 20:10 |
| Michaellaneous | | ID | Name | | 20:10 |
| Michaellaneous | +------------------------------------------------------------------+-----------+ | 20:10 |
| Michaellaneous | | b402b47b00f008e89e48cfb0f94aa91407c4baaafdd0c015335786f7bf253000 | testgroup | | 20:10 |
| Michaellaneous | +------------------------------------------------------------------+-----------+ | 20:10 |
| dstanek | Michaellaneous: that looks to be client output again | 20:10 |
| Michaellaneous | but I don't have a testgroup | 20:11 |
| Michaellaneous | in my default domain client setup | 20:11 |
| bknudson | dstanek: btw - changing to lazy-apps=true worked. | 20:11 |
| bknudson | I'll propose changes to keystone docs and devstack | 20:12 |
| *** code-R has quit IRC | 20:12 | |
| *** chrisshattuck has quit IRC | 20:12 | |
| stevemar | Michaellaneous: the groups are coming from ldap, not from your keystone default domain | 20:13 |
| Michaellaneous | yes | 20:13 |
| Michaellaneous | but thats good | 20:13 |
| stevemar | Michaellaneous: so... that's correct to me | 20:13 |
| Michaellaneous | isnt it? | 20:13 |
| Michaellaneous | i also tried login to dashboard | 20:13 |
| Michaellaneous | it says "Not part of any projects" | 20:13 |
| Michaellaneous | which is also good | 20:13 |
| Michaellaneous | i think | 20:13 |
| dstanek | bknudson: awesome. it also fixes bugs we didn't know we had | 20:13 |
| stevemar | Michaellaneous: did you grant a role on a project to either the ldap user or ldap group? | 20:13 |
| bknudson | dstanek: y, I wonder if opening sockets too early is a bug? (and if there would be any way to detect it for a test) | 20:14 |
| Michaellaneous | not yet | 20:14 |
| bknudson | dstanek: I guess it would save a little bit of memory to be able to lazy-load=false | 20:15 |
| bknudson | could fork a process that imports keystone and loads application and checks for open files | 20:17 |
| bknudson | but then it would depend on the configuration and whatever libs do | 20:18 |
| dstanek | bknudson: back in the day i put a lot of work into flup to deal with these issues. basic architecture was a parent process that we easy to preload with code and data, then children that really did the work | 20:18 |
| stevemar | Michaellaneous: okay, so even though the ldap user can login, they won't be able to do much until they have access to work on a project | 20:18 |
| dstanek | there was actually 2 levels or parents P0 -> P1 -> children | 20:18 |
| openstackgerrit | Chris Spencer proposed openstack/keystone: Add documentation on how to set a user's tenant. https://review.openstack.org/363292 | 20:19 |
| stevemar | Michaellaneous: you, as the cloud admin, can create them a project in the domain we created earlier, and grant individual users access or entire groups | 20:19 |
| bknudson | dstanek: that's a sensible setup | 20:19 |
| dstanek | you could HUP P0 to create a new P1 process with new code and as the number of children grew on the new process it would reap the old ones | 20:19 |
| dstanek | allow for a hotswap of code with no downtime at all | 20:19 |
| bknudson | I wonder if uwsgi's emperor mode allows that | 20:20 |
| Michaellaneous | stevemar, yeah just figuring out how :V | 20:20 |
| bknudson | there's probably 20 uwsgi options about it | 20:20 |
| bknudson | "set the Emperor tolerance about cursed vassals" ?? | 20:21 |
| bknudson | "put the Emperor in Tyrant mode" | 20:21 |
| bknudson | "run the emperor in BroodLord mode" | 20:21 |
| *** esp has joined #openstack-keystone | 20:21 | |
| dstanek | ++ for tyrant | 20:22 |
| bknudson | tyrants are better than broodlords? | 20:22 |
| stevemar | bknudson: no dictator mode option? | 20:22 |
| stevemar | what about overlord? | 20:23 |
| *** chrissha_ has quit IRC | 20:23 | |
| dstanek | --as-Trump? | 20:23 |
| bknudson | "set the maximum time (in seconds) a mule can take" | 20:23 |
| bknudson | "Zerg mode" | 20:24 |
| bknudson | uwsgi is pretty wacky | 20:25 |
| Michaellaneous | stevemar, works! | 20:25 |
| Michaellaneous | well, i still cant view users in group | 20:25 |
| Michaellaneous | but it works | 20:25 |
| *** hockeynut has quit IRC | 20:26 | |
| *** chrisshattuck has joined #openstack-keystone | 20:27 | |
| *** jdennis has quit IRC | 20:37 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Document credential encryption https://review.openstack.org/354497 | 20:37 |
| *** jdennis has joined #openstack-keystone | 20:38 | |
| lbragstad | in order for us to land the grenade change for encrypted credentials (https://review.openstack.org/#/c/362450/) , we need to have a merged release note | 20:41 |
| lbragstad | so I rebased the credential encryption doc patch on master so that it can merge before the implementation | 20:42 |
| lbragstad | since the implementation needs to have the grenade change merged first | 20:42 |
| lbragstad | doc patch is now here - https://review.openstack.org/#/c/354497/15 | 20:42 |
| openstackgerrit | Brant Knudson proposed openstack/keystone: Update sample uwsgi config for lazy-apps https://review.openstack.org/363929 | 20:44 |
| lbragstad | cc stevemar dolphm browne dstanek ^ | 20:45 |
| browne | lgtm | 20:45 |
| lbragstad | browne https://review.openstack.org/#/c/354497/15 | 20:46 |
| browne | lbragstad: thx, i'll rereview | 20:47 |
| mfisch | stevemar: yo here now | 20:48 |
| mfisch | sorry I missed you earlier | 20:48 |
| stevemar | mfisch: yo | 20:48 |
| stevemar | i just DMed ya stuff | 20:48 |
| mfisch | yep I see | 20:48 |
| stevemar | mfisch: lemme know if you have any q's | 20:49 |
| mfisch | ok so backport will be tough | 20:49 |
| mfisch | stevemar: back on battery power ;) | 20:50 |
| mfisch | stevemar: testing this one will not be easy since I need to upgrade all my policy files and driver to | 20:52 |
| mfisch | too | 20:52 |
| mfisch | who was the other person affected? was it browne ? | 20:53 |
| browne | mfisch: the caching issues? if so, yes | 20:54 |
| mfisch | yeah | 20:54 |
| browne | yeah, plan to try brant's fixes out once i get an environment up and running again | 20:54 |
| mfisch | what are those? | 20:55 |
| bknudson | I didn't fix anything. dstanek did it. | 20:55 |
| bknudson | I was able to verify locally | 20:55 |
| browne | oh, right my bad. dstanek | 20:55 |
| *** raildo has quit IRC | 20:56 | |
| lbragstad | dolphm do you think we should just collapse the entire credential encryption doc into configuration.rst? | 20:56 |
| mfisch | stan-bran same thig | 20:56 |
| bknudson | mfisch: do you use uwsgi? | 20:57 |
| *** slberger has quit IRC | 20:58 | |
| mfisch | no | 20:58 |
| mfisch | not anymore | 20:58 |
| mfisch | oh wait sorry yes, I read that backwards | 20:58 |
| bknudson | I'm getting an error in devstack : AttributeError: 'module' object has no attribute 'RegionInvalidationStrategy' | 20:59 |
| bknudson | oddly, from /usr/local/bin/keystone-manage --config-file /etc/keystone/keystone.conf db_sync | 20:59 |
| *** slberger has joined #openstack-keystone | 21:00 | |
| bknudson | oh, I need to update /opt/stack/requirements it keeps loading old libs | 21:00 |
| bknudson | it's strange that devstack overwrites upper-constraints.txt | 21:01 |
| *** tonytan4ever has joined #openstack-keystone | 21:04 | |
| *** ravelar has quit IRC | 21:06 | |
| *** ravelar has joined #openstack-keystone | 21:07 | |
| *** tonytan4ever has quit IRC | 21:09 | |
| *** ayoung has quit IRC | 21:10 | |
| *** ravelar has quit IRC | 21:11 | |
| *** code-R_ has quit IRC | 21:13 | |
| *** pauloewerton has quit IRC | 21:13 | |
| mfisch | browne: let me know your results wrt caching | 21:14 |
| bknudson | mfisch: did you set lazy-apps=true in the uwsgi config? | 21:15 |
| browne | mfisch: will do | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Document credential encryption https://review.openstack.org/354497 | 21:16 |
| dstanek | bknudson: mfisch: browne: may the caching be with you | 21:17 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Document credential encryption https://review.openstack.org/354497 | 21:18 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Document credential encryption https://review.openstack.org/354497 | 21:18 |
| mfisch | bknudson: looking | 21:19 |
| mfisch | bknudson: I dont think so, at least I dont see it | 21:19 |
| *** spzala has quit IRC | 21:21 | |
| lbragstad | dolphm thanks for the doc review - fixed in ^ | 21:21 |
| *** spzala has joined #openstack-keystone | 21:22 | |
| bknudson | according to https://bugs.launchpad.net/keystone/+bug/1537617 , this is fixed already, but devstack is disabling catalog caching ... I'm going to propose changing devstack config since the bug is fixed. | 21:22 |
| openstack | Launchpad bug 1537617 in OpenStack Identity (keystone) "caching of the catalog does not invalidate across processes" [High,Fix released] - Assigned to Morgan Fainberg (mdrnstm) | 21:22 |
| breton | dstanek: https://review.openstack.org/362785 -- right, only if fernet is configured. | 21:24 |
| dstanek | breton: i check and it looks like the repository is configured by default | 21:24 |
| breton | dstanek: configured where? | 21:25 |
| *** spzala has quit IRC | 21:26 | |
| dstanek | in keystone.conf.fernet_tokens | 21:28 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Document credential encryption https://review.openstack.org/354497 | 21:28 |
| breton | dstanek: the check that i remove verifies that the directory exists and not empty | 21:29 |
| breton | dstanek: i don't like it, because i am moving fernet keys to backends | 21:29 |
| dstanek | breton: i like that check early if we can do it | 21:30 |
| breton | dstanek: and in this case fernet_api is instantiated along with token_api | 21:30 |
| dstanek | i'm a believe in failing loud and early as opposed to failure *sometime* later | 21:31 |
| breton | dstanek: in apache it will fail on first request anyway. It will not prevent apache from starting. | 21:31 |
| *** slberger1 has joined #openstack-keystone | 21:31 | |
| *** slberger has quit IRC | 21:31 | |
| breton | dstanek: but i will be happy to hear an option how not to remove the check and have the keys in a backend | 21:32 |
| bknudson | doing the check depends on which backend is chosen, so checking should be a function of the backend. | 21:34 |
| *** thumpba has quit IRC | 21:41 | |
| breton | bknudson: yep. How do i create the keys via the backend then? | 21:44 |
| bknudson | what do you mean? | 21:45 |
| lbragstad | each backend would have to implement an interface | 21:45 |
| lbragstad | I would think? | 21:45 |
| lbragstad | the file based backend would go about creating keys like it does on disk. other backends might require something different | 21:46 |
| bknudson | I assume an sql backend would do an INSERT to put the key there | 21:47 |
| breton | initially there is no keys at all. I need to create them using a backend. To do it, i need to create an instance of fernet_api in load_backends(). load_backends() creates in instance of token_api. | 21:47 |
| *** chris_hultin is now known as spedione|AWAY | 21:47 | |
| lbragstad | bknudson yeah - and it would have to check the rows of the table in order to determine if the keys were valid for example | 21:47 |
| breton | so, in order to create the keys, i need to call call fernet_api.create_keys(). To get fernet_api, i need to call load_backends, which in turn creates token_api, which fails because there are no keys. | 21:50 |
| bknudson | seems like the key store would be a sub-backend of fernet? (like id_generator is a sub-backend of identity) | 21:50 |
| breton | bknudson: what about encryption keys? | 21:51 |
| breton | i was actually hoping to have this discussion at the summit and for now trying to hit all the issues there are :p | 21:51 |
| bknudson | encryption keys for what? the fernet keys? | 21:52 |
| breton | bknudson: encryption keys for credentials encryption | 21:52 |
| breton | bknudson: they are fernet keys too, right? | 21:53 |
| lbragstad | yes | 21:53 |
| bknudson | y, I haven't looked into that much... maybe it's a common backend and not related to the token provider. | 21:53 |
| lbragstad | somewhat common, but it is a separate backend | 21:53 |
| *** chrichip has quit IRC | 21:54 | |
| lbragstad | we don't make the credential api rely on the fernet token provider to encrypt credentials | 21:54 |
| bknudson | breton: please review the credential encryption stuff and make sure it's going to work for you! | 21:54 |
| breton | bknudson: will do | 21:54 |
| bknudson | lbragstad: don't both the credential api and token provider have a key store? | 21:54 |
| lbragstad | bknudson right now they are both stored on disk | 21:55 |
| *** chrichip has joined #openstack-keystone | 21:55 | |
| bknudson | lbragstad: is it the same store? | 21:55 |
| lbragstad | no | 21:55 |
| lbragstad | it can't be | 21:55 |
| lbragstad | fernet tokens are encrypted with different keys than credentials | 21:55 |
| bknudson | do you think you'd want to have tokens in sql and credentials on disk? | 21:56 |
| lbragstad | i would consider sharing the same key between the two providers a security vulnerability | 21:56 |
| bknudson | or vice-versa | 21:56 |
| lbragstad | I would probably prefer them on disk | 21:56 |
| bknudson | too bad | 21:57 |
| lbragstad | since the presence of a staged key doesn't require me to have them replicated using a backend | 21:57 |
| breton | i don't know what i want yet, maaaaybe it will be etcd. | 21:57 |
| lbragstad | also - my credentials key rotation policy might be less frequent than my token key rotation policy | 21:58 |
| *** chrisshattuck has quit IRC | 22:04 | |
| bknudson | are tokens somehow more secure than credentials? | 22:05 |
| rderose | stevemar dolphm: this one is ready: https://review.openstack.org/#/c/362501/ | 22:06 |
| rderose | stevemar dolphm: timestamp type worked like a charm :) | 22:06 |
| lbragstad | bknudson tokens are also cipher text | 22:06 |
| lbragstad | which is leaked to users | 22:06 |
| lbragstad | credential are encrypted and the cipher text is only available to keystone - it should never escape | 22:07 |
| lbragstad | so i would imagine a more frequent rotation policy for tokens | 22:07 |
| lbragstad | but - that's obviously subject to each deployment | 22:07 |
| lbragstad | and nothing should stop an operators from rotating as much as they want | 22:08 |
| *** spzala has joined #openstack-keystone | 22:10 | |
| *** esp has quit IRC | 22:10 | |
| *** adriant has joined #openstack-keystone | 22:11 | |
| *** esp has joined #openstack-keystone | 22:11 | |
| *** ccneill has left #openstack-keystone | 22:12 | |
| *** asettle has joined #openstack-keystone | 22:14 | |
| *** spzala has quit IRC | 22:15 | |
| *** spzala has joined #openstack-keystone | 22:15 | |
| *** asettle has quit IRC | 22:19 | |
| bknudson | mordred: are you saying we should switch keystone to twisted? | 22:24 |
| *** slberger1 has left #openstack-keystone | 22:24 | |
| dstanek | i'd rather die | 22:24 |
| mordred | bknudson: YES! | 22:24 |
| *** su_zhang has quit IRC | 22:32 | |
| *** chrisshattuck has joined #openstack-keystone | 22:32 | |
| *** chrisshattuck has quit IRC | 22:46 | |
| breton | sounds good, +1 :p | 22:58 |
| *** esp has quit IRC | 23:08 | |
| *** edtubill has quit IRC | 23:10 | |
| *** michauds has quit IRC | 23:11 | |
| *** jamielennox|away is now known as jamielennox | 23:14 | |
| *** mlovell has joined #openstack-keystone | 23:14 | |
| *** jamielennox is now known as jamielennox|away | 23:22 | |
| lbragstad | stevemar what's out plan with credential encryption? | 23:23 |
| lbragstad | our* | 23:23 |
| *** gyee has quit IRC | 23:24 | |
| *** hockeynut has joined #openstack-keystone | 23:26 | |
| *** jamielennox|away is now known as jamielennox | 23:28 | |
| *** esp has joined #openstack-keystone | 23:32 | |
| *** chrichip has quit IRC | 23:36 | |
| *** jamielennox is now known as jamielennox|away | 23:36 | |
| *** chrichip has joined #openstack-keystone | 23:37 | |
| openstackgerrit | Adrian Turjak proposed openstack/keystone-specs: Optional MFA via password + TOTP auth plugin https://review.openstack.org/345113 | 23:40 |
| *** gyee has joined #openstack-keystone | 23:41 | |
| *** chrichip has quit IRC | 23:42 | |
| *** chrichip has joined #openstack-keystone | 23:43 | |
| adriant | For security purposes, how do people feel about replacing the default openrc.sh file in Horizon to one that uses token_auth (thus doesn't store the password locally at all)? | 23:46 |
| adriant | example: http://paste.openstack.org/show/565460/ | 23:46 |
| breton | anybody uses openrc files from horizon? | 23:47 |
| adriant | While keeping a password in local memory isn't hugely unsafe, it really should be avoided, and token auth allows a good alternative. | 23:47 |
| adriant | for CLI stuff, yeah | 23:47 |
| adriant | breton: Is there an easier default than using the openrc files? | 23:48 |
| adriant | I'm asking here first because it's a security/auth question really. | 23:49 |
| adriant | breton: http://docs.openstack.org/user-guide/common/cli-set-environment-variables-using-openstack-rc.html | 23:51 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
