| gagehugo | lbragstad I have not | 00:14 |
|---|---|---|
| lbragstad | gagehugo: so your battery hasn't imploded... | 00:16 |
| gagehugo | not yet haha | 00:18 |
| gagehugo | my issue may have already been fixed though if I had it | 00:18 |
| lbragstad | when did you get yours? | 00:20 |
| *** r-daneel has quit IRC | 00:23 | |
| *** brad[] has joined #openstack-keystone | 00:25 | |
| gagehugo | uh | 00:26 |
| gagehugo | mar/april last year | 00:26 |
| gagehugo | mine came up affected | 00:27 |
| gagehugo | I think it's literally a loose screw may be in the laptop | 00:27 |
| lbragstad | ahh | 00:29 |
| *** jessegler has quit IRC | 00:37 | |
| *** zhurong has joined #openstack-keystone | 00:42 | |
| *** gmann_ has joined #openstack-keystone | 01:03 | |
| *** wxy has joined #openstack-keystone | 01:08 | |
| openstackgerrit | Merged openstack/keystoneauth master: Zuul: Remove project name https://review.openstack.org/541080 | 01:08 |
| *** gongysh has joined #openstack-keystone | 01:10 | |
| *** gongysh has quit IRC | 01:20 | |
| wxy | lbragstad: hi | 01:22 |
| lbragstad | wxy: o/ | 01:22 |
| wxy | lbragstad: I don't see JWT topic in https://etherpad.openstack.org/p/keystone-rocky-ptg | 01:23 |
| lbragstad | line 124 | 01:23 |
| lbragstad | :) | 01:23 |
| lbragstad | hackathon | 01:23 |
| wxy | lbragstad: lol. I searched for "JWT".. | 01:23 |
| lbragstad | there - you should be able to search for it now | 01:24 |
| lbragstad | lol | 01:24 |
| *** links has joined #openstack-keystone | 01:28 | |
| *** links has quit IRC | 01:29 | |
| lbragstad | wxy: added some more details about it | 01:30 |
| *** daidv has joined #openstack-keystone | 01:33 | |
| *** gmann_ has quit IRC | 01:46 | |
| *** gmann_ has joined #openstack-keystone | 01:46 | |
| *** masber has quit IRC | 01:59 | |
| *** gyee has quit IRC | 02:01 | |
| *** gongysh has joined #openstack-keystone | 02:10 | |
| *** Dave has quit IRC | 02:11 | |
| *** Dave_ has joined #openstack-keystone | 02:11 | |
| *** links has joined #openstack-keystone | 02:12 | |
| *** itlinux has joined #openstack-keystone | 02:14 | |
| *** Supun has joined #openstack-keystone | 02:45 | |
| *** Supun has quit IRC | 02:48 | |
| *** Supun has joined #openstack-keystone | 02:48 | |
| *** ayoung has joined #openstack-keystone | 02:52 | |
| *** namnh has joined #openstack-keystone | 03:18 | |
| *** itlinux has quit IRC | 03:25 | |
| *** Supun has quit IRC | 03:42 | |
| openstackgerrit | Merged openstack/keystone master: Finish refactoring self.*_api out of tests https://review.openstack.org/541074 | 03:46 |
| openstackgerrit | Merged openstack/keystone master: Validate identity providers during token validation https://review.openstack.org/531915 | 03:46 |
| openstackgerrit | Merged openstack/keystone master: Imported Translations from Zanata https://review.openstack.org/540583 | 03:47 |
| openstackgerrit | Merged openstack/keystone master: Add cache invalidation when delete application credential https://review.openstack.org/540324 | 04:00 |
| *** openstackgerrit has quit IRC | 04:04 | |
| *** itlinux has joined #openstack-keystone | 04:08 | |
| *** mvk has quit IRC | 04:21 | |
| *** harlowja has quit IRC | 04:32 | |
| *** nicolasbock has quit IRC | 04:32 | |
| *** namnh has quit IRC | 04:42 | |
| *** daidv has quit IRC | 04:44 | |
| *** Suramya has joined #openstack-keystone | 04:51 | |
| *** vish_18 has joined #openstack-keystone | 04:52 | |
| *** itlinux has quit IRC | 05:06 | |
| vish_18 | cmurphy: thanks ! | 05:09 |
| *** harlowja has joined #openstack-keystone | 05:37 | |
| *** zhurong has quit IRC | 05:39 | |
| *** Suramya_ has joined #openstack-keystone | 05:40 | |
| *** jaosorior has joined #openstack-keystone | 05:50 | |
| *** zhurong has joined #openstack-keystone | 05:53 | |
| *** openstackgerrit has joined #openstack-keystone | 06:03 | |
| openstackgerrit | Merged openstack/keystone master: Add scope_types to role policies https://review.openstack.org/526171 | 06:03 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/oslo.policy master: Imported Translations from Zanata https://review.openstack.org/541553 | 06:07 |
| *** annp has joined #openstack-keystone | 06:13 | |
| *** vish_18_ has joined #openstack-keystone | 06:18 | |
| vish_18_ | cmurphy: the orignal bug can be reproduced when we uncomment max_active_keys = 3 in master | 06:21 |
| vish_18_ | cmurphy: ad setting driver = blah for fernet provider | 06:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone master: Imported Translations from Zanata https://review.openstack.org/541567 | 06:40 |
| *** harlowja has quit IRC | 06:52 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 07:22 | |
| *** martinus__ has joined #openstack-keystone | 07:42 | |
| *** AlexeyAbashkin has quit IRC | 07:42 | |
| *** rcernin has quit IRC | 07:48 | |
| *** akrzos has quit IRC | 07:50 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 07:54 | |
| *** akrzos has joined #openstack-keystone | 08:02 | |
| *** annp has quit IRC | 08:14 | |
| *** annp has joined #openstack-keystone | 08:14 | |
| *** threestrands has quit IRC | 08:22 | |
| *** tesseract has joined #openstack-keystone | 08:24 | |
| *** zhurong has quit IRC | 08:25 | |
| *** zhurong has joined #openstack-keystone | 08:26 | |
| *** pcaruana has joined #openstack-keystone | 08:38 | |
| *** StefanPaetowJisc has joined #openstack-keystone | 08:48 | |
| *** StefanPaetowJisc has quit IRC | 08:59 | |
| openstackgerrit | Merged openstack/keystone master: Remove all v2.0 APIs except the ec2tokens API https://review.openstack.org/540141 | 09:01 |
| openstackgerrit | Merged openstack/keystone master: Remove v2.0 extension documentation https://review.openstack.org/540525 | 09:01 |
| *** StefanPaetowJisc has joined #openstack-keystone | 09:20 | |
| *** StefanPaetowJisc has quit IRC | 09:25 | |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Delete tokens from DB is not required in case of Fernet. https://review.openstack.org/537322 | 09:29 |
| *** Adri2000 has joined #openstack-keystone | 09:56 | |
| Adri2000 | hello | 09:56 |
| Adri2000 | is there a reason keystone doesn't default to using oslo_cache.memcache_pool? | 09:57 |
| *** pcaruana has quit IRC | 09:57 | |
| *** namnh has joined #openstack-keystone | 09:57 | |
| openstackgerrit | baiwenteng proposed openstack/python-keystoneclient master: pelling error Keytone https://review.openstack.org/541644 | 09:58 |
| *** Exhar has quit IRC | 10:05 | |
| *** sambetts|afk is now known as sambetts | 10:10 | |
| *** Exhar has joined #openstack-keystone | 10:11 | |
| *** pcaruana has joined #openstack-keystone | 10:13 | |
| *** Exhar has quit IRC | 10:15 | |
| Adri2000 | related question: is it a known issue that the default dogpile.cache.memcached backend leaks open connections? (I've found this old bug report https://bugs.launchpad.net/keystone/+bug/1360446) | 10:18 |
| openstack | Launchpad bug 1360446 in keystonemiddleware "client connection leak to memcached under eventlet due to threadlocal" [Medium,Fix released] - Assigned to Morgan Fainberg (mdrnstm) | 10:18 |
| *** annp has quit IRC | 10:22 | |
| *** gongysh has quit IRC | 10:33 | |
| *** zhurong has quit IRC | 10:35 | |
| openstackgerrit | Merged openstack/keystone master: Update sample configuration file for Queens https://review.openstack.org/541447 | 10:37 |
| *** Dave_ is now known as Dave | 10:41 | |
| *** StefanPaetowJisc has joined #openstack-keystone | 10:54 | |
| *** pcaruana has quit IRC | 11:09 | |
| *** dmellado has quit IRC | 11:21 | |
| *** dmellado has joined #openstack-keystone | 11:27 | |
| *** namnh has quit IRC | 11:28 | |
| *** mvk has joined #openstack-keystone | 11:36 | |
| *** nicolasbock has joined #openstack-keystone | 11:41 | |
| *** dmellado has quit IRC | 11:48 | |
| *** StefanPaetowJisc has quit IRC | 11:55 | |
| *** dmellado has joined #openstack-keystone | 11:59 | |
| *** edmondsw has joined #openstack-keystone | 12:02 | |
| *** raildo has joined #openstack-keystone | 12:11 | |
| *** pcaruana has joined #openstack-keystone | 12:13 | |
| *** dave-mcc_ has joined #openstack-keystone | 12:15 | |
| *** AlexeyAbashkin has quit IRC | 12:25 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 12:40 | |
| openstackgerrit | Merged openstack/keystone master: Remove the deprecated "giturl" option https://review.openstack.org/533466 | 12:51 |
| *** jmlowe has quit IRC | 13:07 | |
| *** gongysh has joined #openstack-keystone | 13:15 | |
| openstackgerrit | Suramya proposed openstack/keystone master: Reorganize api-ref: v3-ext trust.inc https://review.openstack.org/531772 | 13:25 |
| openstackgerrit | Suramya proposed openstack/keystone master: Reorganize api-ref: v3-ext trust.inc https://review.openstack.org/531772 | 13:32 |
| *** AlexeyAbashkin has quit IRC | 13:34 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 13:35 | |
| *** AlexeyAbashkin has quit IRC | 13:40 | |
| *** Supun has joined #openstack-keystone | 13:41 | |
| *** jmlowe has joined #openstack-keystone | 13:46 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 13:55 | |
| *** markvoelker has joined #openstack-keystone | 13:57 | |
| *** r-daneel has joined #openstack-keystone | 14:04 | |
| *** links has quit IRC | 14:09 | |
| *** r-daneel has quit IRC | 14:12 | |
| *** daidv_ has joined #openstack-keystone | 14:13 | |
| *** r-daneel has joined #openstack-keystone | 14:31 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Drop domain id foreign key from user table https://review.openstack.org/539347 | 14:33 |
| lbragstad | our in progress column is *so* close to being empty | 14:56 |
| lbragstad | https://trello.com/b/5F0h9Hoe/keystone-queens-roadmap | 14:56 |
| *** spilla has joined #openstack-keystone | 14:57 | |
| cmurphy | \o/ | 14:58 |
| knikolla | o/ | 14:59 |
| lbragstad | we have a few patches with 2 +2s | 15:03 |
| lbragstad | so long as there isn't anything outstanding, we could probably start pushing those through | 15:03 |
| lbragstad | https://review.openstack.org/#/c/540529/ | 15:03 |
| lbragstad | https://review.openstack.org/#/c/540499/ | 15:04 |
| lbragstad | https://review.openstack.org/#/c/539342/ | 15:04 |
| knikolla | lbragstad: nothing like deletion patches to start the day | 15:08 |
| cmurphy | :D | 15:08 |
| lbragstad | mmm fresh cup of coffee, a negative overall diff... what gets better than that? | 15:09 |
| lbragstad | http://lists.openstack.org/pipermail/openstack-dev/2018-February/127039.html is a good idea | 15:10 |
| lbragstad | we used to do that more often when we did midcycle meet ups | 15:11 |
| knikolla | lbragstad: i see references to 35357 in the doc patches, did we reach a consensus on what to do with the second port now that we don't need it? | 15:11 |
| lbragstad | knikolla: yeah - good question.. i have scroll back for you | 15:11 |
| lbragstad | i talked to a couple of the ubuntu maintainers (since cmurphy might be able to help with the suse side and hrybacki might be able to help with the redhat side) | 15:12 |
| lbragstad | knikolla: http://paste.openstack.org/show/664112/ | 15:12 |
| lbragstad | corey is going to work on a patch and reach out when he does | 15:13 |
| lbragstad | once he does that, we should be able to update the second NOTE in our installation guide here - https://docs.openstack.org/keystone/latest/install/keystone-install-ubuntu.html#install-and-configure-components | 15:13 |
| *** Supun has quit IRC | 15:14 | |
| lbragstad | and we can simplify our installation guide | 15:14 |
| *** david-lyle has quit IRC | 15:15 | |
| knikolla | sweet, but that means i have to -1 the docs patch. | 15:15 |
| lbragstad | which one? | 15:15 |
| knikolla | https://review.openstack.org/#/c/540499 | 15:15 |
| knikolla | do a search&replace for 35357 and make it 5000 | 15:16 |
| knikolla | or that can be a follow up patch. | 15:17 |
| lbragstad | the default apache configuration installed from keystone packages will still listen on port 35357 and 5000 | 15:17 |
| lbragstad | so v3 will be available for both | 15:17 |
| lbragstad | s/for/on/ | 15:17 |
| knikolla | but aren't we going to remove 35357? at least that's what i got out of the convo u linked to | 15:18 |
| lbragstad | most likely, it depends on what they decide to use (they could use ports 80, 443, 5000, etc...) | 15:19 |
| knikolla | i kind of like the /identity path instead of a port. | 15:19 |
| knikolla | and use apache just for proxying. | 15:19 |
| lbragstad | yeah | 15:20 |
| lbragstad | that's an option too | 15:20 |
| lbragstad | the ubuntu packages actually set things up like that | 15:20 |
| lbragstad | https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/keystone/tree/debian/keystone.conf#n56 | 15:21 |
| knikolla | lbragstad: our red hat install does the same. | 15:21 |
| lbragstad | cool | 15:22 |
| *** gongysh has quit IRC | 15:23 | |
| knikolla | lbragstad: approved all three doc changes. | 15:23 |
| knikolla | we can worry about ports and paths later. | 15:24 |
| lbragstad | knikolla: thanks - i think we should definitely work through that stuff in rocky, probably talk about it at the PTG | 15:24 |
| *** AlexeyAbashkin has quit IRC | 15:24 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 15:24 | |
| knikolla | sounds good. | 15:25 |
| knikolla | lbragstad: when does master open for rocky? | 15:31 |
| lbragstad | as soon as we get a release candidate cut | 15:32 |
| lbragstad | and we have a stable/queens branch created for keystone | 15:32 |
| lbragstad | then we're technically ready for rocky development | 15:32 |
| openstackgerrit | Merged openstack/keystone master: Fix list users by name https://review.openstack.org/529914 | 15:41 |
| openstackgerrit | Merged openstack/keystone master: Reorganize api-ref: v3-ext trust.inc https://review.openstack.org/531772 | 15:41 |
| *** idlemind has quit IRC | 15:46 | |
| *** idlemind has joined #openstack-keystone | 15:47 | |
| openstackgerrit | Suramya proposed openstack/keystone master: Reorganize api-ref: v3-ext federation identity-provider https://review.openstack.org/533414 | 15:56 |
| openstackgerrit | Merged openstack/keystone master: Remove v2 and v2-admin API documentation https://review.openstack.org/540529 | 16:00 |
| *** pcaruana has quit IRC | 16:03 | |
| *** Supun has joined #openstack-keystone | 16:09 | |
| *** david-lyle has joined #openstack-keystone | 16:18 | |
| *** gyee has joined #openstack-keystone | 16:45 | |
| *** Supun has quit IRC | 16:45 | |
| *** markvoelker_ has joined #openstack-keystone | 16:47 | |
| *** itlinux has joined #openstack-keystone | 16:49 | |
| *** markvoelker has quit IRC | 16:49 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Drop domain id foreign key from user table https://review.openstack.org/539347 | 16:54 |
| openstackgerrit | Merged openstack/keystone master: Remove v2.0 from documentation guides https://review.openstack.org/540499 | 16:56 |
| openstackgerrit | Merged openstack/keystone master: Update curl request documentation to remove v2.0 https://review.openstack.org/539342 | 16:56 |
| *** harlowja has joined #openstack-keystone | 17:11 | |
| *** markvoelker_ has quit IRC | 17:14 | |
| *** markvoelker has joined #openstack-keystone | 17:14 | |
| *** harlowja has quit IRC | 17:17 | |
| *** AlexeyAbashkin has quit IRC | 17:26 | |
| *** tesseract has quit IRC | 17:27 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Reorganize api-ref: v3 os-pki https://review.openstack.org/530459 | 17:44 |
| gagehugo | bah | 17:45 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Reorganize api-ref: v3 os-pki https://review.openstack.org/530459 | 17:46 |
| gagehugo | cmurphy: fixed the merge conflict | 17:47 |
| cmurphy | huh i didn't notice it was merge conflicting | 17:48 |
| *** Supun has joined #openstack-keystone | 17:59 | |
| *** cfriesen has left #openstack-keystone | 18:02 | |
| *** harlowja has joined #openstack-keystone | 18:11 | |
| *** Supun has quit IRC | 18:11 | |
| lbragstad | gagehugo: nice - thanks | 18:15 |
| *** harlowja has quit IRC | 18:18 | |
| *** raildo has quit IRC | 18:19 | |
| lbragstad | does anyone want to do a project interview during the PTG? http://lists.openstack.org/pipermail/openstack-dev/2018-February/127085.html | 18:19 |
| lbragstad | cc wxy unified limits would be a good topic | 18:19 |
| lbragstad | knikolla: cmurphy fyi - https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/keystone/commit/?id=915d787af42096b0fad715e49759cd357e47787e | 18:20 |
| lbragstad | from coreycb | 18:20 |
| knikolla | lbragstad: yay! | 18:20 |
| lbragstad | that will get rolled out with rc1 | 18:21 |
| lbragstad | so - i guess the next question is, do we need an rc2 if that becomes the new documentation? | 18:21 |
| lbragstad | s/documentation/apache configuration/ | 18:21 |
| lbragstad | do we need new documentation for that configuration? | 18:21 |
| *** raildo has joined #openstack-keystone | 18:22 | |
| *** r-daneel_ has joined #openstack-keystone | 18:28 | |
| knikolla | lbragstad: a search and replace for 35357 -> 5000 at a minimum | 18:28 |
| lbragstad | ack | 18:28 |
| lbragstad | we should be able to get that done today | 18:28 |
| lbragstad | and merged before tomorrow | 18:28 |
| knikolla | ack, i'll review | 18:28 |
| *** Supun has joined #openstack-keystone | 18:28 | |
| *** r-daneel has quit IRC | 18:29 | |
| *** r-daneel_ is now known as r-daneel | 18:29 | |
| knikolla | we should also have redhat and suse (cmurphy) remove 35357, so that there's consistency | 18:29 |
| knikolla | in their packages | 18:29 |
| cmurphy | yeah i can work on that this week | 18:29 |
| knikolla | cool :) | 18:30 |
| *** Supun has quit IRC | 18:38 | |
| *** Supun has joined #openstack-keystone | 18:38 | |
| *** sambetts is now known as sambetts|afk | 18:41 | |
| *** Supun has quit IRC | 18:57 | |
| *** Supun has joined #openstack-keystone | 19:00 | |
| mordred | cmurphy, lbragstad: I just found a super-edge-casey bug in keystoneauth version discovery ... requesting version='latest' ona service that does not have any version discovery document but also has an unversioned endpoint in the catalog results in a traceback | 19:04 |
| mordred | instead of falling back to using the endpoint in the catalog | 19:04 |
| kmalloc | mordred: gross | 19:05 |
| mordred | kmalloc: yah. like I said -it's SUPER edge case, and I'll be working with the service in quesiton to get a version discovery document added | 19:05 |
| kmalloc | k | 19:05 |
| *** Exhar has joined #openstack-keystone | 19:05 | |
| kmalloc | we should prob. get a bug on it. | 19:05 |
| mordred | yah | 19:06 |
| cmurphy | nice | 19:06 |
| *** Supun has quit IRC | 19:09 | |
| cmurphy | hey kmalloc i had a crazy idea, can you tell me if it's ridiculous https://review.openstack.org/#/c/539347/ | 19:11 |
| cmurphy | ayoung is not a fan | 19:11 |
| kmalloc | it's ridiculous, but not sure if good or bad until i look at it ;) [ok that doesn't help heheh] | 19:11 |
| kmalloc | looking | 19:11 |
| cmurphy | :P | 19:11 |
| kmalloc | honestly, i prefer to drop the FK. | 19:12 |
| cmurphy | that's what that does | 19:12 |
| kmalloc | yeah | 19:12 |
| kmalloc | it's a tough sell because it changes behavior(ish) in some cases. | 19:12 |
| kmalloc | the FK was a bad call when it was added | 19:13 |
| kmalloc | let me think about the ramifications, but i'd like to see it go if it doesn't impact things too much | 19:13 |
| cmurphy | thanks kmalloc | 19:13 |
| *** r-daneel_ has joined #openstack-keystone | 19:13 | |
| kmalloc | man, i'm cold today. not sure why | 19:14 |
| *** r-daneel has quit IRC | 19:14 | |
| *** r-daneel_ is now known as r-daneel | 19:14 | |
| cmurphy | i'm rationalizing that since it doesn't break tempest then it's not really changing behavior that badly | 19:14 |
| kmalloc | right. | 19:14 |
| cmurphy | but the counter to that is maybe we don't have enough coverage | 19:14 |
| kmalloc | we def. don't have enough coverage :P | 19:15 |
| cmurphy | lol | 19:15 |
| kmalloc | but that aside... | 19:15 |
| *** jmlowe_ has joined #openstack-keystone | 19:16 | |
| *** jmlowe has quit IRC | 19:17 | |
| *** knasim-wrs has joined #openstack-keystone | 19:17 | |
| knasim-wrs | hi folks, I've migrated my deployment to Fernet and see a degradation in performance around VM Live Migration events | 19:18 |
| knasim-wrs | I am tracking the services (Cinder, Neutron, Nova) to see if the AUTH calls they make out to Keystone in Pike, have increased dramatically. In the meantime I was thinking of enabling caching | 19:19 |
| knasim-wrs | does Keystone offer an in-process cache? Something that does not require integration with Memcached or Redis | 19:19 |
| kmalloc | you should enable caching in general, it improves performance for tokens/validation by a large amount | 19:19 |
| openstackgerrit | Monty Taylor proposed openstack/keystoneauth master: WIP Handle unversioned endpoints with no discovery document https://review.openstack.org/541844 | 19:19 |
| kmalloc | don't do in-process caching | 19:19 |
| kmalloc | you will be sad, because something will cache on some requests but not others | 19:20 |
| kmalloc | and you might get failures/successes because tokens expire/revoke and it is only reflected on some processes | 19:20 |
| knasim-wrs | kmalloc: Thanks. So you're saying enable cache/enabled = True | 19:20 |
| mordred | kmalloc, cmurphy, lbragstad: ^^ that fixes the immediate issue - but clearly needs tests and I left a TODO in there about another check that sohuld be added | 19:20 |
| knasim-wrs | but what about the caching backend? | 19:20 |
| kmalloc | i would, but you should be running with memcache or redis, memcache is the one with the most drivetime | 19:20 |
| kmalloc | keystone ALSO does per-request caching for the data from the DB, but that cache is only to minimize SQL round-trips | 19:21 |
| knasim-wrs | won't be able to get Memcached in the product at this stage... shipping out next month | 19:21 |
| kmalloc | each of the services that use keystonemiddleware should also use caching for the tokens | 19:21 |
| kmalloc | that can be done in-memory, but again, memcache is a much much better choice | 19:21 |
| knasim-wrs | k | 19:21 |
| kmalloc | memcache scales keystone and auth/validation by a lot | 19:22 |
| knasim-wrs | so memcache is better, and I can consider that for our next software release. But for now I am going to enable the internal caches | 19:22 |
| kmalloc | be careful on that. do extensive testing and expect some wonky behavior | 19:22 |
| knasim-wrs | 1) enable caching in the keystonemiddleware in all the services | 19:23 |
| kmalloc | in-process caching in openstack is, imo, a mistake | 19:23 |
| kmalloc | we test in the gate with memcache fwiw. | 19:23 |
| kmalloc | but if you're ok with the edge cases, then it can help for sure. | 19:23 |
| kmalloc | (in-process that is) | 19:23 |
| kmalloc | mordred: thanks | 19:24 |
| kmalloc | mordred: yeah def. need tests. but that is a good starting place. | 19:24 |
| mordred | kmalloc: I happened to have a convenient live-against-public-cloud reproduction case locally so I could verify that that fixes the behavior - figured just pushing it up would record the issue | 19:25 |
| mordred | until we cna get it fleshed out more better | 19:25 |
| knasim-wrs | yeah I see that we have the in-process cache enabled in some of our services: | 19:25 |
| knasim-wrs | https://thepasteb.in/p/NxhVxEGNAW8FN | 19:26 |
| knasim-wrs | kmalloc: For the caching backends in keystone I have these options in Pike | 19:27 |
| knasim-wrs | kmalloc: For the caching backends in keystone I have these options in Pike | 19:28 |
| knasim-wrs | https://thepasteb.in/p/P1hvmok0pLkUl | 19:28 |
| *** Supun has joined #openstack-keystone | 19:28 | |
| knasim-wrs | set to dogpile.cache.null | 19:28 |
| lbragstad | i was tinkering with dogpile.cache.memcached recently | 19:28 |
| lbragstad | debugging some openstack-ansible issues | 19:28 |
| knasim-wrs | how was it LanceÉ | 19:29 |
| knasim-wrs | just looking for a stable in-process cache that scales decently | 19:29 |
| knasim-wrs | enough for us to get the product out... right now Live Migration times in Pike after transitioning to Fernet has gone up to 70 seconds | 19:29 |
| knasim-wrs | that is too long | 19:29 |
| knasim-wrs | we were in the 30-45 sec range in Newton | 19:30 |
| lbragstad | the problem we were hitting was with multiple cache backends configured, which keystone should shard data across, making it an even better choice | 19:30 |
| kmalloc | lbragstad: there was some bug around that... i think we fixed it? | 19:30 |
| lbragstad | yep - well, partially | 19:30 |
| lbragstad | i have a link | 19:30 |
| lbragstad | this was the fix https://review.openstack.org/#/c/533314/ | 19:31 |
| kmalloc | yah. that one | 19:31 |
| lbragstad | https://bugs.launchpad.net/keystone/+bug/1743036 | 19:31 |
| openstack | Launchpad bug 1743036 in oslo.cache "Multiple memcached back-end instances breaks caching" [Undecided,Confirmed] - Assigned to Morgan Fainberg (mdrnstm) | 19:31 |
| kmalloc | ugh | 19:31 |
| kmalloc | thought that was it. | 19:31 |
| lbragstad | but - there is a work around for it | 19:31 |
| kmalloc | i have a fix for it, but it's going to be a lot of code | 19:32 |
| lbragstad | it more or less boils down to client issues, based on our last discussions | 19:32 |
| kmalloc | so backend_argument can be used again | 19:32 |
| kmalloc | it is not easy to fix unfortunately | 19:32 |
| *** markvoelker has quit IRC | 19:33 | |
| *** markvoelker_ has joined #openstack-keystone | 19:33 | |
| *** jmlowe has joined #openstack-keystone | 19:33 | |
| *** jmlowe has quit IRC | 19:33 | |
| lbragstad | knasim-wrs: i think one place i might start | 19:33 |
| lbragstad | would be to try and install memcached locally with the keystone servers | 19:33 |
| lbragstad | and just let keystone use it's own local memcached deployment | 19:34 |
| *** jmlowe_ has quit IRC | 19:34 | |
| lbragstad | worst case performance would be a token hitting each server for validation before returning to one that has already validated it | 19:34 |
| lbragstad | (since each memcached instance is going to store it's own copy of the token) | 19:35 |
| knasim-wrs | I floated that idea by our architect. We would need to add memcached as a managed service since our platform supports HA (that is services run active-standby) so you can switch over on node failures | 19:35 |
| lbragstad | then you're not dealing with in-process caching per se | 19:35 |
| knasim-wrs | and so that got turned down fast as its too late in the release cycle | 19:36 |
| lbragstad | :( | 19:36 |
| knasim-wrs | which is why I thought the next best option is in-process caching | 19:36 |
| lbragstad | i see what you mean | 19:36 |
| kmalloc | also, i think newton era had some fernet performance issues... lbragstad correct me if i'm wrong | 19:37 |
| knasim-wrs | now with Fernet tokens I am also getting tons of these error messages: | 19:37 |
| knasim-wrs | https://thepasteb.in/p/RghnlRzyJw5uz | 19:37 |
| lbragstad | it wasn't so much fernet being the problem, but the entire token validation process changed as a result of token writing a token reference to disk | 19:37 |
| kmalloc | knasim-wrs: that is using old tokens with fernet | 19:37 |
| *** AlexeyAbashkin has joined #openstack-keystone | 19:37 | |
| kmalloc | those are uuids | 19:38 |
| lbragstad | so - by way of switching to fernet, we uncovered a lot of performance improvements we could make | 19:38 |
| kmalloc | you have things trying to validate uuid tokens. | 19:38 |
| knasim-wrs | jeez | 19:38 |
| lbragstad | the result was a bunch of fixes to our caching implementation to make it more scalable | 19:38 |
| kmalloc | you can only use a single token backend. | 19:38 |
| kmalloc | and old tokens when you switch backends will fail | 19:38 |
| kmalloc | it's expected. | 19:39 |
| lbragstad | right | 19:39 |
| lbragstad | knasim-wrs: do you have to have a graceful turn over? | 19:39 |
| lbragstad | between the switch from UUID -> Fernet? | 19:39 |
| knasim-wrs | we create the fernet repo during bootstrap and then set the provider etc | 19:39 |
| knasim-wrs | so I don't understand how services are sending over UUID tokens | 19:40 |
| knasim-wrs | it also reports this: | 19:40 |
| knasim-wrs | 2018-02-05 21:42:57.766 48706 WARNING keystonemiddleware.auth_token [-] A valid token was submitted as a service token, but it was not a valid service token. This is incorrect but backwards compatible behaviour. This will be removed in future releases. | 19:40 |
| lbragstad | they could be holding on to a token prior to the switch | 19:40 |
| lbragstad | clients should re-authenticate if they get a 404 | 19:40 |
| lbragstad | which will return a new token | 19:41 |
| lbragstad | in the new format | 19:41 |
| knasim-wrs | yeah clients are fine. It is just blowing up performance to authenticate again via Password | 19:41 |
| knasim-wrs | need to investigate further | 19:41 |
| lbragstad | i did an experiment with a graceful switch | 19:41 |
| lbragstad | and i wrote about the changes you'd need to make if you didn't want "token downtime" during the migration from uuid -> fernet | 19:42 |
| lbragstad | https://www.lbragstad.com/blog/migrating-token-formats-without-downtime in case that helps | 19:42 |
| knasim-wrs | beautiful | 19:42 |
| knasim-wrs | i'll look into this to see if we missed anything | 19:43 |
| lbragstad | but it appears something is trying to pass a uuid token after you make the switch to fernet | 19:43 |
| knasim-wrs | as for in-process caching, is dogpile.cache.null ok? | 19:43 |
| knasim-wrs | yeah | 19:43 |
| kmalloc | null is no caching | 19:43 |
| kmalloc | it's a do-nothing cache | 19:43 |
| kmalloc | dogpile.cache.memory should be in-process | 19:43 |
| lbragstad | unless you have half the cluster running uuid and the other half issuing fernet | 19:43 |
| knasim-wrs | yeah that issue needs to be fixed first | 19:44 |
| lbragstad | then token validation is going to be at the mercy of your load balancer algorithm | 19:44 |
| knasim-wrs | yeah we'd have a lot of cache misses if the transition from UUID -> fernet is not graceful or the cache is not flushed on transition | 19:45 |
| knasim-wrs | what is oslo_cache.dict? In process? | 19:45 |
| kmalloc | yeah | 19:46 |
| kmalloc | same thing i think | 19:46 |
| kmalloc | it does some cleanup though | 19:46 |
| kmalloc | oslo_cache.dict is better than dogpile.cache.memory | 19:46 |
| lbragstad | https://github.com/openstack/oslo.cache/blob/master/oslo_cache/backends/dictionary.py | 19:47 |
| mordred | kmalloc: blast | 19:49 |
| kmalloc | mordred: ? | 19:49 |
| kmalloc | didn't work | 19:49 |
| kmalloc | ? | 19:49 |
| mordred | kmalloc: I'd love to be able to use the stuff in oslo.cache in shade/sdk's caching layer | 19:50 |
| kmalloc | ah | 19:50 |
| kmalloc | yeah | 19:50 |
| kmalloc | =/ | 19:50 |
| mordred | kmalloc: but it pulls in oslo.config and friends | 19:50 |
| kmalloc | yup | 19:50 |
| kmalloc | it's why KSA doesn't use it | 19:50 |
| mordred | I wonder if we could just split those dogpile plugins into their own thing | 19:50 |
| *** pcaruana has joined #openstack-keystone | 19:52 | |
| kmalloc | well i mean... you could. | 19:53 |
| kmalloc | you lose a bunch of the nice-config stuff that we did for the cache regions | 19:54 |
| kmalloc | but those backends should mostly work with just dogpile.cache | 19:54 |
| kmalloc | maybe need to just vendor the backend into shade (the in-mem one) | 19:54 |
| *** blake has joined #openstack-keystone | 19:56 | |
| mordred | kmalloc: yah - I'll poke around at options for that in this next cycle - I could also use some distributed locking support for the rate-limiting task manager | 20:01 |
| kmalloc | right | 20:01 |
| *** jmlowe has joined #openstack-keystone | 20:02 | |
| *** Supun has quit IRC | 20:04 | |
| *** pcaruana has quit IRC | 20:07 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Drop domain id foreign key from user table https://review.openstack.org/539347 | 20:08 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Replace port 35357 with 5000 for ubuntu guide https://review.openstack.org/541857 | 20:09 |
| lbragstad | cmurphy: knikolla ^ | 20:09 |
| knikolla | easy +2 | 20:10 |
| openstackgerrit | Merged openstack/keystone master: Reorganize api-ref: v3 os-pki https://review.openstack.org/530459 | 20:12 |
| cmurphy | when I install keystone from cloud-archive:queens on ubuntu i still get the old vhost file | 20:19 |
| cmurphy | lbragstad: is that commit just in a staging repo or waiting to be synced somewhere? | 20:20 |
| lbragstad | it apparently is going to be sync'd when RC1 goes out | 20:20 |
| lbragstad | at least when i was talking to coreycb in #openstack-pkg earlier | 20:21 |
| cmurphy | ah okay | 20:21 |
| *** AlexeyAbashkin has quit IRC | 20:33 | |
| *** mvk_ has joined #openstack-keystone | 20:34 | |
| *** blake has quit IRC | 20:36 | |
| *** mvk has quit IRC | 20:36 | |
| *** blake has joined #openstack-keystone | 20:36 | |
| *** blake has quit IRC | 20:48 | |
| *** blake has joined #openstack-keystone | 20:48 | |
| *** dave-mcc_ has quit IRC | 20:49 | |
| *** dave-mccowan has joined #openstack-keystone | 20:50 | |
| *** blake__ has joined #openstack-keystone | 20:50 | |
| *** blake has quit IRC | 20:53 | |
| *** blake__ is now known as blake | 20:55 | |
| *** timothyb89 has quit IRC | 21:09 | |
| *** timothyb89 has joined #openstack-keystone | 21:10 | |
| openstackgerrit | Merged openstack/oslo.policy master: Imported Translations from Zanata https://review.openstack.org/541553 | 21:16 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone-specs master: Repropose JWT specification for Rocky https://review.openstack.org/541903 | 21:17 |
| openstackgerrit | Merged openstack/keystone master: Replace port 35357 with 5000 for ubuntu guide https://review.openstack.org/541857 | 21:23 |
| *** Suramya has quit IRC | 21:23 | |
| *** Suramya_ has quit IRC | 21:23 | |
| *** raildo has quit IRC | 21:24 | |
| lbragstad | cmurphy: you don't know if rderose has looked at https://review.openstack.org/#/c/539347/14 yet, do you? | 21:31 |
| lbragstad | it doesn't look like he's commented on it | 21:33 |
| lbragstad | i remember he was in a session with dstanek for like a day figuring out the whole multiple user references thing | 21:34 |
| cmurphy | lbragstad: i haven't talked to him about it | 21:35 |
| lbragstad | i might send an email to see if he'd be able to take a look at it | 21:35 |
| lbragstad | since i don't see him on irc | 21:35 |
| cmurphy | okay | 21:35 |
| *** threestrands has joined #openstack-keystone | 21:39 | |
| *** panbalag has left #openstack-keystone | 21:46 | |
| *** spilla has quit IRC | 22:07 | |
| lbragstad | curious if anyone wants to give https://bugs.launchpad.net/keystone/+bug/1748027 a read | 22:07 |
| openstack | Launchpad bug 1748027 in OpenStack Identity (keystone) "The v3 users API should account for different scopes" [Undecided,New] | 22:07 |
| lbragstad | before i start creating a bunch of them for every FIXME we landed in policies for scope_types | 22:07 |
| *** r-daneel has quit IRC | 22:08 | |
| cmurphy | lbragstad: lgtm | 22:09 |
| lbragstad | sweet | 22:09 |
| *** rcernin has joined #openstack-keystone | 22:25 | |
| lbragstad | stepping away for a bit, i'll be on tonight though | 22:33 |
| *** knasim-wrs has quit IRC | 22:34 | |
| *** blake has quit IRC | 22:45 | |
| *** blake has joined #openstack-keystone | 22:46 | |
| *** itlinux has quit IRC | 22:50 | |
| *** blake has quit IRC | 22:52 | |
| *** martinus__ has quit IRC | 22:53 | |
| *** masber has joined #openstack-keystone | 23:27 | |
| *** edmondsw has quit IRC | 23:37 | |
| *** itlinux has joined #openstack-keystone | 23:43 | |
| *** dave-mccowan has quit IRC | 23:45 | |
| *** markvoelker_ has quit IRC | 23:45 | |
| *** dave-mccowan has joined #openstack-keystone | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!