| *** spzala has quit IRC | 00:00 | |
| *** david-lyle has quit IRC | 00:01 | |
| *** tonytan4ever has joined #openstack-keystone | 00:04 | |
| *** tonytan4ever has quit IRC | 00:06 | |
| *** tonytan4ever has joined #openstack-keystone | 00:07 | |
| *** david-lyle has joined #openstack-keystone | 00:07 | |
| *** esp has quit IRC | 00:07 | |
| *** ddieterly has quit IRC | 00:09 | |
| *** atod has quit IRC | 00:10 | |
| *** atod has joined #openstack-keystone | 00:16 | |
| *** ddieterly has joined #openstack-keystone | 00:17 | |
| *** atod has quit IRC | 00:19 | |
| *** atod has joined #openstack-keystone | 00:21 | |
| *** david-lyle has quit IRC | 00:26 | |
| *** sdake has quit IRC | 00:36 | |
| *** sdake has joined #openstack-keystone | 00:37 | |
| *** cheran has quit IRC | 00:49 | |
| *** code-R has joined #openstack-keystone | 00:56 | |
| *** su_zhang has quit IRC | 00:57 | |
| *** code-R_ has joined #openstack-keystone | 00:57 | |
| *** code-R has quit IRC | 01:01 | |
| *** tonytan_brb has joined #openstack-keystone | 01:11 | |
| *** tonytan4ever has quit IRC | 01:14 | |
| *** tqtran has quit IRC | 01:19 | |
| *** chlong has joined #openstack-keystone | 01:20 | |
| *** itisha has quit IRC | 01:20 | |
| *** gyee has quit IRC | 01:22 | |
| *** code-R_ has quit IRC | 01:26 | |
| openstackgerrit | Jamie Lennox proposed openstack/keystoneauth: Implement caching for the generic plugins. https://review.openstack.org/359506 | 01:28 |
|---|---|---|
| openstackgerrit | Jamie Lennox proposed openstack/keystoneauth: Implement caching for the generic plugins. https://review.openstack.org/359506 | 01:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 01:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware: Updated from global requirements https://review.openstack.org/359514 | 01:32 |
| *** davechen has joined #openstack-keystone | 01:37 | |
| *** EinstCrazy has joined #openstack-keystone | 01:41 | |
| *** tonytan_brb has quit IRC | 01:41 | |
| *** haplo37__ has quit IRC | 01:41 | |
| openstackgerrit | zhufl proposed openstack/keystone: Remove unnecessary __init__ https://review.openstack.org/359063 | 01:44 |
| *** wangqun has joined #openstack-keystone | 01:45 | |
| *** Gorian has quit IRC | 01:53 | |
| *** willise has joined #openstack-keystone | 01:54 | |
| *** willise has left #openstack-keystone | 01:54 | |
| *** Gorian has joined #openstack-keystone | 01:54 | |
| *** dkehn_ has quit IRC | 01:57 | |
| *** EinstCra_ has joined #openstack-keystone | 02:03 | |
| *** tonytan4ever has joined #openstack-keystone | 02:04 | |
| *** EinstCrazy has quit IRC | 02:06 | |
| *** ravelar has quit IRC | 02:06 | |
| *** dkehn_ has joined #openstack-keystone | 02:10 | |
| *** ddieterly has quit IRC | 02:15 | |
| *** spzala has joined #openstack-keystone | 02:25 | |
| *** ravelar has joined #openstack-keystone | 02:26 | |
| *** atod has quit IRC | 02:36 | |
| *** ravelar has quit IRC | 02:42 | |
| *** sdake has quit IRC | 02:46 | |
| *** sdake has joined #openstack-keystone | 02:49 | |
| *** BjoernT has joined #openstack-keystone | 02:56 | |
| *** aswadr_ has joined #openstack-keystone | 03:17 | |
| *** code-R has joined #openstack-keystone | 03:18 | |
| *** BjoernT is now known as Bjoern_zZzZzZzZ | 03:20 | |
| *** Bjoern_zZzZzZzZ is now known as BjoernT | 03:21 | |
| *** BjoernT is now known as Bjoern_zZzZzZzZ | 03:22 | |
| *** Bjoern_zZzZzZzZ is now known as BjoernT | 03:24 | |
| *** EinstCrazy has joined #openstack-keystone | 03:25 | |
| *** BjoernT has quit IRC | 03:27 | |
| *** EinstCra_ has quit IRC | 03:28 | |
| *** iurygregory_ has quit IRC | 03:46 | |
| openstackgerrit | Harini proposed openstack/keystone: EndpointPolicy driver doesn't inherit interface https://review.openstack.org/352586 | 03:49 |
| *** spzala has quit IRC | 03:50 | |
| *** tonytan4ever has quit IRC | 03:50 | |
| *** bigdogstl has joined #openstack-keystone | 04:01 | |
| *** bigdogstl has quit IRC | 04:05 | |
| openstackgerrit | Jamie Lennox proposed openstack/keystoneauth: Allow specifying client and service info to user_agent https://review.openstack.org/357633 | 04:08 |
| *** roxanagh_ has joined #openstack-keystone | 04:13 | |
| *** jraju has joined #openstack-keystone | 04:14 | |
| *** jraju has quit IRC | 04:15 | |
| openstackgerrit | Jamie Lennox proposed openstack/keystoneauth: Allow specifying client and service info to user_agent https://review.openstack.org/357633 | 04:16 |
| *** roxanagh_ has quit IRC | 04:19 | |
| stevemar | rodrigods: i am now | 04:36 |
| stevemar | mordred: notmorgan we didn't -2 using versionedobjects, we just thought using triggers was an easier approach | 04:38 |
| *** asettle has joined #openstack-keystone | 04:51 | |
| *** tonytan4ever has joined #openstack-keystone | 04:51 | |
| *** tonytan4ever has quit IRC | 04:56 | |
| *** asettle has quit IRC | 04:56 | |
| *** jaosorior has joined #openstack-keystone | 05:07 | |
| *** jaosorior has quit IRC | 05:09 | |
| *** jaosorior has joined #openstack-keystone | 05:10 | |
| *** roxanagh_ has joined #openstack-keystone | 05:39 | |
| *** richm has quit IRC | 05:39 | |
| *** roxanagh_ has quit IRC | 05:39 | |
| *** code-R_ has joined #openstack-keystone | 05:51 | |
| *** adriant has quit IRC | 05:52 | |
| *** tonytan4ever has joined #openstack-keystone | 05:52 | |
| *** EinstCrazy has quit IRC | 05:52 | |
| *** code-R has quit IRC | 05:54 | |
| *** EinstCrazy has joined #openstack-keystone | 05:54 | |
| *** tonytan4ever has quit IRC | 05:57 | |
| *** EinstCrazy has quit IRC | 05:57 | |
| *** EinstCrazy has joined #openstack-keystone | 05:58 | |
| *** atod has joined #openstack-keystone | 05:58 | |
| *** EinstCra_ has joined #openstack-keystone | 06:02 | |
| *** jaugustine has quit IRC | 06:02 | |
| *** tqtran has joined #openstack-keystone | 06:03 | |
| *** EinstCrazy has quit IRC | 06:05 | |
| henrynash | stevemar: imho while we approved the spec for the standard approach, we DID (effectively) -2 the implementation - since in the IRC meeting dolphm and others would not support putting in the code since they did not trust oslo.versioned objects would ever be finished | 06:15 |
| henrynash | stevemar: I had the whole implementation kestone-manage for Newtron up for review for several weeks - and peopel would not support it | 06:15 |
| *** rcernin has joined #openstack-keystone | 06:23 | |
| henrynash | stevemar: even though we don't actually need versionedobjects for keystone in Netron (since we only have additive db changes anyway)...but we would not accept even the principle of the command structure in keystone-manage that would be used when we did support versionedobjects | 06:35 |
| *** code-R_ has quit IRC | 06:43 | |
| bigjools | Folks, is there a convenient way to discover all the URLs for which we route API requests? | 06:53 |
| *** tesseract- has joined #openstack-keystone | 06:56 | |
| openstackgerrit | Ha Van Tu proposed openstack/keystone: [api-ref]: Outdated link reference https://review.openstack.org/359631 | 06:59 |
| jamielennox | bigjools: from memory if you hook the right place you can print() the mapper object and it will show you everything | 07:07 |
| jamielennox | you just have to do it on a request or something so that everything is loaded up | 07:07 |
| bigjools | jamielennox: ok, thanks, I just need to work out where the mapper lives inside a request | 07:08 |
| jamielennox | bigjools: yea, i think it comes through as part of the environ in a request or something? i can't remember exactly but it's not too hard to find | 07:08 |
| bigjools | ok cheers, good info | 07:09 |
| bigjools | I'm looking at doing some instrumentation - I might not need all the endpoints but it's useful to see anyway | 07:09 |
| openstackgerrit | Thomas Bechtold proposed openstack/keystone: Fix tempest.conf generation https://review.openstack.org/355723 | 07:20 |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: Use fixtures from keystoneauth https://review.openstack.org/359642 | 07:20 |
| *** code-R has joined #openstack-keystone | 07:29 | |
| *** code-R_ has joined #openstack-keystone | 07:31 | |
| *** code-R has quit IRC | 07:34 | |
| *** ccard has quit IRC | 07:36 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: Use AUTH_INTERFACE object from keystoneauth https://review.openstack.org/359653 | 07:41 |
| *** code-R_ has quit IRC | 07:48 | |
| *** code-R has joined #openstack-keystone | 07:48 | |
| *** tonytan4ever has joined #openstack-keystone | 07:54 | |
| *** tonytan4ever has quit IRC | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-keystone | 08:00 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c https://review.openstack.org/318435 | 08:10 |
| *** aloga has quit IRC | 08:10 | |
| *** aloga has joined #openstack-keystone | 08:10 | |
| openstackgerrit | Divya K Konoor proposed openstack/keystonemiddleware: Globalize authentication failure error https://review.openstack.org/359675 | 08:11 |
| *** woodster_ has quit IRC | 08:19 | |
| *** asettle has joined #openstack-keystone | 08:28 | |
| *** pcaruana has joined #openstack-keystone | 08:29 | |
| *** jed56 has joined #openstack-keystone | 08:33 | |
| *** tqtran has quit IRC | 08:34 | |
| *** dikonoor has joined #openstack-keystone | 08:45 | |
| *** atod has quit IRC | 08:49 | |
| *** code-R_ has joined #openstack-keystone | 08:50 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: Use fixtures from keystoneauth https://review.openstack.org/359642 | 08:52 |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: Use exceptions from Keystoneauth https://review.openstack.org/359705 | 08:52 |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: Remove generic client https://review.openstack.org/359706 | 08:52 |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: [WIP] Remove old method of creating a client https://review.openstack.org/359707 | 08:52 |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: [WIP] Migrate to keystoneauth https://review.openstack.org/359708 | 08:52 |
| *** code-R has quit IRC | 08:53 | |
| *** tonytan4ever has joined #openstack-keystone | 08:55 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: [WIP] Migrate to keystoneauth https://review.openstack.org/359708 | 08:55 |
| openstackgerrit | Jamie Lennox proposed openstack/python-keystoneclient: Remove unauthenticated functions https://review.openstack.org/359710 | 08:56 |
| *** tonytan4ever has quit IRC | 08:59 | |
| *** asettle has quit IRC | 09:01 | |
| *** asettle has joined #openstack-keystone | 09:06 | |
| *** asettle has quit IRC | 09:08 | |
| *** davechen has quit IRC | 09:09 | |
| openstackgerrit | Merged openstack/keystone: Fix credential update to ec2 type https://review.openstack.org/357950 | 09:10 |
| *** Gorian has quit IRC | 09:19 | |
| *** davechen has joined #openstack-keystone | 09:19 | |
| *** jaosorior is now known as jaosorior_lunch | 09:21 | |
| *** davechen has quit IRC | 09:27 | |
| *** davechen has joined #openstack-keystone | 09:33 | |
| *** davechen has left #openstack-keystone | 09:33 | |
| *** atod has joined #openstack-keystone | 09:47 | |
| *** atod has quit IRC | 09:51 | |
| openstackgerrit | Divya K Konoor proposed openstack/keystonemiddleware: Globalize authentication failure error https://review.openstack.org/359675 | 09:54 |
| *** EinstCrazy has joined #openstack-keystone | 09:55 | |
| *** tonytan4ever has joined #openstack-keystone | 09:56 | |
| *** EinstCra_ has quit IRC | 09:56 | |
| *** tonytan4ever has quit IRC | 10:00 | |
| openstackgerrit | Cao Xuan Hoang proposed openstack/keystone: TrivialFix: Remove logging import unused https://review.openstack.org/359760 | 10:05 |
| *** richm has joined #openstack-keystone | 10:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/keystone: [WIP] Testing latest u-c https://review.openstack.org/318435 | 10:10 |
| *** EinstCrazy has quit IRC | 10:16 | |
| *** asettle has joined #openstack-keystone | 10:18 | |
| openstackgerrit | Divya K Konoor proposed openstack/keystonemiddleware: Globalize authentication failure error https://review.openstack.org/359675 | 10:28 |
| *** sdake has quit IRC | 10:28 | |
| *** jaosorior_lunch is now known as jaosorior | 10:30 | |
| *** wangqun has quit IRC | 10:34 | |
| *** rodrigods has quit IRC | 10:34 | |
| *** rodrigods has joined #openstack-keystone | 10:34 | |
| *** amakarov_away is now known as amakarov | 10:41 | |
| *** dikonoor has quit IRC | 10:44 | |
| amakarov | h | 10:46 |
| amakarov | bknudson, hi! Can you please review https://review.openstack.org/#/c/309146/ ? I'd really love to have your opinion, as you had concerns and I can satisfy those only partially. | 10:49 |
| *** d0ugal has quit IRC | 10:59 | |
| *** d0ugal has joined #openstack-keystone | 10:59 | |
| openstackgerrit | Mikhail Nikolaenko proposed openstack/keystone: [WIP] Move fernet utils to backend https://review.openstack.org/356499 | 11:06 |
| samueldmq | morning keystone | 11:07 |
| openstackgerrit | Dave Chen proposed openstack/keystone: Handle the exception from creating access token properly https://review.openstack.org/359795 | 11:08 |
| openstackgerrit | Merged openstack/keystone: Add key repository uniqueness check to doctor https://review.openstack.org/358083 | 11:29 |
| *** asettle has quit IRC | 11:45 | |
| *** sigmavirus|away is now known as sigmavirus | 11:51 | |
| *** jaosorior has quit IRC | 11:51 | |
| *** jaosorior has joined #openstack-keystone | 11:52 | |
| *** tonytan4ever has joined #openstack-keystone | 11:57 | |
| *** jpena is now known as jpena|lunch | 12:01 | |
| *** tonytan4ever has quit IRC | 12:01 | |
| *** markvoelker has joined #openstack-keystone | 12:25 | |
| *** julim has joined #openstack-keystone | 12:28 | |
| openstackgerrit | Mikhail Nikolaenko proposed openstack/keystone: [WIP] Move fernet utils to backend https://review.openstack.org/356499 | 12:31 |
| *** pauloewerton has joined #openstack-keystone | 12:35 | |
| *** asettle has joined #openstack-keystone | 12:40 | |
| *** bjolo has joined #openstack-keystone | 12:43 | |
| *** edmondsw has joined #openstack-keystone | 12:50 | |
| lbragstad | henrynash do you have links to the versionedobjects implementations? | 12:51 |
| dstanek | morning samueldmq | 12:52 |
| *** markvoelker has quit IRC | 12:53 | |
| lbragstad | henrynash was this it? https://review.openstack.org/#/c/330674/ | 12:53 |
| *** tonytan4ever has joined #openstack-keystone | 12:58 | |
| lbragstad | dstanek I see three different patches for cache region fixes... | 13:00 |
| lbragstad | dstanek is there a particular one you need reviewed? | 13:00 |
| *** tonytan4ever has quit IRC | 13:02 | |
| *** ddieterly has joined #openstack-keystone | 13:03 | |
| dstanek | Lb | 13:03 |
| dstanek | lbragstad: fixed the glitch | 13:04 |
| lbragstad | dstanek which glitch? | 13:05 |
| *** jpena|lunch is now known as jpena | 13:06 | |
| dstanek | too many reviews. I abandoned a few | 13:06 |
| lbragstad | dstanek ah - looks like https://review.openstack.org/#/c/349704/ is the only one left | 13:06 |
| *** ddieterly has quit IRC | 13:07 | |
| *** asettle has quit IRC | 13:09 | |
| *** asettle has joined #openstack-keystone | 13:10 | |
| samueldmq | dstanek is that your patch to invalidate across regions ? | 13:11 |
| samueldmq | across processes ? | 13:11 |
| samueldmq | I am not getting how that works | 13:13 |
| *** woodster_ has joined #openstack-keystone | 13:23 | |
| *** BjoernT has joined #openstack-keystone | 13:23 | |
| breton | samueldmq: across processes | 13:25 |
| *** openstackgerrit has quit IRC | 13:26 | |
| *** openstackgerrit has joined #openstack-keystone | 13:26 | |
| breton | samueldmq: a cache entry is a pair of (key, value) | 13:27 |
| dstanek | breton: ++ | 13:28 |
| breton | samueldmq: key gets mangled and pair {mangled_key: value} is set to backend | 13:28 |
| breton | samueldmq: with David's patch a cache entry is a pair of {key+region_id, value} | 13:28 |
| dstanek | samueldmq: lol. answered you in the wrong room | 13:28 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updated from global requirements https://review.openstack.org/359513 | 13:30 |
| samueldmq | dstanek: I did the same yesterday after meeting | 13:30 |
| breton | samueldmq: region_id is ephemeral. It is generated and is stored in the memcached directly, without mangling | 13:30 |
| samueldmq | breton: dstanek : so we're talking about process cache ? or a common cache server | 13:30 |
| breton | samueldmq: when invalidation happens, new region_id is generated | 13:30 |
| samueldmq | that is shared for all processes / keystone servers ? | 13:30 |
| dstanek | samueldmq: memcached | 13:31 |
| lbragstad | it should work across processes | 13:31 |
| samueldmq | dstanek: so let's say a separate memcache server for it | 13:31 |
| breton | samueldmq: so all old {key+old_region_id: value} pairs are forgotten | 13:31 |
| samueldmq | and old caches were still 'valid' because the key matches .. | 13:31 |
| breton | samueldmq: because there is no more old_region_id | 13:31 |
| dstanek | samueldmq: it's all on the same server | 13:32 |
| *** tqtran has joined #openstack-keystone | 13:32 | |
| samueldmq | dstanek: but it's a separate backend, used by keystone, right ? | 13:33 |
| samueldmq | common to all processes | 13:33 |
| samueldmq | in that server | 13:33 |
| dstanek | no, I'm just changing the keys in the existing backend | 13:34 |
| *** sdake has joined #openstack-keystone | 13:34 | |
| samueldmq | dstanek: ok, separate case: we have a proxy, multiple servers behind it | 13:35 |
| samueldmq | one server gets a delete call, invalidate the cache locally | 13:35 |
| *** tonytan4ever has joined #openstack-keystone | 13:35 | |
| dstanek | instead of key/value pairs I made it key: id/value | 13:35 |
| samueldmq | other servers behind the proxy are incosistent | 13:35 |
| lbragstad | that shouldn't be the case of the memcache deployment is sharded, right? | 13:35 |
| dstanek | yes, that's what was happening | 13:36 |
| samueldmq | ok, so in that case I just said, memcache should be shared across keystone servers behind the proxy | 13:36 |
| *** tqtran has quit IRC | 13:37 | |
| samueldmq | so memcache could be put in a separate server, and used by all of them | 13:37 |
| *** sdake_ has joined #openstack-keystone | 13:37 | |
| dstanek | samueldmq: it had to be our tour cache is already broken. | 13:38 |
| lbragstad | samueldmq I believe if you list your memcache servers in the same order in each keystone config, keystone will automatically shard the memcache contents across each | 13:38 |
| dstanek | memcached is designed as a shared cache | 13:38 |
| samueldmq | hmm like a galera db ? | 13:39 |
| samueldmq | or just in a common place and shared to all servers ? | 13:39 |
| * samueldmq goes to https://memcached.org/ rather than keep asking dumb questions | 13:40 | |
| *** sdake has quit IRC | 13:40 | |
| dstanek | samueldmq: :) memcached you be setup as a separate cluster | 13:41 |
| bknudson | we're not going to be able to have a memcache cluster span separate dcs. | 13:41 |
| *** ddieterly has joined #openstack-keystone | 13:41 | |
| samueldmq | dstanek: so yes, I remmeber to set up that once for horizon | 13:41 |
| samueldmq | using pki | 13:41 |
| samueldmq | dstanek: I had to setup a separate server for memcache | 13:41 |
| samueldmq | iirc | 13:41 |
| dstanek | bknudson: then your cache expiration times need to be shorter since invalidation won't work | 13:43 |
| dstanek | or no cache at all if you can manage it | 13:44 |
| *** ddieterly has quit IRC | 13:46 | |
| *** ddieterly has joined #openstack-keystone | 13:46 | |
| *** su_zhang has joined #openstack-keystone | 13:53 | |
| *** raildo has joined #openstack-keystone | 13:54 | |
| *** asettle has quit IRC | 14:05 | |
| samueldmq | dstanek: so you do {key+region_id: value} | 14:06 |
| samueldmq | and you change the region_id, right? | 14:06 |
| dstanek | exactly | 14:07 |
| samueldmq | then region_id needs to be shared by everyone | 14:07 |
| samueldmq | dstanek: you use memcached to store the region_id ? | 14:09 |
| dstanek | yes, the way all processes use the same value | 14:10 |
| dstanek | samueldmq: I'm a little short on the explanation because I'm on my phone at breakfast :) | 14:10 |
| samueldmq | dstanek: that's okay, take your time | 14:10 |
| breton | samueldmq: yes, region_id is stored in the cache backend | 14:17 |
| *** nkinder has joined #openstack-keystone | 14:18 | |
| *** spedione|AWAY is now known as spedione | 14:19 | |
| breton | samueldmq: and fetched from there when get(key) is called | 14:19 |
| *** Jehane has left #openstack-keystone | 14:20 | |
| *** ravelar has joined #openstack-keystone | 14:22 | |
| *** michauds has joined #openstack-keystone | 14:24 | |
| samueldmq | breton: what does expiration_time=-1 mean? | 14:28 |
| samueldmq | that's used for region id | 14:28 |
| *** michauds_ has joined #openstack-keystone | 14:28 | |
| *** michauds has quit IRC | 14:28 | |
| breton | samueldmq: my guess is that memcache keys expire after some time. region_id should not expire unless invalidated. | 14:29 |
| *** ayoung has joined #openstack-keystone | 14:31 | |
| *** ChanServ sets mode: +v ayoung | 14:31 | |
| *** michauds_ is now known as michauds__ | 14:31 | |
| *** michauds__ is now known as michauds | 14:32 | |
| breton | samueldmq: so -1 disabled expiry | 14:32 |
| * breton should finish his thoughts | 14:32 | |
| stevemar | o/ | 14:39 |
| mtreinish | stevemar: https://review.openstack.org/#/c/359931/ | 14:48 |
| dstanek | samueldmq: does that make sense? | 14:52 |
| stevemar | mtreinish: https://review.openstack.org/#/c/355723/ | 14:54 |
| mtreinish | stevemar: https://review.openstack.org/#/c/357543/ | 14:54 |
| *** edtubill has joined #openstack-keystone | 14:55 | |
| lbragstad | dstanek how come we return a function in key_mangler_factory()? | 14:55 |
| dstanek | lbragstad: the keymangler need to be a function | 14:56 |
| dstanek | i use the factory to create it using a closure so it has access to the things passed into the factory | 14:56 |
| lbragstad | interesting | 14:57 |
| breton | dstanek: maybe you should put all that as comments | 14:58 |
| dstanek | for some reason that patch is failing | 15:00 |
| *** jistr is now known as jistr|mtg | 15:00 | |
| lbragstad | dstanek looks like a couple of the jobs failed to stand up | 15:01 |
| dstanek | lbragstad: yeah, maybe directory problems? http://logs.openstack.org/04/349704/11/check/gate-grenade-dsvm-neutron-ubuntu-trusty/6bd4788/logs/devstack-gate-cleanup-host.txt | 15:01 |
| lbragstad | [Errno 32] Broken pipe | 15:02 |
| *** haplo37__ has joined #openstack-keystone | 15:03 | |
| openstackgerrit | Dolph Mathews proposed openstack/keystone: Reduce log level of Fernet key count message https://review.openstack.org/359941 | 15:04 |
| lbragstad | notmorgan henrynash dolphm dstanek will you be availavle to discuss the trigger conversation from yesterday? | 15:04 |
| dstanek | i'm here all date except 2:30-3:30 EST when i pick up my kids | 15:06 |
| *** code-R_ has quit IRC | 15:07 | |
| lbragstad | dstanek cool - we will probably need to air that out in order to continue with the encrypted credential implementation | 15:08 |
| aloga | ( stevemar would you be available in about 10 minutes for a short discussion ) | 15:10 |
| aloga | ( I'm currently in the middle of a meeting :-( ) | 15:10 |
| henrynash | lbragstad: i am here for the next 30 mins, then afk until later. Back on around 4pm EST | 15:11 |
| lbragstad | ok | 15:12 |
| *** hockeynut has joined #openstack-keystone | 15:16 | |
| *** david-lyle has joined #openstack-keystone | 15:17 | |
| stevemar | aloga: i can be | 15:20 |
| *** tonytan4ever has quit IRC | 15:21 | |
| *** asettle has joined #openstack-keystone | 15:21 | |
| *** tonytan4ever has joined #openstack-keystone | 15:24 | |
| aloga | stevemar: :) | 15:25 |
| *** david-lyle_ has joined #openstack-keystone | 15:25 | |
| *** sdake_ has quit IRC | 15:25 | |
| rodrigods | stevemar, hey... available to rapidly discuss something that may affect OSC? | 15:25 |
| *** david-lyle_ has quit IRC | 15:26 | |
| *** hockeynu_ has joined #openstack-keystone | 15:27 | |
| *** hockeynut has quit IRC | 15:29 | |
| aloga | stevemar: I have to leave, so nevermind, thanks anyway | 15:29 |
| *** Ephur has joined #openstack-keystone | 15:29 | |
| *** marekd2 has joined #openstack-keystone | 15:29 | |
| *** slberger has joined #openstack-keystone | 15:32 | |
| *** hockeynu_ has quit IRC | 15:36 | |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Unified delegation model https://review.openstack.org/208488 | 15:36 |
| *** sdake has joined #openstack-keystone | 15:36 | |
| *** BjoernT has quit IRC | 15:37 | |
| stevemar | aloga: same, we can catch up later | 15:38 |
| *** hockeynut has joined #openstack-keystone | 15:40 | |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Unified delegation assignment driver https://review.openstack.org/291318 | 15:41 |
| *** Ephur has quit IRC | 15:43 | |
| *** code-R has joined #openstack-keystone | 15:44 | |
| *** tonytan_brb has joined #openstack-keystone | 15:44 | |
| openstackgerrit | Alexander Makarov proposed openstack/keystone: Unified delegation trust driver https://review.openstack.org/291871 | 15:47 |
| *** tonytan4ever has quit IRC | 15:48 | |
| *** awayne has joined #openstack-keystone | 15:49 | |
| *** su_zhang has quit IRC | 15:53 | |
| *** su_zhang has joined #openstack-keystone | 15:53 | |
| *** adrian_otto has joined #openstack-keystone | 15:55 | |
| *** su_zhang has quit IRC | 15:57 | |
| *** dikonoor has joined #openstack-keystone | 16:01 | |
| *** gyee has joined #openstack-keystone | 16:03 | |
| *** jistr|mtg is now known as jistr | 16:08 | |
| *** chrisshattuck has joined #openstack-keystone | 16:12 | |
| lbragstad | dstanek our sqlite database is initialized for every run right? | 16:15 |
| dstanek | lbragstad: i believe that's true, yes | 16:15 |
| lbragstad | er - every time we run tests - out tests should stand up a new sqlite database | 16:15 |
| lbragstad | ok | 16:15 |
| lbragstad | interesting | 16:15 |
| dstanek | in setUp | 16:15 |
| dstanek | ? | 16:16 |
| bknudson | it's the same for the mysql and postgresql live tests. you get a new db every test. | 16:16 |
| lbragstad | in keystone.tests.unit.test_sql_upgrade.SqlContractSchemaUpgradeTests the setUp is failing because it's saying a trigger is already created | 16:16 |
| lbragstad | which is weird because that would mean we are running the expand repo upgrade in out setup twice somewhere | 16:17 |
| dolphm | lbragstad: in your patch? | 16:17 |
| lbragstad | dolphm not sure if this is related directly to my patch - or one of the previous sql upgrade testing patches | 16:17 |
| dstanek | lbragstad: the upgrade tests may be a little special since we want to use the same DB to go from one version to another. is it possible it's being executed twice in the same test run? | 16:17 |
| lbragstad | sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) trigger credential_read_only already exists [SQL: "\nCREATE TRIGGER credential_read_only BEFORE UPDATE ON credential\nFOR EACH ROW\nBEGIN\n SELECT RAISE (ABORT, 'Credential migration in progress. Cannot pe | 16:18 |
| lbragstad | rform writes to credential table.');\nEND;\n"] | 16:18 |
| lbragstad | the flow is that it will upgrade the migrate_repo (which is the legacy repository) to version 109 | 16:18 |
| lbragstad | then it will go ahead and move on to the expand_repo | 16:18 |
| lbragstad | but when it goes to do that - it fails saying that a specific trigger already exists | 16:19 |
| lbragstad | which is strange because it shouldn't have been created before the migrate_repo was initialized | 16:19 |
| lbragstad | unless we are initializing the legacy repo and running upgrade on the expand repo twice somewhere? | 16:20 |
| lbragstad | this is the setUp https://github.com/openstack/keystone/blob/be7307b20b148892f611f0c3e99b973ed778655d/keystone/tests/unit/test_sql_upgrade.py#L1583-L1595 | 16:21 |
| dstanek | lbragstad: how does upgrade() know what repo to upgrade? | 16:23 |
| dolphm | lbragstad: can you update the commit message on https://review.openstack.org/#/c/355618/ vs https://review.openstack.org/#/c/317169/ -- i keep opening the wrong one because they have the same 1 line summary | 16:23 |
| lbragstad | dstanek you have to initialize it https://github.com/openstack/keystone/blob/be7307b20b148892f611f0c3e99b973ed778655d/keystone/tests/unit/test_sql_upgrade.py#L1590 | 16:23 |
| dolphm | dstanek: it depends on which repo was last initialized :-/ | 16:23 |
| *** chrisshattuck has quit IRC | 16:23 | |
| dstanek | uggg | 16:23 |
| dolphm | dstanek: ideally, we'd have 4 different upgrade methods | 16:23 |
| dolphm | dstanek: agree 110% | 16:23 |
| dolphm | dstanek: i believe that piece is still in review, though | 16:24 |
| lbragstad | self.upgrade() will behave differently depending on what repo was initialized last | 16:24 |
| dolphm | so: luck | 16:24 |
| dstanek | looking at the code you can pass in the repo, it's just not required | 16:24 |
| lbragstad | dolphm what is this luck you speak of? | 16:25 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: WIP: Implement encryption of credentials at rest https://review.openstack.org/317169 | 16:25 |
| dstanek | although you have to pass in both 'repository' and 'current_schema' | 16:25 |
| dolphm | lbragstad: sorry, i guess you don't have any | 16:25 |
| dstanek | lbragstad: can you verify that the correct repo is being upgraded? | 16:26 |
| lbragstad | dstanek yep | 16:26 |
| dstanek | lbragstad: what patch is failing? | 16:26 |
| lbragstad | dstanek let me grab a paste | 16:26 |
| lbragstad | dstanek https://review.openstack.org/#/c/355618 | 16:26 |
| dolphm | if we use the migration helpers to manage things, we could implement sanity checks there that things are being upgraded in order, and we'd probably be able to debug this immediately | 16:26 |
| lbragstad | dstanek http://cdn.pasteraw.com/hk6h0uf6t2e7fzct71ovhphud2cwkxv | 16:27 |
| lbragstad | dstanek yields - http://cdn.pasteraw.com/f7zcwpl36vvonx8nulcwng612dyl8y8 | 16:27 |
| lbragstad | so there we can see that the legacy repository is being updated first | 16:28 |
| lbragstad | and the initial db version for that repository is 66, which makes sense | 16:28 |
| lbragstad | then it gets updated to 109 | 16:28 |
| lbragstad | then we move on to initializing the expand repository - the versions make sense, then it blows up | 16:28 |
| *** rcernin has quit IRC | 16:29 | |
| dstanek | lbragstad: actually it doesn't look like test_sql_upgrade is using the database fixture | 16:30 |
| lbragstad | :( | 16:30 |
| lbragstad | so that would mean that we aren't using a fresh db every time?! | 16:30 |
| dstanek | lbragstad: or that i can't immediately say we are | 16:31 |
| dstanek | if we are not at all then i would have expected this to already be broken | 16:31 |
| *** marekd2 has quit IRC | 16:31 | |
| *** hockeynu_ has joined #openstack-keystone | 16:31 | |
| *** marekd2 has joined #openstack-keystone | 16:32 | |
| *** haplo37__ has quit IRC | 16:33 | |
| lbragstad | dstanek yeah - how would that not fail with just running migrations normally? | 16:33 |
| *** esp has joined #openstack-keystone | 16:34 | |
| *** tqtran has joined #openstack-keystone | 16:35 | |
| *** hockeynut has quit IRC | 16:35 | |
| *** tesseract- has quit IRC | 16:35 | |
| *** marekd2 has quit IRC | 16:36 | |
| *** tqtran has quit IRC | 16:39 | |
| *** hockeynut has joined #openstack-keystone | 16:45 | |
| *** hockeynu_ has quit IRC | 16:45 | |
| *** roxanagh_ has joined #openstack-keystone | 16:47 | |
| *** jaosorior has quit IRC | 16:47 | |
| lbragstad | I think if we are going to have multiple repositories - we should make repository required for _migrate() | 16:47 |
| *** asettle has quit IRC | 16:50 | |
| *** asettle has joined #openstack-keystone | 16:50 | |
| dstanek | lbragstad: i'm getting a DatabaseAlreadyControlled error on your test. it looks like maybe due to the multiple repositories? | 16:53 |
| lbragstad | dstanek actually - just figured it out | 16:53 |
| dstanek | what was it? | 16:53 |
| lbragstad | dstanek it was because I broke the sqlite triggers into two separate triggers | 16:53 |
| lbragstad | one for update and one for insert | 16:53 |
| lbragstad | they were both named the same | 16:53 |
| lbragstad | which was causing the issue | 16:53 |
| lbragstad | dstanek but i can confirm that I get a database controlled error, too | 16:54 |
| lbragstad | not sure what that is all about yet | 16:54 |
| dstanek | i think the repos are all using the same repository id | 16:54 |
| *** asettle has quit IRC | 16:55 | |
| samueldmq | dstanek: I am almost understanding it 100% | 16:55 |
| samueldmq | just left a couple of questions in the review | 16:55 |
| *** mordred has quit IRC | 16:56 | |
| *** tonytan_brb has quit IRC | 16:57 | |
| *** tonytan4ever has joined #openstack-keystone | 16:57 | |
| *** code-R has quit IRC | 16:58 | |
| *** code-R has joined #openstack-keystone | 16:59 | |
| *** mordred has joined #openstack-keystone | 16:59 | |
| *** lamt has quit IRC | 16:59 | |
| dstanek | samueldmq: i just replied | 17:00 |
| *** su_zhang has joined #openstack-keystone | 17:01 | |
| *** hockeynut has quit IRC | 17:10 | |
| rderose | notmorgan stevemar: versioned objects aren't the only alternative for rolling upgrades, we also have a read-only approach | 17:11 |
| *** hockeynut has joined #openstack-keystone | 17:12 | |
| *** code-R_ has joined #openstack-keystone | 17:12 | |
| rderose | notmorgan: https://review.openstack.org/#/c/349700/ | 17:12 |
| rderose | notmorgan stevemar: I'd like to prove out the trigger strategy first though. But if we can't agree, the read-only option gives us a quick win. | 17:14 |
| rderose | * quick and easy :) | 17:14 |
| stevemar | rderose: true, forgot about tht | 17:14 |
| *** code-R has quit IRC | 17:15 | |
| rderose | stevemar: it gets us mostly there for rolling upgrades and is a quick fix in the meantime | 17:15 |
| rderose | stevemar: anyway, something to keep in mind | 17:15 |
| *** thumpba has joined #openstack-keystone | 17:17 | |
| *** thumpba has quit IRC | 17:22 | |
| *** thumpba has joined #openstack-keystone | 17:23 | |
| *** rcernin has joined #openstack-keystone | 17:23 | |
| *** tqtran has joined #openstack-keystone | 17:28 | |
| *** dikonoor has quit IRC | 17:28 | |
| *** code-R_ has quit IRC | 17:34 | |
| *** lamt has joined #openstack-keystone | 17:43 | |
| samueldmq | dstanek: kk, looking | 17:46 |
| *** gyee has quit IRC | 17:47 | |
| *** jpena is now known as jpena|away | 17:50 | |
| *** aswadr_ has quit IRC | 17:51 | |
| *** ddieterly is now known as ddieterly[away] | 17:55 | |
| notmorgan | rderose: as said in the convo yesterday, triggers are generally a bad idea wrt how openstack works and leveraging an ORM. | 17:56 |
| notmorgan | rderose: also it is creating yet another method of doing something that is a unique snowflake compared to the other mechanisms (as much as I am not a fan of them) already in use and accepted by operators | 17:58 |
| notmorgan | writing DDL-specific code is very likely to result in subtle bugs/difficult to maintain code even if it's just for "transitional" states. | 17:59 |
| dolphm | notmorgan: "generally a bad idea wrt how openstack works and leveraging an ORM" -- can you elaborate on why they're a bad idea? yes, they can't be managed by the ORM, but they also don't interfere | 17:59 |
| rderose | notmorgan: the trigger approach seemed to be a common way of dealing with rolling upgrades | 18:00 |
| dolphm | notmorgan: the operators we've spoken to (some VERY large) were enthused, but it was a small sample | 18:00 |
| notmorgan | dolphm: because we are using an ORM that doesn't support it. We're adding in code that is highly dependant on advanced features of the RDBMS that we have historically been extremely lax on specifygin versions that are supported | 18:00 |
| dolphm | notmorgan: and the same fragility argument can be made for maintaining data layer integrity in the application layer | 18:00 |
| samueldmq | dstanek: another question tehre | 18:01 |
| dolphm | notmorgan: i'm not aware of any ORM that supports trigger definitions - are you? | 18:01 |
| notmorgan | dolphm: since we do not test with a variety of versions (we don't) and we don't specify, we have little guarantees of making sure that "Features" used in the triggers will work. | 18:01 |
| notmorgan | dolphm: hit and miss depending on how they are built | 18:01 |
| dolphm | notmorgan: not sure what you mean by "extremely lax on specifygin versions that are supported" though | 18:01 |
| notmorgan | what version of mysql is required for openstack to run | 18:02 |
| notmorgan | minimum | 18:02 |
| notmorgan | same question for PGSQL | 18:02 |
| dolphm | notmorgan: oh, sure. have triggers broken backwards compatibility in the past or what? | 18:02 |
| notmorgan | dolphm: they haven't been used in openstack, but you need to make sure whatever definition you're using conforms to the mysql capabilities | 18:03 |
| notmorgan | same for pgsql | 18:03 |
| dolphm | notmorgan: i imagine no one is going to be upgrading their database version while triggers are in place though, as they're only in place temporary | 18:03 |
| notmorgan | and the code is going to be DDL specific | 18:03 |
| dolphm | notmorgan: ++ we're testing triggers against sqlite, postgres, and mysql in the gate | 18:03 |
| notmorgan | so code for PGSQL, MySQL, etc | 18:03 |
| dolphm | but i guess what versions of those we test are up to infra | 18:03 |
| notmorgan | yep. | 18:03 |
| dolphm | notmorgan: ++ you should check out lance's review | 18:03 |
| dolphm | he's defining 6 triggers for one migration | 18:04 |
| notmorgan | and we also do not test with galera | 18:04 |
| dolphm | focusing on mysql first, but you can see the pattern | 18:04 |
| notmorgan | while galera doesn't say it breaks with triggers, they are not well tested afaict compared to the base featuresets | 18:04 |
| notmorgan | same w/ PGSQL clustering | 18:04 |
| dolphm | notmorgan: do you expect galera or percona to behave any differently with regard to triggers? that hasn't been a concern that's come up thus far | 18:05 |
| notmorgan | i always expect galera/percona to behave subtly different than a non-clustered environment | 18:05 |
| notmorgan | until proven otherwise. | 18:05 |
| notmorgan | usually it doesn't break anything but most places would be testing on a base that mirrors that | 18:05 |
| dolphm | well, worst case, you could do an offline upgrade and the triggers will never fire | 18:05 |
| notmorgan | and could adjust things if an error state/weirdness happens | 18:05 |
| notmorgan | basically, if we only supported MySQL and had a range of versions we specified as supported | 18:07 |
| notmorgan | (more on that last point) | 18:07 |
| notmorgan | I would feel a lot better about the DDL specific code and triggers | 18:07 |
| dolphm | notmorgan: well, mysql is obviously the first class citizen. if we want to gate against a second version of mysql or something, we could totally do that. we'll also be gaining a CI job against OSA (using galera) sometime after release | 18:08 |
| stevemar | jamielennox: SO MUCH DELETE https://review.openstack.org/#/c/359708/2 | 18:08 |
| notmorgan | and as much as i trust lance and henrynash to write good code; when i hear folks like mordred say things to the effect of triggers being the less correct approach (especially since he knows how the stuff internal to mysql works), I tend to err to that sid to start | 18:09 |
| rderose | jamielennox nice! | 18:09 |
| stevemar | notmorgan: dolphm oh nice we're talking about this | 18:09 |
| *** Gorian|work has joined #openstack-keystone | 18:09 | |
| dolphm | notmorgan: i'd love to hear mordred's thoughts (what makes them "less correct" than an application-layer solution?) | 18:09 |
| notmorgan | dolphm: from my discussions it's simply down to less defined versions of the RDBMS backend(s) and that the application layer solution is common regardless of the DDL | 18:10 |
| stevemar | notmorgan: mordred using versionedobjects seems like overkill for keystone | 18:11 |
| rderose | stevemar: agree | 18:11 |
| dolphm | notmorgan: i imagine that's a concern that would be resolved with sufficient testing though, no? | 18:11 |
| notmorgan | mix those two things together, it really is openstack's architecture, use of an ORM to abstract the RDBMS differences, etc, leads to application layer being the most maintainable/supportable/straightforward | 18:11 |
| rderose | stevemar notmorgan dolphm: has any of the core openstack projects implemented rolling upgrades? | 18:11 |
| stevemar | rderose: neutron ? | 18:12 |
| dolphm | notmorgan: exercise each trigger on the database variants we care about | 18:12 |
| dolphm | stevemar: they're trying | 18:12 |
| stevemar | anyone else? | 18:12 |
| dolphm | nova has theoretical support, so far as i can ascertain | 18:12 |
| *** hockeynut has quit IRC | 18:13 | |
| dolphm | ironic has punted to a future release because versioned objects are a lot of work | 18:13 |
| notmorgan | dolphm: possible. in all honesty I would prefer DDL-specific code that could make use of mysql optimisations etc vs the ORM | 18:13 |
| dolphm | i don't think glance has started, but i think triggers would apply there easily | 18:13 |
| dolphm | and none of this is applicable to swift | 18:13 |
| notmorgan | dolphm: and in that case I would have no issues (with version specifics for mysql, and a cluster setup test) of saying "triggers are viable" | 18:13 |
| *** Gorian|work has quit IRC | 18:14 | |
| *** roxanag__ has joined #openstack-keystone | 18:14 | |
| dolphm | not sure how much horizon cares (it doesn't use a db other than for "caching," right?) | 18:14 |
| notmorgan | so it comes down to: I dislike versioned objects, however, they fit openstack architecture a bit better than triggers because of ORM, limited version specification, different DDLs, etc | 18:14 |
| dolphm | they perhaps fit other services better, not necessarily ours | 18:15 |
| notmorgan | and i say this as someone who would love to simply drop support for pgsql | 18:15 |
| rderose | notmorgan: ++ | 18:15 |
| notmorgan | because it's statistically an outlier for use that nets more maintenance/headaches/etc | 18:16 |
| dstanek | notmorgan: is there a simple example of versionedobject abstracting away DB differences. i've only heard about it being used at the RPC level | 18:16 |
| notmorgan | in fact, i'd love if we *only* supported MySQL | 18:16 |
| dolphm | notmorgan: i kinda figure this might push some deployers towards postgres in the near future :-/ http://www.infoworld.com/article/3109213/open-source-tools/open-source-uproar-as-mariadb-goes-commercial.html | 18:16 |
| notmorgan | dstanek: since nova uses a conductor, in almost all cases it is RPC. | 18:16 |
| rderose | stevemar: lets only support mysql! | 18:16 |
| notmorgan | dolphm: doubtful unless oracle makes mysql even worse. | 18:16 |
| rderose | dolphm: oh great | 18:17 |
| dstanek | notmorgan: i don't see how it would work at all for ORM other then writing application code to implement similar patterns | 18:17 |
| notmorgan | dolphm: and/or percona changes. | 18:17 |
| *** roxanagh_ has quit IRC | 18:17 | |
| mordred | stevemar: I disagree. it's not about keystone. it's about openstack | 18:17 |
| *** su_zhang has quit IRC | 18:17 | |
| mordred | stevemar: if keystone was in a vacuum, sure. | 18:17 |
| mordred | stevemar: but keystone is useless without a corresponding openstack | 18:17 |
| notmorgan | mordred: ++ | 18:17 |
| mordred | stevemar: so whatever the openstack-wide solutoin is is what keystone should do | 18:17 |
| *** su_zhang has joined #openstack-keystone | 18:17 | |
| dolphm | dstanek: witness neutron's massive effort https://review.openstack.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db | 18:18 |
| *** dims has quit IRC | 18:18 | |
| rderose | mordred: but openstack doesn't have a solution for rolling upgrades | 18:18 |
| *** dims has joined #openstack-keystone | 18:18 | |
| stevemar | mordred: you're giving us a toolbox when all we want is a screw driver :P | 18:19 |
| dstanek | mordred: should there be some freedom to innovate when the accepted solution isn't ideal? | 18:19 |
| dolphm | it'd be nice if we had a proven solution for real zero downtime upgrades *before* we pushed for every project to implement a solution | 18:19 |
| samueldmq | dstanek: ok, reviewed it (took me ages!) | 18:19 |
| stevemar | dolphm: i was just gonna say that! | 18:19 |
| mordred | dstanek: sure. in the context of the existing not-quite-perfect thing | 18:19 |
| samueldmq | dstanek: I suggested to put some comments to help understanding the code | 18:19 |
| mordred | becuase whatever problems you have with it are probably also applicable to nova | 18:19 |
| rderose | and what's the accepted solution that no one has successfully implemented | 18:19 |
| dstanek | samueldmq: great, thanks | 18:19 |
| stevemar | maybe we wait and see how neutron and nova work out, and what the ops feedback is, before we claim rolling upgrade support in keystone | 18:20 |
| mordred | rderose: whatever it is that nova is doing, that has been developed in consultation with the ops community | 18:20 |
| mordred | like, that's where the ops efforts have been focued | 18:20 |
| mordred | because nova migrations are much more complex | 18:20 |
| dstanek | mordred: or that neutron mess dolph linked to? just seems like the wrong tools for abstracting away database changes | 18:20 |
| rderose | mordred: exactly | 18:20 |
| mordred | dstanek: I'd follow nova, not neutron | 18:20 |
| mordred | I belive neutorn is also trying to follow nova | 18:21 |
| mordred | but is likely not as far along | 18:21 |
| rderose | mordred: so nova needs a really complex (overkill for keystone) solution | 18:21 |
| mordred | so a bad example | 18:21 |
| stevemar | hooooly shiet https://review.openstack.org/#/q/topic:bp/adopt-oslo-versioned-objects-for-db | 18:21 |
| mordred | rderose: but the solution isn't about being a solution for keystone | 18:21 |
| mordred | it's about being a solution for _operators_ who run _openstack_ | 18:21 |
| mordred | so if you make them learn a new tool for keystone when they have a tool for nova already | 18:21 |
| dolphm | mordred: have any operators actually moved to exercising nova's rolling upgrades, or is it all still theory? | 18:21 |
| mordred | they're going to flip their shit | 18:21 |
| mordred | dolphm: tons of them, all the time | 18:21 |
| dolphm | as far as i've seen, gate testing is non-existent | 18:21 |
| mordred | aiui | 18:21 |
| *** Gorian|work has joined #openstack-keystone | 18:22 | |
| mordred | but whatever is there has gone through multiple ops summit sessions and feedback loops | 18:22 |
| dstanek | mordred: afaict they are not using it for DB stuff | 18:22 |
| notmorgan | rderose: if every project implemented their own solution (*cough* wsgi layer *cough*) it makes it hard(er) to run openstack; which is only good if you're selling "we run openstack for you because it's too hard to run" (not a good thing for openstack) | 18:22 |
| dolphm | mordred: the handful of operators i've talked to were on board (mostly at the openstack-ansible midcycle) | 18:22 |
| mordred | so at the very least starting from that point | 18:22 |
| dstanek | mordred: do you now where they are? | 18:22 |
| dstanek | notmorgan: if all solutions used the same commands then the implemenation would matter less | 18:22 |
| rderose | notmorgan: it's a good point, but there has to be some middle ground here, then full blown versioned objects | 18:23 |
| mordred | I do not - I'm just saying please don't develop a new thing in a vacuum. the problem is exactly the same | 18:23 |
| dstanek | it's really a problem if the deployer have to do things a million different ways | 18:23 |
| dolphm | dstanek: ++ that's the most painful part to me at this stage | 18:23 |
| dstanek | mordred: afaict nobody does what we want to do and we are trying to prove out the concept. maybe a ML post is in order here? | 18:23 |
| dolphm | mordred: well, it's not quite the same problem, but i appreciate the notion | 18:23 |
| dolphm | stevemar: have you already started to draft one? | 18:24 |
| notmorgan | dolphm: it kindof is the same probelem, just a separate interface to access the data. | 18:24 |
| stevemar | dolphm: no, shall i open an etherpad? | 18:24 |
| stevemar | dolphm: https://etherpad.openstack.org/p/rolling_upgrade | 18:24 |
| *** jlk has joined #openstack-keystone | 18:25 | |
| jlk | I hear there's talk of online migrations happening in here! | 18:25 |
| dolphm | notmorgan: right, it's not "exactly the same" | 18:25 |
| dolphm | jlk: ++ | 18:25 |
| mordred | look it's a jlk! | 18:25 |
| *** d34dh0r53 is now known as b3rnard0-b0n-h4r | 18:25 | |
| jlk | hi. I figured I'd try to lend an operators viewpoint on the discussion, just to make things interesting. | 18:25 |
| dolphm | jlk: you might be interested in seeing our zero downtime upgrades docs if you want to catch up http://docs.openstack.org/developer/keystone/upgrading.html | 18:26 |
| stevemar | jlk: please do! | 18:26 |
| * jlk reads | 18:26 | |
| jlk | Well, step 9 is wrong, in a pedantic way | 18:28 |
| dolphm | jlk: how so? | 18:28 |
| jlk | wait, no | 18:28 |
| jlk | I misread | 18:28 |
| rderose | Ron looks up the word pedantic | 18:28 |
| jlk | so the difference here between Nova and Keystone | 18:30 |
| dstanek | rderose: In a sentence: "Sometimes my reviews are overly pedantic" :-) | 18:30 |
| jlk | nova continues to read/write to the old location, and a background process is used to fully migrate the data | 18:30 |
| rderose | dstanek: ++ :) | 18:31 |
| jlk | whereas keystone is using triggers to move data written to the old location into the new location | 18:31 |
| dstanek | jlk: and from new to old | 18:31 |
| jlk | mirroring them | 18:31 |
| dstanek | that way we can have 2 vesions of the app running at the same time with no downtime | 18:31 |
| jlk | is the --contract the action that will remove the triggers? | 18:31 |
| dolphm | jlk: correct- nova writes to the old location from the application layer (so the new release has to know about, and detect, the old schema) | 18:31 |
| dolphm | jlk: yes, --contract drops triggers | 18:32 |
| jlk | alright | 18:32 |
| dstanek | dolphm: ++ exactly... but also has to write to the old location so the old version continues to work | 18:32 |
| dolphm | dstanek: ++ triggers maintain sync bidirectionally | 18:32 |
| jlk | operationally, this doesn't seem bad. | 18:32 |
| dolphm | so the new release writes to the new schema, triggers write to the old schema, old release reads old schema | 18:32 |
| jlk | in large environments with very active keystone, how much additional overhead do the triggers create, particularly with databases that are synced across WANs? | 18:33 |
| dolphm | jlk: operationally, the worst part might be that triggers will incur some performance cost, but they're also temporary (they only exist for the duration of the rolling upgrade process) | 18:33 |
| notmorgan | jlk: if using galera, triggers are *supposed* to fire on all nodes. | 18:33 |
| notmorgan | jlk: same as an atomic "write" | 18:33 |
| notmorgan | replication (standard) does not account for triggers directly | 18:33 |
| dstanek | jlk: does nova read from and write to the old columns in addition to the new columns during an upgrade? | 18:34 |
| notmorgan | and leans on replication to cover data changes via binlog post trigger fire | 18:34 |
| samueldmq | dstanek: finally, some comments in the tests, thanks! | 18:34 |
| dolphm | notmorgan: interesting | 18:34 |
| jlk | dstanek: that answer lies somewhere in the nova conductor code | 18:34 |
| notmorgan | (aiui) | 18:35 |
| dolphm | notmorgan: i would have thought they'd fire on one node, and the result of the entire transaction would be replicated | 18:35 |
| jlk | I don't know the internals very well | 18:35 |
| notmorgan | dolphm: since the trigger is low level, it is below the galera level iirc | 18:35 |
| jlk | but from what I understand, nova can look to the old location first, and if it isn't found there, look to a new location | 18:35 |
| notmorgan | dolphm: so it handles the write, which should fire the trigger, hence the fire on all nodes. | 18:35 |
| jlk | so yeah, from an operational and automation flow, this seems workable. | 18:36 |
| notmorgan | dolphm: since a glaera write is "write to X on each node and wait for ack" | 18:36 |
| *** su_zhang has quit IRC | 18:36 | |
| jlk | I'm no db expert, so I can't speak for the impact of divergence from Nova, but it does sound like something Nova could actually make use of in the future to simplify their migration story | 18:36 |
| dstanek | jlk: i'm curious to know what they actually do | 18:37 |
| jlk | they have other concerns due to multiple sub-components that may be at different versions | 18:37 |
| jlk | and passing messages back and forth | 18:37 |
| dolphm | jlk: that bit where nova has to try one thing first, then has to retry something totally different, sounds like an opportunity for a nasty race condition :-/ | 18:37 |
| jlk | whereas keystone is all about the db acess | 18:37 |
| jlk | dolphm: I'm likely paraphrasing badly | 18:37 |
| dolphm | jlk: well, that's my understanding too | 18:37 |
| jlk | actually | 18:37 |
| notmorgan | jlk: the only real concern *I* have is that support for declarations in triggers will differe based on DDL (PGSQL and MySQl, etc) and between versions of the given RDBMS | 18:37 |
| dstanek | jlk: you make a good case for not using the 'nova does it this way' hammer :-) | 18:38 |
| notmorgan | jlk: so the triggers must be written per-DDL since the ORM cannot abstract this. | 18:38 |
| jlk | I think it only writes to old location, and reads from old, until $SOMETHING_HAPPENS so that it starts reading from new. | 18:38 |
| dolphm | notmorgan: that's a developer problem :P | 18:38 |
| jlk | notmorgan: I think most of those words were English... | 18:39 |
| notmorgan | dolphm: it ends up being an operator problem too. | 18:39 |
| notmorgan | dolphm: just from knowning what to run and how to debug | 18:39 |
| jlk | it's an operator problem if the developer screwed it up for the DB the operator is using | 18:39 |
| notmorgan | jlk: ++ | 18:39 |
| dolphm | jlk: ++ | 18:39 |
| jlk | so test coverage is important | 18:39 |
| dolphm | jlk: agree; i don't think any service can claim rolling upgrade support (much less zero downtime) until we've got a whole lot more testing in place | 18:40 |
| dstanek | also testing major DB versions of our supported RDBMS? | 18:40 |
| dstanek | also this still allows for the 100% downtime upgrades | 18:40 |
| notmorgan | dolphm: so to address my concerns: test and specify a minimum version for the RDBMS | 18:40 |
| * dstanek thinks that just sounds bad even though it isn't | 18:40 | |
| dolphm | notmorgan: we do that today, no? | 18:40 |
| notmorgan | we test one version | 18:40 |
| dolphm | notmorgan: (the minimum version bit) | 18:40 |
| notmorgan | doesn't mean we specify the minimum really | 18:40 |
| notmorgan | we say it should be X | 18:41 |
| *** gyee has joined #openstack-keystone | 18:41 | |
| lbragstad | dstanek ++ i had to read it twice just to make sure | 18:41 |
| notmorgan | but nothing we're leaning on really is broken on older versions | 18:41 |
| jlk | honestly I like what I hear, provided y'all can get the code right. The workflow feels right, and is nice and simple. | 18:41 |
| notmorgan | i mean... 3.x mysql it owuld be broken | 18:41 |
| jlk | you know somebody is going to make you make it work on db2 | 18:41 |
| notmorgan | but make sure that whatever version of (mysql|pgsql) you're developing against is the minimum specified version | 18:42 |
| dolphm | jlk: =P | 18:42 |
| jlk | I wish I were joking | 18:42 |
| notmorgan | and/or specify a hard minimum | 18:42 |
| * notmorgan would still love to just drop the ORM completely | 18:42 | |
| dstanek | jlk: db2 users can just upgrade the old way | 18:43 |
| dstanek | notmorgan: from all the code? | 18:43 |
| jlk | that's a fair point | 18:43 |
| jlk | so, the only sad thing I see | 18:43 |
| *** sc68cal has joined #openstack-keystone | 18:43 | |
| notmorgan | dstanek: minimums should be documented | 18:43 |
| jlk | we'll have to be _already_ on Newton in order to make use of htis for Newton+ | 18:43 |
| * sc68cal also heard something about online migrations and rolling upgrades | 18:43 | |
| jlk | so I'll have at least one more "hard" migration | 18:44 |
| notmorgan | dstanek: we can just ignore people who are breaking documented minimums | 18:44 |
| jlk | unless... | 18:44 |
| jlk | wait | 18:44 |
| dstanek | notmorgan: no, i mean about removing ORMs | 18:44 |
| notmorgan | dstanek: and we need to make sure we test the documented minimum version of mysql. | 18:44 |
| jlk | so, if I lay down newton code and config (but don't restart the service), I get the new keystoe-manage | 18:44 |
| notmorgan | dstanek: oh i would LOVE to just drop it and use DDL specific/optimised sql (and only support mysql | 18:44 |
| jlk | which means, I should be able to do this? | 18:44 |
| notmorgan | ) | 18:44 |
| jlk | like Mitaka -> Newton via the live method? | 18:44 |
| notmorgan | dstanek: but that is not some argument i could win. | 18:45 |
| notmorgan | dstanek: as a way forward for keystone :P | 18:45 |
| lbragstad | jlk yeah - that should technically work since Newton code would be used to create the triggers | 18:45 |
| jlk | well I know what I'll be testing | 18:45 |
| dstanek | notmorgan: we'd just write our own, see that it's inferior and then move back to sqla | 18:45 |
| dolphm | jlk: i haven't personally reviewed our "legacy" migrations thus far (mitaka -> master), but if they're all additive, then you'll be able to do zero downtime mitaka -> newton | 18:46 |
| jlk | oh right, because you haven't drawn a line in the sand yet on what kind of migrations you allow? | 18:46 |
| notmorgan | dstanek: possibly not. but since we support > 1 RDBMS, an ORM is the only sane options | 18:46 |
| notmorgan | dstanek: and that isn't to say we wouldn't use SQL-A, just not nessicarily the ORM layer | 18:47 |
| dstanek | notmorgan: fair enough. although even if we use sqla-core we'd implement our own ORM on top to serialize data into our own objects, but that isn't a bad thing | 18:48 |
| *** adrian_otto has quit IRC | 18:48 | |
| dolphm | jlk: we have a patch in review to block different types of operations in each phase of the upgrade, if that's what you mean | 18:49 |
| jlk | nod | 18:49 |
| * dolphm has to step away for a bit | 18:49 | |
| *** amakarov is now known as amakarov_away | 18:50 | |
| notmorgan | dstanek: fair, i mean we would be building highly mysql-specific and optimised operations in that case vs a compromise that works on all (inc. sqlite) RDBMSs | 18:51 |
| dolphm | notmorgan: have you reviewed lbragstad's migration? | 18:52 |
| stevemar | mtreinish: https://review.openstack.org/#/c/359675/ | 18:52 |
| lbragstad | WIP review here - https://review.openstack.org/#/c/355618/15 | 18:55 |
| *** gordc has joined #openstack-keystone | 19:01 | |
| *** ddieterly[away] is now known as ddieterly | 19:04 | |
| *** haplo37__ has joined #openstack-keystone | 19:05 | |
| *** su_zhang has joined #openstack-keystone | 19:07 | |
| *** su_zhang has quit IRC | 19:11 | |
| *** b3rnard0-b0n-h4r is now known as d34dh0r53 | 19:14 | |
| *** ddieterly is now known as ddieterly[away] | 19:15 | |
| lbragstad | thoughts? comments? questions? | 19:16 |
| *** ddieterly[away] is now known as ddieterly | 19:18 | |
| *** sdake has quit IRC | 19:20 | |
| *** thumpba_ has joined #openstack-keystone | 19:39 | |
| *** thumpba_ has quit IRC | 19:39 | |
| *** thumpba has quit IRC | 19:41 | |
| openstackgerrit | Richard Avelar proposed openstack/keystone: POC sql query revoked tokens https://review.openstack.org/359371 | 19:41 |
| lbragstad | i think the test_walk_versions test has a bug in it - too... I have a feeling that it's not upgrading the repositories in order when doing the upgrade of the contract repository | 19:42 |
| lbragstad | it fails to remove the triggers because they don't exist | 19:42 |
| henrynash | lbragstad: back on | 19:42 |
| lbragstad | henrynash o/ | 19:42 |
| lbragstad | henrynash figured you'd be on soon | 19:42 |
| *** ravelar has quit IRC | 19:43 | |
| henrynash | lbragstad: where do we stand....? | 19:43 |
| *** ravelar has joined #openstack-keystone | 19:44 | |
| lbragstad | henrynash looks like stevemar is going to email the -dev list to get some feedback | 19:45 |
| lbragstad | henrynash but dolphm dstanek myself, notmorgan mordred rderose and jlk all visited about it | 19:46 |
| henrynash | stevemar, lbragstad: feel free to use https://review.openstack.org/#/c/357789/ as a working, simple example of the trigger approach | 19:46 |
| lbragstad | jlk was able to provide some useful info from an operator perspective | 19:46 |
| lbragstad | henrynash I do have some questions | 19:47 |
| rderose | yeah, jlk seemed to like it in the end | 19:47 |
| henrynash | lbragstad: shoot | 19:47 |
| henrynash | rderose, lbragstad: I spent today re-looking at the origional (nova-based idea), and actually talked my way back to the trigger idea as it being easier | 19:48 |
| rderose | :) | 19:48 |
| *** hockeynut has joined #openstack-keystone | 19:48 | |
| lbragstad | henrynash https://review.openstack.org/#/c/357789/4/keystone/common/sql/contract_repo/versions/002_contract_created_at_non_nullable.py | 19:49 |
| henrynash | rderose, lbragstad: well, to be accurate, it conatins a small amount of hard stuff which is very localized...and the rest is much easier | 19:49 |
| rderose | henrynash: true | 19:49 |
| lbragstad | I cannot get test_walk_versions to pass to save my life | 19:49 |
| rderose | henrynash: no matter what, there will be some pain points | 19:49 |
| lbragstad | there is always going to be pain | 19:49 |
| henrynash | lbragstad: it's failing for postgresql only? | 19:50 |
| *** ravelar has quit IRC | 19:50 | |
| lbragstad | henrynash no it fails with sqlite | 19:50 |
| lbragstad | henrynash that trace is what happens when I run it locally | 19:50 |
| henrynash | lbragstad: looking it it now | 19:50 |
| lbragstad | henrynash I don't think you're hitting that because you're using 'IF EXISTS' in the statement | 19:51 |
| lbragstad | which won't make it fail hard | 19:51 |
| lbragstad | so - it would appear that test_walk_versions isn't running the expand repository fully before running the contract repository (?) | 19:51 |
| *** asettle has joined #openstack-keystone | 19:53 | |
| henrynash | lbragstad: I did have suspiosions about that, actually...since I think I saw a warning sign sometime to say it didn't exist....I'll experiment with my patch tonight to see if I can debug it | 19:53 |
| henrynash | lbragstad: btw, in your patch, I assume you are using the triggers to block write access becuase it's too hard to embed an encrypt step as part of the stored procesure! | 19:55 |
| lbragstad | henrynash yes - | 19:55 |
| lbragstad | henrynash i have a feeling that would be impossible | 19:55 |
| henrynash | lbragstad: you are probably correct! | 19:55 |
| lbragstad | since it would have to encrypt credentials (and decrypt them) using the same exact crypto implementation and keys used by the cryptography library | 19:56 |
| lbragstad | in order to effectively copy the data back and forth from the old/new schemas | 19:56 |
| henrynash | lbragstad: that's technically what we would want do do in the trigger, but your fallback to RO for a period is also a good example of what you can do if you cannot provide the patch update required in teh trigger | 19:56 |
| *** gordc has quit IRC | 19:57 | |
| lbragstad | henrynash right - it's a strange edge case | 19:57 |
| lbragstad | for the trigger route | 19:57 |
| *** BjoernT has joined #openstack-keystone | 19:58 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 20:01 |
| *** su_zhang has joined #openstack-keystone | 20:01 | |
| *** hockeynut has quit IRC | 20:01 | |
| *** ravelar has joined #openstack-keystone | 20:02 | |
| *** marekd2 has joined #openstack-keystone | 20:02 | |
| *** hockeynut has joined #openstack-keystone | 20:03 | |
| *** ravelar has quit IRC | 20:03 | |
| *** asettle has quit IRC | 20:03 | |
| *** ravelar has joined #openstack-keystone | 20:03 | |
| *** BjoernT has quit IRC | 20:04 | |
| *** su_zhang has quit IRC | 20:06 | |
| *** marekd2 has quit IRC | 20:07 | |
| *** su_zhang has joined #openstack-keystone | 20:07 | |
| *** su_zhang has quit IRC | 20:11 | |
| *** lamt has quit IRC | 20:19 | |
| *** tqtran has quit IRC | 20:22 | |
| *** tonytan4ever has quit IRC | 20:36 | |
| *** nkinder has quit IRC | 20:39 | |
| *** tonytan4ever has joined #openstack-keystone | 20:40 | |
| *** edtubill has quit IRC | 20:41 | |
| *** su_zhang has joined #openstack-keystone | 20:42 | |
| bknudson | I didn't think I'd be able to recreate in a dev environment, but I'm seeing https://bugs.launchpad.net/keystone/+bug/1600394 in my dev system (not devstack) | 20:50 |
| openstack | Launchpad bug 1600394 in OpenStack Identity (keystone) "memcache raising "too many values to unpack"" [Critical,Confirmed] - Assigned to David Stanek (dstanek) | 20:50 |
| *** roxanag__ has quit IRC | 20:51 | |
| *** tqtran has joined #openstack-keystone | 21:01 | |
| openstackgerrit | Ron De Rose proposed openstack/keystone: Relax the requirement for mappings to result in group memberships https://review.openstack.org/358111 | 21:02 |
| lbragstad | henrynash do you happen to get a migrate.exceptions.DatabaseAlreadyControlledError at all? | 21:02 |
| lbragstad | henrynash so far - i've got my patch down to two failures. | 21:03 |
| *** roxanagh_ has joined #openstack-keystone | 21:03 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Modify sql banned operations for each of the new repos https://review.openstack.org/358723 | 21:03 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Implement encryption of credentials at rest https://review.openstack.org/355618 | 21:03 |
| lbragstad | henrynash ^ | 21:05 |
| lbragstad | that just seems to fail because of the repo upgrade order of test_walk_versions and that weird database already controlled error | 21:06 |
| lbragstad | henrynash checking out your patch locally to see if it does the same thing mine does | 21:08 |
| *** edtubill has joined #openstack-keystone | 21:10 | |
| henrynash | lbragstad: it does....and it fails in test_sql_upgrade the same way...only for sqlite | 21:16 |
| lbragstad | henrynash yup | 21:16 |
| lbragstad | henrynash i'm going to paste what I did in the review so it's persisted somewhere other than chat | 21:16 |
| lbragstad | henrynash here are the failures I saw - http://cdn.pasteraw.com/42zam2htmobhucnhuhu0gxqf04onywi | 21:18 |
| lbragstad | nad http://cdn.pasteraw.com/a7hv8t4reqpfmfuy0r7oakqg03worxc | 21:18 |
| lbragstad | and8 | 21:18 |
| lbragstad | ugh... | 21:18 |
| henrynash | lbragstad: agreed...tis a bit weird | 21:20 |
| bknudson | here's what the line is like that memcache gives back that the lib isn't expecting: http://paste.openstack.org/show/563146/ | 21:20 |
| bknudson | it goes on like that for a while. | 21:21 |
| *** roxanagh_ has quit IRC | 21:21 | |
| bknudson | happens with a single client. | 21:22 |
| henrynash | lbragstad: I did also try a create trigger, drop trigger, create trigger sequence in the expand phase and that worked....so the triiger really is getting created....(and other tests would fail if it wasn't)...it just seems that for some reason in the contract phase it's no longer there... | 21:23 |
| lbragstad | henrynash I wonder if that is because the contract phase is only attempting to upgrade the contract repo? | 21:23 |
| henrynash | lbragstad: or as you said...the contract tests are not properly running the expand phas efirst | 21:23 |
| lbragstad | henrynash that would be my wild theory | 21:23 |
| lbragstad | but that's only based on what it smells like | 21:24 |
| *** shaleh has joined #openstack-keystone | 21:24 | |
| henrynash | lbragstad: but I *tried* to run the otehr upgrades first....maybe that's not working for some resain | 21:24 |
| shaleh | ayoung: you around still? | 21:24 |
| lbragstad | henrynash where do you attempt to do that? | 21:24 |
| lbragstad | for test_walk_versions? | 21:24 |
| henrynash | lbragstad: i the setup on the ContractSchema test class | 21:25 |
| lbragstad | henrynash that's not in https://github.com/openstack/keystone/blob/be7307b20b148892f611f0c3e99b973ed778655d/keystone/tests/unit/test_sql_banned_operations.py is it? | 21:26 |
| lbragstad | doens't look like it | 21:26 |
| lbragstad | i'm trying to find where that order is enforced for test_walk_versions | 21:26 |
| lbragstad | henrynash ah - i see you do it here https://github.com/openstack/keystone/blob/be7307b20b148892f611f0c3e99b973ed778655d/keystone/tests/unit/test_sql_upgrade.py#L1584-L1595 | 21:27 |
| shaleh | stevemar: how about you? | 21:28 |
| henrynash | lbragstad: yep... | 21:33 |
| lbragstad | henrynash it doesn't look like we attempt that kind of process in https://github.com/openstack/keystone/blob/be7307b20b148892f611f0c3e99b973ed778655d/keystone/tests/unit/test_sql_banned_operations.py | 21:34 |
| lbragstad | and if test_walk_versions run against repositories individually - it would make sense that we are seeing this | 21:34 |
| henrynash | lbragstad: we do in my version....have you rebased on that? | 21:34 |
| *** nkinder has joined #openstack-keystone | 21:35 | |
| lbragstad | henrynash i have https://review.openstack.org/#/c/355618/17 based on https://review.openstack.org/#/c/358723/4 | 21:35 |
| lbragstad | henrynash is there another patch I should be aware of? | 21:35 |
| *** pauloewerton has quit IRC | 21:35 | |
| henrynash | lbragstad: no, that's enough...if you look at the contract setup of https://review.openstack.org/#/c/358723/4 you'll see I do something similar | 21:36 |
| lbragstad | henrynash line 303? | 21:37 |
| henrynash | lbragstad: yes | 21:38 |
| lbragstad | hmmm | 21:38 |
| henrynash | lbragstad: indeed | 21:38 |
| henrynash | lbragstad: I'm a bit suspsious as to whether my swapping betwen repos, upgradeing the previous ones etc. in both sql_upgrade and sql_banned is really working right all the time...I did struggle a bit to make this work at all | 21:41 |
| lbragstad | henrynash would running this concurrently make a difference? | 21:41 |
| henrynash | lbragstad: how do you mean? | 21:42 |
| lbragstad | all tests running with a single thread versus many | 21:42 |
| henrynash | lbragstad: ah, you mean the tests? hmm | 21:42 |
| henrynash | lbragstad: interesting idea | 21:42 |
| henrynash | lbragstad: would be worth experimenting with that... | 21:43 |
| *** haplo37__ has quit IRC | 21:43 | |
| lbragstad | henrynash what if one thread updates the expand repo before another thread attempts to do something with the contract repo for example? | 21:43 |
| lbragstad | henrynash well - actually | 21:43 |
| lbragstad | that wouldn't make sense | 21:43 |
| lbragstad | because i can run test_walk_versions as a single test and it still fails | 21:43 |
| henrynash | lbragstad: hmm, | 21:43 |
| lbragstad | tox -e py27 -- keystone.tests.unit.test_sql_banned_operations.TestKeystoneContractSchemaMigrationsSQLite.test_walk_versions fails for me in isolation | 21:44 |
| henrynash | lbragstad: off to mull on it....grab a shower and some late food, back on again in a while | 21:44 |
| lbragstad | henrynash sounds good | 21:44 |
| *** sdake has joined #openstack-keystone | 21:48 | |
| *** roxanagh_ has joined #openstack-keystone | 21:49 | |
| *** sdake_ has joined #openstack-keystone | 21:50 | |
| *** nkinder has quit IRC | 21:53 | |
| notmorgan | bknudson: i've been poking at that one | 21:54 |
| notmorgan | bknudson: it's a weird issue honestly | 21:54 |
| *** sdake has quit IRC | 21:54 | |
| bknudson | notmorgan: I can recreate the validate token issue consistently. | 21:55 |
| bknudson | never saw it on devstack so might be interesting to see what's different. | 21:58 |
| *** su_zhang has quit IRC | 21:58 | |
| *** ddieterly is now known as ddieterly[away] | 21:59 | |
| *** su_zhang has joined #openstack-keystone | 21:59 | |
| *** adriant has joined #openstack-keystone | 21:59 | |
| *** Ephur has joined #openstack-keystone | 22:00 | |
| bknudson | 2016-08-24 21:59:29.518 21277 ERROR keystone.common.wsgi Exception: key '1921523d6734d44e88ed58dfc76ef681a36b8e9b' for 'keystone.revoke.core:_list_events|None' gave error parsing line, line is VALUE 1921523d6734d44e88ed58dfc76ef681a3eg | 22:00 |
| lbragstad | cc dstanek ^ | 22:00 |
| bknudson | 2016-08-24 21:59:30.117 21277 ERROR keystone.common.wsgi [req-a6a7154f-de74-4090-8fa8-452d5ca33f99 bb24ac3236014d58ac34b9446b3f0b61 2163b9d114d54da1b682bbcac0608cdf - default default] key '1921523d6734d44e88ed58dfc76ef681a36b8e9b' for 'keystone.revo | 22:01 |
| bknudson | ke.core:_list_events|None' gave No serialization handler registered for type 'type' | 22:01 |
| bknudson | same key different exception | 22:01 |
| *** sdake_ has quit IRC | 22:01 | |
| notmorgan | bknudson: hmmm | 22:02 |
| notmorgan | bknudson, dstanek: responded to comments on https://review.openstack.org/#/c/358872/2 | 22:03 |
| notmorgan | have a question before I correct the 2 real blocking issues. | 22:03 |
| *** sdake has joined #openstack-keystone | 22:03 | |
| bknudson | 2016-08-24 22:07:27.820 32201 ERROR keystone.common.wsgi Exception: key '1921523d6734d44e88ed58dfc76ef681a36b8e9b' for 'keystone.revoke.core:_list_events|None' gave 'unicode' object has no attribute 'payload' | 22:08 |
| bknudson | different error again. | 22:08 |
| bknudson | I guess it's always going to be the same string for the same function call | 22:09 |
| notmorgan | yes. | 22:09 |
| notmorgan | since it's a SHA1 hash | 22:09 |
| bknudson | I don't understand why it doesn't fail all the time because keystone must be using this on every call I'm doing. | 22:10 |
| bknudson | the value that memcache returns changes | 22:11 |
| notmorgan | well | 22:11 |
| notmorgan | no | 22:11 |
| notmorgan | this is going to be related to the request_local cache (in theory) | 22:11 |
| notmorgan | as well | 22:11 |
| notmorgan | so 3 sources of data to validate: thread.local RequestCache, Memcache, DB | 22:12 |
| notmorgan | I'm guessing someone screwed up the data in the RequestLocal cache subtly when refactoring code. | 22:12 |
| *** roxanagh_ has quit IRC | 22:14 | |
| notmorgan | FFS. | 22:14 |
| bknudson | if what's stored in memcache is pickled this does look like pickle string | 22:14 |
| notmorgan | gerrit wont save my changes via the web interface | 22:15 |
| notmorgan | UGH. | 22:15 |
| *** roxanagh_ has joined #openstack-keystone | 22:15 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Filter data when deserializing RevokeEvents https://review.openstack.org/358872 | 22:16 |
| *** sileht has quit IRC | 22:16 | |
| *** spedione is now known as spedione|AWAY | 22:16 | |
| notmorgan | bknudson: hm. | 22:16 |
| notmorgan | bknudson: i mean if somehow it's getting the data from memcache and then trying to de-msgpack it | 22:16 |
| bknudson | I should be able to compare a good value vs a bad one. | 22:16 |
| bknudson | not sure if that will be interesting | 22:17 |
| *** ddieterly[away] is now known as ddieterly | 22:17 | |
| *** sileht has joined #openstack-keystone | 22:18 | |
| bknudson | notmorgan: this one's different: http://paste.openstack.org/show/563150/ | 22:18 |
| notmorgan | hmm. | 22:18 |
| notmorgan | yeah | 22:18 |
| *** marekd2 has joined #openstack-keystone | 22:19 | |
| bknudson | notmorgan: compare with http://paste.openstack.org/show/563151/ -- for some reason get <key> returns a different value. | 22:20 |
| notmorgan | the second one is a sane response/non erroring? | 22:21 |
| notmorgan | right? | 22:21 |
| notmorgan | because it looks like we are getting a response that isn't in a CachedValue wrapper object | 22:21 |
| bknudson | notmorgan: I know the first one is an exception, the 2nd one I don't know if that was ever used. | 22:22 |
| notmorgan | (hence the no attribute .payload) | 22:22 |
| bknudson | I was surprised because I thought they'd be the same. | 22:22 |
| notmorgan | it should be the same net result tbh | 22:22 |
| bknudson | I could probably turn on memcache logging if that would be interesting | 22:23 |
| notmorgan | this is the one with the unpack values from the memcache library, right? | 22:23 |
| bknudson | notmorgan: http://paste.openstack.org/show/563151/ is me connecting to memcached using netcat and doing get <key> | 22:23 |
| *** marekd2 has quit IRC | 22:24 | |
| bknudson | http://paste.openstack.org/show/563150/ is from the keystone log where I added more info to the exception (the value and the key) | 22:24 |
| * notmorgan nods. | 22:24 | |
| notmorgan | and you're debugging the memcache.py too many values to unpack (or not enough) bug? | 22:24 |
| bknudson | I'll try the memcache log and see if I can track the value for a key. | 22:24 |
| notmorgan | or is this a different bug? | 22:24 |
| *** ddieterly has quit IRC | 22:24 | |
| bknudson | yes, I was seeing ValueError: too many values to unpack | 22:25 |
| bknudson | I caught that exception and raised it as "error parsing line, line is %s" | 22:25 |
| notmorgan | where that error is coming from seems like... it's something wrong with the response from memcache server itself/the memcache libarary | 22:26 |
| bknudson | and then I added a catch to dogpile.cache.region so that it prints out the key and orig_key | 22:26 |
| bknudson | I would guess the string here http://paste.openstack.org/show/563150/ is just not a valid one. | 22:26 |
| notmorgan | hm. | 22:26 |
| bknudson | resp, rkey, flags, len = line.split() | 22:27 |
| bknudson | VALUE 302516de06a66748926f0b482c739f437a4b9015 1 710 | 22:27 |
| bknudson | VALUE 302516de06a66748926f0b482c739f437a4b9015 f80292d8939038496 | 22:27 |
| bknudson | first VALUE works, second doesn't | 22:27 |
| bknudson | I assume the len is 710 which is ok. | 22:28 |
| bknudson | interesting that the invalid one still looks like pickle data | 22:28 |
| notmorgan | right | 22:28 |
| notmorgan | and that actually should be fine. | 22:28 |
| bknudson | but the VALUE line is invalid since it doesn't have the length. | 22:29 |
| bknudson | and I assume flags is supposed to be 1 | 22:29 |
| notmorgan | yeah. | 22:29 |
| notmorgan | it's missing something | 22:29 |
| *** markvoelker has joined #openstack-keystone | 22:29 | |
| notmorgan | this really looks like a mistake in python-memcache and / or memcache server returning something weird based on that | 22:29 |
| bknudson | so the invalid one has p13 ..., whereas the valid value has p1 ... | 22:30 |
| bknudson | so it's like it only has the end | 22:30 |
| *** michauds has quit IRC | 22:30 | |
| bknudson | looks like p1 , p2, etc., is just how pickle serializes a list | 22:31 |
| bknudson | print(pickle.dumps({u'abc':u'def'})) | 22:31 |
| bknudson | print(pickle.dumps(['abc', 'def', 'ghi', 'jkl'])) | 22:31 |
| *** jrist has quit IRC | 22:33 | |
| bknudson | looks like the one printed out in the error is just the valid one with the start stuff missing. | 22:33 |
| notmorgan | yeah | 22:35 |
| *** hockeynut has quit IRC | 22:35 | |
| bknudson | so maybe the value in memcache is ok and it's the read that's getting messed up somehow | 22:35 |
| notmorgan | i'm not sure how we would get a different result from a read one time but not the next? | 22:36 |
| notmorgan | unless there is something really wonky happening. | 22:36 |
| bknudson | I'm not going to rule out something really wonky happening | 22:37 |
| notmorgan | bknudson: cosmic rays! | 22:38 |
| notmorgan | :P | 22:38 |
| bknudson | we need to bury the server much deeper. | 22:38 |
| bknudson | Time to try the memcached log. | 22:40 |
| *** jrist has joined #openstack-keystone | 22:45 | |
| bknudson | I wonder if the "keystone.revoke.core:_list_events|None" isn't hit more often because it's a large list and also it's being used all the time (in this test scenario) | 22:49 |
| notmorgan | oh wait | 22:49 |
| notmorgan | hm. oh no the issue i was thinking of would be on write | 22:49 |
| notmorgan | not read | 22:49 |
| notmorgan | =/ | 22:49 |
| *** rcernin has quit IRC | 22:53 | |
| *** bigdogstl has joined #openstack-keystone | 22:54 | |
| *** zigo has quit IRC | 22:56 | |
| bknudson | notmorgan: here's a full stack trace if that helps: http://paste.openstack.org/show/563155/ | 22:56 |
| *** sdake has quit IRC | 22:57 | |
| *** zigo has joined #openstack-keystone | 22:57 | |
| *** shaleh has quit IRC | 22:58 | |
| notmorgan | what was the line you added for debugging/catch? | 22:59 |
| *** bigdogstl has quit IRC | 22:59 | |
| notmorgan | you added it in cache.region.py right? | 22:59 |
| notmorgan | bknudson: uh. weird... from what i can tell VALUE 302516de06a66748926f0b482c739f437a4b9015 1 710 | 23:01 |
| notmorgan | should be the one erroring | 23:01 |
| notmorgan | since we're getting more values to unpack than expected | 23:01 |
| notmorgan | or.. ugh. this is weird | 23:01 |
| bknudson | there's the changes: http://paste.openstack.org/show/563156/ | 23:01 |
| notmorgan | this error doesn't make sense. | 23:03 |
| *** markvoelker has quit IRC | 23:04 | |
| notmorgan | bknudson: can you confirm how much data is in that key? | 23:07 |
| *** sdake has joined #openstack-keystone | 23:07 | |
| notmorgan | bknudson: i wonder if it's an issue with socket and not receiving all the data/abnormal early truncation | 23:07 |
| notmorgan | or something similar (then again... WHY is it "too many values") | 23:07 |
| notmorgan | this is bothering me. | 23:08 |
| bknudson | 35371 Aug 24 22:52 test.txt | 23:08 |
| bknudson | VALUE 1921523d6734d44e88ed58dfc76ef681a36b8e9b 1 35308 | 23:08 |
| notmorgan | hm, ok yeah that shouild be a sane value size | 23:08 |
| bknudson | I wrote the value to a file since it was too big to look at in the window | 23:08 |
| notmorgan | we've done things with 65K+ so that shouldn't be an issue | 23:08 |
| bknudson | that appears to be the entire revocation list or event list or whatever | 23:08 |
| notmorgan | right | 23:08 |
| bknudson | although there's only 146 events | 23:09 |
| bknudson | that's about 241 bytes per event | 23:09 |
| notmorgan | ok so stupid question: can you get the key directly with python-memcache (not through dogpile) | 23:09 |
| bknudson | I'll give it a shot. | 23:10 |
| *** tqtran has quit IRC | 23:10 | |
| *** atod has joined #openstack-keystone | 23:11 | |
| notmorgan | bknudson: because dogpile doesn't do anything interesting besides .get/.set | 23:12 |
| bknudson | >>> mc.get('1921523d6734d44e88ed58dfc76ef681a36b8e9b') | 23:18 |
| bknudson | ([<keystone.models.revoke_model.RevokeEvent object at 0x7ff6aad18650>, <keystone.models.revoke_model.RevokeEvent object at 0x7ff6a566c810>, <keystone.models.revoke_model.RevokeEvent object at 0x7ff6a566c850>, <keystone.models.revoke_model.RevokeEvent object at 0x7ff6a566c890>, <keystone.models.revoke_model.RevokeEvent object at 0x7ff6a566c8d0>, <keystone.models.revoke_model.RevokeEvent object at 0x7ff6a566c910>, <keyston | 23:18 |
| bknudson | looks like it automatically deserializes. | 23:18 |
| bknudson | so I assume I'd get an exception if it failed to get the value | 23:18 |
| notmorgan | as would be correct for pickle | 23:18 |
| notmorgan | also at the end of the datastructure you should have a timestamp | 23:19 |
| bknudson | >>> len(mc.get('1921523d6734d44e88ed58dfc76ef681a36b8e9b')[0]) | 23:19 |
| bknudson | 146 | 23:19 |
| notmorgan | ([values], ts) basically | 23:19 |
| bknudson | {'ct': 1472080588.99701, 'v': 1}) | 23:19 |
| notmorgan | yep | 23:19 |
| notmorgan | ok | 23:19 |
| notmorgan | that is 100% valid | 23:19 |
| bknudson | it's kind of interesting it deserializes because it has to have access to the keystone.models | 23:20 |
| notmorgan | and how it should look. now i just don't know why it's getting too many values to un... | 23:20 |
| bknudson | I didn't import that | 23:20 |
| notmorgan | but you have keystone in ytour python path | 23:20 |
| notmorgan | right? | 23:20 |
| bknudson | sure. | 23:20 |
| notmorgan | pickle does magic | 23:20 |
| notmorgan | it's why it's so bad | 23:20 |
| bknudson | y, scary. | 23:20 |
| *** edmondsw has quit IRC | 23:20 | |
| notmorgan | so... i want to try something | 23:21 |
| notmorgan | see if the issue goes away. | 23:21 |
| notmorgan | but it requires a keystone restart >.< | 23:21 |
| bknudson | this is a dev box so I can restart keystone any time. | 23:21 |
| notmorgan | but i worry about restarts and losing duplicatrion of the issue | 23:21 |
| bknudson | I've restarted several times and had no problem recreating | 23:21 |
| notmorgan | but in short, stop using the memcachepool backend | 23:21 |
| notmorgan | use the normal dogpile memcache backend | 23:22 |
| bknudson | the key for keystone.revoke.core:_list_events|None isn't going to change. | 23:22 |
| bknudson | y, that's an easy change. | 23:22 |
| notmorgan | because the pool does hackery in the memcache library | 23:22 |
| bknudson | I've got a change up to devstack and our installer to switch to that already | 23:22 |
| bknudson | notmorgan: https://review.openstack.org/#/c/357411/ | 23:23 |
| bknudson | ^ devstack change | 23:23 |
| notmorgan | nice | 23:23 |
| jamielennox | bknudson, notmorgan: does that mean in general we want to use dogpile.memcached instead of oslo.memcached_pool? | 23:24 |
| notmorgan | jamielennox: possibly | 23:24 |
| notmorgan | if you're not using eventlet almost assuredly | 23:24 |
| jamielennox | or recommend oslo only when eventlet is running | 23:24 |
| bknudson | jamielennox: if you're using eventlet with 1000 green threads you would want to use the pool | 23:25 |
| notmorgan | well since eventlet is deleted in mitaka | 23:25 |
| notmorgan | ... | 23:25 |
| notmorgan | don't run keystone with eventlet mitaka and later :P | 23:25 |
| jamielennox | yea, but i'm thinking a well that memcache_pool came out of auth_token so its always running there | 23:25 |
| jamielennox | and i need to fix that review that transitions to oslo.cache | 23:26 |
| *** hockeynut has joined #openstack-keystone | 23:30 | |
| bknudson | ok, switched keystone to the dogpile memcache driver. | 23:31 |
| notmorgan | i hope this eliminates the issue | 23:32 |
| notmorgan | because then the fix is "stop doing weird hack-y things" | 23:32 |
| bknudson | nope, hit it again. | 23:33 |
| notmorgan | same exact exception? | 23:33 |
| notmorgan | too many values to unpack? | 23:33 |
| bknudson | yep, same error | 23:33 |
| bknudson | 2016-08-24 23:31:32.913 9976 ERROR keystone.common.wsgi Exception: key '1921523d6734d44e88ed58dfc76ef681a36b8e9b' for 'keystone.revoke.core:_list_events|None' gave error parsing line, line is VALUE 1921523d6734d44e88ed58dfcg15 | 23:33 |
| *** slberger has left #openstack-keystone | 23:34 | |
| bknudson | >>> len(mc.get('1921523d6734d44e88ed58dfc76ef681a36b8e9b')[0]) | 23:34 |
| bknudson | 343 | 23:34 |
| *** lamt has joined #openstack-keystone | 23:34 | |
| bknudson | so can still get the value from the server | 23:34 |
| notmorgan | i just wanted to confirm it's in fact toomany values to unpack error | 23:34 |
| notmorgan | vs. something else | 23:34 |
| notmorgan | weird that mc.get() works but you error when calling via dogpile. | 23:34 |
| bknudson | I could try not having threads for keystone | 23:35 |
| notmorgan | this running under uwsgi? | 23:35 |
| notmorgan | or apache? | 23:35 |
| notmorgan | (mod_wsgi) | 23:35 |
| bknudson | yes uwsgi | 23:35 |
| bknudson | emperor mode | 23:36 |
| bknudson | thanks to jamielennox | 23:36 |
| notmorgan | maybe we really aren't threadsafe | 23:36 |
| jamielennox | hmm? | 23:36 |
| jamielennox | oh, emperor mode seemed a good way to run it | 23:36 |
| bknudson | threads = 6 ! | 23:36 |
| jamielennox | i guessed at most of the values :P | 23:36 |
| bknudson | there was a mailing list thread where apparently people are running other services in uwsgi / mod_wsgi | 23:37 |
| bknudson | I was surprised by that. | 23:37 |
| *** iurygregory_ has joined #openstack-keystone | 23:37 | |
| bknudson | thought they all expected eventlet. | 23:37 |
| notmorgan | they mostly do | 23:38 |
| notmorgan | some folks have done the legwork to make it happen | 23:38 |
| notmorgan | but it requires disabling some stuff | 23:38 |
| bknudson | failed with threads=1 ! | 23:39 |
| notmorgan | ugh | 23:41 |
| bknudson | going to try with enable-threads = False in uwsgi config | 23:41 |
| notmorgan | ok | 23:42 |
| bknudson | notmorgan: promising... have been running for a while and haven't seen the 500 error (seeing 502 errors instead for some reason) | 23:48 |
| *** Gorian|work has quit IRC | 23:49 | |
| bknudson | setting enable-threads = False seems to be pretty stable. | 23:52 |
| bknudson | not sure why I'm getting Bad Gateway errors... must be coming from haproxy. Maybe timing out requests | 23:55 |
| * notmorgan nods. | 23:56 | |
| *** roxanagh_ has quit IRC | 23:57 | |
| bknudson | [Wed Aug 24 23:55:46.420068 2016] [core:notice] [pid 760:tid 140129222494080] AH00051: child pid 8812 exit signal Segmentation fault (11), possible coredump in /etc/apache2 | 23:57 |
| bknudson | Not sure what that's about | 23:57 |
| notmorgan | blink | 23:57 |
| bknudson | [Wed Aug 24 23:26:53.546161 2016] [proxy:error] [pid 15328:tid 140128877999872] (111)Connection refused: AH00957: uwsgi: attempt to connect to 127.0.0.1:5008 (127.0.0.1) failed | 23:58 |
| openstackgerrit | Merged openstack/keystone: Update `href` for keystone extensions https://review.openstack.org/358381 | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!