| *** raildo has quit IRC | 00:17 | |
| *** markvoelker has joined #openstack-keystone | 00:40 | |
| *** markvoelker has quit IRC | 00:59 | |
| openstackgerrit | Merged openstack/keystone master: Drop use opendev.org for tox deps https://review.opendev.org/665313 | 01:52 |
|---|---|---|
| *** markvoelker has joined #openstack-keystone | 01:56 | |
| *** markvoelker has quit IRC | 02:01 | |
| openstackgerrit | zhenmei proposed openstack/keystone master: Fix create nonlocal user issue https://review.opendev.org/661183 | 02:05 |
| openstackgerrit | zhenmei proposed openstack/keystone master: Fix create nonlocal user issue https://review.opendev.org/661183 | 02:07 |
| *** liushuobj__ has joined #openstack-keystone | 02:24 | |
| *** liushuo_ has quit IRC | 02:27 | |
| *** kevinluuuuu has quit IRC | 02:32 | |
| *** markvoelker has joined #openstack-keystone | 02:57 | |
| *** markvoelker has quit IRC | 03:02 | |
| *** liushuo has joined #openstack-keystone | 03:19 | |
| *** liushuobj__ has quit IRC | 03:20 | |
| *** liushuo has quit IRC | 03:44 | |
| *** liushuo has joined #openstack-keystone | 03:44 | |
| *** ayoung has joined #openstack-keystone | 03:49 | |
| *** markvoelker has joined #openstack-keystone | 03:58 | |
| *** joshualyle has joined #openstack-keystone | 04:02 | |
| *** markvoelker has quit IRC | 04:02 | |
| *** dave-mccowan has quit IRC | 04:09 | |
| *** whoami-rajat has joined #openstack-keystone | 04:35 | |
| *** liushuo_ has joined #openstack-keystone | 04:35 | |
| *** liushuo has quit IRC | 04:39 | |
| *** ianw is now known as ianw_pto | 04:44 | |
| *** pcaruana has joined #openstack-keystone | 04:45 | |
| *** markvoelker has joined #openstack-keystone | 04:59 | |
| *** markvoelker has quit IRC | 05:04 | |
| openstackgerrit | zhenmei proposed openstack/keystone master: Fix create nonlocal user issue https://review.opendev.org/661183 | 05:15 |
| *** liushuobj__ has joined #openstack-keystone | 05:40 | |
| *** liushuo_ has quit IRC | 05:43 | |
| *** liushuo_ has joined #openstack-keystone | 05:49 | |
| *** liushuobj__ has quit IRC | 05:53 | |
| *** rcernin has quit IRC | 05:59 | |
| *** markvoelker has joined #openstack-keystone | 06:00 | |
| *** markvoelker has quit IRC | 06:04 | |
| *** vishalmanchanda has joined #openstack-keystone | 06:05 | |
| *** eivis has joined #openstack-keystone | 06:21 | |
| eivis | Hello there, is there any people alive whom might help me to get on my way with keystone and LDAP users policies ? | 06:22 |
| eivis | basically i have a problem, that I created domain where users are authenticated using LDAP. I gave my user Admin role on that domain and admin role on project which is on that domain, but still i can not update users as keystone says "WARNING You are not authorized to perform the requested action." | 06:25 |
| openstackgerrit | Vishakha Agarwal proposed openstack/keystone master: Remove [signing] config https://review.opendev.org/659434 | 06:36 |
| *** dancn has joined #openstack-keystone | 06:48 | |
| *** markvoelker has joined #openstack-keystone | 07:01 | |
| *** tesseract has joined #openstack-keystone | 07:19 | |
| *** Krenair has quit IRC | 07:22 | |
| *** markvoelker has quit IRC | 07:28 | |
| *** tkajinam has quit IRC | 07:31 | |
| *** Krenair has joined #openstack-keystone | 07:31 | |
| *** liushuobj__ has joined #openstack-keystone | 07:40 | |
| *** liushuo_ has quit IRC | 07:44 | |
| *** tkajinam has joined #openstack-keystone | 07:44 | |
| *** liushuo_ has joined #openstack-keystone | 08:18 | |
| *** liushuobj__ has quit IRC | 08:22 | |
| *** tkajinam_ has joined #openstack-keystone | 08:30 | |
| *** tkajinam has quit IRC | 08:33 | |
| *** tkajinam_ has quit IRC | 08:35 | |
| *** imacdonn has quit IRC | 08:41 | |
| *** imacdonn has joined #openstack-keystone | 08:41 | |
| *** Krenair has quit IRC | 08:42 | |
| *** liushuobj__ has joined #openstack-keystone | 08:48 | |
| *** liushuo_ has quit IRC | 08:52 | |
| *** joshualyle has quit IRC | 09:03 | |
| *** dancn has quit IRC | 09:51 | |
| openstackgerrit | Merged openstack/keystone master: Update misleading comment about fernet credential encryption https://review.opendev.org/666600 | 09:59 |
| *** Krenair has joined #openstack-keystone | 10:04 | |
| *** spsurya has joined #openstack-keystone | 10:07 | |
| *** liushuo_ has joined #openstack-keystone | 10:07 | |
| *** liushuobj__ has quit IRC | 10:11 | |
| *** chandra_keystone has quit IRC | 10:24 | |
| *** liushuobj__ has joined #openstack-keystone | 10:50 | |
| *** liushuo_ has quit IRC | 10:53 | |
| *** raildo has joined #openstack-keystone | 11:40 | |
| *** ab-a has quit IRC | 12:18 | |
| *** ab-a has joined #openstack-keystone | 12:24 | |
| *** liushuobj__ has quit IRC | 12:33 | |
| *** liushuobj__ has joined #openstack-keystone | 12:33 | |
| *** vkmc has quit IRC | 12:40 | |
| *** jamielennox has quit IRC | 12:41 | |
| *** Krenair has quit IRC | 12:42 | |
| *** liushuobj__ has quit IRC | 12:50 | |
| *** liushuobj__ has joined #openstack-keystone | 12:50 | |
| openstackgerrit | Merged openstack/keystone master: Fix E731 flake8 https://review.opendev.org/666264 | 12:53 |
| knikolla | o/ | 13:25 |
| *** vishalmanchanda has quit IRC | 13:28 | |
| *** lbragstad has joined #openstack-keystone | 13:28 | |
| *** markvoelker has joined #openstack-keystone | 13:29 | |
| *** dave-mccowan has joined #openstack-keystone | 13:30 | |
| *** FlorianFa has joined #openstack-keystone | 13:30 | |
| lbragstad | o/ | 13:35 |
| *** liushuo_ has joined #openstack-keystone | 13:42 | |
| lbragstad | the first 4 patches starting at https://review.opendev.org/#/c/665708/2 are ready for some eyes | 13:44 |
| lbragstad | (they're pretty trivial) | 13:45 |
| *** liushuobj__ has quit IRC | 13:46 | |
| brtknr | hello, just wondering what happens when a user who created an application credential disappears from a project | 13:54 |
| *** markvoelker has quit IRC | 13:57 | |
| *** jamesmcarthur has joined #openstack-keystone | 13:58 | |
| *** ormandj has joined #openstack-keystone | 13:59 | |
| ormandj | heya folks, i've got a fun one today. we've discovered when creating new credentials (ec2 in this case) with keystone stein on ubuntu 18.04LTS, py3/psql, the encrypted_blob is being stored in the database as hex | 14:00 |
| ormandj | i've added debugging code into various sections of code and determined it gets all the way to the session/db write just fine, even stored in the sqlalchemy model just fine | 14:01 |
| ormandj | but somewhere after that, hex into db. database is utf8 | 14:01 |
| ormandj | decode the hex, and it's the encrypted_blob i'd expect | 14:01 |
| ormandj | it's causing keystone to fail to decrypt the credential/breaking keystone after any new credential is created | 14:02 |
| *** joshualyle has joined #openstack-keystone | 14:02 | |
| ormandj | (ops start 500ing when it can't decrypt that one cred) | 14:02 |
| ormandj | for more clarity, everything else aside from gnocchi and ceilometer is stein, and rocky keystone works just fine, as soon as we move to stein keystone, bam. | 14:07 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| lbragstad | ormandj did anything else besides the version of keystone change? | 14:26 |
| lbragstad | mysql version? | 14:27 |
| ormandj | no, same postgresql | 14:27 |
| ormandj | we don't use mysql | 14:27 |
| lbragstad | sqla versions are the same, too? | 14:28 |
| ormandj | for clarity, we have two keystone nodes. one we have upgraded to stein, one is rocky. without changing anything else in env, if we use the rocky version, works, if we use the stein version, it breaks | 14:28 |
| ormandj | one sec, will get you versions/etc | 14:31 |
| ormandj | https://gist.github.com/ormandj/cabec3c9b07f4672d53c16b9a5a7c0eb | 14:31 |
| lbragstad | the encrypted blob makes it through all of keystone's code path normally? | 14:31 |
| ormandj | yeah let me share my debug code/output | 14:32 |
| ormandj | one sec | 14:32 |
| lbragstad | it looks like sqla packages only changed by three patch versions | 14:32 |
| lbragstad | 1.2.15 -> 1.2.18 | 14:32 |
| lbragstad | 1.2.8 -> 1.2.18 | 14:32 |
| lbragstad | 1.2.8 -> 1.2.15 * | 14:33 |
| lbragstad | sorry - more than three, but.. | 14:33 |
| *** whoami-rajat has quit IRC | 14:34 | |
| ormandj | https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa | 14:35 |
| ormandj | log line below the area i inserted code | 14:36 |
| ormandj | (have plenty more logs from debug logging i added above, but that's the last place i checked to see if the data was as expected) | 14:36 |
| ormandj | that starts at line 44 in keystone/credential/backends/sql.py | 14:37 |
| ormandj | in stein | 14:37 |
| *** dklyle has joined #openstack-keystone | 14:37 | |
| *** joshualyle has quit IRC | 14:39 | |
| lbragstad | ok - so when the credential comes into create_credential, everything is good | 14:39 |
| lbragstad | https://opendev.org/openstack/keystone/src/branch/master/keystone/credential/backends/sql.py#L41 | 14:39 |
| lbragstad | but when it's converted to a ref, it becomes hex? https://opendev.org/openstack/keystone/src/branch/master/keystone/credential/backends/sql.py#L43 | 14:40 |
| lbragstad | ref.encrypted_blob is hex and credential['encrypted_blob'] is not | 14:41 |
| lbragstad | right? | 14:41 |
| ormandj | no, ref.encrypted_blob is still fine | 14:42 |
| lbragstad | oh - actually | 14:42 |
| lbragstad | yeah - you're right | 14:42 |
| ormandj | i stopped debugging when i saw it hit the oslo_db layer | 14:42 |
| lbragstad | i missed the method call out in https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa#file-gistfile1-txt-L16 | 14:43 |
| lbragstad | right | 14:43 |
| lbragstad | i mean, that's the last spot keystone touches that reference | 14:43 |
| ormandj | that was my impression | 14:43 |
| ormandj | so i found one stackoverflow which at least pointed me in some sort of direction | 14:43 |
| lbragstad | i wonder if bnemec or zzzeek have seen anything like this | 14:43 |
| ormandj | https://stackoverflow.com/questions/45613672/why-does-my-text-field-get-hex-encoded-when-storing-a-string-to-postgresql-with | 14:43 |
| *** bnemec is now known as beekneemech | 14:44 | |
| *** liushuo_ has quit IRC | 14:44 | |
| ormandj | but i have no idea what's going on with oslo_db or anything else at that layer or below | 14:44 |
| ormandj | and it may be a completely different issue | 14:44 |
| *** liushuo_ has joined #openstack-keystone | 14:44 | |
| zzzeek | lbragstad: I've observed that keystone has some custom SQL data processesors in use | 14:44 |
| zzzeek | lbragstad: for example there's a bug in one that deals with JSON I need to report for | 14:45 |
| *** liushuo_ has quit IRC | 14:45 | |
| lbragstad | ormandj both rocky and stein are using python3 though? | 14:45 |
| *** TheJulia is now known as needssleep | 14:46 | |
| *** liushuo_ has joined #openstack-keystone | 14:46 | |
| zzzeek | looks like just dates and json though, no hex | 14:46 |
| lbragstad | you mean these? https://opendev.org/openstack/keystone/src/branch/master/keystone/common/sql/core.py | 14:46 |
| zzzeek | lbragstad: yah they dont look suspicious | 14:47 |
| ormandj | fwiw, dbs are utf-8, and let me get you psql version | 14:47 |
| lbragstad | we don't do anything with text either | 14:47 |
| lbragstad | https://opendev.org/openstack/keystone/src/branch/master/keystone/common/sql/core.py#L68 | 14:47 |
| lbragstad | looks like we just alias to sqla directly | 14:47 |
| zzzeek | lbragstad: is this MySQL and "encrypted blob" contains binary data ? | 14:48 |
| zzzeek | lbragstad: that would be where this starts to go wrong perhaps | 14:48 |
| lbragstad | postgres | 14:48 |
| beekneemech | Encoding already unicode data results in a hex value? That's...unintuitive. | 14:48 |
| lbragstad | example debug data | 14:49 |
| lbragstad | https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa#file-gistfile1-txt-L16 | 14:49 |
| zzzeek | lbragstad: issue is only on PG and not mysql? | 14:49 |
| lbragstad | i haven't seen anything like this with mysql, but i haven't tried directly | 14:49 |
| ormandj | https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa has more db info at the bottom | 14:49 |
| ormandj | just added | 14:49 |
| lbragstad | so - when we're here https://opendev.org/openstack/keystone/src/branch/master/keystone/credential/backends/sql.py#L43 | 14:49 |
| lbragstad | the ref.encrypted_blob is what we (as in keystone) want it to be | 14:50 |
| lbragstad | and that's shown here, too https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa#file-gistfile1-txt-L16 | 14:50 |
| ormandj | rocky is python2, stein is python3 | 14:51 |
| ormandj | re: keystone | 14:51 |
| lbragstad | oh - so keystone is running under python2 https://gist.github.com/ormandj/cabec3c9b07f4672d53c16b9a5a7c0eb#file-gistfile1-txt-L20 | 14:52 |
| lbragstad | and you have python3 sqla packages on rocky? https://gist.github.com/ormandj/cabec3c9b07f4672d53c16b9a5a7c0eb#file-gistfile1-txt-L11-L15 | 14:52 |
| *** liushuo_ has quit IRC | 14:53 | |
| ormandj | i'm sure related to some deps for something somewhere | 14:53 |
| zzzeek | lbragstad: uh yeah can you let me know which python version this is happening under | 14:53 |
| ormandj | zzzeek: python3 | 14:53 |
| *** liushuo_ has joined #openstack-keystone | 14:53 | |
| zzzeek | ormandj: python3 only right | 14:53 |
| ormandj | rocky/working is running in py2 | 14:53 |
| ormandj | stein/not-working is running in py3 | 14:53 |
| *** markvoelker has joined #openstack-keystone | 14:54 | |
| *** liushuo_ has quit IRC | 14:54 | |
| *** liushuo_ has joined #openstack-keystone | 14:55 | |
| ormandj | to be clear, we are using the ubuntu cloud archive packaged versions re: gist with dpkg output | 14:55 |
| *** liushuo_ has quit IRC | 14:56 | |
| *** liushuo_ has joined #openstack-keystone | 14:57 | |
| ormandj | consolidated everything into last gist: https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa, deleting the other one | 14:58 |
| *** markvoelker has quit IRC | 14:58 | |
| zzzeek | ormandj / lbragstad it would seem likely the data you are putting in the model is a Python bytes object is that possible ? https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa#gistcomment-2950285 | 14:59 |
| zzzeek | the column is created as TEXT so should not be passed a Python bytes object | 15:00 |
| *** hoonetorg has quit IRC | 15:01 | |
| lbragstad | https://opendev.org/openstack/keystone/src/branch/master/keystone/credential/backends/sql.py#L30 appears to be the only TEXT field in that model | 15:01 |
| ormandj | i'll check for you :) | 15:02 |
| ormandj | ('cred in sql function model type: ', <class 'bytes'>) | 15:04 |
| ormandj | added these lines: | 15:04 |
| ormandj | testout3 = ("cred in sql function model type: ", type(ref['encrypted_blob'])) | 15:04 |
| ormandj | LOG.error(testout3) | 15:04 |
| lbragstad | so - looks like we need patch to make sure we convert from bytes | 15:04 |
| lbragstad | before session.add(ref) | 15:05 |
| ormandj | wonder what that from_dict() function is defined as, could mutate it to handle python3 | 15:05 |
| ormandj | so it'll do the conversion appropriately when building out the model from a dictionary | 15:06 |
| lbragstad | i think session.add(ref) is what persists the credential/model to the actual database | 15:06 |
| lbragstad | i think to_dict() just converts from sqla models to python dictionaries | 15:06 |
| ormandj | it is, but i would assume you'd want the conversion to occur when you populate the model, ie: from_dict() sees a python3 byte class to populate a TEXT type, convert to not-byte | 15:07 |
| ormandj | well, this is the opposite i think, the from_dict is creating the sqla model from a py dict | 15:07 |
| ormandj | https://github.com/openstack/keystone/blob/stable/stein/keystone/credential/backends/sql.py#L43 | 15:08 |
| lbragstad | oh - nvm | 15:09 |
| lbragstad | i thought you were talking about https://github.com/openstack/keystone/blob/stable/stein/keystone/credential/backends/sql.py#L45 | 15:09 |
| lbragstad | yeah - i suppose we can do this one of two ways | 15:10 |
| ormandj | just trying to think what will solve most problems in most code with least lines without being obfuscated | 15:10 |
| lbragstad | see if from_dict() can handle it, or handle it after we have an instance of the model and override what ref.encrypted_blob is to make sure it's not bytes | 15:10 |
| ormandj | yeah, i'd just be worried that everywhere you use sqla in the future you'd have to account for it | 15:11 |
| ormandj | vs. updating the way you populate models so it applies globally now and in future without having to keep in mind that case everywhere | 15:11 |
| lbragstad | https://pasted.tech/pastes/22c35846c1c216495f19d101bc745068ca73263a.raw | 15:12 |
| lbragstad | we'd have to possibly blacklist a version of sqla | 15:12 |
| * lbragstad waits for zzzeek's opinion | 15:13 | |
| zzzeek | looking | 15:13 |
| ormandj | i'll let you two fine folks figure out the best way to skin this cat :) | 15:13 |
| zzzeek | lbragstad: not following, what version of sqlalchemy has a problem ? | 15:14 |
| lbragstad | well - i'm not entirely sure | 15:15 |
| lbragstad | but keystone passes a dictionary into https://opendev.org/openstack/keystone/src/branch/master/keystone/credential/backends/sql.py#L43 | 15:15 |
| lbragstad | and ref comes back with ref.encrypted_blob as bytes | 15:15 |
| ormandj | the dictionary it passes in i assume that encrypted_blob data is also bytes | 15:16 |
| ormandj | i can also check the type on that if you'd like | 15:16 |
| lbragstad | bah... | 15:16 |
| lbragstad | one sec | 15:17 |
| lbragstad | the fix probably needs to go here https://opendev.org/openstack/keystone/src/branch/master/keystone/common/sql/core.py#L191-L203 | 15:18 |
| lbragstad | i don't think it's a sqla issue | 15:18 |
| lbragstad | keystone implements the from_dict and to_dict functionality | 15:19 |
| lbragstad | so we could put the fix there? | 15:19 |
| zzzeek | lbragstad: just the other day someone was complaining why SQLA can't implement various to_dict() schemes | 15:19 |
| zzzeek | this is why! :) | 15:19 |
| lbragstad | so - the fix should live in that method, then | 15:20 |
| lbragstad | i still don't know if i understand where that conversion happens | 15:20 |
| ormandj | https://gist.github.com/ormandj/269c2b6a5baaa6ca79c04e31f6100ffa#file-gistfile1-txt-L98 | 15:20 |
| ormandj | there is no conversion right now | 15:21 |
| ormandj | it's a byte class in the dictionary, and a byte class once from_dict() runs | 15:21 |
| ormandj | in the sqla model | 15:21 |
| lbragstad | aha | 15:21 |
| lbragstad | whelp | 15:22 |
| zzzeek | lbragstad ormandj note the usual place you'd want to correct for this is in a SQLA custom type | 15:22 |
| zzzeek | well, maybe not "usual", but it's the most "bulletproof" | 15:22 |
| zzzeek | but having it explicit at a higher level in your from_dict() routine might be easier to debug in the future b.c. it's more visible | 15:23 |
| lbragstad | a custom type for Credential (as in the model) | 15:23 |
| zzzeek | lbragstad: yes a TEXT that looks for bytes and converts to string if so | 15:24 |
| lbragstad | oh - sure | 15:24 |
| lbragstad | ormandj do you know if there is a keystone bug open for this yet? | 15:24 |
| ormandj | i didn't find one | 15:24 |
| ormandj | (which surprised me) | 15:24 |
| lbragstad | based on this discussion, this would affect mysql, too? | 15:25 |
| lbragstad | i don't see a reason why this would be postgres specific | 15:25 |
| ormandj | i don't know if you're asking me, i don't know if the different drivers would handle this differently | 15:28 |
| lbragstad | keystone only has one sql driver | 15:28 |
| *** cmurphy is now known as cmorpheus | 15:28 | |
| ormandj | mysql vs. postgres i mean, the driver re: backend that sqlalchemy is using | 15:29 |
| ormandj | (or sqlite or w/e) | 15:29 |
| lbragstad | oh - wright | 15:32 |
| lbragstad | right* | 15:32 |
| zzzeek | lbragstad: this is likely somewhat PG specific but you should probably convert from bytes for all drivers regrdless | 15:32 |
| lbragstad | yeah - i mean we only have one keystone driver for sql, we can fix it there for sure | 15:33 |
| lbragstad | ormandj would you be willing to open a bug report? | 15:33 |
| ormandj | sure, i can do that | 15:36 |
| lbragstad | https://bugs.launchpad.net/keystone/+filebug if you need it | 15:36 |
| ormandj | launchpad your place of choice? | 15:36 |
| ormandj | kk | 15:36 |
| lbragstad | yep | 15:36 |
| lbragstad | for now | 15:37 |
| ormandj | let's see if i even have an account :) i'll let you know when i get it posted. | 15:37 |
| *** gyee has joined #openstack-keystone | 15:46 | |
| *** clarkb has quit IRC | 15:53 | |
| ormandj | lbragstad: https://bugs.launchpad.net/keystone/+bug/1833739 | 15:55 |
| openstack | Launchpad bug 1833739 in OpenStack Identity (keystone) "keystone (stein), python3, and postgresql: hex in database" [Undecided,New] | 15:55 |
| ormandj | sufficient for you? | 15:55 |
| *** markvoelker has joined #openstack-keystone | 15:55 | |
| lbragstad | sweet | 15:55 |
| *** jamesmcarthur has joined #openstack-keystone | 15:56 | |
| ormandj | we're hoping for a backport into stein, this is blocking our production upgrades for obvious reasons ;) | 15:58 |
| lbragstad | yeah - this would be a backport potential | 15:59 |
| *** markvoelker has quit IRC | 15:59 | |
| ormandj | awesome. thanks for looking at tihngs | 16:01 |
| ormandj | things, as well | 16:01 |
| ormandj | anything else you feel would be useful to add for you/whomever? | 16:01 |
| *** spsurya has quit IRC | 16:24 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** dancn has joined #openstack-keystone | 16:46 | |
| *** markvoelker has joined #openstack-keystone | 16:56 | |
| *** markvoelker has quit IRC | 17:01 | |
| * lbragstad breaks for lunch | 17:04 | |
| *** hoonetorg has joined #openstack-keystone | 17:06 | |
| *** jamesmcarthur has joined #openstack-keystone | 17:07 | |
| *** lbragstad has quit IRC | 17:09 | |
| *** raildo has quit IRC | 17:09 | |
| *** whoami-rajat has joined #openstack-keystone | 17:09 | |
| *** raildo has joined #openstack-keystone | 17:16 | |
| *** jamesmcarthur has quit IRC | 17:23 | |
| *** tesseract has quit IRC | 17:50 | |
| gagehugo | o/ | 17:51 |
| *** hoonetorg has quit IRC | 17:54 | |
| *** markvoelker has joined #openstack-keystone | 17:56 | |
| *** markvoelker has quit IRC | 18:01 | |
| *** mvkr has quit IRC | 18:06 | |
| *** hoonetorg has joined #openstack-keystone | 18:07 | |
| *** hoonetorg has quit IRC | 18:10 | |
| *** hoonetorg has joined #openstack-keystone | 18:14 | |
| *** Krenair has joined #openstack-keystone | 18:17 | |
| *** mchlumsky has quit IRC | 18:47 | |
| *** awalende has joined #openstack-keystone | 18:52 | |
| *** markvoelker has joined #openstack-keystone | 18:57 | |
| *** hyang has joined #openstack-keystone | 19:10 | |
| *** awalende has quit IRC | 19:11 | |
| *** whoami-rajat has quit IRC | 19:19 | |
| *** lbragstad has joined #openstack-keystone | 19:23 | |
| hyang | Hi keystone team, I'm using Rocky release and trying to configure healthcheck disable by file for keystone following this link https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plugins.html#disable-by-files-ports But it does not work as the healthcheck url always returns 200 even if the disable file exists | 19:25 |
| hyang | wondering is that a known issue or I misconfigured something? | 19:26 |
| *** markvoelker has quit IRC | 19:26 | |
| *** dancn has quit IRC | 19:27 | |
| openstackgerrit | Merged openstack/keystonemiddleware master: Remove Diablo compatibility tests https://review.opendev.org/666668 | 19:37 |
| openstackgerrit | Merged openstack/keystoneauth master: Limit interval between retries to 1 minute https://review.opendev.org/666284 | 19:55 |
| *** ayoung has quit IRC | 19:55 | |
| *** ayoung has joined #openstack-keystone | 20:04 | |
| lbragstad | looking at https://review.opendev.org/#/c/659434/15 | 20:06 |
| lbragstad | i'm pretty sure https://opendev.org/openstack/keystone/src/branch/master/keystone/tests/unit/test_v3_auth.py#L5460 is broken | 20:06 |
| lbragstad | it doesn't look like those tests get run at all.. | 20:07 |
| lbragstad | (they inherit from object and don't actually get invoked by a test class) | 20:07 |
| cmorpheus | :/ | 20:07 |
| lbragstad | if you have them inherit RestfulTestCase, they run, but they break | 20:08 |
| *** mlavalle has joined #openstack-keystone | 20:09 | |
| lbragstad | also... | 20:09 |
| lbragstad | it appears we have a policy that is completely unused | 20:09 |
| lbragstad | https://opendev.org/openstack/keystone/src/branch/master/keystone/common/policies/token_revocation.py#L19 doesn't seem to be protecting anything | 20:10 |
| lbragstad | which is good an misleading for people reading the documentation for that rule :) | 20:12 |
| beekneemech | You're just full of good news today. :-P | 20:15 |
| *** ayoung has quit IRC | 20:16 | |
| lbragstad | yessir | 20:16 |
| lbragstad | ending the week on a good note | 20:17 |
| mlavalle | hey lbragstad. my good friend hyang has a few questions about keystone. Any help is much appreciated. Thanks in advance | 20:17 |
| lbragstad | mlavalle hyang o/ | 20:17 |
| lbragstad | hyang we recently refactored how we load middleware in keystone (rocky i think) | 20:18 |
| hyang | Hi lbragstad | 20:18 |
| lbragstad | kmalloc did a bunch of that work as we moved to flask (and off of paste pipelines) | 20:19 |
| lbragstad | hyang how are you enabling the middleware currently? | 20:19 |
| lbragstad | i think that link still references paste? | 20:20 |
| lbragstad | kmalloc didn't we talk about porting those things to configuration options? | 20:21 |
| kmalloc | Uhm .. maybe | 20:21 |
| lbragstad | kmalloc the flask refactor landed in rocky, right? | 20:21 |
| kmalloc | I am looking at how the middleware works | 20:22 |
| kmalloc | Rocky and stien. | 20:22 |
| lbragstad | yeah - https://docs.openstack.org/releasenotes/keystone/rocky.html#prelude | 20:22 |
| lbragstad | is when we started it | 20:22 |
| kmalloc | Hmm. Are we not pulling in the options for middleware? | 20:23 |
| hyang | so in keystone.conf there is an entry under [paste_deploy] `config_file = /etc/keystone/keystone-paste.ini` | 20:23 |
| kmalloc | Oslo.middleware is weird in some ways. | 20:23 |
| lbragstad | well - it looks like that endpoint was previously enabled through paste | 20:23 |
| *** markvoelker has joined #openstack-keystone | 20:23 | |
| kmalloc | Oh yeah don't ever use paste.ini to pass config options | 20:23 |
| kmalloc | That was broken beyond belief to begin with | 20:24 |
| lbragstad | rocky doesn't actually use paste anymore i don't think | 20:24 |
| kmalloc | We aren't pulling in the conf options from Oslo.middleware | 20:24 |
| kmalloc | Into keystone.conf processing. | 20:24 |
| lbragstad | huh | 20:25 |
| lbragstad | dang... | 20:25 |
| hyang | ok, so wondering how the /healthcheck url is still working for keystone? | 20:25 |
| kmalloc | It is, it just isn't handing the config options from Oslo.middleware | 20:25 |
| lbragstad | so - that endpoint is loaded by default, then? | 20:26 |
| kmalloc | lbragstad: we need to pull in the options the same way we pull in options for say logging | 20:26 |
| lbragstad | ok | 20:26 |
| kmalloc | Yeah it is explicitly enabled | 20:26 |
| lbragstad | and it can't be disabled at all | 20:27 |
| lbragstad | because we don't honor those configuration options | 20:27 |
| kmalloc | https://github.com/openstack/keystone/blob/master/keystone/server/flask/application.py#L177 | 20:27 |
| hyang | aha | 20:28 |
| lbragstad | ah - so it's not configurable | 20:28 |
| kmalloc | So we just need to pull in options and then it is configured in keystone.conf | 20:28 |
| *** markvoelker has quit IRC | 20:28 | |
| *** ayoung has joined #openstack-keystone | 20:28 | |
| lbragstad | using oslo_middleware? | 20:28 |
| kmalloc | Yes | 20:29 |
| kmalloc | We use oslo.middleware's code | 20:29 |
| lbragstad | oh - we need to pass a CONF object here? https://github.com/openstack/keystone/blob/master/keystone/server/flask/application.py#L171 | 20:30 |
| kmalloc | Not sure, will.need to look at it. | 20:30 |
| kmalloc | I think we do need to pass it | 20:31 |
| lbragstad | yeah | 20:31 |
| lbragstad | we do | 20:31 |
| lbragstad | just pulled down a copy of oslo.middleware | 20:32 |
| lbragstad | https://opendev.org/openstack/oslo.middleware/src/branch/master/oslo_middleware/healthcheck/__init__.py#L422 | 20:32 |
| kmalloc | Yeah. | 20:33 |
| hyang | thank you guys for looking at it, so seems like that would be a patch and for the config to support healthcheck disable, it will land in keystone.conf or still paste.ini? | 20:38 |
| lbragstad | we don't support python-paste anymore, so we'll need to patch keystone to handle that | 20:39 |
| hyang | ok got it, thx! | 20:40 |
| lbragstad | this is an example of how ironic does it i think | 20:40 |
| lbragstad | https://opendev.org/openstack/ironic/src/branch/master/ironic/api/app.py#L107-L113 | 20:40 |
| * lbragstad looks at jroll | 20:41 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Pass CONF to healthcheck middleware https://review.opendev.org/666920 | 20:46 |
| lbragstad | ^ no idea if that will work hyang | 20:46 |
| lbragstad | but it might be close? | 20:46 |
| hyang | lbragstad: going to grab some food, will brb | 20:48 |
| lbragstad | ack | 20:48 |
| jroll | lbragstad: seems legit | 21:12 |
| lbragstad | sweet | 21:15 |
| *** pcaruana has quit IRC | 21:16 | |
| *** joshualyle has joined #openstack-keystone | 21:17 | |
| *** lbragstad has quit IRC | 21:20 | |
| *** joshualyle has quit IRC | 21:21 | |
| *** markvoelker has joined #openstack-keystone | 21:24 | |
| *** joshualyle has joined #openstack-keystone | 21:27 | |
| *** markvoelker has quit IRC | 21:29 | |
| *** joshualyle has quit IRC | 21:29 | |
| *** joshualyle has joined #openstack-keystone | 21:29 | |
| openstackgerrit | Merged openstack/keystonemiddleware master: Remove PKI/PKIZ support https://review.opendev.org/613675 | 21:31 |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** joshualyle has quit IRC | 21:34 | |
| cmorpheus | woot ^ | 21:38 |
| *** joshualyle has joined #openstack-keystone | 21:39 | |
| *** joshualyle has quit IRC | 21:44 | |
| *** joshualyle has joined #openstack-keystone | 21:52 | |
| *** joshualyle has quit IRC | 21:56 | |
| *** raildo has quit IRC | 21:56 | |
| hyang | lbragstad: thanks for the patch. I tried that in my Rocky environment and I got "ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option". Then I revert the change and just configured the [healthcheck] section in keystone.conf rather than paste.ini and I was able to disable healthcheck by file. | 22:05 |
| hyang | lbragstad: maybe the patch is only needed for master/stein that have the refactored code I guess | 22:07 |
| cmorpheus | kmalloc: i started implementing the immutable flag for roles https://review.opendev.org/666739 but it steps on some of what is planned for https://review.opendev.borg/624162 I'm wondering where that is on your priority list and whether we should rescope the immutable resources to just focus on roles so they don't depend on other work? | 22:23 |
| *** markvoelker has joined #openstack-keystone | 22:25 | |
| *** markvoelker has quit IRC | 22:30 | |
| *** gyee has quit IRC | 22:40 | |
| *** dave-mccowan has quit IRC | 22:49 | |
| *** markvoelker has joined #openstack-keystone | 23:26 | |
| *** joshualyle has joined #openstack-keystone | 23:27 | |
| *** joshualyle has quit IRC | 23:27 | |
| *** joshualyle has joined #openstack-keystone | 23:27 | |
| *** ormandj has quit IRC | 23:29 | |
| *** markvoelker has quit IRC | 23:30 | |
| *** jamesmcarthur has joined #openstack-keystone | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!