| *** lifeless_ has joined #openstack-keystone | 00:01 | |
| *** oikiki has quit IRC | 00:01 | |
| *** lifeless has quit IRC | 00:02 | |
| *** bigdogstl has joined #openstack-keystone | 00:06 | |
| *** bigdogstl has quit IRC | 00:11 | |
| *** masuberu has joined #openstack-keystone | 00:15 | |
| *** bigdogstl has joined #openstack-keystone | 00:17 | |
| *** masber has quit IRC | 00:19 | |
| *** bigdogstl has quit IRC | 00:24 | |
| *** bigdogstl has joined #openstack-keystone | 00:26 | |
| *** bigdogstl has quit IRC | 00:33 | |
| *** bigdogstl has joined #openstack-keystone | 00:34 | |
| *** rcernin_ has joined #openstack-keystone | 00:37 | |
| *** rcernin has quit IRC | 00:37 | |
| *** bigdogstl has quit IRC | 00:39 | |
| *** bigdogstl has joined #openstack-keystone | 00:41 | |
| *** bigdogstl has quit IRC | 00:46 | |
| *** bigdogstl has joined #openstack-keystone | 00:58 | |
| gagehugo | kmalloc looking | 01:01 |
|---|---|---|
| *** Dinesh_Bhor has joined #openstack-keystone | 01:02 | |
| *** bigdogstl has quit IRC | 01:03 | |
| *** harlowja has quit IRC | 01:07 | |
| *** bigdogstl has joined #openstack-keystone | 01:12 | |
| *** bigdogstl has quit IRC | 01:20 | |
| gagehugo | done | 01:21 |
| wxy | lbragstad: https://www.lbragstad.com/blog/openstack-summit-vancouver-recap reading now. ;) | 01:21 |
| wxy | ops, misoperation. ignore it. | 01:22 |
| openstackgerrit | wangxiyuan proposed openstack/python-keystoneclient master: WIP: functionality for registered limits https://review.openstack.org/572006 | 01:25 |
| *** r-daneel has joined #openstack-keystone | 01:43 | |
| *** namnh has joined #openstack-keystone | 01:53 | |
| *** Dinesh__Bhor has joined #openstack-keystone | 02:00 | |
| *** namnh_ has joined #openstack-keystone | 02:02 | |
| *** namnh has quit IRC | 02:02 | |
| openstackgerrit | Merged openstack/keystone master: Correct test_v3_oauth1.test_bad_authorizing_roles_id https://review.openstack.org/571912 | 02:02 |
| *** namnh_ has quit IRC | 02:02 | |
| *** namnh has joined #openstack-keystone | 02:03 | |
| *** Dinesh_Bhor has quit IRC | 02:03 | |
| *** Rhvs has quit IRC | 02:06 | |
| openstackgerrit | Merged openstack/keystone master: Decouple bootstrap from cli module https://review.openstack.org/558903 | 02:07 |
| *** Rhvs has joined #openstack-keystone | 02:13 | |
| *** itlinux has joined #openstack-keystone | 02:21 | |
| *** liuzz has joined #openstack-keystone | 02:23 | |
| openstackgerrit | Harry Rybacki proposed openstack/keystone master: WIP: Ensure default roles created during bootstrap https://review.openstack.org/572243 | 02:26 |
| *** gongysh has joined #openstack-keystone | 02:27 | |
| *** lifeless_ has quit IRC | 02:38 | |
| *** lifeless has joined #openstack-keystone | 02:39 | |
| *** dave-mccowan has quit IRC | 02:54 | |
| *** alex_xu has quit IRC | 03:00 | |
| *** alex_xu has joined #openstack-keystone | 03:01 | |
| openstackgerrit | Merged openstack/keystone master: Correct test_v3_oauth1.test_change_user_password_also_deletes_tokens https://review.openstack.org/571913 | 03:01 |
| openstackgerrit | Merged openstack/keystone master: Correct test_v3_oauth1.test_deleting_project_also_invalidates_tokens https://review.openstack.org/571914 | 03:06 |
| openstackgerrit | Chason Chan proposed openstack/keystone master: Docs: Remove the TokenAuth middleware https://review.openstack.org/572248 | 03:13 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone master: Convert Keystone to use Flask https://review.openstack.org/568377 | 03:14 |
| *** hoonetorg has quit IRC | 03:23 | |
| *** sonuk has joined #openstack-keystone | 03:26 | |
| *** hoonetorg has joined #openstack-keystone | 03:40 | |
| *** bigdogstl has joined #openstack-keystone | 03:55 | |
| *** bigdogstl has quit IRC | 04:00 | |
| *** germs has quit IRC | 04:01 | |
| *** germs has joined #openstack-keystone | 04:02 | |
| *** germs has quit IRC | 04:02 | |
| *** germs has joined #openstack-keystone | 04:02 | |
| *** germs has quit IRC | 04:06 | |
| *** links has joined #openstack-keystone | 04:22 | |
| *** gongysh has quit IRC | 04:31 | |
| *** sapd has quit IRC | 04:45 | |
| *** sapd has joined #openstack-keystone | 04:45 | |
| *** Dinesh__Bhor has quit IRC | 04:59 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 05:00 | |
| *** gyee has quit IRC | 05:00 | |
| *** mvk has joined #openstack-keystone | 05:05 | |
| *** sonuk_ has joined #openstack-keystone | 05:08 | |
| *** sonuk has quit IRC | 05:11 | |
| *** sonuk has joined #openstack-keystone | 05:12 | |
| *** sonuk_ has quit IRC | 05:15 | |
| *** AlexeyAbashkin has quit IRC | 05:17 | |
| *** lifeless has quit IRC | 05:40 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 05:41 | |
| *** lifeless has joined #openstack-keystone | 05:41 | |
| *** gongysh has joined #openstack-keystone | 05:43 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 05:43 | |
| *** AlexeyAbashkin has quit IRC | 05:44 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 05:44 | |
| *** Dinesh_Bhor has quit IRC | 05:51 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 05:54 | |
| *** AlexeyAbashkin has quit IRC | 05:59 | |
| *** germs has joined #openstack-keystone | 06:03 | |
| *** germs has quit IRC | 06:03 | |
| *** germs has joined #openstack-keystone | 06:03 | |
| *** pooja-jadhav is now known as pooja_jadhav | 06:04 | |
| *** martinus__ has joined #openstack-keystone | 06:05 | |
| *** germs has quit IRC | 06:08 | |
| *** links has quit IRC | 06:08 | |
| *** ispp has joined #openstack-keystone | 06:12 | |
| *** jaosorior has joined #openstack-keystone | 06:14 | |
| *** isssp has quit IRC | 06:14 | |
| *** links has joined #openstack-keystone | 06:26 | |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Fix db model inconsistency for FederatedUser https://review.openstack.org/566242 | 06:32 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Enable Foreign keys for sql backend unit test https://review.openstack.org/558029 | 06:32 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Enable foreign keys for unit test https://review.openstack.org/558193 | 06:32 |
| *** pcaruana has joined #openstack-keystone | 06:54 | |
| *** pcaruana is now known as pcaruana|worksho | 07:03 | |
| *** sonuk_ has joined #openstack-keystone | 07:03 | |
| *** sonuk has quit IRC | 07:07 | |
| openstackgerrit | Adrian Turjak proposed openstack/keystone master: Rename token_utils back to fernet_utils https://review.openstack.org/566208 | 07:34 |
| *** lifeless has quit IRC | 07:40 | |
| *** lifeless has joined #openstack-keystone | 07:42 | |
| openstackgerrit | Adrian Turjak proposed openstack/keystone master: [WIP] Implement auth receipts spec https://review.openstack.org/572286 | 07:43 |
| adriant | lbragstad, cmurphy, mordred: ^ VERY early code for auth receipts. The logic in theory is right, but it entirely untested and has no unit tests. | 07:44 |
| adriant | Was interesting learning how the provider logic actually worked | 07:44 |
| adriant | the actual code for making the auth receipt logic work in the auth process is easy... all the work is in that blasted provider layer. | 07:45 |
| adriant | I'll refine it a bit tomorrow, and start actually trying to get it to work, as well as start adding tests. | 07:46 |
| *** pcaruana|worksho is now known as pcaruana | 07:50 | |
| *** akovi has joined #openstack-keystone | 07:50 | |
| cmurphy | adriant: yay \o/ | 07:50 |
| adriant | my goal is to have the code in a fully ready to review state next week, before feature proposal freeze, so I've got time to refine before feature freeze :P | 07:51 |
| adriant | and hopefully start working on figuring out how to make it play nice in KeystoneAuth. | 07:51 |
| akovi | Hi Keystone team! Is it a possibility that this patch https://review.openstack.org/#/c/568877/ is causing Mistral devstack jobs to break? http://logs.openstack.org/85/527085/29/check/mistral-devstack/56e57ce/ | 07:51 |
| cmurphy | adriant: you are so on top of things | 07:52 |
| adriant | cmurphy: hah, I wish! | 07:52 |
| akovi | It seems like the config changes have not been released? http://logs.openstack.org/85/527085/29/check/mistral-devstack/56e57ce/controller/logs/apache/mistral_api_log.txt.gz#_2018-06-04_19_23_37_223487 | 07:53 |
| cmurphy | akovi: hmm that sure looks related | 07:55 |
| akovi | cmurphy: thanks, could you please ask someone to look at it? | 07:58 |
| openstackgerrit | Adrian Turjak proposed openstack/keystone master: [WIP] Implement auth receipts spec https://review.openstack.org/572286 | 07:59 |
| cmurphy | akovi: better idea, could you file a bug? | 08:02 |
| *** germs has joined #openstack-keystone | 08:04 | |
| *** germs has quit IRC | 08:04 | |
| *** germs has joined #openstack-keystone | 08:04 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 08:05 | |
| *** blake has joined #openstack-keystone | 08:07 | |
| *** germs has quit IRC | 08:08 | |
| akovi | cmurphy: thanks, opened https://bugs.launchpad.net/mistral/+bug/1775140 | 08:11 |
| openstack | Launchpad bug 1775140 in Mistral "Keystoneauth does not consistently add the collect-timing parameter" [Undecided,New] | 08:11 |
| cmurphy | akovi: mistral seems to be doing something weird with keystoneauth, it's manually adding the timeout option here http://git.openstack.org/cgit/openstack/mistral/tree/mistral/utils/openstack/keystone.py#n31 | 08:15 |
| cmurphy | akovi: it shouldn't be doing that, it should be using keystoneauth's oslo.config register methods like in here https://docs.openstack.org/keystoneauth/latest/migrating.html#step-by-step-migration-example | 08:15 |
| cmurphy | that must be why the new option isn't registered | 08:16 |
| akovi | oh, great, I'll try to fix it | 08:17 |
| akovi | one question: what should be fixed here? :) I have no clue why the timeout option is registered here in this way | 08:22 |
| cmurphy | akovi: er i guess that doc doesn't show the example, but here's an example from keystonemiddleware http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/keystonemiddleware/auth_token/_opts.py#n220 | 08:22 |
| cmurphy | it needs to call register_auth_conf_options with the CONF object and the keystone_authtoken group | 08:23 |
| akovi | Ok, I did something. We'll see how far it goes https://review.openstack.org/572300 @cmurphy thanks for your help, very much appreciated | 08:33 |
| cmurphy | akovi: no problem | 08:35 |
| *** namnh has quit IRC | 08:37 | |
| *** gongysh has quit IRC | 09:14 | |
| *** lifeless has quit IRC | 09:25 | |
| *** issp has joined #openstack-keystone | 09:28 | |
| *** lifeless has joined #openstack-keystone | 09:29 | |
| *** bigdogstl has joined #openstack-keystone | 09:35 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/keystone master: Ensure default roles created during bootstrap https://review.openstack.org/572243 | 09:37 |
| *** bigdogstl has quit IRC | 09:40 | |
| jaosorior | hrybacki: Did some of the missing work there ^^ | 09:41 |
| *** Dinesh_Bhor has quit IRC | 09:42 | |
| *** eandersson has quit IRC | 09:50 | |
| *** blake has quit IRC | 10:03 | |
| *** germs has joined #openstack-keystone | 10:04 | |
| *** germs has quit IRC | 10:09 | |
| *** rcernin_ has quit IRC | 10:42 | |
| *** links has quit IRC | 11:09 | |
| *** akovi has left #openstack-keystone | 11:10 | |
| *** liuzz has quit IRC | 11:12 | |
| *** lifeless has quit IRC | 11:12 | |
| *** lifeless has joined #openstack-keystone | 11:13 | |
| *** ianw has quit IRC | 11:13 | |
| *** pooja_jadhav has quit IRC | 11:16 | |
| *** bhagyashri_s has quit IRC | 11:16 | |
| *** bhagyashri_s has joined #openstack-keystone | 11:17 | |
| *** pooja_jadhav has joined #openstack-keystone | 11:17 | |
| *** edmondsw has joined #openstack-keystone | 11:19 | |
| *** links has joined #openstack-keystone | 11:25 | |
| *** sonuk has joined #openstack-keystone | 11:30 | |
| *** sonuk_ has quit IRC | 11:30 | |
| *** ianw has joined #openstack-keystone | 11:32 | |
| *** AlexeyAbashkin has quit IRC | 11:36 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 11:46 | |
| *** raildo has joined #openstack-keystone | 12:02 | |
| *** germs has joined #openstack-keystone | 12:05 | |
| *** sonuk has quit IRC | 12:08 | |
| *** germs has quit IRC | 12:10 | |
| *** eandersson has joined #openstack-keystone | 12:19 | |
| *** neha_alhat has joined #openstack-keystone | 12:30 | |
| neha_alhat | cmurphy mordred: Hi, Want to know that change added in patch[1] is released from keystoneauth library or not? How can I found that? | 12:33 |
| neha_alhat | cmurphy mordred: [1]: https://review.openstack.org/#/c/568878/ | 12:33 |
| cmurphy | neha_alhat: it has been released, which i found out by using `git tag --contains 80323289` | 12:37 |
| neha_alhat | cmurphy: Ok, thanks.. will check | 12:38 |
| neha_alhat | cmurphy: Can you please give me high overview of releases of third party lib and how that are used by core projects like cinder,nova? | 12:48 |
| cmurphy | neha_alhat: when we want to release a library we submit a change to http://git.openstack.org/cgit/openstack/releases and the release team reviews and approves it, and then the automation behind the scenes pushes a git tag and pushes a tarball to pypi, then we make an update to http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt to change the upper bound for the library | 12:52 |
| cmurphy | and then the other projects automatically start using it | 12:52 |
| *** dave-mccowan has joined #openstack-keystone | 12:54 | |
| neha_alhat | cmurphy: ok. thank you for your inputs | 12:55 |
| cmurphy | no problem | 12:56 |
| *** mchlumsky has joined #openstack-keystone | 12:59 | |
| *** r-daneel has quit IRC | 13:12 | |
| lbragstad | adriant: awesome - it's good to see that impl up for review | 13:32 |
| lbragstad | last release we (specifically cmurphy) had a helluva time fighting the gate despite the implementation for application credentials being ready to go :) | 13:33 |
| knikolla | o/ | 13:34 |
| *** ayoung has quit IRC | 13:34 | |
| cmurphy | lol | 13:34 |
| *** ayoung has joined #openstack-keystone | 13:35 | |
| lbragstad | kmalloc: i'll trade you flaskification reviews for a look at the hierarchical limits spec | 13:35 |
| kmalloc | lbragstad: sold | 13:36 |
| lbragstad | sweet | 13:36 |
| kmalloc | lbragstad: but... you need to answer my question re loading middleware/external apps | 13:36 |
| kmalloc | lbragstad: before reviewing | 13:37 |
| kmalloc | ;) | 13:37 |
| kmalloc | just "do i need to do those things or is a release-note sufficient for now, and/or can the relnote be a followup" | 13:37 |
| lbragstad | if we just do a release note - then people's custom middleware won't work once we land the flask patches, right? | 13:38 |
| kmalloc | right, if anyone has custom middleware they're loading via paste-ini | 13:39 |
| kmalloc | most folks layer things via apache modules or via proxy. | 13:40 |
| kmalloc | wsgi middleware is... weird. | 13:40 |
| lbragstad | the other option would be to support that through configuration/ | 13:42 |
| kmalloc | that was the way i';d implement it | 13:47 |
| kmalloc | through keystone config. | 13:47 |
| kmalloc | i'll need to add a conditional for the oslo_debug middleware, but i could make it more generic | 13:47 |
| kmalloc | loading in API endpoints/extensions is something i'm not thrilled to do, but i *could* implement that as well. | 13:47 |
| lbragstad | hmm | 13:47 |
| kmalloc | i'm inclined to say "no, we don't do that" | 13:47 |
| lbragstad | cmurphy: knikolla thoughts* ^ | 13:48 |
| kmalloc | "use a separate wsgi app for that" | 13:48 |
| kmalloc | so 2 questions: 1) Loadable Middleware (before our pipeline) | 13:48 |
| cmurphy | i'm a little too far removed to have an opinion right now | 13:48 |
| kmalloc | and 2) Loadable wsgi routes to some other app not-keystone (something embeded in our APi) | 13:48 |
| kmalloc | ayoung: ^ you have customer like interactions | 13:48 |
| kmalloc | ayoung: thoughts? | 13:49 |
| ayoung | reading | 13:49 |
| kmalloc | ayoung: context, flaskification patches drop paste.deploy, so without added code you can't wedge new middleware into it (as a deployer) and/or cannot load an extension directly in as a routable path | 13:50 |
| kmalloc | paste let people totally muck with what was run before, during, and after keystone | 13:50 |
| cmurphy | it's not just custom middleware though, is it? things like healthcheck or osprofiler or s3 stuff is all configurable in paste | 13:50 |
| ayoung | We do Tripleo. Break that and all hell escapes | 13:50 |
| kmalloc | ayoung: what are you loading in? | 13:50 |
| kmalloc | because paste is a trainwreck | 13:51 |
| ayoung | paste is just another file. Leave it there but empty so the install process does not break and Tripleo will be fine | 13:51 |
| kmalloc | and should have died a long time ago (it's a dead/unmaintained project as well) | 13:51 |
| lbragstad | cmurphy: those things are converted over to flask apps since we include them by default | 13:51 |
| ayoung | I suspect that someone out there is doing custom middleware, but RH would not support it | 13:51 |
| kmalloc | no, no, not talking about paste being empty, i mean we don't use paste.ini at all | 13:51 |
| cmurphy | lbragstad: okay then what if people want to turn them off? | 13:51 |
| kmalloc | ayoung: i don't care about empty unused files. | 13:51 |
| ayoung | So, some alternative way to load it instead is probably OK | 13:51 |
| lbragstad | it's still configurable (at least osprofiler is) | 13:52 |
| cmurphy | ah ok | 13:52 |
| kmalloc | cmurphy: most of the middleware is not really turn-off-able without other headaches. healthcheck, osloprofiler lines up the list | 13:52 |
| kmalloc | oslo/os/ | 13:52 |
| kmalloc | and osprofiler has it's own config options | 13:52 |
| kmalloc | s3 middleware is application side, keystone has the s3 api baked in. | 13:53 |
| ayoung | kmalloc, so I can give the entirely selfish and non-community supporting advice of KILL IT! KILL IT DEAD! | 13:53 |
| kmalloc | ayoung: so, is triple-o adding or changing middleware in keystone? | 13:53 |
| ayoung | but only cuz I know it won't affect me | 13:53 |
| kmalloc | ayoung: i am in the same boat, trying to kill it. but trying to be reasonable | 13:53 |
| ayoung | nah, Tripleo might have to adjust to the lack of the file, but would be otherwise unaffected | 13:53 |
| kmalloc | ok. cool | 13:54 |
| kmalloc | cmurphy: i'm planning on adding a wsgi conditional for debug middleware, but if we need something more configurable, i'll add more configuration-y things | 13:54 |
| ayoung | A replacement paste.ini file saying "this file is no longer used and will be removed in a future release. If you were modifying this file, you were in a state of sin and you are stuck." | 13:55 |
| ayoung | Ort something a littel nicer maybe | 13:55 |
| kmalloc | ayoung: comment here: https://review.openstack.org/#/c/571979/ to keep the file but not use it (with comments in it) | 13:55 |
| kmalloc | ayoung: please. so i don't forget | 13:55 |
| kmalloc | ayoung: and why it's important (just a, so we are nice to triple-o is fine) | 13:56 |
| cmurphy | okay fwiw i just checked and neither of suse's cloud products allow for custom paste configs for keystone | 13:56 |
| kmalloc | cmurphy: almost nobody changes the paste-ini, we found this with the middleware deprecation(s) | 13:56 |
| lbragstad | cmurphy: ++ | 13:57 |
| kmalloc | and if they did, they did changes we no longer support. | 13:57 |
| kmalloc | cmurphy: thanks for checking :) | 13:57 |
| cmurphy | we do seem to muck with it for trove for some reason but that's not really relevant here :) | 13:57 |
| kmalloc | right, i'm concerned with keystone specifically :) | 13:57 |
| *** r-daneel has joined #openstack-keystone | 13:59 | |
| ayoung | kmalloc, done | 14:01 |
| cmurphy | lbragstad: I finally got around to looking at the limits spec, I'm happy with it if kmalloc or ayoung want to push it through but i noted some concerns | 14:02 |
| knikolla | reminder about the edge computing meeting | 14:02 |
| kmalloc | knikolla: thanks | 14:02 |
| kmalloc | knikolla: what was the info on that meeting? | 14:02 |
| ayoung | "strict two-level hierarchical" still? | 14:02 |
| kmalloc | ayoung: yes, that is what was agreed upon initially | 14:03 |
| cmurphy | ayoung: ja | 14:03 |
| kmalloc | but we can add in the "store upwards" comments if they aren't there | 14:03 |
| ayoung | I thought we realized that was neither necessary nor sufficient | 14:03 |
| ayoung | it is the size of the tree, not the depth, that matters | 14:03 |
| kmalloc | right and the store upwards work is relevant in either case | 14:03 |
| *** germs has joined #openstack-keystone | 14:06 | |
| *** germs has quit IRC | 14:11 | |
| ayoung | so, I think that 2 level is broken. Either people are not doing hierarchy at all, or they are making a decent use out of it, and we can't assume that they are 2 level | 14:14 |
| ayoung | that means we have no quota for the normal case | 14:14 |
| ayoung | and, if this is where the effort goes, it is going to cause a lot of churn | 14:15 |
| ayoung | and...I can't see where it addresses the fact that endpoints don't communicate. | 14:16 |
| ayoung | It is one user hard coding a solution that meets their needs at the expense of the normal case. I can't see why we would not reject that. | 14:17 |
| *** vrv_ has joined #openstack-keystone | 14:18 | |
| *** spilla has joined #openstack-keystone | 14:22 | |
| ayoung | lbragstad, how wedded are you to the idea of 2 level? | 14:22 |
| kmalloc | ayoung: we already are planning more enforcement models. | 14:26 |
| kmalloc | this was the agreed upon 1st hierarchical one | 14:26 |
| kmalloc | but, i'm open to expanding to more depth, just the issue is the "fluid" quota part | 14:26 |
| ayoung | kmalloc so the only thing that is fundamentally broken (and this will be for all) is the per-endpoint part | 14:26 |
| kmalloc | where quota might shift from one branch of the tree to another non-intuatively | 14:26 |
| kmalloc | ayoung: right, and I'm going to take a stab at that | 14:27 |
| *** itlinux has quit IRC | 14:27 | |
| ayoung | but...I think 2 level is a mistake in communication | 14:27 |
| kmalloc | but we still need the enforcement model(s) | 14:27 |
| ayoung | people are going to Hear "keystone only supports two levels" | 14:27 |
| ayoung | and I can see doing damage control for that for years | 14:27 |
| kmalloc | that is a fair reasoning to push to a different model | 14:28 |
| ayoung | I'd rather make a push to get the proper tree stuff working, and then 2 level can be an option after | 14:28 |
| kmalloc | cmurphy ^ thoughts? the communication bit is important. | 14:28 |
| ayoung | but we need to solve per enpoint no matter what | 14:28 |
| ayoung | next cup of coffee is going to be Death Wish. Thank you kmalloc | 14:29 |
| cmurphy | i wasn't able to be at the quota session in yvr but in the ops feedback session and in dublin the feedback we got was that 2-level solved the 80% use case | 14:29 |
| * ayoung needs to dig out grinder | 14:29 | |
| kmalloc | ayoung: ^_^ | 14:29 |
| ayoung | cmurphy, that leaves 20% in the lurch. | 14:29 |
| kmalloc | cmurphy: thats fair, i would like to figure a way to avoid the comms issue though | 14:29 |
| cmurphy | ayoung: which we'll address after we get *something* working | 14:29 |
| cmurphy | if we try to hit the 100% we'll never get anything done | 14:30 |
| kmalloc | ayoung: i'm ok with an 80% solution if we can clearly communicate future/forward | 14:30 |
| kmalloc | and not need damage control | 14:30 |
| ayoung | the question is, if the multi level is the same amount of complexity, does 2 level get us anything? | 14:30 |
| kmalloc | software wont be perfect. | 14:30 |
| *** dklyle has joined #openstack-keystone | 14:30 | |
| ayoung | and...could we do 2 level in the context of full? | 14:30 |
| kmalloc | ayoung: i think the 2-level buys us less non-intuative quota affecting the tree in weird ways | 14:30 |
| kmalloc | that was the key issue | 14:30 |
| ayoung | there is that, too. | 14:30 |
| kmalloc | a->b and a->c->d, where d may not have quota because B consumed it all | 14:30 |
| ayoung | UGH | 14:31 |
| kmalloc | that was the biggest concern, more than the technical "this is expensive to calculate" | 14:31 |
| ayoung | yeah, I hit the same issue in my analysis | 14:31 |
| ayoung | Oh, wait | 14:31 |
| ayoung | no, A NEEDS to be able to set Quota on B and D | 14:31 |
| * lbragstad is in a meeting | 14:31 | |
| lbragstad | but i'll read scroll back later | 14:32 |
| ayoung | A can't overcommit on quota | 14:32 |
| kmalloc | right, it's overcommit | 14:32 |
| ayoung | so...I just reread what you wrote and youi are correct | 14:32 |
| kmalloc | lbragstad: the edgecompute one? | 14:32 |
| lbragstad | y | 14:32 |
| ayoung | the quotas are additive per level, so if A messes with his people, they get starved | 14:32 |
| cmurphy | -_- why are community meetings not in irc | 14:33 |
| kmalloc | lbragstad: i want to sync w/ you after, i couldn't get zoom working (need a vm) because i don't trust .deb packages from websites | 14:33 |
| kmalloc | lbragstad: i'll plan to have a VM setup for the next one as needed. | 14:33 |
| kmalloc | cmurphy: because that is what they chose =/ i don't like it | 14:34 |
| kmalloc | but people want video calls | 14:34 |
| cmurphy | the reason we settled on 2-level was we kept running into logic issues trying to work out more than three levels, sdague documented in https://review.openstack.org/441203 | 14:34 |
| cmurphy | we want to start getting the integration with oslo.limits working and so we need something to work with, after we can do that we can have a second stage with user feedback for what actual use cases there are | 14:35 |
| kmalloc | right, i'm not expecting a perfect solution | 14:36 |
| kmalloc | i want to make sure we have a good solution for that 80%, but not shooting ourselves in the foot comm wise | 14:36 |
| *** spilla has quit IRC | 14:36 | |
| lbragstad | cmurphy: i'm not sure :( to be fair... i'm not sure I've seen any of these folks in irc | 14:36 |
| kmalloc | lbragstad: yeah edgecompute is much the same as some of the other groups that are not as irc-centric. a lot of the business world (and this group is business world) tends to video-call. | 14:37 |
| lbragstad | ildikov: is recording things, too | 14:37 |
| * kmalloc nods. | 14:38 | |
| kmalloc | is there a good person to bring up that zoom.us is not super friendly to some folks and that there wasn't a dial-in phone# i could use. | 14:38 |
| cmurphy | kmalloc: we're documenting a limits model endpoint to hit, once this is implemented there will be two different models available, I don't really buy that anyone would assume we're planning to stop there | 14:38 |
| kmalloc | i really wish i could have been there. | 14:38 |
| kmalloc | cmurphy: right. lets just make sure we're very clear on where we aren't stopping :) | 14:39 |
| knikolla | there's the other edge group which does irc meetings IIRC | 14:39 |
| cmurphy | kmalloc: thumbsup | 14:39 |
| kmalloc | cmurphy: i agree with ayoung it's important to make sure we aren't in damage control mode | 14:39 |
| knikolla | the fog edge massively distributed clouds group | 14:39 |
| kmalloc | but having a workable solution for integration purposes is equally important | 14:39 |
| *** spilla has joined #openstack-keystone | 14:40 | |
| *** eandersson_ has joined #openstack-keystone | 14:45 | |
| *** eandersson has quit IRC | 14:45 | |
| ayoung | ERROR, Child limits exceed parent limits. | 14:46 |
| ayoung | That is the heart of it. Which means that you always check both the child limits and the parent limits. Period | 14:47 |
| ayoung | that is from Garbutt's doc | 14:47 |
| ildikov | lbragstad: kmalloc: recording of the meeting will be posted on the Edge wiki soon | 15:03 |
| ildikov | lbragstad: kmalloc: I will try to encourage people in the future to try to take some notes on the IRC channel for easier recap on important info | 15:04 |
| lbragstad | ildikov: kmalloc cmurphy knikolla my rough notes https://gist.github.com/lbragstad/913dcb4e79495beadbb64f855f6a619c | 15:04 |
| cmurphy | ildikov: some official-ish minutes would be nice, it's hard to digest an hour-long recording after the fact :) | 15:05 |
| *** pcaruana has quit IRC | 15:05 | |
| ildikov | cmurphy: I know, I will try to write up a summary some time this week | 15:06 |
| ildikov | cmurphy: I wanted to ask people at the beginning of the meeting to take notes, but forgot | 15:06 |
| ildikov | cmurphy: and despite of being a woman with good multitasking skills, I can't type and talk at the same time... :/ | 15:07 |
| cmurphy | ildikov: that's okay, i can barely do either of those things one at a time ;) | 15:07 |
| ildikov | cmurphy: yeah, I can relate to that too :) | 15:08 |
| *** wxy| has joined #openstack-keystone | 15:15 | |
| openstackgerrit | wangqi proposed openstack/oslo.limit master: fix broken link https://review.openstack.org/572442 | 15:18 |
| *** itlinux has joined #openstack-keystone | 15:19 | |
| openstackgerrit | melissaml proposed openstack/keystone-specs master: Replace Chinese quotes to English quotes https://review.openstack.org/544773 | 15:19 |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/keystone master: Filter by entity_type in get_domain_mapping_list https://review.openstack.org/572446 | 15:25 |
| lbragstad | ayoung: responding to your question about 2 levels versus 3 | 15:30 |
| lbragstad | the decision to go to 2 levels was primarily driven by support use cases | 15:30 |
| lbragstad | and usability | 15:31 |
| lbragstad | if someone has limit use cases that don't fit the strict two level, then they can use flat and manually adjust the knobs how ever they want | 15:32 |
| lbragstad | until we work out an enforcement model that works for their use cases | 15:32 |
| lbragstad | afaik, that was the primary driver behind the concept of enforcement models | 15:32 |
| lbragstad | kmalloc: as far as the flask bits go | 15:34 |
| lbragstad | kmalloc: if we keep things configurable with what we have by default today, then i'm fine | 15:34 |
| *** lifeless_ has joined #openstack-keystone | 15:35 | |
| lbragstad | kmalloc: that said, it might be worth a note to the mailing list raising this issue, just describing what we want to do and what that means for people relying on it - if there are any | 15:35 |
| lbragstad | s/mailing list/operator and development mailing list/ | 15:35 |
| *** lifeless has quit IRC | 15:36 | |
| lbragstad | we're not going to be, or probably aren't the first project faced with this decision | 15:36 |
| lbragstad | with the amount of projects on paste and looking to move from it | 15:36 |
| kmalloc | lbragstad: we wont have the same level of configurability in flask without paste | 15:37 |
| kmalloc | lbragstad: fundamentally | 15:37 |
| kmalloc | lbragstad: so but... | 15:37 |
| kmalloc | lbragstad: so you want me to enable custom middleware via options | 15:38 |
| lbragstad | right - what i meant is things like the ability to turn osprofiler on and off | 15:38 |
| kmalloc | oh ok | 15:38 |
| openstackgerrit | Harry Rybacki proposed openstack/keystone master: Ensure default roles created during bootstrap https://review.openstack.org/572243 | 15:38 |
| kmalloc | then that is a non-issue, i wont be changing that | 15:38 |
| lbragstad | like - what we have by default in paste today | 15:38 |
| kmalloc | perfect | 15:38 |
| kmalloc | i'll add a toggle (followup) for debug | 15:38 |
| kmalloc | since we can do that today | 15:38 |
| kmalloc | i'll create a new [wsgi] group for future things like that | 15:38 |
| lbragstad | cool | 15:38 |
| lbragstad | that'd be good | 15:38 |
| kmalloc | if we need to enable custom middleware, we can add that functionality to [wsgi] group | 15:39 |
| kmalloc | i'll get release note, and toggle done | 15:39 |
| kmalloc | then mail to the ML for /v2.0 | 15:39 |
| kmalloc | and then we can land flaskification | 15:39 |
| kmalloc | and i'll start work on the blueprint stuff and code shuffle | 15:39 |
| kmalloc | let me go review limits | 15:40 |
| kmalloc | now | 15:40 |
| kmalloc | lbragstad: what review # is the limits spec? | 15:41 |
| kmalloc | found it | 15:41 |
| kmalloc | https://review.openstack.org/#/c/540803/ | 15:41 |
| kmalloc | ayoung: that spec cannot explicitly deal with per-endpoint | 15:42 |
| kmalloc | ayoung: it is not in the scope of the spec itself, per-endpoint is a followup | 15:42 |
| ayoung | nope | 15:42 |
| kmalloc | we don't have the mechanism to share the data yet | 15:42 |
| ayoung | gotta be addressed or it is going to be assigned quote * endpoint | 15:42 |
| kmalloc | that is currently, acceptible and how it works with the current quota system | 15:43 |
| ayoung | make it explicit | 15:43 |
| kmalloc | please move your -2 to a -1 and comment i'll not kick it through if that level of explicit is needed | 15:43 |
| kmalloc | heck i'll respin to add that | 15:43 |
| kmalloc | but please don't hold up if that is the -2, that is a -1 worthy comment (and should be addressed before landing) | 15:44 |
| kmalloc | if the -2 is because of insufficiency, we need to get you, lbragstad, and cmurphy on the same page. | 15:44 |
| kmalloc | so we can move it forward. i am not clear on the -2 reason. | 15:44 |
| lbragstad | ayoung: i need more detail about the -2 - otherwise you're leaving me fishing through your blog post hoping wxy and i can figure out what the issue is | 15:45 |
| *** mchlumsky has quit IRC | 15:45 | |
| kmalloc | i am 100% ok making it explicit that this does not address per-endpoint quota and that will be built on top of what we're doing now | 15:45 |
| kmalloc | ftr. and i think we should call that out | 15:46 |
| ayoung | lbragstad, per endpoint is essential. And I would like a decent argument that 2 level is more important than solving multi level other than "we spent more time on it" | 15:46 |
| ayoung | I am not convinced that 2 level is a good thing. | 15:46 |
| lbragstad | ayoung: i already answered that question, it was related to support | 15:46 |
| ayoung | I am convinced that multi-level is needed, and will solve 2 level | 15:46 |
| lbragstad | if you have a tree of more than 2 levels, and a cousin project uses the last 2 cores of the entire tree | 15:47 |
| *** mchlumsky has joined #openstack-keystone | 15:47 | |
| ayoung | lbragstad, what is going to happen is we are going to get 2 level, and then we are going to turn our attention elsewhere | 15:48 |
| kmalloc | lbragstad: it's the fluid quota model with oversub. | 15:48 |
| lbragstad | what am i supposed to put in the support ticket if i'm a user on the other side of the tree?" | 15:48 |
| kmalloc | that is the issue there | 15:48 |
| ayoung | the 2 level as definied here is , I suspect, still vulnerale to gaming | 15:48 |
| kmalloc | it's oversub vs strict. | 15:48 |
| *** wxy| has quit IRC | 15:48 | |
| ayoung | multi level with decent error messages: | 15:48 |
| lbragstad | i isn't susceptible to gaming when you calculate usage of the tree | 15:48 |
| lbragstad | it isn't* | 15:48 |
| *** wxy| has joined #openstack-keystone | 15:49 | |
| kmalloc | strict [no oversub], while game-able [in some cases, see adam's blog and i can explain further], doesn't suffer from the weird error state | 15:49 |
| ayoung | "The action would exceed your quota assigned by 'Parent'." Versus "the action would exceed quote for 'Parent' assigned by 'System'. | 15:50 |
| openstackgerrit | wangqi proposed openstack/oslo.limit master: fix link https://review.openstack.org/572442 | 15:50 |
| ayoung | so..get the endpoint thing at a minimum. But with edge, I really see that as a deal breaker | 15:50 |
| *** jmlowe has quit IRC | 15:50 | |
| ayoung | we need to be able to assign quota per endpoint. | 15:50 |
| kmalloc | ayoung: ok, so if we solve the per-endpoint bit [forward looking/explicit statement on what is implemented there] can the -2 be removed | 15:50 |
| kmalloc | ? | 15:50 |
| ayoung | does not have to be for all quota | 15:50 |
| kmalloc | i can add in the per-endpoint stuff | 15:51 |
| ayoung | I don't love it, but sure | 15:51 |
| ayoung | I'd rather you did it right, as it is going to cause more work for everyone, but not a hill for me to stand on | 15:51 |
| kmalloc | honestly i don't understand the need for oversub on children | 15:52 |
| kmalloc | but my view is largely oversub is bad(tm) in quotas. | 15:53 |
| lbragstad | i'm still missing context as to why everything is terribly broken | 15:53 |
| kmalloc | lbragstad: it isn't. | 15:53 |
| lbragstad | it certainly sounds like it is | 15:53 |
| ayoung | kmalloc, is oversub a "two layer" thing ? | 15:53 |
| kmalloc | oversub is the reason for two-layer limit | 15:53 |
| ayoung | what is it called in there? | 15:54 |
| kmalloc | it is about UX. | 15:54 |
| kmalloc | [end user ux] | 15:54 |
| kmalloc | it is the argument that a->b and a->c->d causes issues if b consumes all the quota | 15:54 |
| kmalloc | and d doesn't get any when it should have available | 15:54 |
| kmalloc | because D's quota + b + C > a | 15:54 |
| kmalloc | (limit) | 15:55 |
| kmalloc | communicating that with N depth is very very very hard to do clearly | 15:55 |
| lbragstad | right | 15:55 |
| kmalloc | without oversub on children, non-issue | 15:55 |
| kmalloc | N depth is 100% a-ok | 15:55 |
| kmalloc | and not hard to communicate clearly | 15:55 |
| kmalloc | "you are over quota" | 15:55 |
| kmalloc | done. | 15:55 |
| kmalloc | ;) | 15:55 |
| kmalloc | lbragstad: i'll add details about per-endpoint sharing of quota | 15:55 |
| kmalloc | data | 15:56 |
| lbragstad | i'd like to understand it before you push a new version | 15:56 |
| kmalloc | should be straightforward to do, which should reolve the bulk of the -2 | 15:56 |
| ayoung | Endpoints are going to break this | 15:56 |
| ayoung | same issues with gaming | 15:56 |
| kmalloc | basically, "this initially doesn't share quota across endpoints, so each nova is granted the entire volume of quota" | 15:56 |
| ayoung | I want to split my quota over two endpoint, I want to delet quote from one endpoint, etc | 15:56 |
| ayoung | we had, like 5000 meetings on edge at the summit | 15:56 |
| kmalloc | "we will build on top of this, use of etcd to store the shared quota state" | 15:57 |
| ayoung | is that really a viable approach? | 15:57 |
| ayoung | anyway, I'll downgrade to -1 | 15:57 |
| kmalloc | ayoung: ^ is that sufficient (better words to come)? | 15:57 |
| *** bigdogstl has joined #openstack-keystone | 15:57 | |
| kmalloc | before i start generating all the verbiage and updating the spec. | 15:57 |
| kmalloc | and yes, i will commit to writing the bulk of that etcd bit. | 15:58 |
| ayoung | kmalloc, I really don't know what is the right solution. The division of services we have in OpenStack may make it an impossible problem to solve, and I don't have the cycles to spend on it now | 15:58 |
| kmalloc | i am asking from a unblock the spec perspective. | 15:58 |
| ayoung | making changes in keystone without communicating them to the services seems "broken by design" | 15:59 |
| ayoung | I unblocked | 15:59 |
| kmalloc | but you're still not really answering my question | 15:59 |
| kmalloc | if that had been in the spec, would you have -2'd based upon per-endpoint data sharing issues | 15:59 |
| kmalloc | ? | 15:59 |
| ayoung | kmalloc, I think that per endpoint might be too hard to solve, so the comment you have there is fine. | 16:00 |
| kmalloc | ok. | 16:00 |
| ayoung | And that was the deal breaker that lead to the -2 | 16:00 |
| *** gyee has joined #openstack-keystone | 16:00 | |
| ayoung | I don't think 2 level really is the right approach, but I'm not implementing, so I can live with it | 16:00 |
| *** bigdogstl has quit IRC | 16:02 | |
| *** links has quit IRC | 16:03 | |
| *** germs has joined #openstack-keystone | 16:07 | |
| *** germs has quit IRC | 16:07 | |
| *** germs has joined #openstack-keystone | 16:07 | |
| *** germs has quit IRC | 16:09 | |
| *** germs has joined #openstack-keystone | 16:09 | |
| *** germs has quit IRC | 16:09 | |
| *** germs has joined #openstack-keystone | 16:09 | |
| *** germs has quit IRC | 16:09 | |
| *** germs has joined #openstack-keystone | 16:10 | |
| *** harlowja has joined #openstack-keystone | 16:28 | |
| lbragstad | rm_work: ping | 16:33 |
| *** issp has quit IRC | 16:44 | |
| *** harlowja has quit IRC | 16:45 | |
| *** bigdogstl has joined #openstack-keystone | 16:53 | |
| knikolla | kmalloc: the x1? | 16:58 |
| knikolla | yoga or carbon | 16:58 |
| *** jaosorior has quit IRC | 16:58 | |
| lbragstad | rm_work: fyi - http://eavesdrop.openstack.org/meetings/keystone/2018/keystone.2018-06-05-16.00.log.html#l-164 | 16:58 |
| kmalloc | knikolla: i can have an x1c6 [1080p], P50, T480 [1080p], or macbook pro 15/13 or macbook air | 16:59 |
| knikolla | kmalloc: no hidpi screens? | 16:59 |
| kmalloc | knikolla: i think i'm going to go x1c6, but honeslty it wont replace my current x1c4 because 1080p | 17:00 |
| kmalloc | knikolla: not offered as the "standard" configs | 17:00 |
| kmalloc | all 1080p | 17:00 |
| kmalloc | makes me sad. | 17:00 |
| kmalloc | my x1c4 is a personal machine. | 17:00 |
| kmalloc | and hidpi is a hard requirement for me to use anything else. | 17:00 |
| knikolla | damn :/ non hidpi is a blocker for me. | 17:00 |
| knikolla | i asked for an xps 13 with a 4k screen. but answer was let's talk again end of summer. | 17:01 |
| kmalloc | i'll buy a laptop for myself and use it over a corp one without hidpi | 17:02 |
| *** masuberu has quit IRC | 17:02 | |
| kmalloc | though, $2k+ isn't in my budget right now. | 17:02 |
| knikolla | ++, that was what i was gonna do until i realized i couldn't afford it. | 17:02 |
| kmalloc | so, x1c4 for the time being [even though it's on it's last legs] | 17:02 |
| *** masuberu has joined #openstack-keystone | 17:02 | |
| kmalloc | speaker busted, WWAN sim no longer reading, screen issues, battery issues, and warranty is up | 17:03 |
| knikolla | that sucks. | 17:03 |
| kmalloc | and you saw the laptop, it wasn't in bad shape/abused (at the summit) | 17:03 |
| knikolla | yeah, i'm suprised. thinkpads usually last forever. | 17:03 |
| knikolla | macbook pro (late 2013) is still working flawlessly | 17:04 |
| knikolla | and ubuntu works amazingly well | 17:04 |
| lbragstad | #startmeeting keystone-office-hours | 17:04 |
| openstack | Meeting started Tue Jun 5 17:04:49 2018 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:04 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:04 |
| openstack | The meeting name has been set to 'keystone_office_hours' | 17:04 |
| *** wxy| has quit IRC | 17:11 | |
| rm_work | lbragstad: ah yeah, i thought we basically came to a conclusion on that in-channel at the time, and that we'd review the available developer volunteers and approaches in Denver at the PTG, but that the general thought was "YES WE WANT THIS" it just wasn't prioritized | 17:25 |
| rm_work | so i didn't think we needed to discuss it in the meeting :P | 17:25 |
| rm_work | but, reading through that now | 17:25 |
| *** lifeless has joined #openstack-keystone | 17:26 | |
| rm_work | i THINK people had the right idea | 17:28 |
| *** AlexeyAbashkin has quit IRC | 17:28 | |
| *** lifeless_ has quit IRC | 17:28 | |
| rm_work | we really shouldn't be *removing records*. and yeah, obviously a deleted project would not be significantly different than a disabled one -- honestly delete just being an alias to disable is practically acceptable from my PoV, it's just a terminology issue | 17:29 |
| rm_work | we can't retrain end users to "please use disable instead of delete" because that just isn't feasible | 17:29 |
| rm_work | imagine trying to get every random user of your cloud (who you mostly have no power over, they're just people who sign up / pay you) to use a different method that doesn't mesh with every other project in existence | 17:30 |
| rm_work | "When we don't want a resource anymore, we delete it" <-- is what everyone thinks, for everything | 17:30 |
| rm_work | i wasn't ever saying we need an undelete (though I'm sure it could essentially be done manually without too much trouble, just to get the project itself back in an urgent case) | 17:31 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone-specs master: Hierarchical Unified Limits https://review.openstack.org/540803 | 17:31 |
| lbragstad | kmalloc: i was just about to start reading this | 17:32 |
| kmalloc | lbragstad: ^ addressing ayoung's concerns with the explicit endpoint data sharing and some reasoning for the strict two-level, calling out we allow oversubscription | 17:32 |
| rm_work | i think ayoung and I agree in principle and he's on the same page as I am generally | 17:32 |
| kmalloc | rm_work: the way i would accept this change (soft delete): change how delete works to always be soft delete, still do the "delete" process but maintain records | 17:33 |
| rm_work | yep | 17:33 |
| rm_work | that's basically all i want | 17:33 |
| kmalloc | rm_work: i am a hard -2 for a "new and special delete" | 17:33 |
| kmalloc | all deletes should become soft (for projects and other must maintain records) | 17:33 |
| rm_work | delete should just ... set the disabled flag, hide the record from returning in lists | 17:33 |
| kmalloc | no, do not used "disabled" | 17:33 |
| kmalloc | please | 17:33 |
| kmalloc | delete should still remove grants | 17:34 |
| kmalloc | etc | 17:34 |
| rm_work | ah, k, yeah, that's prolly fine | 17:34 |
| rm_work | i just mean "it should also do all the same things disable does" | 17:34 |
| rm_work | IE, can't create tokens | 17:34 |
| kmalloc | right. | 17:34 |
| kmalloc | disable is a lot less invasive | 17:34 |
| ayoung | what is the word for the trivial case of something | 17:34 |
| rm_work | and whatever else, because a deleted project is a superset of disabled | 17:34 |
| kmalloc | disable is toggle, soft delete should never [really] expected to be toggled back except <extreme> cases | 17:35 |
| ayoung | like...when a complicated solution has a base case that could be covered by a simpler solution? | 17:35 |
| kmalloc | you will also need a new API to purge records [sanely] or a keystone-manage command | 17:35 |
| kmalloc | based upon <before datetime> | 17:35 |
| rm_work | yeah i imagined something like what ... someone brought up, one sec | 17:35 |
| lbragstad | stepping away to grab lunch quick and i'm going to go through the unified limit stuff | 17:35 |
| *** isssp has joined #openstack-keystone | 17:35 | |
| rm_work | ah yeah, you | 17:36 |
| kmalloc | hehe | 17:36 |
| kmalloc | ;) | 17:36 |
| ayoung | kmalloc, I'm with you on the "don't just use disabled" etc. | 17:36 |
| rm_work | 16:43:25 <kmalloc> and track deleted time with a "cleanup" option so "remove records for projects with deleted and deleted before <datetime> | 17:36 |
| kmalloc | ayoung: ++ disabled has it's use. | 17:36 |
| rm_work | actually i just said that because i thought it'd be MORE palatable | 17:36 |
| ayoung | rm_work I hear ya | 17:36 |
| kmalloc | "you didn't pay, we're turning your spigot off" | 17:36 |
| rm_work | i don't really care how it works, as long as the project DB record stays in the DB, and doesn't show up to users but does show up to admins | 17:36 |
| kmalloc | delete is "you still haven't paid and we're closing your account" | 17:37 |
| kmalloc | rm_work: you'll probably want a new api for deleted record introspection | 17:37 |
| ayoung | and we should be able to disable, but still get tokens to delete resources for a disables project...maybe with app creds or something. But that is a different feature | 17:37 |
| rm_work | and further deletes and such should probably 404? but that's getting deep into implementation details | 17:37 |
| kmalloc | ayoung: yep, different features | 17:37 |
| kmalloc | rm_work: if it's already deleted it works the same as already deleted today ;) | 17:37 |
| rm_work | yeah, for most other projects you can do a ?deleted=True and it'll allow showing/listing deleted objects | 17:38 |
| kmalloc | rm_work: sure, just make sure it's independent of disabled=True or whatever we use for that | 17:38 |
| rm_work | yes, for all intents and purposes to a regular user it should seem like nothing changed from the way it works now | 17:38 |
| rm_work | but for admins, we should be able to go in and still see the project if we want | 17:38 |
| kmalloc | so the (implementation detail warning) change is a going to be a new column and some constraint changes for project_name | 17:39 |
| kmalloc | as delete frees project name | 17:39 |
| kmalloc | and some changes to how delete propagates (delete from -> update) | 17:39 |
| rm_work | hmmm | 17:39 |
| *** ispp has quit IRC | 17:40 | |
| kmalloc | if you make a unique constraint and deleted can be null, be aware that null is never considered to collide in mysql | 17:40 |
| kmalloc | so (PName, Null) doesn't collide with (Pname, Null) | 17:40 |
| rm_work | heh, noted | 17:40 |
| rm_work | yeah i honestly am not 100% sure whether I WILL be able to work on this or not, by the time Denver rolls around. will see. | 17:40 |
| kmalloc | sure. | 17:41 |
| rm_work | I would like to. I don't at first glance feel like this should be too involved | 17:41 |
| rm_work | but i am sure it will rabbit-hole quickly ;P | 17:41 |
| ayoung | so what is the word. The 2 level quota is the (blank) of the multi level approach | 17:41 |
| ayoung | logical extrame? | 17:41 |
| rm_work | distillation? | 17:42 |
| ayoung | rm_work, not quite the word I am looking for...that means the essense, and I am looking for the "trivial case" or something | 17:44 |
| ayoung | and maybe that is the term I want? | 17:44 |
| ayoung | anyway, I see no benefit of 2 level. It requires the same overhead as the multilevel | 17:44 |
| ayoung | just forces you to a flat, wide tree, but it will be just as expensive to calculate and query | 17:45 |
| ayoung | For any project, we need to know its parent | 17:45 |
| ayoung | inside of nova | 17:45 |
| ayoung | the multi level will then chain from parent to grand parent and so on | 17:46 |
| ayoung | the two level will not, but the same amount of info needs to be synced between keysteon and e.g. nova | 17:46 |
| ayoung | we can put restrictions on people moving quota around such that you can never allocate more quota to your child projects than you yourself have, and then there is no "you parent quote was exceeded" | 17:47 |
| ayoung | that is just a gotcha that we need to make sure we cover, not fundamental to the two level | 17:47 |
| ayoung | so, yeah, 2 level dumb | 17:47 |
| kmalloc | except even in the case of that, you run into | 17:47 |
| kmalloc | non-strict enforcement | 17:47 |
| kmalloc | or does it. | 17:48 |
| kmalloc | *me drinks more coffee* | 17:48 |
| kmalloc | right so no oversub quota | 17:48 |
| ayoung | no over sub quota | 17:49 |
| kmalloc | ayoung: oversub is the fundamental piece | 17:49 |
| kmalloc | that dictates need for two-level | 17:49 |
| ayoung | kmalloc, ah, I reemmber | 17:50 |
| ayoung | yeah, the oversub is a side effect of not squaring things with Keystone | 17:50 |
| ayoung | it is kinda fundamental, but I think 2 level suffers from it as well | 17:50 |
| kmalloc | sortof... | 17:50 |
| kmalloc | but oversub is a method that people do use | 17:50 |
| ayoung | say A->B A->C each has 4 units of a parent 8 unit quota | 17:50 |
| kmalloc | "i give you 100 cores, and each child 100 cores even though i only have 100" | 17:51 |
| kmalloc | the quota system can tell youre in aggregate over | 17:51 |
| ayoung | B maexs out | 17:51 |
| kmalloc | but people like being able to oversub and then just buy up when children aggregate hit limit | 17:51 |
| ayoung | maxes out at 4, C has 0 | 17:51 |
| ayoung | A then removes from B, creates D | 17:51 |
| kmalloc | oversub is a headache, but common practice | 17:51 |
| ayoung | A->D gets 4 | 17:51 |
| ayoung | D allocates all | 17:52 |
| kmalloc | no, the not squaring with keystone is solved with the store upwards concept | 17:52 |
| ayoung | C still has 4 quota, but it puts A over the limit | 17:52 |
| kmalloc | so we can be sure we can (cheaply?) calculate the total usage | 17:52 |
| ayoung | meh | 17:52 |
| kmalloc | oversub is a choice made on representation to child projects | 17:52 |
| ayoung | store upwards means what? That A cannot remove quota from B? | 17:53 |
| ayoung | Cuz that is really where the oversub comes from | 17:53 |
| kmalloc | no, consumption of quota is stored up to the parent | 17:54 |
| kmalloc | in aggregate | 17:54 |
| ayoung | so A cannot assign quota to B? | 17:55 |
| ayoung | no hierarchy? That seems to contradict the name of the spec | 17:55 |
| kmalloc | so when i make a claim on quota in C (a->b->c) it stores a record upwards that consumption for C is used and A gets a record of aggregate of b+c | 17:55 |
| kmalloc | it's about storing claim aggregate usage upwards | 17:55 |
| kmalloc | in the hierarchy | 17:55 |
| ayoung | right] | 17:55 |
| kmalloc | it means worst case to check usage we go to the top of the tree | 17:56 |
| kmalloc | that doesn't have to square with keystone ever | 17:56 |
| ayoung | but the cost is based on the number of nodes in the tree, not the depth | 17:56 |
| kmalloc | because if A tries to game the system i am fine with saying "over quota" | 17:56 |
| kmalloc | i am less ok with something D does affecting a different branch of the tree | 17:56 |
| kmalloc | under a | 17:56 |
| kmalloc | a->b->c and a->d | 17:56 |
| ayoung | yeah, I don't think that is possible | 17:57 |
| kmalloc | with oversub it is | 17:57 |
| ayoung | its omnly possible if A plays games, not D | 17:57 |
| kmalloc | oversub means C+B+D might have more "allowed [not consumed]" quota than A | 17:57 |
| kmalloc | so D could use all the quota for B and C | 17:57 |
| kmalloc | with no games being played by A | 17:57 |
| ayoung | Let me mull it over to see if there are peer-to-peer games, but I think that only A can do *that* to its Pledges | 17:57 |
| kmalloc | that is bad UX | 17:57 |
| ayoung | oversub is only possible if A plays games | 17:58 |
| ayoung | by not removing actual resources when it removes quota | 17:58 |
| kmalloc | in the model proposed oversub is explicitly allowed without games | 17:58 |
| kmalloc | A may allocate it's entire quota to A, B, C, and D | 17:58 |
| kmalloc | at the same time | 17:58 |
| ayoung | that is just "don't divide the quota on sub projects" and is suppored by multi level as well | 17:58 |
| kmalloc | but the consumption check prevents use once A's quota is hit | 17:58 |
| ayoung | yeah, that is covered in multilevel | 17:59 |
| kmalloc | so, 2 fundamental issues: | 17:59 |
| ayoung | you are saying the fact that A can have deep trees just makes it more surprising? | 17:59 |
| kmalloc | 1) Oversub makes for icky ux beyond ~2 layers | 17:59 |
| ayoung | Got it | 17:59 |
| kmalloc | yeah | 17:59 |
| ayoung | what if we don't allow oversub beyond to levels? | 17:59 |
| kmalloc | because D has no insight to it's peers really | 17:59 |
| ayoung | or... | 17:59 |
| kmalloc | so it's very surprising when D runs out of quota | 17:59 |
| ayoung | that was the idea of quota pools | 18:00 |
| ayoung | the pool id is the id of the project that owns the quota | 18:00 |
| ayoung | which is you or someone in your tree | 18:00 |
| kmalloc | i think the solution to store aggregate consumption to the parent(s) is easier than needing to defrag quota pools. | 18:00 |
| ayoung | kmalloc, perhaps | 18:00 |
| kmalloc | it's net the same thing. | 18:00 |
| ayoung | got ameeting | 18:00 |
| kmalloc | sure. | 18:00 |
| kmalloc | chat later | 18:01 |
| *** jmlowe has joined #openstack-keystone | 18:01 | |
| lbragstad | ok - does quota mean limit or usage? | 18:03 |
| lbragstad | we've been referring to limit and usage as two distinct, but related, things | 18:04 |
| lbragstad | and i'm not exactly sure what people mean by quota now | 18:04 |
| *** bigdogstl has quit IRC | 18:04 | |
| *** bigdogstl has joined #openstack-keystone | 18:09 | |
| kmalloc | lbragstad: i always specify Limit as what is stored in keystone (allowance) | 18:09 |
| kmalloc | lbragstad: and quota claim/consumption as what is used (not stored in keystone) | 18:09 |
| kmalloc | just for clarity, quota is BOTH things, so it needs a specifier | 18:10 |
| kmalloc | you can't say quota without being explicit what side you're looking at | 18:10 |
| lbragstad | https://docs.openstack.org/keystone/latest/admin/identity-unified-limits.html#limits-and-usage | 18:10 |
| ayoung | kmalloc, OK, store up can and should be done even in the multi-level approach | 18:10 |
| kmalloc | we may need to update those docs to reflect it | 18:10 |
| ayoung | there mechanism is this: | 18:11 |
| kmalloc | ayoung: totally i added a comment to the spec to highlight it | 18:11 |
| lbragstad | kmalloc: how so? those docs look fine to me | 18:11 |
| ayoung | there is no distinction between allocating quota for a new resource or splitting for a sub quota | 18:11 |
| kmalloc | lbragstad: if we are unclear | 18:11 |
| lbragstad | they describe keystone as being the place for limits and usage being calculated by services | 18:11 |
| kmalloc | lbragstad: right i haven't dug deep, i said "may" assuming we crossed the definitions somewhere | 18:11 |
| kmalloc | it wasn't a "it's wrong" comment, sorry, split attention between the two docs (new spec, old current documentation) | 18:12 |
| ayoung | so A->B->C add a resource to C we add the resource to C, reduce 1 fro C available, bubble up, reduce 1 from B available, bubble up reduce 1 from A. | 18:12 |
| lbragstad | so - is a "quota system" still referring to the coordination between limits and keystone and usages at the service? | 18:12 |
| kmalloc | ayoung: basically. | 18:12 |
| kmalloc | lbragstad: both. | 18:12 |
| lbragstad | limits in keystone* | 18:13 |
| kmalloc | lbragstad: you really can't have one without the other (well you can, but it's silly) | 18:13 |
| kmalloc | you need both limits (allowance) and usage | 18:13 |
| lbragstad | right - and when people reference quota, that's what they are talking about | 18:13 |
| lbragstad | ? | 18:13 |
| kmalloc | yes. | 18:13 |
| lbragstad | ok - that helps | 18:13 |
| kmalloc | if you're tracking usage without limits for billing purposes, it's just "usage" | 18:14 |
| kmalloc | quota adds in a allowance/cap | 18:14 |
| ayoung | lbragstad, so you still want to pursue 2 level? And if so, only due to pressure to get spec approved? | 18:14 |
| lbragstad | ayoung: i'm still about 10 steps behind understanding the reason why you don't like it | 18:14 |
| lbragstad | i'm trying to catch up | 18:15 |
| ayoung | lbragstad, it think the 2 level restriction is not really buying anything, and is as expensive as the multi-level | 18:15 |
| ayoung | why half-ass something when you can whole-ass it? | 18:15 |
| * ayoung apologizes | 18:16 | |
| lbragstad | ayoung: ok - so 1.) you don't see the value in it 2.) you consider it still expensive, yeah? | 18:16 |
| ayoung | lbragstad, and 3) we will be doing damage control once we expand to the multi level | 18:17 |
| lbragstad | what makes you think it doesn't buy us anything? | 18:17 |
| kmalloc | lbragstad: if we have to look up every child, it is expensive beyond a very small number of children | 18:17 |
| lbragstad | ok - let's just start with one | 18:17 |
| ayoung | right, and that is not based on depth of tree, but on number of children | 18:17 |
| lbragstad | for the sake of me being slow | 18:17 |
| ayoung | If we have a wide flat tree with 1000 nodes the work is the same as a deep tree with 1000 nodes | 18:18 |
| kmalloc | so, i wont comment to the two/not-two level. i am not opposed/for it [with exception of oversub/non-oversub ux issues] | 18:18 |
| ayoung | And I won't hold things up, just want it to be a deliberate decision | 18:19 |
| lbragstad | by work are you referring usage calculation? | 18:19 |
| kmalloc | yes | 18:19 |
| kmalloc | collection and calculate usage when making a new claim | 18:19 |
| kmalloc | the "are we over quota" check | 18:19 |
| ayoung | and same for storage requirements | 18:19 |
| lbragstad | fwiw - the reasonsing being two-level wasn't because calculating usage was harder for either | 18:19 |
| lbragstad | if you have 1000 projects, usage calculation will be the same regardless of what the tree looks like | 18:20 |
| lbragstad | so - sure, i'll agree there | 18:20 |
| lbragstad | the main reason for the two-level bit was from dublin | 18:20 |
| kmalloc | and that issue, i have a solution for [see comment on spec] the wide tree issue | 18:20 |
| lbragstad | in filling out support tickets | 18:20 |
| lbragstad | if someone can't create an instance because someone in a third cousin project used up the last of the limit allocation, what is that person supposed to put in a support ticket? | 18:21 |
| lbragstad | that is actually useful for operators? | 18:22 |
| lbragstad | with two level project hierarchies, everyone knows the parent | 18:22 |
| kmalloc | is oversubscription of children actually useful? i don't have an answer | 18:22 |
| *** fiddletwix has joined #openstack-keystone | 18:22 | |
| kmalloc | i'm not being rhetorical here, is it useful, if it is, then the UX issue you describe is real | 18:23 |
| lbragstad | if i'm an operator | 18:23 |
| kmalloc | if it isn't useful, then we can not allow oversub and dodge the issue. | 18:23 |
| lbragstad | and I manage a deployment with a depth of 30 levels | 18:23 |
| lbragstad | and someone at the 29th level reports a ticket like that | 18:24 |
| lbragstad | i'm going to have a *real* long day tracking down where in the tree in need to make adjustments | 18:24 |
| kmalloc | right, but this only comes up if the aggregate limit (allowance) for all children is large than the parent | 18:24 |
| kmalloc | (width/depth included) | 18:24 |
| kmalloc | so is oversubscription useful? | 18:24 |
| kmalloc | s/large/larger | 18:25 |
| lbragstad | according to the requirements johnthetubaguy proposed from CERN, it sounds like it is | 18:25 |
| kmalloc | ok, that is fine then, i didn't have a good answer | 18:25 |
| lbragstad | let me see if i can findit | 18:25 |
| kmalloc | that to me tells us that in the case we allow oversub we need to be either limiting depth or adding smarts like "quota=100, oversub_children_allowance=[true|false]" so the operator can divine where the top of that "omg something borked" is | 18:26 |
| lbragstad | http://lists.openstack.org/pipermail/openstack-dev/2017-February/111999.html | 18:26 |
| kmalloc | and i don't really like oversub as a dynamic property of the individual limit | 18:26 |
| lbragstad | "Sub project over commit is OK (i.e. promising your sub projects more is OK, sum of the commitment to subprojects>project is OK but should be given an error if it actually happens)" | 18:26 |
| kmalloc | ok, overcommit is a requirement, that answers my question | 18:27 |
| kmalloc | overcommit is something we need to address in the quota limit model. | 18:27 |
| lbragstad | i imagine it being useful for deployments looking for resources to flow between sub-projects | 18:28 |
| kmalloc | and that makes deep trees tricky | 18:28 |
| lbragstad | and not having to be tightly coupled to tinkering with limits | 18:28 |
| kmalloc | i agree | 18:28 |
| lbragstad | when things are in flux | 18:28 |
| kmalloc | but i wanted to be sure we had that clearly delineated | 18:28 |
| kmalloc | because if it wasn't, i was going to ask more critically why we are allowing overcommit | 18:29 |
| lbragstad | that was the reason wxy and johnthetubaguy included it in their proposals, afaict | 18:29 |
| * kmalloc nods. | 18:29 | |
| kmalloc | soooooo then. | 18:29 |
| *** jmlowe has quit IRC | 18:30 | |
| kmalloc | i see two ways out of this.. well 3 | 18:30 |
| kmalloc | 1) as is, two level limit | 18:30 |
| kmalloc | 2) overcommit=true (on the limit definition) | 18:30 |
| kmalloc | 3) config value for max limit and hang the ux issues [please don't pick this one] | 18:30 |
| kmalloc | 4) clever solution to error reporting to end users | 18:31 |
| kmalloc | (see, off-by one errors, common) | 18:31 |
| kmalloc | and honestly, i am fine with any of those options. | 18:31 |
| kmalloc | except 3 | 18:31 |
| kmalloc | 3 is a bad option | 18:31 |
| kmalloc | i'm inclined to say option 1 is the most direct. | 18:32 |
| *** fiddletwix has quit IRC | 18:32 | |
| lbragstad | ok - so what is #1 | 18:32 |
| lbragstad | pretty much what we have detailed in the specification? | 18:33 |
| *** jmlowe has joined #openstack-keystone | 18:34 | |
| kmalloc | yep | 18:34 |
| kmalloc | as the spec sits | 18:34 |
| kmalloc | not much changes. | 18:34 |
| *** fiddletwix has joined #openstack-keystone | 18:34 | |
| lbragstad | ok - what about #2 | 18:35 |
| lbragstad | does the not have the two-level requirement? | 18:35 |
| kmalloc | nothing specific | 18:38 |
| kmalloc | just allows for operators to know where in the tree to start looking if 29 deep you say you're out of quota | 18:39 |
| kmalloc | maybe overcommit only starts ate level 28 | 18:39 |
| kmalloc | at* | 18:39 |
| kmalloc | i am not a huge fan of that btw. | 18:39 |
| kmalloc | just figured i'd float options as I saw them | 18:39 |
| kmalloc | i think the 2-level bit is about as good as we're going to get for now. | 18:39 |
| *** bigdogstl has quit IRC | 18:40 | |
| lbragstad | and #3 restricts creating new projects that exceed two levels deep? | 18:40 |
| kmalloc | via config | 18:41 |
| kmalloc | which is just bad design | 18:41 |
| kmalloc | but... we do that kind of stuff elsewhere | 18:41 |
| kmalloc | "oh i want 20 deep, cool i set the config" | 18:41 |
| lbragstad | well - is that in a way similar to option #1? | 18:41 |
| kmalloc | sortof | 18:41 |
| kmalloc | but i'd rather have it hard-set | 18:41 |
| kmalloc | 3 means API behavior is different based on config | 18:41 |
| *** edmondsw has quit IRC | 18:41 | |
| kmalloc | which i REALLY hate | 18:41 |
| lbragstad | #1 does that in a way by opting into the model | 18:42 |
| kmalloc | i'm more inclined to give it a pass if it's not a 3 way config (am i using quota enforcement, what level deep, and keystone side) | 18:42 |
| kmalloc | s/3/2 | 18:42 |
| kmalloc | #1 is "I opt into enforcement" cool, that changes api behavior anyway | 18:42 |
| kmalloc | because enforcement is centrally managed | 18:42 |
| kmalloc | (sortof) | 18:43 |
| kmalloc | 3 is "I opt into enforcement and enforcement may behave differently depending on config" | 18:43 |
| *** EmilienM is now known as EmilienM|PTO | 18:43 | |
| *** mvk has quit IRC | 18:43 | |
| lbragstad | mm | 18:43 |
| kmalloc | i prefer the more specific "X enforcement means X behavior" | 18:43 |
| kmalloc | than "X enforcement could be X, Y, Z, Q, R, G" behavior | 18:43 |
| kmalloc | pick one, good luck knowing | 18:44 |
| lbragstad | so would we modify the project hierarchy depth to be ignored in option #1? | 18:44 |
| kmalloc | we could. | 18:44 |
| kmalloc | or we create a new enforcement model that does explicitly that | 18:44 |
| lbragstad | we do similar things with the security_compliance configuration section | 18:44 |
| kmalloc | "multi-level-overcommit-enabled" | 18:44 |
| kmalloc | vs "two-level-overcommit-enabled" | 18:44 |
| kmalloc | security compliance is a bit more special because of how PCI-DSS works | 18:45 |
| lbragstad | well - in that we explicity say " | 18:45 |
| lbragstad | "this only matters if you are using the sql driver" | 18:45 |
| kmalloc | quota has no implications with outside certification on if you can process credit cards...if for some reason your cloud is in scope | 18:45 |
| kmalloc | quota is very much internal to openstack | 18:46 |
| kmalloc | vs potentially very externally impacting | 18:46 |
| lbragstad | we could have something similar for the tree depth configuration option saying "this option is ignored if you're using the two-level strict enforcement model" | 18:46 |
| kmalloc | correct. | 18:46 |
| kmalloc | and in fact, i'd deprecate the option in keystone to change the depth | 18:47 |
| kmalloc | set it to some reasonable max | 18:47 |
| kmalloc | and let the quota enforcement model dictate the amount | 18:47 |
| kmalloc | if it changes from the "Reasonable max" | 18:47 |
| kmalloc | lbragstad: i'd increase the max_depth to 10, and deprecate the option | 18:49 |
| kmalloc | and if someone has extra deep tree(s), we let them stay just no new extra deep ones | 18:50 |
| kmalloc | then the quota model dictates the max depth, no multiple options to reconcile. | 18:50 |
| * lbragstad thinks | 18:50 | |
| kmalloc | also keep in mind what happens if someone enables this enforcement model and already has a tree 5 projects deep | 18:51 |
| kmalloc | does it error and say "NO WAY" | 18:51 |
| kmalloc | or? | 18:52 |
| lbragstad | cmurphy: brought that up in review | 18:52 |
| lbragstad | the specification review* | 18:52 |
| kmalloc | yep | 18:52 |
| lbragstad | we do something like that with the fernet token provider | 18:52 |
| lbragstad | if you're using fernet tokens and we can't find the key repository we fail | 18:53 |
| lbragstad | on start up | 18:53 |
| kmalloc | thats fine. | 18:53 |
| kmalloc | just as long as we have the expected behavior documented | 18:53 |
| kmalloc | anyway, i've covered my view | 18:54 |
| lbragstad | do we keep that same view if a deployment has a tree 4 projects deep? | 18:54 |
| kmalloc | it should be consistent | 18:54 |
| lbragstad | and they attempt to use the strict-two-level enforcement model? | 18:54 |
| kmalloc | if we are saying this enforcement only works with two-level, it only works with two level | 18:55 |
| lbragstad | fail on start up, have them switch back to flat, fix the project structure, and try again? | 18:55 |
| kmalloc | error with clear indication on all the failures | 18:55 |
| kmalloc | not just the first one | 18:55 |
| kmalloc | project heirarchy is too deep at locations X, y, z, | 18:55 |
| kmalloc | [i might make it a doctor command to check before swaping to an enforcement model] | 18:56 |
| kmalloc | so you don't have to play "are we going to fail to start games" | 18:56 |
| lbragstad | yeah | 18:56 |
| kmalloc | but basically, "run this command, it will tell you if the quota enforcement model can work" | 18:56 |
| kmalloc | if not, you need to fix the places it wont work | 18:57 |
| *** links has joined #openstack-keystone | 18:59 | |
| *** spilla has quit IRC | 19:02 | |
| lbragstad | ok | 19:04 |
| *** edmondsw has joined #openstack-keystone | 19:07 | |
| *** vrv_ has quit IRC | 19:08 | |
| lbragstad | i'm going to re-parse the spec again | 19:11 |
| *** edmondsw_ has joined #openstack-keystone | 19:12 | |
| *** edmondsw has quit IRC | 19:15 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone-specs master: Update blueprint link in default roles specification https://review.openstack.org/572528 | 19:17 |
| lbragstad | hrybacki: ^ | 19:18 |
| *** AlexeyAbashkin has joined #openstack-keystone | 19:19 | |
| *** Alexey_Abashkin has joined #openstack-keystone | 19:22 | |
| *** AlexeyAbashkin has quit IRC | 19:23 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 19:23 | |
| *** bigdogstl has joined #openstack-keystone | 19:25 | |
| *** itlinux has quit IRC | 19:26 | |
| *** bigdogstl has quit IRC | 19:30 | |
| lbragstad | kmalloc: in your first comment here - https://review.openstack.org/#/c/540803/12/specs/keystone/rocky/strict-two-level-enforcement-model.rst | 19:36 |
| lbragstad | where are we writing the usage information? | 19:36 |
| *** AlexeyAbashkin has quit IRC | 19:38 | |
| *** links has quit IRC | 19:46 | |
| *** mvenesio has joined #openstack-keystone | 19:47 | |
| kmalloc | that is data stored in the service layer [oslo.limit] | 19:48 |
| lbragstad | ack | 19:50 |
| kmalloc | if you look at the convo between me and melwitt it's a tough nut to crack | 19:50 |
| lbragstad | i just left a response to that | 19:50 |
| lbragstad | and my head hurts | 19:51 |
| *** timothyb89_ is now known as timothyb89 | 19:51 | |
| melwitt | well, fwiw I think if oslo.limit calls the per project count functions in parallel, maybe we don't really have a problem | 19:52 |
| melwitt | er, or we should be more okay without doing the usage caching thing than I was originally thinking | 19:53 |
| lbragstad | yeah - that's a good point | 19:56 |
| lbragstad | another thing | 19:56 |
| lbragstad | CERN has pretty much been asking for this for a while | 19:56 |
| lbragstad | and they've done a good job of stretching the legs on other big initiatives | 19:56 |
| lbragstad | (cells v2.0?) | 19:56 |
| lbragstad | say we try this, we will likely get good feedback from them we can use to improve the system and refine it | 19:57 |
| kmalloc | melwitt: oh hi! | 19:57 |
| kmalloc | i really think we're going to see a wide tree issue more commonly than expected, wide enough that even parallel lookup is going to be painful | 19:57 |
| kmalloc | but, that said, i put that as a comment so we weren't holding anything up besides having a discussion | 19:58 |
| *** bigdogstl has joined #openstack-keystone | 19:58 | |
| kmalloc | i dind't want to encode that behavior as part of the spec without some level of agreememnt on the approach. | 19:58 |
| kmalloc | also we can revisit if we hit performance issues, i seriously hope i'm wrong. | 19:59 |
| melwitt | yeah, I definitely think there should be a plugin or choice that does not do the usage caching as a first cut | 19:59 |
| kmalloc | for what james was proposing [i keep forgetting his IRC nick], because multi-level was a thing we def. need a report up to avoid "game the system" issues. | 20:00 |
| lbragstad | melwitt: ftr - you're talking about making requests to calculate usage in parallel, right? | 20:01 |
| kmalloc | but that is out of scope of this spec. | 20:01 |
| lbragstad | not requests to kesytone? | 20:01 |
| kmalloc | right in nova in this case | 20:01 |
| kmalloc | usage lookup not allowance lookup | 20:01 |
| melwitt | lbragstad: yes, so oslo.limit would spin up some green threads to count usage for project A, B, C in parallel | 20:01 |
| lbragstad | ack | 20:01 |
| lbragstad | just making sure i was reading things right | 20:02 |
| lbragstad | factoring that in, i'm not sure where i would guess things to tip over without seeing the system working or pushing it | 20:02 |
| openstackgerrit | Merged openstack/keystone-specs master: Update blueprint link in default roles specification https://review.openstack.org/572528 | 20:03 |
| kmalloc | the real tip over is just how many children there are, even in parallel. | 20:03 |
| *** bigdogstl has quit IRC | 20:03 | |
| kmalloc | and concurrency to nova making new vms. | 20:03 |
| kmalloc | or whatever service :) | 20:04 |
| * kmalloc goes back to flask stuffs i think we beat this spec into submission | 20:04 | |
| melwitt | there was a thread awhile back where someone did some testing in cinder with the "counting quotas" re-architecting we did in nova and the tip over was having a lot of resources to count in one project (in the absence of caching). example, 30k volumes in one project and things started slowing down to do the "enforce" call | 20:04 |
| lbragstad | interesting | 20:04 |
| kmalloc | and i'm happy to upgrade to a +2 as it sits as long as other folks weigh in. | 20:04 |
| melwitt | I do wonder if hierarchy is available, people would be more likely to create smaller projects, that would help things a lot | 20:05 |
| kmalloc | melwitt: i think people will. | 20:05 |
| lbragstad | to a certain extent though | 20:05 |
| lbragstad | since we're still limiting to two levels | 20:05 |
| lbragstad | maybe? | 20:05 |
| melwitt | this was the thread, though really hard to read in plain text because there are graphics http://lists.openstack.org/pipermail/openstack-dev/2018-March/128096.html | 20:06 |
| melwitt | the resource count is a database query sum(column) over the rows for a project and as you get a lot more records, that slows down | 20:07 |
| melwitt | well, in the cases where it has to be a sum() (like vcpus, ram). the count() are faster if that is possible (one resource per row) | 20:08 |
| lbragstad | interesting | 20:08 |
| *** r-daneel has quit IRC | 20:08 | |
| lbragstad | so - there is the resource per project tip-over point and the total projects to calculate usage for tip-over point | 20:09 |
| *** edmondsw_ is now known as edmondsw | 20:10 | |
| lbragstad | nova could slow down calculating that there are 10000 instances in a single project, or that there are 100 instead in 100 projects, yeah? | 20:10 |
| melwitt | instances would be fast because that's a count() in an index but if it were vcpus, that would be a sum() and the former (10k) would be slower than the latter (100 x 100) I think | 20:11 |
| lbragstad | oh - sure, good point | 20:12 |
| lbragstad | is this enough to classify what we have in the spec as a unsupportable design? | 20:16 |
| lbragstad | or do we keep things marked as experimental just until we get a better idea at where things fall over? | 20:17 |
| lbragstad | and then iterate from there | 20:18 |
| melwitt | I'm writing up a response but I was wondering if we can't design this in a way such that the initial model is the simplest and could be potentially chosen via config option if we add a new model that writes upwards, for example | 20:21 |
| melwitt | in the future | 20:21 |
| lbragstad | i think that is reasonable | 20:22 |
| lbragstad | and it still leaves the "flat" enforcement we have today | 20:22 |
| lbragstad | which let's system operators enforce whatever hierarchical model they want if they're will to modify the limits manually | 20:23 |
| lbragstad | which doesn't help james as much :( | 20:23 |
| lbragstad | since he's looking for the ability for domain/project administrator to do a certain level of that on their own | 20:23 |
| *** bigdogstl has joined #openstack-keystone | 20:24 | |
| melwitt | if I understood correctly, they wanted to be able to delegate limit setting via hierarchy and that each project delegated to would have to manage their own limits manually | 20:24 |
| *** pcichy has joined #openstack-keystone | 20:25 | |
| lbragstad | yeah | 20:25 |
| lbragstad | i'm not sure we'll be able to do that with flat enforcement | 20:25 |
| lbragstad | since the hierarchy isn't part of the limit validation process in keystone | 20:25 |
| melwitt | I see | 20:25 |
| melwitt | ah | 20:25 |
| lbragstad | if you have A and it has two children B and C as a single tree | 20:26 |
| lbragstad | then D is a separate tree with one child, D | 20:26 |
| lbragstad | E* | 20:26 |
| lbragstad | if you give project admin on D the ability update limits, they'd be able to modify limits on A's tree | 20:26 |
| melwitt | I see, yeah, so flat means flat on both sides, the limit setting and the enforcement | 20:27 |
| lbragstad | right - at least right now? | 20:27 |
| lbragstad | like, flat RBAC enforcement and flat limit validation | 20:27 |
| melwitt | which makes sense. but they were hoping for a hybrid | 20:27 |
| lbragstad | that might still be useful though | 20:28 |
| lbragstad | if you have hierarchical RBAC enforcement without the hierarchical limit validation, does that work? | 20:28 |
| melwitt | yeah, I think it'd be useful, just a matter of whether it's too many knobs or other UX issue. to be able to choose hierarchical limit validation + flat RBAC enforcement | 20:28 |
| lbragstad | yeah - flat being only system administrator can modify limits | 20:29 |
| lbragstad | right? | 20:29 |
| melwitt | I guess I didn't see why not but that could well be my keystone limit ignorance. I had been thinking of a toggle in oslo.limit, either it walks the hierarchy or it doesn't | 20:29 |
| kmalloc | also, keep in mind that hierarchical limit and flat enforcement needs write-upwards or the system can be gamed | 20:29 |
| kmalloc | or we have to do the entire depth search/validate anyway | 20:30 |
| kmalloc | add quota to child, spin up instanced, remove quota, add removed quota to other child, spin up instances, rinse, repeat | 20:30 |
| melwitt | I didn't think it would need to write upwards. if we're checking quota for project E, only call the count function for project E and then compare it to project E's limit and the parent limit but don't go and count all the siblings to compare with the parent limit. maybe I'm missing something | 20:31 |
| kmalloc | right but what if you take quota away from E and give it to F | 20:31 |
| kmalloc | and then spin up instances on F without turning down e | 20:31 |
| kmalloc | then do it for D and Q | 20:31 |
| lbragstad | doing this from the perspective of an administrator of D | 20:32 |
| kmalloc | you end up with as many instances on as many projects as you want. | 20:32 |
| *** bigdogstl has quit IRC | 20:32 | |
| lbragstad | so long as it's under the limit the system administrator gave your tree at D | 20:32 |
| kmalloc | as many instances = max quota for the ultimate parent | 20:32 |
| melwitt | yeah, that is what you're trusting your delegated party with. if they do that, it would be over quota until resources are freed by users | 20:32 |
| kmalloc | i make it a habit to not write CVEs by design ;) | 20:33 |
| melwitt | well, in this scenario these are trusted sub-admins but I see your point | 20:33 |
| kmalloc | but that feels like something that will be a CVE pretty quickly, because someone will not expect it... even if it's documented | 20:33 |
| melwitt | in their case, they just want to alleviate the load of putting all the limit setting work on one admin | 20:34 |
| lbragstad | if it wasn't a trusted sub admin and a malicious customer | 20:34 |
| kmalloc | i still don't trust a sub-admin in the grand scheme of things [have to play the "have security hat, will critique"] - it may even be a malicious subadmin, you want to keep quota for a given tree under X | 20:34 |
| melwitt | fair | 20:35 |
| kmalloc | the admin has users saying "OMG NEED MORE VMS" and they, with good intentions, give the quota knowing they can game the system | 20:35 |
| kmalloc | somewhat malicious compliance to the limits delegated to them | 20:35 |
| melwitt | I know penick would be okay with that if he could get the flat enforcement but I definitely see your point that this would open up a lot of problems for other use cases, so maybe it's a nonstarter | 20:35 |
| kmalloc | i view this as outside of the scope of the spec proposed though | 20:35 |
| melwitt | yeah, it's outside the scope for sure | 20:36 |
| lbragstad | penick would use the flat enforcement? | 20:36 |
| kmalloc | and i am happy to hammer that issue into the dead horse that it is when we start working on it | 20:36 |
| *** itlinux has joined #openstack-keystone | 20:36 | |
| kmalloc | penick said flat enforcement, strict commit [no overcommit], heirarchy | 20:36 |
| kmalloc | hierarchical limits* | 20:36 |
| melwitt | that's what he said. but maybe with parallel queries the hierarchical enforcement would be fine. would have to check with him | 20:37 |
| kmalloc | ++ we should check with him on that front. | 20:37 |
| lbragstad | yeah - if he's will to manage requests :( | 20:37 |
| lbragstad | willing* | 20:38 |
| melwitt | kmalloc: what do you think about the idea of the first model not caching usage designed in a way to allow swapping in a different model (via config option) in the future? is that something that would be possible? | 20:38 |
| kmalloc | melwitt: i'm inclined to just make that a different model (maybe make it a model family, that can be swapped between) | 20:38 |
| *** raildo has quit IRC | 20:38 | |
| lbragstad | so long as the project structure adheres to the requirements of the new model | 20:38 |
| lbragstad | that seems reasonable | 20:39 |
| melwitt | I agree with you that we're going to hit performance issues but as lbragstad mentioned, things get a lot more complicated that way (claim releases services are responsible for) and out-of-sync possibilities | 20:39 |
| melwitt | as far as trying to provide something that doesn't have a high barrier for entry | 20:40 |
| kmalloc | model_family(hierarchical_enforcement, hierarchical_limits) => [parallel_check_model, cached_check_model, cached_check_model_with_oob_timed_updates], model_family(flat_enforcement, flat_limits) => [flat_enforcer_model], model_family(flat_enforcement, heirarchical_limits) => [...] | 20:40 |
| kmalloc | and anything in a model_family is assumed to be compatible (with maybe a seed-the-cache command) | 20:40 |
| kmalloc | so to start we end up with flat_enforcement and heirarchical/hierarchical [ugh i can spell/type] | 20:40 |
| kmalloc | and we expand from there | 20:41 |
| kmalloc | flat is what is merged today, hierarchical/hierarchical without caching is the current spec | 20:41 |
| kmalloc | and we expand from there | 20:41 |
| *** jmlowe has quit IRC | 20:41 | |
| melwitt | sounds cool | 20:41 |
| lbragstad | if james ends up using flat or two-level-strict | 20:42 |
| kmalloc | and i can then add some etcd, endpoint-sync data storage bits for this as well | 20:42 |
| lbragstad | i'd be interested to hear his feedback, because i wouldn't be surprised if we could us it for a new model | 20:42 |
| lbragstad | use* | 20:42 |
| kmalloc | so multi-endpoint could have a single limit set that is enforced. | 20:42 |
| kmalloc | just develop for each family of limit/enforcer combinations | 20:42 |
| *** r-daneel has joined #openstack-keystone | 20:45 | |
| *** pcichy has quit IRC | 20:47 | |
| *** martinus__ has quit IRC | 20:48 | |
| *** lifeless has quit IRC | 20:51 | |
| *** lifeless has joined #openstack-keystone | 20:51 | |
| *** bigdogstl has joined #openstack-keystone | 20:53 | |
| *** bigdogstl has quit IRC | 20:58 | |
| *** dave-mccowan has quit IRC | 21:16 | |
| lbragstad | last unified limit question for the day | 21:23 |
| lbragstad | we currently support the ability to create multiple limits in a single POST request | 21:24 |
| lbragstad | do we want to expose that through the CLI some how, or would that be weird? | 21:24 |
| *** lifeless has quit IRC | 21:25 | |
| lbragstad | or do we just leave CLI support as a single limit per request for the sake of not cluttering the command line or reading multiple limits from a file | 21:26 |
| melwitt | like for multiple resources? or multiple projects? or both? I could see someone maybe wanting to do the former, but not sure tbh | 21:26 |
| lbragstad | we currently have this https://developer.openstack.org/api-ref/identity/v3/index.html#create-registered-limits | 21:26 |
| lbragstad | and https://developer.openstack.org/api-ref/identity/v3/index.html#create-limits | 21:27 |
| lbragstad | so you have create limits in batches | 21:27 |
| melwitt | okay, so the request format can be repeated, I see | 21:27 |
| *** spilla has joined #openstack-keystone | 21:27 | |
| melwitt | or it's a list rather | 21:27 |
| lbragstad | yeah - like "here are all the limits i want to create, go" | 21:28 |
| melwitt | the only data point I have is the nova quotas let you do multiple resources for one project in one go https://developer.openstack.org/api-ref/compute/#update-quotas | 21:28 |
| *** bigdogstl has joined #openstack-keystone | 21:28 | |
| melwitt | not sure how many people do that, but it's there and it makes sense to want to set limits for several resources for a project | 21:28 |
| lbragstad | are users able to create multiple quotas in a single go from the CLI? | 21:29 |
| melwitt | I believe so. let me double check | 21:29 |
| melwitt | osc does it https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/quota.html#quota-set | 21:30 |
| melwitt | (per project) | 21:30 |
| lbragstad | oh - wow | 21:31 |
| lbragstad | so it is possible string together settings per project for multiple projects in a single request | 21:32 |
| *** edmondsw has quit IRC | 21:34 | |
| melwitt | and this is the old novaclient ref https://docs.openstack.org/ocata/cli-reference/nova.html#nova-quota-update | 21:34 |
| *** edmondsw has joined #openstack-keystone | 21:34 | |
| *** bigdogstl has quit IRC | 21:34 | |
| lbragstad | so you could do openstack quota set $SETTINGS $PROJECT_1 $SETTINGS $PROJECT_2 | 21:35 |
| melwitt | I doubt it, I think it's one project at a time only | 21:35 |
| lbragstad | oh - got it | 21:35 |
| lbragstad | i was thinking you could do it for multiple projects at one | 21:36 |
| lbragstad | once* | 21:36 |
| melwitt | nah | 21:36 |
| melwitt | just multiple resources | 21:36 |
| melwitt | I don't think multiple projects makes much sense but that's just MHO | 21:36 |
| lbragstad | from a CLI perspective you mean? | 21:36 |
| lbragstad | imo - if an operator wanted to do that i could see them having a large .json file with all their requests ready to go and just curling it to keystone directly | 21:38 |
| *** edmondsw has quit IRC | 21:38 | |
| *** r-daneel_ has joined #openstack-keystone | 21:39 | |
| *** r-daneel has quit IRC | 21:39 | |
| *** r-daneel_ is now known as r-daneel | 21:39 | |
| melwitt | yeah from CLI perspective | 21:41 |
| lbragstad | makes sense | 21:42 |
| melwitt | mayve I'm wrong too :) | 21:43 |
| melwitt | *maybe | 21:43 |
| lbragstad | i'm not aware of any CLI commands that let you do that | 21:43 |
| lbragstad | or support batch creation | 21:43 |
| * melwitt nods | 21:44 | |
| *** bigdogstl has joined #openstack-keystone | 21:45 | |
| lbragstad | thanks for the help today melwitt | 21:45 |
| lbragstad | #endmeeting | 21:46 |
| openstack | Meeting ended Tue Jun 5 21:46:32 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:46 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-06-05-17.04.html | 21:46 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-06-05-17.04.txt | 21:46 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/keystone_office_hours/2018/keystone_office_hours.2018-06-05-17.04.log.html | 21:46 |
| *** lifeless has joined #openstack-keystone | 21:47 | |
| *** itlinux has quit IRC | 21:53 | |
| *** itlinux has joined #openstack-keystone | 22:03 | |
| *** itlinux has quit IRC | 22:13 | |
| *** r-daneel has quit IRC | 22:21 | |
| *** mvenesio has quit IRC | 22:23 | |
| *** ayoung has quit IRC | 22:25 | |
| *** rcernin has joined #openstack-keystone | 22:25 | |
| *** ayoung has joined #openstack-keystone | 22:36 | |
| *** boris_42_ has joined #openstack-keystone | 22:47 | |
| *** bigdogstl has quit IRC | 22:53 | |
| *** bigdogstl has joined #openstack-keystone | 23:00 | |
| *** edmondsw has joined #openstack-keystone | 23:03 | |
| *** bigdogstl has quit IRC | 23:05 | |
| *** edmondsw has quit IRC | 23:07 | |
| *** bigdogstl has joined #openstack-keystone | 23:07 | |
| *** threestrands has joined #openstack-keystone | 23:08 | |
| *** harlowja has joined #openstack-keystone | 23:10 | |
| *** bigdogstl has quit IRC | 23:12 | |
| *** bigdogstl has joined #openstack-keystone | 23:22 | |
| *** bigdogstl has quit IRC | 23:31 | |
| *** annp has quit IRC | 23:38 | |
| *** annp has joined #openstack-keystone | 23:38 | |
| *** bigdogstl has joined #openstack-keystone | 23:44 | |
| *** bigdogstl has quit IRC | 23:49 | |
| *** lifeless has quit IRC | 23:54 | |
| *** lifeless has joined #openstack-keystone | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!