Wednesday, 2014-04-23

*** daneyon has joined #openstack-keystone00:00
*** wchrisj has joined #openstack-keystone00:05
*** browne1 has quit IRC00:07
*** wchrisj has quit IRC00:09
*** browne has joined #openstack-keystone00:09
*** dstanek_zzz is now known as dstanek00:19
*** nkinder has joined #openstack-keystone00:30
ayounggyee, done.00:31
*** daneyon_ has joined #openstack-keystone00:34
*** daneyon has quit IRC00:38
*** browne has quit IRC00:41
*** richm has quit IRC00:48
*** wchrisj has joined #openstack-keystone01:06
*** wchrisj has quit IRC01:10
*** diegows has quit IRC01:28
*** marcoemorais has quit IRC01:29
*** lbragstad has joined #openstack-keystone01:32
*** wchrisj has joined #openstack-keystone01:43
*** gokrokve_ has quit IRC01:45
*** gokrokve has joined #openstack-keystone01:47
*** praneshp has quit IRC01:47
*** wchrisj has quit IRC01:48
*** gokrokve has quit IRC01:51
*** wchrisj has joined #openstack-keystone01:54
*** wchrisj has quit IRC01:59
*** daneyon has joined #openstack-keystone02:04
*** daneyon has quit IRC02:05
*** daneyon_ has quit IRC02:06
*** gokrokve has joined #openstack-keystone02:16
*** gokrokve has quit IRC02:21
*** mberlin1 has joined #openstack-keystone02:27
*** mberlin has quit IRC02:28
ayoungdolphm, morganfainberg, care to kick this one over the threshold   ?    bknudson wrote it, so it is not going to get any better reviewed than it is now02:35
ayoungdstanek, or you ^^02:35
lbragstadayoung: ++ that will be nice to have02:36
ayounglbragstad, saw you +1ed it already02:36
lbragstadayoung: I think I had one concern but it could be properly addressed in a separate patch02:37
openstackgerritA change was merged to openstack/keystone: Code which gets elements of tree in ldap moved to a common method
morganfainbergayoung, also +2/+A the universal newline stuff02:38
morganfainbergayoung, so that should be gating now02:39
ayoungsaw that.  thanks02:39
ayoungjamielennox, what do you think about a temporary auth plugin for keystoneclient:  negotiate-external.  It will set up the negotiate call on the client side, and will set method = "external"02:40
jamielennoxayoung: can do, we still need requests-kerberos to get updated though right02:41
jamielennoxwhat are you trying to do?02:41
jamielennoxwe don't have a CLI or auth_token loading from a plugin02:41
*** stevemar has joined #openstack-keystone02:43
*** harlowja is now known as harlowja_away02:43
gyeeayoung, thanks02:46
openstackgerritA change was merged to openstack/python-keystoneclient: remove universal_newlines
*** gyee has quit IRC02:47
ayoungjamielennox, I was just thinking through.  I probably will be writing a hack to Horizon to do S4U2Proxy02:47
jamielennoxayoung: you may as well use it for testing02:52
jamielennoxi still think we should just do a 'kerberos' thing for longer term02:52
jamielennoxgiven that we can't load it anyway02:52
ayoungjamielennox, I'll make the change, based on Jose's, and submit it.  We can keep it as a WIP02:52
jamielennoxalso if you look at the from_conf stuff you should be able to see how to load it externally02:53
*** praneshp has joined #openstack-keystone02:55
*** wchrisj has joined #openstack-keystone02:55
*** wchrisj has quit IRC02:57
*** wchrisj_ has joined #openstack-keystone02:57
*** praneshp_ has joined #openstack-keystone03:00
*** wchrisj_ has quit IRC03:01
*** praneshp has quit IRC03:01
*** praneshp_ is now known as praneshp03:01
*** nkinder has quit IRC03:02
*** openstackgerrit has quit IRC03:04
*** openstackgerrit has joined #openstack-keystone03:04
*** nkinder has joined #openstack-keystone03:08
openstackgerritayoung proposed a change to openstack/python-keystoneclient: Compressed Signature and Validation
*** gokrokve has joined #openstack-keystone03:19
*** david-lyle has joined #openstack-keystone03:19
*** gokrokve has quit IRC03:24
openstackgerritayoung proposed a change to openstack/python-keystoneclient: Regions Management
*** zhiyan_ is now known as zhiyan03:48
*** rwsu has quit IRC03:50
*** marcoemorais has joined #openstack-keystone03:52
*** david-lyle has quit IRC03:54
*** marcoemorais1 has joined #openstack-keystone03:56
*** marcoemorais has quit IRC03:57
*** topol has joined #openstack-keystone04:04
*** ilives has joined #openstack-keystone04:10
*** gokrokve has joined #openstack-keystone04:20
*** gokrokve has quit IRC04:24
*** ayoung is now known as ayoung_ZZZzz04:25
openstackgerritDavid Stanek proposed a change to openstack/keystone: Identity authentication now uses rotated passwords
openstackgerritDavid Stanek proposed a change to openstack/keystone: Adds table and model for storing rotated passwords
openstackgerritDavid Stanek proposed a change to openstack/keystone: Adds an extension to enable password rotation
openstackgerritDavid Stanek proposed a change to openstack/keystone: Ignore broken endpoints in get_catalog
openstackgerritDavid Stanek proposed a change to openstack/keystone: Ignore broken endpoints in get_v3_catalog
*** andreaf has joined #openstack-keystone04:48
*** andreaf_ has joined #openstack-keystone04:49
*** andreaf has quit IRC04:53
openstackgerritDavid Stanek proposed a change to openstack/keystone: Identity authentication now uses rotated passwords
openstackgerritDavid Stanek proposed a change to openstack/keystone: Adds an extension to enable password rotation
*** bvandenh has joined #openstack-keystone05:11
*** shakamunyi has quit IRC05:14
*** gokrokve has joined #openstack-keystone05:20
*** gokrokve_ has joined #openstack-keystone05:22
*** gokrokve has quit IRC05:25
*** gokrokve_ has quit IRC05:27
*** topol_ has joined #openstack-keystone05:31
*** topol has quit IRC05:34
*** lbragstad has quit IRC05:36
*** topol_ has quit IRC05:37
openstackgerritA change was merged to openstack/keystone: Configurable token hash algorithm
*** chandan_kumar has joined #openstack-keystone05:54
openstackgerritSergey Nikitin proposed a change to openstack/keystone: Code which gets and deletes elements of tree was moved to one method
*** bvandenh has quit IRC06:00
*** bvandenh has joined #openstack-keystone06:01
openstackgerritOpenStack Proposal Bot proposed a change to openstack/keystone: Imported Translations from Transifex
*** ilives has quit IRC06:07
*** ilives has joined #openstack-keystone06:07
*** bvandenh has quit IRC06:15
*** bvandenh has joined #openstack-keystone06:15
*** gokrokve has joined #openstack-keystone06:23
*** gokrokve has quit IRC06:27
openstackgerritA change was merged to openstack/keystone: Include extra attributes in list results
*** gokrokve has joined #openstack-keystone06:42
*** gokrokve has quit IRC06:42
*** gokrokve has joined #openstack-keystone06:42
*** gokrokve has quit IRC06:48
*** tomoiaga has joined #openstack-keystone06:54
openstackgerritSteve Martinelli proposed a change to openstack/python-keystoneclient: Add request/access token and consumer support for keystoneclient
openstackgerritSteve Martinelli proposed a change to openstack/python-keystoneclient: Add request/access token and consumer support for keystoneclient
*** dstanek is now known as dstanek_zzz07:16
*** stevemar has quit IRC07:20
*** ilives has quit IRC07:29
*** ilives has joined #openstack-keystone07:30
*** morganfainberg is now known as morganfainberg_Z07:38
*** ilives has quit IRC07:47
*** ilives has joined #openstack-keystone07:53
*** praneshp has quit IRC08:01
*** leseb has joined #openstack-keystone08:09
*** ilives has quit IRC08:20
*** ilives has joined #openstack-keystone08:22
tomoiagaI see that v3/users/ {user_id} /roles is listed as a supported API call, however it seems to return a 404 when called. Looking at the code I can't find an implementation for it, maybe someone knows something about this.08:23
*** marcoemorais1 has quit IRC08:35
*** ilives has quit IRC08:42
*** ilives has joined #openstack-keystone08:43
openstackgerritMatthieu Huin proposed a change to openstack/python-keystoneclient: Limited use trusts
*** ilives has quit IRC08:50
*** ilives has joined #openstack-keystone08:52
*** bada has quit IRC09:00
*** marcoemorais has joined #openstack-keystone09:03
*** dstanek_zzz is now known as dstanek09:05
*** marcoemorais1 has joined #openstack-keystone09:05
*** marcoemorais has quit IRC09:08
*** marcoemorais1 has quit IRC09:09
*** Chicago has joined #openstack-keystone09:20
*** Chicago has joined #openstack-keystone09:20
*** dstanek is now known as dstanek_zzz09:23
*** ilives has quit IRC09:34
*** zhiyan is now known as zhiyan_09:42
*** marcoemorais has joined #openstack-keystone10:06
*** marcoemorais has quit IRC10:10
*** fmarco76 has joined #openstack-keystone10:10
*** fmarco76 has left #openstack-keystone10:11
*** dstanek_zzz is now known as dstanek10:15
*** dstanek is now known as dstanek_zzz10:24
*** leseb has quit IRC10:33
*** leseb has joined #openstack-keystone10:34
*** leseb has quit IRC10:38
*** jaosorior has joined #openstack-keystone10:56
*** jamielennox is now known as jamielennox|away11:02
*** marcoemorais has joined #openstack-keystone11:07
*** leseb has joined #openstack-keystone11:09
*** marcoemorais has quit IRC11:11
*** leseb has quit IRC11:14
*** dstanek_zzz is now known as dstanek11:15
*** kashyap has joined #openstack-keystone11:15
jaosoriorHey, I've noticed that the keystone.common.models is barely used anywhere except for the ldap backend11:21
jaosoriorI was thinking of refactoring some modules in order to use more the information in the models11:21
jaosoriorfor example, keystone.common.identity.core contains a function called filter_user, where some fields in the User model are hardcoded, instead, I would like to get these attributes from the model itself. Is there any desire for something like this?11:23
*** dstanek is now known as dstanek_zzz11:25
kashyapHi, while setting up Keystone endpoint, is v3.0 recommended or should I go w/ v2.0? This is w/ IceHouse11:31
tomoiagakashyap: v3 is not yet supported by services like nova and the rest. You can setup both if you want to play around with v3 or give access to v3 to others but you will also need to have v2 for the other services11:32
kashyaptomoiaga, Okay, I'm just setting up a minimal IceHouse setup to do some bug triage, so I'll go w/ V211:33
kashyapThank you for your response11:33
tomoiagakashyap: np. I personally use both, but v2 needs to remains at least for services until Juno at least11:34
kashyaptomoiaga, Hmm, my minimal setup involves - Nova, Keystone, Neutron, Glance. Cinder is not really necessary in a minimal env, right?11:34
tomoiagakashyap: no, unless you want to support volumes (centralized storage). You will be able to have local storage (ephemeral) without cinder so it should be ok. I have a setup working fine without cinder since I don't need volumes.11:36
kashyaptomoiaga, Yeah, I mostly use qcow2 files (preallocated), and nested virt (KVM-based). My entire setup is virtual. Thanks for this detail, again.11:37
tomoiagakashyap: does it work ok for you in nested mode ? I never got the chance to test this but I would soon need to since I don't have unlimited hardware :)11:38
* kashyap on the phone, be back in one minute11:38
tomoiagakashyap: I think I found your blog and I am going to go through your findings now :)11:39
kashyaptomoiaga, Yes, it does work for me :-) . I've been using nested virt consistently on Intel (some new hardware) ~ 2 years11:41
kashyaptomoiaga, I also have some newer notes here on a staging blog --
*** diegows has joined #openstack-keystone11:42
tomoiagakashyap: thank you! I'll read your blog and the newer notes since I need to catch up on some things.11:42
kashyaptomoiaga, Sure, no rush. Thanks to you too.11:43
openstackgerritLi Ma proposed a change to openstack/keystone: Password trunction makes password insecure
*** lbragstad has joined #openstack-keystone11:55
*** leseb has joined #openstack-keystone11:58
kashyaptomoiaga, If you still have a moment, creating a Keystone service throws me --
kashyapThat's the command I invoked: $ keystone service-create --name keystone --type identity     --description "Keystone Identity Service"12:01
tomoiagakashyap: did you synced the db ?12:01
tomoiagakashyap: I believe this is a new keystone installation, done manually right ?12:02
kashyaptomoiaga, That's what I ran -- keystone-manage db_sync keystone12:02
kashyaptomoiaga, Yes, done manually, I'm following my own notes from Havana -
tomoiagakashyap: can you run explain on the service column in your keystone db ?12:05
kashyaptomoiaga, Sorry, "run explain"?12:05
*** dstanek_zzz is now known as dstanek12:05
tomoiagakashyap: login to mysql, use keystone (or the name of your db) and execute: explain service;12:05
* kashyap tries12:06
tomoiagakashyap: explain will show the columns in the service table12:06
kashyaptomoiaga, I'm logged into mysql, do you have an exact command that I can try? (/me is db crippled)12:07
tomoiagakashyap: you won't need openstack-neutron-openvswitch either on the controller node, unless you plan on adding instances there too.12:07
*** marcoemorais has joined #openstack-keystone12:07
kashyaptomoiaga, Good to know, will remove it from my IceHouse notes :-)12:08
tomoiagakashyap: execute: use keystone (if keystone is the name of your db). After that execute: explain service12:08
tomoiagakashyap: if you want to see all the databases, execute: show databases12:09
kashyaptomoiaga, Still trying, I'm on a remote machine, and I'm using a slow USB internet. . .12:09
kashyapMariaDB [keystone]> explain service;12:10
kashyap| Field | Type         | Null | Key | Default | Extra |12:10
kashyap| id    | varchar(64)  | NO   | PRI | NULL    |       |12:10
kashyap| type  | varchar(255) | YES  |     | NULL    |       |12:10
kashyap| extra | mediumtext   | YES  |     | NULL    |       |12:10
kashyap3 rows in set (0.01 sec)12:10
*** marcoemorais has quit IRC12:11
tomoiagakashyap: the enabled column seems to be missing. Try to run from ssh again and see if you notice any errors/warnings: keystone-manage db_sync12:12
* kashyap re-tries: keystone-manage db_sync12:13
kashyaptomoiaga, Yeah, it's now there:12:13
kashyap| enabled | tinyint(1)   | NO   |     | 1       |       |12:14
tomoiagakashyap: good, db_sync needs to be executed when you first install keystone or after an upgrade12:14
*** dstanek is now known as dstanek_zzz12:15
kashyaptomoiaga, That seems weird -- why "sync" a db when you're installing for the first time12:16
* kashyap now re-tries creating Keystone service12:17
tomoiagakashyap: well, the rpm package may not sync the db for you or it may, it depends on who configured the rpm, but in any case, if you have a remove mysql server, someone creating the rpm package can't know where that mysql server is to configure keystone and run the sync for you12:17
kashyaptomoiaga, I see, fair enough. Thanks for your time to help debug this.12:19
tomoiagakashyap: glad it's working12:20
openstackgerritMatthieu Huin proposed a change to openstack/keystone: More random values for oAuth1 verifier
*** dstanek_zzz is now known as dstanek12:26
*** andreaf_ has quit IRC12:48
*** andreaf has joined #openstack-keystone12:48
*** marcoemorais has joined #openstack-keystone13:08
*** marcoemorais1 has joined #openstack-keystone13:10
*** dstanek is now known as dstanek_zzz13:11
*** marcoemorais has quit IRC13:12
*** dstanek_zzz is now known as dstanek13:14
*** marcoemorais1 has quit IRC13:14
*** daneyon has joined #openstack-keystone13:19
*** daneyon has quit IRC13:22
*** daneyon has joined #openstack-keystone13:23
*** jzl_ has joined #openstack-keystone13:24
*** jzl_ has quit IRC13:24
*** afaranha has quit IRC13:34
*** thiagop has quit IRC13:34
*** afaranha has joined #openstack-keystone13:40
*** joesavak has joined #openstack-keystone13:42
*** thiagop has joined #openstack-keystone13:43
*** chandan_kumar is now known as chandankumar13:54
*** wchrisj has joined #openstack-keystone13:56
*** stevemar has joined #openstack-keystone13:57
*** gokrokve has joined #openstack-keystone14:02
openstackgerritSteve Martinelli proposed a change to openstack/python-keystoneclient: Add request/access token and consumer support for keystoneclient
*** topol has joined #openstack-keystone14:04
*** marcoemorais has joined #openstack-keystone14:11
openstackgerritayoung proposed a change to openstack/keystone: Remember the DN
*** rwsu has joined #openstack-keystone14:14
*** marcoemorais has quit IRC14:15
*** thedodd has joined #openstack-keystone14:22
*** bada has joined #openstack-keystone14:48
openstackgerritSteve Martinelli proposed a change to openstack/keystone: Add detailed federation configuration docs
*** david-lyle has joined #openstack-keystone14:55
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Sync test_migrations
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Redundant unique constraint
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Corresponding `nullable` value.
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Migration DB_INIT_VERSION in common place
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Compatible server default value in the models.
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Explicit foreign key indexes.
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Sync on-demand database schemas
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Comparision of database models and migrations.
*** dstanek is now known as dstanek_zzz15:17
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Sync test_migrations
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Redundant unique constraint
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Corresponding `nullable` value.
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Migration DB_INIT_VERSION in common place
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Compatible server default value in the models.
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Explicit foreign key indexes.
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Sync on-demand database schemas
openstackgerritIlya Pekelny proposed a change to openstack/keystone: Comparision of database models and migrations.
*** gokrokve_ has joined #openstack-keystone15:31
*** gokrokve has quit IRC15:34
*** browne has joined #openstack-keystone15:34
openstackgerritSteve Martinelli proposed a change to openstack/keystone: Rename extension docs from configure to enable
*** gyee has joined #openstack-keystone15:40
stevemarmarekd|away, thanks for the comments :)15:43
*** jaosorior has quit IRC16:01
*** marekd|away is now known as marekd16:02
marekdstevemar: i wrote them in the plane, just got connected and uploaded without actually looking that you uploaded another patchset.16:03
stevemarmarekd, i'll forgive you, being on a plane and all16:03
*** marcoemorais has joined #openstack-keystone16:04
marekdstevemar: i am not sure if we can say that there is a possibility to map federated users to existing users...16:11
marekdstevemar: i am assuming this would be happening just because the user ids match, but do you think it's enough?16:11
*** jsavak has joined #openstack-keystone16:12
marekdstevemar: AFAIR the whole federation thing was made and we avoided user lookups in the backend...16:13
*** joesavak has quit IRC16:13
marekduser_id produced by the maping engine is just an information, something that may help in accounting or something like that.16:14
*** gokrokve has joined #openstack-keystone16:14
*** joesavak has joined #openstack-keystone16:15
*** richm has joined #openstack-keystone16:15
*** gokrokv__ has joined #openstack-keystone16:16
stevemarmarekd, i agree, I thought I wrote that back in my comment. It's just for auditing16:17
*** Chicago has quit IRC16:18
*** jsavak has quit IRC16:18
*** gokrokve_ has quit IRC16:18
marekdstevemar: ok, that 'sort of' was confusing :P16:18
stevemarmarekd, ah, that was a nice way for me to say 'no'16:20
*** gokrokve has quit IRC16:20
marekdok ok :-)16:21
*** tomoiaga has quit IRC16:22
*** Anju_ has joined #openstack-keystone16:22
*** marekd is now known as marekd|afk16:27
*** thiagop has quit IRC16:28
*** erecio has joined #openstack-keystone16:33
*** kashyap has left #openstack-keystone16:33
*** leseb has quit IRC16:40
*** amcrn has joined #openstack-keystone16:47
openstackgerritDoug Hellmann proposed a change to openstack/keystone: Register all backend classes as entry points
*** chandankumar has quit IRC16:53
openstackgerritOpenStack Proposal Bot proposed a change to openstack/keystone: Updated from global requirements
*** ayoung_ZZZzz is now known as ayoung16:55
ayoungdhellmann, when I hear Entrypoints I think exported symbols, and I never quite got why.  Why?16:59
openstackgerritOpenStack Proposal Bot proposed a change to openstack/python-keystoneclient: Updated from global requirements
*** harlowja_away is now known as harlowja17:06
*** spligak has joined #openstack-keystone17:09
*** gokrokv__ has quit IRC17:15
openstackgerritSteve Martinelli proposed a change to openstack/python-keystoneclient: Add request/access token and consumer support for keystoneclient
*** gokrokve has joined #openstack-keystone17:17
*** dstanek_zzz is now known as dstanek17:20
*** gokrokve has quit IRC17:23
lbragstadayoung: had a couple very minor/trivial comments on
lbragstadI can push a new patch if you want17:24
ayounglbragstad, must be something with my editor.  I copied and pasted the "good" one over the overly spaced one.17:25
*** praneshp has joined #openstack-keystone17:25
ayoungNah, I'll repush...17:25
lbragstadayoung: cool, what are you using for editor? just curious17:26
dstanekayoung: how do you like PyCharm?17:26
lbragstadah gotcha17:26
ayoungdstanek, I like it for the most part.  Good browsing.  Its a bit of a pain wokring with multiple projects, as each needs to be open in its own window, but that is almost more a feature.  Debugging support is good17:29
ayoungautomated refactorings are better than in Eclipse pydev.17:30
openstackgerritguang-yee proposed a change to openstack/keystone: Make sure all the auth plugins agree on the shared identity attributes.
ayoungnkinder, I'm getting lost looking at the rebase of "Remember the DN"  llist_group_users
ayounglbragstad, the copyright headers are formatted the same as other files in the same subdir17:36
ayoungI guess it is a question of which spacing is right...17:37
lbragstadayoung:  yeah I'm guess I'm not sure which spacing is 'right'17:37
ayoung  lbragstad17:37
*** chandan_kumar has joined #openstack-keystone17:39
lbragstadso two spaces before text17:40
nkinderayoung: checking...17:42
nkinderayoung: ok, where are you getting lost?17:43
ayoungnkinder, so the code that messed up the rebase is new17:43
ayoungand I don;t think it was there before you did your work.  So which approach do we follow17:43
ayoung group_dn = self._id_to_dn(group_id)  is what we want to repalce, right?17:44
ayoungI dropped the "query" param on the rebase, which I think was just a mistake in merging17:44
nkinderayoung: which code is new?  The use of _ldap_get_list()?17:45
ayoungnkinder, I think so, yeah17:45
nkinderayoung: so here's how it was when I made my patch -
nkinderayoung: I was avoiding _id_to_dn()17:46
ayoungnkinder, so keep the existing code for the most part, but17:46
ayounguse  group_ref = self.get(group_id)17:46
ayoung388        group_dn = group_ref['dn']17:47
*** morganfainberg_Z is now known as morganfainberg17:48
nkinderayoung: keep the use of _ldap_get_list(), but extract the group_dn as I did in my patch17:49 patch in a moment17:49
morganfainbergbknudson, ping looks like there is possibly an error win a setup script w/ the db2 ci.17:51
morganfainbergbknudson, looking at some of the db changes and want to make sure db2 ci is running sanely (or know if we should ignore it in this case)17:51
bknudson/opt/stack/new/devstack-gate/ line 123: [: : integer expression expected17:52
morganfainbergbknudson, yep17:52
morganfainbergbknudson, not sure if that is something we (read: openstack-infra) did or something the team over there needs to look at17:52
bknudsonmorganfainberg: our team should look at it ... unless all the jobs are failing17:53
morganfainbergbknudson, let me check.17:53
bknudsonSaving to: `/tmp/cacert.pem' -- that looks safe!17:53
morganfainbergbknudson, yeah wow, super safe!17:55
bknudsonit's the public key, so not the worst thing17:55
bknudsonjust don't write a test that dumps that file!17:55
ayoungnkinder, yep, thjat is what I have, just testing17:55
morganfainbergbknudson, yeah looks like db2 ci is the one failing, upstream gate doesn't look to be having errors17:57
openstackgerritayoung proposed a change to openstack/keystone: Remember the DN
morganfainbergbknudson, i'll look at the code and give a +1 with a +2 note if db2 runs clean (either manually ot the ci) before merge.18:04
bknudsonmorganfainberg: which was the review?18:05
bknudsonI'll make sure they do a recheck18:05
morganfainbergbknudson, there are a bunch by ilya pekelny that are around the DB.18:05
morganfainbergbknudson, i'll give you a list as i review them if it helps.18:06
bknudsonI know which they are.18:06
ayoungnkinder, ^^ look right?18:06
bknudsonI don't live under a rock.18:06
morganfainbergbknudson, hehe but rocks are cool18:08
*** gokrokve has joined #openstack-keystone18:16
*** gokrokve has quit IRC18:21
morganfainbergbknudson, somedays i think living under a rock might be better >.>18:21
dstanekmorganfainberg: technology is just a dead end18:24
morganfainbergdstanek, i KNEW IT!18:24
nkinderayoung: what's the point of calling self.get_group(group_id) in add_user_to_group()?18:25
* morganfainberg invests heavily in rocks.18:25
nkinderayoung: that call was already there, but is it really needed?18:25
ayoungnkinder, no idea. You wrote that code, and I was just trying to rebase.18:26
nkinderno, I didn't add it IIRC18:26
ayoungI don't think I did either, but let me look18:26
dstaneknkinder: a lot of times we do a get_* call and don't do anything with the value to make sure it actually exists18:27
ayoungnkinder, its probably checking for existance.18:27
morganfainbergnkinder, that is to raise an exception to check existence18:27
nkinderwon't we find out when the MOD operation fails?18:27
morganfainbergnkinder, erm, s/raise exception//18:27
nkinderthat will get an LDAP err=32 (LDAP_NO_SUCH_OBJECT), which should trigger an exception18:27
ayoungnkinder, might be the wrong exception18:28
nkinderall this does is cause an extra search18:28
bknudsonwrite the tests and make sure they pass.18:28
ayoungyeah, we can probably streamline it18:28
morganfainbergayoung, ++ probably GROUP_NOT_FOUND vs LDAP_ERROR18:28
bknudsonhave a test that the op fails if the entry doesn't exist18:28
nkinderless operations == profit18:28
ayoungbknudson, we need LDAP in the gate18:28
ayounglive LDAP18:28
nkinderayoung: I'll test it out18:28
bknudsonI think this one is handled by fakeldap. it's a pretty easy one18:29
bknudsonit shouldn't be hard to write a fakeldap that fails if the entry doesn't exist, anyways18:29
morganfainbergnkinder, this also might just be an artifact of SQLisms where we do need to check for existence18:29
bknudsonalthough it's pretty much impossible to compare DNs correctly18:29
morganfainbergnkinder, because the way RDBMs works/stores data vs. LDAP (you could create the row in sQL...)18:30
nkinderI want to see what this patch does with wireshark anyway, so I'll test without those calls with a live LDAP server and see what happens18:31
morganfainbergbknudson, is the DN comparison pain just python-isms? or a universal problem?18:31
bknudsonmorganfainberg: it's LDAP and not python-isms.18:32
bknudson1) attribute names can take several forms (short names, long names, OIDs)18:32
morganfainbergbknudson, i've only worked with LDAP in C/C++ and Python.18:32
morganfainbergbknudson, i know it was a pain in both, but it might have been the libs we used in C/C++18:33
bknudson2) attribute values have different comparison rules (case-sensitive, case-insensitive, whatever)18:33
morganfainbergbknudson, ah yeah18:33
bknudsonso you need the schema and also lots of code to be able to compare DNs18:33
bknudsonand I don't think LDAP has a standard even for how to fetch the schema or what the schema looks like18:33
morganfainbergbknudson, yeah sounds about right18:33
bknudsonsomething you'd think would be simple and fundamental to working with LDAP is pretty much impossible18:34
morganfainbergbknudson, sure it does... if you're an LDAP server and already know how to parse a schema... :P18:34
morganfainbergand using an opensource LDAP server with it's own protocol for that.18:34
bknudsonmaybe ldap has an extended op for comparing dns18:35
bknudsonwould expect that to be slow since it requires a roundtrip to the server, though18:35
nkinderayoung: I found one error in the patch.  I'll run some tests with it now and will submit a new rev. afterwards18:35
morganfainbergbknudson, hm.18:35
nkinderbknudson: there is a compare operation18:36
nkinderit's not commonly used though18:36
bknudsonattribute compare, e.g., for the password18:36
morganfainbergbknudson, i always found the python ldap impl (python-ldap lib) more kludgy than some other options18:36
morganfainbergnkinder, it sounds expensive... lets ask the LDAP server to compare these things.18:36
morganfainbergnkinder, oh another OP every time.18:36
nkinderyeah, definitely18:36
nkinderDNs can be easily compared by doing a base level search18:37
nkinderstill a round-trip18:37
morganfainbergnkinder, hm. still yeah18:37
bknudsonand the entry has to exist.18:37
nkinderIf you pass a legal DN, an LDAP server is supposed to normalize it and do a syntax-aware comparison when searching18:37
nkinderbknudson: yes, absolutely18:38
bknudsonlet's say I want to find out if this entry would be in this subtree, for example.18:38
bknudson(although that might not even make a lot of sense if you've got aliases and references)18:38
nkinderthe client would be responsible to be smart about the DN syntax in that case18:39
nkinderwhich is easier said than done18:39
bknudsonluckily DNs tend to only use a few attrs, so if you can limit yourself to that then you can make something that works client-side18:39
*** chandan_kumar has quit IRC18:41
*** gokrokve has joined #openstack-keystone18:45
ayoung  I likes my new Icon!18:47
morganfainbergayoung, nice18:47
ayoungmorganfainberg, I love howlaunchpad calls it "rebranding"18:48
morganfainbergwait... they call it rebranding?18:48
* morganfainberg snickers18:48
nkinderdo I remember correctly that a change was made a little while back that eliminates the need to mount keystone/tests/tmp as tmpfs to speed up the tests?18:52
morganfainbergnkinder, correct.18:52
nkinderyay, I'm not 100% crazy18:52
morganfainbergnkinder, we do all the SQL-lite migrations/db work in-memory now18:52
morganfainbergthe #1 reason to mnt tmpfs was for the db file18:53
morganfainbergthere are some files that are still dropped on disk, but they are minor and the tiny improvements (unless you're really io-bound) shouldn't be noticable18:53
nkinderdoes this test failure look familiar to anyone?
nkinderI have a freshly installed OS, so it may be environment related.  That said, I was able to run all of the unit tests in stable/icehouse earler this morning on this system.18:58
morganfainbergnkinder, looking now19:00
dstaneknkinder: that's pretty odd19:00
morganfainbergnkinder, yeah i know what that issue is19:00
dstanekmorganfainberg: ?19:01
morganfainbergdstanek, he didn't use 'usedevelop' to install keystone19:01
morganfainbergit's the way we load the files on the backend19:01
morganfainbergit's a bug19:01
morganfainbergthat patch fixes it19:01
morganfainbergor should19:01
morganfainbergbasically it is looking in the wrong place for the code to do reflection based schema building19:02
morganfainbergthe code assumes you're using the pip -e . style setup (or similar) for running tests19:02
morganfainbergthere are other ways it can conflict, the answer is to use stevedore and entry points instead19:02
morganfainbergmakes things a lot less likely to break in odd ways19:02
dstanekmorganfainberg: really? it should be looking relative to the tests19:03
morganfainbergdstanek, dhellmann ran across the same issue19:03
nkinderfor setup, I created a venv, then just did 'pip -r' for requirements.txt and test-requirements.txt19:03
nkinderafter that, I'm just using run_tests.sh19:04
morganfainbergnkinder, if you did pip install -e .19:04
morganfainbergthat would then fix it.19:04
morganfainbergi think that is the issue.19:04
nkindermorganfainberg: ++19:04
nkindermorganfainberg: that step won't be necessary once the above patch is merged?19:05
dstaneki'll have to try to recreate - i use tox and have not actually installed keystone into a venv19:05
morganfainbergnkinder, hopefully19:05
morganfainbergnkinder, though to be fair, that is how gate does it, (run_tests should do the same)19:06
morganfainbergnkinder, iirc tox also works like that.19:06
dstanekmorganfainberg: how does that patch know to import from the backends package?19:07
dstanekthe thing i don't like about that patch is i think it'll need a test to make sure as new sql modules are added then get added to the list19:07
morganfainbergdstanek, it's based on the namespace list and the use of entrypoints19:08
morganfainbergdstanek, the entrypoint says "keystone.assignment" and setup.cfg has the mapping to keystone.assignment.backends for each module19:08
dstanekmorganfainberg: ah i didn't look in the setup.cfg19:08
*** bvandenh has quit IRC19:09
morganfainbergdstanek, we might need a test to make sure things end up in that lilst, but there was a developing.rst change as well, and the schema wont exist (you'll get other failures for SQL tests)19:09
morganfainbergdstanek, as in SQLA - no such table19:09
dstanekmorganfainberg: that's the problem - you won't :-(19:09
dstaneka test will likely import the sql before it's used19:09
morganfainbergdstanek, the test absolutely would fail if you don't import the module19:09
dstanekbut that mean the schema would be different between the tests and depend on the ordre19:10
dstanekthe point of what i did was to make sure the schema is always the same19:10
bknudsondstanek: could you write a check to ensure that no new models were imported by the test?19:12
dstanekbknudson: not really because by the time the test runs they have already been imported19:12
bknudsoncheck before and after19:12
morganfainbergthere has to be abetter way than walking the directory.19:12
morganfainbergor stevedore19:12
morganfainbergi think i know why nova sticks all it's models into a single module now.19:13
dstanektechnically you could check and the begining of each test and the first test could have off the list19:13
dstanekbknudson: the sql modules will be import before the test runs, not during the test run19:14
morganfainbergmaybe the right answer is to have a single location all models go.19:14
dstanekbknudson: well, not all of the time; but generally speaking19:14
morganfainbergmove them to like keystone/common/sql/models/<module_name>.py ?19:14
bknudsonmorganfainberg: what about extensions?19:14
dstanekmorganfainberg: what about extensions?19:14
bknudsonquit reading my mind19:15
morganfainbergbknudson, dstanek, for in-tree extensions we have prior art for that - the doc tree19:15
morganfainbergbknudson, dstanek, and i was actually typing the response to that before you guys asked :P19:15
morganfainbergnot sure i like that structure, but it could be worse.19:16
dstaneki have to think about that a bit19:16
dstaneki'm just not sure about stevedor based on the way we wire extensions via paste19:17
bknudsonsince we test with all the models we should probably run with all the models19:17
morganfainbergbknudson, that is kindof my thought19:17
bknudsonand we test with all the tables created so should probably run that way19:17
bknudsonor fix the tests so that it works the way the server runs19:17
bknudsonthis is why global state is a disaster waiting to happen19:18
*** leseb has joined #openstack-keystone19:18
morganfainbergbknudson, the way SQLA registers models / table defs19:20
morganfainbergbknudson, i think the answer is to run with 100% of the models and the schema in the db19:20
dstanekmorganfainberg: so for real deployments have all tables for all extensions loaded?19:21
morganfainbergbknudson, it doesn't hurt anything to have the extra tables for tests (we don't check that tables exist/dont)19:21
morganfainbergdstanek, that is a migrate question19:21
morganfainbergdstanek, real deployments use migrate.19:21
morganfainbergdstanek, we could go back to migrating everytime.19:22
ayoungmorganfainberg, is stevedore the path forward?  dstanek it seems to be in the same general field as your snake-guice approach19:22
dstanekwhat problem are we trying to solve now? seems like we got away from the directory crawling19:23
morganfainbergdstanek, it probably wouldn't be bad to just create the tables for all extensions everytime anyway - it doesn't cost a lot and it is less likely ot cause issues with DBs having a new extension enabled and not having the schema to support it19:23
ayoungPub crawling!19:23
ayoungoops, sorry19:23
*** bach has joined #openstack-keystone19:23
morganfainbergayoung, stevedore for loading the backend is the "openstack" way (vs. importutils?)19:23
dstanekayoung: not really; it's about extensions not about IoC19:23
ayoungdstanek, not as I read it19:23
morganfainbergayoung, it just happens to work for this usecase (sortof)19:23
ayoungdstanek, they are two sides of the same coin19:23
ayoungdstanek, ok, lets start with entrypoints.  What are they good for?19:24
dstanekayoung: i don't think they are - it doesn't construct object graphs19:24
morganfainbergayoung, entrypoints solve a different issue, simplified namespace + objects (and/or scripts)19:24
morganfainbergayoung, it also could allow other modules (3rd party) to supply backends to keystone by using the correct namespace19:25
morganfainbergayoung, e.g. keystone.assignment backendModule My_Cool_assignment_backend19:25
dstanekmorganfainberg: the thing with that is that we currently put the fully qualified module path in the config19:25
morganfainbergand you just ask for that vs needing to know <where>.<did>.<i>.<install>.my_cool_backend19:26
ayoungthe thing is, I think entrypoints are just another way of naming dependencies19:26
morganfainbergayoung, uhm could be?19:26
ayoungso  " I need a database connection"  vs "I am a database connection"19:26
dstanekayoung: it's much higher level19:26
morganfainbergayoung, sure you could do that19:26
morganfainbergbut i think dstanek is right19:26
bknudsonmorganfainberg: I just did a recheck db2-test and it worked this time.19:27
dstanekyou wouldn't create an entrypoint for every interface19:27
bknudsonmust have healed itself19:27
morganfainbergbknudson, cool. ok no concerns then.19:27
dstanekand you wouldn't want more than one thing registered for an entrypoint19:27
morganfainbergbknudson, thanks :) (hadn't gotten to review yet, was on other discussions first)19:27
ayoungmorganfainberg, dstanek I don't want two mechanisms where one will do, and I want the one that is the most "core"19:28
dstanekayoung: a quick example is ICatalogBackend (i hate the I naming convention, but it's easier to visualize)19:28
ayoungFrom what I've read of entrypoints, its for componentn naming19:28
dstanekwhich one do you pick? all 3 are registered under stevedore (and maybe thirdpary ones too)19:28
morganfainbergayoung, you need to be specific about which one tyou're loading, the entry point just saves needing to know the full path for every possible variation19:29
ayoungdstanek, you, how do you distinguish between three components that all fulfill the same interface contract?19:29
dstanekayoung: exactly. we do that in the config file19:29
bknudsonso would we have something like [assignment] driver=sql ?19:29
morganfainbergayoung, you ask for <namespace [ like keystone.assignment ]> with the module SQL(or KVS)19:29
morganfainbergbknudson, yep. and we'd look for keystone.assignment namespace, module sql19:30
dstanekentrypoints are really good for dynamic discovery questions like 'show me a list of all identity backends'19:30
morganfainbergbknudson, the namespace would be known and we always look in that namespace for an assignment module19:30
dstaneknot so good at instantiating classes19:30
ayoungI'm just learning stevedore, and I have not yes groked it, nor do I have a position on how well or poorly it is designed, but since it looks like it is already integrated its way into opnestack, I'm trying to understand it.  Stevedore seems to make heavy use of entrypoints, and so I m trying to get a good sense of them.19:30
morganfainbergso rax could have keystone.assignment <RAX>19:30
bknudsonand [token] provider=uuid19:30
bknudsonthat would be nice.19:30
morganfainbergbknudson, ++ yep19:31
ayoungFrom what little I've read, there is a concept of namespaces.  I would say that you use namespaces as one way to distinguish19:31
morganfainbergbknudson, that solves a separate problemspace from i think the test one19:31
dstanekbknudson: that's how we would probably use it19:31
ayoungIts not an easy problem.  At its most complex you have a query language, and you use that to select the component you want19:31
morganfainbergwould it hurt us to always migrate all extensions? and run with all models?19:31
ayoungbased on "criteria"19:32
dstanekthen code would look to see what was registered as sql for the tokenbackend19:32
bknudsonso we're already using entry points, for the cli19:32
morganfainbergback to the origional discussion19:32
bknudsonis that stevedore already?19:32
ayoungalternatively, you annotate a component.  For example, with databse conections, in an ecommerce site you can have "catalog, orders, browsing"19:32
morganfainbergbknudson, i don't think so w/ keystoneclient19:32
ayoungeach are tuned differently19:33
dstanekbknudson: no that setuptools19:33
morganfainbergbknudson, but we might be?19:33
ayoungso a given component that needs both catalog and orders should be able to say "I need two different data base connections.  One is the order one..."19:33
dstanekayoung: that is what you would do in an IoC framework19:34
ayoungdstanek, RIGHT19:34
ayoungdstanek, but the component naming standard seems to be "entrypoints"19:34
morganfainbergayoung, i think entrypoints are more for providing options in a given namespace but not providing _the_ option.19:34
dstanekmorganfainberg: exactly19:34
dstanekyou still need something to say "i use the sql catalog"19:35
morganfainbergayoung, you're looking for a single option here, stevedore allows you to register something clearly into a namespace and make it available without needing to know
morganfainbergdstanek, ++19:35
morganfainbergayoung, exactly what dstanek just said19:35
ayoungmorganfainberg, yes.  Which to me sounds like "neither necessary nor sufficient"19:35
ayoungso why use it?19:35
morganfainbergwe can table that discussion and bring in common code folks (dhelmann) on why/why-not use that19:36
dstanekwe can use entrypoints without stevedor; including what dhellmann's patch does19:37
dstaneki have to look into stevedor to see what it actually does on top on entrypoints19:37
morganfainbergi think we need to (more importantly) solve the model issue wetherornot it's entrypoints / stevedore19:37
ayoungWhat is the "model" issue?19:37
morganfainbergayoung, we need to import all models (SQLA) for testing so db schema matches with reflection19:38
morganfainbergayoung, erm so db schema matches in all tests because we use reflection to create it19:39
ayoungcuz I have to say, I really don't like the look of that setup.cfg19:39
morganfainbergexcluding the migration_specific tests19:39
morganfainbergotherwise the db schema could vary on any given test.19:39
morganfainbergif a model has been imported or not19:39
dstanekmorganfainberg: the real problem is that you don't like crawling through dirs during the tests and what (if anything) do we do in deployments19:40
morganfainbergdstanek, i think crawling the directories and hoping things are named sanely is bad19:41
ayoungmorganfainberg, that was the argument for using migrations to setup the schema for the tests.  We had that a year+ ago19:42
morganfainbergdstanek, i would rather have models in one place vs scattered all over if we need to centralize like that19:42
ayoungthe issue was really just speed19:42
morganfainbergayoung, i can move us back to migrations, it's just slow19:42
ayoungbut migrations are the cannonical approach19:42
morganfainbergayoung, and it doesn't really buy us a lot imo19:42
ayoungIt means we test our production code path19:42
morganfainbergi would almost argue we should always migrate in all extensions19:42
morganfainbergrather than having varying schemas19:43
morganfainbergayoung, we do test that with the SQL migration tests.19:43
ayoungwe can do that19:43
dstanekmorganfainberg: the hard part there is that sometimes you only need catalog sql and not identity sql. does that matter?19:43
morganfainbergdstanek, not really19:43
ayoungthe migrations are called by the command line.  The default there if called with no params can be "migrate all exensions to their max"19:43
morganfainbergayoung, no i mean, always migrate in extensions (in tree ones) even for prod19:44
morganfainbergayoung, is there a reason not to add the schema information19:44
ayoungAh.   I think that was the origianal intention19:44
morganfainbergayoung, oh ok then :)19:44
ayoungI'd be OK with that, too.19:44
morganfainbergayoung, the second question, would you be opposed to in-tree (even extensions) keeping their models in a common module19:44
morganfainberge.g. keystone.common.sql.models.<module>19:45
ayoungeach backend should have its own migrate repo.  Then again, we should treat core modules as nothing more than extensions that have been blessed19:45
morganfainbergso extension would have it's own module for that19:45
morganfainbergayoung, each backend would keep it's own migrate repo for sure19:45
ayounglets keep the migrations split19:45
morganfainbergayoung, just the SQLA models themselves19:45
ayoungthat way wehn we cut an extension we don't end up with "oops, FK"19:45
ayounghmmm, I don't think I like that.19:46
ayoungI think I like the current approach.  It seems more like the problem is figuring out which extensions have SQL models19:46
ayoungwould you just lump "core" together, or would it require extensions as well?19:47
morganfainbergayoung, i'd make assignment be ""19:47
morganfainbergor similar19:47
morganfainbergsame w/ identity19:47
morganfainbergeach subsystem would be explicitly named out19:48
ayoungso...   backends/sql/  ?19:48
morganfainbergwe could do that19:48
dstanekwhat's the difference between that and what we have?19:48
ayoungI think that is the right format if we do.  What does that buy us?19:48
bknudsonwe already have a keystone.common.sql19:48
morganfainbergdstanek, ^19:49
dstanekmorganfainberg: i just want a way to "discover" the models in a reliable way19:49
morganfainbergit's more of a question, do we want models to be separate from the extensions (and core) in a common location or do we want to walk the tree to find them19:49
dstaneknaming convention seemed to be that way since we seem to be good at enforcing it19:50
morganfainbergdstanek, i'm with you, i don't know if looking for "" is the best option19:50
dstaneki like them separate just because it keeps the code closer to where it's used19:50
morganfainbergdstanek, i think the issue here is when you're using a build system you aren't guaranteed to be running in the keystone directory19:51
morganfainbergyou might be running elsewhere and end up with "home" or some other thing and home.keystone.thing doesn't exist19:51
morganfainbergor worse19:51
dstanekmorganfainberg: why would you need to be in the keystone directory?19:51
morganfainbergor in nkinder's issue keystone wasn't installed in the venv and so it tried to import a non-existant model19:52
morganfainbergdstanek, who knows where the build system sticks files before trying do things.19:52
morganfainbergdstanek, or if "." is in the python path19:52
morganfainbergi just don't like "guessing" this is the file to import19:52
morganfainbergi'd rather it be a bit more rigid.19:53
dstanekthe keystone package has to be in the path for anything to work. i think that is all my code depends on19:53
morganfainbergdstanek, then i'm not sure how dhellmann ended up with "home" in his path or nkinder's test run failed19:54
morganfainbergerm home in the module path19:54
dstanekit's possible then that the line that does the import needs to do a slice - i'll try think from outside of keystone19:54
morganfainbergdstanek, sure.19:55
morganfainbergdstanek, if you think the os walk is the best possible option19:56
dstanekmorganfainberg: i don't, but that's the only one i had :-)19:56
morganfainbergdstanek, i'm happy to continue with it, but i think i'd rather see if we can do something better :)19:56
morganfainbergok s/best possible option/current best plan/19:56
ayoungmorganfainberg, I think the sql code is explicitly part of the Driver.  Spliiting it from the driver seems like splitting along the wrong axis19:57
dstanekmorganfainberg: i don't know how else to do it besides forcing all sql module to be in one place or to make them register somewhere19:57
morganfainbergdstanek, perhaps registering is the right answer?19:57
morganfainbergayoung, solid enough argument for not splitting it up for me19:58
morganfainbergdstanek, i kindof with SQLA didn't automagically register the model when you defined it on the declarative base19:58
morganfainbergdstanek, s/with/wish19:59
ayoungmorganfainberg, why did dean submit the entrypoint patch?  What does stevedor propose to do for us here?19:59
morganfainbergdstanek, we could be more programatic then.19:59
dstanekayoung: in this case nothing - with can use and inspect entrypoints without it19:59
morganfainbergayoung, it solves an issue wtih bad logic slicing up the name and trying to import the module to get the SQLA models when you don't use_develop (or similar) in build systems19:59
dstanekit just saves us a bit of code19:59
ayoungwhat is use_develop?20:00
morganfainbergayoung, but it's mostly saving some code on one front20:00
ayoungit seems brittle?20:00
morganfainbergayoung, use_develop is a python-ism to symlink to the local directory vs. instealling the stuff into the packages, it's a setuptools ism20:00
*** jsavak has joined #openstack-keystone20:01
morganfainbergayoung, stevedore is good for loading modules consistently, but i don't know (after this discussion) if it's the option we want for this problem20:01
morganfainbergayoung, i'd like to use stevedore for loading the backends, but that is a different reason than getting the models for SQL into the python namespace20:02
dstanekmorganfainberg: what does stevedor do on top on entrypoints? does it just replace the __import__?20:03
morganfainbergdstanek, it replaces the __import__ and can instantiate the object directly as well w/ arguments/config20:03
ayoungmorganfainberg, so my take is the entrypoints are like the explicitl exported symbols you get from dll_loadibrary or the elf equivalent, versus python class names being more like the native symbols you would get from the symbol table.  One is more "public" than the other, but if you try hard enough you can get access to either20:04
morganfainbergdstanek, it's mostly a standardized framework that is a bit more straightforward than using importutils everywhere20:04
*** joesavak has quit IRC20:04
dstanekmorganfainberg: i wonder if it gives us a hook to do DI20:04
morganfainbergdstanek, that is a good question20:05
morganfainbergdstanek, that'd be useful20:05
morganfainbergayoung, that sounds about right. you can be more/less granular than objectclasses20:05
dstanekwhat concerns me there is the way i'd like to see DI work20:06
morganfainbergayoung, it's a little more open than the exported symbols in DLLs (you could have other packages register into the same namespace) but yes.20:06
morganfainbergdstanek, even if we only use stevedore for specifying the driver, i think it's a win. we could even package (not saying we should) the assignment backends separate from keystone if we wanted in that case.20:07
morganfainbergdstanek, it may not be worth using for DI at all.20:08
morganfainbergdstanek, at worst is brings us more in line with other OpenStack projects on importing modules and using those imports as a driver/object. at best we can do soemthing far cooler20:08
dstanekthat would be interesting if that were the case20:09
morganfainbergs/importing/dynamically importing based on config/20:09
*** topol has quit IRC20:09
morganfainbergdstanek, so lets get back to the SQL thing :)20:10
morganfainbergdstanek, i like this convo, but lets solve the small issue first if we can.20:10
*** andreaf has quit IRC20:11
morganfainbergdstanek, oh i found the bug.20:11
ayoungIs there a "real" problem with the sql thing?20:11
morganfainbergayoung, minor problem.20:11
morganfainbergdstanek, i see where nkinder ran afoul20:11
dstanekmorganfainberg: what was the issue?20:12
nkinderwhat'd I do this time? :)20:12
morganfainbergnkinder, you did everything right and showed us a bug,20:12
morganfainbergdstanek, oh wait i'm wrong :(20:13
morganfainbergsomehow we end up with '' as the module name after root.replace(os.sep, '.')20:14
morganfainbergnkinder, you still didn't do anything different than you should have.20:14
dstanekmorganfainberg: my guess is that root needs to have the first part ripped off base on keystone_root20:15
dstanekdo you know how i can replicate?20:15
morganfainbergwell for dhellmann's case, set use_develop in tox.ini to false (it looks like)20:16
morganfainbergi'm trying it now20:16
morganfainbergdstanek, and yea, looks like a slice should work20:17
morganfainbergstil not sure why you end up with '' as the module_name20:18
morganfainbergack i gotta go, running late20:19
nkinderayoung: I'm testing the remember the dn patch, and I'm seeing something odd (unrelated I think/hope)20:21
nkinderayoung: I run "keystone user-update --name=foo --email=bar <id>" for a valid id20:21
nkinderit hits ldap, searches for the user and gets a result, then searches the enabled group (using emulated enabled option)20:22
nkinderthen I get a 404 and no mod occurs20:22
nkinderayoung: create, delete, and list are working fine20:22
nkinderget too20:23
ayoungnkinder, that is way too close to our code to be coincidental20:23
nkinderinteresting that all of the unit tests pass though20:23
nkinder...though the unit tests are using fakeldap and I'm using live now20:23
ayoungyeah, fake is , well, fake20:25
nkinderayoung: which keystoneclient commands map to the group stuff in identity?20:27
ayoungnkinder, um...let me see...20:28
ayoungI thinkthat stuff was never added to CLI20:28
ayoungyou'll have to Curl it20:28
ayoungnkinder, least, I don't see any group operations in the CLI args list20:29
nkinderayoung: me either20:29
nkinderI suppose I'll need to use curl then20:29
ayoungI might have an example20:29
*** bach has quit IRC20:29
ayoungbut  this might start you on the way:
ayoungyou could also probably do it from python using the API directly even easier20:31
ayoungthat is why we need ^^ patch20:32
*** dstanek is now known as dstanek_zzz20:33
*** bach has joined #openstack-keystone20:33
nkinderayoung: yeah, I've used your example before.  I can tweak it and use the group API20:33
*** joesavak has joined #openstack-keystone20:37
*** leseb has quit IRC20:39
*** jsavak has quit IRC20:41
*** leseb has joined #openstack-keystone20:41
*** harlowja is now known as harlowja_away20:44
*** dstanek_zzz is now known as dstanek20:50
nkinderayoung: that 404 error is interesting.  I added a pdb.set_trace() at the top of update_user() in the LDAP driver, and it never gets hit.20:54
nkinderayoung: debug logging shows that it searches and finds the user, but they it gets this:20:55
nkinder2014-04-23 13:52:18.978 27998 INFO eventlet.wsgi.server [-] - - [23/Apr/2014 13:52:18] "PUT /v3/users/141812f41b67434296b5fd6eebceedcd HTTP/1.1" 404 228 0.00186920:55
ayoungnkinder, let me see...20:55
*** leseb has quit IRC20:56
ayoungnkinder, v2 or v3 update_user?20:56
ayoungusing the cli?20:56
nkinderayoung: yes, cli20:57
*** leseb has joined #openstack-keystone20:57
ayoungnkinder, is there a project in that request?  There looks like a lot of project logic in the controller20:57
nkinderayoung: using v3 for OS_SERVICE_ENDPOINT20:57
bknudsongroup commands are in the openstack CLI20:57
nkinderayoung: I didn't explicitly set one, so it's just "default"20:58
ayoungbknudson, not in the help message20:58
bknudsonopenstack --os-identity-api-version=3 --os-auth-url=http://localhost:5000/v3 group create admin20:58
bknudsonayoung: the help message is magic and changes based on --os-identity-api-version !20:58
nkinderbknudson: ok, thanks.  I'll use that for testing the group side of things20:58
nkindermagic == smart20:58
*** leseb has quit IRC20:58
bknudsonmight be nicer if it said (only available with identity-api-version=3)20:59
*** leseb has joined #openstack-keystone20:59
*** Anju_ has quit IRC21:01
*** dstanek is now known as dstanek_zzz21:02
nkinderayoung: I like your new shirt21:02
ayoungI always wished I could grow my hair that long, but gravity never takes effect, so it only grows up21:03
*** leseb has quit IRC21:03
*** harlowja_away is now known as harlowja21:05
*** erecio has quit IRC21:07
*** david-lyle has quit IRC21:09
*** andreaf has joined #openstack-keystone21:21
*** derek_c has joined #openstack-keystone21:27
*** jamielennox|away is now known as jamielennox21:40
*** joesavak has quit IRC21:41
ayoungnkinder, is there some sort of SELinux reason that keystone couldn't make an LDAP call?21:43
nkinderayoung: yes, if you're running it within httpd21:44
*** bach has quit IRC21:45
ayoungnkinder, probably same rules for eventlet then21:45
nkinderayoung: not necessarily (unless it's confined)21:45
nkindergetsebool -a | grep ldap21:45
nkinderayoung: httpd_can_connect_ldap --> off21:45
nkinderayoung: user setseboot to allow it to connect21:46
ayoungyeah, I have that21:46
nkinderyeah, off means it's not allowed21:46
*** derek_c has quit IRC21:46
ayoungnkinder, I'm getting "2014-04-23 21:46:57.853 25944 TRACE keystone.common.wsgi SERVER_DOWN: {'desc': "Can't contact LDAP server"}21:47
nkinderayoung: what ldap server are you using?21:48
ayoungi can do an ldapsearch from thesmae machine from the command line, so I know the URL is good21:48
ayoungits an internal IPA21:48
nkinderayoung: ps -efZ and grep for keystone21:48
nkinderwhat context is it running as?21:48
*** med_ has joined #openstack-keystone21:48
ayoungsystem_u:system_r:keystone_t:s0 keystone 26057     1  0 21:47 ?        00:00:00 /usr/bin/python /usr/bin/keystone-all21:49
nkinderah, keystone_t21:49
ayoungits a packstack install, so its run from systemd21:49
ayoungI haven't switched this one over to running in httpd yet.  One step at a time21:49
nkinderinteresting... It would need it's own rule to connect21:49
nkinderayoung: you can manually create a rule, but a boolean would be nice21:50
nkinderayoung: might be worth a selinux-policy bug if you don21:50
nkinderdon't see an existing boolean for it21:50
nkinderayoung: do you see AVC messages?21:51
*** bach has joined #openstack-keystone21:51
nkinderayoung: you might need to turn off dontaudit rules to be sure21:51
nkinderayoung: semanage dontaudit off21:51
mfischCan I get a +2 from someone on this?
ayoungnkinder,  yep type=AVC msg=audit(1398289617.851:119991): avc:  denied  { name_connect } for  pid=25944 comm="keystone-all" dest=389 scontext=system_u:system_r:keystone_t:s0 tcontext=system_u:object_r:ldap_port_t:s0 tclass=tcp_socket21:53
nkinderayoung: using wireshark to watch keystone->ldap is very enlightening21:53
ayoungwhat is dontaudit?21:53
nkinderayoung: 8 operations (not counting binds) just to add a user to a group21:53
nkinderwith 8 individual connections21:53
nkinderand ssl/tls handshakes, etc.21:54
ayoungnkinder, whomever added LDAP  support to Keystone obviosuly had no idea what he was  doing21:54
nkinderayoung: counting binds and unbinds, it's 24 LDAP operations21:54
nkinderthough unbind is essentially free21:54
nkinderayoung: I'm producing a little report on the operations for user/group CRUD21:54
nkinderdontaudit is on by default.  It causes certain AVC denials to be silent (no logging)21:55
nkinderit's for common cases that might occur but not be fatal21:55
*** bach has quit IRC21:56
jamielennoxmorganfainberg: did you ever look at jsonschema stuff?21:58
morganfainbergjamielennox, not in depth22:00
ayoungnkinder, its def selinux.  setenforce 0 makes it work22:00
morganfainbergjamielennox, been on other tasks here (limited bandwidth last couple days)22:00
nkinderayoung: yeah, I'd just add a custom policy module to allow it.  It is worth a bug to ask for a boolean22:00
ayoungI'm going to leave it on permissive until I straighten out the database, then I'll move to keystone in httpd and re-enfroce22:00
*** bach has joined #openstack-keystone22:00
ayoungnkinder, myabe...but I would rather drive on to httpd-ness instead22:00
ayoungthe more work we do with the borken eventlet approach them ore pain it will be to wean ourselves.22:01
nkinderayoung: but others may run into this.  The selinux devs could add it very easily22:01
ayoungYes they could22:01
ayoungbut you can't polish a t***22:02
jamielennoxmorganfainberg: i was playing with it yesterday - essentially i want to know if we could build something similar to WSME but based on jsonschema22:02
jamielennoxmorganfainberg: thought it'd be good to get someone else's opinion to see if i'm way off22:02
morganfainbergjamielennox, that would be cool22:02
ayoungnkinder, actually, this is just one of a slew of issues going to LDAP in packstack22:02
morganfainbergjamielennox, looking at the paste now22:02
jamielennoxi think it's promising but it feels slightly off in implementation22:04
*** derek_c has joined #openstack-keystone22:04
ayoungI'll open the bug, though22:04
ayoungnkinder, what is the component that has default policy?22:06
ayoungis it just policy?22:07
morganfainbergjamielennox, hm. ah i see what you're doing22:08
morganfainbergtook me a bit to switch into metaprogramming mindset22:08
jamielennox i tried to get around it22:08
morganfainbergsometimes you just can't22:08
jamielennoxWSME doesn't use it, but it does some hacky things to not22:08
jamielennoxall the metaclass should be doing is collecting class properties22:08
morganfainbergif it's easier to do with metaprogramming, besides the immidiate "are you sure metaprogramming is the right way" question, it's not bad to use it for clarity22:09
morganfainbergreally hacky to avoid metaclasses is just as bad22:09
jamielennoxi agree22:09
uvirtbotayoung: Error: Could not parse XML returned by HTTP Error 404: Not Found22:10
morganfainberguvirtbot, nice.22:10
uvirtbotmorganfainberg: Error: "nice." is not a valid command.22:10
ayoungmorganfainberg, we just put on some XSRF protection, I suspect that is what is breaking it22:10
morganfainbergayoung, that makes sense22:10
ayoungnkinder, so packstack with LDAP/IPA needs to be planned out.   I talked a little with simo about setting passwords.  but the user-adds are all going to have to be IPA commands, not keystone.22:12
*** marcoemorais has quit IRC22:13
morganfainbergjamielennox, so i'd be concerned with some of the choices of instance variable names in array_validator22:14
jamielennoxmorganfainberg: sure22:14
*** harlowja has quit IRC22:14
bknudsonstevemar: any reason isn't +A?22:14
jamielennoxmost of the naming is off22:14
*** marcoemorais has joined #openstack-keystone22:14
morganfainbergjamielennox, yeah22:14
jamielennoxmorganfainberg: it just doesn't feel right that it's so based around the Model22:15
bknudsonmorganfainberg: ?22:15
jamielennoxand that the Model is an object and not a Type like the others22:15
morganfainbergjamielennox, agree22:15
bknudsonmorganfainberg: (you had reviewed it earlier)22:15
morganfainbergbknudson, looking, i think this is a simple spot check for me.22:15
morganfainbergjamielennox, let me look at this review rally quickly22:16
jamielennoxi don't really know a way around it because of the way the __get__ and __set__ work22:16
jamielennoxmorganfainberg: np22:16
*** stevemar has quit IRC22:17
*** marcoemorais has quit IRC22:18
morganfainbergbknudson, +2/+A lgtm, my concerns were addressed with the clearer message on the exception22:20
morganfainbergjamielennox, ok looking at the __get__ and __set__ bits22:20
morganfainbergjamielennox, noriced you also had the ObjectValidator commented out22:22
jamielennoxright, have you seen the __get__ and __set__ before?22:22
morganfainbergjamielennox, totally aside, was that not working?22:22
jamielennoxi don't know22:22
morganfainbergthe descriptor special methods?22:22
morganfainbergi've not used __get__ and __set__ before22:23
jamielennoxit means that you an define the property and set the value on the object22:23
*** marcoemorais has joined #openstack-keystone22:24
jamielennoxso when you set m = Model() m.prop_int the data is actually being set on the m object22:24
jamielennoxso the 'types' don't store any actual data22:25
morganfainbergjamielennox, yeah i see that now.  this is a good writeup btw22:25
jamielennoxbut that means you have to have an actual object at the root22:25
jamielennoxwhich is why the base object is not a 'type' like the others and why i comment out the object one22:25
jamielennoxit means though that everything is based around the object though - you couldn't validate an array or an integer directly22:27
morganfainbergjamielennox, right you can't just say do_validate(thing) it needs to be passed through22:28
morganfainbergjamielennox, hm.22:28
jamielennoxI was wondering if it's worth building them as actual types - but i think that's overthinking it22:29
*** bach has quit IRC22:33
morganfainbergjamielennox, yeah not sure building this as a type would be beneficial here22:33
morganfainbergit would be more consistent, but not exactly "better"22:33
jamielennoxwas thinking if the outcome of integer = Integer(minimum=1, maximum=5) then you could do i= integer(4)22:34
jamielennoxbecause it feels like it's mixing the type of the object and the data of the object22:34
morganfainbergjamielennox, get full comparitor functionality22:34
*** amcrn has quit IRC22:34
morganfainbergdamn it22:35
morganfainbergbrain crossed words22:35
jamielennoxcould do that with __call__ but i don't know what that returned object would be22:35
*** thedodd has quit IRC22:36
morganfainbergjamielennox, i like __call__ less.22:36
jamielennoxso the __get__ __set__ make sense22:37
morganfainbergjamielennox, yeah22:37
morganfainbergjamielennox, it does.22:37
jamielennoxbecause if you do i = Integer() i = 5 then you are replacing the variable - not doing a __set__22:37
morganfainbergjamielennox, i think with comments, it will make more sense than using __call__22:37
morganfainbergrebinding issue22:37
*** bach has joined #openstack-keystone22:38
jamielennoxand Object could be a Type, if we could have somewhere else to actually store the data22:38
jamielennoxcould just do a Validator object or something22:38
morganfainbergjamielennox, this argues for simply making a clever response class that acts like we want it to22:38
morganfainbergjamielennox, erm response/data22:38
morganfainbergjamielennox, which is what your model class is now.22:39
jamielennoxmorganfainberg: except that at the moment that's being used instead of an Object type22:39
jamielennoxand it implies an object22:39
*** derek_c has quit IRC22:39
jamielennoxit wouldn't let us validate an integer or anything directly22:39
morganfainbergjamielennox, fair.22:39
morganfainbergjamielennox, lets start with what points are we validating?22:40
morganfainberginput on the API, right?22:40
jamielennoxthe problem isn't the base one though, it's when you get nested objects22:40
*** bach has quit IRC22:40
morganfainbergah, ahhh22:40
jamielennoxso what i was hoping to do was provide an @input(UserRequest) and @output(UserResponse)22:41
jamielennoxand figure out how we could hook up an Accept to return get_schema()22:41
jamielennox(that bit's later)22:42
jamielennoxto start with the object would have to be similar to the dict of now22:43
morganfainbergwhich is really fine22:43
jamielennoxmorganfainberg: yes and no22:43
jamielennoxi'm not sure how you cross those object boundaries but it can be figured out22:44
morganfainbergthis is a ton of moving parts, :(22:45
jamielennoxthis is what i mean, i don't think there is enough logical seperation between data and type22:45
morganfainbergyeah i can see that now22:45
jamielennoxi was thinking about this for a day or so before i started but i didn't think it'd be this hard to keep seperate22:46
morganfainbergat face value this looks "easy"*22:46
morganfainberg* = some value of easy22:46
jamielennoxyea, i thought it'd be mostly figuring out what parts of jsonschema to support22:46
morganfainberglets step back, is there any way we can simplify this a bit22:47
jamielennoxwhether you do validation on __get__ and __set__ or on validate()22:47
morganfainbergbear with me, trying to step through it22:48
morganfainbergso we want input validation - user sends a request, we need to make sure it conforms to the correct data22:48
jamielennoxmorganfainberg: this is why i was getting someone else to bounce ideas off :)22:48
morganfainbergare we also aiming to do output validation "output is supposed to look like XXXX"? or mostly input22:49
jamielennoxmorganfainberg: i think input is the most obvious, if we can get that working then i thinnk we would pick up a number of bugs doing output22:49
jamielennoxbut maybe that's a debug option22:49
morganfainbergjust making sure, output validation sounds great for debug / tests22:50
morganfainbergbut it could be very odd responses if output validation failed in production22:50
morganfainbergrequest comes in, we go json -> python22:51
morganfainbergnow we're at where we are today22:51
morganfainbergcould we invert this?22:53
morganfainbergcould we instead of going json -> object, we do validate(json) -> object22:53
morganfainbergjsonschema seems to support that kind of concept22:53
morganfainbergoh wait. nvm misreadinfg it22:53
jamielennoxno, you start with python objects22:54
morganfainbergjust saw that22:54
jamielennoxwhich is good cause it means you can do xml as well22:54
jamielennoxor any input that comes through to a python object we understand22:54
morganfainbergjamielennox, right.22:55
morganfainbergjamielennox, this is a tough one23:04
jamielennoxmorganfainberg: it seems though that there should be an elegant solution23:04
jamielennoxotherwise i'd probably just have scrapped the idea and used jsonschema directly23:04
morganfainbergjamielennox, take the json and use something like protobuf as the validator (deseriealize it into the protobuf object) and use that for validation (only partly serious/joking)23:05
morganfainbergjamielennox, i mean, that is what it feels like it should be: define simple schema, take serialized data and deserialize to <object>23:05
morganfainbergjamielennox, ???23:05
morganfainbergjamielennox, profit!23:05
morganfainbergdeserialization should implicitly validate23:06
nkinderok, the extra searches for users and groups in the ldap driver are basically just there for exceptions23:07
*** richm has quit IRC23:07
nkinderif you try to check if a user is in a group, you get one exception if they are not a member (but are a valid user)23:07
nkinderyou get a different exception is the user doesn't exist23:07
nkinderis that behavior really required?23:08
nkinderjamielennox, morganfainberg: perhaps you guys have some thoughts on it? ^^^23:09
morganfainbergnkinder, we often look for a usernotfound error vs a groupnotfound etc23:10
morganfainbergnkinder, we do similar things in the other backends.23:10
morganfainbergnkinder, or uhm. whatver the other error really is23:10
nkinderUserNotFound vs NotFound23:11
morganfainbergnkinder, one is more specific23:11
morganfainbergnkinder, it's useful to differentiate, but they're both 404s23:12
nkinderhere's what sucks about how we do this... this is what we do when adding a member to a group with the ldap driver:23:12
nkindersearch user, search enabled, search group, search user, search enabled, search group, search group, modify group23:12
morganfainbergnkinder, we can probably streamline that.23:13
nkinderaround each of those, there is connect, bind, <op>, unbind23:13
morganfainbergnkinder, we've already searched user, so we _might_ get away with knowing we did that23:13
nkindermorganfainberg: yeah, I see value in the separate exceptions23:13
morganfainbergbut isn't that the DN comparison issue?23:13
nkindermorganfainberg: no, this is with the DN fixes23:13
nkinderI need to see what it is without them23:13
jamielennoxyea, there are a lot of lookups that are just validation23:14
morganfainbergi bet we can streamline those operations (similar to how some SQLisms are moved into joins behind the scenes)23:14
jamielennoxi was hoping that we could put some of this onto the object so fetch user only if it's not saved on the object23:14
morganfainbergit's a naive approach that more closely mirrors the KVS style system23:14
jamielennoxthe request object23:14
morganfainbergjamielennox, that is one option23:14
morganfainbergjamielennox *nod*23:15
jamielennoxthat's easier with pecan where the request is available everywhere23:15
jamielennoxbut the pecan thing is harder / less desirable than i would have liked23:15
morganfainbergdstanek_zzz, ok i think i have a simple alternative fix for dhellmann's and nkinder's problem. it is relative/abs path issues23:16
morganfainbergdstanek_zzz, i think i can fix this in 2 lines... or so23:17
morganfainbergdstanek_zzz, but it isn't pretty still23:17
*** andreaf has quit IRC23:18
*** diegows has quit IRC23:19
morganfainbergdstanek_zzz, actually 1 line, just a slice, but easy to duplicate23:19
*** marcoemorais has quit IRC23:21
*** marcoemorais has joined #openstack-keystone23:22
*** gokrokve has quit IRC23:23
morganfainbergnkinder, in short, i think we could make the driver a lot smarter without changing much else.23:24
morganfainbergnkinder, and know if we need to do the lookups or not depending on operations.23:24
nkindermorganfainberg: yeah.  I can eliminate a good chunk of them without changing the exceptions23:25
nkindermorganfainberg: I think the only unavoidable one is when checking if a user is a member of a group23:25
morganfainbergnkinder, might even be able to work around that one.23:26
morganfainbergnkinder but it might require another method that is called that bypasses the extra lookup(s) in some cases.23:26
nkindermorganfainberg: we can iterate on it.  I'll tackle what is straight-forward right now and will compare before/after23:27
morganfainbergnkinder, ++ sounds good23:27
jamielennoxmorganfainberg: that descriptors tutorial is really good - thats not the ne i was originally reading and came to most of those conclusions seperately23:30
morganfainbergjamielennox, yeah it's very good23:31
morganfainbergjamielennox, easy to read and understand23:31
jamielennoxwhich is good that it's a standard pattern - just should have found that first23:31
*** lbragstad has quit IRC23:47
*** gokrokve has joined #openstack-keystone23:58

Generated by 2.14.0 by Marius Gedminas - find it at!