18:02:57 #startmeeting keystone 18:02:58 Meeting started Tue Oct 15 18:02:57 2013 UTC and is due to finish in 60 minutes. The chair is dolphm_. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:59 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:03:01 The meeting name has been set to 'keystone' 18:03:08 gyee, not favorite, just one of the many things to go through in order to get back to standards 18:03:21 is keystone going to support application/x-www-form-urlencoded 18:03:25 umm 18:03:29 #topic Havana 18:03:31 bknudson, wait till we get there 18:03:32 #yay RC2 18:03:43 on to RC3 18:03:49 lol 18:03:51 dolphm_, one thing came up from Fedora packaging is memcached version 18:03:54 AFAIK we won't have an RC3 18:04:05 RC3?! 18:04:10 if that's the case, then RC2 *is* havana 18:04:19 2013.2.0 18:04:19 like final final? 18:04:23 like final final final 18:04:30 ayoung, fedora has what 1.47? keystone was assuming 1.48 for cas? 18:04:32 * gyee waves his hands like a casino dealer 18:04:33 #link https://bugs.launchpad.net/keystone/+bug/1239892 18:04:35 Launchpad bug 1239892 in keystone "Version for python-memcached not specified" [Low,New] 18:04:43 gyee: i don't know what that looks like, but yes 18:04:54 morganfainberg, yeah, and there is no way to communicate it beside requirements.txt 18:05:00 but memcached is just a tes-requires 18:05:30 doesn't that bug apply to keystoneclient as well? 18:05:32 morganfainberg, the numbers were 1.48 and 1.53 I thought, but you are roughly correct 18:05:45 dolphm_, client doesn't much care 18:05:47 ayoung, hrm i just chased this down a couple weeks ago for bknudson 18:05:56 it is a test-and-set thing, and client doesn't make use of it 18:06:29 morganfainberg, if you have more recent data, please update the bug. I am just going from memory. Dealt with this back in July I think 18:06:48 dolphm_, before a certain version the memcache python lib leaks memory like a sieve, so they added an option to no track cas ids unless explicitly "turned" on in later versions 18:07:08 ah, i remember that 18:07:12 ayoung, before whatever version added that init kwarg all transactions saved cas ids, regardless, and nothing ever cleans that up 18:08:03 #topic Reserved migration numbers for havana 18:08:14 morganfainberg, yeah, and since Keystone server depends on that now with the memcached backends, it breaks on Fedora. I suspect Ubuntu LTS too 18:08:28 ayoung, nope LTS is fine 18:08:36 well, lucid no, precise yes 18:08:48 lucid is likely broken…but iirc that is EOL now 18:09:12 just need to add some extra migrations? is it just for core or for extensions too? 18:09:19 oh. apr 2015 =/ boo 18:10:28 dolphm_, what is the impetus for reserving the migration numbers? 18:10:42 wondering if we can accommodate https://review.openstack.org/#/c/50488/ in upcoming RC? 18:10:58 allowing ourselves room to backport migrations as part of bug fixes 18:10:58 you can do cleanups post-grizzly and pre-havana necessary 18:11:27 dolphm_, that will mess up people tracking the main 18:12:06 ayoung, i know nova did it for (i think) grizzly. not sure if i like the idea though 18:12:13 ayoung: the idea is that we merge a bunch of empty migrations to master right now 18:12:17 before we have a real migration 18:12:20 did nova actually use it? 18:12:38 bknudson, don't know, but they had the empty ones. 18:12:48 they've done it again for havana 18:12:58 nova milestone-proposed: https://github.com/openstack/nova/tree/milestone-proposed/nova/db/sqlalchemy/migrate_repo/versions 18:13:07 nova master: https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/migrate_repo/versions 18:13:21 looks like nova used it a couple time? 18:13:37 so many empty migrations? 18:13:42 wait, no they didn't... 18:13:42 i'd like the headroom-- i was wondering if anyone was strongly opposed 18:13:45 since wasn't updated in 7 months 18:13:50 https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/migrate_repo/versions 18:13:52 gyee: i think 10 would be a lot for us 18:13:53 dolphm_, then if someone on a dev branch runs the migrations, they are SOL 18:14:07 ayoung: a dev branch? 18:14:10 dolphm_, master 18:14:23 dolphm_, someone like, oh, Racksapce, that tracks master 18:14:30 Rackspace 18:14:51 remember how much trouble we caused when we went back and rewrote migrations in Grizzly 18:15:23 dolphm_: does it matter? if we are backporting these fixes what is wrong with just tacking the migrations on to the end? 18:15:33 rackspace certainly doesn't track master on keystone 18:15:36 jamielennox: the numbers are sequential 18:15:49 couldn't have a 161 and another 161 18:15:49 and i have no idea whether rackspace uses nova's migrations 18:15:50 does it have to be sequential? 18:16:02 can we have a gap? 18:16:03 gyee: that's how sqlalchemy-migrate tracks it, it keeps the version #. 18:16:09 that'd be a good question for the next 'we track master' presentation at the summit lol 18:16:10 gyee: we can have a gap 18:16:10 dolphm_, I'm sure you know better than I do, but one of the people that yelled loudest was a Racker 18:16:11 right, it means that a backported migration number may not be the same in dev/stable 18:16:29 bknudson, why can we say for IceHouse we start with 100 or something 18:16:33 ayoung: i'm curious if you remember who that was 18:16:36 jamielennox, how do you reconsile the change then… if say rev 037 and 045 do teh same thing? 18:16:57 gyee: hmm, not sure why that wouldn't work. 18:17:16 bknudson, you need placeholders or you need to set the config to start at a fixed "known" number 18:17:21 so...this is why we really should have each of the various modules in their own migration repo. It minimizes the possbility of clash. Doesn't remove it completely, but it is a lot more granular. Identity should not depend on token etc. That is why the extensions at least are now split out 18:17:26 you can't have just a gap. 18:17:51 gyee: not sure if gaps are a good idea, because later if you come along and db_sync it will say ok i'm at eg 50 and start from there, missing all the migrations from the gap 18:18:01 gaps would be confusing, if nothing else 18:18:04 jamielennox, +1 18:18:15 that probably throws out changing migration numbers as well 18:18:38 on the other hand, 10 migrations that run successfully give the illusion of stability (they just happen to be no-ops) 18:18:52 nova has a big gap at beginning (it starts at 133) 18:19:01 bknudson, the config says to start there though. 18:19:01 bknudson: that's not a gap though 18:19:12 bknudson, they collapse each release 18:19:13 we haven't needed this so far. What happens if we don't skip numbers? 18:19:21 we should wrap db_sync to skip the prompt if we know the gap number 18:19:23 bknudson: that's a collapsed migration of 1 -> 133 18:19:26 https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migrate_repo/migrate.cfg doesn't have 133 ? 18:19:32 bknudson: ish 18:19:45 IIRC we were going to collapse all of the pre-grizz revisions this time around. 18:19:51 ^ 18:19:54 dolphm_: but the same thing would happen right? if you upgrade from stable to dev or to the next stable your db will think it has run a bunch of migrations that are no-ops and not do it next tim 18:20:05 jamielennox, that is correct 18:20:28 jamielennox: the no-ops land in master 18:20:33 jamielennox: not in stable 18:20:56 dolphm_, I don't think we have that freedom 18:21:12 I am pretty sure that skipping is a mistake 18:21:19 ayoung: it's not skipping 18:21:27 it is 18:21:27 how about do the gap and modify db_sync to automagically generate the no-opts? 18:21:45 gyee, it will break people that track master 18:21:46 ayoung: empty migrations in master mean that you can write migrations in stable 18:21:56 no you can't 18:21:58 dolphm_: so the same thing will happen in master then, if you bug fix a migration in one of these no-ops it won't be run next time you do a db_sync 18:22:04 the migrations that a stable -> master upgrade will skip are just empty migrations 18:22:14 would be interesting to know if nova ever used this and what it was for. 18:22:14 but as far as I can tell nova did not use it. 18:22:21 it means that the state of the database is not reflected buy the version number 18:22:41 so two dbs at version 50 are in different states, with no way to tell which 18:22:42 you would have to be very careful 18:22:57 and you cannot rerun the migrations without corrupting the database 18:23:03 from memory this was something that alembic solved nicely, but not pushing that argument again 18:23:15 might need to do both a placeholder and a new migration in havana 18:23:20 oops, icehouse I mean 18:23:41 bknudson: but then stable icehouse would have two migrations doing the same thing 18:23:42 jamielennox, sort of. It marks each migrations with a hash, like git 18:23:50 k, placeholder then, seems like that's less of the two SOLs 18:23:51 :) 18:23:57 ayoung, i am thinking we should do the work in icehouse to split up the modules, regardless if we have placeholders. 18:23:58 the new migration would have to check if the placeholder ran first 18:24:17 ya'll are way over thinking this 18:24:18 jamielennox, and...that might be the best solution. We do alembic for Icehouse, and add no-new migrations via sql-a 18:24:20 (easier once/if we do the collapse) 18:25:03 I'd prefer to have the placeholders since it doesn't hurt anything. 18:25:12 are we running short of numbers for sqlalchemy-migrate? 18:25:13 then we make a SQL-A migration for stable, we can optionally apply a commit in alembic. Not sure how that will work in practice, though 18:25:24 bknudson, yeah, lopusy supplier ran out 18:25:28 pousy 18:25:29 lousy 18:25:32 pony! 18:25:36 #link http://lists.openstack.org/pipermail/openstack-dev/2013-March/006827.html 18:26:03 everyone go read ^ this week, and don't approve any db migrations until next meeting :) 18:26:17 #topic PEP8 public/internal 18:26:20 bknudson: o/ 18:26:30 I added this one, just wanted to point out that I started work on it. 18:26:39 turns out python-keystoneclient is smaller than I thought 18:26:48 lol 18:26:51 * morganfainberg cheers for bknudson on starting this. 18:26:58 first step was updating __init__.py's with __all__ = [ ... ] 18:27:16 so this lists which modules or packages have public parts 18:27:23 also, started a etherpad... 18:27:33 link? 18:27:34 which has a list of all modules/packages so you can see the list easier 18:27:37 bknudson: love most of it, disagree with renaming tests -> _tests 18:27:51 #link https://etherpad.openstack.org/KeystonePep8PublicInternal 18:27:56 #link https://review.openstack.org/#/c/51415/ 18:28:27 well, the PEP8 standard says if it's not public then should be prefix with _ 18:28:35 I don't make the rules 18:28:50 bknudson, tests are public 18:28:57 bknudson, they just don't get packaged up 18:29:01 ayoung: if they're public then we can't change them. 18:29:04 but tempest shoudl be able to call our tests 18:29:08 ayoung, public interface 18:29:18 tests need to get run by package maintainers, then optionally get excluded from the eventual package 18:29:27 bknudson, yes we can 18:29:33 bknudson: where does PEP8 talk about "internal modules" 18:29:47 #link http://www.python.org/dev/peps/pep-0008/#public-and-internal-interfaces 18:29:48 bknudson, public in this case is different from the public API. tests are meant tbe dynamically enumerated and run 18:30:10 but the standard for enumerating tests is that the names must start with test_ 18:30:27 ayoung, tempest shouldn't be calling our tests, that would compromise the integrity of the tests 18:30:30 the _tests module is internal so everything in it is internal 18:30:33 gyee, wrong 18:30:40 i think i consider keystoneclient.tests itself to be public 18:30:58 tests isn't part of the api. 18:31:02 tempest is integrated tests no? 18:31:05 i agree with ayoung that test really are public 18:31:09 I'm fine with leaving tests as it is. 18:31:39 gyee, if you mean that we can change a commit and change a test at the same time, then, yes. But that doesn not mean that temptest should not run our tests as well. 18:31:43 It is a migration path 18:31:47 bknudson: it's part of *an* API, sort of... just not python lib's api 18:32:12 so we write a test in, say keystone for keystone client, and make sure it runs in tempests. Later on, we can move the test from the keystone repo to tempest 18:32:19 ayoung, why two different gates runs the same tests? I don't get it 18:32:29 that's redundant 18:32:35 also there is a lot of precedent for having a package level tests directory 18:32:35 i don't understand either 18:32:37 gyee, well, because we don't ruyn unit tests against the live DBs 18:32:44 gyee, but tempest should 18:32:46 same with LDAP 18:33:26 bknudson: i do agree with jamielennox's notion that as little should be public as we can get away with 18:33:26 unit tests = SQLite. Functional tests = Postgresql, MySQL, OpenLDAP, and to make the IBMers happy, DB2 18:34:03 IBMers are people too 18:34:03 they are not unit tests by definition 18:34:04 thanks for making us happy ayoung 18:34:08 live tests are not unit tests 18:34:18 stevemar, it is purely selfish. I want you to review my code 18:34:28 we'll have db2 tempest tests in icehouse 18:34:45 live tests are integrated tests 18:35:13 anyway, please take look at the etherpad and update if you know more about the client. 18:35:22 gyee, exactly. SQLite is not a real DB. So test run against it are unit tests. TEsts run against MySQL are integrating our code with the production DB 18:35:29 we can discuss more later if there's further questions. 18:35:35 we just happen to use the unit test code as part of that integration testing 18:35:52 hence, tempest should consume keystone/tests and python-keystonclient/tests 18:35:56 horse is dead 18:36:07 sounds pretty messy 18:36:14 anyway, move on 18:36:22 uhh 18:36:28 #topic open discussion 18:36:28 bknudson, rest of it is good. I hand over the conch 18:36:36 there's LOTS of code reviews on the meeting agenda 18:36:37 HTML. Please remove the -2 18:36:42 #link https://wiki.openstack.org/wiki/Meetings/KeystoneMeeting#Agenda_for_next_meeting 18:36:58 let the masses discuss and we can decide as a group 18:37:11 I'm not a fan of adding complexity of HTML. 18:37:13 I am not sure about opening the can of HTML :) 18:37:18 dolphm_: i had the topic up for the keystoneclient blueprints 18:37:22 we barely support XML 18:37:30 maybe if we dropped XML I'd be OK with HTML 18:37:30 bknudson: ++ 18:37:33 lol 18:37:35 XML is pretty messy as is 18:37:36 bknudson, complexity is contained within the marshalling code 18:37:46 i really want either approval or changes to the APIclient and discovery blueprints - and then get some reviews happening 18:37:46 XML is used by some people for integration 18:37:59 if the marshalling code was cleaner then maybe I wouldn't have a problem 18:38:00 there are first patches for both and they see may 1 review / 2 weeks 18:38:10 jamielennox: i try to maintain the "Code reviews" section of the meeting agenda with stuff that's current -- it doesn't have to be specific to keystone 18:38:32 ayoung: are you going to support IE6? 18:38:36 bknudson, well, that is what the code review is there for. dolphm_ had -2ed it last release, saying it needed to be self contained. It is now, but still has the -2 on it 18:38:54 wondering if we can accommodate https://review.openstack.org/#/c/50488/ in upcoming RC? 18:38:55 dolphm_: I was more looking for a yes/no on blueprints at the moment - then i can at least hassle people to follow allong 18:38:57 dolphm_, Personally? No 18:39:03 what do you think about using jsonschema for input validation? 18:39:03 ayoung: -- 18:39:07 dolphm_, we should support IE4 for mac. 18:39:10 ayoung: how can i apply my company's branding to keystone's HTML interface? 18:39:16 I'm a little worried about our lack of input validation today as is 18:39:16 dolphm_, I am going to support a minimal HTML format 18:39:45 bknudson: i'm a fan, i'd be interested in looking at the wsme and such that is being proposed as well 18:39:45 if we have jsonschema for input validation then that might also be helpful for html 18:39:46 bknudson, i am for validation w/ jsonschema. heck, more validation ++ 18:40:01 bknudson, IMHO that is backwards 18:40:07 regardless of tech. 18:40:09 JSON is only one marshalling format 18:40:22 we could start adding validation all over the place, but doing it centrally would be saner 18:40:24 JSON schema should be driven off the code and not the other way around 18:40:44 the problem is we have no codification of what structures look like and what are the types allowed 18:40:54 e.g., if you pass an int for your password 18:41:16 hey, no fair hacking my password. What is wrong with 1234 18:41:45 ayoung: which html standard? 18:42:00 ayoung: is there going to be a mobile friendly interface? 18:42:07 dolphm_, no, and no 18:42:09 ayoung: how do i apply my own CSS? 18:42:18 dolphm_, ah 18:42:28 ayoung: i'd like the background color to be cornflower blue 18:42:33 here was my example, did "methods": "password", 18:42:34 hahaha 18:42:36 that is a good question. We provide a config option which is the css file. The deployes can optionally add their own 18:42:39 ayoung: no no sunflower yellow 18:42:45 result is "Expecting to find p in identity. The server could not comply with the request since it is either malformed 18:42:50 Keystone core does not support the CSS 18:42:52 I'd like a animated git 18:42:57 so I used a string for methods in auth request 18:42:58 gyee: ++ 18:43:05 gyee, just what we need! 18:43:06 ayoung: where do i put my static files? 18:43:09 gyee, I think I've been called an animated git before 18:43:13 and then it looked for each of the string elements in the methods 18:43:20 dolphm_, its an URL. where ever they can be reached 18:43:22 animated gif 18:43:24 my bad 18:43:39 dolphm_, I have not BPed up that, but it is not hard to solve. 18:43:59 dolphm_, I am just first stpe getting HTML rendering supported in the simplest fashion 18:44:11 is there a lot of demand for HTML? 18:44:12 ayoung: which html standard? 18:44:16 bknudson: i'm having the same thing with the KDS stuff, some things have to be integers, some fields if you don't pass a date you'll get a 500 error 18:44:22 dstanek_: i count one demand 18:44:39 ayoung, all kidding aside, with federation, identity provider may serve up html form for login 18:44:44 bknudson: and looking at past code i think there are a bunch of places where that sort of this would happen with TypeErrors 18:44:58 i can say that the way it's implemented with FreeIPA (html acces), is pretty damn slick. 18:44:59 gyee, and with oauth, Keystone might need to do the same 18:45:16 ayoung: where do i put my google analytics ID? 18:45:26 jamielennox: it's not fun to write that code or tests... I think the only way is to centralize it, and use a jsonschema or something. 18:45:31 dolphm_, where the sun don't shine 18:45:50 I don't really care if we jsonschema or something new for keystone 18:46:23 jsonschema happens on dictionaries after json.loads right, so if it's json/xml it doesn't matter 18:46:25 dolphm_, there are a million things that can be extended. If your objection is that we need to support a specific version of HTML and valirdate that, please provide that as feedback for the BP. 18:46:43 jamielennox: ? 18:46:46 jamielennox: I'm not sure how it works... would be a waste to have to convert it back to json! 18:47:18 bknudson, agreeds, and we force that to be the case now 18:47:19 bknudson: this is mainly going from the guy who was trying to do email validation in keystoneclient with jsonschema 18:47:25 bknudson, kesytone internal operate on json only 18:47:29 everything goes content -> JSON -> python 18:47:42 gyee, not really. It operates on Python 18:47:47 bknudson: so i'm not sure either, but i think it does 18:47:53 just that we force tit to go through JSON body first 18:48:12 its not necessary 18:48:20 maybe it's in the JSON body middleware 18:48:22 ayoung, its a dict anyway you slice it :) 18:48:36 maybe validation is in the JSON body middleware 18:48:41 dolphm_: i mean that i think jsonschema looks at the contents of a python dictionary, not the raw json, so the JSON and XML middlewares should produce the same thing and so jsonschema should handle both 18:48:46 seems like it should be in controllers 18:49:10 bknudson: the problem is that our wsgi layer mangles the input to our controllers 18:49:36 so if i have input { "a": 1, "b": 2} it will come through to the controller as def method(self, a, b): 18:49:36 dolphm_,gyee and ayoung: any agreement on https://review.openstack.org/#/c/37141 yet? 18:49:57 howdy, guys, sorry for interrupting you... I just wanna make a note to draw your attention to quotas: the work on them has been resumed, the deficiencies in the design have been addressed: 18:50:04 #link https://review.openstack.org/#/c/37545/ 18:50:34 atiwari, it looked good when we discusssed it yesterday. I do want to raise the question of global vs service spevfifc roles being disjoint sets 18:51:31 or can we have an id for global IDM roles 18:51:32 ? 18:51:38 atiwari's API review specifies that, in order to link a role to a service, it must always be linked to that service. THus, we couldn't do, say, give a user admin, but only on glance, whereas another user gets admin on all services 18:52:02 are we OK with this approach? It seems somewhat arbitrary 18:52:22 inkerra_: nice, that spec is looking pretty good 18:52:23 role assignments are typically where we were linking to scope in the past 18:52:49 so a user gets arole on a project or on a domain. project/service seems like a reduction in scope that is reasonable 18:54:03 ayoung, it would be hard to answer questions like what are all the Glance roles out there without iterating through all the assignments 18:54:22 ayoung, how do you handle role collision in that case ? 18:54:52 suppose there are n number of service trying to have "admin" role 18:55:23 ok, so keystoneclient plug, can everyone have a look at the version discovery blueprint and review: https://review.openstack.org/#/c/38414/ let me know if you have any questions or concerns about the approach but something like this is needed 18:55:53 #link https://review.openstack.org/#/c/38414/ 18:56:13 dolphm_, tnx, welcome to review :) 18:56:13 some eyes on the session extraction review would be appreciated too 18:56:17 #link https://review.openstack.org/#/c/43829/ 18:56:39 jamielennox, just to be clear, discover is optional, right? If I specify /v2.0 on the URL it won't force me through discovery, right? 18:57:17 atiwari, there is no collision. The role name is admin. If you have admin on glance, you can do admin tasks. 18:57:18 if you call the discovery factory you will always hit the server, if you specify version=2 to the factory you will always get a v2 client 18:57:34 it seems to really not work now to leave off /v2.0 in the service catalog 18:57:39 dolphm_, can we add a session for fixing catalog in the upcoming summit if we haven't created one already? 18:57:45 I can submit a bp if you like 18:57:51 if you specify the url with /v2.0 it will hit the server and make sure that /v2.0 is supported at that endpoint 18:57:54 in short, pretty much need to use v2.0 still :( 18:58:10 bknudson, that is part of what he is trying to fix 18:58:10 gyee: bp or summit discussion? 18:58:21 dolphm_, don't we need both? 18:58:24 bknudson: right, crusade for Icehouse is start moving people to v3 api and we need a discovery mechanism to do it 18:58:32 I think it will have an impact on version discovery 18:58:33 gyee: i mean, is there not already a bp? 18:58:44 jamielennox, one round trip. If I go directly to a web page that doesn't exist, either 404 or redirect. The Client should act the same way. 18:58:45 discovery in general 18:58:51 jamielennox: +++ 18:59:05 ayoung, that means keystone is going define all the roles even for future service? 18:59:39 gyee: yea, the service catalog bugs me, but we need to have discovery in place otherwise when we point people to the root url they won't be able to use it 18:59:51 Now, if I do discovery, the client should be able to remember it for a given service, at least for a limited time. I'd hate to have to do multiple round trips for each operation, especially from auth_token middleware 19:00:21 ayoung: the client does nothing, if you know what you need you can instantiate a client and it will just be (as now), discovery will do one round trip 19:00:37 atiwari, no, it means that the services might potentially both require the same role. 19:00:43 dolphm_, I see dns catalog and service relationship bps 19:01:07 ayoung: gee i would love to have like a session object where i could cache these discovery requests then for future: https://review.openstack.org/#/c/43829/ 19:01:09 a young, do we use cache control headers? 19:01:26 jamielennox, right, but if auth_token does discovery, it should remember it, if only for a limitied time...but that won't work too well in the prefork model anyway 19:01:26 dstanek_, we do not at the moment 19:01:37 dstanek_, i believe we should. 19:01:53 #endmeeting