18:02:46 #startmeeting Keystone 18:02:47 startmeeting? 18:02:48 Meeting started Tue Mar 17 18:02:46 2015 UTC and is due to finish in 60 minutes. The chair is morganfainberg. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:49 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:51 The meeting name has been set to 'keystone' 18:03:03 gyee, so i like to type extra spaces >> 18:03:07 ha 18:03:13 Ok, lets just get to it. 18:03:25 K3 this is this week. Please review the last fernet token patches. 18:03:37 #topic DB2 CI 18:03:45 that was fast. 18:03:54 bknudson, o/ (since yanfengxi is not in channel) 18:04:09 yanfenxi is in beijing and wasn't able to stay awake. 18:04:17 understandable 18:04:19 so he asked me to cover this... 18:04:28 2:04 AM 18:04:32 in Beijing 18:04:34 so the DB2 CI was disabled a while ago. 18:04:45 why? 18:04:54 i am still awake, 2:04 here. :) 18:04:56 it was disabled because it was reporting merge failures 18:04:59 o/ 18:05:03 and nobody was around who could fix it. 18:05:07 I don't have access. 18:05:18 ayoung, noise, consistently broken, and was not presenting logs for 30 days, and no one was found to fix it 18:05:34 still in training, but i'll try to pay attention here too 18:05:41 who owns it? We can get you access 18:06:11 so the request is to re-enable it, just so that it can get notifications so we can verify it works, 18:06:17 it won't be posting reviews 18:06:28 was it a voting gate before? 18:06:30 when we think it's ready to post reviews we'll come ask fo rthat. 18:06:38 gyee, no. 18:06:51 bknudson, ok so you want to move it back to sandbox capable 18:07:30 bknudson, i'm ok with that and will ask -infra, as long as it isn't posting anything until we're sure it is working. 18:07:31 morganfainberg: right. 18:07:53 how much time does db2 add to jenkins? 18:07:57 morganfainberg: ok, thanks. 18:08:09 bknudson, before it posts stuff again, we need to have a clear escalation when it's broken, it needs to retain logs for 30 days. 18:08:18 bknudson: is that DB2 CI running on VM provided by Rackspace,HP and others? 18:08:26 gyee: obviously it runs on its own, and it's hard to compare since we've got our own system... 18:08:32 bknudson, preferably someone looking at it, vs. keystone having to ask them why it's broken. 18:08:40 I vote yes...any additional data points on our DB side are valuable 18:08:46 marekd: we've got our own systems. 18:09:01 one of the issues is that we didn't have enough systems and we've added more. 18:09:14 bknudson: so, all VM's simply configure keystone to point to that remote backend somewhere else, ok 18:09:31 I believe these vms have their own db2 instance. 18:09:32 ok i'll ask -infra to re-enable as long as it isn't posting anything. 18:09:40 until other issues are addressed. 18:09:42 bknudson, that's fine so as long as it doesn't slow down jenkins considerably overall 18:09:56 gyee, it's 3rd party CI. 18:10:20 it actually doesn't run all of the tests, only ones marked for keystone. 18:10:44 gyee, so it should run mostly faster than the rest of the gate based on a smaller subset of tests. 18:11:00 gyee, and it wont blockup the actual gate pipeline (3rd party ci is check-only) 18:11:03 Can we all agree yes and move on? I think this is a non issue 18:11:05 morganfainberg, sure lets reenable then 18:11:17 ayoung: yes 18:11:17 thanks! 18:11:24 coo 18:11:27 I'll let yanfengxi know. 18:11:42 bknudson, he'll need to fix failures at 2am :) 18:11:56 he needs to give perms to bknudson to do it 18:12:02 #info DB2 CI will be requested to be re-enabled but will not post until other issues (failures and 30 day log retention) have been addressed. 18:12:08 #topic Feature Freeze Exceptions 18:12:09 ++ 18:12:31 since kilo3 is this week.. we have a couple FFEs that will be requested. 18:12:34 how much work is left for Reseller? 18:12:45 I've spoken to henrynash to sponsor the two major FFEs 18:12:53 the first FFE is reseller. 18:13:00 ayoung, we need to finish just one more patch (bye bye domain table) 18:13:15 yep, happy to sponsor 18:13:15 rodrigods, raildo, please send a message to the -dev mailing list requesting the FFE. 18:13:24 raildo, and how close is that...link? 18:13:29 morganfainberg, ok 18:13:31 a moment of silence for domains 18:13:33 please say henrynash will sponsor, link to the code, and show that it's mostly reviewing. 18:13:39 morganfainberg: http://paste.openstack.org/show/192951/ this is also after couple iterations of reviews. and since we were blocked by a bug we couldn't merge earlier: https://review.openstack.org/#/c/152156/ 18:13:39 ayoung, https://review.openstack.org/#/c/161854/ 18:13:48 morganfainberg, great, thanks 18:13:58 why WIP? 18:14:18 ayoung,because we don't finish the implementation yet 18:14:25 OK....how much work? 18:14:29 The second FFE, is the rest of henrynash's domainSQL, henrynash please send the FFE request to the -dev mailing list. I know it's just the final couple patches. 18:14:46 morganfainberg: will do 18:14:48 adn we have a problem when we're trying drop the table 18:15:04 marekd, you or marco are welcome to send an FFE request for those. 18:15:05 raildo, ok, hit me up after the meeting if you need help 18:15:13 morganfainberg: ok 18:15:13 marekd, s/those/that 18:15:16 that should be relatively simple to knock out 18:15:17 ayoung, ok, thanks :) 18:15:34 or impossible...we'll see. 18:15:39 when sending to the dev mailing list be sure to use [keystone] and include FFE request in the subject 18:15:44 be descriptive 18:16:06 a few other BPs have been moved to post kilo3 since they are non-API impacting (and not really "Features") 18:16:07 morganfainberg, ok 18:16:20 morganfainberg: ack. 18:16:42 please respond to the ML topics (everyone) for/against/concerns once tehy are sent. 18:16:43 raildo, do it as two migrations. One for the FK, one for the table drop 18:16:50 will make it easier to sort out/ 18:16:51 bknduson, stevemar, lbragstad: there are two domain config ones that we can probably get in ahead of k3 since they have been extensively reviewed: https://review.openstack.org/#/c/159928/ and https://review.openstack.org/#/c/165075/ 18:17:32 ayoung, the FK part is working, but when we try drop the table, we got a error. i'll talk with you later :) 18:17:35 the ML topic will be used to either grant the exception or push to liberty. Realize it is not exclusively the keystone team involved in granting the FFE, i will be conferring with ttx and release management as well. 18:17:50 are all these reviews on the https://gist.github.com/dolph/651c6a1748f69637abd0 page? 18:18:06 bknudson, no. those reviews are just for k3 now [except the domain SQL one] 18:18:09 that's where I've been going to find reviews. 18:18:16 bknudson, since someone else has starred it 18:18:38 should we put them on the review page? 18:18:42 I can star them 18:18:42 bknudson, so basically the fernet tokens only 18:18:46 morganfainberg: dolphm bknudson : https://review.openstack.org/#/c/152156/ is not 18:18:46 lbragstad, no. 18:18:49 lbragstad, wait till the FFEs are granted 18:18:54 morgainfainberg: remind me, keystone client library additions don’t need an FFE, right? 18:18:55 morganfainberg: ok 18:19:03 henrynash, nope 18:19:15 henrynash, client and middleware and pycadf have no FFEs needed 18:19:17 at some point we're going to release a keystoneclient, so they have to be in before that. 18:19:21 ok 18:19:32 there will be a release of client right before RC 18:19:37 yeah...let's queue up the client questions for later 18:19:40 bknduson: indeed 18:19:45 I have a few... 18:19:48 that client release will coincide with the release (final tagging) 18:20:02 but for today, the Fernet token patches are the priority 18:20:02 scary. 18:20:21 btw, this means that bp https://blueprints.launchpad.net/openstack/?searchtext=auth-token-use-client isn't going to be done for K. 18:20:24 morganfainberg, remove " Group role revocation invalidates all user tokens" 18:20:25 #info Fernet Token Final Reviews are priority for k3 18:20:28 we are not going to address that 18:20:37 since it requires changes to auth_token middleware that depend on client changes. 18:20:38 ayoung, i can't someone else has it starred 18:20:55 ayoung, it's not exclusively my list. 18:21:20 Clone it 18:21:22 Its git! 18:21:34 ayoung, i don't have the bot that updates it. 18:21:39 ayoung, that's dolphs code. 18:21:45 and i haven't seen it. 18:21:49 I know, but for the k3 list on launchpad it should be gone 18:22:07 https://launchpad.net/keystone/+milestone/kilo-3 18:22:25 ayoung, it was tagging 2 bugs, i missed one 18:22:31 cool 18:22:33 ok 18:22:50 #topic requirements.txt: Should requirements.txt be for the expected installation 18:22:57 #undo 18:22:57 wait 18:22:57 Removing item from minutes: 18:22:59 skipped one 18:23:04 https://blueprints.launchpad.net/keystone/+spec/mapping-enhancements 18:23:15 ayoung, that is a patch dstanek needed to update a comment on 18:23:16 that is on the k3 list, and tagged as needs code review 18:23:27 other than that it is good>? 18:23:28 it's done, just a test enhancement that could land now/post k3 18:23:29 yes 18:23:31 ayoung, i think we can close that one out now 18:23:47 wasn't going to close it till the last patch was landed but that is minimal/doesn't need massive eyes atm. 18:24:12 OK... 18:24:20 #topic requirements.txt: Should requirements.txt be for the expected installation? 18:24:27 bknudson, o/ 18:24:32 ah, so I posted a review... 18:24:37 that got -1d 18:24:41 link 18:25:04 #link https://review.openstack.org/#/c/162360/ 18:25:06 bknudson, you were finally on the receiving end 18:25:12 y, that was it. 18:25:24 essentially moving fernet requirements to test-requirements.txt 18:25:39 since my understanding was that we only have default config requirements in requirements.txt 18:25:55 Do we have mysql in requirements? 18:25:58 my view is test-requirements should only be for testing. 18:26:16 now, if we change that policy ... maybe we should also have ldap and ldappool in requirements.txt? 18:26:19 anything that could be used for runtime should be in main requirements.txt 18:26:25 bknudson, i would support that 18:26:28 bknudson, we probably should 18:26:36 bknudson +++ 18:26:57 vote on it? 18:27:04 #vote 18:27:09 damn enter key. 18:27:09 so why we even need test-requirements.txt then? 18:27:18 there will still be test-only requirements... 18:27:19 since now everything will be enabled by default 18:27:19 testtools 18:27:20 * topol yay a vote. lets see who is paying attention 18:27:23 gyee, yes there are test-only things 18:27:27 mock 18:27:28 How about different providers for dogpile? where does it fit? 18:27:32 fixtures 18:27:51 the documentation stuff will be there too 18:28:00 we don't even have mySQL in test-requirements 18:28:08 haneef, we would be adding any runtim dep - the only special case is liky mongo for licensing reasons 18:28:17 haneef, and that is a special case across openstack 18:28:19 using test-req for 'optional' isn't right 18:28:19 I don't think mysql is on pypi? 18:28:26 b/c oracle. 18:28:39 bknudson, mysqldb is, but i think we get that another way 18:28:55 So...on the client side, we are splitting things into separate repos due to dependencies. This is where the python packaging is dumb 18:29:26 i always thought that test-requirements was actually for the optional stuff 18:29:37 if we have a plugin/extension/driver that needs a special piece of code, we should be able to package that separately without having to split the repo 18:29:39 dstanek, that's what we kept telling people :P 18:29:52 well, it is for installing things we need to run tests 18:30:05 so it has the optional flavor, since you can do a minimal install without it 18:30:05 whaat? test-requirements is for test. come on man! 18:30:10 #startvote Should all runtime requirements (that don't have specific licensing concerns) move from test-requirements.txt to requirements.txt (test-requirements no longer used for "optional")? yes,no,obligatory-yes-but-i-want-to-be-a-snowflake-answer 18:30:11 Begin voting on: Should all runtime requirements (that don't have specific licensing concerns) move from test-requirements.txt to requirements.txt (test-requirements no longer used for "optional")? Valid vote options are yes, no, obligatory-yes-but-i-want-to-be-a-snowflake-answer. 18:30:12 Vote using '#vote OPTION'. Only your last vote counts. 18:30:24 dstanek, is python really going to continue to be so stiffnecked about this? 18:30:29 #vote no 18:30:33 #vote yes 18:30:38 #vote yes 18:30:43 #vote yes 18:30:45 #vote yes 18:30:49 It's not licensing, it is dependencies on install 18:30:59 #vote yes 18:30:59 #vote no 18:31:02 who will be the original one? 18:31:06 #vote yes 18:31:08 #vote no 18:31:10 #vote no 18:31:11 voting no just because this is how we've always worked so why change it. 18:31:11 ayoung, magically knowing what to install to turn something on is a terrible experience 18:31:12 but torn 18:31:20 #vote no 18:31:23 ayoung, it doesn't prevent redhat from bundling deps differently 18:31:29 morganfainberg, having to find the LDAP deps if you are not running LDAP is also a bad one 18:31:32 or postgresql 18:31:35 DB2? 18:31:36 #vote no 18:31:41 has anyone talked to distributions about this? 18:31:44 morganfainberg: it expects (for example redhat) package maintainer to go and find all this for themselves 18:31:46 #showvote 18:31:47 yes (6): gyee, morganfainberg, marekd, samueldmq, topol, breton 18:31:48 no (6): dstanek, ayoung, bknudson, haneef, dolphm, jamielennox 18:31:59 evenly spread 18:32:03 test-requirements is the wrong place for "optional" python dependencies 18:32:03 ayoung: this isn't really a python thing 18:32:04 hmmm... 18:32:11 so ayoung brings up a valid consumability point 18:32:11 dstanek, distributions are going to have to update how they idenityf what to install with what...it will likely break their scripts\ 18:32:16 #vote yes 18:32:26 dstanek, yes it is, cuz I can only deploy one package from one repo 18:32:29 stevemar: the tiebreaker. 18:32:32 #vote yes 18:32:33 and that is due to python, not git, and not packaging 18:32:45 dstanek, packagers have told me both sides, the one that got me was "i can't make X a dependency in my package cause it's in test-requirements, move it to requirements or i can't support it because $policy" 18:32:48 we should not put something in requirements if its not always required and can easily break stuff 18:33:00 morganfainberg: that's a broken $policy 18:33:04 topol, how would it break stuff? 18:33:10 topol: changing your vote? 18:33:11 can we hold off on this one until we get feedback please? 18:33:13 ayoung: normally you would use the extras_require to say 'hey, i'm using ldap and here are the extra requirements' 18:33:15 dolphm, putting things in test-requirements because they are optional is worse imo 18:33:18 #vote no 18:33:19 topol: if it's in the main tree and covered with tests, it's working 18:33:26 #vote no 18:33:30 dolphm, that is very much the wrong place for these things 18:33:31 breton, ++ 18:33:32 for the reasons bknudson listed 18:33:34 ayoung: you know someone who can provide feedback on your side? 18:33:40 bknudson, yes 18:33:42 apevec 18:33:44 ayoung can you elaborate on your ldap concern 18:33:45 this includes cryptography, msgpack, ldap, ldappool, etc 18:33:50 morganfainberg: is there a broader community concern on that? 18:33:54 topol, LDAP pulls in a native library 18:34:06 morganfainberg: isn't this a TC question anyway? 18:34:09 morganfainberg: (i wouldn't want to see keystone deviate from the community consensus on usage of test-requirements) 18:34:14 jamielennox, not really. 18:34:16 ayoung and if the library isnt there? 18:34:17 so by putting LDAP in the python tree, you pull in pythn-ldaop, which pulls in openldap libs 18:34:25 morganfainberg have an excellent point on *user experience* 18:34:27 dolphm, i think we're flat out doing it wrong in python. 18:34:40 topol, I think with PIP, it is even worse 18:34:44 #showvote 18:34:44 it would suck if I enabled a feature and then found out I need to explicitly install more packages 18:34:45 yes (8): gyee, morganfainberg, marekd, samueldmq, topol, raildo, breton, stevemar 18:34:47 no (8): dstanek, ayoung, lhcheng, bknudson, haneef, lbragstad, dolphm, jamielennox 18:34:56 how contentious 18:34:57 gyee: ++ 18:34:57 for MySQL and LDAP you have to have the devel version of the native libs installed to do the native bindings 18:34:59 I vote to put it off, let's have an action to hear back from packagers. 18:35:01 gyee makes a good point 18:35:10 dolphm, this isn't openstack community this is doing it wrong in python world. if we want a separate dependency tree, we should be splitting the stuff up into separate python packages 18:35:13 I agree with bknudson 18:35:19 gyee, that's always been the case 18:35:20 #vote no 18:35:29 stevemar, then lets make it less suck 18:35:29 gyee: some of those packages might require more binary deps - which sucks in python land imo 18:35:35 gyee: that mean you always have to install everything? what about things that have a dep on a system package? 18:35:38 as far as our packaging goes, I don't think it will cause a problem either way. 18:35:40 gyee, thats why i voted yes 18:35:46 gyee: it would also suck if my installation is stuck due to a dependency which I won't be using it 18:35:50 morganfainberg: that'd be fantastic 18:35:52 most of us are shipping a distro 18:36:01 dolphm, i have $ideas for liberty ;) 18:36:03 we split the python-keystoneclient repo for just this reason 18:36:05 henrynash, what sis your concern? 18:36:05 morganfainberg: test-requirements.txt was invented by the openstack community 18:36:05 a distro, not distro and a bunch of optional deps 18:36:21 the client is relatively consistant, but we pushed kerberos and SAML codes to separate repos 18:36:26 #vote no 18:36:35 Im scared we will rush this decision and really break folks 18:36:59 Let's float it to the -dev list and get some wider feedback, and revisit 18:37:03 topol, it wont really break anyone shipping a distro, and anyone doing CI/CD already has addressed most of this. 18:37:10 but in short. we'll defer this till liberty 18:37:13 ++ayoung 18:37:14 * topol trying to my yes back in the bottle 18:37:14 there are some discussion already going on in community. 18:37:16 as a user, I would expect a distro contains everything that I need 18:37:17 based on this. vote 18:37:22 topol: I’m worried about a) deviating from openstack standard, and b) loading stuff that’s not strictly needed into production systems (less you install the better for secuirty) 18:37:27 we'll do further talking at the summit and on the list 18:37:35 bknudson, i am going to -2 that patch though for now. 18:37:54 wait, my patch was about following the current policy... 18:37:58 not deviating. 18:38:04 henrynash, make sense 18:38:13 makes sense :-) 18:38:20 #showvote 18:38:21 yes (8): gyee, morganfainberg, marekd, samueldmq, topol, raildo, breton, stevemar 18:38:22 no (10): dstanek, ayoung, lhcheng, bknudson, davechen_, haneef, lbragstad, dolphm, jamielennox, henrynash 18:38:23 bknudson, because i don't want to shuffle everything around right now. 18:38:24 (that s makes a world of difference) 18:38:30 #vote no 18:38:35 #endvote 18:38:36 Voted on "Should all runtime requirements (that don't have specific licensing concerns) move from test-requirements.txt to requirements.txt (test-requirements no longer used for "optional")?" Results are 18:38:38 yes (7): gyee, morganfainberg, marekd, samueldmq, raildo, breton, stevemar 18:38:39 no (11): dstanek, ayoung, lhcheng, bknudson, davechen_, haneef, lbragstad, dolphm, jamielennox, henrynash, topol 18:38:46 it's 2 lines. 18:38:51 bknudson, and if we're doing massive policy adherance lots of stuff needs to shuffle. 18:38:57 really? 18:39:07 bknudson, yes. we have lots of non-default things in there. 18:39:11 relating to PKI tokens as well 18:39:20 Oy Vey! 18:39:20 genesis in keystone? https://github.com/openstack/keystone/commit/72e4edcc12f4741bd945adf777fe43f3ffcd62d6 18:40:03 moving optional to test-requirements was an awful thing. it has caused a lot of questions and "why can't X work" with a response of "go install thing because we don't make it a hard requirement" 18:40:06 morganfainberg: i might change my vote to yes after seeing my own commit :P 18:40:27 wow things were simpler in 2011 18:40:31 morganfainberg, lets work with dstank to have a better modular install story 18:40:45 ayoung, liberty. 18:40:48 topol, 1969, nothing by peace and love 18:40:54 if we can get away from "separate git repo to avoid depednecy hell" I'll be happy 18:40:59 ok, so for K we're going to stick with fernet requirements in requirements.txt ? 18:41:07 ayoung, we wont be able to with pypi 18:41:09 morganfainberg, on the client first, and then Liberty for Server. 18:41:19 ayoung, and pbr. 18:41:20 morganfainberg, THEN WE BREAK THEM! 18:41:23 we can just make it a policy where we pick based on what we know. 18:41:24 WE BREAK THAT TOO 18:41:34 BREAK ALL THE THINGS! 18:41:48 rewrite everythin from scratch 18:41:49 bknudson, ok so let me change my stance 18:41:50 ayoung's ptl slogan 18:41:51 this is really a packager issue - 'apt-get install keystone keystone-ldap ... etc' - right? 18:41:54 please remember the user experience concerns 18:42:02 dstanek, pip install and apt-get install 18:42:17 jamielennox, life philosophy really. I was, and will always in my heart be, an Infantryman 18:42:33 there's a man who lived 18:42:35 bknudson, so my stance is: if it's pure python it can always go in requirements. if it has exceptional binary deps lets put it in test-requirements for now (ldap) 18:42:56 morganfainberg that makes sense 18:42:58 That is better 18:43:01 morganfainberg: pki & openssl? 18:43:05 let's write a pure-python ldap client. 18:43:09 dolphm, opensssl isn't exceptional. 18:43:14 morganfainberg, we shouldprobably get MySQL into test-requirements then 18:43:18 and xmlsec1 18:43:24 pki and openssl is even worse 18:43:36 remember lxml? 18:43:39 those can't be done by package requirements except at the distro level 18:43:44 if it has historically been in requirements DONT MOVE IT until we decide how we're handling the splitup/new install etc 18:43:45 yes that one too 18:43:50 this is for new requirements 18:43:57 morganfainberg: to me that is harder to draw the line - do i now have to know how projects are implemented? 18:43:58 so crypto and msgpack should be evaluated 18:44:08 we need to get the cryptography.py solution in place for PKI etc. 18:44:10 cryptography must rely on openssl. 18:44:12 But not today 18:44:19 dstanek, I am on the user experience side 18:44:23 * dolphm finds it best to refer to the library as pypi/cryptography to avoid confusion 18:44:32 dstanek, try and install it, is it awful to pip install it? 18:44:41 bknudson: it does not, afaik 18:44:51 dstanek did it require a bunch of extra apt-gets (e.g. ldap libs dev) 18:44:56 ayoung: There was a review last week to replace openssl system calls with a crypto lib 18:45:09 morganfainberg: things will go fine if i have the required system packages already so i may not think twice 18:45:25 dstanek, since we aren't adding more g-rs this cycle, this is largely deferred 18:45:31 dstanek, the only question is msgpack and crypto 18:45:33 haneef, link? 18:45:47 morganfainberg: fair enough 18:45:51 btw - we need a newer cryptography, we're using features that aren't in 0.4 18:45:54 dstanek, so, for this cycle address those two and in liberty we work on better stuff. 18:45:56 https://review.openstack.org/#/c/163088/ 18:46:00 bknudson, yes there should be a g-r update 18:46:10 browne, yay! 18:46:18 browne, thats cool. 18:46:36 that must rely on openssl? 18:46:40 Ah...cool. that is for the setup. Need to -2 that one 18:46:42 ayoung: https://review.openstack.org/#/c/163088/ 18:46:46 Its right, but not right enopguh 18:46:49 browne, Really? we can do that now? Thats awesome! 18:46:51 wow, at long last! 18:46:53 browne: does it also spawns new process? 18:46:59 morganfainberg: i believe alex gaynor implemented everything we needed to replace openssl while sitting at the keystone pod in atlanta 18:47:12 marekd: no 18:47:18 bknudson, https://review.openstack.org/#/c/164289/ 18:47:23 marekd: no, library calls. this patch only replace rsa key generation 18:47:25 pypi/cryptography implements it's own primitives 18:47:32