Thursday, 2018-08-09

*** gyee has quit IRC00:00
*** r-daneel has quit IRC00:10
*** zhurong has joined #openstack-keystone00:13
*** elibrokeit has quit IRC00:17
*** elibrokeit has joined #openstack-keystone00:32
*** gagehugo_ has quit IRC01:07
*** gagehugo has joined #openstack-keystone01:22
*** zhurong has quit IRC01:47
*** zhurong has joined #openstack-keystone02:34
*** dklyle has joined #openstack-keystone02:46
*** dklyle has quit IRC03:42
*** zhurong has quit IRC03:53
*** _ix has quit IRC03:54
openstackgerritVishakha Agarwal proposed openstack/keystone master: Implement Trust Flush via keystone-manage.  https://review.openstack.org/58937804:37
*** markvoelker has joined #openstack-keystone04:41
*** shyambiradar has joined #openstack-keystone05:21
*** nicolasbock has joined #openstack-keystone05:34
*** shyambiradar has quit IRC05:53
openstackgerritOpenStack Proposal Bot proposed openstack/keystone master: Imported Translations from Zanata  https://review.openstack.org/59011106:07
*** Emine has joined #openstack-keystone06:28
*** nicolasbock has quit IRC06:35
*** pcaruana has joined #openstack-keystone06:38
*** nicolasbock has joined #openstack-keystone06:41
*** evrardjp has joined #openstack-keystone06:55
mbuillbragstad, cmurphy: what tool you use to generate a PKI key pair?06:56
cmurphymbuil: for the saml metadata? I just use the openssl CLI07:00
*** ispp has joined #openstack-keystone07:06
openstackgerritzhengliuyang proposed openstack/keystone master: More accurate explanation in api-ref:application credentials  https://review.openstack.org/58913507:07
*** shyambiradar has joined #openstack-keystone07:11
mbuilcmurphy: yes. I am not familiar with key generation. I used "openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:2048" to generate the private key and "openssl rsa -pubout -in private_key.pem -out public_key.pem" to generate the public key. Is certfile the public key and keyfile the private key?07:12
cmurphymbuil: yep07:12
mbuilcmurphy: thanks07:13
cmurphyalso i like to use this oneliner https://stackoverflow.com/a/1017668507:13
cmurphywith -nodes07:14
*** rcernin has quit IRC07:16
mbuilcmurphy: this generates a certificate which I should add as certfile?07:16
cmurphymbuil: it generates the certfile and keyfile in one go07:17
mbuiloh that's good! If I remember well, a certificate was a public key signed by a private key. I was wondering if with the commands I wrote you, was enough because I did not generate any certificate but a public and a private key07:18
openstackgerritOpenStack Proposal Bot proposed openstack/oslo.policy master: Imported Translations from Zanata  https://review.openstack.org/59014507:19
mbuilcmurphy: keystone-manage does not exist. What pip package should I install? I have keystoneauth1==3.4.0 and python-keystoneclient==3.15.007:24
mbuiloh wait07:25
cmurphymbuil: keystone-manage comes from the server package, you'll need to install keystone itself07:25
mbuilcmurphy: I forgot things are installed in venvs, sorry07:25
mbuilI see it now :)07:25
*** shyambiradar has quit IRC07:38
*** shyambiradar has joined #openstack-keystone07:39
mbuilcmurphy: I need a bit of help. In my deployment, I had Keystone working under nginx. I have been following this guide to change to Apache ==> https://docs.openstack.org/keystone/latest/install/keystone-install-obs.html#configure-the-apache-http-server but I guess it assumes that there is a clean environment. I stopped nginx and started apache successfully but when using the openstack cli I get problems07:57
mbuilthis is what I get ==> Failed to discover available identity versions when contacting http://192.168.122.3:5000/v3. Attempting to parse version from URL. Service Unavailable (HTTP 503). I guess I am missing some config in the apache part07:58
cmurphymbuil: is there anything in the apache logs or the keystone logs that would indicate why it's returning a 503?07:59
mbuilcmurphy: in apache2 logs ==> client denied by server configuration:.... let me investigate this. thanks08:07
*** _ix has joined #openstack-keystone08:11
*** shyambiradar has quit IRC08:11
*** aojea has joined #openstack-keystone08:22
*** shyambiradar has joined #openstack-keystone08:26
*** dtantsur|afk is now known as dtantsur08:33
mbuilcmurphy: I am stuck again and still in the nginx -> apache step :(. When trying 'openstack network list' things are ok, until it does a GET call to .../v2.0/networks, where it gets a HttpException. This are the logs in the client: https://hastebin.com/efalosiseg.sql08:48
mbuilcmurphy: and this is the error I see in neutron logs ==> https://hastebin.com/suvivaxobi.rb "DiscoveryFailure: Could not determine a suitable URL for the plugin"08:49
mbuilI am using the same openrc as before... do I need to change anything there?08:49
cmurphymbuil: that looks like an issue with neutron to me, you can see on the lines before the failure that it was successful in talking to keystone08:52
cmurphyso the openrc should be fine, seems like a server error with neutron08:53
cmurphyoh sorry08:53
cmurphyyou are looking at the neutron logs08:53
cmurphyso i'd check the keystone_authtoken section in neutron.conf08:54
*** aojea has quit IRC08:54
cmurphyand make sure it is pointing to the right keystone endpoint and also make sure it has a user_domain_name and project_domain_name set to 'Default'08:54
mbuilcmurphy: I also get problems when listing images and glance logs give a bit more info ==> https://hastebin.com/jowigoxica.js08:56
mbuillet me check that08:57
cmurphymbuil: is your keystone listening on port 35357? we changed most of our docs to stop using that port and only use port 500008:57
mbuilcmurphy: good point. I have just realized that nginx was listening to 5000, 80 and 35357 and apache is listening to 5000 and 8009:04
*** shyambiradar has quit IRC09:06
*** ispp has quit IRC09:10
mbuilcmurphy: when using nginx I had a .../conf.d/keystone-wsgi-public.conf and a .../conf.d/keystone-wsgi-admin.conf. Now I only have .../conf.d/wsgi-keystone.conf and the content points to keystone-wsgi-public. Is that enough or should there be a config for keystone-wsgi-admin too?09:12
*** josecastroleon has quit IRC09:14
*** shyambiradar has joined #openstack-keystone09:15
cmurphymbuil: that's enough, the two different endpoints are legacy from the keystone v2 API which used different access control for each endpoint, for keystone v3 all of the access control is done in code with policy and it can all go through the one endpoint09:17
*** josecastroleon has joined #openstack-keystone09:20
*** shyambiradar has quit IRC09:38
mbuilcmurphy: I am at the step "Configure Apache to use a federation capable authentication method". Any preference between Shibboleth and Mellon? Remember I am planning to do Keystone to Keystone (perhaps that limits the option to one)09:50
cmurphymbuil: the last time I tried, mellon didn't work properly in the keystone to keystone scenario, there is some bug either in mellon or in keystone that made it unable to parse the saml response properly and i never got to the bottom of it09:52
cmurphyshibboleth is a safe bet even though it's annoying to configure09:52
mbuilcmurphy all right! thanks09:53
*** Emine has quit IRC10:04
*** Emine has joined #openstack-keystone10:04
mbuilcmurphy: regarding shibboleth config, I am following: https://docs.openstack.org/keystone/latest/advanced-topics/federation/shibboleth.html. I am about to add the "<Location " config but I am not sure what should I write there10:20
mbuilcmurphy: first question, should I install and configure Shibboleth in both deployments or only in the one acting as SP?10:21
*** shyambiradar has joined #openstack-keystone10:28
cmurphymbuil: only on the SP10:34
cmurphymbuil: the <Location /Shibboleth.sso> you can copy verbatim, for the <Location /v3/OS-FERATION/...> you can also copy it verbatim but the name of the identity provider and protocol, which is 'myidp', and 'saml2' in the example, is important and will come up later in the documentation10:36
cmurphyyou should probably keep the name of the protocol as 'saml2' but you might want to change the name of the identity provider10:36
mbuilcmurphy: ah! ok, I was wondering whether I should change myidp with what I wrote in the IdP as "idp_entity_id". Let me read further then10:37
mbuilcmurphy: BTW, there was a shib.conf that appeared in the conf.d/ directory right after installing Shibboleth. Should I leave it there?10:50
cmurphymbuil: yes10:50
mbuilcmurphy: ok. I need to stop here and focus on something different. I'll try to progress tomorrow. Thanks for the help! :)10:51
cmurphyno problem :)10:52
*** neiljerram has joined #openstack-keystone10:54
*** shyambiradar has quit IRC11:07
*** shyambiradar has joined #openstack-keystone11:10
*** aojea has joined #openstack-keystone11:20
*** shyambiradar has quit IRC11:24
*** shyambiradar has joined #openstack-keystone11:25
*** shyambiradar has quit IRC11:32
*** aojea has quit IRC11:33
*** aojea has joined #openstack-keystone11:45
*** s10 has joined #openstack-keystone11:52
*** shyambiradar has joined #openstack-keystone11:56
*** aojea has quit IRC11:58
*** raildo has joined #openstack-keystone12:19
*** edmondsw has joined #openstack-keystone12:45
*** lbragstad has joined #openstack-keystone12:45
*** ChanServ sets mode: +o lbragstad12:45
*** _ix has quit IRC12:52
lbragstadildikov: is there a specific zoom link floating around? or should we use the same one as before?12:53
ildikovlbragstad: https://zoom.us/j/67123614812:53
ildikovlbragstad: all the relevant info is here: https://wiki.openstack.org/wiki/Edge_Computing_Group#Keystone12:54
*** shyambiradar has quit IRC12:55
*** josecastroleon has quit IRC12:57
*** josecastroleon has joined #openstack-keystone12:57
*** _ix has joined #openstack-keystone13:00
*** josecastroleon has quit IRC13:03
*** josecastroleon has joined #openstack-keystone13:04
knikollajoining, now, sorry i'm late.13:12
lbragstadzzzeek: about your galera work, do you know if there was ever a spec pushed for that? i remember you brought it to a meeting once and the next steps were to document the approach a bit13:28
zzzeeklbragstad: the spec I was working on is at https://review.openstack.org/#/c/566448/13:28
lbragstadoh - great13:29
lbragstadthanks!13:29
*** edmondsw has quit IRC13:29
zzzeeklbragstad: current POC is at https://github.com/zzzeek/stretch_cluster13:29
*** _ix has quit IRC13:29
*** edmondsw has joined #openstack-keystone13:37
*** jistr is now known as jistr|call13:39
ildikovknikolla: no worries, Tnx for joining13:41
*** edmondsw has quit IRC13:41
*** edmondsw has joined #openstack-keystone13:44
*** jistr|call is now known as jistr13:59
*** _ix has joined #openstack-keystone14:03
*** josecastroleon has quit IRC14:08
*** josecastroleon has joined #openstack-keystone14:19
kmalloco/14:24
lbragstadmornin'14:26
gagehugoo/14:41
*** dtantsur is now known as dtantsur|brb14:43
*** aojea has joined #openstack-keystone14:55
lbragstadFYI - https://review.openstack.org/58995015:02
lbragstadi've proposed rc115:02
lbragstaddepending on the state of porting various APIs we can assess if we want an RC2 next week15:02
*** aojea has quit IRC15:06
*** aojea has joined #openstack-keystone15:07
lbragstadalso - i'll be traveling tomorrow and unavailable15:07
lbragstadif anything urgent comes up i should be available saturday-ish?15:08
lbragstadbut at that point i'll be on opposite timezones15:08
*** s10 has quit IRC15:18
*** dtantsur|brb is now known as dtantsur15:29
*** dklyle has joined #openstack-keystone15:39
openstackgerritOpenStack Release Bot proposed openstack/keystone master: Update reno for stable/rocky  https://review.openstack.org/59040515:49
*** dtantsur has left #openstack-keystone15:56
*** dklyle has quit IRC15:59
*** dklyle has joined #openstack-keystone16:01
kmallocweird, i am getting a failure on policy for "get_domain_role" and i don't see why16:12
kmallocoh... i see what is happening.16:14
kmallocmissing target data because of magic stuff.16:14
kmallocgot it16:14
lbragstadi added cycle-highlights for keystone https://review.openstack.org/#/c/590411/16:15
lbragstadwould love feedback there if anyone has any16:15
gagehugolbragstad: done16:24
gagehugomy main concern is the whole case-insensitive issue we had with "Member" vs "member" when we merged the default roles16:24
gagehugothat may affect people16:24
lbragstadgagehugo: done16:27
lbragstadgagehugo: where did we put that case-sensitity statement in docs?16:28
gagehugohttps://review.openstack.org/#/c/576640/16:29
lbragstadhttps://docs.openstack.org/keystone/latest/admin/identity-case-insensitive.html ah16:29
gagehugoyeah16:29
*** s10 has joined #openstack-keystone16:38
*** aojea has quit IRC16:51
*** s10 has quit IRC16:57
*** itlinux has joined #openstack-keystone16:58
*** raildo has quit IRC17:07
*** raildo has joined #openstack-keystone17:31
*** raildo_ has joined #openstack-keystone17:35
*** raildo has quit IRC17:36
*** gyee has joined #openstack-keystone17:47
*** felipemonteiro has joined #openstack-keystone17:52
*** felipemonteiro has quit IRC18:30
kmalloclbragstad: interesting, role api and role_implication api is the first place i had to convert an enforcement callback. it went super easily18:32
kmalloclbragstad: fwiw, enforce_call made it almost painless.18:32
*** spilla has joined #openstack-keystone18:33
openstackgerritGage Hugo proposed openstack/keystone master: WIP - Add details and clarify examples on casing  https://review.openstack.org/59047718:35
*** ayoung has joined #openstack-keystone18:41
ayoungorange_julius, ever solve your problem?  Sounds like HA proxy is sending one request to each Keystone server for some reason.18:43
kmalloclbragstad: hm.19:01
kmalloclbragstad: we broke HTTP spec again in implied roles *sigh*19:01
kmalloclbragstad: we issue a NO_CONTENT for HEAD.19:01
kmalloc*eyeroll*19:01
kmallocchecking our docks.19:01
kmallocdocs*19:01
kmallocyep, we messed that one up =/19:21
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert Roles API to flask native dispatching  https://review.openstack.org/59049419:23
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert Roles API to flask native dispatching  https://review.openstack.org/59049519:23
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert Roles API to flask native dispatching  https://review.openstack.org/59049419:24
*** rmascena__ has joined #openstack-keystone19:29
*** raildo_ has quit IRC19:33
*** pcaruana has quit IRC19:33
*** aojea has joined #openstack-keystone19:47
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert role_inferences API to flask native dispatching  https://review.openstack.org/59050219:59
*** rmascena__ has quit IRC20:09
*** rmascena__ has joined #openstack-keystone20:09
*** nicolasbock has quit IRC20:11
*** rmascena__ has quit IRC20:16
openstackgerritGage Hugo proposed openstack/keystone master: Set initiator id as user_id for auth events  https://review.openstack.org/58808620:19
orange_juliusayoung: no I have not yet. Trying to replicate the issue. I can lock out a user by sending a POST with curl to a backend keystone node though.. so I don't think its haproxy20:28
orange_juliussince the curl never hits haproxy20:28
openstackgerritMerged openstack/keystone master: Imported Translations from Zanata  https://review.openstack.org/59011120:39
kmalloclbragstad: almost have role_assignments converted. the assignment subsystem is getting there.20:50
*** lbragstad has quit IRC20:51
ayoungorange_julius, so we do one bind as the use to authenticate, and the rest of the work is done as an admin user.  If there are multiple simple-bind calls from Keystone to LDAP, it should show up in the Keystone log.20:53
ayoungNeed to turn on tracing20:53
orange_juliusI can't turn that on in prod. Trying to replicate in test. Hopefully I'll have something soon20:57
ayoungorange_julius, I wonder if it is something fun like:  pooling mechanism treats a failure as a reason to retry.20:59
kmallocayoung: hm. that would be odd, but i could see ldappool having such an issue21:00
orange_juliusYup thats what the theory is. I have a bug report opened with ldappool21:00
orange_juliusJust need verification21:00
kmallocayoung: wonder if we're setting a RETRY value on the connection as well21:00
orange_juliusldappool catches ldap.LDAPError which is the superclass for all ldap errors. I'm pretty sure its happening there... line 251-268 of ldappool21:00
ayoungsimple-bind is dumb21:01
ayoungit really is an anti-pattern.  Share your password with every app....21:01
kmallochmm.. this looks suspect21:01
kmallochttps://github.com/openstack/ldappool/blob/master/ldappool/__init__.py#L9921:01
kmallocayoung, orange_julius: https://github.com/python-ldap/python-ldap/blob/f3ff4a320a21a3ac42bc7587eaed09c4f9b2f9f5/Lib/ldap/ldapobject.py#L1193-L120321:05
kmalloclooks to me like what is happening is the failure is being re-tried21:06
ayoungexcept ldap.SERVER_DOWN:21:06
kmalloclooks like LDAPpool needs a bunch more logic and we need to do connect independant of simple_bind, and ensure we don't retry on a legit simple bind error21:06
ayoungyou should not get that on an auth failure, tho21:06
kmallocunless the simple_bind failure to AD maybe drops the connections?21:07
kmallocin this case21:07
kmallocwe can't discount that as a possibility21:07
orange_juliusThis is what we had been taking a look at: https://git.openstack.org/cgit/openstack/ldappool/tree/ldappool/__init__.py#n255 Is this only to establish the pool then?21:07
ayoungldap.TIMEOUT maybe?21:07
kmallocayoung: maybe as well21:07
kmalloci am guessing it is an interaction with the reconnect object and ldappool's use21:08
ayoungoh, that last one looks like a suspect21:08
kmallocah yeah that might do it21:08
ayoungthat is so not my code21:08
ayoungwho wrote that...21:09
kmallocthat is mostly inherited from mozilla21:09
orange_juliusAre you guys looking at ldappoll stuff?21:09
kmallocwhen we took over ldappool21:09
kmallocorange_julius: yeah.21:09
kmallocayoung: so, we own ldappool but it was originally a mozilla project.21:09
kmallocayoung: my guess is that is historical.21:09
ayoungI still wanna know who wrote dat21:10
kmallocayoung: git blame?21:10
ayoung5f674821 (Steve Martinelli   2016-05-12 12:16:34 -0700 255)             except ldap.LDAPError as error:21:11
ayoung5f674821 (Steve Martinelli   2016-05-12 12:16:34 -0700 256)                 exc = error21:11
ayoungDun dun DUN!21:11
ayoungnot really21:11
ayoung-            except ldap.LDAPError as exc:21:12
ayoung+            except ldap.LDAPError as error:21:12
kmallocayoung: how about https://github.com/openstack/ldappool/blame/6350323b7f6ecbb54f70fd2e847190c99b826d94/ldappool/__init__.py#L21721:12
ayoungpredates this repo....21:13
kmallocyep21:13
ayounginitial-import....21:14
kmallocso long long ago21:14
kmallocin a galaxy far away21:14
ayoungso, I wonder if an auth failure is not supposed to raise an exception21:15
ayoungexception ldap.INSUFFICIENT_ACCESS21:16
ayounghttps://www.python-ldap.org/en/latest/reference/ldap.html21:16
ayoungI wonder if we were not using that code before to authenitcate21:17
*** dave-mccowan has quit IRC21:17
kmallocwe might have been21:17
ayoung conn = self.user.get_connection(user_ref['dn'],21:19
ayoung                                            password, end_user_auth=True)21:19
ayoungI was young.  I needed the money.21:19
ayoungI'd like to state for the record that I did not do the pool thing.21:20
ayoungcommit 22b114f64724a551df5d32075b6a2d93c394b0d321:21
ayoungAuthor: Dolph Mathews <dolph.mathews@gmail.com>21:21
ayoungDate:   Fri Feb 26 01:22:22 2016 +000021:21
ayoung    Enable LDAP connection pooling by default21:21
ayounghttps://review.openstack.org/#/c/285008/21:21
ayoungI +2ed it21:21
kmallocLOL21:22
ayoungcourse, that was not the initial pool commit21:22
ayoungcommit ea689ff78f47ca762a4c46a726917b290c52cfef21:22
ayoungAuthor: Arun Kant <arun.kant@hp.com>21:22
ayoungDate:   Fri May 23 15:25:38 2014 -070021:22
ayounghttps://review.openstack.org/#/c/95300/21:23
ayoungI +2ed that one, too21:23
*** spilla has quit IRC21:23
ayoungorange_julius, you can blame me21:23
orange_juliusfire and brimstone upon you!!!21:24
ayoungIt would cool things off right now.  Heat wave in New England21:24
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert role_assignments API to flask native dispatching  https://review.openstack.org/59051821:24
ayoungI don't think that is fixable21:25
ayoungyou'd have to turn off pooling21:25
orange_juliusSo is this issue indeed with https://bugs.launchpad.net/ldappool/+bug/1785898 ?21:25
openstackLaunchpad bug 1785898 in ldappool "Connection Pooling Retries Failed Passwords" [Undecided,New]21:25
ayoungit sure is21:26
ayoungSwitch to Kerberos21:26
ayoungDo we still support kerberos with LDAP? Seems to me Dolph wanted to kill that21:26
orange_julius=(   What effect does turning off connection pooling have?21:27
ayoungslower connection to the LDAP server/21:27
ayoungbut you would not be turning it off21:27
ayoungjust stopping retrys21:27
ayoungwhich means poor failover semantics, I think21:28
ayoungthe hack would be to change ldappool to not retry on auth failure21:28
ayoungyou have the power to change Python code in production?21:28
orange_juliusYes but we won't be doing that =P. I think disabling retries would be fine. Does that involve just setting the retry number to 0 in the ldap options?21:29
*** lbragstad has joined #openstack-keystone21:55
*** ChanServ sets mode: +o lbragstad21:55
orange_juliusmight've replicated the issue. I can't unlock my own account though so I gotta wait =(21:56
*** mchlumsky has quit IRC21:58
*** neiljerram has quit IRC22:08
*** rcernin has joined #openstack-keystone22:09
kmallocTurning off the pool is much slower, but this legitimately is an icky bug22:14
kmallocI'd go with slower over lockout :P22:14
*** itlinux has quit IRC22:23
*** _ix has quit IRC22:29
*** aojea has quit IRC22:46
*** evrardjp has quit IRC22:55
*** gyee has quit IRC23:06
openstackgerritMerged openstack/oslo.policy master: Imported Translations from Zanata  https://review.openstack.org/59014523:11
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert role_assignments API to flask native dispatching  https://review.openstack.org/59051823:31
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert role_assignments API to flask native dispatching  https://review.openstack.org/59051823:32
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert role_assignments API to flask native dispatching  https://review.openstack.org/59051823:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!