18:03:14 <stevemar> #startmeeting keystone
18:03:15 <openstack> Meeting started Tue Mar 22 18:03:14 2016 UTC and is due to finish in 60 minutes.  The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:03:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:03:20 <openstack> The meeting name has been set to 'keystone'
18:03:27 <stevemar> #topic keystone RC status
18:03:48 <stevemar> #link https://launchpad.net/keystone/+milestone/mitaka-rc1
18:04:00 <stevemar> we cut RC1 last week and created the stable/mitaka branch
18:04:08 <lbragstad> whoop
18:04:09 <stevemar> that is where mitaka proper will be released from
18:04:24 <stevemar> if you have any fixes that should be backported, propose them to stable/mitaka
18:05:10 <stevemar> i think the only one that may be included is bug 1558670, but it's definitely not a show stopper
18:05:11 <openstack> bug 1558670 in OpenStack Identity (keystone) "Internal server error when updating an identity provider" [Medium,In progress] https://launchpad.net/bugs/1558670 - Assigned to Rodrigo Duarte (rodrigodsousa)
18:05:21 <rodrigods> ++
18:05:44 <stevemar> any other issues with the current RC ?
18:05:49 <bknudson> is there a tag like rc-potential?
18:05:56 <stevemar> bknudson: yep
18:06:19 <stevemar> just tagged 1558670 with it
18:06:37 <stevemar> no other bugs should have it
18:07:19 <stevemar> if no other questions here, we can move on
18:07:30 <stevemar> #topic Docs and DocImpact
18:07:44 <stevemar> i thought we covered this last meeting...
18:07:48 <stevemar> maybe not
18:07:55 <ayoung> DOC DOC MOOSE!
18:07:55 * morgan still dislikes docimpact opening a bug against keystone.
18:08:05 <morgan> i think it's kindof absurd.
18:08:15 <ayoung> Keystone is absurd, so it fits
18:08:20 <ayoung> theater of the absurd
18:08:30 <stevemar> so if a patch is going to impact the openstack manuals create a DocImpact tag in the commit message
18:08:53 <ayoung> stevemar, shouldn't it open a bug in  openstack manuals?
18:09:08 <stevemar> it'll open a bug against keystone, and we can use that to create keystone dev docs + it's up to us to also open it against the openstack-manuals team
18:09:28 <ayoung> that sounds like not good
18:09:30 <stevemar> release notes are not changed, we should keep doing what we've been doing there
18:09:33 <dstanek> back to the stone age with manual processes!
18:09:42 <ayoung> yeah, can we just do the right thing here?
18:09:55 <morgan> this is why i think docimpact is absurd to open a bug against keystone
18:09:55 <stevemar> it's part of an effort to reduce the amount of bugs the docs team has to deal with
18:09:59 <stevemar> they can't keep up
18:10:06 <bknudson> I think they want us to fill in whatever more info we can and then transfer it
18:10:13 <ayoung> assign the bug to the committer
18:10:16 <bknudson> or just close it
18:10:19 <ayoung> but don't put it in the keystone db
18:10:55 <morgan> stevemar: so... shove more bugs on the projects and then make us manually move it over -mwhich doesn't really reduce their bug count. just adds manual steps that will make the bugs / updates more likely to get lost
18:11:12 <dstanek> stevemar: the thing i don't like is that since we push our patches to include dev docs the only time i expect a docimpart is a manuals bug
18:11:34 <stevemar> dstanek: yep, which is why i wasn't enforcing the DocImpact tag
18:11:51 <stevemar> and it was just 2 weeks ago that the docs team learned about keystone-manage bootstrap :(
18:11:57 <ayoung> yeah, let's can this one
18:12:34 <stevemar> we can chat about it with the docs team in person at the summit
18:12:42 <stevemar> but for now, this was just a PSA
18:12:51 <ayoung> and let's open the bugs where they belong.  IN the docs repo, but assign them to the keystone user that commited the patch, or to a default keystone contact if that is all we can do
18:12:57 <ayoung> ok...move on?
18:13:03 <stevemar> yep
18:13:10 <stevemar> happily
18:13:13 <stevemar> #topic Define driver interface
18:13:16 <stevemar> bknudson: ^
18:13:31 <bknudson> so today we essentially define the driver interface by test_backends
18:13:41 <bknudson> and test_backends is actually mostly testing the managers, not the drivers
18:13:50 <bknudson> so the proposal is to have component tests for the drviers
18:14:13 <bknudson> which also means updating the docstrings for the drivers to actually specify how they're supposed to work
18:14:25 <bknudson> for example, the docstring for create_user should say that it returns the user
18:14:26 <stevemar> bknudson: do you have an example / first patch that we can look at ?
18:14:27 <rodrigods> bknudson, so mock the driver responses?
18:14:31 <bknudson> because otherwise keystone isn't going to work.
18:14:33 <stevemar> wait found it
18:14:35 <stevemar> #link https://review.openstack.org/#/c/291950/
18:14:52 <rodrigods> bknudson, i mean, for the manager tests
18:15:14 <bknudson> rodrigods: for the manager tests what I plan to do is run them with sqlite.
18:15:19 <amakarov> can't imagine how driver unit test may look like
18:15:27 <rodrigods> so no sense do have driver tests?
18:15:31 <bknudson> sqlite only, and the driver tests will run against all the drivers we support.
18:15:33 <rodrigods> since they are being tested with the manager
18:15:43 <rodrigods> bknudson, hmm
18:15:44 <rodrigods> got it
18:15:48 <amakarov> maybe revive functional testing idea?
18:16:01 <rodrigods> amakarov, we are going to get there :)
18:16:18 <rodrigods> in today's meeting
18:16:18 <bknudson> so for example we will have driver tests that test against real sql
18:16:22 <bknudson> and hopefully real ldap.
18:16:29 <ayoung> WOO!
18:16:42 <rodrigods> bknudson, great, so it is aligned with the tempest plugin bp
18:16:52 <rderose> bknudson: sounds good to me
18:17:01 <bknudson> functional tests are different. They test from the REST API in
18:17:04 <dstanek> bknudson: i'd love to see a fake driver that acts exactly as a real driver would to test the managers
18:17:17 <bknudson> the driver tests will be calling the driver methods directly
18:17:40 <ayoung> bknudson, meaningless without a real back end though
18:17:53 <ayoung> functional tests need to be there to set up LDAP and SQL
18:17:59 <ayoung> otherwise it is a finger drill
18:18:00 <bknudson> for SQL a real backend should be pretty easy.
18:18:05 <ayoung> same for LDAP
18:18:11 <ayoung> devstack supports now
18:18:12 <bknudson> this is the next topic
18:18:18 <dstanek> ayoung: we can do that without having functional tests
18:18:33 <bknudson> LDAP will be a little harder I'll have to convince infra to have LDAP in the unit test image like they do with mysql and postgresql
18:18:46 <ayoung> dstanek, now we are arguing over the semantics of functional tests.  So, yes, sure
18:18:51 <rodrigods> bknudson, long term is a great effort
18:19:30 <shaleh_> bknudson: why is ldap harder than two sql packages?
18:19:51 <ayoung> shaleh_, cuz only Keystone needs it, and thus only we understand it.
18:19:52 <amakarov> shaleh_, imagine loading fixtures there ))
18:19:54 <stevemar> shaleh_: i imagine some push back since we're the only project that'll use ldap and those images are for every project
18:19:56 <bknudson> shaleh_: it's harder because the sql packages are already there whereas for openldap I'll have to find out if they can do it.
18:20:29 <ayoung> bknudson, can we install post image build?
18:20:36 <stevemar> looks like we have a lot of different testing strategies to pick from
18:20:59 <ayoung> it means that the Keystone process will take  a little longer , but, so what...
18:21:01 <bknudson> ayoung: I think there's a way to do the install ourselves.
18:21:23 <ayoung> bknudson, OK. We can link that in with the LDAP cleanup effort, then.
18:21:32 <ayoung> Worst case, we battle to get it in to the base effort
18:21:38 <stevemar> bknudson: mind if we jump to the next topic?
18:21:39 <bknudson> yes, part of this was to help with the LDAP cleanup effort
18:21:48 <amakarov> bknudson, one tricky thing about LDAP: enterprises use everything besides OpenLDAP :)
18:21:54 <bknudson> ok, only question is if others wanted to work on a driver.
18:22:02 <bknudson> if so, get in touch.
18:22:19 <rodrigods> ++
18:22:20 <bknudson> enterprise ldap isn't going to be installed by infra since it's not open source
18:22:25 <shaleh_> amakarov: true, but how many of our problems are really due to different LDAP impls?
18:22:34 <stevemar> guys, bknudson is asking for help, it must be hard
18:22:36 <bknudson> you'll need to provide external CI for that.
18:22:38 <rodrigods> shaleh_, some...
18:22:49 <rodrigods> not much, though
18:22:58 <ayoung> amakarov, I am well aware...especially active directory, but testing that is non-free
18:23:14 <ayoung> shaleh_, which is also the answer to your question
18:23:34 <stevemar> #topic Opportunistic DB tests
18:23:40 <stevemar> bknudson: ^ you're up again
18:23:44 <stevemar> this is awesome btw
18:23:47 <bknudson> so this is somewhat related to the prev discussion
18:23:55 <bknudson> turns out infra installs mysql and postgresql
18:24:03 <bknudson> with a known username and password
18:24:04 <amakarov> ayoung, can we provide some way to configure functional test to run against come configurable LDAP backend?
18:24:20 <bknudson> and oslo.db has test_db which supports these databases
18:24:27 <bknudson> and runs with them if they're available
18:24:39 <amakarov> I think that can be handy for deployers
18:24:43 <ayoung> amakarov, yes,  and we should also treat it like another database
18:24:44 <bknudson> so the work here is to get our migration tests working with mysql and postgresql, too.
18:25:00 <bknudson> anyways, I'm making good progress
18:25:11 <bknudson> and I'll try to add some docs so that devs can set up their env to run them, too.
18:25:13 <ayoung> we probably should focus on LDAP via domain specific backends, as that is the easiest:  if it is there, mount it....
18:25:27 <stevemar> which would be great, since we find bugs very often related to non-sqlite backends
18:25:44 <rderose> ++
18:25:44 <shaleh_> stevemar: no doubt
18:25:47 <bknudson> the migrations are not working now with mysql
18:25:51 <bknudson> and postgresql
18:26:00 <stevemar> bknudson: ?
18:26:01 <bknudson> but there's not too many failures. I'll post the changes.
18:26:08 <amakarov> bknudson, neat
18:26:10 <bknudson> the migration tests, not the migrations themselvs.
18:26:13 <stevemar> bknudson: the tests or the
18:26:14 <stevemar> oh good
18:26:22 <stevemar> scared me for a sec
18:26:53 <bknudson> that was it, just wanted others to know that this is available.
18:27:15 <bknudson> I think the infra team is changing things up a bit so that the dbs are only available on a periodic job.
18:27:19 <shaleh_> bknudson: what will toggle between sqlite and the others?
18:27:31 <bknudson> there are 3 test classes, they all try to run
18:27:54 <bknudson> oh, what toggles is there's a fixture that sets the connection string so sqlalchemy picks the right driver
18:27:55 <stevemar> shaleh_: oslo.db magic to figure out whats installed
18:28:03 <amakarov> shaleh_, foreign keys cascading operations for sure
18:28:22 <shaleh_> bknudson: nice
18:28:43 <stevemar> rodrigods: ready for next topic?
18:28:46 <rodrigods> yep!
18:28:49 <rodrigods> should be quick
18:28:57 <stevemar> bknudson: i am really looking forward to the different sql tests
18:29:05 <stevemar> bknudson: let me know if i can help
18:29:17 <stevemar> #topic Specless bp for tempest plugin interface tests
18:29:21 <stevemar> rodrigods: ^
18:29:31 <rodrigods> so... tempest won't accept negative tests anymore
18:29:35 <rodrigods> that are component specific
18:29:41 <rodrigods> dstanek was in the meeting as well
18:29:53 <rodrigods> the idea is to have tests in keystone using the tempest plugin interface
18:30:02 <rodrigods> this tests will run only against keystone
18:30:24 <bknudson> this sounds cool
18:30:28 <rodrigods> since are tests, don't need a spec, right?
18:30:44 <bknudson> does it replace the functional testing we've been discussing? e.g., federation tests?
18:30:45 <dstanek> this stemmed from QA wanting to get rid of certain types of tests from tempest and push them back into the projects
18:30:47 <stevemar> is this the keystone meeting or the QA meeting, it's all tests!
18:30:50 <bknudson> no spec is needed.
18:31:00 <rodrigods> stevemar, lol
18:31:04 <stevemar> ++ no spec
18:31:11 <dstanek> bknudson: i've been toying with a plugin to do the functional testing
18:31:13 <rodrigods> should be sending a patch this week
18:31:16 <rodrigods> dstanek, ++
18:31:28 <bknudson> how many tests are they getting rid of?
18:31:33 <dstanek> bknudson: it looks like it could, but we'd still need to carry all the devstack things that i wrote
18:31:38 <rodrigods> bknudson, none
18:31:44 <rodrigods> but new ones won't be accepted
18:31:46 <bknudson> I've always found tempest's keystone tests to be pretty wacky, e.g., they have a v3.admin package???
18:31:54 <rodrigods> they even denied a test where a found the idp update bug
18:32:09 <rodrigods> bknudson, yes
18:32:39 <bknudson> so are the plugin tests only for negative tests or any tests we want?
18:32:45 <dstanek> bknudson: rodrigods: right, they decided to keep them all (they were primarily talking about negative tests)
18:32:47 <rodrigods> bknudson, any tests we want
18:33:13 <bknudson> when would we choose our plugin vs tempest proper?
18:33:33 <rodrigods> i think the answer is: whenever we communicate with other components -> tempest
18:33:34 <stevemar> maybe we should have all these tests in it's own repo, i think ayoung mentioned this
18:33:41 <rodrigods> only keystone -> keystone
18:33:53 <dstanek> bknudson: it sounds like most things we want to test would be in our plugin since we don't use may other services
18:33:59 <rodrigods> stevemar, makes sense... but is a bit more work to enable them in this case
18:34:12 <dstanek> rodrigods: also i imagine tests of defcore?
18:34:41 <shaleh_> do we plan to pull tests in from Tempest once the infrastructure is in place to accept them?
18:34:58 <rodrigods> shaleh_, maybe?
18:35:00 <rodrigods> makes sense to me
18:35:06 <stevemar> we could
18:35:22 <shaleh_> rodrigods: at least on the negative/positive side right?
18:35:29 <rodrigods> shaleh_, yes
18:35:34 <shaleh_> Tempest validates it works, we look for where it does not.
18:36:29 <stevemar> i really hope this talk continues up to and at the summit
18:36:33 <shaleh_> so even though each project will carry these tests they will still be executed by a Tempest run?
18:36:56 <breton> what's a negative test?
18:37:19 <rodrigods> breton, the tests an error, for example
18:37:24 <bknudson> breton: a negative test is one that shows what happens when you do something wrong
18:37:43 <bknudson> e.g., try to specify an ID when you create a user
18:38:34 <dstanek> breton: the most valuable tests!
18:39:22 <ayoung> stevemar, the rationale for its own repo was to be able to to double book entry accounting
18:39:39 <ayoung> any change that affects tests has to be done in 3 commits
18:39:43 <ayoung> 1.  change the tests
18:39:47 <ayoung> 2. change the main code
18:39:54 <ayoung> 3. change the tests to validate the new code
18:40:12 <ayoung> usually step 1 is "disable old tests that are checking the wrong thing"
18:40:27 <rodrigods> hmm
18:40:27 <rodrigods> true
18:40:48 <ayoung> But...the other issue I had with tempest is that their test infrastructure essentially has its own client implementation
18:40:51 <bknudson> it should be easy to split out the plugin later since it's self-contained.
18:40:56 <stevemar> ayoung: yep, it's got merit, but we can probably just add to our own projects for now and when it gets too big we can toss them in their own repo
18:41:13 <ayoung> I'd much rather we focused on functional testing in the keystone client project most of all
18:41:17 <stevemar> the tests are not public interfaces, so we can rip them out whenever
18:41:23 <ayoung> client->server->real backends
18:41:29 <amakarov> ayoung, ++
18:41:53 <rodrigods> ayoung, it's an option, but we need to test keystone directly
18:42:08 <amakarov> ayoung, the problem is that such tests are hard to investigate on their failure
18:42:26 <rodrigods> and tempest has a really great infrastructure in its plugins
18:42:30 <rodrigods> to write the tests
18:42:41 <ayoung> rodrigods, the KC is such a think wrapper , I disagree
18:42:56 <ayoung> in order to break something, it would take a change to both Keystone and client in sync
18:43:05 <rodrigods> ayoung, we need both
18:43:07 <shaleh_> rodrigods: help them make it available to all now that they are asking each project to carry tests?
18:43:18 <bknudson> ok, my vote is no spec is required. Might help but not a big deal.
18:43:31 <bknudson> we can review the code.
18:43:45 <rodrigods> will put the docs in the commit message too
18:43:51 <rodrigods> it should be enough
18:44:03 <dstanek> bknudson: i don't think we need a spec either
18:44:08 <stevemar> docs in the commit msg?
18:44:15 <rodrigods> stevemar, link :P
18:44:25 <stevemar> ahhh
18:44:30 <stevemar> rodrigods: i see
18:44:39 <stevemar> rodrigods: keep the docs in the docs folder :P
18:44:50 <rodrigods> stevemar, tempest plugin interface docs
18:44:51 <stevemar> #topic TLS Gate
18:44:57 <rodrigods> thanks o/
18:44:58 <ayoung> OK, so Rob can't make it
18:45:01 <stevemar> anyone know rcritten ?
18:45:06 <stevemar> ayoung does
18:45:09 <ayoung> stevemar, he's from my team
18:45:11 <bknudson> we've all met rcritten
18:45:16 <ayoung> we are trying to get TLS everywhere to work
18:45:21 <ayoung> and this is part of that effort.
18:45:30 <ayoung> So, the question here is, can Keystone run with this setup
18:45:31 <bknudson> my only complaint about this is I don't think keystone needs a TLS proxy
18:45:41 <bknudson> we run in httpd, so just configure httpd with TLS
18:45:43 <ayoung> yeah, but do we set up Keystone with TLS at all?
18:46:09 <morgan> bknudson: ++
18:46:17 <ayoung> I figure the setup would be roughly the same...and many sites do front things with a proxy
18:46:19 <bknudson> so I'd really like to see this, but I think we'd be better off waiting until we get rid of eventlet and uwsgi
18:46:20 <morgan> HTTPD can handle the TLS layer just fine.
18:46:27 <bknudson> and then just set httpd to TLS and skip the proxy
18:46:29 <ayoung> right morgan ?  HA Proxy as a TLS terminator is common
18:46:31 <stevemar> how does devstack setup keystone with tls now if we run under httpd?
18:47:06 <morgan> ayoung: it's all the same really - but i don't see a big benefit to adding a proxy for the sake of proxy
18:47:08 <bknudson> if someone wants to propose haproxy in devstack that would be interesting, too.
18:47:09 <ayoung> does it even support that? stevemar ?
18:47:09 <morgan> in gate
18:47:15 <ayoung> bknudson, ++
18:47:30 <morgan> but haproxy is better than generic tls proxy
18:48:06 <amakarov> bknudson, we use it already
18:48:13 <ayoung> morgan, so in this case it is more acase of validating the approach so that we can get the other services to use it, too
18:48:18 <browne> we use haproxy as ssl terminator.  does the load balancing and active-active for us very nicely
18:48:37 <amakarov> browne, ++
18:48:44 <bknudson> browne: how do you run keystone? eventlet?
18:48:50 <ayoung> I can answer back that we think HAProxy would be the better approach.  I think that is not a bad answer
18:48:53 <dstanek> browne: is haproxy running on each node?
18:48:54 <ayoung> would we then use it?
18:48:59 <browne> bknudson: yes today (kilo based)
18:49:23 <browne> dstanek: haproxy runs on its own vm separate from the controller.
18:49:29 <dstanek> browne: so traffic from haproxy -> keystone isn't over TLS
18:49:36 <amakarov> bknudson, we run haproxy + apache mod_wsgi (looking towards uwsgi)
18:49:48 <browne> dstanek:  no, its just http on private network
18:49:55 <ayoung> dstanek, it can be, but that is a different (but important) thing to test
18:50:22 <browne> and we use keepalived for virtual IPs
18:50:30 <dstanek> ayoung: that's why i was asking :-) i like TLS everywhere, not just between users and services
18:51:36 <stevemar> 10 minute warning
18:51:36 <ayoung> OK...that is all I have.
18:51:37 <bknudson> I think we should start with (for keystone) configuring apache for TLS.
18:51:45 <ayoung> bknudson, ++
18:51:47 <dstanek> bknudson: ++
18:51:48 <amakarov> https://httpd.apache.org/docs/2.4/ssl/
18:51:49 <bknudson> ignore eventlet case since that's dead
18:52:49 <dstanek> bknudson: if they want to shoot themselves in the foot with eventlet then they can use haproxy/stud/whatever to terminate
18:52:52 <breton> letsencrypt?
18:53:07 <ayoung> but actually, what about the Keystone gate jobas
18:53:21 <stevemar> this is where we setup the different keystone deployment methods: https://github.com/openstack-dev/devstack/blob/master/lib/keystone#L283-L323
18:53:21 <bknudson> we're getting rid of keystone-all, so there won't be eventlet
18:53:21 <ayoung> shouldn't we be running with the other services behind tls proxy or comparable?
18:53:48 <bknudson> other services will have to use tls proxy unless they can run in apache
18:54:12 <ayoung> right, so can we enble that gate job for Keystone?
18:54:17 <ayoung> https://review.openstack.org/#/c/293090/
18:54:47 <ayoung> we can modify it to not do proxy for Apache based services.  Is that the feedback?
18:54:49 <bknudson> does it work?
18:54:52 <bknudson> the gate job?
18:55:24 <bknudson> I'd prefer it if the devstack config for tls "proxy" was configuring apache to do TLS directly
18:55:36 <stevemar> ++
18:55:37 <bknudson> and of course running keystone in apache and not eventlet
18:55:40 <ayoung> bknudson, according to Rob (who is far more thorough than I am on most things) yes it does
18:56:10 <ayoung> So, to be honest, I would have to look at what he means by "most services"
18:56:35 <bknudson> is rcrit going to be around so we can ask him questions? I'm trying to change things in this area and might have questions or might ask him to do stuff.
18:57:01 <ayoung> bknudson, yeah, I asked him to be here, but too late for him to change another appointment.  I'll drag him in next week
18:57:12 <ayoung> he'll be back later on today, too
18:57:25 <bknudson> have him ping me when he's on
18:57:40 <ayoung> bknudson, will do.  He's got two accoutns active in #freeipa right now
18:57:51 <ayoung> just I know he's OOTO ATM
18:57:59 <ayoung> all I have
18:58:25 <stevemar> bknudson follow up with rcritten and see if you have overlap i guess
18:58:39 <stevemar> last topic ...
18:58:41 <stevemar> #topic Apply for stable:follows-policy tag
18:58:48 <stevemar> i will submit a patch for this
18:58:53 <stevemar> end topic :)
18:58:54 <ayoung> bknudson, waht does that tag give us?
18:58:56 <bknudson> I don't have any info here, it was just asked this morning in the mtg.
18:59:11 <shaleh_> bknudson: someone was looking for people to talk about it last week
18:59:18 <stevemar> ayoung: it just asserts that we will follow the stable policy
18:59:29 <ayoung> stevemar, no way!
18:59:32 <bknudson> I guess stevemar will fill us in next week.
18:59:33 <ayoung> times up
18:59:33 <stevemar> so we won't backport features or nonsense
18:59:47 <stevemar> yep
18:59:50 <stevemar> #endmeeting