17:59:51 <stevemar> #startmeeting keystone
17:59:52 <openstack> Meeting started Tue Sep  6 17:59:51 2016 UTC and is due to finish in 60 minutes.  The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:59:53 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:59:55 <openstack> The meeting name has been set to 'keystone'
17:59:59 <samueldmq> hi all
18:00:09 <jaugustine> o/
18:00:25 <dolphm> o/
18:00:26 <NishaYadav> o/
18:00:32 <shaleh> \o
18:00:46 <gagehugo> hey
18:01:04 <stevemar> #agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting
18:01:08 <stevemar> o/
18:01:09 <browne> o/
18:01:15 <bknudson> hi
18:01:27 <ayoung> Holla
18:01:43 <henrynash> hi
18:01:43 <stevemar> ayoung: i moved your thing to the end of the agenda OK
18:01:59 <ayoung> stevemar, not really, no.
18:02:02 <topol> o/
18:02:15 <stevemar> ayoung: we will get there by 2:05 :)
18:02:16 <ayoung> We break the other projects, it is kindof like the worst thing to do
18:02:52 <stevemar> just push the pause button for 5 minutes
18:03:00 <stevemar> #topic Announcements
18:03:06 <stevemar> rderose is now core :)
18:03:13 <raildo> rderose, congrats :)
18:03:14 <henrynash> yeah!
18:03:15 <lbragstad> ++
18:03:16 <knikolla> congrats!
18:03:17 <stevemar> thanks rderose for your hard work
18:03:23 <bknudson> nice
18:03:23 <rderose> yeah!  thanks guys :)
18:03:24 <gagehugo> congrats!
18:03:29 <gyee> rderose, u da man!
18:03:37 <rderose> gyee: :)
18:03:43 <henrynash> another rabbit bites the dust?
18:03:48 <henrynash> just kiddin
18:03:50 <rderose> hahaha
18:03:58 <stevemar> there are many other great contributors helping make keystone awesome, you're all great in my eyes
18:04:08 <henrynash> stevemar:++
18:04:19 <ayoung> We just lost another productive community member to the burden of code reviews.
18:04:20 <stevemar> rderose: thanks for all your coding and reviewing
18:04:38 <rderose> stevemar: thanks man, really appreciate all of your support
18:04:43 <dolphm> ayoung: ++
18:04:45 <gyee> ayoung, then try to write less code damn it!
18:05:00 <stevemar> #topic Release status
18:05:36 <stevemar> we're in the RC period, so only critical bug fixes should be merging now
18:05:45 <stevemar> docs and tests are always OK to merge
18:06:09 <stevemar> please go and play with the latest master level and find bugs :)
18:06:17 <stevemar> especially in the PCI support :D
18:06:27 <ayoung> https://bugs.launchpad.net/keystone/+bug/1619758  should be on that list
18:06:27 <openstack> Launchpad bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] - Assigned to Adam Young (ayoung)
18:06:27 <dolphm> and rolling upgrades :)
18:06:39 <stevemar> dolphm: yes that too!
18:06:50 <stevemar> dolphm: have prometheanfire test it for postgres :S
18:07:05 <ayoung> where is the bug list?
18:07:23 <stevemar> ayoung: the only one targeted for rc1 is yours
18:07:48 <stevemar> RC bugs will be tracked here: https://bugs.launchpad.net/keystone/+milestone/newton-rc1
18:07:49 <dolphm> #link https://launchpad.net/keystone/+milestone/newton-rc1
18:08:13 <ayoung> dolphm,  thanks
18:08:17 <stevemar> i encourage everyone to go and triage some of the open keystone bugs
18:08:25 <stevemar> make sure we are not missing any
18:08:31 <stevemar> that should go into newton
18:08:55 <stevemar> alright, done all the organization stuff
18:09:05 <stevemar> #topic credential encryption breaking the world
18:09:14 <stevemar> some background on this
18:09:29 <stevemar> actually, someone else want to take this one? dolphm? it's a lot of typing
18:09:32 <stevemar> :P
18:09:36 <dolphm> ayoung: ?
18:09:42 <stevemar> we broke tripleo
18:09:48 <bknudson> did grenade break or did a change have to be made to devstack?
18:10:00 <lbragstad> a change was made to both devstack and grenade
18:10:02 <dolphm> we broke everyone that is not following our upgrade release notes and attempting to upgrade anyway
18:10:15 <stevemar> bknudson: we upgraded grenade https://github.com/openstack-dev/grenade/blob/master/projects/10_keystone/from-mitaka/upgrade-keystone
18:10:15 <ayoung> yeah, firedrill
18:10:45 <lbragstad> it fails because if there are credentials in the backend when the migration is run - it will attempt to encrypt them
18:10:49 <ayoung> So...distribution of keys is hard
18:10:53 <bknudson> do you need to do this even if you don't have any credentials?
18:10:58 <lbragstad> bknudson no
18:10:58 <stevemar> in an attempt to make keystone more secure, by encrypting credentials, we are *forcing* an extra upgrade step for mitaka -> newton, you must run `keystone-manage --credential_setup`
18:11:04 <henrynash> dolphm: so an offline migration fails?
18:11:04 <lbragstad> bknudson it's conditional if you have credentials
18:11:12 <ayoung> doing it right is pretty much beyond the scope of what Tripleo should be trying to do between now and release
18:11:19 <lbragstad> bknudson https://github.com/openstack/keystone/blob/b47f10290ed83415149f3d2ab6b0dc64646e578a/keystone/common/sql/data_migration_repo/versions/003_migrate_unencrypted_credentials.py#L26
18:11:29 * stevemar waves at EmilienM
18:11:33 <dolphm> lbragstad: the error in bug 1619758 is not super helpful, btw
18:11:33 <openstack> bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] https://launchpad.net/bugs/1619758 - Assigned to Adam Young (ayoung)
18:11:40 <dolphm> should fix that with a better message
18:11:40 <EmilienM> o/
18:11:48 <ayoung> To use the Credentials backend now requires a setup of the keys to encrypt the credentials
18:12:04 <stevemar> ayoung: yes, as it should have been from day 1
18:12:04 <lbragstad> dolphm sure - we can catch that and report something more useful
18:12:05 <ayoung> and, this key needs to be in sync across all keystone servers that talk to the same database
18:12:30 <lbragstad> dolphm bug report?
18:12:32 <dolphm> stevemar: ++ we should never have been storing secrets in plaintext anywhere, but since we're already in that business, we need to correct our behavior ASAP
18:12:39 <dolphm> lbragstad: https://bugs.launchpad.net/keystone/+bug/1619758
18:12:39 <openstack> Launchpad bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] - Assigned to Adam Young (ayoung)
18:12:40 <stevemar> lbragstad: https://bugs.launchpad.net/keystone/+bug/1619758
18:12:41 <ayoung> dolphm, that is what happens when you record bugs after close of business on a Friday evening
18:13:07 <lbragstad> dolphm want me to just tack the fix on to that?
18:13:12 <stevemar> dolphm: what are the OSA guys doing?
18:13:27 <dolphm> ayoung: i'm referring to needing to encrypt things, and not let deployers continue to store secrets in plain text using keystone
18:13:38 <stevemar> lbragstad: you can make the error message improved related to the bug, not close it though
18:13:42 <dolphm> stevemar: running credential_setup ?
18:13:43 <ayoung> EmilienM, check me on this, the breakage was due to Tempest run, rigjht?
18:13:47 <lbragstad> stevemar yep
18:13:49 <lbragstad> stevemar I can do that
18:13:55 <lbragstad> stevemar i'll leave a comment
18:13:59 <stevemar> lbragstad: thx
18:14:15 <lbragstad> stevemar andymccr is working on it for OSA
18:14:24 <lbragstad> i already had a conversation with him about it this morning
18:14:32 <EmilienM> ayoung: tempest could not validate keystone without credentials
18:14:35 <dolphm> stevemar: OSA is not broken because they're not attempting to upgrade to master or newton-rc1 yet
18:14:41 <stevemar> ayoung: yes, so we had to decide... if someone is not encrypting their credentials, do we break them on upgrade, or when they are using the API
18:14:49 <dolphm> stevemar: at least, not in their gate or anything
18:14:53 <ayoung> stevemar, the answer is "Neither"
18:14:55 <ayoung> but...
18:15:08 <lbragstad> dolphm andymccr was hitting some issues on some tests he is working on locally
18:15:41 <ayoung> So, lets get out of the way the real sin is in storing and retrieving credentials, and encrypting them is a bandaid (albeit necessarY) on a sucking chest wound.
18:15:52 <ayoung> Yes, unencrypted is worse
18:16:24 <ayoung> so, we should have no-opped this from the beginning, and given people at least one dev cycle to catch up
18:16:30 <dolphm> ayoung's patch provides a no-op credential provider, that basically doesn't require any configuration, and he's proposing it as the default. that would let operators switch to the fernet-based credential encryption provider later, but its probably broken by rolling upgrades now (?)
18:16:30 <ayoung> the dropped the no-op
18:17:07 <ayoung> now...I would be happy with having no-op as an option, but not the default option, as that would only require a single config change
18:17:14 <ayoung> we can live with that if necessary
18:17:17 <bknudson> can we add a warning to say to switch it?
18:17:23 <ayoung> but, that is just me being selfish
18:17:33 <dolphm> bknudson: i'd suggest marking the no-op as deprecated immediately to do exactly that
18:17:46 <lbragstad> the no-op thing will only work if there aren't any credentials stored in the backend
18:17:46 <ayoung> dolphm, that makes sense
18:17:52 <lbragstad> when the upgrade takes place
18:18:02 <samueldmq> dolphm: ++
18:18:06 <bknudson> I like the deprecated no-op as a solution.
18:18:22 <stevemar> bknudson: but will the noop driver work?
18:18:31 <ayoung> passes unit tests
18:18:35 <stevemar> i think lbragstad has doubts
18:18:39 <bknudson> nobody knows if anything works.
18:18:46 <ayoung> apevev said it failed due to a different problem
18:18:47 <dolphm> ayoung: the rolling upgrade process migrates plaintext data in the database to be encrypted -- that's not something operators can postpone
18:18:49 <ayoung> something about entrypoints
18:18:59 <lbragstad> it won't work if someone has credentials stored in plaintext and they upgrade
18:19:11 <dolphm> lbragstad: ++
18:19:15 <stevemar> right, the upgrade will fail
18:19:18 <lbragstad> if they do that and then switch to the noop provider they will be return cipher text to the end user
18:19:20 <ayoung> lbragstad, won't that migration fail if there is no key?
18:19:39 <dolphm> ayoung: yes, you can't upgrade without running credential_setup first
18:20:21 <bknudson> are there grenade tests with credentials?
18:20:34 <dolphm> "In order to upgrade successfully to Newton, deployers must encrypt all credentials currently stored before contracting the database. Deployers must run keystone-manage credential_setup in order to use the credential API within Newton, or finish the upgrade from Mitaka to Newton. This will result in a service outage for the credential API where credentials will be read-only for the duration of the upgrade process. Once
18:20:34 <dolphm> the database is contracted credentials will be writeable again. Database contraction phases only apply to rolling upgrades." http://docs.openstack.org/releasenotes/keystone/unreleased.html#upgrade-notes
18:20:41 <ayoung> so noop really is not going to help, then, is it
18:20:55 <lbragstad> bknudson now - but after the upgrade grenade will exercise the credential api
18:20:58 <lbragstad> bknudson no*
18:21:02 <dolphm> ayoung: i was hoping it would, but lbragstad is right -- it's only useful if you don't have any credentials anyway
18:21:29 <lbragstad> bknudson the credential api was failing to do stuff because it didn't have any keys to encrypt with
18:22:09 <ayoung> dolphm, so I don't think there are any easy answers here.
18:22:31 <stevemar> ayoung: are tripleo users upgrading?
18:22:49 <dolphm> ayoung: i'm afraid to ask, but what's the complexity blocking tripleo from dropping fernet keys on disk prior to upgrading?
18:22:51 <ayoung> stevemar, they will, so we need to cover that, too
18:23:08 <ayoung> dolphm, good question, here it goes:
18:23:26 <dolphm> ayoung: there's no reason to *require* credential_setup or any syncing if you can distribute them any other way
18:23:32 <dolphm> them = fernet keys
18:23:36 <ayoung> Tripleo does not have an easy way of distributing the credentials in a way that is not world readable
18:23:44 <stevemar> dolphm: i think EmilienM is looking into that https://bugs.launchpad.net/keystone/+bug/1619758/comments/3
18:23:44 <openstack> Launchpad bug 1619758 in OpenStack Identity (keystone) "Credential Encryption breaks deployments without Fernet" [Undecided,In progress] - Assigned to Adam Young (ayoung)
18:23:50 <ayoung> if it were just a single entry, then hiera can hide it from the world
18:23:58 <dolphm> ayoung: how are service user passwords and things distributed? or database passwords?
18:24:10 <ayoung> but the fact that it is a directory can only, at the moment, make use of a crude mechanism
18:24:13 <EmilienM> yes I'm currently working on tripleoclient to generate the keys for managing keystone credentials
18:24:25 <EmilienM> so we can use puppet to put the files on keystone servers
18:24:34 <ayoung> IN general, Tripleo is Heat generating Hiera which is used to configure puppet
18:24:39 <stevemar> dolphm: the complexity is unknown :(
18:24:45 <ayoung> so if the puppet modules can't do soemthing, we have limited tools available
18:24:59 <ayoung> which means poor EmilienM is responsible for making everythign work
18:25:11 <antipsychiatry> Destroy " Israel" !!!!!!! They are antichrist !!!!!!!!!!!!!!!!!!! They are synagogue of Satan!!!!!!!!!!
18:25:57 <ayoung> the keystone tooling assumes it is executed on the keystone node, but we need to generate the keys on the undercloud and then deploy them in sync to all the other nodes.
18:26:38 <ayoung> hence my Proof of concept using the undercloud's keystone to generate the Repo for the overcloud.
18:26:48 <stevemar> thanks infra <3
18:27:13 <fungi> (any time)
18:27:19 <dolphm> ayoung: you should be able to generate them on either cloud, or outside the cloud, doesn't matter. it's a single line of python to generate a key
18:27:26 <lbragstad> ayoung if you do use the keystone tooling, is there a way to sync them after they are generated?
18:27:36 <lbragstad> like how osa does it with fernet keys?
18:27:49 <ayoung> So, changes to fix tripleo usually require 1) Puppet changes and 2)triple-heat-template changes.  In this case, I think a third change to generate the Keys might also be required
18:27:54 <dolphm> i'm pretty sure mfisch has public puppet code out there to illustrate fernet key configuration and distribution without using keystone-manage at all
18:28:10 <lbragstad> #link https://github.com/pyca/cryptography/blob/master/src/cryptography/fernet.py#L46
18:28:29 <ayoung> lbragstad, not a good way.
18:28:33 <dolphm> lbragstad: don't even need to install cryptography!
18:28:43 <ayoung> Possibly sufficient to get through gate, but not one I would support live
18:28:45 <lbragstad> nope just need standard lib stuff
18:28:52 <ayoung> yeah, that is the easy part
18:28:58 <lbragstad> i believe EmilienM was working on something using that
18:29:10 <lbragstad> generating the keys and storing them in puppet somewhere?
18:29:35 <stevemar> while EmilienM hacks on his PoC, are there any other avenues we should look at ?
18:29:51 <stevemar> revert the credential encryption? :)
18:30:03 <ayoung> its not an impossible task, just one we should not expect the installers to deal with post Milestone 3
18:30:28 <stevemar> i agree the timing is terrible, which is why i put the revert on the table
18:30:38 <ayoung> stevemar, so, I'd be OK with a No op solution
18:31:22 <stevemar> what about a noop and a fernet, but another command to actually do the encrypting of all credentials?
18:31:39 <ayoung> lbragstad, what if we skip the encryption until the no-op provider is disabled, and do it as an offline operation
18:31:43 <dolphm> stevemar: the problem there is that a database migration is required
18:31:45 <ayoung> stevemar, that
18:31:57 <dolphm> stevemar: and we can't do database migrations outside of the upgrade process
18:32:08 <dolphm> it's not a standalone, at-will operation like credential rotation is
18:32:12 <dolphm> or fernet key rotation
18:32:17 <lbragstad> right
18:32:34 <bknudson> how about don't encrypt/decrypt if there are no keys?
18:32:43 <ayoung> dolphm, so, if the no-op provider is enabled, skip the encryption.  If the no-op provider is enabled, and encryption has not happened, report an error
18:32:49 <stevemar> bknudson: oh, like at all?
18:33:08 <stevemar> lbragstad: dolphm is that possible?
18:33:15 <ayoung> do we drop the old column?
18:33:18 <bknudson> right, currently it's an error if there are no keys, so the proposal would be to allow no keys to work (don't decrypt/encrypt)
18:33:25 <lbragstad> ayoung yes
18:33:36 <dolphm> bknudson: that'd require either rewriting our migration history or introducing a new migration after the one that ayoung is struggling with to re-introduce a plaintext column or something
18:33:43 <dolphm> ayoung: yes
18:33:51 <ayoung> ok...what if we put the keys in the database or have a default key that gets removed later?
18:34:00 <ayoung> encrypt, but poorly
18:34:04 <dolphm> keys in the database defeats everything
18:34:08 <ayoung> agreed
18:34:15 <ayoung> and a default key in the config file does, too
18:34:21 <dolphm> ayoung: hardcode your upgrade to use a null key
18:34:27 <dolphm> ayoung: then credential rotate later
18:34:30 <dolphm> #solved
18:34:33 <Anticimex> (as a deployer, hashicorp vault seems better than puppet / puppet-db for distributing secrets - or similar)
18:34:33 <ayoung> but...adding a key to the config file that we generate is probably easier than managing the repo...
18:35:10 <breton> the situation with requiring keys is also bad because the keys will be in the backend in the next cycle (i hope)
18:35:22 <ayoung> Anticimex, we are not even using the puppet mechanism here
18:35:45 <EmilienM> this is the link of my PoC https://review.openstack.org/366287 (it's only a start and unit tests will fail but save the URL)
18:36:27 * lbragstad starred
18:36:50 <dolphm> ayoung: EmilienM: seriously, hardcode tripleo to drop two credentials keys on disk that are just a bunch of nulls 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
18:37:03 <EmilienM> I also need to patch puppet-keystone and tripleo but I'll do it after my terrible dentist apt :(
18:37:14 <dolphm> ayoung: EmilienM: finish the upgrade, and then at your leisure when running newton, you can rotate new keys in with credential_setup
18:37:20 <ayoung> dolphm, that is essentially what EmilienM is saying, only actually generating a key instead.  Its the file managment that is the fireddrill here
18:37:26 <dolphm> or in a future release when you solve the syncing problem
18:37:36 <ayoung> but if Tripleo has to do it, everyone will have to, and keystone will have broken the world
18:37:54 <dolphm> ayoung: i assume the complexity in the file management is in coordinating the contents of the files?
18:38:00 <bknudson> it would be great if tripleo was reporting CI failures
18:38:02 <dolphm> not in writing a file with a known value
18:38:04 <lbragstad> i imagine osa is going to use the same mechanism that it uses for fernet rotation
18:38:22 <ayoung> dolphm, probably more important is to identify the existence of certain files and get them to the servers.
18:38:35 <ayoung> the contents are always in sync across all nodes
18:38:57 <dolphm> ayoung: /etc/keystone/credentials-keys/0 should always contain AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
18:39:00 <dolphm> ayoung: /etc/keystone/credentials-keys/1 should always contain AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
18:39:00 <bknudson> how do you re-encrypt the credentials with the new keys?
18:39:14 <dolphm> bknudson: keystone-manage credential_rotate does the db work to do that
18:39:36 <breton> the requirement to have the keys before keystone starts bugs me
18:40:00 <dolphm> bknudson: or keystone-manage credential_migrate? (help, lbragstad)
18:40:08 <breton> i already tried to address it in https://review.openstack.org/362785 with low-priority
18:40:50 <lbragstad> bknudson keystone-manage credential_migrate assumes you have keys on disk in order to re-encrypt your credentials
18:41:15 <lbragstad> bknudson https://gist.github.com/lbragstad/ddfb10f9f9048414d1f781ba006e95d1#encrypted-credential-key-management
18:41:18 <dolphm> breton: that's only for the fernet token provider?
18:41:25 <lbragstad> explained in more detail there ^
18:41:26 <EmilienM> ayoung: sorry I have to leave now, I'm back in 1h30
18:41:27 <dolphm> breton: not for the fernet credential provider
18:41:55 <breton> dolphm: yes. But i guess it would have to be done for credential provider when the keys move to backend.
18:42:27 <bknudson> lbragstad: neat
18:43:44 <stevemar> bknudson: will your time be able to handle credential encryption?
18:43:51 <ayoung> so we could run the encrypt with a null key
18:43:56 <stevemar> bknudson: i assume you guys are using fernet tokens now anyway
18:44:02 <ayoung> er
18:44:04 <ayoung> upgrade
18:44:13 <bknudson> stevemar: arrrsula supports fernet tokens.
18:44:23 <bknudson> not sure what's going to happen with this change.
18:44:34 <stevemar> bknudson: i'm assuming you'll use the same mechanisms then
18:44:39 <bknudson> nobody has reported any failures
18:44:40 <dolphm> ayoung: does that make sense?
18:45:10 <dolphm> ayoung: it lets you postpone the hard part, and we can document it as a minimum viable upgrade solution
18:45:15 <dolphm> (in keystone)
18:45:24 <ayoung> dolphm, I think you misunderstand what I am saying
18:45:38 <ayoung> we could have keystone operate using the null key until one is provided
18:45:51 <ayoung> if the key database is not populated
18:46:23 <ayoung> and deprecate that, too
18:46:45 <ayoung> dolphm, we just need to give people breathing room, not maintain the no-change upgrade forever
18:47:00 <lbragstad> why not just have tripleo drop two null keys on disk as /etc/keystone/credential-keys/0 and /etc/keystone/credential-keys/1 ?
18:47:06 <dolphm> i'm a little lost on how why supporting insecure upgrades needs to be keystone's responsibility
18:47:07 <ayoung> If we had landed this first thing in the cycle, I would be right there with you
18:47:22 <ayoung> it is purely the timing that leads me to want to compromise here
18:48:02 <breton> lbragstad: what would be the contents of the null keys? AAA...=?
18:48:22 <dolphm> breton: literally AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
18:48:28 <lbragstad> breton yeah
18:48:29 <dolphm> breton: it's the null byte base64 encoded
18:48:38 <dolphm> well, several null bytes :)
18:48:39 <ayoung> The insecurity was Keystone's in the first place. A Credential Database is a scary thing, and it should never have existed, certainly not the way it was implemented.  The encryption of credentials with a single key is still medicocre at best.
18:48:43 <lbragstad> or if there is a syncing mechanism in place they could be randomly generated
18:49:26 <breton> dolphm: lbragstad: :( why should tripleo know such implementation details about keystone?
18:49:31 <ayoung> It is not just Tripleo.  It is all of the installers that are going to be messed up buy this. Tripleo was just the one that reported it
18:50:02 <bknudson> we already knew about it because of the grenade change
18:50:32 <lbragstad> breton i thought we considered encryption keys as configuration
18:50:43 <dolphm> breton: this is something we can document "if you'd like to opt-out of securely storing credentials in your database, then put these two well-known keys in these two files on disk to operate insecurely"
18:51:49 <ayoung> dolphm, I know it seems simple, but until you've worked through the mechanisms, it is a change that has significant impact
18:52:14 <ayoung> "here are two file that have cryptographic info in them, but use defaults..."
18:52:18 <dolphm> yeah, it seems like this is an option that could have been implemented before this meeting was over
18:52:45 <ayoung> what I did in my POC for Fernet tokens  would also work for the credentials. it just would be world readable
18:53:26 <ayoung> We broke it, we own it.
18:54:02 <dolphm> ayoung: if you're implying that keystone broke tripleo, i don't think that's fair to say at all
18:54:18 <ayoung> dolphm, I am saying keystone broken Tripleo.
18:54:36 <dolphm> tripleo broke itself by attempting upgrades entirely blindly. we write release notes, documentation, examples, configuration docs, etc, for a reason
18:54:38 <ayoung> and doing so is OKish early in the development cycle
18:54:59 <ayoung> dolphm, Keystone broke every installer out there that uses this stuff
18:55:22 <ayoung> We essentially deprecated no-op with no lead time and no transition time
18:55:36 <ayoung> The end state, with encryption, is probably worth it
18:55:49 <dolphm> i've literally never heard of anyone attempting to upgrade without at least reading release notes
18:56:05 <ayoung> dolphm, it wasn't an upgrade
18:56:07 <ayoung> it was an install
18:56:18 <ayoung> that install used the credentials basckend
18:56:25 <stevemar> the install is fine, it's trying to use the credentials backend that failed
18:56:31 <ayoung> and that fails without the keys in place
18:56:33 <stevemar> it's tempest that fails
18:56:47 <stevemar> its the tests *after* install
18:57:17 <ayoung> As I said, it is probably the right thing to do, in the absolute sense.  Just that the timing sucks
18:57:19 <dolphm> ayoung: an installation of master?
18:57:32 <ayoung> dolphm, yep
18:57:39 <bknudson> IBM public cloud process is going to automate upgrading. We're not going to have people reading release notes on every change.
18:57:46 <ayoung> ++
18:58:03 <bknudson> at least that's the plan
18:58:06 <breton> why... do we write them then
18:58:20 <dolphm> bknudson: i'm all for automating it, but expect it to fail and to require maintenance to keep it working when changes to your moving target require additional steps
18:58:22 <breton> git rm -fr release_notes
18:58:28 <stevemar> breton: so when upgrades fail, we can point bugs to them
18:58:59 <lbragstad> some folks rely on them for their upgrade process
18:59:01 <bknudson> dolphm: yes, we'll have tests in place and do expect to do maintenance when the tests fail.
18:59:13 <stevemar> for most deployments, it'll be a one-time step
18:59:23 <breton> 1 minute left
18:59:25 <stevemar> that's the point of the scripts in grenade
18:59:44 <stevemar> tripleo is unfortunately, not a routine deployment of openstack
19:00:04 <stevemar> otherwise it would be a 1 line command
19:00:09 <stevemar> #endmeeting