18:00:24 <SlickNik> #startmeeting trove
18:00:25 <openstack> Meeting started Wed Sep 10 18:00:24 2014 UTC and is due to finish in 60 minutes.  The chair is SlickNik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:28 <openstack> The meeting name has been set to 'trove'
18:00:29 <grapex> o/
18:00:39 <mattgriffin> o/
18:00:44 <amrith> ./
18:00:46 <schang> o/
18:00:51 <SlickNik> Agenda at:
18:00:53 <kevinconway> o/
18:00:54 <SlickNik> #link https://wiki.openstack.org/wiki/Meetings/TroveMeeting#Agenda_for_Sep_10
18:01:51 <SlickNik> giving folks a few minutes to trickle in before we start.
18:03:13 <vipul> o/
18:03:55 <SlickNik> #topic  Clusters Migration and Downgrades
18:04:26 <SlickNik> schang: floor is yours
18:04:34 <schang> ok ...
18:04:54 <amcrn> o/
18:05:07 <schang> A little while ago, I discovered that Trove do not have a migration test script when I was working on a bug.
18:05:12 <cp16net> o\
18:05:14 <vgnbkr> o/
18:05:23 <schang> So I ported the script from Nova into Trove.
18:05:55 <schang> When I run my test script against the Trove migration script, it reveals issues in the downgrades.
18:06:57 <schang> We need to decide on what we actually should be testing in the migration test script.
18:07:12 <schang> The agenda item outlines a few options we have to move forward.
18:07:15 <schang> Any thoughts?
18:07:58 <dougshelley66> o/
18:07:59 <peterstac> o/
18:08:27 <georgelorch> o/
18:08:44 <SlickNik> schang: I don't believe HP actually uses any of the downgrade scripts (while deploying).
18:09:23 <dougshelley66> SlickNik, the key question is whether we want to test that infra. When schang fixed that bug we decided to implement upgrade/downgrade testing
18:09:23 <SlickNik> would be good to get a sense if other folks use them at all.
18:09:38 <vgnbkr> What is "downgrade" supposed to do in a wider OpenStack context?   What is the expected state after a downgrade?
18:09:44 <amcrn> when this was first being discussed, the question that arose was: should the purpose of a downgrade script be for development purposes, or should it be a means of temporarily downgrading in a production deployment (i.e. rollback)
18:09:58 <dougshelley66> but that testing isn't really going to work because folks are introducing downgrade scripts with issues
18:10:22 <schang> SlickNik: I think the downgrade's purpose should be for devs to reset db's.
18:10:54 <schang> Therefore, I think it's OK to clean up tables on downgrade.
18:11:41 <SlickNik> amcrn: good point if that's the case — we're not going to catch any of the production issues during a migrations test on devstack, so I'd question the intent of testing downgrades in a devstack install.
18:12:30 <amcrn> SlickNik: +1
18:12:44 <grapex> Testing the upgrades with grenade are bad enough- I can't imagine how much of a hassle it would be to test downgrades too
18:12:45 <cp16net> i agree as well we do not use downgrades
18:12:57 <peterstac> I think that in a production environment, if a downgrade is necessary then it would probably entail replacing the db with a backup taken before the upgrade
18:13:25 <vgnbkr> Looking at the OPenstack docs, a downgrade is only supposed to be done if no changes have been made to the system after the upgrade.  That being the case, the downgrade scripts should remove tables created by the upgrade scipts.
18:13:34 <cp16net> i think that would be the safe way peterstac
18:13:45 <grapex> Also a downgrade implies something doesn't work in the upgrade. If the upgrade itself produced a state that was undesirable, what's the hope the downgrade scripts is totally compatible with whatever the upgrade did?
18:13:53 <kevinconway> yeah, you can't actually run some of the downgrade scripts in a production db
18:14:01 <kevinconway> imaging rolling back datastore versions and types
18:14:38 <SlickNik> vgnbkr: link?
18:15:06 <schang> Seems like all of us is aggreeing that downgrade is for the convenience of devs to reset db.
18:15:09 <vgnbkr> SlickNik: I deleted the tab - I'll see if I can find it again.
18:15:32 <amrith> schang, I don't know that is the case
18:15:40 <amrith> if it is a documented feature of openstack.
18:15:41 <vgnbkr> http://docs.openstack.org/openstack-ops/content/ops_upgrades-roll-back.html
18:15:55 <cp16net> so there is another cmd to reset the db called wipe_db
18:16:04 <cp16net> DONT DO THAT IN PROD :)
18:16:20 <grapex> cp16net: But what if I really wanted to?
18:16:33 <grapex> Is there a way to modify that command to also save all of my user's data?
18:16:33 * cp16net has evil laugh
18:17:00 <SlickNik> vgnbkr: According to that link — "Restore databases from the grizzly-db-backup.sql backup file that you created with the mysqldump command during the upgrade process"
18:17:12 <grapex> wipe just implies cleaning. If I bought wipes for my car and it stopped running after I used them I'd feel very upset!
18:17:19 <grapex> Ok I'll stop now.
18:17:27 <SlickNik> vgnbkr: So I don't think they're advocating running the dowgrade scripts either.
18:17:32 <kevinconway> schang: when you hit issues with downgrades, was that during/after a failed upgrade?
18:17:37 <amcrn> the statement "Make sure that you did not make any state changes after attempting the upgrade process: no new instances, networks, storage volumes, and so on." is one heck of an assumption, that nearly no deployer can make.
18:17:49 <amrith> amcrn, that statement is funny
18:18:00 <amrith> because it comes after the one that says you found a problem not found in testing
18:18:02 <peterstac> SlickNik: Correct - it says "Restore databases from the <backup_name>.sql backup file that you created"
18:18:13 <kevinconway> amcrn: well, as soon as you use a feature that required the upgrade you can no longer run the downgrade without causing lossy changes
18:18:14 <schang> kevinconway, I hit issues when I test downgrade from latest all the way back to v1.
18:18:20 <cp16net> i agree i rather just keep moving forward
18:18:50 <vgnbkr> The key point is that downgrade is for correcting a failed upgrade, not for deciding 3 months later that you don't like it.
18:19:03 <amcrn> i think we're all in agreement that a db_downgrade is an awful idea in production, and irrespective of whether the official literature claims it *can* be safe in some scenarios, it shouldn't be trusted. therefore, i think schang is correct that a downgrade should drop.
18:19:20 <amcrn> might as well make the development-path a happy path.
18:20:00 <dougshelley66> do folks agree that having a test that runs the upgrade and downgrade code is a useful thing?
18:20:12 <amcrn> dougshelley66: i don't
18:20:14 <cp16net> i dont see much use of it
18:20:22 <amcrn> to me db_downgrade is a dev utility
18:20:24 <SlickNik> amcrn: +1 I agree I would not recommend running thees downgrade scripts on any live deployment.
18:20:48 <amrith> even if it is a dev thing, shouldn't we at least test it?
18:20:49 <cp16net> amcrn: i've never used it
18:20:53 <amrith> and make sure it works
18:21:01 <amrith> if not, let's eliminate it.
18:21:04 <schang> amcrn: +1 on dropping the tables on downngrade. That will make the test simple.
18:21:44 <vgnbkr> Do we have the option to eliminate downgrade, or is it a mandated feature?
18:21:45 <SlickNik> Agreed. I'm fine with schang updating the downgrade scripts to drop data, and do what needs to be done to return the system to what state it was in before the upgrade.
18:22:06 <dougshelley66> SlickNik, and also put up the automated test?
18:22:14 <SlickNik> We just have to be clear that these downgrade scripts are not an operator feature, but rather a dev convenience for testing.
18:22:32 <cp16net> who has ever used the downgrade ?
18:23:11 <kevinconway> my downgrade is typically drop the db and run the migrate up
18:23:18 <kevinconway> for dev, of course
18:23:21 <dougshelley66> so why do we have all the code?
18:23:23 <cp16net> exactly.
18:23:23 <dougshelley66> that doesn't work
18:23:25 <kevinconway> i try not to do that in production too often
18:23:31 <dougshelley66> and is untested
18:23:40 <peterstac> cp16net: well, you'd use downgrade when you're testing an upgrade, right?
18:23:45 <cp16net> my point is why does it need to be tested?
18:23:48 <SlickNik> cp16net: I've run it occasionally to check that upgrades from multiple versions work when I'm authoring a new migrate script.
18:24:19 <cp16net> SlickNik: i'm surprised..
18:24:22 <cp16net> :-P
18:24:39 <dougshelley66> so my view is we either test the code or get rid of it and don't test it
18:24:48 <amrith> dougshelley66, +1
18:25:01 <amrith> well, if you get rid of it, it would be hard to test it.
18:25:06 <amrith> but +1 nonetheless.
18:25:28 <grapex> SlickNik: So is the utility you get that you *can* create some instance data or other stuff, run the downgrade, then test upgrade multiple times?
18:25:44 <grapex> Even though officially the downgrades shouldn't preserve existing data?
18:25:48 <amrith> grapex, you can't necessarily do that
18:25:55 <amrith> downgrades don't clean up data
18:25:58 <amrith> the drop tables
18:26:07 <amrith> they don't necessarily delete rows added in existing tables
18:26:14 <grapex> I guess I really don't see what a downgrade buys even devs over just dropping the tables and re-creating them
18:26:27 <amcrn> grapex:i manually drop tables fwiw
18:26:44 <SlickNik> grapex: Nope, don't create anything — Say you're authoring migrate script v10. I've used it to verify that I can upgrade correctly from multiple v3, v5, v7, and so on.
18:26:53 <robertmyers> typically a downgrade is only used during a deploy to revert a change
18:26:54 <grapex> amcrn: At Rax we all be dropping tables like gangsters
18:27:04 <robertmyers> so there shouldn't be data in it
18:27:06 <amcrn> grapex: pft, we hired Bobby Droptables
18:27:12 <grapex> amcrn: lol
18:27:20 <grapex> SlickNik: So is this when you're doing grenade testing?
18:27:41 <grapex> or is this all by hand, running each command?
18:28:10 <schang> Ok ... should I drop the tables on downgrades?
18:28:21 <SlickNik> grapex: nope, last time I did it was when I wrote a new migration script (backups I think) by hand.
18:29:05 <grapex> SlickNik: So you wrote it by hand- it sounds like we're ok w/ dropping tables
18:29:28 <SlickNik> grapex: Yes, that's what I'm hearing.
18:29:38 <schang> +1?
18:30:09 <SlickNik> +1 to dropping the tables to leave the db at the exact state you found it at before the upgrade.
18:30:30 <schang> ok, thanks SlickNik, I'll submit a new patch.
18:30:31 <SlickNik> schang: that should solve your issues, correct?
18:30:40 <schang> SlickNik: yes
18:31:14 <schang> That's all I have for this agenda item. We can move on.
18:31:43 <SlickNik> Okay, thanks schang!
18:32:01 <schang> thx
18:32:01 <SlickNik> #topic trove-specs is underway for Kilo
18:32:59 <SlickNik> So I've been working on getting a trove-specs repo going.
18:33:31 <cp16net> so all BPs will need a spec review
18:33:33 <SlickNik> And that should be done in the next couple of days.
18:33:37 <cp16net> ?
18:34:02 <cp16net> all BPs going into kilo
18:34:05 <SlickNik> So this is just a heads up — any new BPs starting week after next will need a spec attached.
18:34:20 <cp16net> ok
18:35:05 <SlickNik> cp16net: Yes, I believe it's a good idea to retroactively require even BPs that we approved to have a spec.
18:35:19 <SlickNik> So all Kilo BPs will be in the spec repo.
18:35:36 <SlickNik> (There's only the SUSE one really)
18:35:42 <SlickNik> What do folks think?
18:35:49 <amrith> SlickNik, +1
18:35:50 <kevinconway> do we write them in .rst format?
18:35:52 <amcrn> SlickNik: sounds good.
18:35:55 <kevinconway> i'm new the specs-repo
18:35:59 <peterstac> SlickNik: +1
18:36:02 <dougshelley66> I think that all the BPs for kilo should be in the spec repo even if they were wiki before
18:36:22 <SlickNik> kevinconway: Yes, it should be in rst format
18:36:45 <SlickNik> FWIW, I have the specs repo seed at: https://github.com/SlickNik/trove-specs
18:37:01 <SlickNik> Just need to submit the infra changes to get that sucked into the openstack repos
18:37:41 <SlickNik> But you guys can start taking a look at that location for format, template, etc.
18:37:47 <amcrn> SlickNik: https://github.com/SlickNik/trove-specs/blob/master/specs/template.rst should go a long way in making our blueprints more robust, i'm a fan.
18:38:19 <SlickNik> #link https://github.com/SlickNik/trove-specs
18:38:32 <grapex> amcrn: +1
18:38:44 <SlickNik> I'll let the channel know when the repo made its way to openstack/trove-specs
18:39:18 <cp16net> coo
18:39:18 <SlickNik> #info All Kilo blueprints will require a link to a spec in trove-specs
18:39:48 <amrith> and the review process will be through gerrit, right?
18:40:03 <SlickNik> Also this link talks a bit about about the process —https://wiki.openstack.org/wiki/Blueprints#Spec_.2B_Blueprints_lifecycle
18:40:05 <SlickNik> #link https://wiki.openstack.org/wiki/Blueprints#Spec_.2B_Blueprints_lifecycle
18:40:09 <SlickNik> amrith: correct
18:40:19 <amrith> tx
18:40:49 <SlickNik> That's all I had for this. Will keep you guys updated on the progress.
18:41:07 <SlickNik> Any questions?
18:41:26 <SlickNik> .
18:41:29 <cp16net> anyone have a screw driver?
18:41:38 <SlickNik> #topic Open Discussion
18:42:24 <dougshelley66> cp16net, the drink?
18:42:34 <cp16net> no phillips tool
18:42:41 <cp16net> need one to fix my monitor
18:42:43 <cp16net> :-P
18:42:51 <amrith> the one you broke?
18:43:07 <amrith> laptop monitor
18:43:24 <amrith> I don't think SlickNik meant this when he said "open discussion" ;)
18:43:46 <SlickNik> lol, Anything else?
18:44:00 <amrith> going once
18:44:58 <SlickNik> going twice
18:45:10 <amrith> SOLD!
18:45:11 <SlickNik> #endmeeting