13:02:34 <HenryG> #startmeeting neutron_db
13:02:35 <openstack> Meeting started Tue May 27 13:02:34 2014 UTC and is due to finish in 60 minutes.  The chair is HenryG. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:02:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:02:38 <openstack> The meeting name has been set to 'neutron_db'
13:02:51 <HenryG> #topic Introduction
13:03:11 <HenryG> salv-orlando has called into question the need for this weekly meeting and forming a "subteam".
13:03:24 <HenryG> The TC identified the Neutron DB migrations has the #0 priority Gap (higher than the #1 priority Gap) for nova-network parity.
13:03:31 <HenryG> #link https://wiki.openstack.org/wiki/Governance/TechnicalCommittee/Neutron_Gap_Coverage
13:03:40 <HenryG> We need to make sure this work gets done correctly and on time.
13:03:53 <HenryG> I agree it's not really a subteam. If we are efficient the meetings and "subteam" can be dissolved soon.
13:03:59 <mestery> I think a sub-team operating until #0 is fixed is warranted HenryG.
13:04:46 <HenryG> Sure. Let's keep the red tape to a minimum though.
13:04:50 <HenryG> I'll try to stay out of everyone's way while tracking progress and removing obstacles where I can.
13:05:01 <HenryG> Do not wait on me or postpone work until the next meeting if there is something that needs to be hashed out.
13:05:13 <HenryG> Please use the mailing list and IRC effectively.
13:05:22 <HenryG> Allright, let's go through the agenda.
13:05:28 <HenryG> #link https://wiki.openstack.org/wiki/Meetings/NeutronDB#Agenda
13:05:38 <HenryG> #topic Plan/Design
13:05:49 <HenryG> The plan we discussed at the summit is written in the etherpad
13:05:56 <HenryG> #link https://etherpad.openstack.org/p/neutron-db-migrations
13:06:11 <HenryG> I am writing a spec for the work.
13:06:18 <HenryG> #action HenryG to write a spec and submit to neutron-specs.
13:06:29 <jlibosva> oh, I was active and tried to write it: https://review.openstack.org/#/c/95738/
13:06:29 <HenryG> It is not clear to me what the final design will be.
13:06:34 <akamyshnikova> I've got some drafts on this topic.
13:06:48 <jlibosva> it's really a very first draft though
13:07:38 <HenryG> jlibosva: Great! No problem. We can use yours. We'll build on it until it is OK.
13:08:08 <akamyshnikova> I create a kind of plan where are shown some options #link https://docs.google.com/a/mirantis.com/document/d/10p6JKIQf_rymBuNeOywjHiv53cRTfz5kBg8LeUCeykI/edit
13:08:35 <HenryG> Sorry I missed it. My laptop was stolen and I lost track of some happenings in the community for a week or so.
13:09:15 <HenryG> akamyshnikova: Thanks for that!
13:10:16 <HenryG> So let's continue discussing on the ML and IRC about those options.
13:10:17 <rpodolyaka> akamyshnikova: 2 looks hard to implement (as you would need to write this non-trivial conditional logic in SQL)
13:10:54 <rpodolyaka> akamyshnikova: we could probably assume we stick to online migrations for this one
13:11:22 <rpodolyaka> akamyshnikova: 1 is doable, but I don't think we should go through all these steps
13:11:31 <akamyshnikova> I created some test changes too. But today I was told about one more option that I didn't check how to use, it is number 3 in the list, but it could be the best decision.
13:11:59 <rpodolyaka> as still you would need to compare every single column of every single table and do an 'update' (which can be: change type, change default value, etc)
13:12:07 <HenryG> Excuse my ignorance, but other than developers, who uses offline migrations?
13:12:24 <rpodolyaka> HenryG: in theory, DBAs :)
13:12:59 <rpodolyaka> akamyshnikova: so what I'm trying to say here is that all this is already implemented in alembic
13:13:11 <rpodolyaka> so alembic provides support for autogeneration of migration scripts
13:13:35 <rpodolyaka> it basically takes your metadata (which you defined in terms of model definitions) and compares it to the current state of the database
13:13:51 <rpodolyaka> (it reflects the DB schema by the means of SQLAlchemy)
13:14:08 <rpodolyaka> then it compares two SQLAlchemy MetaData instances and generate a list of differences
13:14:26 <rpodolyaka> new indexes, new tables, changes of types, default values, etc
13:14:32 <jlibosva> rpodolyaka: will autogenerate work after removing register_models() ?
13:14:46 <jlibosva> rpodolyaka: sorry, stupid question. Disregard
13:14:49 <rpodolyaka> jlibosva: yep, it has nothing to do with it
13:15:01 <akamyshnikova> rpodolyaka, yeah, I see this. I think I will research this option tomorrow.
13:15:42 <rpodolyaka> then, for each difference it generates a corresponding alembic operation
13:16:00 <rpodolyaka> op.alter_column(), op.create_constraint(), etc
13:16:12 <rpodolyaka> I think, this will cover 90% of problems we have
13:16:20 <rpodolyaka> there is probably just a few cases left
13:16:45 <rpodolyaka> e.g. ENUMs in PostgreSQL (AFAIK, these are not reflected by SQLAlchemy)
13:17:11 <rpodolyaka> not sure if have migrations changing ENUMs though
13:17:22 <rpodolyaka> so I'd try to use alembic for this first
13:17:32 <HenryG> rpodolyaka: And it will generate a "downgrade" too, for those who chicken out and want to go back?
13:17:56 <rpodolyaka> HenryG: haven't checked that, but probably yes
13:18:12 <akamyshnikova> rpodolyaka, there are migrations that change enums
13:19:03 <jlibosva> aren't all/most of current migration scripts created by autogenerate?
13:19:09 <rpodolyaka> akamyshnikova: ah... so we should either add fixes for those manually or teach SQLAlchemy/alembic how to treat ENUMs
13:19:15 <rpodolyaka> jlibosva: yep
13:19:43 <HenryG> How does this fit with salv-orlando's idea here: http://lists.openstack.org/pipermail/openstack-dev/2014-May/035567.html ?
13:21:17 <rpodolyaka> I like salv-orlando's idea that this should be done in a migration script rather than by a separate call
13:21:40 <HenryG> There are many good ideas, but we need to converge.
13:22:25 <rpodolyaka> should not be a problem, but I'll need to take a look at alembic API first. there are a few cases, when you have to call alembic commands instead of doing calls programatically from Python
13:22:38 <jlibosva> it could be a separate executable module with ability to import and re-use in neutron-db-manage
13:23:14 <jlibosva> this way you can heal on demand and be part of automatic upgrade
13:23:29 <rpodolyaka> fwiw, this is just my two cents. I believe, this will be easier to implement than write schema comparison from scratch
13:24:36 <akamyshnikova> I think that I need to look deeply in alembic.
13:24:41 <HenryG> rpodolyaka: It certainly makes sense to me.
13:25:29 <rpodolyaka> we use alembic to do schema comparison in oslo-db so that we can show differences between your models definitions and migration scripts
13:26:14 <HenryG> #action rpodolyaka, akamyshnikova, jlibosva to investigate and propose solution using alembic for schema comparison and base script.
13:26:14 <akamyshnikova> this script is used in my work about sync models and migrations
13:26:23 <rpodolyaka> so far it worked good for us (just a few tweaks needed, e.g. tell it to compare server default values, or tell it that TINYINT(1) is a valid BOOLEAN column for MySQL, and things like that)
13:26:37 <HenryG> Let's keep everyone in the loop with discussions in the ML and IRC.
13:27:02 <rpodolyaka> ML is probably easier to follow
13:28:38 <akamyshnikova> I've another big question. What should we do with new migrations that appear  in Neutron? Should they be unconditional from very beginning?
13:29:15 <HenryG> akamyshnikova: My understanding is that we want to start that now.
13:29:29 <rpodolyaka> +1
13:29:33 <jlibosva> -1. I don't think so. They would fail because most of them are dependent on previous migrations.
13:29:41 <HenryG> akamyshnikova: By "now" I mean from Icehouse.
13:29:43 <jlibosva> it will break deployment with devstack
13:30:15 <rpodolyaka> jlibosva: hmm, what would it break new deployments?
13:31:17 <jlibosva> e.g. you deploy neutron with ml2 and someone makes a change in column for other specific plugin. Table would be missing, migration will fail
13:31:36 <jlibosva> because this particular migration would be executed
13:32:51 <akamyshnikova> If it should be started from now. I think that it should be announced for everyone.
13:32:57 <jlibosva> I'd suggest to keep conditional migrations for now until we have something that brings idempotent operation above db
13:33:01 <rpodolyaka> won't we runn ALL migrations unconditionally?
13:33:16 <HenryG> My understanding is you would have the old conditional migrations until Icehouse. Then the healing script. Then all tables will be present and all migrations from Icehouse on are unconditional.
13:33:35 <jlibosva> since we want to get rid of conditional migration, there will be other patch removing conditions themselves.
13:33:38 <akamyshnikova> jlibosva, where you find such changes of columns?
13:34:07 <jlibosva> HenryG: agreed, question is what to do with current patches coming to master
13:34:46 <HenryG> jlibosva: If I understood Mark correctly, we "undo" them and replace them with unconditional ones.
13:35:19 <HenryG> We undo the ones that are committed in the Juno cycle.
13:35:53 <jlibosva> yes, that was my idea. But not starting using '*' in all patches going to master
13:35:57 <HenryG> Once we have laid down the code for uncondtional migrations, we no longer allow patches with conditional migrations.
13:36:32 <jlibosva> e.g. https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/versions/35c7c198ddea_lbaas_healthmon_del_status.py#L46 if lbaas is not present, patch like this won't pass deployment
13:36:41 <jlibosva> HenryG: agreed
13:38:17 <akamyshnikova> yes, in this case. But if some new tables create?
13:39:07 <jlibosva> akamyshnikova: you mean not using "if not migration.should_run(active_plugins, migration_for_plugins):" at all?
13:39:48 <jlibosva> akamyshnikova: if so, then I agree. It's a good thing. Good poing.
13:39:51 <jlibosva> point
13:40:24 <akamyshnikova> If some new tables created - yes
13:40:41 <jlibosva> ok, +1
13:41:21 <HenryG> I would like to move on in the agenda.
13:41:26 <akamyshnikova> of course if migration is changing table it should be conditional for now.
13:41:40 <HenryG> We can discuss more at the end or in the ML/IRC.
13:41:49 <akamyshnikova> ok
13:42:03 <HenryG> #topic Blueprints
13:42:25 <HenryG> What blueprints do we have? Here is one:
13:42:31 <HenryG> #link https://blueprints.launchpad.net/neutron/+spec/neutron-robust-db-migrations
13:42:57 <jlibosva> HenryG: here is my draft https://review.openstack.org/#/c/95738/
13:43:11 <akamyshnikova> this should be updated neutron-robust-db-migrations
13:43:22 <HenryG> jlibosva: "Blueprint" vs. "Spec". :)
13:43:23 <akamyshnikova> as it was only for service plugins
13:43:54 <jlibosva> HenryG: aha, it's difference. sorry :)
13:43:59 <akamyshnikova> I can do that tomorrow. If there is no other bp
13:44:40 <HenryG> I think one spec is enough, but it may refer to a couple of blueprints.
13:45:39 <HenryG> I think we need a blueprint for the healing script.
13:46:07 <HenryG> #action HenryG to file BP for healing script.
13:46:15 <HenryG> #topic Testing
13:46:25 <HenryG> Currently there is no unit testing for migrations.
13:46:29 <HenryG> But akamyshnikova is syncing some framework for this from oslo.
13:46:35 <HenryG> #link https://review.openstack.org/76519
13:46:41 <HenryG> Let's see if we can use this to improve migration unit testing.
13:47:03 <HenryG> I will discuss the "if and how" of functional testing with marun after the meeting.
13:47:08 <HenryG> #action HenryG to discuss functional testing with marun.
13:47:17 <HenryG> Most important in the topic of testing is tempest, specifically grenade coverage.
13:47:28 <HenryG> Can someone summarize the current status and plan? jlibosva?
13:48:20 <jlibosva> currently icehouse->master is broken. There are two issues I'm aware of.
13:49:06 <jlibosva> there is an issue with mounting some image for swift, probably because some neutron service is not stopped. I have a testing patch for getting more info https://review.openstack.org/#/c/94351/
13:49:42 <jlibosva> second issue is probably more reproducible wrt policy - I don't know much about this yet, looked at it yesterday afternoon: http://logs.openstack.org/79/95479/2/check/check-grenade-dsvm-neutron/89b999e/logs/new/screen-q-svc.txt.gz?#_2014-05-26_10_49_46_452
13:50:05 <jlibosva> if someone is familiar with such an error, I would be glad to get help
13:50:12 <HenryG> jlibosva: Will the conversion to unconditional migrations help to fix these?
13:50:46 <jlibosva> HenryG: I don't think so. It's rather related to Neutron configuration than db
13:51:08 <HenryG> jlibosva: OK, thanks.
13:51:33 <jlibosva> Also I see time to time failure on stable/icehouse tempest even before upgrade starts
13:51:43 <jlibosva> seems like the stable is not that much stable :)
13:51:49 <HenryG> Please everyone who can, help jlibosva to look at these issues.
13:51:54 <jlibosva> thanks
13:51:57 <HenryG> Thanks jlibosva
13:52:21 <HenryG> #topic Division of labour
13:52:35 <HenryG> Code contributors (so far): akamyshnikova, salv-orlando, jlibosva, HenryG
13:52:43 <HenryG> Core mentors: markmcclain, salv-orlando, marun
13:52:59 <HenryG> New: rpodolyaka :)
13:53:11 <HenryG> Personally I am no expert on databases, so I would prefer to work more on the python side of things.
13:53:42 <HenryG> Any objections or additions?
13:54:04 <akamyshnikova> I think i will continue my researching for creating a script
13:54:16 <rpodolyaka> akamyshnikova: I'll help you on alembic side :)
13:54:33 <HenryG> jlibosva: focus on grenade/tempest?
13:54:56 <jlibosva> HenryG: ok, I'd like to also participate on the healing script
13:55:01 <akamyshnikova> rpodolyaka, thank a lot! I rely on your help
13:55:18 <HenryG> jlibosva: of course
13:56:37 <HenryG> jlibosva: we are accustomed to turning to you (and salv-orlando) for grenade wisdom :)
13:57:31 <HenryG> #topic Mid-cycle sprint
13:57:37 <HenryG> Just wanted to ensure everyone is aware of the mid-cycle sprint to focus on nova-networking parity.
13:57:43 <HenryG> #link https://etherpad.openstack.org/p/neutron-juno-mid-cycle-meeting
13:58:37 <akamyshnikova> yes, I've heard about it
13:59:21 <HenryG> For those who can't attend, I will try to set up some interactive thing with a webcam.
14:00:15 <HenryG> Everyone try to squeeze some travel funds out of their bosses. :)
14:00:31 <rpodolyaka> :)
14:00:39 <HenryG> #topic Open discussion
14:00:56 <sc68cal> HenryG: we have the ipv6 meeting
14:01:11 <HenryG> #endmeeting