19:00:03 <dripton> #startmeeting DB
19:00:04 <openstack> Meeting started Thu Jul 11 19:00:03 2013 UTC.  The chair is dripton. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:07 <openstack> The meeting name has been set to 'db'
19:00:08 <dripton> Who's here for the DB meeting?
19:00:47 <rpodolyaka1> Hello, guys!
19:00:54 <vsergeyev> hi All
19:01:07 <dripton> Is boris-42 around?
19:01:14 <boris-42> Hi all
19:01:15 <dripton> russellb said he'd come
19:01:19 <boris-42> vsergeyev hI
19:01:39 <dripton> #topic sqlalchemy-migrate vs. alembic
19:01:42 <boris-42> vsergeyev pls remove code that is connected with db archiving from our sola utilites
19:01:43 <russellb> o/
19:02:06 <dripton> I don't know if you guys saw the thread on the openstack-dev ML but sqlalchemy-migrate is looking for a new maintainer.  So we have to choose.
19:02:07 <boris-42> sqlalchemy-migrate is death and it sucks..
19:02:08 <boris-42> sory
19:02:09 <boris-42> sorry
19:02:16 <dripton> Do we want to maintain it or do we want to switch to Alembic?
19:02:21 <boris-42> no
19:02:21 <russellb> so when can we switch?  :-)
19:02:38 <boris-42> russellb at the moment we are switching in ceilometer
19:02:50 <russellb> neutron, at least, already uses alembic
19:02:55 <boris-42> russellb there is not so much bugs in migrations and not so much migrations and models
19:02:58 <dripton> How many migrations does ceilometer have?
19:03:02 <boris-42> 10
19:03:13 <boris-42> But we are working on step by step solution
19:03:36 <dripton> okay, so the hope is to finish that first and hope it scales to Nova after?
19:03:38 <boris-42> so that in one moment we have migrations with alembic and sql migate
19:03:57 <boris-42> dripton yes we will continue our work in all projects
19:04:09 <boris-42> cinder, glance, nova
19:04:11 <boris-42> at least
19:04:48 <boris-42> russellb what do you think about switching to alembic after fixing current models and migrations?)
19:04:56 <mordred> should we take over sqlalchmey-migrate anyway?
19:05:11 <mordred> our stable branches will still use it
19:05:11 <boris-42> mordred I would strongly preferred to do that
19:05:26 <boris-42> only old stable branches
19:05:36 <russellb> the issues with models/migrations are separate from which tool we're using right?
19:05:53 <boris-42> yes but we are not able to switch until we fix them
19:06:10 <russellb> right, so makes sense to keep doing that?
19:06:11 <boris-42> we should know exactly what our migrations produces
19:06:15 <mordred> sure. but still, it'll be at least a year before we as a project could possibly not have any needs for sqla-migrate... that seems like too long for it to be abandonware
19:06:26 <russellb> mordred: that's a good point...
19:06:29 <russellb> mordred: as much as i hate it
19:06:50 <dripton> mordred: I agree but I hate to volunteer to maintain code that's 1. bad 2. has lots of open bugs on it 3. not compatible with SQLAlchemy 0.8
19:06:50 <mordred> russellb: and then, once we stop caring - if someone else wants to care, it's not like we don't have the infrastructure to let them
19:07:00 <boris-42> mordred it is not enough big reason to shoot your legs..
19:07:18 <mordred> well, we probably need to get it up to sqlalchemy 0.8 - I think 0.8 is kinda inevitable
19:07:33 <mordred> how bad is it? /me cringes
19:07:35 <dripton> there is allegedly a patch on github that brings it to 0.8, but I haven't tested it.
19:07:42 <mordred> cool. well, that's a start
19:07:49 <boris-42> mordred it is f**cking terrible=)
19:07:56 <boris-42> i hate it=)
19:08:01 <dripton> It's ugly but not huge.  So at least it's a small about of ugly.
19:08:01 <mordred> boris-42: excellent. my favorite kind of terrible code
19:08:30 <dripton> I do not want to volunteer to maintain it but if someone else does they have my full support.  :->
19:08:37 <mordred> I'll work with zigo to get it moved somewhere. if we never patch it, then it's no different than now
19:08:44 <mordred> but if we do, then great
19:09:04 <boris-42> at moment we would like to write some monkey patches
19:09:05 <dripton> What's wrong with leaving it at code.google.com?  It's been there a long time and there are many bugs there.  Moving it might be bad.
19:09:26 <russellb> if we're maintaining it, it needs to be in our infrastructure
19:09:31 <boris-42> dripton mordred sqlalchemy-migrate is bad thing
19:09:32 <mordred> yup
19:09:38 <boris-42> bad bad bad very bad thing
19:09:53 <russellb> boris-42: but we're stuck using it for at least another year
19:09:54 <boris-42> and there is no communtiy
19:10:04 <russellb> that's reality
19:10:08 <dripton> if we do that then we should consider moving all those bug reports to launchpad so we don't lose them.
19:10:22 <mordred> russellb: I agree.
19:10:24 <boris-42> btw
19:10:27 <mordred> dripton: good point
19:10:29 <boris-42> there is some trick
19:10:37 <boris-42> we could back port
19:10:40 <boris-42> all migrations
19:11:06 <boris-42> if they do the same thing, and I will try to make garantie of that
19:11:17 <boris-42> we could just replace it all in stable branches
19:11:40 <boris-42> and use in stable branches Alembic
19:11:42 <russellb> that's not going to fly ...
19:11:46 <boris-42> why?
19:11:54 <boris-42> just one patch set
19:11:56 <russellb> nobody is going to be ok making such a drastic change to stable branches
19:12:08 <russellb> well, most poeople :-)
19:12:11 <russellb> obviously you're ok with it, ha
19:12:15 <boris-42> it could be really good tested
19:12:23 <russellb> i really don't think that's an option
19:12:27 <dripton> yeah, we have to keep it alive for grizzly. If we can get to Alembic by havana final we've at least limited our pain.
19:12:29 <boris-42> on real data step by step
19:12:32 <russellb> we don't let *any* new features in stable branches
19:12:46 <boris-42> it is not *new feature*
19:12:58 <russellb> if we switched by havana, then it's less than a year until we're no longer using it
19:13:22 <boris-42> btw why is it more safe to switch it in havana
19:13:32 <boris-42> and it is not safe to switch it in grizzly stable branch?
19:13:36 <russellb> because it's a new release, and that's when we make disruptive changes?
19:13:37 <dripton> Havana is still in development.  Nobody has production clusters on it yet.
19:13:55 <boris-42> russellb let me try to describe situation
19:14:03 <boris-42> russellb you have could that runs grizzly
19:14:11 <boris-42> russellb you have done a tons of migrations
19:14:23 <boris-42> russellb then you are trying to move to havana
19:14:32 <russellb> i'm sorry, but there's no way you can convince me that switching this in stable/grizzly is ok
19:14:33 <boris-42> russellb and half of migrations are replaced=)
19:15:14 <boris-42> ok, I just would like to explain situation
19:15:39 <boris-42> why I am so afraid to switch to something else without fully tested migrations and synced models with tons of tests=)
19:15:50 <boris-42> even in Havana
19:16:13 <boris-42> we should be very careful with this things =)
19:16:16 <boris-42> such*
19:16:19 <dripton> We all agree that we should be really careful even in Havana.  But there's no such thing as careful enough for Grizzly stable, since we promise not to add new features there.
19:16:32 <boris-42> Ok
19:16:52 <boris-42> I agree that it could be dangerous =)
19:18:06 <dripton> So do we have consensus that we're moving to Alembic in Havana, and taking over maintainership of sqlalchemy-migrate for legacy use?
19:18:44 <russellb> i personally don't necessarily have a strong tie to Alembic.  It's just obvious that we need to *not* be using migrate
19:19:05 <dripton> I don't know of a third option unless we want to write our own.  Which would be well behind Alembic.
19:19:31 * russellb nods
19:19:35 <boris-42> Alembic is good because it has a community around it
19:19:44 <russellb> and it's cool if there are 10 steps that need to be completed before we can move
19:19:48 <boris-42> And it is really much simpler to extend
19:19:49 <russellb> but as long as it's on our radar for asap
19:19:58 <russellb> using a dead upstream is a really really bad position to be in
19:20:30 <dripton> I think I already have a blueprint for migrate -> alembic.  I will update it since it has more urgency now, and send mail to the ML.
19:20:51 <dripton> Unless someone else has a competing blueprint that they'd rather use?
19:21:07 <boris-42> we don't have but we have it on our road map
19:21:37 <dripton> #agreed move to Alembic in Havana
19:21:47 <dripton> next topic anyone?
19:21:52 <boris-42> I am not sure that we will be able to do it in Havana
19:21:56 <boris-42> without a tons of reviews
19:22:13 <boris-42> russellb ^
19:22:19 <dripton> It sounds like we have to try hard or we're in danger of being stuck on sqlalchemy-migrate for another year.
19:22:34 <russellb> yeah, it may be too late at this point
19:22:59 <boris-42> russellb we are able to make all code in Havana
19:23:07 <russellb> we can only review so much
19:23:20 <boris-42> if you can
19:23:21 <russellb> and it may be the kind of change that's better to land early in the cycle than right at the end
19:23:24 <boris-42> there is no problem+)
19:23:35 <russellb> need time to find any possible regressions
19:23:56 <boris-42> I could ask QA guys to test in every possible way
19:24:06 <boris-42> on real data
19:24:11 <boris-42> from real deployements
19:24:30 <boris-42> chose you destiny =)
19:24:44 <boris-42> russellb ^
19:25:27 <russellb> if the code was ready this week maybe
19:25:32 <russellb> otherwise i'm not terribly optimistic
19:25:37 <russellb> we already have 80+ blueprints on havana-3
19:25:40 <russellb> no way we can get that many in
19:25:45 <russellb> adding another high risk one probably isn't an option
19:26:04 <boris-42> is it so problem to implement in I cycle
19:26:12 <boris-42> and wait for 1.5 year? for this moment?)
19:26:24 <boris-42> from*
19:26:48 <russellb> that's probably the only option right now
19:26:59 <russellb> for nova anyway
19:27:09 <boris-42> russellb ok then we will get some experience in ceilometer
19:27:21 <dripton> #agreed move nova to alembic in Icehouse.  Too late for Havana.
19:27:47 <dripton> next topic?
19:28:14 <boris-42> hm
19:28:19 <boris-42> db-archiving
19:28:21 <dripton> boris-42 did we break the roadblock on getting your db code into oslo now that the archiving functions are out?
19:28:33 <dripton> #topic db-archiving
19:28:50 <boris-42> Ok we decide that db-archiving is not good enough for oslo DB code
19:29:00 <boris-42> so it should stay only in Nova for a moment
19:29:16 <boris-42> so we will remove our code from oslo sqlalchemy utilites
19:29:43 <boris-42> and I hope our utilities will be land to oslo
19:30:20 <boris-42> So we will have, our sqlalchemy utilities and test_migrations in oslo
19:30:26 <boris-42> Also I have some question
19:30:40 <boris-42> As we are not able to move from sqlalchemy-migrate for a long period of time
19:30:52 <boris-42> it makes sense to move to oslo also code that makes migrations
19:30:57 <boris-42> russellb ^
19:31:53 <boris-42> dripton ^
19:31:55 <dripton> But only some projects are using sqlalchemy-migrate, so there might be resistance to having it in oslo.
19:32:06 <dripton> Like Neutron and now ceilometer won't use it.
19:32:31 <boris-42> yeah but to have N different wrappers around sqlalchemy-migarte is also not so good idea
19:33:12 <dripton> Well, I guess it's okay to propose it and see if they're okay with it.  Though it would be good if there were an obvious parallel path for alembic.
19:34:39 <dripton> I have a controversial proposal.  Is it okay to remove FK constraints so that db-archiving can clean up deleted instances with metadata (wrongly but necessarily) still pointing to them?
19:34:59 <dripton> I kind of think no, because the tail of archiving should not wag the dog of the Nova db schema
19:35:12 <dripton> bug it would let us fix bug #1183523
19:36:24 <dripton> #topic open discussion
19:36:28 <dripton> Anyone have anything else?
19:36:50 <rpodolyaka1> I've been working on implementation of db-api-tests-on-all-backends
19:36:58 <vsergeyev> I have a question about db-reconnect
19:37:10 <rpodolyaka1> mordred: could please take a look at https://review.openstack.org/#/c/33236/?
19:37:31 <rpodolyaka1> comments from anyone else are welcome too!
19:38:35 <mordred> looking
19:38:39 <dripton> rpodolyaka1: that looks great.  We've needed that for a while.  I'll take a close look later.
19:38:48 <rpodolyaka1> (as well as for any patches for this topic in Gerrit https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/db-api-tests-on-all-backends,n,z)
19:39:43 <rpodolyaka1> mordred: I'm not sure that's the best way to run those tests...
19:41:24 <rpodolyaka1> mordred: probably a better explanation is here http://lists.openstack.org/pipermail/openstack-dev/2013-June/010763.html
19:42:57 <boris-42> rpodolyaka1 I think that we shouldn't fix UC in Nova
19:43:07 <boris-42> rpodolyaka1 because that code is already in oslo
19:43:36 <rpodolyaka1> boris-42: yep, and don't we need to import those code from Oslo?
19:43:50 <boris-42> I think that we should wait until other code
19:43:58 <rpodolyaka1> I've been making a slow progress on running DB API tests on MySQL and PostgreSQL, there are a few patches on review fixing various issues. There are about 15-20 test cases still failing
19:44:24 <boris-42> that will be merged tomorrow when vsergeyev remove from it utilities that work with db archiving
19:44:58 <boris-42> as we discussed before today (it it not ready to be implemented in all proejcts)
19:45:21 <boris-42> So we will rethink it
19:45:36 <boris-42> rpodolyaka1 nice work
19:46:12 <dripton> you might want to ask in #openstack-infra about the issues testing multiple DBs, since the cabal of infra geniuses hangs out there.
19:47:02 <rpodolyaka1> dripton: actually, I tried, but haven't received enough feedback
19:47:23 <dripton> rpololyaka1: I fear that means the answer isn't obvious to anyone because you're doing trailblazing work.
19:47:46 <dripton> but if we all go look at your patches, maybe someone will have good ideas.
19:48:23 <rpodolyaka1> dription: sure! that's exactly what I'm looking for
19:48:30 <vsergeyev> please can somebody look at blueprint db-reconnect implementation? https://review.openstack.org/#/c/33831/
19:48:38 <boris-42> russellb dripton this is the almost patch set in db-api-tests https://review.openstack.org/#/c/33962/
19:48:48 <boris-42> almost last*
19:49:08 <russellb> cool, may be able to have it done in h2 then
19:49:38 <boris-42> and Unique Constraints also
19:49:46 <boris-42> how much days we have?
19:50:28 <rpodolyaka1> boris-42: 7?
19:50:41 <boris-42> i don't know=)
19:50:42 <rpodolyaka1> boris-42: if you are talking about H2
19:50:48 <boris-42> yeah about H2
19:51:19 <dripton> https://launchpad.net/nova/+milestone/havana-2 says 2013-07-18
19:51:24 <dripton> So very close
19:51:24 <russellb> needs to be merged by the end of Tuesday
19:51:27 <russellb> 18 is the release
19:51:28 <russellb> so, 16
19:51:39 <boris-42> If you help with reviews
19:52:22 <boris-42> I think that all patches are already on review
19:52:33 <boris-42> But even if they are not we will finish it tomorrow
19:52:46 <dripton> #todo dripton send meeting minutes link to openstack-dev ML to try to get more reviewers
19:53:27 <dripton> Anyone have anything else?
19:53:45 <vsergeyev> question about db reconnect )
19:54:32 <vsergeyev> do we need procced situation when method could execute successfully in the database, but the connection could be dropped before the final status is sent to the client?
19:55:25 <rpodolyaka1> vsergevev: it
19:55:43 <rpodolyaka1> vsergeyev: it's a real edge case, but data corruption is still possible...
19:56:19 <dripton> I don't know if databases let you handle that case where the connection drops at the very end.  There would have to be a final handshake telling the server that the client got the status.
19:57:27 <boris-42> sorry guys I have to go
19:57:28 <vsergeyev> dripton, yes that's a question
19:57:32 <dripton> It's a real concern but I don't think we should hold up your patch over it.  If it's a problem in the real world there could be another fix later.
19:57:36 <boris-42> dription russellb Thanks for meeting
19:57:38 <dripton> thanks boris-42
19:57:48 <vsergeyev> but its  most unlikely situation
19:57:59 <vsergeyev> boris-42, bye
19:58:11 <dripton> If your patch only fixes 90% of the problem and we need another one for the other 10% later, that seems like it's still going in the right direction.
19:59:03 <dripton> Anything else?  We're almost out of time.
19:59:12 <dripton> Thanks everyone.
19:59:15 <dripton> #endmeeting