00:02:59 <_0x44> #startmeeting
00:03:00 <openstack> Meeting started Fri Nov 11 00:02:59 2011 UTC.  The chair is _0x44. Information about MeetBot at http://wiki.debian.org/MeetBot.
00:03:01 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic.
00:03:11 <dragondm> Heh
00:03:12 <_0x44> #topic Nova DB subteam
00:04:25 <dragondm> Ok.
00:04:58 <_0x44> I'm looking to see if there are any bugs that we should tackle before redoing everything.
00:06:41 <_0x44> There are two, but one is a sqlalchemy bug that needs patching, the other Nachi has claimed.
00:06:52 <_0x44> https://bugs.launchpad.net/nova/+bug/884837 and https://bugs.launchpad.net/nova/+bug/845076
00:06:53 <uvirtbot> Launchpad bug 884837 in openstack-qa "Unused db.api in /nova/db/sqlalchemy/api.py" [Medium,Confirmed]
00:07:00 <dragondm> ok.
00:07:30 <_0x44> So since no one else is here, let's talk about serializing
00:07:41 <_0x44> Vish suggested serializing to dicts, you suggested serializing to objects.
00:08:21 <_0x44> There was a huge session at the Diablo summit in Santa Clara where justinsb was pushing for serializing to objects so he could use static analysis tools to do something.
00:08:47 <_0x44> The end result of that discussion was that most people weren't comfortable with the idea of moving to objects since dictionaries are so pythonic and prevalent in the codebase.
00:08:58 <dragondm> Ay. it would be nice to have an actual, real model with actual, real objects.
00:09:22 <_0x44> I'm a bit worried that serializing DB records to objects is kind of doing an end-run around that quasi-decision.
00:10:09 <dragondm> Really, if we are just pulling dicts from the db, why do we have an ORM at all?
00:11:09 <dragondm> Plus, it's been getting obvious that nova's lack of object design is causing messy code.
00:11:22 <_0x44> The ORM is to abstract the places where we were writing redis calls directly
00:11:59 <dragondm> I have had discussions w/ sandywalsh and jkoelker  abt this, and there has been much concurrance.
00:12:00 <_0x44> For whatever reason, someone felt that redis wasn't a good tool and decided to replace it with mysql
00:12:39 <_0x44> So they fixed the datastore calls being written by hand problem with sqlalchemy
00:13:01 <_0x44> Part of the db-cleanup is to make it so that we have an abstracted db api that can support datastores that aren't SQL
00:13:15 <_0x44> anotherjesse would like to see zookeeper in nova
00:13:16 <dragondm> Yah, but they are really just simulating using the DB-API directly.
00:13:20 <dragondm> Ay.
00:14:04 <dragondm> If we made the db datastore use sqlalchemy properly, we could map to plain old python objects.
00:14:30 <dragondm> Then any other datastore (redis, zk, ...) could use the same objects
00:14:42 <dragondm> thus we could have real methods on our models
00:15:56 <dragondm> (note that I am not being pro relational db, here. Redis and/or zookeeper are likely better solutions for this system)
00:16:23 <_0x44> We'd need to have a db model layer that just consumed the raw data from either backend for that though.
00:16:31 <_0x44> (Not objecting, just musing)
00:16:41 <dragondm> Basically.
00:17:05 <dragondm> We'd have a  nova.model that was separate from any datastore.
00:17:06 <_0x44> So db cleanup basically means writing our own model layer/api and a migration generator engine?
00:17:28 <_0x44> Because migrations would need to be the same across datastores also
00:17:42 <dragondm> need?
00:17:52 <dragondm> they wouldn't need to.
00:18:27 <dragondm> Tho having to maintain n different migrations (where n = datastores) would be annoying
00:18:42 <dragondm> Fortunately, the model layer would be trivial.
00:18:47 <_0x44> They would, otherwise we end up with another keystone db problem or we're maintaining n migrations.
00:20:32 <dragondm> Yah,  as said above, that would be annoying.
00:21:18 <dragondm> THo migrating nosql datastores would likely be much easier than sql.
00:22:05 <_0x44> Yeah, most of them probably wouldn't require more than a model update...
00:22:36 <dragondm> Yup.   And in many cases there woudn't need to be an all-at once migration.
00:22:57 <dragondm> if the objects were versioned, they could update at load.
00:23:20 <dragondm> sql is the pain, coz of the fixed schema.
00:24:57 <dragondm> right now, as I understand the code,  if we actually had multiple datastores, we'd need a migration engine per store.
00:25:11 <_0x44> You're correct.
00:26:41 <_0x44> But that would be easier than maintaining specific migrations for each store.
00:27:24 <_0x44> Especially when you consider the problem's we're having keeping up feature parity with libvirt vs. xenserver
00:27:33 <dragondm> Yah,
00:28:53 <dragondm> Hm.....  In many cases,  I'm not sure how we could avoid that.   But yes, libvirt v xs shows that is a problem.
00:29:28 <dragondm> (I should also say, that the opinion of folks on  ozone team is the use of  sqllight for tests blows due to migration issues.  we wind up with multiple migrations due to that, too)
00:29:50 <_0x44> Just tell _cerberus_ to fix it.
00:29:56 <dragondm> Heh.
00:29:56 <_0x44> That's his favourite thing.
00:30:28 <dragondm> yah, I can tell by the cursing.
00:33:06 <dragondm> Still, the representations in different stores is likely to be different enough, it'd be difficult to create migrations that would work for all of them.
00:33:36 <_0x44> We can probably defer that until after we've gotten the model layer done
00:33:50 <_0x44> And after we've abstracted the db.api
00:34:03 <dragondm> ya.
00:34:16 <dragondm> THe model layer would be much easier.
00:35:25 <dragondm> effectively just db.sqlalchemy.models  minus the sqlalchemy cruft.
00:35:45 <dragondm> which would got in mappers in db.sqlalchemy.something
00:35:57 <dragondm> er /got/go/
00:36:28 <dragondm> Breaking up the db.api  would be a major win.
00:37:08 <dragondm> and, imho,  nuking the useless db.api  module as it is.
00:37:34 <dragondm> We should just use unittests  to make sure the datastore apis are alike
00:37:50 <dragondm> (we can do that to some degree by introspection)
00:38:14 <_0x44> That would require that we write something like RedisAlchemy or ZookeeperAlchemy
00:38:18 <_0x44> Which would kind of suck
00:38:22 <dragondm> ?
00:38:24 <_0x44> (The redisalchemy one is pretty small)
00:38:57 <dragondm> howzat?
00:39:43 <dragondm> I don't follow.
00:39:49 <_0x44> If the datastore apis need to be the same, we need to wrap them with something like SQLAlchemy... because that's the one we have right now.
00:40:26 <dragondm> the data store apis are the  db.api implementations.
00:41:59 <_0x44> Which won't exist after you've nuked the module.
00:42:17 <dragondm> I ment in it's current form
00:43:18 <dragondm> as in get rid of the code that just turns around does:  def foobar():  return IMPL.foobar()
00:43:31 <_0x44> And replace it with method_missing!
00:43:38 <dragondm> ?
00:43:51 <_0x44> It's a ruby method :P
00:43:54 <dragondm> make the db apis classes
00:44:15 <dragondm> (yah, I know... I've done much ruby) :->
00:44:32 <dragondm> and 'the' db api is duck typed.
00:45:16 <dragondm> sou you ask for a db.api,  and get an  db.sqlalchemy.api or db.redis.api, or such.
00:46:29 <dragondm> and meanwhile, there are unittests that load all 'known' db apis, and make sure they all have the same 'public' (i.,e. non _blah  ) methods and properties.
00:47:29 <dragondm> makle sense ?
00:48:10 <_0x44> Yep
00:48:28 <_0x44> That seems reasonable and in line with what the blueprint requested:
00:48:28 <_0x44> https://blueprints.launchpad.net/nova/+spec/nova-db-api-objects
00:49:26 <dragondm> (and  by db.api, I mean an assortment of classes like db.api.instance , db.api.network .....    )
00:50:27 <dragondm> Hmm... yah, what I've been suggesting is basically option b)
00:50:47 <_0x44> I was going to make a lazy-loading dict class for a) but then I got distracted
00:50:51 <_0x44> Which is probably for the best.
00:50:57 <dragondm> hah.
00:51:13 <dragondm> lazy lazy loading
00:51:55 <dragondm> Or the power of creative distraction. ;>
00:52:42 <_0x44> Yeah.
00:53:04 <_0x44> We should make blueprints for the model api and the db api.
00:53:15 <dragondm> yah.
00:53:43 <_0x44> Okay, want to take the db api changes one? I'll take the models api one
00:53:51 <dragondm> Ok.
00:53:56 <dragondm> Sounds good.
00:54:07 <_0x44> #action _0x44 to write models api blueprint.
00:54:17 <_0x44> #action dragondm to write db api changes blueprint.
00:55:12 <_0x44> Since no one else is here to object, do you want to make Friday at 00:00UTC (Thursday 1800CST, 1600PDT) the regular meeting time for the nova-db subteam?
00:55:43 <dragondm> sure.  Suits me.
00:56:50 <bcwaldon> I object!
00:56:56 <bcwaldon> no, that works
00:57:13 <dragondm> Oh, NOW he talks :>
00:57:14 <_0x44> Awesome. I'll update the meetings page.
00:57:32 <bcwaldon> I came in half way through. Thought the meeting started at 8 est
00:57:57 <_0x44> Sorry about that.
00:58:11 <dragondm> DST is the tool of Eris.
00:58:18 <bcwaldon> its not a big deal. You guys came to the conclusion I wanted anyways
00:58:21 <_0x44> We've only discussed serialization.
00:58:28 <_0x44> Perl scripts and flat files?
00:58:38 <bcwaldon> Jesus no
00:58:42 <bcwaldon> what's happening
00:58:52 <_0x44> :D
00:58:56 <_0x44> #endmeeting