00:02:59 <_0x44> #startmeeting 00:03:00 Meeting started Fri Nov 11 00:02:59 2011 UTC. The chair is _0x44. Information about MeetBot at http://wiki.debian.org/MeetBot. 00:03:01 Useful Commands: #action #agreed #help #info #idea #link #topic. 00:03:11 Heh 00:03:12 <_0x44> #topic Nova DB subteam 00:04:25 Ok. 00:04:58 <_0x44> I'm looking to see if there are any bugs that we should tackle before redoing everything. 00:06:41 <_0x44> There are two, but one is a sqlalchemy bug that needs patching, the other Nachi has claimed. 00:06:52 <_0x44> https://bugs.launchpad.net/nova/+bug/884837 and https://bugs.launchpad.net/nova/+bug/845076 00:06:53 Launchpad bug 884837 in openstack-qa "Unused db.api in /nova/db/sqlalchemy/api.py" [Medium,Confirmed] 00:07:00 ok. 00:07:30 <_0x44> So since no one else is here, let's talk about serializing 00:07:41 <_0x44> Vish suggested serializing to dicts, you suggested serializing to objects. 00:08:21 <_0x44> There was a huge session at the Diablo summit in Santa Clara where justinsb was pushing for serializing to objects so he could use static analysis tools to do something. 00:08:47 <_0x44> The end result of that discussion was that most people weren't comfortable with the idea of moving to objects since dictionaries are so pythonic and prevalent in the codebase. 00:08:58 Ay. it would be nice to have an actual, real model with actual, real objects. 00:09:22 <_0x44> I'm a bit worried that serializing DB records to objects is kind of doing an end-run around that quasi-decision. 00:10:09 Really, if we are just pulling dicts from the db, why do we have an ORM at all? 00:11:09 Plus, it's been getting obvious that nova's lack of object design is causing messy code. 00:11:22 <_0x44> The ORM is to abstract the places where we were writing redis calls directly 00:11:59 I have had discussions w/ sandywalsh and jkoelker abt this, and there has been much concurrance. 00:12:00 <_0x44> For whatever reason, someone felt that redis wasn't a good tool and decided to replace it with mysql 00:12:39 <_0x44> So they fixed the datastore calls being written by hand problem with sqlalchemy 00:13:01 <_0x44> Part of the db-cleanup is to make it so that we have an abstracted db api that can support datastores that aren't SQL 00:13:15 <_0x44> anotherjesse would like to see zookeeper in nova 00:13:16 Yah, but they are really just simulating using the DB-API directly. 00:13:20 Ay. 00:14:04 If we made the db datastore use sqlalchemy properly, we could map to plain old python objects. 00:14:30 Then any other datastore (redis, zk, ...) could use the same objects 00:14:42 thus we could have real methods on our models 00:15:56 (note that I am not being pro relational db, here. Redis and/or zookeeper are likely better solutions for this system) 00:16:23 <_0x44> We'd need to have a db model layer that just consumed the raw data from either backend for that though. 00:16:31 <_0x44> (Not objecting, just musing) 00:16:41 Basically. 00:17:05 We'd have a nova.model that was separate from any datastore. 00:17:06 <_0x44> So db cleanup basically means writing our own model layer/api and a migration generator engine? 00:17:28 <_0x44> Because migrations would need to be the same across datastores also 00:17:42 need? 00:17:52 they wouldn't need to. 00:18:27 Tho having to maintain n different migrations (where n = datastores) would be annoying 00:18:42 Fortunately, the model layer would be trivial. 00:18:47 <_0x44> They would, otherwise we end up with another keystone db problem or we're maintaining n migrations. 00:20:32 Yah, as said above, that would be annoying. 00:21:18 THo migrating nosql datastores would likely be much easier than sql. 00:22:05 <_0x44> Yeah, most of them probably wouldn't require more than a model update... 00:22:36 Yup. And in many cases there woudn't need to be an all-at once migration. 00:22:57 if the objects were versioned, they could update at load. 00:23:20 sql is the pain, coz of the fixed schema. 00:24:57 right now, as I understand the code, if we actually had multiple datastores, we'd need a migration engine per store. 00:25:11 <_0x44> You're correct. 00:26:41 <_0x44> But that would be easier than maintaining specific migrations for each store. 00:27:24 <_0x44> Especially when you consider the problem's we're having keeping up feature parity with libvirt vs. xenserver 00:27:33 Yah, 00:28:53 Hm..... In many cases, I'm not sure how we could avoid that. But yes, libvirt v xs shows that is a problem. 00:29:28 (I should also say, that the opinion of folks on ozone team is the use of sqllight for tests blows due to migration issues. we wind up with multiple migrations due to that, too) 00:29:50 <_0x44> Just tell _cerberus_ to fix it. 00:29:56 Heh. 00:29:56 <_0x44> That's his favourite thing. 00:30:28 yah, I can tell by the cursing. 00:33:06 Still, the representations in different stores is likely to be different enough, it'd be difficult to create migrations that would work for all of them. 00:33:36 <_0x44> We can probably defer that until after we've gotten the model layer done 00:33:50 <_0x44> And after we've abstracted the db.api 00:34:03 ya. 00:34:16 THe model layer would be much easier. 00:35:25 effectively just db.sqlalchemy.models minus the sqlalchemy cruft. 00:35:45 which would got in mappers in db.sqlalchemy.something 00:35:57 er /got/go/ 00:36:28 Breaking up the db.api would be a major win. 00:37:08 and, imho, nuking the useless db.api module as it is. 00:37:34 We should just use unittests to make sure the datastore apis are alike 00:37:50 (we can do that to some degree by introspection) 00:38:14 <_0x44> That would require that we write something like RedisAlchemy or ZookeeperAlchemy 00:38:18 <_0x44> Which would kind of suck 00:38:22 ? 00:38:24 <_0x44> (The redisalchemy one is pretty small) 00:38:57 howzat? 00:39:43 I don't follow. 00:39:49 <_0x44> If the datastore apis need to be the same, we need to wrap them with something like SQLAlchemy... because that's the one we have right now. 00:40:26 the data store apis are the db.api implementations. 00:41:59 <_0x44> Which won't exist after you've nuked the module. 00:42:17 I ment in it's current form 00:43:18 as in get rid of the code that just turns around does: def foobar(): return IMPL.foobar() 00:43:31 <_0x44> And replace it with method_missing! 00:43:38 ? 00:43:51 <_0x44> It's a ruby method :P 00:43:54 make the db apis classes 00:44:15 (yah, I know... I've done much ruby) :-> 00:44:32 and 'the' db api is duck typed. 00:45:16 sou you ask for a db.api, and get an db.sqlalchemy.api or db.redis.api, or such. 00:46:29 and meanwhile, there are unittests that load all 'known' db apis, and make sure they all have the same 'public' (i.,e. non _blah ) methods and properties. 00:47:29 makle sense ? 00:48:10 <_0x44> Yep 00:48:28 <_0x44> That seems reasonable and in line with what the blueprint requested: 00:48:28 <_0x44> https://blueprints.launchpad.net/nova/+spec/nova-db-api-objects 00:49:26 (and by db.api, I mean an assortment of classes like db.api.instance , db.api.network ..... ) 00:50:27 Hmm... yah, what I've been suggesting is basically option b) 00:50:47 <_0x44> I was going to make a lazy-loading dict class for a) but then I got distracted 00:50:51 <_0x44> Which is probably for the best. 00:50:57 hah. 00:51:13 lazy lazy loading 00:51:55 Or the power of creative distraction. ;> 00:52:42 <_0x44> Yeah. 00:53:04 <_0x44> We should make blueprints for the model api and the db api. 00:53:15 yah. 00:53:43 <_0x44> Okay, want to take the db api changes one? I'll take the models api one 00:53:51 Ok. 00:53:56 Sounds good. 00:54:07 <_0x44> #action _0x44 to write models api blueprint. 00:54:17 <_0x44> #action dragondm to write db api changes blueprint. 00:55:12 <_0x44> Since no one else is here to object, do you want to make Friday at 00:00UTC (Thursday 1800CST, 1600PDT) the regular meeting time for the nova-db subteam? 00:55:43 sure. Suits me. 00:56:50 I object! 00:56:56 no, that works 00:57:13 Oh, NOW he talks :> 00:57:14 <_0x44> Awesome. I'll update the meetings page. 00:57:32 I came in half way through. Thought the meeting started at 8 est 00:57:57 <_0x44> Sorry about that. 00:58:11 DST is the tool of Eris. 00:58:18 its not a big deal. You guys came to the conclusion I wanted anyways 00:58:21 <_0x44> We've only discussed serialization. 00:58:28 <_0x44> Perl scripts and flat files? 00:58:38 Jesus no 00:58:42 what's happening 00:58:52 <_0x44> :D 00:58:56 <_0x44> #endmeeting