Friday, 2011-11-11

_0x44Nova db meeting anyone?00:01
* dragondm listens to crickets. 00:01
*** jog0 has left #openstack-meeting00:02
*** jog0 has joined #openstack-meeting00:02
_0x44dragondm: Eh, let's just have it. I mostly want to argue with you anyway.00:02
openstackMeeting started Fri Nov 11 00:02:59 2011 UTC.  The chair is _0x44. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic.00:03
_0x44#topic Nova DB subteam00:03
*** openstack changes topic to "Nova DB subteam"00:03
_0x44I'm looking to see if there are any bugs that we should tackle before redoing everything.00:04
*** zns has quit IRC00:05
_0x44There are two, but one is a sqlalchemy bug that needs patching, the other Nachi has claimed.00:06
_0x44 and
uvirtbotLaunchpad bug 884837 in openstack-qa "Unused db.api in /nova/db/sqlalchemy/" [Medium,Confirmed]00:06
_0x44So since no one else is here, let's talk about serializing00:07
_0x44Vish suggested serializing to dicts, you suggested serializing to objects.00:07
_0x44There was a huge session at the Diablo summit in Santa Clara where justinsb was pushing for serializing to objects so he could use static analysis tools to do something.00:08
_0x44The end result of that discussion was that most people weren't comfortable with the idea of moving to objects since dictionaries are so pythonic and prevalent in the codebase.00:08
dragondmAy. it would be nice to have an actual, real model with actual, real objects.00:08
_0x44I'm a bit worried that serializing DB records to objects is kind of doing an end-run around that quasi-decision.00:09
dragondmReally, if we are just pulling dicts from the db, why do we have an ORM at all?00:10
dragondmPlus, it's been getting obvious that nova's lack of object design is causing messy code.00:11
_0x44The ORM is to abstract the places where we were writing redis calls directly00:11
dragondmI have had discussions w/ sandywalsh and jkoelker  abt this, and there has been much concurrance.00:11
_0x44For whatever reason, someone felt that redis wasn't a good tool and decided to replace it with mysql00:12
_0x44So they fixed the datastore calls being written by hand problem with sqlalchemy00:12
_0x44Part of the db-cleanup is to make it so that we have an abstracted db api that can support datastores that aren't SQL00:13
_0x44anotherjesse would like to see zookeeper in nova00:13
dragondmYah, but they are really just simulating using the DB-API directly.00:13
dragondmIf we made the db datastore use sqlalchemy properly, we could map to plain old python objects.00:14
dragondmThen any other datastore (redis, zk, ...) could use the same objects00:14
dragondmthus we could have real methods on our models00:14
dragondm(note that I am not being pro relational db, here. Redis and/or zookeeper are likely better solutions for this system)00:15
_0x44We'd need to have a db model layer that just consumed the raw data from either backend for that though.00:16
_0x44(Not objecting, just musing)00:16
dragondmWe'd have a  nova.model that was separate from any datastore.00:17
_0x44So db cleanup basically means writing our own model layer/api and a migration generator engine?00:17
_0x44Because migrations would need to be the same across datastores also00:17
dragondmthey wouldn't need to.00:17
dragondmTho having to maintain n different migrations (where n = datastores) would be annoying00:18
dragondmFortunately, the model layer would be trivial.00:18
_0x44They would, otherwise we end up with another keystone db problem or we're maintaining n migrations.00:18
*** bcwaldon has joined #openstack-meeting00:19
dragondmYah,  as said above, that would be annoying.00:20
dragondmTHo migrating nosql datastores would likely be much easier than sql.00:21
_0x44Yeah, most of them probably wouldn't require more than a model update...00:22
dragondmYup.   And in many cases there woudn't need to be an all-at once migration.00:22
dragondmif the objects were versioned, they could update at load.00:22
dragondmsql is the pain, coz of the fixed schema.00:23
dragondmright now, as I understand the code,  if we actually had multiple datastores, we'd need a migration engine per store.00:24
_0x44You're correct.00:25
_0x44But that would be easier than maintaining specific migrations for each store.00:26
_0x44Especially when you consider the problem's we're having keeping up feature parity with libvirt vs. xenserver00:27
dragondmHm.....  In many cases,  I'm not sure how we could avoid that.   But yes, libvirt v xs shows that is a problem.00:28
dragondm(I should also say, that the opinion of folks on  ozone team is the use of  sqllight for tests blows due to migration issues.  we wind up with multiple migrations due to that, too)00:29
_0x44Just tell _cerberus_ to fix it.00:29
_0x44That's his favourite thing.00:29
dragondmyah, I can tell by the cursing.00:30
dragondmStill, the representations in different stores is likely to be different enough, it'd be difficult to create migrations that would work for all of them.00:33
_0x44We can probably defer that until after we've gotten the model layer done00:33
_0x44And after we've abstracted the db.api00:33
dragondmTHe model layer would be much easier.00:34
dragondmeffectively just db.sqlalchemy.models  minus the sqlalchemy cruft.00:35
dragondmwhich would got in mappers in db.sqlalchemy.something00:35
dragondmer /got/go/00:35
dragondmBreaking up the db.api  would be a major win.00:36
dragondmand, imho,  nuking the useless db.api  module as it is.00:37
dragondmWe should just use unittests  to make sure the datastore apis are alike00:37
dragondm(we can do that to some degree by introspection)00:37
_0x44That would require that we write something like RedisAlchemy or ZookeeperAlchemy00:38
_0x44Which would kind of suck00:38
_0x44(The redisalchemy one is pretty small)00:38
dragondmI don't follow.00:39
_0x44If the datastore apis need to be the same, we need to wrap them with something like SQLAlchemy... because that's the one we have right now.00:39
dragondmthe data store apis are the  db.api implementations.00:40
_0x44Which won't exist after you've nuked the module.00:41
dragondmI ment in it's current form00:42
dragondmas in get rid of the code that just turns around does:  def foobar():  return IMPL.foobar()00:43
_0x44And replace it with method_missing!00:43
_0x44It's a ruby method :P00:43
dragondmmake the db apis classes00:43
dragondm(yah, I know... I've done much ruby) :->00:44
dragondmand 'the' db api is duck typed.00:44
dragondmsou you ask for a db.api,  and get an  db.sqlalchemy.api or db.redis.api, or such.00:45
dragondmand meanwhile, there are unittests that load all 'known' db apis, and make sure they all have the same 'public' (i.,e. non _blah  ) methods and properties.00:46
dragondmmakle sense ?00:47
_0x44That seems reasonable and in line with what the blueprint requested:00:48
dragondm(and  by db.api, I mean an assortment of classes like db.api.instance , .....    )00:49
dragondmHmm... yah, what I've been suggesting is basically option b)00:50
_0x44I was going to make a lazy-loading dict class for a) but then I got distracted00:50
_0x44Which is probably for the best.00:50
dragondmlazy lazy loading00:51
dragondmOr the power of creative distraction. ;>00:51
_0x44We should make blueprints for the model api and the db api.00:53
_0x44Okay, want to take the db api changes one? I'll take the models api one00:53
dragondmSounds good.00:53
_0x44#action _0x44 to write models api blueprint.00:54
_0x44#action dragondm to write db api changes blueprint.00:54
_0x44Since no one else is here to object, do you want to make Friday at 00:00UTC (Thursday 1800CST, 1600PDT) the regular meeting time for the nova-db subteam?00:55
dragondmsure.  Suits me.00:55
bcwaldonI object!00:56
bcwaldonno, that works00:56
dragondmOh, NOW he talks :>00:57
_0x44Awesome. I'll update the meetings page.00:57
bcwaldonI came in half way through. Thought the meeting started at 8 est00:57
_0x44Sorry about that.00:57
dragondmDST is the tool of Eris.00:58
bcwaldonits not a big deal. You guys came to the conclusion I wanted anyways00:58
_0x44We've only discussed serialization.00:58
_0x44Perl scripts and flat files?00:58
bcwaldonJesus no00:58
bcwaldonwhat's happening00:58
*** openstack changes topic to "Openstack Meetings: | Minutes:"00:58
openstackMeeting ended Fri Nov 11 00:58:56 2011 UTC.  Information about MeetBot at . (v 0.1.4)00:58
openstackMinutes (text):
_0x44I guess I should have asked if there was anything else to add.00:59
_0x44Thanks dragondm, bcwaldon01:00
bcwaldonI don't deserve any thanks here, but I'll take what I can get ;)01:00
dragondmRandom thought: what did you think of my notion of making the db.api objects a property of the context?01:04
*** bcwaldon has quit IRC01:07
_0x44dragondm: That might be difficult since the context gets serialized onto the queue.01:08
*** bcwaldon has joined #openstack-meeting01:12
dragondmthe context serializer would have to ignore certain internal fields.  Easily done.   The reason I thought of that (besides the fact that context is needed in all db calls anyway) is that would allow for a  dbapi  object that lived throughout  our call, be it a wsgi web request, or rpc call, and was passed into it from outside (eaither earlier in the wsgi stack, or in the rpc machinery) and thus would be a place for session data (like sqlalchemy01:13
dragondmlive and be able to handle setup/teardown in the dbapi object at beginning/ending of calls for dbstores that needed such.01:13
_0x44I guess I don't yet have an opinion on that one way or the other.01:15
dragondmok.  just pondering.01:15
*** jog0 has left #openstack-meeting01:15
dragondmI'll throw my random thoughts on implementations onto an  etherpad when I write up the bp.01:16
_0x44Awesome, send the link to the list and I'll read it and form an opinion :)01:18
*** gyee has quit IRC01:29
*** oubiwann has quit IRC01:30
*** novas0x2a|laptop has quit IRC01:40
*** Gordonz has quit IRC01:40
*** bcwaldon has left #openstack-meeting01:42
*** vladimir3p has quit IRC01:50
*** shang has quit IRC02:02
*** dragondm has quit IRC02:04
*** dolphm has joined #openstack-meeting02:09
*** dolphm has quit IRC02:15
*** dolphm has joined #openstack-meeting02:16
*** dolphm has quit IRC02:20
*** df1 has quit IRC02:25
*** dwalleck has joined #openstack-meeting02:27
*** dragondm has joined #openstack-meeting02:31
*** vladimir3p has joined #openstack-meeting02:36
*** vladimir3p has quit IRC02:45
*** shang has joined #openstack-meeting03:00
*** df1 has joined #openstack-meeting03:21
*** jakedahn has joined #openstack-meeting04:09
*** mmetheny_ has quit IRC04:09
*** mmetheny has joined #openstack-meeting04:09
*** jakedahn has quit IRC04:14
*** jakedahn has joined #openstack-meeting04:39
*** dwalleck has quit IRC04:55
*** blamar__ has joined #openstack-meeting04:56
*** blamar__ has quit IRC05:04
*** mdomsch has quit IRC05:16
*** sleepsontheflo-1 has joined #openstack-meeting05:26
*** jdurgin has quit IRC05:35
*** adjohn has joined #openstack-meeting06:13
*** sleepsontheflo-1 has quit IRC06:19
*** sleepsontheflo-1 has joined #openstack-meeting06:20
*** danwent has joined #openstack-meeting06:34
*** reed has quit IRC06:55
*** jog0 has joined #openstack-meeting07:04
*** jog0 has quit IRC07:08
*** dragondm has quit IRC07:30
*** danwent has left #openstack-meeting07:30
*** jakedahn has quit IRC07:33
*** adjohn has quit IRC08:17
*** nati2 has quit IRC08:30
*** sleepsontheflo-2 has joined #openstack-meeting09:05
*** sleepsontheflo-1 has quit IRC09:09
*** darraghb has joined #openstack-meeting10:08
*** darraghb has quit IRC11:23
*** darraghb has joined #openstack-meeting11:46
*** edconzel has joined #openstack-meeting14:27
*** Gordonz has joined #openstack-meeting15:05
*** mdomsch has joined #openstack-meeting15:06
*** Gordonz has joined #openstack-meeting15:06
*** troytoman-away is now known as troytoman15:14
*** reed_ has joined #openstack-meeting15:21
*** AlanClark has joined #openstack-meeting15:30
*** dendrobates is now known as dendro-afk15:35
*** dendro-afk is now known as dendrobates15:35
*** dendrobates has joined #openstack-meeting15:35
*** AlanClark has quit IRC15:35
*** Gordonz has quit IRC15:38
*** dolphm has joined #openstack-meeting15:39
*** troytoman is now known as troytoman-away15:50
*** reed_ is now known as reed15:54
*** edconzel has quit IRC15:56
*** dolphm has quit IRC16:01
*** dolphm has joined #openstack-meeting16:01
*** adjohn has joined #openstack-meeting16:02
*** dolphm has quit IRC16:06
*** mmetheny has quit IRC16:09
*** mmetheny_ has joined #openstack-meeting16:09
*** danwent has joined #openstack-meeting16:14
*** dolphm has joined #openstack-meeting16:15
*** sleepsontheflo-2 has quit IRC16:23
*** dolphm has quit IRC16:28
*** dolphm has joined #openstack-meeting16:29
*** dolphm_ has joined #openstack-meeting16:29
*** dolphm has quit IRC16:33
*** dragondm has joined #openstack-meeting16:54
*** nati2 has joined #openstack-meeting16:54
*** dprince has joined #openstack-meeting16:55
*** danwent has quit IRC16:57
*** darraghb has quit IRC16:58
*** danwent has joined #openstack-meeting17:01
*** danwent has quit IRC17:05
*** dragondm has quit IRC17:06
*** adjohn has quit IRC17:09
*** shang has quit IRC17:11
*** danwent has joined #openstack-meeting17:15
*** sleepsontheflo-1 has joined #openstack-meeting17:16
*** mdomsch has quit IRC17:23
*** sec_prof has joined #openstack-meeting17:29
*** jog0 has joined #openstack-meeting17:31
*** sec_prof has quit IRC17:36
*** jakedahn has joined #openstack-meeting17:42
*** hggdh has quit IRC17:44
*** nati2 has quit IRC17:44
*** ohnoimdead has joined #openstack-meeting17:44
*** ohnoimdead_ has joined #openstack-meeting17:48
*** ohnoimdead has quit IRC17:48
*** ohnoimdead_ is now known as ohnoimdead17:48
*** ohnoimdead_ has joined #openstack-meeting17:48
*** ohnoimdead has quit IRC17:48
*** ohnoimdead_ is now known as ohnoimdead17:48
*** ohnoimdead_ has joined #openstack-meeting17:49
*** ohnoimdead has quit IRC17:49
*** ohnoimdead_ is now known as ohnoimdead17:49
*** ohnoimdead has joined #openstack-meeting17:50
*** ohnoimdead has quit IRC17:51
*** ohnoimdead_ has joined #openstack-meeting17:51
*** hggdh has joined #openstack-meeting17:51
*** ohnoimdead has joined #openstack-meeting17:52
*** ohnoimdead_ has quit IRC17:52
*** ohnoimdead_ has joined #openstack-meeting17:56
*** ohnoimdead has quit IRC17:56
*** ohnoimdead_ is now known as ohnoimdead17:56
*** ohnoimdead_ has joined #openstack-meeting18:00
*** ohnoimdead has quit IRC18:00
*** ohnoimdead_ has quit IRC18:00
*** ohnoimdead has joined #openstack-meeting18:01
*** ohnoimdead_ has joined #openstack-meeting18:02
*** ohnoimdead has quit IRC18:02
*** ohnoimdead_ is now known as ohnoimdead18:02
*** ohnoimdead has quit IRC18:05
*** ohnoimdead has joined #openstack-meeting18:06
*** ohnoimdead has quit IRC18:06
*** ohnoimdead has joined #openstack-meeting18:06
*** ohnoimdead_ has joined #openstack-meeting18:07
*** ohnoimdead has quit IRC18:07
*** ohnoimdead_ is now known as ohnoimdead18:07
*** ohnoimdead has joined #openstack-meeting18:09
*** dragondm has joined #openstack-meeting18:11
*** dragondm has joined #openstack-meeting18:12
*** littleidea has joined #openstack-meeting18:14
*** novas0x2a|laptop has joined #openstack-meeting18:15
*** dolphm_ has quit IRC18:28
*** dolphm has joined #openstack-meeting18:28
*** dolphm has quit IRC18:33
*** adjohn has joined #openstack-meeting18:38
*** littleidea has quit IRC18:46
*** mdomsch has joined #openstack-meeting18:48
*** sleepsontheflo-1 has quit IRC18:54
*** jakedahn has quit IRC18:54
*** adjohn has quit IRC18:54
*** sleepsontheflo-1 has joined #openstack-meeting18:54
*** jakedahn has joined #openstack-meeting18:55
*** zul has quit IRC18:56
*** zul has joined #openstack-meeting18:56
*** dolphm has joined #openstack-meeting18:58
*** jakedahn has quit IRC19:01
*** jakedahn has joined #openstack-meeting19:02
*** dprince has quit IRC19:06
*** gyee has joined #openstack-meeting19:18
*** sleepsontheflo-2 has joined #openstack-meeting19:20
*** sleepsontheflo-1 has quit IRC19:22
*** jog0 has left #openstack-meeting19:28
*** Gordonz has joined #openstack-meeting19:34
*** dendrobates is now known as dendro-afk19:55
*** adjohn has joined #openstack-meeting20:09
*** sandywalsh_ has joined #openstack-meeting20:13
*** adjohn has quit IRC20:27
*** littleidea has joined #openstack-meeting20:35
*** df1 has joined #openstack-meeting20:41
*** danwent_ has joined #openstack-meeting20:51
*** danwent has quit IRC20:51
*** danwent_ is now known as danwent20:51
*** troytoman-away is now known as troytoman21:33
*** dolphm has quit IRC21:48
*** dolphm has joined #openstack-meeting21:48
*** dolphm_ has joined #openstack-meeting21:49
*** dolphm has quit IRC21:53
*** danwent has quit IRC21:59
*** troytoman is now known as troytoman-away22:03
*** ameade_ has quit IRC22:06
*** ameade has joined #openstack-meeting22:06
*** dragondm has quit IRC22:06
*** dragondm has joined #openstack-meeting22:09
*** dendro-afk is now known as dendrobates22:11
*** dolphm_ has quit IRC22:19
*** danwent has joined #openstack-meeting22:28
*** littleidea has quit IRC22:28
*** reed has quit IRC22:31
*** mdomsch has quit IRC22:34
*** jdg has joined #openstack-meeting22:39
*** reed has joined #openstack-meeting22:39
*** jakedahn has quit IRC22:41
*** jakedahn has joined #openstack-meeting22:42
*** sandywalsh_ has quit IRC22:43
*** anotherjesse has joined #openstack-meeting23:04
*** sleepsontheflo-2 has quit IRC23:10
*** sleepsontheflo-1 has joined #openstack-meeting23:10
*** jdg has quit IRC23:14
*** df1 has quit IRC23:16
*** dolphm has joined #openstack-meeting23:19
*** df1 has joined #openstack-meeting23:25
*** dolphm has quit IRC23:28
*** jakedahn_ has joined #openstack-meeting23:30
*** jakedahn has quit IRC23:31
*** jakedahn_ has quit IRC23:31
*** jakedahn has joined #openstack-meeting23:31
*** df1 has quit IRC23:32
*** dolphm has joined #openstack-meeting23:32
*** novas0x2a|laptop has quit IRC23:32
*** dolphm has quit IRC23:35
*** Gordonz has quit IRC23:37

Generated by 2.14.0 by Marius Gedminas - find it at!