| *** gnrfan has quit IRC | 00:00 | |
| *** gustavomzw has joined #openstack | 00:04 | |
| anotherjesse | kinda getting tired of not having a real orm :( | 00:11 |
|---|---|---|
| *** binaryWarrior has joined #openstack | 00:14 | |
| *** dmd17 has joined #openstack | 00:14 | |
| *** filler has quit IRC | 00:14 | |
| mtaylor | anotherjesse: spoke with michael bayer from sqlalchemy - he seemed to think that getting redis support wouldn't be terrible | 00:20 |
| *** jc_smith has quit IRC | 00:28 | |
| *** justinsb has joined #openstack | 00:32 | |
| justinsb | vish: You around? | 00:33 |
| *** RobertLJ_ has joined #openstack | 00:36 | |
| *** abecc has joined #openstack | 00:46 | |
| *** RobertLJ_ has quit IRC | 00:51 | |
| *** RobertLJ_ has joined #openstack | 00:55 | |
| *** kallistec has quit IRC | 00:56 | |
| *** gustavomzw has quit IRC | 01:01 | |
| *** silassewell has quit IRC | 01:01 | |
| *** silassewell has joined #openstack | 01:17 | |
| *** flamingoA has quit IRC | 01:18 | |
| *** mrayzenoss has joined #openstack | 01:20 | |
| *** mrayzenoss is now known as mray | 01:20 | |
| *** mray has joined #openstack | 01:20 | |
| *** zenmatt has quit IRC | 01:30 | |
| *** kallistec has joined #openstack | 01:37 | |
| *** miclorb_ has quit IRC | 01:39 | |
| *** zul has quit IRC | 01:41 | |
| *** pcrews has quit IRC | 01:43 | |
| *** zul has joined #openstack | 01:43 | |
| *** pcrews has joined #openstack | 01:43 | |
| *** mray has quit IRC | 01:44 | |
| *** zul has quit IRC | 01:46 | |
| *** zul has joined #openstack | 01:47 | |
| *** zul_ has joined #openstack | 01:49 | |
| *** clayg has quit IRC | 01:54 | |
| *** silassewell has quit IRC | 01:54 | |
| *** clayg_ has joined #openstack | 01:55 | |
| *** admboom has quit IRC | 02:05 | |
| *** dmd17_ has joined #openstack | 02:06 | |
| *** dmd17 has quit IRC | 02:07 | |
| *** dmd17_ is now known as dmd17 | 02:07 | |
| *** jamiew has quit IRC | 02:07 | |
| *** zenmatt has joined #openstack | 02:19 | |
| *** gundlach has joined #openstack | 02:24 | |
| *** jbryce has quit IRC | 02:51 | |
| *** miclorb has joined #openstack | 02:52 | |
| *** pcrews has quit IRC | 02:55 | |
| *** gundlach has quit IRC | 02:56 | |
| *** maximilien has quit IRC | 02:59 | |
| *** metoikos has quit IRC | 03:08 | |
| *** pvo has quit IRC | 03:10 | |
| *** zenmatt has quit IRC | 03:14 | |
| *** zenmatt has joined #openstack | 03:14 | |
| *** jdmaturen has quit IRC | 03:16 | |
| *** codejunkie has quit IRC | 03:18 | |
| *** RobertLJ_ has quit IRC | 03:21 | |
| *** adjohn has joined #openstack | 03:27 | |
| *** pvo has joined #openstack | 03:27 | |
| *** ChanServ sets mode: +v pvo | 03:27 | |
| *** zenmatt has quit IRC | 03:28 | |
| *** londo__ has joined #openstack | 03:53 | |
| *** londo has quit IRC | 03:53 | |
| *** binaryWarrior has quit IRC | 03:54 | |
| *** londo__ has quit IRC | 04:02 | |
| *** londo__ has joined #openstack | 04:03 | |
| *** tonywolf has joined #openstack | 04:05 | |
| *** kashyapc has joined #openstack | 04:10 | |
| *** pvo has quit IRC | 04:13 | |
| *** londo__ has quit IRC | 04:20 | |
| *** abecc has quit IRC | 04:22 | |
| *** londo__ has joined #openstack | 04:22 | |
| *** tr3buchet has quit IRC | 04:26 | |
| *** sophiap has joined #openstack | 04:27 | |
| *** rajijoom has joined #openstack | 04:31 | |
| *** jdmaturen has joined #openstack | 04:34 | |
| *** zenmatt has joined #openstack | 04:40 | |
| *** pvo has joined #openstack | 04:40 | |
| *** ChanServ sets mode: +v pvo | 04:40 | |
| *** pvo has left #openstack | 04:57 | |
| *** clayg_ is now known as clayg | 05:07 | |
| *** sirp1 has quit IRC | 05:07 | |
| *** pharkmillups has joined #openstack | 05:13 | |
| *** cglee has joined #openstack | 05:19 | |
| *** jdmaturen has quit IRC | 05:28 | |
| *** BartVB has joined #openstack | 05:29 | |
| *** BartVB has quit IRC | 05:34 | |
| *** pharkmillups has quit IRC | 05:34 | |
| *** BartVB has joined #openstack | 05:40 | |
| *** miclorb has quit IRC | 05:42 | |
| *** miclorb has joined #openstack | 05:42 | |
| *** miclor___ has joined #openstack | 05:42 | |
| *** miclorb has quit IRC | 05:43 | |
| *** jdmaturen has joined #openstack | 05:47 | |
| *** rajijoom has quit IRC | 06:02 | |
| *** Mr_T has quit IRC | 06:13 | |
| *** BartVB has quit IRC | 06:13 | |
| *** Mr_T has joined #openstack | 06:14 | |
| *** almaisan-away is now known as al-maisan | 06:20 | |
| *** allsystemsarego has joined #openstack | 06:26 | |
| *** dmd17_ has joined #openstack | 06:42 | |
| *** adjohn_ has joined #openstack | 06:42 | |
| *** adjohn has quit IRC | 06:43 | |
| *** adjohn_ is now known as adjohn | 06:43 | |
| *** jdmaturen has quit IRC | 06:43 | |
| *** dmd17 has quit IRC | 06:44 | |
| *** dmd17_ is now known as dmd17 | 06:44 | |
| *** CommunityString has joined #openstack | 07:01 | |
| *** CommunityString has left #openstack | 07:02 | |
| *** DogWater has quit IRC | 07:18 | |
| *** BartVB has joined #openstack | 07:21 | |
| *** stewart has quit IRC | 07:35 | |
| *** cglee has quit IRC | 07:35 | |
| *** miclor___ has quit IRC | 08:03 | |
| *** rajijoom has joined #openstack | 08:15 | |
| *** rajijoom has quit IRC | 08:19 | |
| *** kashyapc has quit IRC | 08:29 | |
| *** rajijoom has joined #openstack | 08:35 | |
| *** kashyapc has joined #openstack | 08:42 | |
| *** kashyapc_ has joined #openstack | 08:52 | |
| *** kashyapc has quit IRC | 08:56 | |
| *** hazmat has joined #openstack | 09:04 | |
| *** jtdowney has joined #openstack | 09:06 | |
| *** calavera has joined #openstack | 09:11 | |
| *** dmd17 has quit IRC | 09:12 | |
| *** adjohn has quit IRC | 09:12 | |
| *** matclayton has joined #openstack | 09:34 | |
| *** stewart has joined #openstack | 09:48 | |
| *** metoikos has joined #openstack | 09:55 | |
| *** tomo__ has joined #openstack | 09:56 | |
| *** jtdowney has quit IRC | 10:24 | |
| *** jonesy_ has quit IRC | 10:30 | |
| *** tomo__ has quit IRC | 10:47 | |
| *** tomo__ has joined #openstack | 10:51 | |
| *** tomo__ has joined #openstack | 10:52 | |
| *** silassewell has joined #openstack | 10:59 | |
| *** jonesy_ has joined #openstack | 11:08 | |
| *** miclorb_ has joined #openstack | 11:15 | |
| *** justinsheehy has quit IRC | 11:20 | |
| *** ctennis has quit IRC | 11:21 | |
| *** cjwilson has quit IRC | 11:24 | |
| *** justinsheehy has joined #openstack | 11:26 | |
| *** justinsheehy has quit IRC | 11:36 | |
| *** miclorb_ has quit IRC | 11:43 | |
| *** ctennis has joined #openstack | 11:43 | |
| *** calavera has quit IRC | 11:43 | |
| *** calavera has joined #openstack | 11:43 | |
| *** gundlach has joined #openstack | 11:46 | |
| *** calavera has quit IRC | 11:48 | |
| *** silassewell has quit IRC | 11:49 | |
| *** calavera has joined #openstack | 11:50 | |
| *** jaypipes has quit IRC | 11:51 | |
| *** carolinad28 has joined #openstack | 11:53 | |
| *** calavera has quit IRC | 11:54 | |
| *** calavera has joined #openstack | 12:06 | |
| *** justinsheehy has joined #openstack | 12:14 | |
| *** mattf has quit IRC | 12:14 | |
| *** rnewson has quit IRC | 12:14 | |
| *** calavera_ has joined #openstack | 12:15 | |
| *** calavera has quit IRC | 12:16 | |
| *** jsmith-away is now known as jsmith | 12:16 | |
| *** calavera_ has quit IRC | 12:19 | |
| *** justinsheehy has quit IRC | 12:20 | |
| *** metoikos has quit IRC | 12:32 | |
| *** Podilarius has joined #openstack | 12:32 | |
| *** jdarcy has joined #openstack | 12:33 | |
| *** cjwilson has joined #openstack | 12:38 | |
| *** DogWater has joined #openstack | 12:49 | |
| *** justinsheehy has joined #openstack | 12:49 | |
| *** jsmith is now known as jsmith-away | 12:56 | |
| *** rnewson has joined #openstack | 13:00 | |
| *** jsmith-away is now known as jsmith | 13:05 | |
| *** codejunkie has joined #openstack | 13:06 | |
| *** aghaster has joined #openstack | 13:12 | |
| aghaster | Hi | 13:12 |
| notmyname | morning aghaster | 13:22 |
| *** calavera has joined #openstack | 13:39 | |
| *** StylusEater_work has joined #openstack | 13:39 | |
| *** jaypipes has joined #openstack | 13:40 | |
| StylusEater_work | gm | 13:40 |
| jaypipes | StylusEater_work: morning. :) | 13:40 |
| *** dendrobates is now known as dendro-afk | 13:43 | |
| *** kashyapc_ has quit IRC | 13:50 | |
| *** maximilien has joined #openstack | 13:50 | |
| StylusEater_work | jaypipes: maybe you know the answer to this... but am I correct in assuming hudson pushes the latest successful build for swift to the ppa packages area on launchpad? I'm writing an SAIO script and wanted to know of a reliable place to grab the distributable? | 13:55 |
| creiht | StylusEater_work: I wouldn't rely on the PPAs for swift itself, we are still working through some packaging stuff | 13:57 |
| creiht | The dependencies in the PPA should be fine though (webob, eventlet, etc.) | 13:57 |
| StylusEater_work | +creiht: so the safest would be grab the latest from stable? | 13:59 |
| creiht | StylusEater_work: For swift, the best thing currently is to grab the latest code from lp:swift | 13:59 |
| creiht | For the dependencies the debs in the swift PPA should be fine | 14:00 |
| StylusEater_work | +creiht: ok (was just gonna say if there is such a thing... :-) ) thanks | 14:00 |
| *** tonywolf has quit IRC | 14:02 | |
| *** mef has joined #openstack | 14:09 | |
| alekibango | anotherjesse: talking about orm, i very much like sqlalchemy. but to make it redundant you would need to use cluster of sql servers -- which would take its toll for cloud management purposes :D | 14:09 |
| redbo | no no, it's great having 20 or 30 db servers. no availability problems or anything. | 14:10 |
| StylusEater_work | +redbo: need a sarcasm tag... haha | 14:13 |
| alekibango | mtaylor: please PUSH IT, sqlalchemy would rock! | 14:16 |
| *** pcrews has joined #openstack | 14:16 | |
| creiht | How much db type stuff actually has to be done? | 14:18 |
| creiht | Does it really buy you much using sqlalchemy? | 14:18 |
| * creiht notes that he likes sqlalchemy, but it does have costs | 14:18 | |
| redbo | like your soul | 14:18 |
| creiht | haha | 14:18 |
| *** abecc has joined #openstack | 14:19 | |
| redbo | I'm not a big fan of ORMs these days, unless you have a bunch of big complicated businessy objects that you need to manage. | 14:20 |
| jaypipes | redbo: better than hacking a specific dkvs' API all over your codebase ;) | 14:21 |
| creiht | jaypipes: I think there can be a happy medium | 14:22 |
| jaypipes | alekibango: the problem with an ORM like SQLAlchemy (even if you use a wrapper like Elixir) is that they simply are not built for async operations. The trick with Nova is going to be creating an ORM-*like* wrapper that enables us to work with all the persistent objects in a consistent manner regardless of whether the backing data store in a DKVS, a KVS, or an RDBMS... | 14:23 |
| creiht | and since when did Bayer start liking KVS? :) | 14:23 |
| jaypipes | alekibango: I've been hacking on that for 2 weeks and made some progress, but the current codebase is very Redis-specific. | 14:23 |
| creiht | an ORM also seems like a lot of overkill for a KVS | 14:24 |
| jaypipes | creiht: depends on how you are using the KVS. In Nova, a lot of the Redis API is used, and lots of objects are associated with each other...not sure Swift is similar. | 14:25 |
| creiht | jaypipes: http://bazaar.launchpad.net/~hudson-openstack/swift/trunk/annotate/head:/swift/common/db.py | 14:26 |
| creiht | Is what we do currently in swift | 14:26 |
| gholt | Swift's KVs in memcached are definitely not relational. | 14:27 |
| jaypipes | creiht: right, that's very SQLiter specific. | 14:27 |
| gholt | I don't think he meant our sqlite stuff, but maybe I'm off. That isn't exactly KVs, but isn't overly relational either, heh. | 14:27 |
| creiht | It is indeed, but if we wanted to use a different backend (like berkley db) then you can rewrite that one section with the same interface, and you are good to go | 14:28 |
| creiht | Still some work | 14:28 |
| creiht | But you don't have the ORM overhead | 14:28 |
| redbo | That's probably a pretty leaky abstraction | 14:29 |
| creiht | It is | 14:29 |
| jaypipes | creiht: which is precisely what something like SQLalchemy already does for you... | 14:29 |
| creiht | but if we wanted to, we could make it better | 14:29 |
| creiht | It wasn't really meant to be made like that | 14:29 |
| creiht | jaypipes: If you are alright with the overhead, and it supports the backends, then yes that is fine | 14:29 |
| redbo | actually I made that file because EJ didn't like seeing SQL all over the rest of the code. | 14:29 |
| creiht | hehe | 14:29 |
| creiht | I like having the SQL in one file :) | 14:29 |
| redbo | he was like, "move it all in one place." | 14:29 |
| redbo | and I was like FIEN | 14:30 |
| creiht | SQLAlchemy has a measureable amount of overhead | 14:30 |
| jaypipes | creiht: have you actually measured that? | 14:30 |
| creiht | yes | 14:30 |
| jaypipes | creiht: results? | 14:30 |
| StylusEater_work | +redbo: why would you put sql all over the code? eeek | 14:30 |
| creiht | It is one of the reasons why we don't use SQLAlchemy any more | 14:30 |
| jaypipes | creiht: what were the results of your findings? | 14:31 |
| StylusEater_work | +redbo: concentrate it one place... make a clean interface, then you can use other dbs much more easily, no? | 14:31 |
| creiht | redbo: do you still have those results? | 14:31 |
| creiht | StylusEater_work: that is closer to what we ended up with | 14:31 |
| redbo | I'm pretty sure swift isn't ever going to use a different database | 14:31 |
| creiht | yeah | 14:31 |
| * gholt watches creiht punt to redbo | 14:31 | |
| creiht | It wasn't really designed with that in mind | 14:32 |
| *** silassewell has joined #openstack | 14:32 | |
| creiht | jaypipes: if you are planning on using it, shouldn't you do the due dilligence :) | 14:32 |
| jaypipes | redbo: you are thinking in terms of rackspace only. | 14:32 |
| jaypipes | creiht: I already have. | 14:32 |
| jaypipes | creiht: the vast majority of performance issues with SQLAlchemy have to do with poorly written schemas and SQL queries. At least in my experience. | 14:33 |
| gholt | I can't remember the specific overhead amount for our prior project. It was at least 10% or we probably wouldn't have cared. However, our prior project also had things way over.... relationalized? | 14:33 |
| StylusEater_work | +redbo: why wouldn't one leave it open to other db's? | 14:34 |
| redbo | we didn't replace sqlalchemy because of performance anyway (though it sucked). It's because we were tired of trying to trick sqlalchemy into doing the queries that we wanted it to. | 14:34 |
| jaypipes | gholt: hehe, good term :) | 14:34 |
| creiht | And that's true as well :) | 14:34 |
| gholt | Ah, that's right. I'm glad redbo has a memory, unlike me. | 14:34 |
| * creiht notes that a lot of this part was before I started on the project | 14:35 | |
| *** sirp1 has joined #openstack | 14:35 | |
| gholt | I remember letterj and torch grumbling at us for hours to quite doing stupid queries that crashed things. :) | 14:35 |
| redbo | the DBAs would sit and watch every query the app made and say "WHY IS IT DOING THAT!" and we never had a good answer. | 14:35 |
| gholt | And we're like: It's not me... It's the ORM! | 14:36 |
| jaypipes | gholt: yup, that's been my experience as well (not just SQLAlchemy, but many ORMs...) | 14:36 |
| StylusEater_work | gholt: hehe | 14:37 |
| *** errr has quit IRC | 14:37 | |
| jaypipes | gholt: it's not the overhead of the ORM really, but that they have a tendency to produce poor SQL queries when the schema is not properly designed or is outside of the "traditional enterprise application SQL schemas" | 14:37 |
| creiht | I'm not totally agains ORMs, though it gets harder to sell me on them now a days | 14:37 |
| gholt | I like magic. In my movies. :) | 14:38 |
| jaypipes | gholt: and it comes to a point of "man, getting this SQL to be good with our schema just isn't worth the effort...let's just write SQL by hand" ;) | 14:38 |
| letterj | In some cases we would see queries making two trips to the DB. | 14:38 |
| redbo | and yeah, when profiling the app, the query compiler in sqlalchemy was always a decent percentage of the runtime. I wouldn't have cared about that so much, because we can always add more web servers. But we just weren't doing anything complicated where we needed an object-based interface. | 14:38 |
| *** calavera has quit IRC | 14:38 | |
| termie | joshuamckenty: https://code.launchpad.net/~termie/nova/mega_flags/+merge/31312 | 14:38 |
| joshuamckenty | danke | 14:39 |
| *** mrayzenoss has joined #openstack | 14:39 | |
| *** mrayzenoss is now known as mray | 14:39 | |
| *** mray has joined #openstack | 14:39 | |
| creiht | jaypipes: I guess where I was getting at, is the DB related stuff that you guys are doing in Nova sufficient enough that the ORM provides enough benefit? | 14:40 |
| creiht | If it is, then cool | 14:40 |
| * creiht is still interested to see how an ORM is going interact with a KVS | 14:40 | |
| jaypipes | creiht: well...not sure :) first things first should be an abstraction layer above Redis/SQL. Then perhaps look at using an ORM ;) | 14:41 |
| jaypipes | creiht: problem with Nova is that the authorization objects are stored in LDAP while the rest of the objects are not. So, you can't "associate" some objects with each other (in other words you can't associate a project (stored in LDAP) with a Vlan (stored in Redis) :( | 14:42 |
| jaypipes | creiht: at least not naturally, through the same interface... | 14:42 |
| *** pvo has joined #openstack | 14:43 | |
| *** ChanServ sets mode: +v pvo | 14:43 | |
| creiht | hah... sqla should work awesome with LDAP :) | 14:43 |
| jaypipes | creiht: heh | 14:44 |
| jaypipes | creiht: LDAP is complete ass-baggery. | 14:44 |
| * creiht would love to see the look on Mike's face when someone asks him to write an SQLA backend for ldap | 14:44 | |
| jaypipes | creiht: hehe, I've already had an email conversation with Bayer about a Redis backend...was met with a tepid response, but he felt our pain :) | 14:45 |
| creiht | I wonder if he is finally caving to pressure to support KVS, as he has avoided them like the plague in the past | 14:46 |
| creiht | though I'm still curious as to how KVS are going to work in an ORM | 14:46 |
| clayg | creiht: http://github.com/iamteem/redisco - object mapper for redis | 14:47 |
| *** metoikos has joined #openstack | 14:49 | |
| jaypipes | creiht: yeah, he is... http://www.sqlalchemy.org/trac/ticket/1518 | 14:50 |
| creiht | haha | 14:50 |
| jaypipes | creiht: you can tell from that he's slightly less than thrilled about it ;) | 14:51 |
| creiht | Oh I know he is :) | 14:51 |
| creiht | I actually side with him... I don't think SQLAlchemy should support KVS, but that could be a whole other debate :) | 14:52 |
| redbo | there's such a big difference in features between the kvs | 14:53 |
| hazmat | sqlalchemy is not for kvs systems | 14:53 |
| creiht | It seems like there will be so many problems trying to emulate SQL functionality on top of a KVS | 14:54 |
| clayg | KVSAlchemy? | 14:54 |
| *** al-maisan is now known as almaisan-away | 14:54 | |
| jaypipes | clayg: :) | 14:54 |
| clayg | it just SOUNDS like a disaster? | 14:54 |
| hazmat | effectively folks doing kvs systems right their own object mapping layer.. python mongodb has a few nice ones.. | 14:54 |
| jaypipes | creiht: ya. there's that dang transactional thang. | 14:54 |
| jaypipes | hazmat: yup. that's what we're trying to do in Nova... | 14:55 |
| hazmat | well most of the kvs aren't transactional.. with some exceptions.. so doing object mapping scoped to sessions and transactions is just a programming mismatch waiting for bugs | 14:55 |
| jaypipes | hazmat: ++ | 14:55 |
| creiht | hazmat++ | 14:55 |
| hazmat | jaypipes, in terms of contributing to nova.. is there a list of things to do..? just go find a bug and implement something? | 14:56 |
| hazmat | really that question is for anyone ;-) | 14:56 |
| creiht | someone needs to make an impedence mismatch abstraction pattern :) | 14:57 |
| hazmat | i'd still like to hear some resolution of the eventlet/twisted thing.. at least that would give me an idea of stuff to do (like rip out sync calls)... or drop carrot | 14:58 |
| gundlach | hazmat: check out https://blueprints.launchpad.net/nova | 14:58 |
| hazmat | but maybe just a decent build systems | 14:58 |
| _0x44 | gundlach: Has the low-hanging fruit blueprint been created yet? | 14:58 |
| hazmat | i was thinking of a buildout | 14:58 |
| hazmat | http://www.buildout.org/ | 14:58 |
| gundlach | _0x44: no idea | 14:58 |
| hazmat | also does anyone know why the current install thingy hardcodes a custom twisted tarball? | 14:59 |
| _0x44 | IIRC, it's because twisted needed to be patched or the unit tests would flip out over something silly. | 14:59 |
| hazmat | i guess i could do the pyflakes/lint thingy, it would be a good exercise to poke around some more of the code base. | 14:59 |
| *** gundlach has quit IRC | 15:00 | |
| *** gundlach has joined #openstack | 15:00 | |
| StylusEater_work | hazmat: I have some idle hosts if you need some for a build farm. | 15:01 |
| *** gundlach_ has joined #openstack | 15:03 | |
| *** avsm has joined #openstack | 15:03 | |
| *** gundlach has quit IRC | 15:05 | |
| *** brd_from_italy has joined #openstack | 15:06 | |
| hazmat | StylusEater_work, thanks, no need atm, i've got a few.. | 15:07 |
| *** hazmat has quit IRC | 15:08 | |
| *** hazmat has joined #openstack | 15:08 | |
| notmyname | creiht: I'm starting to work on merging the stats system for swift. would it make sense to put it under a "stats" dir under swift (/swift/stats/*)? | 15:17 |
| creiht | hrm | 15:18 |
| notmyname | or ./logging/? or what? | 15:18 |
| notmyname | log_processing? | 15:18 |
| creiht | Maybe we should talk about that this evening? | 15:18 |
| notmyname | really, I don't feel that way about you. I'm happily married ;-) | 15:19 |
| creiht | hah | 15:19 |
| creiht | this afternoon? | 15:19 |
| creiht | :) | 15:19 |
| notmyname | ah, are you in the office today? | 15:19 |
| creiht | yes | 15:19 |
| creiht | Will be all week | 15:20 |
| notmyname | ah, ok | 15:20 |
| creiht | I have some ideas, but probably easier to hash it out in person | 15:20 |
| *** mray has quit IRC | 15:20 | |
| *** jbryce has joined #openstack | 15:22 | |
| notmyname | ok. fwiw, I was thinking that it makes more sense to put all the stats stuff in one place rather than spread out (account stats under account, access logs under proxy, and the collating somewhere else) | 15:22 |
| creiht | k | 15:22 |
| creiht | In it's own project? :) | 15:22 |
| creiht | jk | 15:22 |
| notmyname | heh | 15:23 |
| *** silassewell has quit IRC | 15:23 | |
| *** dendro-afk is now known as dendrobates | 15:23 | |
| notmyname | of course, if we go that route, why stop there? why not have each component in its own project and swift becomes the aggregate of each ;-) | 15:24 |
| notmyname | (that's not a serious request) | 15:24 |
| creiht | notmyname: be careful what you ask for :) | 15:24 |
| *** mrayzenoss has joined #openstack | 15:26 | |
| *** mrayzenoss is now known as mray | 15:26 | |
| *** mray has joined #openstack | 15:26 | |
| _0x44 | If stats was it's own project, it could be shared between nova and swift. | 15:31 |
| *** kashyapc has joined #openstack | 15:32 | |
| notmyname | that's an interesting thought | 15:33 |
| * creiht ponders | 15:33 | |
| notmyname | in porting this to swift (from internal cloud files stuff), I'm trying to make it a little more generic | 15:33 |
| notmyname | the basic idea is that log files are uploaded to cloud files then parsed later according to their type (stat logs have their own parser, proxy [access] logs have their own parser, and, internally, we'll have another for CDN logs) | 15:34 |
| notmyname | so I'm not sure if the first stab at it will be generic enough for nova, but it may work. the biggest hurdle will be storing the logs. it's not too bad for swift, since it is a storage product :-) | 15:35 |
| _0x44 | So in order for nova to share the stats piece, it would have to write logs to swift. | 15:35 |
| _0x44 | Which probably couples them too closely, unless it's optional. Is it worth the work for optional? | 15:36 |
| notmyname | perhaps, but that's not a key piece. it's just really convenient for swift to do that | 15:36 |
| gholt | Doesn't nova have a backing store? That they're considering using swift for? | 15:37 |
| notmyname | the point is being able to access the logs (which are large) from a single place | 15:37 |
| _0x44 | gholt: Yes, but I don't know if considering using swift for means requiring swift to use nova | 15:37 |
| gholt | But yeah, I guess thinking outside just swift/nova is a good idea, heh. | 15:37 |
| gholt | I'm not sure if diluting stats that far would end up making it a whole 'nother project just to use it with swift though. | 15:39 |
| creiht | I say for now we put it in swift, and when you guys are ready to talk about stats for nova, we can see if what we have is usable, and if so we can extract it then | 15:40 |
| notmyname | I like that idea. it will certainly allow something to get out there sooner | 15:41 |
| creiht | notmyname: never thought you would hear me say that, would ya? :) | 15:42 |
| notmyname | it is a little surprising ;-) | 15:42 |
| *** rnewson has quit IRC | 15:46 | |
| *** rnewson has joined #openstack | 15:47 | |
| eday | notmyname, _0x44: Having stats/logs/... be a shared component is certainly on the roadmap, this was the whole 'openstack firehose' discussion. Swift could be one place to store that data (probably the default for large installations that use swift for other things too) | 15:48 |
| _0x44 | eday: How far along the roadmap? 3 months or 9? | 15:48 |
| eday | but we don't want to *require* logs be stored there at this generic layer, you may not care about saving this data anywhere and instead just want to run it through filters for notifications | 15:48 |
| eday | _0x44: I have no idea where, but just something to keep in mind when working out the APIs needed now :) | 15:49 |
| notmyname | eday: I think the idea for the stats system in swift is as an additional component that's not required for a system that stores data. but it's something that's sure nice to have as a starting point and one of the first things "management" is going to ask for in a company that installs swift/openstack | 15:50 |
| _0x44 | If it's not something we're working on for the current release, then doing a semi-generic stats thing in swift and abstracting it out when we start working on the firehose is probably the best course of action _now_ | 15:50 |
| creiht | eday: That's why I'm saying that we keep it in swift now, and when we get to the point to talk about shared stats, then we can decide if we want to use what we have in swift as a base, or start from scratch | 15:50 |
| * creiht doesn't want to abstract just for abstraction's sake :) | 15:51 | |
| eday | creiht, _0x44: Agreed, don't do too much now, but just something to keep in mind. | 15:52 |
| eday | or if you guys are feeling really ambitious and have the time, start working out the abstraction so Nova could start using it sooner :) | 15:54 |
| notmyname | the "have the time" is the kicker, isn't it :-) | 15:55 |
| creiht | We're doing good just to get the stats stuff into swift :) | 15:56 |
| aghaster | quick question, is swift supported on windows server 2008? | 15:57 |
| * creiht has no idea :) | 15:57 | |
| notmyname | I like the idea of having some general stats integration, but I don't want to delay things waiting for perfect abstraction and portability. put it in swift now, port to a generic swift/nova solution later | 15:57 |
| gholt | We knew someone would ask that eventually. :) | 15:58 |
| notmyname | aghaster: we develop against ubuntu currently | 15:58 |
| gholt | As a client, windows has been used just fine. Not as a swift cluster however. | 15:58 |
| eday | notmyname: well, nothing is going to be perfect, just something to start iterating over as folks have time.. but yeah, def don't worry if there is no time for it | 15:58 |
| StylusEater_work | aghaster: why not use the native OS? | 15:59 |
| aghaster | StylusEater: I'm all for ubuntu, but I just wanted to make sure that it's required. It'd be for work, and they're all using windows here. Getting the IT guy to set up a Linux box is just more effort than a windows box | 16:00 |
| StylusEater_work | aghaster: hrm...ok | 16:01 |
| notmyname | aghaster: it's never been done. there are a few linux-specific things that may have to be patched (fallocate), and you would need a FS that supports xattrs | 16:01 |
| creiht | aghaster: We haven't targeted windows, though that isn't to say that it could be made to work on windows with some changes | 16:01 |
| notmyname | but if you get it working, awesome | 16:02 |
| creiht | Of course patches are welcome :) | 16:02 |
| *** rajijoom has quit IRC | 16:02 | |
| notmyname | perhaps if those linux-specific things were worked around, other OSs (like BSD) could be used too | 16:03 |
| StylusEater_work | +notmyname: +1 | 16:03 |
| creiht | notmyname: I only OpenBSD variants don't support xattrs very well | 16:03 |
| *** gundlach has joined #openstack | 16:03 | |
| creiht | or I guess I should say, FreeBSD supports them | 16:04 |
| *** gundlach_ has quit IRC | 16:05 | |
| *** brd_from_italy has quit IRC | 16:06 | |
| notmyname | seems like the answer is the same as the previous discussion: "creiht doesn't want to abstract just for abstraction's sake :)" | 16:07 |
| notmyname | if there are useful patches that make lack of xattrs and fallocate work, yay. but not if it makes everything else worse | 16:07 |
| creiht | notmyname: actually redbo already fixed the fallocate issue | 16:08 |
| notmyname | oh, cool | 16:08 |
| notmyname | well then, ignore everything I said about that | 16:08 |
| creiht | If it can't load the lib, it logs a warning and continues | 16:08 |
| * notmyname is off to lunch | 16:08 | |
| *** avsm has quit IRC | 16:10 | |
| *** jdmaturen has joined #openstack | 16:12 | |
| *** gnrfan has joined #openstack | 16:21 | |
| *** mef is now known as _mattf | 16:37 | |
| gundlach | any reason http://wiki.openstack.org/InstallationNova20100729 points people to bazaar download site instead of sudo apt-get install bzr? | 16:48 |
| gundlach | like, do we need to be working with the tip or something? | 16:48 |
| gundlach | i'm going to change it -- feel free to change it back if i'm wrong | 16:50 |
| jk0 | there was a reason for that | 16:52 |
| jk0 | something in the newer release that's not in the Ubuntu repo, I think | 16:52 |
| gundlach | k, if you're sure, i'll revert it -- any idea who to ask what the specific feature was that we needed? i remember talk about new features going into the tip but didn't remember them being vital for working with nova | 16:55 |
| hazmat | do each of the openstack projects require different contributor agreements or is the swift one good for nova? | 16:55 |
| *** avsm has joined #openstack | 16:56 | |
| *** codejunkie has quit IRC | 16:57 | |
| jk0 | gundlach: sorry, I don't recall | 16:58 |
| *** avsm has quit IRC | 16:58 | |
| notmyname | hazmat: there is one for all of openstack | 16:58 |
| gundlach | jk0: k | 16:58 |
| hazmat | notmyname, thks | 16:58 |
| *** avsm has joined #openstack | 16:58 | |
| *** kallistec has quit IRC | 17:02 | |
| *** gustavomzw has joined #openstack | 17:04 | |
| *** aghaster has quit IRC | 17:08 | |
| *** Sander__ has quit IRC | 17:13 | |
| uvirtbot | New bug: #613075 in nova "Support authentication" [Undecided,New] https://launchpad.net/bugs/613075 | 17:15 |
| uvirtbot | New bug: #613076 in nova "Support JSON or XML in API" [Undecided,New] https://launchpad.net/bugs/613076 | 17:15 |
| zul | did nova land in ubuntu archive yet? | 17:17 |
| uvirtbot | New bug: #613077 in nova "Support content compression in API" [Undecided,New] https://launchpad.net/bugs/613077 | 17:20 |
| uvirtbot | New bug: #613079 in nova "Support persistent connections in the API" [Undecided,New] https://launchpad.net/bugs/613079 | 17:20 |
| *** kashyapc has quit IRC | 17:21 | |
| *** avsm has quit IRC | 17:22 | |
| *** avsm has joined #openstack | 17:22 | |
| eday | gundlach: If we create those as blueprints instead of bugs, we could do dependency tracking in LP (new API components) | 17:22 |
| gundlach | oh geez, i'm going to be spamming this room | 17:23 |
| gundlach | eday: sorry that i'm new to launchpad -- so what's the difference between a blueprint and a bug, then? | 17:23 |
| gundlach | are you saying bugs don't support dependency, marking as duplicate, things like that? they're more lightweight? | 17:23 |
| eday | gundlach: well, I see bugs as things that are broken in the system (existing functionality), where blueprints are todo items | 17:24 |
| uvirtbot | New bug: #613082 in nova "Support efficient polling in API" [Undecided,New] https://launchpad.net/bugs/613082 | 17:26 |
| eday | gundlach: both allow for branch/dev tracking, but blueprints also allow for dependency tracking. For example, "open stack API" could be an umbrella blueprint with the others listed. Obviously we don't need to do all that if its just a couple folks working on it, but it can be useful for new folks/managers/... to see whats going on | 17:26 |
| *** sophiap has quit IRC | 17:26 | |
| gundlach | eday: well, i'll do whatever you think is best in launchpad's mindset. i usually treat a bug tracker as a place for any kind of issue -- bug/enhancement/task/other. | 17:27 |
| gundlach | i'm tagging all these bugs as "api" so they are in an "umbrella" in that sense -- is that enough management, do you think? | 17:28 |
| gundlach | (see https://bugs.launchpad.net/nova/+bugs?field.tag=api) | 17:29 |
| eday | gundlach: I think either will be fine, just wanted to let you know about blueprints in case you had not used them :) | 17:29 |
| eday | gundlach: since you've already got some in here, probably doesn't make sense to move them now | 17:30 |
| gundlach | eday: thanks :) i had seen blueprints but forgot their existence. | 17:30 |
| gundlach | ok, will stick with bugs. | 17:30 |
| gundlach | the dichotomy makes me itch for a unified system, but that's probably just because i spend all my spare time in google project hosting, which puts everything into the same basket. | 17:30 |
| gundlach | (http://adblockforchrome.googlecode.com) | 17:31 |
| eday | gundlach: yeah. I see them as something broken in current vs new feature todo. that could just be a tag though in the same system. | 17:31 |
| *** maximilien has quit IRC | 17:31 | |
| eday | gundlach: mtaylor also has opinions on bugs vs blueprints, as you probably saw on the mailing list :) | 17:32 |
| *** kallistec has joined #openstack | 17:32 | |
| gundlach | eday: that's a pretty classic argument -- isn't the lack of a required feature a bug? etc :) tdd makes all that fuzzy | 17:32 |
| eday | gundlach: yup :) | 17:32 |
| gundlach | eday: probably did, but i guess it washed over me as i didn't have enough familiarity with launchpad to really grasp it | 17:32 |
| gundlach | anyway, thx for the input, and i guess i'll steamroll ahead with bug reporting :) | 17:32 |
| eday | gundlach: have you, or have you seen other apps, that stacked WSGI interfaces in the same app? | 17:38 |
| gundlach | eday: maybe i'm misunderstanding you, but yes -- one of the major selling points of WSGI (and Ruby's equivalent, Rack) is that you can stack them to make "middleware" | 17:39 |
| gundlach | http://wsgi.org/wsgi/Middleware_and_Utilities | 17:39 |
| *** codejunkie has joined #openstack | 17:40 | |
| eday | gundlach: aha, I just stumbled upon that page too. I see | 17:40 |
| eday | gundlach: I've not done any WSGI stuff before, didn't realize it was designed for this | 17:40 |
| gundlach | gotcha | 17:40 |
| *** rajijoom has joined #openstack | 17:40 | |
| gundlach | yeah, it helps tease apart layers of functionality while not adding new server "binaries" to the chain | 17:41 |
| gundlach | so i was hoping this would ease your mind about added complexity for small installations, while helping break the system into smaller pieces, and making it easy to pull a part into its own binary if needed for scaling | 17:41 |
| gundlach | at a previous gig i did authentication as a Rack middleware calling sideways to an authentication service | 17:42 |
| gundlach | the middleware could be used by lots of different systems that needed to authenticate, and it itself used an authentication library that could be embedded deeper in the system. | 17:42 |
| gundlach | man, until i just said that i didn't realize how germaine that architecture was to our discussion :) | 17:42 |
| eday | gundlach: I think WSGI middleware is exactly what we need looking at it. easy to chain either in process or as proxies | 17:44 |
| redbo | swift works the same | 17:44 |
| gundlach | eday: oh good, i'm glad, as will be jorge i expect :) | 17:45 |
| uvirtbot | New bug: #613090 in nova "Support rate limiting in API" [Undecided,New] https://launchpad.net/bugs/613090 | 17:45 |
| uvirtbot | New bug: #613092 in nova "Support programmatic rate limit querying in API" [Undecided,New] https://launchpad.net/bugs/613092 | 17:51 |
| *** zebziggle has joined #openstack | 17:51 | |
| mtaylor | gundlach: I see them as descrbing what vs. how ... bug == what's wrong/what's missing - is a description of the problem - the blueprint is the how - is a description of the implementation or solution | 17:54 |
| gundlach | mtaylor: ah, i see. | 17:55 |
| *** jbryce has quit IRC | 17:55 | |
| gundlach | though it seems then like every bug would require a blueprint? (maybe just the large enough bugs) | 17:56 |
| mtaylor | gundlach: I think just the large enough bugs - some bugs the description of the problem is really all you need | 17:56 |
| gundlach | got it. so is it normal practice for a (large enough bug) report to then link to a blueprint for its solution? | 17:56 |
| *** avsm has quit IRC | 17:57 | |
| mtaylor | gundlach: but if blueprints had a decent conversation mechanism like merge requests did, it would be a great place to hold things like "what do we do for an orm" or something | 17:57 |
| mtaylor | gundlach: in theory. in practice we don't want to kill devs and I think it's still a little clunkier than it could be | 17:57 |
| mtaylor | gundlach: but I think that would be the goal if we can get the tooling worked out properly | 17:58 |
| gundlach | mtaylor: i see. "we" -- are you a contributor to launchpad? (pardon if we've met already, there are too many irc handles about :) | 17:58 |
| *** avsm has joined #openstack | 18:00 | |
| _0x44 | mtaylor: I thought the solution to blueprints sucking was blueprints + etherpad? | 18:00 |
| mtaylor | gundlach: hehe. in theory ... my job description at rackspace involves dev tooling which extends to hacking on launchpad - but so far my todo list hasn't gotten cleared down to the launchpad fixes yet | 18:00 |
| mtaylor | _0x44: I think that's an excellent solution to the conversation/document side of things | 18:01 |
| gundlach | mtaylor: devil's advocate -- why would it be bad if we deleted the bug tracker concept from launchpad entirely? is it just that blueprints require more work to enter (aka the UI needs streamlining)? | 18:03 |
| eday | mtaylor, _0x44: THat's what I've been doing (spec URL is a etherpad link) | 18:03 |
| mtaylor | gundlach: well, the bug tracker interface is much more advanced for people reporting bugs and interacting with devs and stuff | 18:04 |
| _0x44 | mtaylor, eday: Yeah, I thought the consensus was that's what we should be doing. | 18:04 |
| eday | mtaylor, gundlach: I think I'm starting to converge on wanting a single tool that can suit both. | 18:04 |
| gundlach | mtaylor: ok. in the 5 or 6 bugs i entered i saw a paucity of controls (tie to another project or milestone, add an attachment, and leave a comment) but i must have not discovered all the features. | 18:05 |
| redbo | win 2 | 18:06 |
| gundlach | eday: certainly would make launchpad feel less busy to a newbie like me. i feel like it's covered in links that all do different flavors of the same job, and i don't know quite which to use for what purpose. | 18:06 |
| mtaylor | eday: I think (I hope) there are some UI things that could be done to make them behave more sensibly | 18:07 |
| mtaylor | eday: and to feel like one integrated tool rather that two different somewhat overlapping tools | 18:07 |
| anotherjesse | jaypipes: around? | 18:07 |
| gundlach | mtaylor: +1 | 18:08 |
| eday | mtaylor: yeah | 18:08 |
| *** jsmith is now known as jsmith-away | 18:17 | |
| *** anotherjesse is now known as anotherjesse-foo | 18:21 | |
| *** devcamcar has joined #openstack | 18:22 | |
| *** adrian_otto has joined #openstack | 18:24 | |
| *** devcamcar has quit IRC | 18:24 | |
| eday | gundlach: so, WSGI on twisted looks to be suboptimal at this point. it uses a different thread pool for wsgi apps (not part of the main event loop). There is some guy doing async-wsgi interface, but that was just in April | 18:25 |
| creiht | eday: WSGI is a no go for twisted, even more of a reason to use eventlet ;) | 18:26 |
| *** stewart has quit IRC | 18:26 | |
| uvirtbot | New bug: #613117 in nova "Support versioning in the API" [Undecided,New] https://launchpad.net/bugs/613117 | 18:26 |
| *** jsmith-away is now known as jsmith | 18:28 | |
| *** devcamcar has joined #openstack | 18:31 | |
| *** rajijoom has quit IRC | 18:33 | |
| StylusEater_work | bbl | 18:33 |
| *** StylusEater_work has left #openstack | 18:33 | |
| *** jamiew has joined #openstack | 18:34 | |
| *** ctennis has quit IRC | 18:41 | |
| *** stewart has joined #openstack | 18:44 | |
| *** devcamcar has quit IRC | 18:44 | |
| eday | creiht: so, looking at the swift wsgi code, it looks like you fork and each child accept()s directly on the shared socket? | 18:44 |
| creiht | correct | 18:44 |
| eday | creiht: is the main reason for this just to use all the cores on a system? or do you see children dieing from bugs? | 18:48 |
| redbo | both, though we don't have children die these days. | 18:48 |
| *** stewart has quit IRC | 18:49 | |
| *** stewart has joined #openstack | 18:49 | |
| *** rdw has quit IRC | 18:51 | |
| *** devcamcar has joined #openstack | 18:51 | |
| *** ctennis has joined #openstack | 18:52 | |
| *** ctennis has joined #openstack | 18:52 | |
| * creiht notes that we only currently "stack" auth with wsgi in swift, but there are other things that we would like to pull out into wsgi middleware | 18:54 | |
| *** tr3buchet has joined #openstack | 18:57 | |
| eday | creiht: perhaps the to-be-abstracted openstack-auth project will provide both the raw API and a WSGI middleware layer | 19:00 |
| *** mrayzenoss has joined #openstack | 19:01 | |
| *** mrayzenoss has joined #openstack | 19:01 | |
| *** brd_from_italy has joined #openstack | 19:02 | |
| creiht | eday: possibly, but depends on what you want for the domain knowledge for authing for the products | 19:02 |
| creiht | so for example, the auth would need to understand swift's url structure to correctly know which part is the account to send in auth | 19:02 |
| creiht | I imagine that auth could provide a library, or module that gives primative auth functions, that then swift calls from its wsgi | 19:03 |
| *** mray has quit IRC | 19:03 | |
| creiht | eday: http://bazaar.launchpad.net/~hudson-openstack/swift/trunk/annotate/head:/swift/common/auth.py | 19:04 |
| *** devcamcar has quit IRC | 19:04 | |
| creiht | In the above file, the middleware would be swift's responsibility, but the auth method could come from a shared auth library | 19:05 |
| creiht | (conceptually) | 19:05 |
| eday | creiht: yeah. if a WSGI middleware layer could not be cleanly abstracted, no point. but I thought it might be with a couple params (I've not looked yet) | 19:08 |
| eday | creiht: either way, probably just a few lines and then use the auth API :) | 19:08 |
| *** devcamcar has joined #openstack | 19:11 | |
| *** adrian_otto has quit IRC | 19:25 | |
| *** devcamcar has quit IRC | 19:25 | |
| uvirtbot | New bug: #613137 in nova "Use stable host keys for SSH" [Undecided,New] https://launchpad.net/bugs/613137 | 19:26 |
| *** BartVB has quit IRC | 19:28 | |
| *** devcamcar has joined #openstack | 19:32 | |
| *** adrian_otto has joined #openstack | 19:36 | |
| jaypipes | anotherjesse-foo: back now...on vacation in Florida..just back from a "lovely" afternoon of sweating my ass off. | 19:38 |
| eday | jaypipes: haha, it's not a dry heat down there? :) | 19:45 |
| *** devcamcar has quit IRC | 19:45 | |
| _0x44 | jaypipes: I'm writing my political representatives demanding that we switch from Fahrenheit to Celsius so the summers will be cooler. | 19:50 |
| *** rdw has joined #openstack | 19:51 | |
| *** devcamcar has joined #openstack | 19:52 | |
| *** sirp1 has quit IRC | 19:54 | |
| *** devcamcar has quit IRC | 19:54 | |
| *** anotherjesse-foo is now known as anotherjesse | 19:55 | |
| anotherjesse | jaypipes: so - we are debating moving to sqlalchemy or sqlobject for our backend | 19:55 |
| *** maximilien has joined #openstack | 19:55 | |
| *** silassewell has joined #openstack | 20:00 | |
| eday | anotherjesse: do you have a backend picked out yet that will give you sql? | 20:00 |
| *** devcamcar has joined #openstack | 20:00 | |
| anotherjesse | eday: well, we are debating between mysql & postgres - with the preference being postgres, but devcamcar will disagree | 20:01 |
| alekibango | anotherjesse: jaypipes: SQLAlchemy is really good | 20:01 |
| anotherjesse | alekibango: I've used it a couple years ago for a django project where the default orm wasn't good enough | 20:01 |
| creiht | eday: yup | 20:01 |
| alekibango | i got in touch with sqlalchemy via turbogears | 20:02 |
| alekibango | and i must say i still am in aw | 20:02 |
| alekibango | they did really good job | 20:02 |
| creiht | sqlobject is still around? | 20:02 |
| anotherjesse | http://www.sqlobject.org/ - dead? | 20:02 |
| creiht | Well the question was more meant in terms of actively supported? | 20:03 |
| alekibango | http://www.sqlalchemy.org/ FTW | 20:03 |
| anotherjesse | understood - checking mailing lists / ... | 20:03 |
| alekibango | never seen sqlobject but imho this speaks volumes: http://google.com/trends?q=sqlalchemy%2Csqlobject | 20:04 |
| creiht | sqlobject has been around a while, but lost a lot of intrest when sqlalchemy came out | 20:04 |
| creiht | The only other real orm people push a lot is storm | 20:05 |
| anotherjesse | is "south" still the way to do migrations or was that a django thing? | 20:05 |
| alekibango | storm manual under construction is not looking good | 20:06 |
| alekibango | sqlalchemy is there and its real | 20:06 |
| eday | anotherjesse: have you though about making the data store a bit less shared yet? for example, not letting API poke into it? This is one of the things i've been thinking about lately | 20:06 |
| alekibango | just imagine having all data in replicated database available via sqlalchemy - that would ROCK :) | 20:07 |
| anotherjesse | eday: hmm, I was thinking the oposite - that the workers shouldn't have write priviledges | 20:07 |
| anotherjesse | eday: since then if a host node was compromised it limits the damage | 20:07 |
| eday | anotherjesse: is that really a concern for you? (host node being comp.) what about if API server was? or message queue? | 20:08 |
| eday | anotherjesse: in other words, do you see that as your weakest link? | 20:09 |
| *** mrayzenoss has quit IRC | 20:09 | |
| anotherjesse | eday: it isn't that the api server or queue is inherently secure, but we can improve each - for the queue I was talking with the chef guys and the idea of messages being signed by the sender | 20:10 |
| anotherjesse | eday: and the api server has the weakness of being exposed to the world - so definitely need to work on securing it | 20:11 |
| *** sirp1 has joined #openstack | 20:11 | |
| eday | anotherjesse: I was thinking in terms of data consistency and less dependencies. If host nodes were the source of data (and datastore was really just a backup for them), then scheduler/API nodes could just keep a subset of data reported from the host worker (whatever they need t answer/route requests). | 20:11 |
| anotherjesse | heh, that was how nova 0.1 worked | 20:11 |
| eday | anotherjesse: why did you move away from that model? | 20:11 |
| anotherjesse | the api server got updates from host nodes - and if the api server dies it takes 10-20 seconds to repopulate it | 20:11 |
| anotherjesse | the reason for moving away was data consistency | 20:12 |
| anotherjesse | specifically of the network information | 20:12 |
| anotherjesse | when you allocate an instance you need to know what IP is safe to give out, ... | 20:12 |
| eday | anotherjesse: the host nodes can cache and only ask for updates after a timestamp on start, don't need to dump all state from each worker | 20:12 |
| *** jamiew has quit IRC | 20:12 | |
| *** sirp1 has quit IRC | 20:12 | |
| *** jamiew has joined #openstack | 20:13 | |
| eday | anotherjesse: ahh, I see network address delegation almost as another app that could have workers and shared data store | 20:13 |
| *** sirp1 has joined #openstack | 20:14 | |
| anotherjesse | eday: we spent some time thinking about zookeeper (and other paxos systems) as a way of keeping that "configuration" - which would be separate from most of the instance state | 20:14 |
| anotherjesse | there were other questions like - who owns the fact that an instance & volume are associated | 20:15 |
| anotherjesse | eday: so - you are thinking a "central" sql store isn't a good idea | 20:16 |
| anotherjesse | for each "region" | 20:17 |
| *** matclayton has left #openstack | 20:17 | |
| eday | anotherjesse: ahh, I'm thinking about something simpler than paxos-like systems. just a simple pool of workers with a shared datastore for specific things like network and volume assignment. but host/guest info would be kept on host only (and subsets of that data bubbled up from there) | 20:17 |
| eday | anotherjesse: yeah.. trying to reduce dependencies on API/schedler layers | 20:18 |
| *** mshadle has left #openstack | 20:18 | |
| anotherjesse | eday: comprendo - vish has been working on the first of the api layer changes - which removes the "network controller" from the api | 20:18 |
| * mtaylor tosses out his approval of sqlalchemy as an orm ... and his agreement with devcamcar about postgres not being the right choice for a sql solution for this type of env | 20:18 | |
| eday | anotherjesse: I'm in the process of forming an email proposal, but here is a draft of the arch: http://oddments.org/tmp/architecture_new.png | 20:19 |
| anotherjesse | which means that network address allocation happens during "run instances" but all the other network instanciation can occur later during the host launching the instances | 20:19 |
| eday | anotherjesse: this would be a region | 20:19 |
| *** brd_from_italy has quit IRC | 20:20 | |
| *** brd_from_italy has joined #openstack | 20:20 | |
| anotherjesse | eday: when you say "guest table" in a worker - does that mean a sqlite table? | 20:20 |
| eday | anotherjesse: and a request like create guest would be a message that bounces from worker to worker until done (image verifucation, scheduler, IP assignment, then host) | 20:20 |
| eday | anotherjesse: I'm thinking just the list of VMs running with metadata, could be built into hypervisor, could be another data store | 20:21 |
| anotherjesse | so - the thing that made us move away from that is the requirement that IP assignment occurs during the HTTP request cycle for run instance | 20:22 |
| eday | anotherjesse: the idea is that the canonical data store for instances doesn't need to be accessed anywhere but from the host | 20:22 |
| anotherjesse | and needing to be certain that we don't re-allocate the ip | 20:22 |
| alekibango | eday: that last article of yours is really interesting | 20:22 |
| alekibango | http://oddments.org/ | 20:22 |
| devcamcar | mtaylor: thoughts on sqlalchemy vs sqlobject? | 20:23 |
| mtaylor | alchemy | 20:23 |
| mtaylor | object is essentially dead | 20:23 |
| anotherjesse | mtaylor: why no postgres? | 20:23 |
| eday | anotherjesse: if a HTTP request needs that, it could wait until the network worker assigns it. the openstack API will probably have a non-blocking version too that just gives a urn that can be used to lookup IP later too | 20:23 |
| *** avsm has quit IRC | 20:24 | |
| * jsmith wonders why no postgres as well | 20:24 | |
| *** justinsheehy has quit IRC | 20:24 | |
| alekibango | alchemy +100 | 20:24 |
| devcamcar | anotherjesse: i'm not anti-postgres, just have a lot more experience with mysql | 20:24 |
| mtaylor | anotherjesse: postgres isn't designed for lightweight scaleout architectures - it scales most effectively in the same way that oracle scales effectively - by throwing more spindles at a problem | 20:24 |
| devcamcar | i do like that its open source | 20:24 |
| eday | alekibango: yeah, I'd still like to test more frameworks too, not enough time :) | 20:24 |
| mtaylor | anotherjesse: postgres is a great database - it's just designed to handle a different set of problems | 20:24 |
| anotherjesse | mtaylor: I was under the (untested) impression that 9.x changed that | 20:24 |
| jsmith | mtaylor: In times past, that was true. With modern versions of PostgreSQL, it scales almost as well as MySQL | 20:25 |
| alekibango | devcamcar: for years postgres is superior... just mysql worked better on windows for years so it got popular | 20:25 |
| creiht | 9.x does change that | 20:25 |
| devcamcar | mtaylor: i just was looking at sqlobject earlier. it looks a bit simpler but if development is stalled then its a moot point | 20:25 |
| jsmith | mtaylor: Especially if you add foreign keys, transactions, etc. | 20:25 |
| * alekibango found this today: http://postgres-xc.sourceforge.net/ | 20:25 | |
| mtaylor | jsmith: well. you shouldn't do that if you want to scale :) | 20:25 |
| * creiht notes that the drizzle devs might be a little mysql biased :) | 20:25 | |
| mtaylor | not at all | 20:26 |
| jsmith | mtaylor: It all depends on your workload | 20:26 |
| *** jonesy_ has quit IRC | 20:26 | |
| anotherjesse | heh | 20:26 |
| mtaylor | I tel people to use postgres when it's appropriate all the time | 20:26 |
| * jsmith used to run one of the five (known) largest MySQL installations in the world | 20:26 | |
| mtaylor | as a mysql consultant, I used to tell clients to use postgres | 20:26 |
| devcamcar | mtaylor: seems that its down to between sqlalchemy and sqlobject. sqlalchemy is probably the best choice given the ecosystem around it | 20:26 |
| alekibango | even i use both, but i must say i like postgresql more | 20:26 |
| alekibango | mysql is just simple hack which is better than grep a bit | 20:26 |
| jsmith | I learned enough about MySQL to prefer PostgreSQL | 20:26 |
| eday | just fyi, I don't think we'll be hitting many scaling issues with the data store, especially if we don't share it between components :) | 20:26 |
| alekibango | jsmith: my thoughts :D | 20:27 |
| creiht | mtaylor: anyways... shouldn't it be using drizzle anyways? ;) | 20:27 |
| eday | we're not talking about that much data, even for large installations | 20:27 |
| mtaylor | creiht: well, that's sort of where I was going to go ... | 20:27 |
| jsmith | From a licensing standpoint, PostgreSQL makes things much easier as well | 20:27 |
| alekibango | eday: but we need reliable HA database on multiple servers right? | 20:27 |
| mtaylor | the thing here is that postgres has a decent amount of overhead in terms of spinning up a postgres on each node | 20:27 |
| alekibango | mysql might even die soon as a project, but some forks are on the way anyway | 20:27 |
| creiht | Why are we even talking about rational databases for scalable distributed systems anyways ;) | 20:28 |
| mtaylor | creiht: because nothing else actually works yet :) | 20:28 |
| * mtaylor stops being snarky | 20:28 | |
| *** jonesy_ has joined #openstack | 20:28 | |
| devcamcar | creiht: i doubt we'll be scaling to billions of instances any time soon :) | 20:28 |
| eday | alekibango: yeah, HA is still an issue for some parts, but even then we don't need strong consistency | 20:28 |
| mtaylor | devcamcar: yes - and the sqlalchemy dev folks are really good to work with - very responsive | 20:28 |
| alekibango | devcamcar: think BIG | 20:28 |
| eday | creiht: ++ :) | 20:28 |
| alekibango | if you can make sqlalchemy work with that replicated storage i never worked with - (forgot the name) - that would be very nice. something small and reliable | 20:29 |
| jsmith | If "eventually consistent" is OK, what about Cassandra? | 20:30 |
| alekibango | devcamcar: http://www.wikivs.com/wiki/MySQL_vs_PostgreSQL | 20:30 |
| * jsmith throws out a random idea | 20:30 | |
| anotherjesse | jsmith: it is ok for reads, but not for things like network allocation | 20:30 |
| mtaylor | cassandra devs will happily tell you cassandra is not a general purpose solution | 20:30 |
| devcamcar | alekibango: sweet, thanks | 20:30 |
| uvirtbot | New bug: #613156 in nova "Support GET /servers and GET /servers/detail in API" [Undecided,New] https://launchpad.net/bugs/613156 | 20:30 |
| uvirtbot | New bug: #613157 in nova "Support POST /servers in the API" [Undecided,New] https://launchpad.net/bugs/613157 | 20:30 |
| *** miclorb has joined #openstack | 20:31 | |
| alekibango | devcamcar: http://www.wikivs.com/wiki/MySQL_vs_PostgreSQL#Pro_PostgreSQL <- this part | 20:31 |
| alekibango | and yes its old info, but the score is similar during all last 10 years | 20:32 |
| devcamcar | nod | 20:32 |
| redbo | I'm a postgres fan, but it's been a huge headache for us. | 20:32 |
| redbo | (not that I mean mysql would be better) | 20:33 |
| mtaylor | redbo: what have the headaches been? | 20:33 |
| alekibango | yes - the new sqlite might be interestng too :D | 20:33 |
| mtaylor | and I wonder if I could solve them for you in drizzle? | 20:33 |
| *** mrayzenoss has joined #openstack | 20:33 | |
| *** mrayzenoss is now known as mray | 20:33 | |
| *** mray has joined #openstack | 20:33 | |
| anotherjesse | does sqlalchemy do a good job of abstracting the need for us deciding mysql vs. postgres? | 20:33 |
| mtaylor | anotherjesse: yes | 20:33 |
| alekibango | they have very nice feature in last version - write ahead logging | 20:34 |
| anotherjesse | mtaylor: then the only major question I have - is how do people manage schema in the sqlalchemy world? | 20:34 |
| alekibango | sqlalchemy supports mysql, postgresql, sqlite | 20:34 |
| mtaylor | anotherjesse: and I spoke with the sqlalchmey guys about adding redis support | 20:34 |
| alekibango | if taht would be solition, go ahead FTW | 20:34 |
| mtaylor | anotherjesse: it's been a couple of years since I've managed a full-on sqlalchemy install myself, so I don't know what the state of the art is there | 20:35 |
| alekibango | anotherjesse: i can help with that. | 20:36 |
| uvirtbot | New bug: #613158 in nova "Support <personality> when creating a server via API" [Undecided,New] https://launchpad.net/bugs/613158 | 20:36 |
| uvirtbot | New bug: #613159 in nova "Support GET /servers/<id> in API" [Undecided,New] https://launchpad.net/bugs/613159 | 20:36 |
| uvirtbot | New bug: #613162 in nova "Support PUT /servers/<id> in API" [Undecided,New] https://launchpad.net/bugs/613162 | 20:36 |
| uvirtbot | New bug: #613164 in nova "Support DELETE /servers/<id> in API" [Undecided,New] https://launchpad.net/bugs/613164 | 20:36 |
| mtaylor | I know they were working on a sqlalchemy-based migrations tool a couple of years ago - I can't imagine that's not come along nicely :) | 20:36 |
| anotherjesse | mtaylor: given that we are running a verison of this for users, we need to have a way to migrate as we deploy new versions | 20:36 |
| mtaylor | anotherjesse: totally | 20:36 |
| anotherjesse | which is where the KVS kinda falls down - just like we had to create our own ORM, we have to create our own migration system | 20:36 |
| * creiht has yet to see an orm migration tool that actually worked fairly well | 20:37 | |
| anotherjesse | too easy to just change things | 20:37 |
| comstud | soren around? | 20:37 |
| redbo | mtaylor: oh you know, per-server limits on write transactions, servicing real time requests on multi-TB databases, that sort of stuff. When a server goes down, it's a multi-hour maintenance to get a new replica online. | 20:37 |
| devcamcar | mtaylor: http://code.google.com/p/sqlalchemy-migrate/ | 20:37 |
| redbo | as we add shards, the maintenance doesn't scale | 20:37 |
| devcamcar | compatible with current 0.6.x | 20:37 |
| mtaylor | redbo: yeah - management of N db servers is certainly something high on our roadmap :) | 20:38 |
| redbo | well, I mean like the administration is O(n) for the number of servers | 20:38 |
| devcamcar | creiht: south migrations for django is actually pretty decent | 20:38 |
| devcamcar | but thats pretty specific | 20:38 |
| mtaylor | redbo: yeah | 20:38 |
| mtaylor | redbo: which is suck once you're talking about having N=1000 | 20:38 |
| alekibango | anotherjesse: schema migration in SA works... | 20:39 |
| redbo | our big db servers are really expensive, too | 20:39 |
| alekibango | but i choosed to not use it :D | 20:39 |
| mtaylor | redbo: well yeah - if management of them is O(n) - then it makes much more sense to have less, bigger shards | 20:40 |
| *** carolinad28 has quit IRC | 20:40 | |
| mtaylor | redbo: if they play nicely with each other and management is less easier, then having more smaller cheaper shards is easier to deal with | 20:40 |
| anotherjesse | alekibango: what would you choose then? | 20:40 |
| alekibango | anotherjesse: for what? | 20:40 |
| redbo | mtaylor: it'd also be nice if I had a cookie | 20:41 |
| anotherjesse | alekibango: for schema migrations | 20:41 |
| *** devcamcar has left #openstack | 20:41 | |
| alekibango | i am still reading above, this goes so fast :D | 20:41 |
| *** devcamcar has joined #openstack | 20:41 | |
| *** achew22 has joined #openstack | 20:41 | |
| uvirtbot | New bug: #613165 in nova "Support GET /servers/<id>/ips in API" [Undecided,New] https://launchpad.net/bugs/613165 | 20:41 |
| uvirtbot | New bug: #613168 in nova "Support PUT and DELETE /servers/<id>/ips/public/<address> in API" [Undecided,New] https://launchpad.net/bugs/613168 | 20:41 |
| uvirtbot | New bug: #613169 in nova "Support server reboot in API" [Undecided,New] https://launchpad.net/bugs/613169 | 20:41 |
| alekibango | sqlalchemy-migrate <- i tried this one, but decided not to use it | 20:41 |
| redbo | we used that, it was okay | 20:41 |
| anotherjesse | alekibango: what was the reason behind not using it? | 20:41 |
| alekibango | no need to hack more stuff that i need | 20:42 |
| alekibango | :D | 20:42 |
| alekibango | it works | 20:42 |
| redbo | Now I'm not saying you shouldn't use a database, just it didn't work for us for real time transactions with billions of records. It caused a lot of availability problems and was a major bottleneck on write speed. | 20:43 |
| alekibango | for me without migrating all db - i have tools to migrate parts i need and that makes me happy enough to kill the rest of the db i have when i migrate to new version | 20:43 |
| alekibango | imho i lost myself in the chat, what should be the database for? | 20:43 |
| mtaylor | redbo: well yeah. there's only one database out there I know of that can handle real time transactions with billions of records - but most people have problems administering it | 20:43 |
| anotherjesse | alekibango: we're getting frustrated with not having a real orm | 20:44 |
| redbo | mtaylor: what we have with swift is basically our giant tables sharded out to a bazillion sqlite dbs. :) | 20:44 |
| mtaylor | redbo: ah yes. :) | 20:44 |
| alekibango | for openstack metadata? | 20:44 |
| anotherjesse | alekibango: aye | 20:45 |
| *** miclorb has quit IRC | 20:45 | |
| alekibango | i would say sqlalchemy + some light backend, which will work reliably | 20:45 |
| redbo | with automatic replication and replica placement and it mostly manages itself | 20:45 |
| alekibango | sqlalchemy supports some different databases | 20:45 |
| mtaylor | redbo: at some point I'd love to make a patch to test using ndb ... just for shits and giggles... but I have _way_ too much on my plate to do that right now | 20:45 |
| *** jonesy_ has quit IRC | 20:45 | |
| anotherjesse | alekibango: I don't like that sqlite reuses primary keys | 20:45 |
| alekibango | i dont like sqlite also - concurrent access might be a problem | 20:46 |
| alekibango | and how to make it replicated? | 20:46 |
| anotherjesse | so - mysql/postgres via sqlalchemy for now | 20:46 |
| devcamcar | anotherjesse: i don't like sqlite because it doesn't support alter table | 20:46 |
| alekibango | how to make operations atomic? | 20:46 |
| redbo | We had a bsddb based database at one point. I was looking at HailDB last night. sqlite works extremely well. | 20:46 |
| uvirtbot | New bug: #613171 in nova "Support server rebuild in API" [Undecided,New] https://launchpad.net/bugs/613171 | 20:46 |
| alekibango | yes imho SA + mysql/postgres | 20:46 |
| uvirtbot | New bug: #613172 in nova "Support server resize (& confirm & revert) in API" [Undecided,New] https://launchpad.net/bugs/613172 | 20:46 |
| uvirtbot | New bug: #613175 in nova "Support Flavors operations in API" [Undecided,New] https://launchpad.net/bugs/613175 | 20:46 |
| mtaylor | SA + mysql/postgres ++ | 20:47 |
| creiht | sqlite can do a lot more than most people give it credit for :) | 20:47 |
| alekibango | anotherjesse: mysql supports some cluster based operations but its kinda hackety hack | 20:47 |
| alekibango | yes i use it right now :D | 20:47 |
| mtaylor | alekibango: depends on which backend you're talking about | 20:47 |
| alekibango | with SA... but i hate the fact that i cant (sqlite) modify columns | 20:47 |
| alekibango | i mean tables | 20:47 |
| alekibango | it lacks some really good features | 20:48 |
| alekibango | so sqlalchemy migration is not that easy with it | 20:48 |
| mtaylor | SA + mysql/postgres means that folks who know how to adminster postgres clusters can do that, and folks who know how to do the same with mysql or drizzle can do that too | 20:48 |
| *** justinsheehy has joined #openstack | 20:48 | |
| alekibango | mtaylor: right | 20:48 |
| anotherjesse | so, then we have eday's question - which is - is this the wrong direction | 20:49 |
| creiht | Is there a blueprint or wiki page talking more about the problem being solved? | 20:50 |
| alekibango | i found SA to make me much more productive - when i got used to it (which took few days) | 20:50 |
| mtaylor | could be. I think it depends on where we're imagining this database or set of databases to go ... are we talking about replacing the current redis that's there? or is this for something different? | 20:50 |
| alekibango | i still am not a guru on SA but i works pretty well for simple cases without much headache | 20:50 |
| uvirtbot | New bug: #613178 in nova "Support POST /images in API" [Undecided,New] https://launchpad.net/bugs/613178 | 20:50 |
| uvirtbot | New bug: #613180 in nova "Support GETting backup schedules in API" [Undecided,New] https://launchpad.net/bugs/613180 | 20:50 |
| uvirtbot | New bug: #613182 in nova "Support creation/deletion of backup schedules in API" [Undecided,New] https://launchpad.net/bugs/613182 | 20:50 |
| * alekibango needs to study redis | 20:50 | |
| alekibango | aha, so not SQL, but key value :D | 20:51 |
| alekibango | oh mistake :D | 20:51 |
| uvirtbot | New bug: #613177 in nova "Support Image GET operations in API" [Undecided,New] https://launchpad.net/bugs/613177 | 20:51 |
| uvirtbot | New bug: #613179 in nova "Support DELETE /images/<id> in API" [Undecided,New] https://launchpad.net/bugs/613179 | 20:51 |
| alekibango | redis is weaker compared to mysql/postgres... but if redis is light, fast and needs no administration, might be way to go... | 20:53 |
| *** devcamcar has quit IRC | 20:53 | |
| alekibango | http://pgsnake.blogspot.com/2010/04/postgres-91-release-theme.html <-- this is much fun. the guy almost scared me | 20:55 |
| uvirtbot | New bug: #613187 in nova "Support GETting shared ip groups in API" [Undecided,New] https://launchpad.net/bugs/613187 | 20:56 |
| uvirtbot | New bug: #613189 in nova "Support creating shared IP groups in API" [Undecided,New] https://launchpad.net/bugs/613189 | 20:56 |
| uvirtbot | New bug: #613190 in nova "Support deleting shared IP groups in API" [Undecided,New] https://launchpad.net/bugs/613190 | 20:56 |
| *** avsm has joined #openstack | 20:59 | |
| *** justinsheehy_ has joined #openstack | 21:00 | |
| *** justinsheehy has quit IRC | 21:00 | |
| *** justinsheehy_ is now known as justinsheehy | 21:00 | |
| *** devcamcar has joined #openstack | 21:01 | |
| mtaylor | devcamcar: you're in wa? | 21:01 |
| alekibango | maybe - idea - having ORM used with redis makes little sense as rdis is not R - it would be OKV (object key value)... am i right? | 21:01 |
| achew22 | I'm going through the instructions at http://wiki.openstack.org/NovaInstallFestInstructions and I got to the part where you start bin/nova-api and it is giving me the error gflags.UnrecognizedFlagError: Unknown command line flag 'fake_users' should I not be passing that in? | 21:02 |
| devcamcar | mtaylor: in seattle | 21:02 |
| mtaylor | devcamcar: me too | 21:02 |
| devcamcar | mtaylor: hah, nice! i'm about 8 blocks north of pike market | 21:02 |
| mtaylor | devcamcar: hah. I'm about 2 blocks south of pike :) | 21:03 |
| mtaylor | it is truly a small world :) | 21:03 |
| eday | before deciding on a data store, I think we should decide what we are storing in there exactly, and the operations around it. it may make sense if keep the same data model, but I'm not convinced that is the one we want long term | 21:03 |
| *** avsm has quit IRC | 21:03 | |
| devcamcar | mtaylor: i'm watching you right now, mwahah | 21:03 |
| alekibango | eday: good point | 21:03 |
| * mtaylor goes to put on pants | 21:03 | |
| devcamcar | the killer is in the house! | 21:03 |
| alekibango | this needs to be clarified | 21:03 |
| creiht | eday: yeah that is what I was trying to get at :) | 21:03 |
| eday | anotherjesse: What major concerns do you have around the arch I am drafting up? | 21:03 |
| *** brd_from_italy has quit IRC | 21:03 | |
| alekibango | maybe we should start some etherpad page about this? | 21:04 |
| alekibango | writing it all down in one place | 21:04 |
| eday | alekibango: I'm already drafting up text around the data flow, canonical sources, and all that. was going to post to mailing list later today | 21:05 |
| alekibango | http://etherpad.openstack.org/pieKQT8fRB <-- etherpad page would not hurt | 21:05 |
| alekibango | and i never used etherpad and this is my oportunity to have some fun with it | 21:06 |
| eday | gundlach: geez, lots of bug notifications in my email :) | 21:07 |
| *** justinsb has quit IRC | 21:08 | |
| *** devcamcar has quit IRC | 21:08 | |
| eday | gundlach: in other news, I have a eventlet server running with layered wsgi interfaces and patched amqp lib so everything is non-blocking | 21:08 |
| eday | gundlach: so, I think this would work quite well | 21:08 |
| achew22 | are there any openstack ppas? | 21:09 |
| creiht | eday: woot | 21:09 |
| eday | achew22: yes, looking... | 21:10 |
| eday | achew22: https://launchpad.net/~nova-core for nova stuff | 21:10 |
| eday | achew22: https://launchpad.net/~swift-core for swift | 21:11 |
| mtaylor | achew22: ppa:nova-core/ppa and ppa:swift-core/ppa if you're an add-apt-repository fan | 21:11 |
| alekibango | SQLALCHEMY supported DBS: http://www.sqlalchemy.org/docs/dbengine.html#supported-databases | 21:12 |
| achew22 | is nova-core/ppa acceptable? | 21:12 |
| mtaylor | achew22: what do you mean? | 21:13 |
| achew22 | is that new enough that it would run? | 21:13 |
| achew22 | it was built 2010-07-16 | 21:13 |
| mtaylor | oh - no, we need to upload new packages to that one | 21:14 |
| achew22 | hey! mtaylor someone told me to look for your PPA | 21:14 |
| mtaylor | achew22: all of the dev depends in there are good | 21:14 |
| *** justinsheehy has quit IRC | 21:15 | |
| *** devcamcar has joined #openstack | 21:15 | |
| mtaylor | achew22: yeah - we're working on getting those updated. ... the ubuntu python guys decided to do a full rebuild of every bloody python package across all of ubuntu - so it sort of stole most of the build slaves | 21:15 |
| achew22 | Ah, but some day soon it will get up to date? (i.e. that is the ppa to pull from because it will get updated eventually) | 21:16 |
| *** justinsheehy has joined #openstack | 21:21 | |
| vish1 | hi everyone, did i miss anyting? | 21:21 |
| eday | vish1: hey! you missed everything :) | 21:22 |
| vish1 | eday: I'm in the process of doing a very difficult refactor around networking | 21:22 |
| eday | vish1: yeah, anotherjesse was telling me about that | 21:23 |
| vish1 | but it sparked a question that i wanted to run by some people | 21:23 |
| gundlach | eday: just caught up with long irc logs to see your msgs to me. great to hear that you've got eventlet up -- post the code somewhere so we can try making the API in eventlet to see how it compares to twisted | 21:23 |
| vish1 | configuration for plugabble classes in python | 21:23 |
| vish1 | i'm using the following: | 21:23 |
| eday | gundlach: writing it up now in an email | 21:24 |
| vish1 | package.path.to.ClassName | 21:24 |
| vish1 | for example | 21:24 |
| *** _mattf has quit IRC | 21:24 | |
| mtaylor | achew22: yes | 21:24 |
| vish1 | --auth_driver=nova.auth.ldapdriver.LdapDriver | 21:24 |
| vish1 | the advantage of this is you can plug in any python class regardless of module | 21:25 |
| *** gustavomzw has left #openstack | 21:25 | |
| vish1 | the disadvantage is that it exposes a lot of inner workings to users | 21:25 |
| achew22 | mtaylor: thank you. I assume that after I apt-get install nova-* the next step would be nova-manage creating users and projects then using eucatools to set up nodes? | 21:25 |
| vish1 | who might want to use --auth_driver=ldap or something simple | 21:25 |
| gundlach | vish1: yagni? | 21:26 |
| *** justinsheehy has quit IRC | 21:26 | |
| vish1 | to which? | 21:26 |
| gundlach | sorry, the exposed classnames. | 21:26 |
| gundlach | i'd go with =ldap (less flexible but simpler to use) until at least 2 customers have complained about the lack of plugging in arbitrary classes :) | 21:27 |
| mtaylor | vish1: I agree with gundlach | 21:27 |
| vish1 | ok | 21:27 |
| gundlach | aka, don't make a user-level plugin framework until we need it. | 21:27 |
| vish1 | should i automagically generate classname? | 21:27 |
| vish1 | or just do ewans version | 21:27 |
| gundlach | no? | 21:27 |
| vish1 | of an if statement | 21:27 |
| alekibango | gundlach: LDAP simple to use? omg | 21:27 |
| mtaylor | vish1: also, somewhere in between is the setuptools entrypoints system, since we're on setuptools now | 21:27 |
| creiht | Isn't auth the one thing that you know for sure will need to be user pluggable? | 21:27 |
| vish1 | if 'ldap' driver==xxx | 21:28 |
| vish1 | i'm doing the same with network | 21:28 |
| gundlach | i'd do the if statement until it's clear that we will need to be dropping in more and more classes. | 21:28 |
| vish1 | cuz we have two network classes now | 21:28 |
| achew22 | mtaylor: how do you intend to create debian packages from setup.py? | 21:28 |
| gundlach | alekibango: no, i'm talking about the nova cmd line args, not LDAP itself | 21:28 |
| vish1 | instead of random FLAG checks in multiple places | 21:28 |
| mtaylor | achew22: we have a debian packaging branch | 21:28 |
| achew22 | is that a launchpad feature? | 21:29 |
| mtaylor | achew22: from which the packages are made | 21:29 |
| *** justinsheehy has joined #openstack | 21:29 | |
| vish1 | so is that the general consensus? short flags that get turned into classes internally? | 21:29 |
| mtaylor | achew22: well, the nightly builds into the ppa is | 21:29 |
| mtaylor | achew22: except it's all borked right now :) | 21:29 |
| _0x44 | vish1: That's my preference. | 21:29 |
| achew22 | can you link me to this branch? I've been trying to create .debs out of my python packages for a few weeks and have had NO luck | 21:29 |
| alekibango | i would think that with ORM, having users and everything in SQL database would make sense to me | 21:30 |
| vish1 | ok i'll do it in networking and convert the auth to short flags in a patch | 21:30 |
| mtaylor | achew22: ah. sure. it's lp:~nova-core/nova/debian-packaging | 21:30 |
| creiht | Here's an idea... why not make auth wsgi middleware, and poof, problem goes away :) | 21:31 |
| achew22 | mtaylor: thank you sir you might be the most helpful person I've run into all day | 21:31 |
| redbo | creiht: how are you going to specify the middleware? with short flags or full class path? | 21:31 |
| mtaylor | achew22: there's a set of bzr tools for managing the whole process... you may want to look at http://www.advogato.org/person/robertc/diary/130.html | 21:31 |
| creiht | well gholt seems to like the full class path :) | 21:32 |
| mtaylor | achew22: but feel free to ping me with questions | 21:32 |
| achew22 | mtaylor: thank you for the offer | 21:32 |
| creiht | If we really want configurable middleware, there are tools that do a lot of that work already | 21:32 |
| creiht | like paste | 21:32 |
| redbo | no, we're not having a requirement on paste :) | 21:32 |
| creiht | If you were to want that type of configurability | 21:32 |
| creiht | hehe | 21:32 |
| creiht | I'm just using it as an example | 21:32 |
| creiht | But might be a good place to get ideas | 21:33 |
| gholt | I thought redbo liked paste. | 21:33 |
| *** pvo has quit IRC | 21:33 | |
| gundlach | nite all | 21:33 |
| *** gundlach has quit IRC | 21:33 | |
| gholt | Took me a bit to depaste swift, back in the day. But I might be misremembering. I'm good at that. | 21:33 |
| redbo | you might be thinking of routes | 21:34 |
| gholt | Oh, yeah, maybe. For all the downsides, a bad memory does have upsides. Like.. Uhm. Well, you know. | 21:34 |
| *** londo__ has quit IRC | 21:36 | |
| achew22 | mtaylor: also, I don't know who to report this to but there isn't an upstart script for redis in the nova-core ppa | 21:36 |
| mtaylor | achew22: is there an upstart script in other redis packages? | 21:36 |
| achew22 | off hand I do not know | 21:37 |
| mtaylor | achew22: file it as a bug on nova and assign it to me (if it will let you assign it) ... I'm mordred in launchpad | 21:37 |
| *** RobertLJ has joined #openstack | 21:37 | |
| achew22 | mtaylor: I don't see a way to assign it so I'm going to make it generic | 21:42 |
| uvirtbot | New bug: #613210 in nova "Redis does not have an upstart script" [Undecided,New] https://launchpad.net/bugs/613210 | 21:51 |
| *** devcamcar has quit IRC | 21:51 | |
| *** RobertLJ has quit IRC | 21:52 | |
| jaypipes | anotherjesse, alekibango, mtaylor, eday: sorry, been away from keyboard...we should have the discussion on the mailing list I think... | 21:52 |
| * jaypipes kinda out of it until Thursday this week.. :( | 21:53 | |
| *** devcamcar has joined #openstack | 21:57 | |
| mtaylor | jaypipes: bah! I'm sick with strep throat and taking all sorts of antibiotics and I'm still in here :) | 21:58 |
| _0x44 | mtaylor: That's what you get for going on vacation. | 21:58 |
| mtaylor | _0x44: yes indeed... I'm just glad I didn't get swine flu, or bird flu or horse flu or whatever | 21:59 |
| _0x44 | mtaylor: You're lucky you didn't catch a case of Dread Candiru. | 21:59 |
| _0x44 | That always ruins a vacation. | 21:59 |
| mtaylor | sounds like it! | 22:00 |
| achew22 | which service is the one that runs on 8773? | 22:02 |
| *** tr3buchet has quit IRC | 22:02 | |
| *** londo has joined #openstack | 22:04 | |
| soren | mtaylor: Um... Is it ok if I reassign bug 613210 to ubuntu/redis? | 22:06 |
| *** devcamcar has quit IRC | 22:06 | |
| uvirtbot | Launchpad bug 613210 in nova "Redis does not have an upstart script" [Medium,Triaged] https://launchpad.net/bugs/613210 | 22:06 |
| *** cjwilson has quit IRC | 22:11 | |
| *** devcamcar has joined #openstack | 22:14 | |
| vish1 | achew22: api | 22:15 |
| *** devcamcar has quit IRC | 22:15 | |
| mtaylor | soren: sure | 22:16 |
| *** allsystemsarego has quit IRC | 22:16 | |
| alekibango | jaypipes: mail is dead | 22:19 |
| soren | mtaylor: Done. Phew. My sanity was at risk there for a minute. :) | 22:19 |
| alekibango | i dont like reading talking on mail lists - rather irc + wiki -> this is much better | 22:20 |
| comstud | soren, had you started any work on nova scheduler? someone said you might have | 22:20 |
| soren | comstud: Nope. | 22:20 |
| soren | comstud: I was kind of waiting for the architecture discussion to finish first. | 22:20 |
| comstud | sounds good.. just checking, since pvo has asked me to look at it | 22:20 |
| soren | comstud: It's way up high on my todo-list. As soon as my daughter feels better (here's hoping), I should get to it real soon. Like this week-ish. | 22:21 |
| alekibango | jaypipes: http://etherpad.openstack.org/pieKQT8fRB i tried to write some parts of our discussion here | 22:21 |
| vish1 | i think anotherjesse is looking at it as well | 22:21 |
| vish1 | at least a really dumb/simple version | 22:22 |
| comstud | ok, any branches yet? i've not seen any | 22:22 |
| *** devcamcar has joined #openstack | 22:22 | |
| comstud | for some reason https://code.launchpad.net/~nova is blank for me right now | 22:23 |
| alekibango | comstud: remove ~ sign | 22:24 |
| comstud | ah | 22:24 |
| comstud | duh | 22:24 |
| comstud | thanks | 22:24 |
| alekibango | np it happens all the time | 22:24 |
| achew22 | Is there anything equivalent to amazon's management console for openstack? | 22:26 |
| *** dendrobates is now known as dendro-afk | 22:26 | |
| *** justinsheehy has quit IRC | 22:27 | |
| *** abecc has quit IRC | 22:27 | |
| notmyname | achew22: since consoles generally tie in to id management and billing and ..., that is probably more left to the one implementing openstack | 22:28 |
| achew22 | It stands to reason that a generic console could exist with plugins for billing and management | 22:28 |
| notmyname | perhaps. but does a generic billing system even exist? ;-) | 22:29 |
| alekibango | achew22: i would like to have such console | 22:29 |
| achew22 | I hope not ;) | 22:29 |
| alekibango | achew22: if there is no other effort to create WEB based gui i am willing to help with making one (but cant do this alone) | 22:30 |
| achew22 | alekibango: if I were used to EC2 and all of it's components (lets be honest I have 0 hours logged on EC2 or competitors) I would say I could help but I probably am not the guy to talk to | 22:31 |
| creiht | I believe it is planned, and I believe that there is some proof of concept work for that done | 22:31 |
| creiht | I also saw a release a bit ago that some billing service said that they were planning on working with openstack | 22:31 |
| notmyname | that's cool | 22:31 |
| alekibango | creiht: there must be such thing... the future is here. its just not evenly distributed | 22:31 |
| mtaylor | that may be my favorite quote of the day... | 22:32 |
| mtaylor | "the future is here. its just not evenly distributed" | 22:32 |
| alekibango | its not mine, i just used it | 22:32 |
| alekibango | :D | 22:32 |
| creiht | mtaylor: credit - William Gibson | 22:32 |
| alekibango | thanks creiht | 22:33 |
| *** gustavomzw has joined #openstack | 22:33 | |
| mtaylor | mmm. I knew it was good | 22:33 |
| achew22 | Are there plans to implement something like Amazon's SQS? | 22:33 |
| *** justinsheehy has joined #openstack | 22:34 | |
| alekibango | achew22: what is sqs? | 22:34 |
| creiht | achew22: I would imagine that openstack would have more projects included in the future, but can't comment directly on sqs | 22:34 |
| achew22 | Simple Queue Service | 22:34 |
| alekibango | achew22: something like ampq? | 22:35 |
| creiht | I don't think there is a roadmap yet for other projects | 22:35 |
| achew22 | It is similar | 22:35 |
| creiht | alekibango: much simpler | 22:35 |
| creiht | alekibango: http://aws.amazon.com/sqs/ | 22:35 |
| alekibango | creiht: thanks, already reading it now | 22:35 |
| alekibango | amqp that is, i never write that one down correctly, hehe | 22:37 |
| *** devcamcar has quit IRC | 22:37 | |
| creiht | ->out | 22:39 |
| alekibango | achew22: anyway, come tomorow to ask about console again. maybe i will knew more, i ma mailing someone who might tell me something about it | 22:39 |
| alekibango | know* | 22:42 |
| notmyname | sounds like some sort of gangster thing: "Come back tomorrow, and I'll tell you more. I know a guy, see? And no cops!" | 22:43 |
| achew22 | I fear for my electronic life | 22:43 |
| alekibango | notmyname: i have heard here that someone is working on some gui for public api servers, thats all | 22:44 |
| alekibango | notmyname: but i have few bridges to sell if you have enought $$ :D | 22:44 |
| *** mray has quit IRC | 22:44 | |
| notmyname | "I can't tell you who he is, see? They'll kill me!" | 22:44 |
| *** devcamcar has joined #openstack | 22:44 | |
| alekibango | mail was given to me via pvtmsg | 22:45 |
| alekibango | yesterday or so, and now i sent him my question (similar to his) | 22:45 |
| alekibango | the guy is not on irc | 22:45 |
| alekibango | i dont know more except the email :D | 22:45 |
| alekibango | notmyname: i will msg you what he will reply to me | 22:47 |
| alekibango | if his mail will not end in trash | 22:47 |
| alekibango | :D | 22:47 |
| alekibango | i have 10000 spams a day | 22:47 |
| notmyname | I think the idea of a console is great, but I don't know how it would be useful for anyone outside of the developer's use case. | 22:47 |
| achew22 | In looking at the SQS page on Amazon I think i might be able to use it for free for a good deal of time | 22:48 |
| notmyname | what about some sort of cPanel extension or something like that | 22:48 |
| alekibango | who is that developer in your sentence? | 22:48 |
| * notmyname doesn't know anything about such things | 22:48 | |
| notmyname | alekibango: the developer that makes the generic console | 22:48 |
| alekibango | aha... smart idea.. generic console... do you know such consoles? | 22:49 |
| *** Glace has joined #openstack | 22:49 | |
| notmyname | no. that's my point | 22:49 |
| alekibango | they all sell it as part of the solution | 22:49 |
| alekibango | see cloud.com | 22:49 |
| alekibango | or amazon | 22:49 |
| Glace | Is Cloud.com based on openstack? | 22:50 |
| alekibango | nope, but they started to cooperate somehow | 22:50 |
| alekibango | they have own cloud solution, and they also have ENTERPRISE version (for spaceships) | 22:50 |
| alekibango | plus nice gui | 22:50 |
| gustavomzw | who runs irclogs.openstack.org? | 22:50 |
| achew22 | alekibango: is that targeted at shuttleworth? | 22:50 |
| notmyname | the console becomes a client of the system. it makes sense that it is individually developed as part of a product. | 22:51 |
| Glace | yeah.. but openstack.. is there a gui? | 22:51 |
| alekibango | Glace: ask tomorow | 22:51 |
| Glace | i didnt try openstack yet | 22:51 |
| alekibango | maybe there might be someone working on it | 22:51 |
| mtaylor | gustavomzw: me | 22:51 |
| mtaylor | gustavomzw: is it borked again? | 22:52 |
| alekibango | notmyname: i see. but there is none. i think one of partners should OPEN his own | 22:52 |
| Glace | heheh.. well if its nice.. i might start building one with my guys. but i didnt have time to read all the openstack doc | 22:52 |
| Glace | is it really better than cloud.com? | 22:52 |
| gustavomzw | mtailor: it returns 403 since the 29th | 22:52 |
| mtaylor | hrm | 22:52 |
| Glace | cloud.com have been there for a while I think.. their api is pretty solid | 22:52 |
| mtaylor | gustavomzw: thanks... will fix | 22:52 |
| Glace | but.. what seems very nice is the.. storage api of openstack :0 | 22:53 |
| Glace | no one does something similar i think.. right? | 22:53 |
| alekibango | Glace: there are few nice ways on openstack that are making me wanna go deeper with it | 22:53 |
| alekibango | 1) nice code | 22:53 |
| alekibango | 2) python | 22:53 |
| alekibango | 3) people here in channel | 22:53 |
| notmyname | Glace: glad you like it :-) | 22:53 |
| alekibango | 4) OPEN | 22:54 |
| Glace | hehe well it seems really nice.. cloud.com dont have astorage api.. and i was really looking for one | 22:54 |
| gustavomzw | alekibango: agreed 100% | 22:54 |
| Glace | yeah.. seems nice alek.. from what i read about openstack | 22:54 |
| alekibango | and its gaining momentum, so imho it might be ahead soon | 22:54 |
| Glace | a lot of people seem to be contributing to it | 22:55 |
| devcamcar | Glace: the nasa nebula dashboard will be open sourced very soon, which is a django based web front end | 22:55 |
| *** hazmat has quit IRC | 22:55 | |
| Glace | very nice devcamcar | 22:55 |
| alekibango | devcamcar: so it will become part of openstack? | 22:55 |
| Glace | I am developing right now a silverlight interface for cloud.com | 22:55 |
| alekibango | that would make me go on including glusterfs support in openstack :D | 22:55 |
| Glace | but I am really looking foward to use the openstack storage api | 22:56 |
| *** jonesy_ has joined #openstack | 22:56 | |
| Glace | hehe anyone here have tried glusterfs with infiniband? | 22:56 |
| alekibango | Glace: that live silver thingy is bad for your stomach | 22:56 |
| Glace | any good benchmark? | 22:56 |
| alekibango | Glace: as i have no infiniband, i didnt | 22:56 |
| alekibango | its not very fast - but it works and its easy to play with | 22:57 |
| Glace | oki.. yeah.. wanted to test glusterfs with inifniband and get some benchmarks | 22:57 |
| devcamcar | alekibango: that's the plan | 22:57 |
| alekibango | devcamcar: thanks for those news | 22:57 |
| alekibango | i hope it will be soon | 22:57 |
| *** justinsb has joined #openstack | 22:58 | |
| mtaylor | gustavomzw: done. thanks for pointing it out | 22:58 |
| * notmyname out | 22:59 | |
| gustavomzw | mtaylor: cool, it is back - night reading | 23:01 |
| *** londo has joined #openstack | 23:07 | |
| *** londo has quit IRC | 23:09 | |
| *** _mattf has joined #openstack | 23:09 | |
| *** rnewson has quit IRC | 23:09 | |
| *** miclorb_ has joined #openstack | 23:11 | |
| *** londo has joined #openstack | 23:14 | |
| *** devcamcar has quit IRC | 23:14 | |
| *** achew22 has quit IRC | 23:16 | |
| *** metoikos has quit IRC | 23:17 | |
| *** justinsheehy has quit IRC | 23:26 | |
| *** devcamcar has joined #openstack | 23:29 | |
| *** justinsheehy has joined #openstack | 23:32 | |
| *** devcamcar has quit IRC | 23:32 | |
| *** cjwilson has joined #openstack | 23:34 | |
| *** devcamcar has joined #openstack | 23:40 | |
| *** aliguori has quit IRC | 23:40 | |
| *** dmd17 has joined #openstack | 23:43 | |
| *** devcamcar has quit IRC | 23:43 | |
| *** strtok has quit IRC | 23:47 | |
| *** devcamcar has joined #openstack | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
