Tuesday, 2010-08-17

*** abecc has quit IRC00:00
*** ali``` has joined #openstack00:02
mtaylorcreiht, gholt: fixed00:05
*** abecc has joined #openstack00:06
*** ali```_ has joined #openstack00:08
*** silassewell has joined #openstack00:10
*** ali``` has quit IRC00:10
*** ali```_ is now known as ali```00:10
*** zooko has quit IRC00:11
*** pcrews has quit IRC00:12
*** jdmaturen has quit IRC00:15
*** tobym has joined #openstack00:15
*** ali```_ has joined #openstack00:16
*** maplebed has quit IRC00:17
*** ali``` has quit IRC00:19
*** ali```_ is now known as ali```00:19
*** adjohn has joined #openstack00:25
*** heckj has quit IRC00:38
*** e1mer has joined #openstack00:53
*** RobertLJ has joined #openstack01:01
*** zooko has joined #openstack01:01
*** pvo has joined #openstack01:06
*** ChanServ sets mode: +v pvo01:06
*** RobertLJ has quit IRC01:21
*** RobertLJ has joined #openstack01:23
*** lmcdowell has joined #openstack01:28
*** pvo has quit IRC01:32
*** binaryWarrior has joined #openstack01:35
*** binaryWarrior has joined #openstack01:36
*** RobertLJ has quit IRC01:36
*** RobertLJ has joined #openstack01:38
*** zooko has quit IRC01:41
*** jamiew has quit IRC01:49
creihtmtaylor: awesome, thanks!01:50
*** jdmaturen has joined #openstack01:51
*** e1mer has quit IRC01:56
gholtOkay cool, I'm going to try to get the new functional tests in now.01:58
*** gustavomzw has quit IRC02:09
*** RobertLJ has quit IRC02:10
*** binaryWarrior has quit IRC02:16
*** RobertLJ has joined #openstack02:25
gholtOkay, looks like it's all cleaned up now. Sorry for the lotsa-emails.02:35
*** e1mer has joined #openstack02:38
mtaylorgholt: thanks, btw - that problem helped fix up a part of how we were using tarmac02:38
gholtHeh, I guess I'm good at doing the unexpected.02:39
*** RobertLJ has quit IRC02:41
*** e1mer has quit IRC02:42
*** ali``` has quit IRC02:45
*** zooko has joined #openstack03:06
*** e1mer has joined #openstack03:17
*** pvo has joined #openstack03:21
*** ChanServ sets mode: +v pvo03:21
*** sirp1 has quit IRC03:38
*** sirp1 has joined #openstack03:49
*** kashyapc has joined #openstack03:51
*** littleidea has quit IRC03:59
*** littleidea has joined #openstack04:01
*** miclorb has quit IRC04:01
*** adjohn has quit IRC04:02
*** jeevan_ullas has quit IRC04:09
*** jeevan_ullas has joined #openstack04:09
*** miclorb has joined #openstack04:15
*** pvo has quit IRC04:20
*** silassewell has quit IRC04:22
*** abecc has quit IRC04:25
*** miclorb has quit IRC04:43
*** miclorb has joined #openstack05:12
*** adjohn has joined #openstack05:30
*** ibarrera has joined #openstack05:53
*** Podilarius has joined #openstack05:57
*** guigui has joined #openstack06:02
*** zooko has quit IRC06:05
*** jdmaturen has quit IRC06:09
*** ChrisAM has quit IRC06:09
*** timrc_ has joined #openstack06:09
*** chromakode has quit IRC06:10
*** comstud has quit IRC06:10
*** stewart_ has joined #openstack06:11
*** jpipes has joined #openstack06:11
*** allsystemsarego has joined #openstack06:12
*** brainproxy has quit IRC06:13
*** stewart has quit IRC06:13
*** jaypipes has quit IRC06:13
*** ChrisAM` has joined #openstack06:13
*** ChrisAM` has joined #openstack06:13
*** ChrisAM` is now known as ChrisAM06:13
*** chromakode has joined #openstack06:13
*** littleidea has quit IRC06:14
*** littleidea has joined #openstack06:15
*** brainproxy has joined #openstack06:29
*** littleidea has quit IRC06:34
*** adjohn has quit IRC06:50
*** adjohn has joined #openstack06:54
*** calavera has joined #openstack07:04
*** sirp1 has quit IRC07:05
*** almaisan-away is now known as al-maisan07:12
*** rnewson has joined #openstack07:15
*** rnewson has joined #openstack07:15
*** justinsheehy has quit IRC07:21
*** adjohn has quit IRC07:22
*** justinsheehy has joined #openstack07:27
*** brd_from_italy has joined #openstack07:28
*** brd_from_italy has quit IRC07:35
*** miclorb has quit IRC08:11
*** zheng_li has joined #openstack08:19
*** rnewson has quit IRC08:20
*** justinsheehy has quit IRC08:23
*** rnewson has joined #openstack08:25
*** rnewson has joined #openstack08:25
*** justinsheehy has joined #openstack08:30
*** fsoyer has joined #openstack08:33
*** fsoyer has left #openstack08:35
*** rajijoom has joined #openstack08:36
*** fsoyer has joined #openstack08:41
*** ptremblett has quit IRC08:55
*** miclorb has joined #openstack09:54
*** miclorb has quit IRC09:56
*** miclorb_ has joined #openstack09:58
*** ptremblett has joined #openstack10:01
*** Guest96517 has joined #openstack10:11
*** rsampaio has joined #openstack10:19
*** gustavomzw has joined #openstack10:42
*** justinsheehy has quit IRC10:54
*** justinsheehy has joined #openstack11:00
fsoyerHi all. Sorry il it's a stupid question, but after reading all I found about OpenStack (nice and so interesting project !) I don't understand what is the last layer of the architecture : which virtualization software can be used ??11:05
redbofsoyer: I believe it's anything supported by libvirt, which means most hypervisors  http://libvirt.org/11:27
*** Guest96517 has quit IRC11:37
*** ctennis has quit IRC11:38
*** JPP has joined #openstack11:38
*** JPP is now known as Guest2262311:38
*** miclorb_ has quit IRC11:42
*** kashyapc has quit IRC11:42
*** justinsheehy has quit IRC11:46
*** jpopi has joined #openstack11:48
fsoyer@redbo : thanks. So KVM and Xen, maybe some others ? Is VMware definitely excluded ? (technically or/and phylosophically ?)11:55
larissafsoyer: Error: "redbo" is not a valid command.11:55
*** sandywalsh has joined #openstack11:56
*** ctennis has joined #openstack11:58
*** kashyapc has joined #openstack11:59
alekibangofsoyer:  nothing is excluded. feel free to add support, like some people did with virtualbox12:00
*** sophiap has quit IRC12:00
alekibangofor now it supports xen, kvm (qemu) and virtualbox12:01
*** sophiap has joined #openstack12:04
*** sophiap has quit IRC12:14
fsoyeralekibango : OK I see. I'm more a technician and (beta) tester, not a developer, so I hope that some geek will find an interest in vmware :) But maybe it's time to re-test kvm or xen. 3 years ago I examine them but was not convinced by the functionalities.12:17
*** abecc has joined #openstack12:21
*** jpopi has quit IRC12:26
*** jpopi has joined #openstack12:26
sorentermie: Is this still on your radar? https://code.edge.launchpad.net/~justin-fathomdb/nova/check-subprocess-exit-code/+merge/3070712:32
*** justinsheehy has joined #openstack12:32
sorenfsoyer: It also supports user-mode-linux.12:32
fsoyersoren : OK but my needs include Windows VMs... I see on the libvirt page that VMWare ESX is supported ??? Well, I didn't think it was possible, in my mind libvirt was only attach to linux... And forgot that ESX *is* now linux based !12:39
sorenfsoyer: Mind you, using ESX with Nova isn't just a click of a button just because libvirt supports it.12:42
*** DuSTIX has joined #openstack12:44
sorenfsoyer: libvirt is an API for managing (running, stopping, starting, etc) virtual machines (and a bunch of things wrt. to storage, but that's separate to this). *Creating* VM's is a different matter.12:45
sorenfsoyer: Different hypervisors require different formats for their VM's and require different information to get them running. Right now, nova's libvirt support handles QEmu style hypervisors and user-mode linux. Nova also has a Xen API backend that talks to Xen server.12:46
sorenfsoyer: Anything other than that will require work.12:46
sorenfsoyer: Probably not much work, but still work.12:46
sorenfsoyer: It's not just a configuration issue.12:46
*** pcrews has joined #openstack12:47
fsoyersoren : I understand , it's why I think that maybe, yes, if I find some time I'll try to play with vmware, after trying OpenStack/Xen or KVM, and if no heavy development is necessary...12:49
fsoyerBut let start with a "default" Nova/Switch architecture !12:51
*** zheng_li has quit IRC12:55
*** gdusbabek has left #openstack12:56
*** jdarcy has joined #openstack12:57
*** zheng_li has joined #openstack12:59
*** aliguori has joined #openstack13:01
*** holoway has quit IRC13:01
*** pvo has joined #openstack13:02
*** ChanServ sets mode: +v pvo13:02
*** sophiap has joined #openstack13:02
*** holoway has joined #openstack13:03
*** holoway has quit IRC13:06
*** holoway has joined #openstack13:08
*** pvo has quit IRC13:11
*** zooko has joined #openstack13:12
*** dendro-afk is now known as dendrobates13:25
termiesoren: nope, i'll look at it again13:26
*** pvo has joined #openstack13:26
*** ChanServ sets mode: +v pvo13:26
*** DuSTIX has quit IRC13:28
*** fsoyer has quit IRC13:35
*** rajijoom has quit IRC13:46
*** kashyapc has quit IRC13:47
*** zooko has quit IRC13:55
*** kashyapc has joined #openstack13:55
*** mrayzenoss has joined #openstack13:56
*** mrayzenoss is now known as mray13:56
*** mray has joined #openstack13:56
gholtmtaylor: How should I test debian packaging? The swift package is failing due to a new .png. I was going to fix the problem by doing what it suggests (adding it to include-binaries) but I'd like to be able to test things out before pushing.13:57
*** gundlach has joined #openstack13:57
gundlachAnyone know how to tell Hudson that there's a new package requirement?  He can't merge lp:~gundlach/nova/rsapi because he doesn't have 'routes' installed, but it's in tools/pip-requires.13:59
*** pcrews is now known as pcrews_errand14:02
*** dendrobates is now known as dendro-afk14:13
*** dendro-afk is now known as dendrobates14:16
*** holoway has quit IRC14:20
*** holoway has joined #openstack14:21
*** johnpur has joined #openstack14:24
*** ChanServ sets mode: +v johnpur14:24
sorengundlach: You tell mtaylor, and magic happens.14:29
gundlachsoren: is that the way we want it, or should i put in a bug to get that fixed?14:29
gundlachsoren: also: thanks :)14:29
sorengundlach: I'm reasonably sure that's how we want it.14:33
*** JimCurry has joined #openstack14:33
sorenHm... I think.14:34
*** Glace has quit IRC14:37
*** sirp1 has joined #openstack14:40
*** Glace has joined #openstack14:42
mtaylorgundlach: adding now - will figure out a better way to do that moving forward14:43
gundlachtyvm!  i emailed you the full list (I think there are 3)14:43
*** holoway has quit IRC14:44
*** holoway has joined #openstack14:45
*** mray has quit IRC14:47
*** holoway has quit IRC14:49
zulis there a way to increase the logging level of nova-compute i have the sneaky suscpion that something is wrong14:49
*** holoway has joined #openstack14:50
*** littleidea has joined #openstack14:54
sorenzul: What do you have now?15:01
zulim just using --nodaemon and --verbose15:01
zulbut the image isnt loading or anything15:02
zulthis is with the latest bzr btw15:02
*** zooko has joined #openstack15:03
*** mrayzenoss has joined #openstack15:05
*** mrayzenoss is now known as mray15:05
*** mray has joined #openstack15:05
*** guigui has quit IRC15:07
soren--nodaemon --verbose should give you everything.15:07
*** tobym has quit IRC15:08
*** JimCurry has left #openstack15:10
mtaylorgholt: pull the lp:swift/debian - make the change, add an entry to the changelog with dch --nmu (you'll need the nmu since you're not listed in the uploaders) , then run bzr bd -S15:12
*** JimCurry has joined #openstack15:17
*** ibarrera has quit IRC15:23
*** tobym has joined #openstack15:26
*** pcrews_errand is now known as pcrews15:29
gholtmtaylor: Okay, I'll work through that today.15:29
*** calavera has quit IRC15:30
*** fsoyer has joined #openstack15:31
*** tr3buchet has joined #openstack15:31
mtaylorgholt: we can also add you to the uploaders list in debian/control if you're going to be hacking on it and stuff15:33
*** holoway has quit IRC15:37
*** holoway has joined #openstack15:38
*** hazmat has quit IRC15:40
gholtmtaylor: Well, I'm very newb when it comes to debian packaging, but yeah I'll try to keep on it.15:41
* mtaylor needs to get nova and swift submitted to debian ...15:41
gholtAlso, the bzr bd -S failed on dpkg-buildpackage -rfakeroot -d -us -uc -S with little info about why, but I can run that manually just fine.15:41
*** holoway has quit IRC15:42
*** holoway has joined #openstack15:43
*** JimCurry has quit IRC15:49
*** jpopi has left #openstack15:52
gholtmtaylor: What do you think of: http://bazaar.launchpad.net/~gholt/swift/debiancyberduckhowtopng/revision/5415:54
*** holoway has quit IRC15:56
mtaylorgholt: two things - now that you added yourself to debian/control, you don't need the --nmu for dch ... and you can go ahead and make the version number 1.0.2-3 ... we use the nmu revs for the automatic hudson merges15:56
mtaylorgholt: also, if you could change the distro series from lucid to unstable - the packaging trunk is targeted at debian and then there's an ubuntu branch we keep for targetting ubuntu15:57
mtaylorgholt: other than that - it looks great!15:57
gholtAh, heh. Didn't even notice the lucid, okay.15:58
*** holoway has joined #openstack15:58
gundlachanyone know what would cause ReactorAlreadyRunning errors in twisted?  I must have touched something related to twisted in my branch, because I'm getting these errors in the unittests.16:05
*** RobertLJ has joined #openstack16:09
mtaylorgholt: yeah - dch on ubuntu is ... 'helpful'16:10
*** metoikos has quit IRC16:10
edaygundlach: eventlet can use twisted under the hood, so if you try to run them both you may get issues16:11
*** metoikos has joined #openstack16:13
gundlacheday: thanks, i just figured that out right when you said that :)  the only place we import eventlet is nova.wsgi, but that must get pulled in somewhere else that is called by the unittests.16:13
*** avsm has joined #openstack16:14
gundlachyep, nova.endpoint.__init__16:16
creihtgundlach: eventlet will detect if twisted is imported, and if so by default will try to use the twisted hub, if that is the problem, you can tell it which hub to use to override it16:17
gundlachcreiht: i think the problem is that eventlet starts up the twisted reactor? because then when the unittest tries to start the reactor manually it bombs16:18
creihtthat part, I'm not entirely sure, as I have never actually used that part16:19
creihtgundlach: What I'm wondering is if you add something like this right after the imports in the test file, if it fixes the issue: http://paste.openstack.org/show/14/16:20
gundlachcreiht: will try it.  though do we want to be hacking the test to avoid this problem?  i'm making an existing test (many AWS EC2 API tests) fail, so i don't know that i should just work around it in the test file.16:22
*** littleidea has quit IRC16:22
gundlachi think this is showing me that as the code stands now, eventlet is getting in the way of twisted's normal operation, or something16:22
gundlach(didn't seem to make a difference anyway :) )16:23
gundlachi'll keep digging16:23
creihtgundlach: That depends if the error is being introduced because of how twisted and eventlet are being used in the tests, or how they are being used in the actual code16:24
gundlachtx for the suggestions16:24
*** zaitcev has joined #openstack16:25
*** jdmaturen has joined #openstack16:26
*** jdmaturen has joined #openstack16:27
*** littleidea has joined #openstack16:28
gundlachcreiht: what happens when in a nova venv you do this:16:36
gundlach>>> import twisted.internet.reactor16:37
gundlach>>> import eventlet16:37
termiegundlach: same thing that would happen if you had them both installed16:37
creihtgundlach: I have no idea, I don't have nova set up :)16:37
*** RobertLJ has quit IRC16:37
gundlachcreiht: heh, ok16:37
gundlachtermie: ImportError: cannot import name getcurrent16:37
gundlachwhat am i missing?16:38
gundlach[from eventlet/hubs/twistdr.py's line "from eventlet import getcurrent, greenlet"16:38
termiegundlach, i am guessing that eventlet is trying to override the twisted reactor16:38
termiei don't get that error however16:39
gundlachtermie: glad to hear that the error seems weird.  guess i'll go looking in eventlet's source.16:39
*** aliguori has quit IRC16:39
termiegundlach: it may be a 0.9.10 error for eventlet16:39
termiemy eventlet is 0.9.9 since i haven't updated the venv since bfore eventlet was added i guess16:40
gundlachtermie: is 0.9.10 supposed to be flaky?  (It's a "stability release" according to their blog)16:40
creihtgundlach: The eventlet/twisted integration was abandoned by the original author, and hasn't really been maintained16:40
termieone sec16:40
creihtso it may very well cause weird issues16:40
creihtgundlach: .9.10 is stable16:41
termieupdating mine and checking16:41
gundlachcreiht: good to know, thanks.  maybe the right thing to do is explicitly keep eventlet from using twisted, in the actual code not just the tests16:41
gundlachtermie: ty16:41
creihtgundlach: yeah that may be a good idea, I'm install twisted on my laptop so I can see what happens16:41
termieyup, newer version gets that error16:42
creihtgundlach: yeah I reproduced it as well16:43
gundlachtermie: but not 0.9.9?16:43
gundlachthere was a "fix for twisted hub" that went into 0.9.916:43
termienot that i can tell16:43
gundlachthanks both for your help!16:43
gundlachi'll see if i can work around it, and worst case we'll use 0.9.9 until we've teased apart twisted and eventlet more16:43
termiegetcurrent should be in the greenthread library16:44
termieor rather, module16:44
gundlachtermie: looks like it's a circular import problem -- getcurrent isn't defined in eventlet/__init__.py before the import loop cycles16:45
termieit doesn't happen in the opposite order, for the record16:45
termieand e comes before t16:46
*** pvo has quit IRC16:46
termiecoincidence? I THINK NOT16:46
creihtgundlach: In my python shell if I import eventlet.hubs, and then do eventlet.hubs.use_hub('poll'), import eventlet works for me16:46
gundlachcreiht: i'll try it again, i must have put it in the wrong place. thx.16:46
gundlachtermie: deep16:47
termiegundlach: it is good sometimes to explore the rich meaning in the worl daround us16:47
*** RobertLJ has joined #openstack16:47
*** stewart_ has quit IRC16:53
*** thobbs has joined #openstack16:56
gundlachcreiht: i think your "import eventlet.hubs" thing works because that implicitly imports eventlet, so you're importing eventlet before you import twisted.internet.reactor.16:58
*** RobertLJ has quit IRC16:59
*** holoway has quit IRC17:01
*** holoway has joined #openstack17:03
*** zooko has quit IRC17:05
*** holoway has quit IRC17:06
*** stewart_ has joined #openstack17:06
*** holoway has joined #openstack17:08
*** maplebed has joined #openstack17:09
*** jamiew has joined #openstack17:10
*** zheng_li has quit IRC17:15
sorenmtaylor: Were you working on an eventlet package for Ubuntu?17:17
sorenmtaylor: Or did I just dream that?17:17
*** pvo has joined #openstack17:19
*** ChanServ sets mode: +v pvo17:19
mtaylorsoren: I believe you just dreamt that - although I could certainly do if needed17:21
sorenmtaylor: That would be magnificent.17:21
sorenmtaylor: We're already past feature freeze in Ubuntu, so getting new stuff like this in is getting harder by the day.17:22
sorenmtaylor: ...and I'm working on other packaging bits (as you may have noticed if you're following the ubuntu-packaging branch at all)17:22
mtayloryes. I noticed17:23
sorenAnyways, /me calls it a day17:23
mtaylorsoren: I was going to suggest we make a slight change to how we're doing packaging branches - but I'll send you an email about it17:23
sorenWho am I kidding? I'll go eat some dinner and be back in a few hours.17:23
mtaylorsoren: heh17:23
sorenmtaylor: We can chat in a few hours if you prefer.17:23
* soren heads dinnerwards17:23
*** zheng_li has joined #openstack17:34
rdwgundlach: ah, I think perhaps the problem is that twistedr is importing getcurrent from eventlet, and it should be importing from eventlet.greenthread17:38
gundlachrdw: hm, ok.  in any case, i filed a bug at eventlet.com, and have moved our code around so eventlet and twisted don't have to touch -- so the tests pass now and my branch merged (finally :D )17:39
*** tobym has quit IRC17:39
rdwso,wait, you're not even using the twisted stuff?17:39
rdwI didn't realize it got initialized unless you explicitly enabled it17:40
creihtgundlach: In my tests above, I had imported twisted reactor before importing anything from eventlet17:40
creihtrdw: If it detects twisted as imported, it will select it by default17:40
rdwok yeah so two things then I guess17:41
gundlachcreiht: really!  weird; if i tried importing twisted.internet.reactor and then "from eventlet import hubs" it barfed17:42
gundlach(in 0.9.10 of course)17:42
rdwoh yep, it sure does17:42
creihtgundlach: interesting as I did a import eventlet.hubs :)17:42
rdwthat is some fucked up code17:42
gundlachcreiht: shouldn't make a difference, should it?  they're equivalent except for what name ends up in yoru namespace17:43
creihtI dunno17:43
creihtJust noting the difference17:43
creihtWorked for me due to another sideffect :/17:44
creihtIt was in the same session that it had errored on a previously on import eventlet17:45
creihtif you try to import again, it doesn't give you the error17:45
rdwok well I'm disabling the automatic twisted-hub usage17:46
creihtrdw: :)17:46
gundlachrdw: how?17:46
creihtgundlach: rdw is the eventlet maintainer17:46
gundlachoh, hey, you're ryan williams17:46
* gundlach asks for autograph17:46
* rdw writes "Ryan Williams" illegibly17:47
* gundlach rebuttons his shirt17:47
gundlacherm, thanks!17:47
edaygundlach: what are you hacking on now? don't want to overlap with your work if I start hacking API stuff too17:47
rdwthanks for finding this weirdness!17:47
gundlacheday: images.py.  just merged rsapi into trunk17:47
gundlachrdw: no prob17:47
gundlachand thanks for fixing it :)17:48
edaygundlach: ok, cool17:50
gundlacheday: lemme know if you have any more specific questions17:50
*** kashyapc has quit IRC17:51
edaygundlach: will do. I've been getting a real nova+kvm setup going on a machine so I can test things end-to-end... I think it's all there now :)17:51
gundlacheday: cool, put your instructions on the wiki somewhere so i can follow them :)17:51
*** comstud has joined #openstack17:52
edaygundlach: hah, well there are 4 sets of instructions on the wiki already... I'm thinking we need to consolidate (I didn't follow any of them)17:52
gundlacheday: yesyesyes17:52
gundlacheday: or remove all links to them, put yours up as the official one, and at the bottom in a footnote link to the old instructions17:53
gundlachuntil a few people have followed yours and been successful17:53
edayI need to polish up the method I'm using too17:54
*** stewart_ is now known as stewart18:00
*** mray has quit IRC18:03
edayzul: I am just running firectly out of bzr source so I can hack/restart easily. Just got libvirtd installed and setup network bridges, and am using boto to control it18:07
edayerr, directly18:08
*** tobym has joined #openstack18:11
*** holoway has quit IRC18:15
*** holoway has joined #openstack18:17
*** avsm has quit IRC18:19
*** brd_from_italy has joined #openstack18:20
*** sirp1 has quit IRC18:21
*** aliguori has joined #openstack18:22
*** sirp1 has joined #openstack18:23
*** sirp1 has quit IRC18:26
*** tobym has quit IRC18:30
*** tobym has joined #openstack18:31
*** al-maisan is now known as almaisan-away18:39
comstudi followed the 'install from source' to run directly from a bzr checkout18:45
comstud(install from source wiki page)18:46
sorenzul: You expressed an interest in the user-mode-linux stuff. Are you still running lucid anywhere?18:46
zulsmoser: nope18:46
zuldoh...i mean soren18:47
sorenzul: alright, never mind.18:47
zaitcevHas anyone succeeded in running a smaller Swift cluster than the 4x that SAIO provides? Say, if I have an old laptop and I want to run an instance to test clients... Is it doable?18:50
notmynamezaitcev: you need at least 3 nodes (even if they are all on th same box, as in the SAIO) to get swift to run18:51
zaitcevnotmyname: Thanks.18:51
*** rajijoom has joined #openstack18:51
notmynamezaitcev: the 4th node in the SAIO allows for handoffs in failure scenarios18:51
*** thobbs has left #openstack18:53
jerois there documentation about how high availability is handled in OpenStack ?18:57
*** ptremblett has quit IRC18:58
notmynamejero: compute or storage?18:58
jeronotmyname: for all the critical components18:59
jeroi'll take any link you have :-)18:59
notmynameI can only speak for swift (the storage piece). http://swift.openstack.org/overview_architecture.html is a good place to start for knowing how swift works19:00
notmynamehttp://swift.openstack.org/ is the swift doc site19:00
creihtjero: high available is quite a broad term, is there anything specifically you were wondering or about, or just generally?19:00
jerocreiht: what I'm wondering, is if openstack is desingned from the bottom up to be highly available19:01
creihtnext question :)19:02
jeroand horizontally scalable ?19:02
creihtat least I can speak for swift, but I'm pretty sure those are the goals of nova as well19:02
notmynamejero: on the storage side, we have often said that a storage system is easy. what we have made is a storage system that handles failures19:03
creihtjero: A little more specifically, swift stores 3 copys of every object, and ensures on a write at least 2 copies have been written to return a success19:03
jerodoes swift also store the persistent disk volumes ? or just system images/stuff like that19:03
notmynamehowever, understand that error handling in the code is only part of the full picture. the other part is how your deployment is built out and how good your ops guys are :-)19:04
creihtjero: swift is "just" an object store, similar in scope to S319:04
*** rsampaio has quit IRC19:04
jerothen how are the persistant disk volumes managed ?19:05
* creiht passes off to the nova guys19:05
jerook I found a documentation pointer19:07
creihtnova.openstack.org and wiki.openstack.org are good starting places19:07
jeroanyone here on the choice of AoE for storage ?19:08
sorenmtaylor: So... packaging.19:09
mtaylorsoren: yes.19:10
mtaylorsoren: so - I set up something for the swift guys that I wound up liking - namely that hudson merges trunk into the packaging branch on every trunk push so that there is always a branch that one can pull and build packages from19:11
mtaylorsoren: I was thinking that if we did that for nova as a sort of trunk-packaging branch, and then had the debian and ubuntu branches be forks of that - we can roll up all of the extra changelog entries when it's time to release to deb or ubuntu19:12
mtaylorbut I wanted to run it by you first before just doing it19:13
redboage of empires for storage?19:16
_0x44redbo: ATA over Ethernet?19:16
_0x44redbo: But I like the way you're thinking.19:17
_0x44With four nodes, we could probably get swift to play a pretty good game of Risk. ;)19:17
jerothis should be implemented as an easter egg.19:18
creihtIt would be easy with wsgi middleware :)19:18
redboit's like farmville, but with attack mode19:18
*** rajijoom has quit IRC19:19
jeroso AoE is the current stuff, iSCSI is on the way,19:21
creihtjero: From what I hear that seems correct19:21
creihtmtaylor: can you take a look at: http://etherpad.openstack.org/FPJmxV34yP19:21
creihtand see if that seems reasonable19:21
mtaylorcreiht: looking19:22
sorenmtaylor: Sure, that sounds like fun.19:22
mtaylorsoren: cool. I'll put it on my todo list19:22
edayvishy: you around? how are things with the sql work going?19:23
mtaylorcreiht: the ~/.bazaar/bazaar.conf part can be replaced just with the bzr whoami command19:23
sorentermie: You've been on top of the gflags business before. Would you mind taking a peek at https://code.edge.launchpad.net/~soren/nova/derootification/+merge/32888? I need it to move forward on a fresh upload to Ubuntu.19:24
zaitcevswift/README says "To build documentation run `python setup.py build_sphinx`", but this ends in "error: invalid command 'build_sphinx'". Any ideas?19:26
sorenzaitcev: Do you have sphinx installed?19:26
creihtzaitcev: Yeah you have to have sphinx installed19:26
*** rdw has quit IRC19:28
zaitcevAww, how silly of me. But of course.19:30
creihtzaitcev: and btw, docs are also available online at swift.openstack.org19:30
*** Steve0 has joined #openstack19:37
*** Robi_ has joined #openstack19:39
*** theron has joined #openstack19:40
* Robi_ waves at theron 19:40
theronme waves from the brownbag podcast19:41
Robi_nod, good questions19:41
theronHi Robi_19:41
*** sirp1 has joined #openstack19:41
theronalready lovin' swift,  the idea of having nodes strung across multiple providers with openvpn tying nodes together = awesome.19:42
Robi_nice.. any migrations possible yet?19:42
Robi_as in workload19:45
Robi_vmotion type19:46
creihtWhat brownbag podcast is this?19:47
theronRobi_ still evolving, I'm pining the VMs to hosts... scaling out, really just leveraging vmware for templates, quick deployments, the idea of migrations or failover isn't required obviously.19:47
creihttheron: and thanks about the comment with swift19:47
theroncreiht: someone needs to say "welcome to the community, just make sure to RTFM"19:47
creihttheron: A small note about the openvpn thing, while that will probably work, I'm not sure that would give the desired performance for large data sets19:48
theroncreiht: certainly not.19:48
theroncreight: thinking more along the lines of "private cloud storage" files email blah blah in public cloud.19:49
gholtCurrently, swift has no concept of 'closest node'. But that'd be a cool feature if someone ever gets to it. :)19:49
therongholt as long as I can tweak dns, closest node can be done externally, but yea, that would make life easier :)19:50
Robi_heh, i had to do some dns hacking recently migrating a live app19:51
*** sirp1 has quit IRC19:51
Robi_hosts file manipulation ftw19:51
creihttheron: You might be interested in http://www.nasuni.com/19:51
creihtThey can't put files accross different providers yet, but it sounded like something they were interested in19:52
Robi_seems you get a choice of your provider though19:52
gholttheron: Well, I mean on the backend. All the backend process talk to each other with IPs (DNS would lag). For instance, only the proxy servers need to be publicly accessible. When a proxy is asked for an object it can retrieve that from any one of the backend replica nodes. If it knew which was closet to it....19:52
*** ptremblett has joined #openstack19:53
Robi_hmm this sounds a lot like the distributed storage model LaCie has been working on for some time with Wuala19:53
theroncreiht nasuni looks interesting.19:53
Robi_it's a dropbox like free service, but has a very neat distribution model19:54
Robi_i remember talks at google years ago from the wuala team19:54
Robi_invite code if wanted: http://is.gd/em5gk19:55
creihtI saw a presentation from Nasuni just a couple weeks ago and was impressed19:55
theroncreiht a swift backed node where management is retained would be more palitable for certain customers, at least making an easy bridge to a public service down the road. :)19:56
uvirtbotNew bug: #607235 in swift "Add cloudfiles developer docs to the swift docs" [Undecided,New] https://launchpad.net/bugs/60723519:56
therongholt ah. understood.19:56
creihttheron: indeed19:57
gholtuvirtbot: That's only a new bug to you. :P19:57
uvirtbotgholt: Error: "That's" is not a valid command.19:57
*** sirp1 has joined #openstack20:01
*** Xenith has joined #openstack20:03
*** theron_ has joined #openstack20:03
*** theron has quit IRC20:05
eday_cerberus_: hey! hows API work going? Have you had much time for it?20:07
*** ppetraki has joined #openstack20:07
*** theron__ has joined #openstack20:08
_cerberus_eday: slowly :-) I just picked it up today for the first time in a few days.20:09
*** theron__ has quit IRC20:09
*** theron_ has quit IRC20:11
eday_cerberus_: ahh20:14
*** ThisDB has quit IRC20:14
*** gustavomzw has quit IRC20:15
_cerberus_eday: I plan to look at testing in a little bit. Looking through all the stuff that gundlach merged in over the last couple days first to see if I can get a clear picture in my head20:15
gundlach_cerberus_: feel free to ask me any q's you got20:15
_cerberus_gundlach: I never hesitate :-)20:15
*** gustavomzw has joined #openstack20:16
*** littleidea has quit IRC20:26
*** rdw has joined #openstack20:28
*** gustavomzw has quit IRC20:40
*** e1mer has quit IRC20:40
*** brd_from_italy has quit IRC20:44
*** jbd has joined #openstack20:44
*** ppetraki has quit IRC20:46
*** sophiap has quit IRC20:47
*** sophiap has joined #openstack20:48
*** ctennis has quit IRC20:51
*** ctennis has joined #openstack21:03
*** ctennis has joined #openstack21:03
gundlachout, bye21:06
*** gundlach has quit IRC21:07
termiesoren: done21:09
*** e1mer has joined #openstack21:09
termiesoren: just style fixes, otherwise looks great21:09
cwcreiht: poke21:18
*** allsystemsarego has quit IRC21:19
*** jdarcy has quit IRC21:20
gholtHe's out for a little while, probably on again in a few hours.21:29
sorentermie: Lovely, thanks. Will fix it up tomorrow. thanks for the review.21:37
* soren heads bedwards21:37
termiesoren: gnight21:38
*** aliguori has quit IRC21:48
*** justinsheehy has quit IRC21:48
*** zaitcev has quit IRC21:50
*** ppetraki has joined #openstack21:50
*** avsm has joined #openstack21:57
creihtcw: howdy22:04
creihtmtaylor: https://launchpad.net/~swift-core/+archive/ppa22:11
creihtswift ppa fail22:11
mtaylorcreiht: oh wow. yeah. that's pretty mcuh fail22:12
*** aliguori has joined #openstack22:15
*** zaitcev has joined #openstack22:20
cwcreiht: wrt to hashing a path to get a ring location ... surely i can game that?22:26
cwie. generate paths to always git the same spot in the ring and flood that with large objects22:26
creihtcw: you can surely try :)22:26
cwsince there is no salt in the process22:26
creihtThere is indeed a salt22:26
*** sophiap has quit IRC22:26
cwok good22:26
cwi must have missed that22:27
cwwhere is that?22:27
* creiht looks22:27
creihtAt least kind of a salt22:27
cwsalt makes things tasty22:27
creihtA configurable string of chars is added to the path22:27
cwthat suffices22:27
creihtWe worked through it abit, and even without the salt it would be very difficult, be we added it just to make everyone happy :)22:28
cww/o it my math said it would be trivial ... 100ms or so effort for a 24-bit ring22:28
cwmd5 on modern cpus is fast ... and everyone has lotsa cores now22:28
creihtcw: plus we figured that we could catch onto it pretty fast, and stop the user22:29
cwupload speed will limit you22:29
creihtand we would notice a node being off balance before it became an issue22:29
cwcreiht: so you just use stochastic spreading to avoid filling up partitions?22:29
creihtcw: swift/common/utils.py hash_path()22:30
creihtnot the HASH_PATH_SUFFIX22:30
cwok, awesome22:31
cwso when adding new physical space ... you rebalance the ring ... this means most things end up with 2 copies?22:31
cwand you let it replicate out?22:31
creihtcw: So we recommend that you configure the number of partitions so that there are at least 100 partitions per disk (estimating the max number of disks that you would use), which gives on average a 1% variance22:32
cwi saw that .... 100 per disk seemed to stem from some old rsync nasties22:32
cwi wondered if it might be relaxed22:32
redboit's to keep things balanced with a larger sample22:33
creihtIt can be relaxed if you don't mind more variance22:33
creiht(and less even distribution)22:33
cwthe rign sized is fixed presently surely?22:33
cwso as you add more hw the # partitions per spindle will decrease22:34
creihtWhen adding new physical space, the ring rebalances moving at most 1 replica of a partition, and everything then replicates out22:34
creihtcw: correct22:34
cw(btw, partitions is an over-used and much confusing mataphore)22:34
cwright, so you have to plan for some target22:34
creihtOnce you set the partition count, you can't change it22:34
cwand if you're at 10% of that to start with ... you have 1000 / spindle22:34
* cw waves hands with vague numbers22:34
cwdo you have any idea on the access patters for the data?22:35
cwthis is tangential22:35
cwie. what objects are accessed hot-cache?22:35
cwlatency would tell you22:35
cwthis goes back to earlier talks about limited RAM and slab vs page-cache value22:35
creihtcw: In our use case, I don't think we get many cache hits, as that type of traffic is pushed to the CDN22:35
cwis that true for metadata too?22:36
creihtI don't know that for sure, but my guess would be that it is true as well22:36
cwagain, latency would tell you22:37
redboRight now we don't use the page cache for file data, because it's not good for our use case.  I expect other people will wanna change that eventually.22:37
cwstat from cache is low usecs (faster maybe) ... hitting the disk is 5ms+22:37
*** justinsheehy has joined #openstack22:37
cwredbo: you use regular IO22:37
cwso it does use the page-cache22:37
cwwhich tends to push the slab down22:37
redbookay, we use the page cache, then throw the page cache away22:37
cwhow do you throw it away?22:38
cwthere is no syscall to purge it unless you unlink the inode22:38
redbosure there is22:38
cwthat's advise :-)22:38
cwi'm not 100% sure that works that well ... at least it never did in the past22:38
cwcat /proc/meminfo ... and see how much is in the page-cache22:38
redboif you advise it that you don't need the page cache, it throws it away22:38
redboit'll be full of inodes, I hope22:39
redboand directories22:39
cwok, you win :-)22:40
* cw owes more beer22:40
redboI'm thinking we might remove that fadvise call, because it's bad for other peoples use cases.  And use the vfs cache pressure settings on our install so it'll tend to cache inodes but not file data on its own.22:40
cwthere is code that will invalidate on POSIX_FADV_DONTNEED22:40
*** jc_smith has joined #openstack22:40
creihtor at least make it configurable22:41
cwvfs cache pressure from page cache means your inodes will get cached very poorly22:41
cwi did tweaks in the past to try avoid this22:41
cwon other stuff i use DIO22:41
cwand avoid the page cache completely22:41
redboI think it depends on which way you tune it?22:42
cwdio or page-cache vs inodes?22:42
cwthe latter is does a tiny bit22:42
cwbut not as much as you think ... there is a vfs pressure knob from 100 to 0 ... but it doesn't work very well22:43
redbo"Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches.  Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes."22:43
cwit's not linear22:43
cwnon-zero still means page-cache puts a lot of pressure on slab22:43
cwit's basically a count of loops into the reclaim code in some cases22:43
redboI'd still like to test it.22:44
redbobut yeah, maybe leaving it as an option is better22:44
redbothere will probably be other people who don't have repeated reads to a single file very often22:45
redboand that might be changing for us soon anyway, if we do public access containers22:45
cwredbo: ok, so i am showing my age ... i checked, the 'fadvise problem' was fixed april 2004 :)22:46
redboI know it makes re-reading files slower, so I assume it's dumping the cache.22:46
edaycomstud: any word on your scheduler branch? I know jesse was hacking on it, but probably good to get that merged sooner than later :)22:48
cwredbo: it invalidates the pages yes22:48
*** rnewson has quit IRC22:49
comstudeday, had to work on these couple of tickets22:49
cwi have code that uses mincore to measure how much of a file is cached, i'll try it out later and verify it does as it claims22:49
comstudjesse got caught up with other things, so I'm doing some additional work22:49
comstudi'll try to propose a merge tomorrow22:49
*** justinsheehy has quit IRC22:49
*** johnpur has quit IRC22:50
*** tobym has quit IRC22:55
*** justinsheehy has joined #openstack22:56
cwredbo: removing the fadvise would be ugly actually22:56
cwredbo: w/o that things will initially be _really_ fast22:57
cwonce your working set gets larger though ... you'll be like 10x to 100x slower22:57
cwand people will pelt you with angry kittens22:57
*** avsm has quit IRC23:05
*** sophiap has joined #openstack23:06
redboI expect our working set to be a tiny percentage of our total storage.  But if we have people serving images directly from cloudfiles, it could get important.23:07
*** jamiew is now known as jamiew_moviez23:07
*** pcrews has left #openstack23:08
redbowhy can't we just have 128gb of ram on the storage machines like I wanted?23:09
*** ppetraki has quit IRC23:09
cw16gb dimms are like unicorm hair23:09
cwright now23:09
redboThey also didn't like my networking idea.  "Everything needs to talk to everything else really fast."23:09
*** pvo has left #openstack23:10
_0x44redbo: TCP over New Jersey nannies isn't as fast as you'd think.23:10
cwhigh bi-section bandwidth is getting affordable now23:10
*** anm_ has quit IRC23:11
*** justinsb has joined #openstack23:12
*** rdw has joined #openstack23:13
*** miclorb_ has joined #openstack23:13
cwredbo: it seems you should be able to increase the ring size actually23:14
cwdouble it on demand23:14
*** aliguori has quit IRC23:16
*** sophiap has quit IRC23:30
*** sandywalsh has quit IRC23:33
*** sophiap has joined #openstack23:39
*** jero- has joined #openstack23:45
*** tr3buchet has quit IRC23:52
*** silassewell has joined #openstack23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!