Saturday, 2010-08-07

*** miclorb has quit IRC00:05
*** miclorb has joined #openstack00:07
comstudhmmm00:12
comstudvish- i don't see that ewan's xenapi branch fixes bug 61409000:12
uvirtbotLaunchpad bug 614090 in nova "instances fail to launch due to nova.virt.images.fetch error" [Undecided,Fix committed] https://launchpad.net/bugs/61409000:12
comstudi see you approved it but it's not merged yet00:13
*** blpiatt has joined #openstack00:14
edayahh, it doesn't. local_image fix, not s300:15
edaymarked approved00:15
comstudthnx00:15
*** miclorb has quit IRC00:19
*** lbieber has joined #openstack00:22
*** miclorb_ has joined #openstack00:25
*** gundlach has quit IRC00:28
*** littleidea has quit IRC00:35
*** justinsheehy has quit IRC00:42
*** nettok has joined #openstack00:47
*** miclorb_ has quit IRC00:48
*** justinsheehy has joined #openstack00:48
*** jamiew has quit IRC00:54
*** lbieber has left #openstack00:57
*** pvo has joined #openstack01:08
*** ChanServ sets mode: +v pvo01:08
*** skeptomai has quit IRC01:31
*** blpiatt has quit IRC02:01
*** pvo has quit IRC02:20
*** chainey_ has joined #openstack02:25
*** chainey has quit IRC02:25
*** chainey_ is now known as chainey02:25
*** toddmorey has joined #openstack02:30
*** chainey has quit IRC02:31
*** carolinad28 has joined #openstack02:31
*** carolinad28 has quit IRC02:49
*** PythonPup has joined #openstack03:35
*** sophiap has quit IRC03:40
*** sophiap has joined #openstack03:41
*** benoitc has quit IRC03:42
*** pvo has joined #openstack03:44
*** pvo has joined #openstack03:44
*** ChanServ sets mode: +v pvo03:44
*** sophiap_ has joined #openstack04:04
*** sophiap has quit IRC04:08
*** sophiap_ is now known as sophiap04:08
*** skeptomai has joined #openstack04:30
*** toddmorey has quit IRC04:35
*** skeptomai has quit IRC04:39
*** sophiap_ has joined #openstack05:03
*** sophiap has quit IRC05:03
*** sophiap_ is now known as sophiap05:04
*** sophiap has quit IRC05:08
*** pvo has quit IRC05:50
*** hornbeck has quit IRC05:52
*** benoitc has joined #openstack06:21
*** ttx has quit IRC06:43
*** justinsheehy has quit IRC06:45
*** justinsheehy has joined #openstack06:51
*** allsystemsarego has joined #openstack07:21
*** nettok has quit IRC07:23
*** metoikos has joined #openstack08:21
*** ttx has joined #openstack08:45
*** ttx has joined #openstack08:45
*** rajijoom has joined #openstack09:09
*** metoikos has quit IRC10:31
*** BartVB has joined #openstack10:31
*** rnewson has joined #openstack10:33
*** rajijoom has quit IRC10:36
*** rajijoom has joined #openstack10:48
*** rajijoom has quit IRC10:52
*** rajijoom has joined #openstack10:52
*** jonesy_ has quit IRC11:24
*** jonesy_ has joined #openstack11:25
*** kuttan_ has joined #openstack11:42
*** ctennis has quit IRC12:04
*** ctennis has joined #openstack12:07
*** ctennis has joined #openstack12:07
*** silassewell has joined #openstack12:30
*** sophiap has joined #openstack13:06
*** toddmorey has joined #openstack13:10
*** toddmorey has quit IRC13:12
*** tjyang has quit IRC13:16
*** blpiatt has joined #openstack13:29
*** justinsheehy has quit IRC13:43
*** justinsheehy has joined #openstack13:50
*** rajijoom has quit IRC13:52
*** dendrobates is now known as dendro-afk14:10
*** dendro-afk is now known as dendrobates14:12
*** sophiap has quit IRC14:14
*** sophiap has joined #openstack14:27
*** gholt has quit IRC14:30
*** clayg has quit IRC14:30
*** clayg has joined #openstack14:31
*** gholt has joined #openstack14:32
*** sophiap has quit IRC14:38
*** sophiap has joined #openstack14:38
*** carolinad28 has joined #openstack14:45
*** sophiap has quit IRC14:46
*** pvo has joined #openstack14:47
*** pvo has joined #openstack14:47
*** ChanServ sets mode: +v pvo14:47
*** dendrobates is now known as dendro-afk14:49
*** carolinad28 has quit IRC15:12
*** pvo has quit IRC15:19
*** pvo has joined #openstack15:37
*** ChanServ sets mode: +v pvo15:37
*** dendro-afk is now known as dendrobates15:40
*** sophiap has joined #openstack15:47
*** pvo has quit IRC15:54
*** dendrobates is now known as dendro-afk16:08
*** chainey has joined #openstack16:19
*** chainey has quit IRC16:22
*** blpiatt has quit IRC16:31
*** nettok has joined #openstack16:32
*** hazmat has quit IRC16:46
*** codejunkie has quit IRC16:55
*** chmouel has quit IRC17:22
*** chmouel has joined #openstack17:24
*** rdw has quit IRC17:29
*** Jim_Shew has joined #openstack17:38
*** rdw has joined #openstack17:39
*** RobertLJ has joined #openstack17:48
*** binaryWarrior has joined #openstack17:49
*** chmouel has left #openstack17:49
*** chmouel has joined #openstack17:50
*** pvo has joined #openstack18:01
*** pvo has joined #openstack18:01
*** ChanServ sets mode: +v pvo18:01
*** binaryWarrior has quit IRC18:01
*** rnewson has quit IRC18:13
*** rnewson has joined #openstack18:14
*** rnewson has joined #openstack18:14
*** henriquetft has joined #openstack18:55
*** Jim_Shew has quit IRC18:58
*** codejunkie has joined #openstack19:02
*** termie has quit IRC19:06
*** termie has joined #openstack19:08
*** Jim_Shew has joined #openstack19:16
*** alexs_ has quit IRC19:25
*** alexs_ has joined #openstack19:25
*** kuttan_ has left #openstack19:27
*** pvo has quit IRC19:46
*** justinsb has joined #openstack19:59
justinsbeday: I have a question about your benchmarks... http://oddments.org/?p=49420:00
*** justinsheehy has quit IRC20:00
edayjustinsb: shoot20:01
justinsbeday: thanks!  So I'm trying to recreate your findings (as I want to benchmark native threads, including on "other Python runtimes").20:02
justinsbeday: Is this the right command for the flood server @ 512KB, 32k connections?20:02
justinsbeday: time LD_LIBRARY_PATH=/usr/local/lib scalestack echo::flood::tcp.port=12345 echo::flood::tcp.iterations=16 echo::flood::tcp.count=3276820:03
creihtjustinsb: redbo has an interesting graph of threads vs. eventlet20:03
justinsbcreiht: Interesting... do you have a link?20:04
creihtno, just prodding redbo to post it  :)20:04
justinsbcreiht: My motivation is to try to figure out where Threads break down20:04
creihtshort storry is that as concurrency went up throughput suffered quite a bit20:04
justinsbcreiht: I think that's the normal argument against threads, but I'd like to know where the limits are20:05
justinsbcreiht: And further, whether this is because of the GIL or threads themselves20:05
edayjustinsb: you need a dev branch I have to get it working properly.. trunk flood tool has a bug20:05
creihtMy guess is some combination, but it seemed to be along the same lines as david beazely's observations20:06
*** ctennis has quit IRC20:06
justinsbeday: Ah!  Let me check out that code and retry!20:06
edayjustinsb: but threads break down due to 2 reasons usually: kernel context switching overhead adds up and wasted memory for all the threads (even with minimal stack size settings)20:06
*** justinsheehy has joined #openstack20:06
justinsbeday: Have you seen the talk which preceeded David Beazely's?  http://blip.tv/file/223234920:07
edayjustinsb: nope20:07
justinsbeday: I stumbled across it - it basically says "just use threads"20:07
justinsbeday: I'd just like to get a better understanding with some hard benchmarks20:07
justinsbeday: So I thought I'd start with yours :-)20:08
creihtjustinsb: Sounds great in theory, but in practice it doesn't work :)20:08
justinsbcreiht: Why doesn't it work?20:08
justinsbcreiht: If it's because of the GIL (as David Beazely observed) that might well be fixable20:09
*** ctennis has joined #openstack20:09
creihtBut redbo was writing a simple utility to transfer some data, and did the first implementation with threads, and the performance wasn't great at all.  Monkeypatching with eventlet immediately gave a huge boost to throughput20:09
justinsbcreiht: Well, I want to work from experiments, not dogma, so hopefully redbo will post his code20:09
creihtUnfortunatley I can't remember the numbers, but hopefully when he gets on he can shed some more light20:10
creihtjustinsb: No dogma here, we want to do things the simplist most efficient way possible20:10
justinsbeday: Which is the dev branch I should get?20:10
edayjustinsb: so, it really all comes down to how state is stored for each connection, and how efficient that is (both in size, event detection, and switching states). OS threads can be a bit heavy for this (os thread stacks, kernel context switching, ...) where polling mechanisms and smaller data structures scale better (more context, not as much overhead)20:10
edayjustinsb: thats not saying an OS thread could be more efficient, just current implementations are still not20:11
justinsbeday: Absolutely agree with the idea that it should be possible to write a purpose built scheduler in userland and beat the kernel (generic) scheduler20:11
creihtjustinsb: I'm also unclear on how you plan on fixing the GIL20:12
justinsbeday: The question in my mind are: is it really worth the effort?  And can it be done in Python (or other non-multithreaded languages)?20:12
edayin fact, lightweight kernel-based co-routines are probably the best thing, they just don't exist :)20:12
creihtjustinsb: Python is multithreaded, the GIL just prevents them from being as useful as they could20:12
justinsbeday: Really, I want to find where the limits are20:12
justinsbcreiht: In terms of fixing the GIL, it's really just about fixing the problems that David Beazely observed.  http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorities-and-new.html20:13
creiht*just* :)20:14
creihtAnd I hope that is true20:14
creihtBut it hasn't been yet20:14
justinsbI mean 'just' as opposed to removing the GIL entirely20:14
creihtand that was 6 months ago20:14
*** dendro-afk is now known as dendrobates20:15
creihtAnd let me say this again, I would love simple *threading* to work for these use cases, but the fact is that they don't yet20:16
creihtAnd I'm not trying to disuade you from doing the testing20:17
creihtJust letting you know what our observations have been20:17
edayjustinsb: right now it's CPU bound assuming you have lots of memory to throw at it (~80k/thread)20:17
justinsbcreiht: Right!  Benchmarks can stop people like me bringing this up again and again :-)20:18
edayjustinsb: and the CPU is busy with switching between all the different threads, this is heavy because it needs to swap entire stacks and thread context in and out20:18
creihtjustinsb: another interesting test would also be to once you have your threaded version done, use the exact same code, and have eventlet patch threading, and see what type of difference there is :)20:19
justinsbcreiht: Awesome idea20:19
creihtredbo did the same and was quite surprised at how much difference it made20:19
justinsbeday: I know that threads are fairly heavy20:20
justinsbeday: They're also much easier to program with than (say) Twisted20:20
* creiht also notes that programming eventlet code is very close to how you would probram threaded code20:20
justinsbeday: So I just want to run the benchmarks to see when we can get away with using threads and when we have to get smart20:20
justinsbeday: Anyway, I branched from lp:superstack before running the benchmarks20:20
justinsbeday: Is that right?20:20
justinsbeday: Oops lp:scalestack20:21
edayjustinsb: lp:~eday/scalestack/style-updates  (rev 62)20:21
justinsbeday: Ah ... thanks!20:21
edayjustinsb: and then:  /usr/bin/time ./bin/scalestack p=scalestack/.libs echo_flood_tcp.address=localhost:12345 v echo::flood.iterations=16 echo_flood_tcp.count=3276820:22
justinsbeday: Thank you again :-)20:23
justinsbeday: Much better... it's actually taking some time now :-)20:26
creihthttp://github.com/opscode/openstack-cookbooks20:26
edayjustinsb: of course! keep me posted with what you find :)20:26
creihtprops to holoway20:28
*** Jim_Shew has quit IRC20:30
justinsbeday: The python servers work fine (twisted and evented), but with the scalestack server the flood tool is giving "Connection reset by peer: 104" errors20:36
justinsbeday: "ERROR [echo::flood::tcp] Received error on read: Connection reset by peer:104"20:37
*** hornbeck has joined #openstack21:01
*** dendrobates is now known as dendro-afk21:03
*** dendro-afk is now known as dendrobates21:04
*** RobertLJ has quit IRC21:07
edayjustinsb: make sure you specify max backlog of something high (i use 32k just to be safe)21:23
justinsbeday: Thanks - will do21:23
edayjustinsb: I run the server with: ./bin/scalestack p=scalestack/.libs echo_server_tcp.addresses=':12345' echo_server_tcp.backlog=3276821:24
redbopython threads on a multi-core machine are just horrible.21:27
justinsbredbo: Do you have the benchmark that creiht mentioned earlier?21:27
*** Daviey has quit IRC21:28
redboumm... I have a little stuff.  http://bit.ly/bp3Yzn21:28
redbohttp://bit.ly/cSJTxj21:29
justinsbredbo: Cool... do you have the source code for the benchmark you ran?21:29
*** hazmat has joined #openstack21:30
*** metoikos has joined #openstack21:30
redboNot put together yet.  I was thinking about doing a blog post or something.  The script was only a few lines, it created n threads that each spun urllib2.urlopen across a 10GbE connection.21:31
justinsbAwesome... well, let me know if you put together a blog post!21:31
justinsbWhat was the target?21:31
justinsbi.e. what was the webserver?21:32
redbonginx21:32
justinsbGot it.  So just a check of performance, similar to Eric Day's, but using HTTP rather than an echo test21:33
*** Daviey has joined #openstack21:33
redboyeah, mostly.  the thing was, I was doing something like this for work, and I did the first version with threads and was confused that I could only get 1gbps over that link.  I didn't realize threads were *that* bad.  But when I dropped in eventlet, it jumped up to 7gbps with no real modifiction.21:35
redboSo I kind of wanted a simplified version to show that effect.21:35
redboWith the GIL trickery, you wind up using more CPU to get less throughput as you add concurrent threads.  It's very pronounced on these 8-core machines.21:39
edaymust have some nasty write barriers to drop it down that much21:42
redbo(which is why eventlet peaks around 13% or whatever on that one graph, it's 100% of 1 core)21:42
*** Daviey has quit IRC21:44
*** jtimberman has left #openstack21:45
*** Daviey has joined #openstack21:47
edayjustinsb: http://oddments.org/?p=64  something I wrote a while ago to keep in mind if you find lots of shared data structures between threads (besides the GIL)21:49
*** Daviey has quit IRC21:50
*** Daviey has joined #openstack21:50
justinsbeday: Yeah, definitely something to watch out for.  But probably not as much of a problem for a webserver as it is for DB - much less heavily-contended shared state!21:52
edayjustinsb: yep. I'm thinking more about internal python data structures, not so much application code21:55
justinsbeday: Good point!21:55
*** allsystemsarego has quit IRC22:04
*** alekibango has quit IRC22:17
*** notmyname has quit IRC22:21
*** notmyname has joined #openstack22:23
*** ChanServ sets mode: +v notmyname22:23
*** alekibango has joined #openstack22:24
justinsbeday: Here are my results...22:26
*** Glace has quit IRC22:26
justinsbPython libevent: 1:0422:26
justinsbPython twisted: 1:1922:26
justinsbScaleStack (1 thread): 47s22:26
justinsbPython threaded: 1:0422:26
justinsbThis is at 32768 threads, 512KB22:27
justinsbSo this benchmark doesn't support using twisted or libevent22:27
justinsb(Unless I've screwed up!)22:28
*** Glace has joined #openstack22:30
*** alekibango has quit IRC22:30
*** alekibango has joined #openstack22:31
*** dendrobates is now known as dendro-afk22:48
justinsbOoops: s/Python libevent/Python eventlet/g22:48
*** pvo has joined #openstack22:59
*** ChanServ sets mode: +v pvo22:59
edayjustinsb: how many cores are you on?23:03
justinsbeday: 4.  AMD Phenom(tm) 9550 Quad-Core Processor23:04
justinsbeday: That could be why you saw a disadvantage from threads.  If one core is running the flood tool, you only have one thread left to run the server.  I have 3.23:04
justinsbeday: Which presumably offsets some of the overhead of threads23:05
*** blpiatt has joined #openstack23:08
edayjustinsb: possibly. this was on my laptop (intel core2duo 2.4)23:11
*** randybias has joined #openstack23:13
justinsbeday: I'm thinking that maybe redbo's benchmark will demonstrate some reason to use an eventing framework23:14
justinsbeday: Though realistically, I think we'll need something that mixes I/O and CPU23:14
edayjustinsb: yup, the echo server was mainly to test i/o and framework overhead23:15
justinsbeday: I used a bare-bones socket Python implementation for my echo server; it might be that's not fair?23:16
edayjustinsb: most of the time for all the python progs is going to be in syscalls, which will be in the lower levels (outside the GIL)23:16
edayjustinsb: as soon as you process more in the app (and need the GIL), boom23:16
justinsbeday: But isn't that true of twisted / eventlet also?23:16
edayjustinsb: yes, but those are only single threaded, so as you said you'll be bound by a single thread23:17
edayjustinsb: if you start up twisted/eventlet mulit-process all sharing the accept fd, you'll probably see similar better numbers than python threaded23:18
justinsbeday: Wouldn23:18
justinsbSorry23:18
justinsbeday: But you could do the same with threads23:18
edayjustinsb: but threads are already going over multiple cores23:19
edayjustinsb: twisted/eventlet dont23:19
justinsbeday: I think that's an orthogonal argument.23:21
justinsbeday: In Python, there's not true multi-threading for CPU bound tasks because of the GIL23:21
justinsbeday: Multi-threading in Python just makes the programming easier in terms of things that would otherwise block23:21
justinsbeday: Then, if you want to work around the GIL problem - in Twisted, Eventlet or Threading - you probably need multi-process23:22
justinsbeday: Or even Jython :-)23:22
justinsbeday: Just benchmarked Jython @ 1:34 on the same benchmark23:22
edayjustinsb: well, if we're talking about the GIL not being the limiting factor here yet (since we're spending more time in syscalls), I think it has some merit23:23
edayjustinsb: just try putting a couple os.fork() calls after the listen() but before the eventlet loop start23:23
edayjustinsb: and then you could do the same for the threaded version to see if it helps at all23:24
*** metoikos has quit IRC23:25
*** rdw has quit IRC23:28
*** pvo has quit IRC23:31
*** rdw has joined #openstack23:32
justinsbeday: With two os.fork (i.e. 4 processes).  Twisted goes from 1:19 -> 52s23:37
justinsbeday: Threaded 1:04 -> 1:0923:37
justinsbeday: Eventlet I couldn't get to work23:37
justinsbeday: In my opinion, this benchmark just says 'just stick with threads'23:38
justinsbeday: So to justify Twisted, I'll have to find another benchmark23:39
justinsbeday: I looked for Twisted's  'bragging benchmark', but I couldn't find it23:39
justinsbeday: Any idea what the canonical case-study benchmark for twisted actually is?23:39
edayjustinsb: well, twisted improved, but threaded didn't. don't you think that gives twisted a leg up?23:45
*** nettok has quit IRC23:46
edayjustinsb: what happened with eventlet? my eventlet forks fine23:46
*** randybias has quit IRC23:46
rdwthe main reason to use an eventing framework is to save on memory, not to reduce CPU consumption23:48
rdw(just by the way)23:48
*** hazmat has quit IRC23:48
redbordw: http://bit.ly/bp3Yzn http://bit.ly/cSJTxj  not having GIL contention makes a huge difference in CPU and throughput when you have a bunch of cores.23:50
rdwyeah, but we're "lucky" that python threads suck so much23:51
rdwin the future that may not be the case, though, if dabeaz does his thing  :)23:52
redbothat's true, I don't think you'd be able to see much difference using a similar model vs threads in straight C.  but it's also the python world as it exists.23:52
rdwtrue23:53
redboyou're crazy to use threads for concurrent networking in python right now.  In java or something, async vs threaded is a much closer argument.23:54
rdwin my opinion, though, focusing on CPU is focusing on the elephant's tail; the other benefits you get from eventing frameworks are also huge:  much lower memory consumption, and a better data access story23:54
*** henriquetft_ has joined #openstack23:55
*** henriquetft has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!