*** miclorb has quit IRC | 00:05 | |
*** miclorb has joined #openstack | 00:07 | |
comstud | hmmm | 00:12 |
---|---|---|
comstud | vish- i don't see that ewan's xenapi branch fixes bug 614090 | 00:12 |
uvirtbot | Launchpad bug 614090 in nova "instances fail to launch due to nova.virt.images.fetch error" [Undecided,Fix committed] https://launchpad.net/bugs/614090 | 00:12 |
comstud | i see you approved it but it's not merged yet | 00:13 |
*** blpiatt has joined #openstack | 00:14 | |
eday | ahh, it doesn't. local_image fix, not s3 | 00:15 |
eday | marked approved | 00:15 |
comstud | thnx | 00:15 |
*** miclorb has quit IRC | 00:19 | |
*** lbieber has joined #openstack | 00:22 | |
*** miclorb_ has joined #openstack | 00:25 | |
*** gundlach has quit IRC | 00:28 | |
*** littleidea has quit IRC | 00:35 | |
*** justinsheehy has quit IRC | 00:42 | |
*** nettok has joined #openstack | 00:47 | |
*** miclorb_ has quit IRC | 00:48 | |
*** justinsheehy has joined #openstack | 00:48 | |
*** jamiew has quit IRC | 00:54 | |
*** lbieber has left #openstack | 00:57 | |
*** pvo has joined #openstack | 01:08 | |
*** ChanServ sets mode: +v pvo | 01:08 | |
*** skeptomai has quit IRC | 01:31 | |
*** blpiatt has quit IRC | 02:01 | |
*** pvo has quit IRC | 02:20 | |
*** chainey_ has joined #openstack | 02:25 | |
*** chainey has quit IRC | 02:25 | |
*** chainey_ is now known as chainey | 02:25 | |
*** toddmorey has joined #openstack | 02:30 | |
*** chainey has quit IRC | 02:31 | |
*** carolinad28 has joined #openstack | 02:31 | |
*** carolinad28 has quit IRC | 02:49 | |
*** PythonPup has joined #openstack | 03:35 | |
*** sophiap has quit IRC | 03:40 | |
*** sophiap has joined #openstack | 03:41 | |
*** benoitc has quit IRC | 03:42 | |
*** pvo has joined #openstack | 03:44 | |
*** pvo has joined #openstack | 03:44 | |
*** ChanServ sets mode: +v pvo | 03:44 | |
*** sophiap_ has joined #openstack | 04:04 | |
*** sophiap has quit IRC | 04:08 | |
*** sophiap_ is now known as sophiap | 04:08 | |
*** skeptomai has joined #openstack | 04:30 | |
*** toddmorey has quit IRC | 04:35 | |
*** skeptomai has quit IRC | 04:39 | |
*** sophiap_ has joined #openstack | 05:03 | |
*** sophiap has quit IRC | 05:03 | |
*** sophiap_ is now known as sophiap | 05:04 | |
*** sophiap has quit IRC | 05:08 | |
*** pvo has quit IRC | 05:50 | |
*** hornbeck has quit IRC | 05:52 | |
*** benoitc has joined #openstack | 06:21 | |
*** ttx has quit IRC | 06:43 | |
*** justinsheehy has quit IRC | 06:45 | |
*** justinsheehy has joined #openstack | 06:51 | |
*** allsystemsarego has joined #openstack | 07:21 | |
*** nettok has quit IRC | 07:23 | |
*** metoikos has joined #openstack | 08:21 | |
*** ttx has joined #openstack | 08:45 | |
*** ttx has joined #openstack | 08:45 | |
*** rajijoom has joined #openstack | 09:09 | |
*** metoikos has quit IRC | 10:31 | |
*** BartVB has joined #openstack | 10:31 | |
*** rnewson has joined #openstack | 10:33 | |
*** rajijoom has quit IRC | 10:36 | |
*** rajijoom has joined #openstack | 10:48 | |
*** rajijoom has quit IRC | 10:52 | |
*** rajijoom has joined #openstack | 10:52 | |
*** jonesy_ has quit IRC | 11:24 | |
*** jonesy_ has joined #openstack | 11:25 | |
*** kuttan_ has joined #openstack | 11:42 | |
*** ctennis has quit IRC | 12:04 | |
*** ctennis has joined #openstack | 12:07 | |
*** ctennis has joined #openstack | 12:07 | |
*** silassewell has joined #openstack | 12:30 | |
*** sophiap has joined #openstack | 13:06 | |
*** toddmorey has joined #openstack | 13:10 | |
*** toddmorey has quit IRC | 13:12 | |
*** tjyang has quit IRC | 13:16 | |
*** blpiatt has joined #openstack | 13:29 | |
*** justinsheehy has quit IRC | 13:43 | |
*** justinsheehy has joined #openstack | 13:50 | |
*** rajijoom has quit IRC | 13:52 | |
*** dendrobates is now known as dendro-afk | 14:10 | |
*** dendro-afk is now known as dendrobates | 14:12 | |
*** sophiap has quit IRC | 14:14 | |
*** sophiap has joined #openstack | 14:27 | |
*** gholt has quit IRC | 14:30 | |
*** clayg has quit IRC | 14:30 | |
*** clayg has joined #openstack | 14:31 | |
*** gholt has joined #openstack | 14:32 | |
*** sophiap has quit IRC | 14:38 | |
*** sophiap has joined #openstack | 14:38 | |
*** carolinad28 has joined #openstack | 14:45 | |
*** sophiap has quit IRC | 14:46 | |
*** pvo has joined #openstack | 14:47 | |
*** pvo has joined #openstack | 14:47 | |
*** ChanServ sets mode: +v pvo | 14:47 | |
*** dendrobates is now known as dendro-afk | 14:49 | |
*** carolinad28 has quit IRC | 15:12 | |
*** pvo has quit IRC | 15:19 | |
*** pvo has joined #openstack | 15:37 | |
*** ChanServ sets mode: +v pvo | 15:37 | |
*** dendro-afk is now known as dendrobates | 15:40 | |
*** sophiap has joined #openstack | 15:47 | |
*** pvo has quit IRC | 15:54 | |
*** dendrobates is now known as dendro-afk | 16:08 | |
*** chainey has joined #openstack | 16:19 | |
*** chainey has quit IRC | 16:22 | |
*** blpiatt has quit IRC | 16:31 | |
*** nettok has joined #openstack | 16:32 | |
*** hazmat has quit IRC | 16:46 | |
*** codejunkie has quit IRC | 16:55 | |
*** chmouel has quit IRC | 17:22 | |
*** chmouel has joined #openstack | 17:24 | |
*** rdw has quit IRC | 17:29 | |
*** Jim_Shew has joined #openstack | 17:38 | |
*** rdw has joined #openstack | 17:39 | |
*** RobertLJ has joined #openstack | 17:48 | |
*** binaryWarrior has joined #openstack | 17:49 | |
*** chmouel has left #openstack | 17:49 | |
*** chmouel has joined #openstack | 17:50 | |
*** pvo has joined #openstack | 18:01 | |
*** pvo has joined #openstack | 18:01 | |
*** ChanServ sets mode: +v pvo | 18:01 | |
*** binaryWarrior has quit IRC | 18:01 | |
*** rnewson has quit IRC | 18:13 | |
*** rnewson has joined #openstack | 18:14 | |
*** rnewson has joined #openstack | 18:14 | |
*** henriquetft has joined #openstack | 18:55 | |
*** Jim_Shew has quit IRC | 18:58 | |
*** codejunkie has joined #openstack | 19:02 | |
*** termie has quit IRC | 19:06 | |
*** termie has joined #openstack | 19:08 | |
*** Jim_Shew has joined #openstack | 19:16 | |
*** alexs_ has quit IRC | 19:25 | |
*** alexs_ has joined #openstack | 19:25 | |
*** kuttan_ has left #openstack | 19:27 | |
*** pvo has quit IRC | 19:46 | |
*** justinsb has joined #openstack | 19:59 | |
justinsb | eday: I have a question about your benchmarks... http://oddments.org/?p=494 | 20:00 |
*** justinsheehy has quit IRC | 20:00 | |
eday | justinsb: shoot | 20:01 |
justinsb | eday: thanks! So I'm trying to recreate your findings (as I want to benchmark native threads, including on "other Python runtimes"). | 20:02 |
justinsb | eday: Is this the right command for the flood server @ 512KB, 32k connections? | 20:02 |
justinsb | eday: time LD_LIBRARY_PATH=/usr/local/lib scalestack echo::flood::tcp.port=12345 echo::flood::tcp.iterations=16 echo::flood::tcp.count=32768 | 20:03 |
creiht | justinsb: redbo has an interesting graph of threads vs. eventlet | 20:03 |
justinsb | creiht: Interesting... do you have a link? | 20:04 |
creiht | no, just prodding redbo to post it :) | 20:04 |
justinsb | creiht: My motivation is to try to figure out where Threads break down | 20:04 |
creiht | short storry is that as concurrency went up throughput suffered quite a bit | 20:04 |
justinsb | creiht: I think that's the normal argument against threads, but I'd like to know where the limits are | 20:05 |
justinsb | creiht: And further, whether this is because of the GIL or threads themselves | 20:05 |
eday | justinsb: you need a dev branch I have to get it working properly.. trunk flood tool has a bug | 20:05 |
creiht | My guess is some combination, but it seemed to be along the same lines as david beazely's observations | 20:06 |
*** ctennis has quit IRC | 20:06 | |
justinsb | eday: Ah! Let me check out that code and retry! | 20:06 |
eday | justinsb: but threads break down due to 2 reasons usually: kernel context switching overhead adds up and wasted memory for all the threads (even with minimal stack size settings) | 20:06 |
*** justinsheehy has joined #openstack | 20:06 | |
justinsb | eday: Have you seen the talk which preceeded David Beazely's? http://blip.tv/file/2232349 | 20:07 |
eday | justinsb: nope | 20:07 |
justinsb | eday: I stumbled across it - it basically says "just use threads" | 20:07 |
justinsb | eday: I'd just like to get a better understanding with some hard benchmarks | 20:07 |
justinsb | eday: So I thought I'd start with yours :-) | 20:08 |
creiht | justinsb: Sounds great in theory, but in practice it doesn't work :) | 20:08 |
justinsb | creiht: Why doesn't it work? | 20:08 |
justinsb | creiht: If it's because of the GIL (as David Beazely observed) that might well be fixable | 20:09 |
*** ctennis has joined #openstack | 20:09 | |
creiht | But redbo was writing a simple utility to transfer some data, and did the first implementation with threads, and the performance wasn't great at all. Monkeypatching with eventlet immediately gave a huge boost to throughput | 20:09 |
justinsb | creiht: Well, I want to work from experiments, not dogma, so hopefully redbo will post his code | 20:09 |
creiht | Unfortunatley I can't remember the numbers, but hopefully when he gets on he can shed some more light | 20:10 |
creiht | justinsb: No dogma here, we want to do things the simplist most efficient way possible | 20:10 |
justinsb | eday: Which is the dev branch I should get? | 20:10 |
eday | justinsb: so, it really all comes down to how state is stored for each connection, and how efficient that is (both in size, event detection, and switching states). OS threads can be a bit heavy for this (os thread stacks, kernel context switching, ...) where polling mechanisms and smaller data structures scale better (more context, not as much overhead) | 20:10 |
eday | justinsb: thats not saying an OS thread could be more efficient, just current implementations are still not | 20:11 |
justinsb | eday: Absolutely agree with the idea that it should be possible to write a purpose built scheduler in userland and beat the kernel (generic) scheduler | 20:11 |
creiht | justinsb: I'm also unclear on how you plan on fixing the GIL | 20:12 |
justinsb | eday: The question in my mind are: is it really worth the effort? And can it be done in Python (or other non-multithreaded languages)? | 20:12 |
eday | in fact, lightweight kernel-based co-routines are probably the best thing, they just don't exist :) | 20:12 |
creiht | justinsb: Python is multithreaded, the GIL just prevents them from being as useful as they could | 20:12 |
justinsb | eday: Really, I want to find where the limits are | 20:12 |
justinsb | creiht: In terms of fixing the GIL, it's really just about fixing the problems that David Beazely observed. http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorities-and-new.html | 20:13 |
creiht | *just* :) | 20:14 |
creiht | And I hope that is true | 20:14 |
creiht | But it hasn't been yet | 20:14 |
justinsb | I mean 'just' as opposed to removing the GIL entirely | 20:14 |
creiht | and that was 6 months ago | 20:14 |
*** dendro-afk is now known as dendrobates | 20:15 | |
creiht | And let me say this again, I would love simple *threading* to work for these use cases, but the fact is that they don't yet | 20:16 |
creiht | And I'm not trying to disuade you from doing the testing | 20:17 |
creiht | Just letting you know what our observations have been | 20:17 |
eday | justinsb: right now it's CPU bound assuming you have lots of memory to throw at it (~80k/thread) | 20:17 |
justinsb | creiht: Right! Benchmarks can stop people like me bringing this up again and again :-) | 20:18 |
eday | justinsb: and the CPU is busy with switching between all the different threads, this is heavy because it needs to swap entire stacks and thread context in and out | 20:18 |
creiht | justinsb: another interesting test would also be to once you have your threaded version done, use the exact same code, and have eventlet patch threading, and see what type of difference there is :) | 20:19 |
justinsb | creiht: Awesome idea | 20:19 |
creiht | redbo did the same and was quite surprised at how much difference it made | 20:19 |
justinsb | eday: I know that threads are fairly heavy | 20:20 |
justinsb | eday: They're also much easier to program with than (say) Twisted | 20:20 |
* creiht also notes that programming eventlet code is very close to how you would probram threaded code | 20:20 | |
justinsb | eday: So I just want to run the benchmarks to see when we can get away with using threads and when we have to get smart | 20:20 |
justinsb | eday: Anyway, I branched from lp:superstack before running the benchmarks | 20:20 |
justinsb | eday: Is that right? | 20:20 |
justinsb | eday: Oops lp:scalestack | 20:21 |
eday | justinsb: lp:~eday/scalestack/style-updates (rev 62) | 20:21 |
justinsb | eday: Ah ... thanks! | 20:21 |
eday | justinsb: and then: /usr/bin/time ./bin/scalestack p=scalestack/.libs echo_flood_tcp.address=localhost:12345 v echo::flood.iterations=16 echo_flood_tcp.count=32768 | 20:22 |
justinsb | eday: Thank you again :-) | 20:23 |
justinsb | eday: Much better... it's actually taking some time now :-) | 20:26 |
creiht | http://github.com/opscode/openstack-cookbooks | 20:26 |
eday | justinsb: of course! keep me posted with what you find :) | 20:26 |
creiht | props to holoway | 20:28 |
*** Jim_Shew has quit IRC | 20:30 | |
justinsb | eday: The python servers work fine (twisted and evented), but with the scalestack server the flood tool is giving "Connection reset by peer: 104" errors | 20:36 |
justinsb | eday: "ERROR [echo::flood::tcp] Received error on read: Connection reset by peer:104" | 20:37 |
*** hornbeck has joined #openstack | 21:01 | |
*** dendrobates is now known as dendro-afk | 21:03 | |
*** dendro-afk is now known as dendrobates | 21:04 | |
*** RobertLJ has quit IRC | 21:07 | |
eday | justinsb: make sure you specify max backlog of something high (i use 32k just to be safe) | 21:23 |
justinsb | eday: Thanks - will do | 21:23 |
eday | justinsb: I run the server with: ./bin/scalestack p=scalestack/.libs echo_server_tcp.addresses=':12345' echo_server_tcp.backlog=32768 | 21:24 |
redbo | python threads on a multi-core machine are just horrible. | 21:27 |
justinsb | redbo: Do you have the benchmark that creiht mentioned earlier? | 21:27 |
*** Daviey has quit IRC | 21:28 | |
redbo | umm... I have a little stuff. http://bit.ly/bp3Yzn | 21:28 |
redbo | http://bit.ly/cSJTxj | 21:29 |
justinsb | redbo: Cool... do you have the source code for the benchmark you ran? | 21:29 |
*** hazmat has joined #openstack | 21:30 | |
*** metoikos has joined #openstack | 21:30 | |
redbo | Not put together yet. I was thinking about doing a blog post or something. The script was only a few lines, it created n threads that each spun urllib2.urlopen across a 10GbE connection. | 21:31 |
justinsb | Awesome... well, let me know if you put together a blog post! | 21:31 |
justinsb | What was the target? | 21:31 |
justinsb | i.e. what was the webserver? | 21:32 |
redbo | nginx | 21:32 |
justinsb | Got it. So just a check of performance, similar to Eric Day's, but using HTTP rather than an echo test | 21:33 |
*** Daviey has joined #openstack | 21:33 | |
redbo | yeah, mostly. the thing was, I was doing something like this for work, and I did the first version with threads and was confused that I could only get 1gbps over that link. I didn't realize threads were *that* bad. But when I dropped in eventlet, it jumped up to 7gbps with no real modifiction. | 21:35 |
redbo | So I kind of wanted a simplified version to show that effect. | 21:35 |
redbo | With the GIL trickery, you wind up using more CPU to get less throughput as you add concurrent threads. It's very pronounced on these 8-core machines. | 21:39 |
eday | must have some nasty write barriers to drop it down that much | 21:42 |
redbo | (which is why eventlet peaks around 13% or whatever on that one graph, it's 100% of 1 core) | 21:42 |
*** Daviey has quit IRC | 21:44 | |
*** jtimberman has left #openstack | 21:45 | |
*** Daviey has joined #openstack | 21:47 | |
eday | justinsb: http://oddments.org/?p=64 something I wrote a while ago to keep in mind if you find lots of shared data structures between threads (besides the GIL) | 21:49 |
*** Daviey has quit IRC | 21:50 | |
*** Daviey has joined #openstack | 21:50 | |
justinsb | eday: Yeah, definitely something to watch out for. But probably not as much of a problem for a webserver as it is for DB - much less heavily-contended shared state! | 21:52 |
eday | justinsb: yep. I'm thinking more about internal python data structures, not so much application code | 21:55 |
justinsb | eday: Good point! | 21:55 |
*** allsystemsarego has quit IRC | 22:04 | |
*** alekibango has quit IRC | 22:17 | |
*** notmyname has quit IRC | 22:21 | |
*** notmyname has joined #openstack | 22:23 | |
*** ChanServ sets mode: +v notmyname | 22:23 | |
*** alekibango has joined #openstack | 22:24 | |
justinsb | eday: Here are my results... | 22:26 |
*** Glace has quit IRC | 22:26 | |
justinsb | Python libevent: 1:04 | 22:26 |
justinsb | Python twisted: 1:19 | 22:26 |
justinsb | ScaleStack (1 thread): 47s | 22:26 |
justinsb | Python threaded: 1:04 | 22:26 |
justinsb | This is at 32768 threads, 512KB | 22:27 |
justinsb | So this benchmark doesn't support using twisted or libevent | 22:27 |
justinsb | (Unless I've screwed up!) | 22:28 |
*** Glace has joined #openstack | 22:30 | |
*** alekibango has quit IRC | 22:30 | |
*** alekibango has joined #openstack | 22:31 | |
*** dendrobates is now known as dendro-afk | 22:48 | |
justinsb | Ooops: s/Python libevent/Python eventlet/g | 22:48 |
*** pvo has joined #openstack | 22:59 | |
*** ChanServ sets mode: +v pvo | 22:59 | |
eday | justinsb: how many cores are you on? | 23:03 |
justinsb | eday: 4. AMD Phenom(tm) 9550 Quad-Core Processor | 23:04 |
justinsb | eday: That could be why you saw a disadvantage from threads. If one core is running the flood tool, you only have one thread left to run the server. I have 3. | 23:04 |
justinsb | eday: Which presumably offsets some of the overhead of threads | 23:05 |
*** blpiatt has joined #openstack | 23:08 | |
eday | justinsb: possibly. this was on my laptop (intel core2duo 2.4) | 23:11 |
*** randybias has joined #openstack | 23:13 | |
justinsb | eday: I'm thinking that maybe redbo's benchmark will demonstrate some reason to use an eventing framework | 23:14 |
justinsb | eday: Though realistically, I think we'll need something that mixes I/O and CPU | 23:14 |
eday | justinsb: yup, the echo server was mainly to test i/o and framework overhead | 23:15 |
justinsb | eday: I used a bare-bones socket Python implementation for my echo server; it might be that's not fair? | 23:16 |
eday | justinsb: most of the time for all the python progs is going to be in syscalls, which will be in the lower levels (outside the GIL) | 23:16 |
eday | justinsb: as soon as you process more in the app (and need the GIL), boom | 23:16 |
justinsb | eday: But isn't that true of twisted / eventlet also? | 23:16 |
eday | justinsb: yes, but those are only single threaded, so as you said you'll be bound by a single thread | 23:17 |
eday | justinsb: if you start up twisted/eventlet mulit-process all sharing the accept fd, you'll probably see similar better numbers than python threaded | 23:18 |
justinsb | eday: Wouldn | 23:18 |
justinsb | Sorry | 23:18 |
justinsb | eday: But you could do the same with threads | 23:18 |
eday | justinsb: but threads are already going over multiple cores | 23:19 |
eday | justinsb: twisted/eventlet dont | 23:19 |
justinsb | eday: I think that's an orthogonal argument. | 23:21 |
justinsb | eday: In Python, there's not true multi-threading for CPU bound tasks because of the GIL | 23:21 |
justinsb | eday: Multi-threading in Python just makes the programming easier in terms of things that would otherwise block | 23:21 |
justinsb | eday: Then, if you want to work around the GIL problem - in Twisted, Eventlet or Threading - you probably need multi-process | 23:22 |
justinsb | eday: Or even Jython :-) | 23:22 |
justinsb | eday: Just benchmarked Jython @ 1:34 on the same benchmark | 23:22 |
eday | justinsb: well, if we're talking about the GIL not being the limiting factor here yet (since we're spending more time in syscalls), I think it has some merit | 23:23 |
eday | justinsb: just try putting a couple os.fork() calls after the listen() but before the eventlet loop start | 23:23 |
eday | justinsb: and then you could do the same for the threaded version to see if it helps at all | 23:24 |
*** metoikos has quit IRC | 23:25 | |
*** rdw has quit IRC | 23:28 | |
*** pvo has quit IRC | 23:31 | |
*** rdw has joined #openstack | 23:32 | |
justinsb | eday: With two os.fork (i.e. 4 processes). Twisted goes from 1:19 -> 52s | 23:37 |
justinsb | eday: Threaded 1:04 -> 1:09 | 23:37 |
justinsb | eday: Eventlet I couldn't get to work | 23:37 |
justinsb | eday: In my opinion, this benchmark just says 'just stick with threads' | 23:38 |
justinsb | eday: So to justify Twisted, I'll have to find another benchmark | 23:39 |
justinsb | eday: I looked for Twisted's 'bragging benchmark', but I couldn't find it | 23:39 |
justinsb | eday: Any idea what the canonical case-study benchmark for twisted actually is? | 23:39 |
eday | justinsb: well, twisted improved, but threaded didn't. don't you think that gives twisted a leg up? | 23:45 |
*** nettok has quit IRC | 23:46 | |
eday | justinsb: what happened with eventlet? my eventlet forks fine | 23:46 |
*** randybias has quit IRC | 23:46 | |
rdw | the main reason to use an eventing framework is to save on memory, not to reduce CPU consumption | 23:48 |
rdw | (just by the way) | 23:48 |
*** hazmat has quit IRC | 23:48 | |
redbo | rdw: http://bit.ly/bp3Yzn http://bit.ly/cSJTxj not having GIL contention makes a huge difference in CPU and throughput when you have a bunch of cores. | 23:50 |
rdw | yeah, but we're "lucky" that python threads suck so much | 23:51 |
rdw | in the future that may not be the case, though, if dabeaz does his thing :) | 23:52 |
redbo | that's true, I don't think you'd be able to see much difference using a similar model vs threads in straight C. but it's also the python world as it exists. | 23:52 |
rdw | true | 23:53 |
redbo | you're crazy to use threads for concurrent networking in python right now. In java or something, async vs threaded is a much closer argument. | 23:54 |
rdw | in my opinion, though, focusing on CPU is focusing on the elephant's tail; the other benefits you get from eventing frameworks are also huge: much lower memory consumption, and a better data access story | 23:54 |
*** henriquetft_ has joined #openstack | 23:55 | |
*** henriquetft has quit IRC | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!