C++ vs. Java 1.6 - A fair Benchmark
Posted on:January 29 2007
For one of my small private projects, I wanted to know if I could implement the program in Java instead of C++. The only critical feature for this program was speed, so I needed to know how fast Java is today using the latest VM, compared to C++ using a modern optimizing compiler.
There are lots of tests and benchmarks available on the web, but usually I don't trust any statistics I did not fake myself. So I did my own benchmark with 3 simple real-world test cases and one additional academic test. The tests I found on the web have been mostly usesless, especially if you are programming games. These are the ones I used:
- Sorting a huge array (5 mio doubles) using Heapsort, testing usual, random memory access quite well.
- Calling multiple nested loops (4), each one 280 times, computing a number in the inner loop based on the loop counters.
- Copying an array (8 MB of doubles) into another one, multiple times (100 times).
- Calculating 44 fibonacci numbers, testing recursive calls. Totally useless test for real world programs, but because every benchmark seems to use it, it was interesting to include it as well.
The results
To make it short: As I expected C++ outperforms Java in every single test. But interestingly Java is getting closer now, here are my average results:
| Java | C++ | |
| Fibonacci | 10767 ms | 7468 ms |
| Heapsort | 5454 ms | 4344 ms |
| NestedLoops | 8022 ms | 3453 ms |
| CopyData | 1204 ms | 1125 ms |
Details and Findings
Knowing a bit about the C++ and Java optimizers, I was also able to revert the results completely. For example although using equivalent Java and C++ code, I was able to make the C++ version take 16 seconds for the nested loops-test while Java only needed 1 second by adding some virtual method calls which I knew Java could optimize. I also was able to make the C++ version run in only 78 ms while the equivalent Java version needed even 10 seconds by only using static loops - the C++ optimizer then would not even loop anything and just add the values together. This is partially what some of the wrong benchmarks out there are doing - even if not intentional. If you take a look at the sourcecode of my benchmark, you'll also see that I only measured the actual calculations, I was not interested in the start up times at all. This would have been very unfair because java needs some more time to start up, and it doesn't matter at all if an end user program (especially game) needs some extra seconds in the beginning. That's also the reason I used the client and not the server java VM: No end user is using the server VM at all. Another detail was interesting: Java was running out of heap memory very fast. I even had to reduce the sizes of the arrays in the tests, Java wasn't even able to allocate two arrays of several million doubles. Of course this wasn't a problem on the C++ side at all.
My conclusion: Java is not as fast as C++ although asserted by several people. Simply as that. But with today's incredible hardware, nothing will prevent you from writing simple games with it, unless you are not trying to create a fast Software Rasterizer or similar.
Comments:
interesting stuff. java has gotten quite fast as it seems. at least in some areas...
bakkdoor
Quote2007-01-29 22:13:00
Many thanks for that.
Funnily enough I really can use that right now :)
I wouldn't have thought using virtual mehtods is such a slow-downer... still I think the Java programs are still slow in comparement to C++. And I hate to live without default parameters or (at a smaller scale) without operator overloading.
I just grew up with that :)
Funnily enough I really can use that right now :)
I wouldn't have thought using virtual mehtods is such a slow-downer... still I think the Java programs are still slow in comparement to C++. And I hate to live without default parameters or (at a smaller scale) without operator overloading.
I just grew up with that :)
Gauntlet
Quote2007-01-29 22:21:00
Well nice figures. Thanks for that!
But I have to add that though being very accurate and more fair than other tests I saw already - this is only half the truth again. I would admit that it is correct to say java is slower than c++ when it comes to primitve datatypes and optimizing nested loops but there are real objects also where the numbers may change somewhat. In fact it will depend upon the tasks.
By the way: the server vm is used often on servers's - that is what it is optimzed for. The client VM is faster for running on machines using swing ui. Would be interesting to do some benchmerks to see what happens if the server vm is to be tested with jirr - the java binding of irrlicht.
To make it short: the server vm is ahead of the client vm by some frames (1-3%) in the tech demo (running java 6) when using default values - though it is possible to tune the client vm to reach even higher numbers.
I just was asthonished to see a decrease within the fibonacci test duration by roughly 20% by using the server vm whereas the nested loops were signifcant slower.
Also please note that java is limiting its maximum memory usage to 128MB by default: -Xmx256m (or any higher number) will increase that ... There is also a switch for the initial heapsize: -Xms. http://performance.netbeans.org/howto/jvmswitches/index.html lists some more paramters though with every new java version there seem to be more flags - please ask sun for any hidden ones ;)
PS: Do not blame java for being slow on the client when it is the programmer's fault. The swing ui is somewhat critical to running cpu intensive tasks without launching such a task in a separate thread. That will stop rendering of the gui until the main task returns. And guess what several coders are hacking in ...
But I have to add that though being very accurate and more fair than other tests I saw already - this is only half the truth again. I would admit that it is correct to say java is slower than c++ when it comes to primitve datatypes and optimizing nested loops but there are real objects also where the numbers may change somewhat. In fact it will depend upon the tasks.
By the way: the server vm is used often on servers's - that is what it is optimzed for. The client VM is faster for running on machines using swing ui. Would be interesting to do some benchmerks to see what happens if the server vm is to be tested with jirr - the java binding of irrlicht.
To make it short: the server vm is ahead of the client vm by some frames (1-3%) in the tech demo (running java 6) when using default values - though it is possible to tune the client vm to reach even higher numbers.
I just was asthonished to see a decrease within the fibonacci test duration by roughly 20% by using the server vm whereas the nested loops were signifcant slower.
Also please note that java is limiting its maximum memory usage to 128MB by default: -Xmx256m (or any higher number) will increase that ... There is also a switch for the initial heapsize: -Xms. http://performance.netbeans.org/howto/jvmswitches/index.html lists some more paramters though with every new java version there seem to be more flags - please ask sun for any hidden ones ;)
PS: Do not blame java for being slow on the client when it is the programmer's fault. The swing ui is somewhat critical to running cpu intensive tasks without launching such a task in a separate thread. That will stop rendering of the gui until the main task returns. And guess what several coders are hacking in ...
java protagonist
Quote2007-01-29 23:55:00
I think that Java can be closer to C++ in big real world examples, but of course it is difficult to have more than an impression about that, as almost no big C++ app has been translated to Java. May be it is because it is very easier for an average programmer to optimize a Java program than a C++ one (in C++, you have also to take care of memory leaks, pointers, etc... this can be a lot of work).
The only two examples that I know of that have been implemented both in C++ (or C) and Java are http://www.bytonic.de/html/jake2.html (Jake2) ,a
The only two examples that I know of that have been implemented both in C++ (or C) and Java are http://www.bytonic.de/html/jake2.html (Jake2) ,a
Hervé
Quote2007-01-29 23:59:00
(sorry, some mistyping).
The only two examples that I know of that have been implemented both in C++ (or C) and Java are http://www.bytonic.de/html/jake2.html (Jake2), a clone of Quake2 in Java, and http://cmusphinx.sourceforge.net/sphinx4/ (Sphynx), a speech recognizer first coded in C, then in Java. Jake2 performance in Java 5.0 are identical to the original C versions (in some configurations), and Sphynx4 (in Java) performs better than the C version.
But of course, to be fair, I have to say that Jake2 still binds to a low-level C OpenGL wrapper, and the algorithms in the Java version have surely been improved for performance...
The only two examples that I know of that have been implemented both in C++ (or C) and Java are http://www.bytonic.de/html/jake2.html (Jake2), a clone of Quake2 in Java, and http://cmusphinx.sourceforge.net/sphinx4/ (Sphynx), a speech recognizer first coded in C, then in Java. Jake2 performance in Java 5.0 are identical to the original C versions (in some configurations), and Sphynx4 (in Java) performs better than the C version.
But of course, to be fair, I have to say that Jake2 still binds to a low-level C OpenGL wrapper, and the algorithms in the Java version have surely been improved for performance...
Hervé
Quote2007-01-30 00:05:00
@Hervé: C++ programmers who really have to care about memory leaks and memory management in general have completely messed up (and I'm not talking about using goehm-gc et al).
ak
Quote2007-01-30 00:23:00
next up, c++ vs. C#, the ultimate benchmark throwdown!!
that's one i wanna see :-)
that's one i wanna see :-)
buhatkj
Quote2007-01-30 00:50:00
Yes me too Niko I wonder how C# would perform against C++ and Java... After all its only fair that someone did this...
leo
Quote2007-01-30 07:51:00
And it would be interesting to see memory consumption too ^^ :P
Raedwulf
Quote2007-01-30 09:35:00
I've never heard of anyone claiming Java is faster than C++ - that would be pretty much impossible since there is still a layer of interpretation and garbage collection going on, no matter how well optimised the VM is.
Most of the time when I worked extensively in (server-side) Java I had to defend that Java was fast _enough_ for the types if application that should be written in it. Most people judging it did it with either very old VMs or on application tests which were totally unsuited for the sort of application you would typically build in Java. It's like saying scripting languages are slower than C++ when running tight loops - well, duh. ;) Too many developers look at the minutae without seeing the big picture - that it's worth trading a known amount of performance in certain cases for benefits in other areas.
Most of the time when I worked extensively in (server-side) Java I had to defend that Java was fast _enough_ for the types if application that should be written in it. Most people judging it did it with either very old VMs or on application tests which were totally unsuited for the sort of application you would typically build in Java. It's like saying scripting languages are slower than C++ when running tight loops - well, duh. ;) Too many developers look at the minutae without seeing the big picture - that it's worth trading a known amount of performance in certain cases for benefits in other areas.
Steve
Quote2007-01-30 12:10:00
here is one claiming java is faster: http://kano.net/javabench/index
erik
Quote2007-01-30 12:36:00
This prooved Java is really fast... for an interpreted language. Of course it can not be nearly as fast as C++. But a C++ vs C# would be cool, yes :)
jasper
Quote2007-01-30 12:59:00
It would be interesting to hear about compiler version and compilation flags. Moreover you should Java grant some startup time, so better inline several calls to the test function and average over the inner call times. Also the latest gccs still optimize the loops away with many optimization settings.
hybrid
Quote2007-01-30 14:34:00
I used the VisualStudio 2005 compiler with the standard release mode settings. Hm, another test with different compilers/settings would be interesting as well, right :)
But I don't think it's completely impossible that java programs are faster than c++: Java has the advantage that it can inline virtual functions during runtime and also has the GC, which can compact the heap resulting in better caching where C++ is forced to work with a fragmented heap. Lots of applications can profit from this resulting in possible faster speed than C++.
But I don't think it's completely impossible that java programs are faster than c++: Java has the advantage that it can inline virtual functions during runtime and also has the GC, which can compact the heap resulting in better caching where C++ is forced to work with a fragmented heap. Lots of applications can profit from this resulting in possible faster speed than C++.
niko
Quote2007-01-30 18:13:00
Java Vs C# which one is better for making a 3D game.
leo
Quote2007-01-31 16:40:00
ok..c++ out performs java...i would like to know how fast c# is,'cos i am working on a project with c#..
tunde
Quote2007-02-02 12:48:00
You never mentioned which C++ compiler you're using. That could make a big difference. Using g++ 3.4.4 on Windows under Cygwin with -O3 option, I get these times:
fib: 17341
heap: 4770
nest: 4628
copy: 1932
Here is the code to use the non-MS-specific gettimeofday() call:
struct timeval tv1;
struct timeval tv2;
cout << "testing " << testName << "...\t\t";
gettimeofday(&tv1, 0);
// run test
int retValue = toTest->test();
gettimeofday(&tv2, 0);
delete toTest;
long diff = (tv2.tv_sec - tv1.tv_sec)*1000 +
(tv2.tv_usec - tv1.tv_usec)/1000;
cout << diff <<
"ms. (calculated result: " << retValue << ")\n";
Here are the times I get for Java:
fib: 10282
heap: 4796
nest: 13641
copy: 1922
And for java -server:
fib: 6375
heap: 4796
nest: 13204
copy: 1891
This is using jdk1.5 on a 3Ghz machine.
So the compiler can make a big difference: gcc -O3 takes
17s compared to 10s for whatever you're using.
And the -server
can make a big difference: it brought the Java time down from
10s to 6s for the fibinocci test. And it's perfectly reasonable to use -server for this kind of thing. The "client" VM is really intended for GUI apps where startup time is to be minimized. Any app where non-GUI performance matters would use -server.
So Java is about 3x faster on one test, C++ about 3x faster on another, and they're about the same on two tests. That seems about right to me - I wouldn't say that either one is generally faster than the other - it all depends.
fib: 17341
heap: 4770
nest: 4628
copy: 1932
Here is the code to use the non-MS-specific gettimeofday() call:
struct timeval tv1;
struct timeval tv2;
cout << "testing " << testName << "...\t\t";
gettimeofday(&tv1, 0);
// run test
int retValue = toTest->test();
gettimeofday(&tv2, 0);
delete toTest;
long diff = (tv2.tv_sec - tv1.tv_sec)*1000 +
(tv2.tv_usec - tv1.tv_usec)/1000;
cout << diff <<
"ms. (calculated result: " << retValue << ")\n";
Here are the times I get for Java:
fib: 10282
heap: 4796
nest: 13641
copy: 1922
And for java -server:
fib: 6375
heap: 4796
nest: 13204
copy: 1891
This is using jdk1.5 on a 3Ghz machine.
So the compiler can make a big difference: gcc -O3 takes
17s compared to 10s for whatever you're using.
And the -server
can make a big difference: it brought the Java time down from
10s to 6s for the fibinocci test. And it's perfectly reasonable to use -server for this kind of thing. The "client" VM is really intended for GUI apps where startup time is to be minimized. Any app where non-GUI performance matters would use -server.
So Java is about 3x faster on one test, C++ about 3x faster on another, and they're about the same on two tests. That seems about right to me - I wouldn't say that either one is generally faster than the other - it all depends.
Andy Tripp
Quote2007-02-07 03:57:00
I see many people are interested in c# results... so was I. Here are my results with java (1.6), c# (vs2005sp1, release), c++ (vs2005sp1, release) on Athlon 64 3200+ Windows XP machine:
java: fib: 9531, heap: 3640, loops: 12688, copy: 1500
c# : fib: 12671, heap: 3765, loops: 9406, copy: 1453
c++ : fib: 8750, heap: 3594, loops: 6328, copy: 1500
c# port is pretty straightforward from java... One more thing. None of this languages is interpereted... c# and java use just-in-time compiler, while c++ uses "classic" one. All results are surprisingly close to each other and all languages should now be usable for most purposes. (Someone could try compiling java source to native (GCJ)?)
java: fib: 9531, heap: 3640, loops: 12688, copy: 1500
c# : fib: 12671, heap: 3765, loops: 9406, copy: 1453
c++ : fib: 8750, heap: 3594, loops: 6328, copy: 1500
c# port is pretty straightforward from java... One more thing. None of this languages is interpereted... c# and java use just-in-time compiler, while c++ uses "classic" one. All results are surprisingly close to each other and all languages should now be usable for most purposes. (Someone could try compiling java source to native (GCJ)?)
bd
Quote2007-02-12 17:34:00
Add comment:
|
|
Possible Codes
| Feature | Code |
| Link | [url] www.example.com [/url] |
| Bold | [b]bold text[/b] |
| Quote | [quote]quoted text[/quote] |
| Code | [code]source code[/code] |
Emoticons
 |  |  |  |  |  |  |  |
 |  |  |  |  |