The new version of the benchmark is incomplete. It leaves out several servlet engines. FYI, here's the old data. All results are in operations per second. The results are sorted by Hello performance.

• Resin 1.1 b2 (internal web server)
• Orion 0.7.6b (internal web server)
• Tomcat (internal web server)
• Resin 1.1 b2/Apache 1.3.9
• mod_php 4.0b2
• mod_perl 1.21
• JRun 2.3.3/Apache 1.3.9
• ServletExec 2.2/Apache 1.3.9
• CGI using Perl

The server is a 266 Mhz Pentium II running RedHat 6.0 with 64 meg of ram. The client is a 300 Mhz Celeron also running RedHat 6.0 with 32 meg. The two machines are connected with a 100bT ethernet connection.

The client is a simple C program (benchmark.jar or benchmark.tgz) acting as a browser. It sends requests, including typical header fields, and retrieves the results. The results are hashed and compared to an expected value.

Before each test, the client sends and discards a single request. This takes care of startup costs. In the test, 1000 requests were sent sequentially; the client waited for the request to complete before sending another request. Each test was repeated three times and the median result chosen.

Apache configurations used Apache 1.3.9. All modules were compiled with DSO support.

Where possible, the benchmark used the IBM 1.1.8 JDK. When the servlet engine didn't work with the IBM JDK, the Blackdown 1.2 was substituted instead. For example, Orion needs JDK 1.2 and JRun spun using the IBM JDK.

Database benchmarks used Mysql using the mm.mysql JDBC driver.

This benchmark didn't include a full application for two reasons. First, we're somewhat crunched for time. Second, what would be a representative benchmark? The home page of slashdot? Also, should be application be written as most people would write it, or the most maintainable, or the fastest hacked up version the best programmers can come up with? So a full application would raise more questions than it answered.

For example, compare the "Big" numbers with the "Loop" numbers. They both send the same amount of data back to the client, but the performance is difference is large. Based on this information alone, an evil benchmarker could tweak the benchmark to the benefit of any engine.

Based on this benchmark, the Perl and PHP database drivers are faster than the JDBC driver. This appears to be a JDBC implementation issue, not anything intrinsic to Java. The internal Caucho prototype JDBC driver gives much faster results, even faster than Perl and PHP. Since real projects can't use internal JDBC prototypes, the DB performance difference may matter to some projects.

Also, the 'Big' benchmark also gives the advantage to Perl. It may be interesting to note that changing the benchmark slightly makes Resin faster than Perl. If the benchmark is a simple for loop printing 1024 lines, with the same output size, Resin is faster than Perl. The benchmarks will be updated in a few weeks including the new loop test.

So is Resin or PHP or Perl faster? It really depends on your application. As a first cut, it's probably best to assume they have about the same performance unless you can benchmark your specific application. A difference of only 30% in a toy benchmark should be interpreted as having little or no difference.

These results are primarly important because previous servlet engines were much slower than PHP or Perl. With Resin, servlet performance is now about the same. Projects should use other considerations, like maintainability or ease of implementation to decide between the languages.

The near-static performance means that many sites, who never considered JSP or servlets because of performance, can now take advantage of the Java platform's reliability and ease of programming benefits.