Overhead measurements are always valid and often the most useful measurement. It tells you the best you can possibly do, i.e. the speed limit. Because it doesn't exercise any functionality except the overhead, it's impossible to defeat the measurement. RPC mechanisms like CORBA, NFS, or message passing kernel OS often quote the 'null-RPC' measurement.

The bulk of a servlet's work is identical no matter which JSP engine is chosen. The Java code runs at the same speed and the database drivers are identical. So creating a "more realistic" benchmark would only add a constant time to each request. This benchmark is less effective comparing languages and drivers.

The second point does become important when comparing JSP/Java to JSP/JavaScript or Perl or PHP or ASP. This benchmark would be less effective in comparing technologies with different languages and drivers. It still would give a sureful overhead result, but would not give a good idea of how real applications compare. In addition, because the JDK is the same, the JSP engines can't do the equivalent of optimizing out loops.

Third, the benchmark isn't cacheable without heroic measures. When comparing computers or static web servers, the performance when the results come from a cache are better and unrepresentitive of real results. That's one reason the sieve and n-queens benchmark are bad, and why SpecWeb goes to great lengths to defeat web server caching. (Of course, in the last case, many sites can fit the most popular pages into a memory cache, so the memory results may actually be more indicative of the real performance.)

Finally, tweaking configurations typically only gets at most 20% improvement. Resin is three times faster. No amount of tweaking will change that significantly. For example, JDK 1.1.7 was used because it's the most stable on the Linux platform and because all the engines ran successfully with it. Internally, I did spot checks with JDK 1.2 and the numbers were similar. The full results for JDK 1.2 will be available shortly; time pressures limited their inclusion.

The near-static performance means that many sites, who never considered JSP or servlets because of performance, can now take advantage of the Java platform's reliability and ease of programming benefits.