Performance/stability issues

We have the following set-up:

5 * Sun E250 webservers running apacheloadbalanced via a hardware
loadbalancer with an Oracle cluster backend - each of the webservers runs
it's own resin/jvm and we preserve session consistancy by using resin
clustering

  <!-- add clustering, define ws1-5 hosts -->
  <srun id="d" host='10.240.12.4' port='6802' srun-index='4' />
  <srun-backup id="a" host='10.240.12.1' port='6802' srun-index='1' />
  <srun-backup id="b" host='10.240.12.2' port='6802' srun-index='2' />
  <srun-backup id="c" host='10.240.12.3' port='6802' srun-index='3' />
  <srun-backup id="e" host='10.240.12.5' port='6802' srun-index='5' />

The problem is that we are having some major issues with performance on the
site. A typical scenario is that one of the webservers will start to slow
down and eventually stop serving any pages via resin. This seems to have a
knock on effect on the other webservers [who seem to be still sending more
traffic to the lame server] and eventually all the servers will tie
themselves in knots and stop serving.

We see behaviour similar to this thread:
http://www.caucho.com/support/resin-interest/0209/0146.html

The wierd thing is that the servers are not overloaded and resin never logs
any errors, nor is anything logged in the applications logs- we simply can't
connect on port 80. Netstat shows hundreds of established connections both
on port 80 and to the other servers on 6802.

Our resin configs are something like:

  <thread-max>300</thread-max>
  <thread-keepalive id='150'/>
  <request-timeout id='30s'/>
  <accept-buffer-size id='256'/>
  <thread-min id='5'/>
 <class-update-interval id='120'/>
<cache-mapping url-pattern='/' expires='15m'/>

So far we have tried:

        * Updating Java to 1.4
        * Tweaking the application code
        * Tweaking Apache configs

Any ideas of areas we should look at?

Dave

-- 
David Campbell
Granada Sport and Interactive
+44(0)207 578 4415
david.campbell@xxx.com
********************************************************************** 
Please visit the official Granada website at www.granadamedia.com
for the latest company news.
This email and any files transmitted are confidential and intended 
solely for the use of the individual or entity to which they are 
addressed.  If you have received this email in error, please notify 
postmaster@xxx.com 
Thank you.
**********************************************************************