resin clustering

As traffic increases beyond a single server, Resin's clustering lets you add new machines to handle the load and simultaneously improves uptime and reliability by failing over requests from a downed or maintenance server to a backup transparently.

A session needs to stay on the same JVM that started it. Otherwise, each JVM would only see every second or third request and get confused.

To make sure that sessions stay on the same JVM, Resin encodes the cookie with the host number. In the previous example, the hosts would generate cookies like:

On the web-tier, Resin will decode the cookie and send it to the appropriate host. So bacX8ZwooOz would go to app-b.

In the infrequent case that app-b fails, Resin will send the request to app-a. The user might lose the session but that's a minor problem compared to showing a connection failure error.

The following example is a typical configuration for a distributed server using an external hardware load-balancer, i.e. where each Resin is acting as the HTTP server. Each server will be started as -server a or -server b to grab its specific configuration.

In this example, sessions will only be stored when the server shuts down, either for maintenance or with a new version of the server. This is the most lightweight configuration, and doesn't affect performance significantly. If the hardware or the JVM crashes, however, the sessions will be lost. (If you want to save sessions for hardware or JVM crashes, remove the <save-only-on-shutdown/> flag.)

<resin xmlns="http://caucho.com/ns/resin">
<cluster id="app-tier">
  <server-default>
    <http port='80'/>
  </server-default>

  <server id='app-a' address='192.168.0.1'/>
  <server id='app-b' address='192.168.0.2'/>
  <server id='app-c' address='192.168.0.3'/>

  <web-app-default>
    <!-- enable tcp-store for all hosts/web-apps -->
    <session-config>
      <use-persistent-store/>
      <save-only-on-shutdown/>
    </session-config>
  </web-app-default>

  ...
</cluster>
</resin>

<resin xmlns="http://caucho.com/ns/resin">
  ...

  <cluster id="app-tier">
    <server id="app-a" host="192.168.0.1" port="6802"/>
    <server id="app-b" host="192.168.0.2" port="6802"/>

    ...
  </cluster>
</resin>

<web-app xmlns="http://caucho.com/ns/resin">
  <session-config>
    <use-persistent-store="true"/>
  </session-config>
</web-app>

<resin xmlns="http://caucho.com/ns/resin">
<cluster id="app-tier">

  <server-default>
    <http port='80'/>
  </server-default>

  <server id="app-a" address="192.168.2.10" port="6802"/>
  <server id="app-b" address="192.168.2.11" port="6803"/>

  <host id=''>
  <web-app id=''>

  <session-config>
    <use-persistent-store="true"/>
  </session-config>

  </web-app>
  </host>
</cluster>
</resin>

<resin xmlns="http://caucho.com/ns/resin">
  ...
  
  <cluster id="app-tier">

    <server id="app-a" host="192.168.0.1"/>
    <server id="app-b" host="192.168.0.2"/>
    <server id="app-c" host="192.168.0.3"/>
    <server id="app-d" host="192.168.0.4"/>

    ...
    <host id="">
    <web-app id='myapp'>
      ...
      <session-config>
        <use-persistent-store/>
      </session-config>
    </web-app>
    </host>
  </cluster>
</resin>

resin-4.0.x> java -jar lib/resin.jar -conf conf/resin.xml -server c start

package test;

public class Counter implements java.io.Serializable {
  private int _count;

  public int nextCount() { return _count++; }
}

...
<web-app id="/foo">
...
<session-config>
  <use-persistent-store/>
  <always-save-session/>
</session-config>
...
</web-app>

Protocol Examples

Session Request

To see how cluster sessions work, consider a case where the load balancer sends the request to a random host. Server C owns the session but the load balancer gives the request to Server A. In the following figure, the request modifies the session so it must be saved as well as loaded.

The session id encodes the owning host. The example session id, ca8MbyA, decodes to an server index of 3, mapping to Server C. Resin determines the backup host from the cookie as well. Server A must know the owning host for every cookie so it can communicate with the owning srun. The example configuration defines all the sruns Server A needs to know about. If Server C is unavailable, Server A can use its configuration knowledge to use Server D as a backup for ca8MbyA instead..

When the request first accesses the session, Server A asks Server C for the serialized session data (2:load). Since Server A doesn't cache the session data, it must ask Server C for an update on each request. For requests that only read the session, this TCP load is the only extra overhead, i.e. they can skip 3-5. The always-save-session flag, in contrast, will always force a write.

At the end of the request, Server A writes any session updates to Server C (3:store). If always-save-session is false and the session doesn't change, this step can be skipped. Server A sends the new serialized session contents to Server C. Server C saves the session on its local disk (4:save) and saves a backup to Server D (5:backup).

Sticky Session Request

Smart load balancers that implement sticky sessions can improve cluster performance. In the previous request, Resin's cluster sessions maintain consistency for dumb load balancers or twisted clients like the AOL browsers. The cost is the additional network traffic for 2:load and 3:store. Smart load-balancers can avoid the network traffic of 2 and 3.

Server C decodes the session id, caaMbyA. Since it owns the session, Server C gives the session to the servlet with no work and no network traffic. For a read-only request, there's zero overhead for cluster sessions. So even a semi-intelligent load balancer will gain a performance advantage. Normal browsers will have zero overhead, and bogus AOL browsers will have the non-sticky session overhead.

A session write saves the new serialized session to disk (2:save) and to Server D (3:backup). always-save-session will determine if Resin can take advantage of read-only sessions or must save the session on each request.

Disk copy

Resin stores a disk copy of the session information, in the location specified by the path. The disk copy serves two purposes. The first is that it allows Resin to keep session information for a large number of sessions. An efficient memory cache keeps the most active sessions in memory and the disk holds all of the sessions without requiring large amounts of memory. The second purpose of the disk copy is that the sessions are recovered from disk when the server is restarted.

Failover

Since the session always has a current copy on two servers, the load balancer can direct requests to the next server in the ring. The backup server is always ready to take control. The failover will succeed even for dumb load balancers, as in the non-sticky-session case, because the srun hosts will use the backup as the new owning server.

In the example, either Server C or Server D can stop and the sessions will use the backup. Of course, the failover will work for scheduled downtime as well as server crashes. A site could upgrade one server at a time with no observable downtime.

Recovery

When Server C restarts, possibly with an upgraded version of Resin, it needs to use the most up-to-date version of the session; its file-saved session will probably be obsolete. When a "new" session arrives, Server C loads the saved session from both the file and from Server D. It will use the newest session as the current value. Once it's loaded the "new" session, it will remain consistent as if the server had never stopped.

No Distributed Locking

Resin's cluster sessions does not lock sessions. For browser-based sessions, only one request will execute at a time. Since browser sessions have no concurrently, there's no need for distributed locking. However, it's a good idea to be aware of the lack of distributed locking.

Adding a dynamic server to a cluster is a simple two-step process:

1. Register the dynamic server with a triad server via JMX.
2. Start the new dynamic server using the registration in the previous step.

Before adding a dynamic server, you must:

• Set up and start a cluster with a triad, e.g.
  Example: conf/resin.xml

  <resin xmlns="http://caucho.com/ns/resin">
  
  <cluster id="app-tier">
    ...
    <server id="triad-a" address="234.56.78.90" port="6800"/>
    <server id="triad-b" address="34.56.78.90" port="6800"/>
    <server id="triad-c" address="45.67.89.12" port="6800"/>
  

• Install at least one admin password, usually in admin-users.xml
• Enable the RemoteAdminService for the cluster, e.g.

  <resin xmlns="http://caucho.com/ns/resin">
  
  <cluster id="app-tier">
    ...
    <admin:RemoteAdminService xmlns:admin="urn:java:com.caucho.admin"/>
    ...
  

• Enable the dynamic servers for the cluster, e.g.

  <resin xmlns="http://caucho.com/ns/resin">
  
  <cluster id="app-tier">
    ...
    <dynamic-server-enable>true</dynamic-server-enable>
    ...
  

Check the main Clustering section for more information on this topic.

For the first step of registration, you can use a JMX tool like jconsole or simply use the Resin administration web console. We'll show how to do the latter method here. For registration, you'll specify three values:

With these three values, browse to the Resin administration application's "cluster" tab. If you have enabled dynamic servers for your cluster, you should see a form allowing you to register the server in the "Cluster Overview" table.

Once you have entered the values and added the server, it should show up in the table as a dead server because we haven't started it yet. The dynamic server's registration will be propagated to all the servers in the cluster.

Now that we've registered the dynamic server, we can start it and have it join the cluster. In order for the new server to be recognized and accepted by the triad, it needs to start with the same resin.xml that the triad is using, the name of the cluster it is joining, and the values entered in the registration step. These can all be specified on the command line when starting the server:

dynamic-server> java -jar $RESIN_HOME/lib/resin.jar -conf /etc/resin/resin.xml \
                     -dynamic-server app-tier:123.45.67.89:6800 start

Specifying the configuration file allows the new server to configure itself using the <server-default> options, to find the triad servers of the cluster it is joining, and to authenticate using the administration logins. This command starts the server, which immediately contacts the triad to join the cluster. Once it has successfully joined, the "Cluster" tab of the administration application should look like this:

• See the index for a list of all the tags.
• See Web Application configuration for web.xml (Servlet) configuration.
• See Server tags for ports, threads, and JVM configuration.
• See Resource configuration for resources: classloader, databases, connectors, and resources.
• See Log configuration for access log configuration, java.util.logging, and stdout/stderr logging.

<access-log> configures a HTTP access log for all virtual hosts in the cluster. See access-log in the <host> tag for more information.

<cache> configures the proxy cache (requires Resin Professional). The proxy cache improves performance by caching the output of servlets, jsp and php pages. For database-heavy pages, this caching can improve performance and reduce database load by several orders of magnitude.

The proxy cache uses a combination of a memory cache and a disk-based cache to save large amounts of data with little overhead.

Management of the proxy cache uses the ProxyCacheMXBean.

element cache {
  disk-size?
  & enable?
  & enable-range?
  & entries?
  & path?
  & max-entry-size?
  & memory-size?
  & rewrite-vary-as-private?
}

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">
        <cache entries="16384" disk-size="2G" memory-size="256M"/>

        <server id="a" address="192.168.0.10"/>

        <host host-name="www.foo.com">
    </cluster>
</resin>

<cluster> configures a set of identically-configured servers. The cluster typically configures a set of <server>s, each with some ports, and a set of virtual <host>s.

Only one <cluster> is active in any on server. At runtime, the <cluster> is selected by the <server> with id matching the -server-id on the command line.

element cluster {
  attribute id { string }
  & environment resources
  & access-log?
  & cache?
  & connection-error-page?
  & ear-default*
  & error-page*
  & host*
  & host-default*
  & host-deploy*
  & ignore-client-disconnect?
  & invocation-cache-size?
  & invocation-cache-max-url-length?
  & machine*
  & persistent-store?
  & ping*
  & redeploy-mode?
  & resin:choose*
  & resin:import*
  & resin:if*
  & rewrite-dispatch?
  & root-directory?
  & server*
  & server-default*
  & server-header?
  & session-cookie?
  & session-sticky-disable?
  & url-character-encoding?
  & url-length-max?
  & web-app-default*
}

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">
        <server-default>
            <http port="8080"/>
        </server-default>

        <server id="a" address="192.168.0.10"/>
        <server id="b" address="192.168.0.11"/>

        <host host-name="www.foo.com">
          ...
        </host>
    </cluster>
</resin>

<cluster-default> defines default cluster configuration for all clusters in the <resin> server.

<resin xmlns="http://caucho.com/ns/resin">
    <cluster-default>
        <cache entries="16384" memory-size="64M"/>
    </cluster-default>

    <cluster id="web-tier">
        ...
    </cluster>

    <cluster id="app-tier">
        ...
    </cluster>
</resin>

<connection-error-page> specifies an error page to be used by IIS when it can't contact an app-tier Resin. This directive only applies to IIS.

element connection-error-page {
  string
}

<development-mode-error-page> enables browser error reporting with extra information. Because it can expose internal data, it is not generally recommended in production systems. (The information is generally coped to the log.

<ear-default> configures defaults for .ear resource, i.e. enterprise applications.

<error-page> defines a web page to be displayed when an error occurs outside of a virtual host or web-app. Note, this is not a default error-page, i.e. if an error occurs inside a <host> or <web-app>, the error-page for that host or web-app will be used instead.

See webapp: error-page.

element error-page {
  (error-code | exception-type)?
  & location?
}

<host> configures a virtual host. Virtual hosts must be configured explicitly.

• See host tags for configuration details.

<host host-name="www.foo.com">
  <host-alias>foo.com</host-alias>
  <host-alias>web.foo.com</host-alias>

  <root-directory>/opt/www/www.foo.com</root-directory>

  <web-app id="/" document-directory="webapps/ROOT">
    
  </web-app>
  ...
</host>

<host regexp="([^.]+)\.foo\.com">
  <host-name>${host.regexp[1]}.foo.com</host-name>

  <root-directory>/var/www/hosts/www.${host.regexp[1]}.com</root-directory>

  ...
</host>

It is recommended that any <host> using a regexp include a <host-name> to set the canonical name for the host.

Defaults for a virtual host.

The host-default can contain any of the host configuration tags. It will be used as defaults for any virtual host.

Configures a deploy directory for virtual host.

The host-deploy will add an EL variable ${name}, referring to the name of the host jar file.

ignore-client-disconnect configures whether Resin should ignore disconnection exceptions from the client, or if it should send those exceptions to the application.

element ignore-client-disconnect {
  r_boolean-Type
}

Configures the number of entries in the invocation cache. The invocation cache is used to store pre-calculated servlet and filter chains from the URLs. It's also used as the basis for proxy caching.

element invocation-cache-size {
  r_int-Type
}

Configures the longest entry cacheable in the invocation cache. It is used to avoid certain types of denial-of-service attacks.

element invocation-cache-max-url-length {
  r_int-Type
}

Sets limit on longest URIs that can be served by Resin.

element max-uri-length {
  r_int-Type
}

Defines the cluster-aware persistent store used for sharing distributed sessions. The allowed types are "jdbc", "cluster" and "file". The "file" type is only recommended in single-server configurations.

The <persistent-store> configuration is in the <server> level because it needs to share update information across the active cluster and the <cluster> definition is at the <server> level. Sessions activate the persistent store with the <use-persistent-store> tag of the <session-config>.

See Persistent sessions for more details.

element persistent-store {
  type
  & init?
}

element persistent-store {
  type { "cluster "}

  element init {
    always-load?
    & always-save?
    & max-idle-time?
    & triplicate?
    & wait-for-acknowledge?
  }
}

<resin xmlns="http://caucho.com/ns/resin">
<cluster>
  <server id="a" address="192.168.0.1" port="6800"/>
  <server id="b" address="192.168.0.2" port="6800"/>

  <persistent-store type="cluster">
    <init>
      <triplicate>true</triplicate>
    </init>
  </persistent-store>

  <web-app-default>
    <session-config use-persistent-store="true"/>
  </web-app-default>
</cluster>
</resin>

<resin xmlns="http://caucho.com/ns/resin">
<cluster>
  <server id="a" address="192.168.0.1" port="6800"/>
  <server id="b" address="192.168.0.2" port="6800"/>

  <persistent-store type="jdbc">
    <init>
      <data-source>jdbc/session</data-source>

      <max-idle-time>24h</max-idle-time>
    </init>
  </persistent-store>

  <web-app-default>
    <session-config use-persistent-store="true"/>
  </web-app-default>
</cluster>
</resin>

Starts a thread that periodically makes a request to the server, and restarts Resin if it fails. This facility is used to increase server reliability - if there is a problem with the server (perhaps from a deadlock or an exhaustion of resources), the server is restarted.

A failure occurs if a request to the url returns an HTTP status that is not 200.

Since the local process is restarted, it does not make sense to specify a url that does not get serviced by the instance of Resin that has the ping configuration. Most configurations use url's that specify 'localhost' as the host.

This pinging only catches some problems because it's running in the same process as Resin itself. If the entire JDK freezes, this thread will freeze as well. Assuming the JDK doesn't freeze, the PingThread will catch errors like deadlocks.

<resin xmlns="http://caucho.com/ns/resin"
       xmlns:resin="http://caucho.com/ns/resin/core">
    <cluster id="app-tier">
        <ping url="http://localhost/"/>
        ...
    </cluster>
</resin>

<resin xmlns="http://caucho.com/ns/resin"
       xmlns:resin="http://caucho.com/ns/resin/core">
    ...
    <cluster id="app-tier">
        <ping>
            <url>http://localhost:8080/index.jsp</url>
            <url>http://localhost:8080/webapp/index.jsp</url>
            <url>http://virtualhost/index.jsp</url>
            <url>http://localhost:443/index.jsp</url>

            <sleep-time>5m</sleep-time>
            <try-count>5</try-count>
    
            <!-- a very busy server -->
            <socket-timeout>30s</socket-timeout>
        </ping>
      ...
    </cluster>
</resin>

The class that corresponds to <ping> is PingThread.

<resin xmlns="http://caucho.com/ns/resin"
       xmlns:resin="http://caucho.com/ns/resin/core">
  ...
  <cluster id="web-tier">
    <ping resin:type="com.caucho.server.admin.PingMailer">
      <url>http://localhost:8080/index.jsp</url>
      <url>http://localhost:8080/webapp/index.jsp</url>

      <mail-to>fred@hogwarts.com</mail-to>
      <mail-from>resin@hogwarts.com</mail-from>
      <mail-subject>Resin ping has failed for server ${'${'}server.name}</mail-subject>
    </ping>
    ...
  </server>
</resin>

  <system-property mail.smtp.host="127.0.0.1"/>
  <system-property mail.smtp.port="25"/>

All Environment tags are available to the <cluster>. For example, resources like <database>.

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="app-tier">
        <database jndi-name="jdbc/test">
            <driver type="org.postgresql.Driver">
                <url>jdbc:postgresql://localhost/test</url>
                <user>caucho</user>
            </driver>
        </database>

        <server id="a" ...>
          ...

        <host host-name="www.foo.com">
          ...
    </cluster>
</resin>

<rewrite-dispatch> defines a set of rewriting rules for dispatching and forwarding URLs. Applications can use these rules to redirect old URLs to their new replacements.

See rewrite-dispatch for more details.

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">

        <rewrite-dispatch>
            <redirect regexp="^http://www.foo.com"
                      target="http://bar.com/foo"/>
        </rewrite-dispatch>
  
    </cluster>
</resin>

<root-directory> configures the root directory for files within the cluster. All paths in the <cluster> will be relative to the root directory.

element root-directory {
  r_path-Type
}

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="app-tier">
        <root-directory>/var/www/app-tier</root-directory>

        <server id="a" ...>

        <host host-name="www.foo.com">
    </cluster>
</resin>

The <server> tag configures a JVM instance in the cluster. Each <server> is uniquely identified by its id attribute. The id will match the -server-id command line argument.

See the full server configuration for more details of the <server> tag and its children.

The current server is managed with a ServerMXBean. The ObjectName is resin:type=Server.

Peer servers are managed with ServerConnectorMXBean. The ObjectName is resin:type=ServerConnector,name=server-id.

element server {
  attribute id { string }
  & address?
  & bind-ports-after-start?
  & cluster-port*
  & group-name?
  & http*
  & java-exe?
  & jvm-arg?
  & jvm-classpath?
  & keepalive-connection-time-max?
  & keepalive-max?
  & keepalive-select-enable?
  & keepalive-timeout?
  & load-balance-connect-timeout?
  & load-balance-idle-time?
  & load-balance-recover-time?
  & load-balance-socket-timeout?
  & load-balance-warmup-time?
  & load-balance-weight?
  & memory-free-min?
  & ping?
  & port?
  & protocol?
  & shutdown-wait-max?
  & socket-timeout?
  & thread-max?
  & thread-executor-task-max?
  & thread-idle-max?
  & thread-idle-min?
  & user-name?
  & watchdog-jvm-arg*
  & watchdog-port?
}

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">
        <server id="a" address="192.168.0.10" port="6800">
          <http port="8080"/>
        </server>

        <server id="b" address="192.168.0.11" port="6800">
          <http port="8080"/>
        </server>

        <server id="c" address="192.168.0.12" port="6800">
          <http port="8080"/>
        </server>

        <host id="">
          ...
    </cluster>
</resin>

Defines default values for all <server> instances. See <server> configuration for more details.

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">
        <server-default>
            <server-port>6800</server-port>

            <http port="8080"/>
        </server-default>

        <server id="a" address="192.168.0.10"/>
        <server id="b" address="192.168.0.11"/>
        <server id="c" address="192.168.0.12"/>

        <host id="">
          ...
    </cluster>
</resin>

Configures the HTTP Server: header which Resin sends back to any HTTP client.

element server-header {
  string
}

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">
        <server-header>MyServer/1.0</server-header>
    </cluster>
</resin>

Configures the cookie used for servlet sessions.

element session-cookie {
  string
}

Disables sticky sessions from the load balancer.

element session-sticky-disable {
  r_boolean-Type
}

Configures the URL prefix used for session rewriting.

element session-cookie {
  string
}

Defines an alternative session cookie to be used for a SSL connection. Having two separate cookies increases security.

element ssl-session-cookie {
  string
}

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="web-tier">
        <ssl-session-cookie>SSLJSESSIONID</ssl-session-cookie>
        ...
    </cluster>
</resin>

Defines the character encoding for decoding URLs.

The HTTP specification does not define the character-encoding for URLs, so the server must make assumptions about the encoding.

element url-character-encoding {
  string
}

<web-app-default> defines default values for any web-app in the cluster.

<resin xmlns="http://caucho.com/ns/resin">
    <cluster id="app-tier">

        <web-app-default>
            <servlet servlet-name="resin-php"
                     servlet-class="com.caucho.quercus.servlet.QuercusServlet"/>

            <servlet-mapping url-pattern="*.php"
                             servlet-name="resin-php"/>
        </web-app-default>

        <host id="">
          ...
    </cluster>
</resin>