http://blog.akquinet.de/2012/06/21/clustering-in-jboss-as7eap-6/
Overview
The ability to combine different servers to a cluster that hides its internal servers from the clients and offers a virtual platform for an application is important for enterprise applications. It can be used to provide
- high scalability by adding cheap computational resources to the cluster on demand or
- high availability by using a transparent failover that hides faults of the internal servers.
Usually high scalability limits high availability and vice versa, but it is also possible to get both. The JBoss application server can be configured to support both features.
This post is the first one of a series about clustering with the JBoss AS 7. Here, we focus on the basic concepts behind JBoss AS 7 clustering and show you how to setup a basic clustered environment with a simple Java EE application.
In the series, we concentrate on the JBoss AS 7 respectively the EAP 6 which is the Red Hat-supported version of the JBoss application server. Future posts will be about particular subsystems of the JBoss AS, such as HornetQ or Infinispan.
Clustering subsystem of the JBoss AS 7
To support high availability, application data has to be replicated inside the cluster. This ensures that the crash of one server does not result in the loss of data. In the AS 7, the distributed cache of Infinispan is used as fundament for replication. A Java EE application stores data in different layers. Accordingly the AS 7 comes with four preconfigured cache container for replication between the cluster nodes:
- web – replication of HTTP sessions
- sfsb – replication of stateful session beans
- hibernate – second level entity cache for JPA/Hibernate
- cluster – general purpose replication of objects in a cluster
To actually transmit replicated data between the cluster nodes, Infinispan uses JGroups as the underlying subsystem. JGroups supports the creation of groups from distributed nodes. It provides operations to add new nodes, to remove nodes explicitely and to automatically abandon faulty nodes. The JBoss AS 7 includes two protocol stacks for reliable communication in a group of nodes. A UDP based protocol stack utilizing multicasts and a TCP based protocol stack for environments that do not support multicast communication. UDP is the default protocol stack in JBoss AS 7.
To support high scalability the requests from the clients have to be distributed among the cluster nodes. This is called load balancing. For load balancing of HTTP-clients, AS 7 supports the mod_cluster module for the Apache httpd server. It provides intelligent, dynamic load balancing for web applications. It utilizes the AJP connector and communicates over UDP with the httpd server module. Contrary to the older mod_jk, the httpd worker is dynamically configured.
Load balancing for EJB remote clients is supported by the JBoss Client library for EJB applications. The library will receive the topology of the cluster from the underling remoting implementation. By default a random node selector is used for load balancing of remote EJB invocations. A remote client uses JBoss remote naming as directory service with a JNDI interface to access the RMI-stub.
The example-application
To demonstrate clustering, we start with a simple example-application. It includes one stateful session bean that counts the invocations and one stateless session bean implementation, which returns the name of its cluster node. Both session beans are called from a JSF page. You can find the application on github, in the cluster-example/ directory. The application is built with Maven by executing the mvn package command. The packaged Web-archive can be found in the cluster-example/target directory.
The objective of the example-application is to demonstrate how to configure your application for clustering at different levels.
Clustering EJB Session Beans
Enterprise Java Beans (EJB) are the core components of a Java EE application. EJBs provide transaction and security management and most EJB-containers also support clustering.
Clustering a Stateless Session Bean
Clustering a stateless session bean is really easy, just annotate the bean-class with @org.jboss.ejb3.annotation.Clustered or add the corresponding tags in the JBoss-specific deployment descriptor for EJB components.
@Stateless
@Clustered
@Named
public class ClusteredStatelessBean {
private final static Logger LOG =
Logger.getLogger(ClusteredStatelessBean.class.getName());
public String getNodeName() {
LOG.info("invoke getNodeName()");
try {
String jbossNodeName =
System.getProperty("jboss.node.name");
return jbossNodeName != null ? jbossNodeName :
InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
throw new RuntimeException(e);
}
}
}
The example stateless session bean from our application has one simple method getNodeName() which returns the name of the cluster node that is processing the request.
But why cluster a stateless session bean? The simple reason is that, invocation of a stateless session bean can load balanced over the cluster nodes. This can be good for beans that do computationally intensive tasks. Thus, clustering of stateless session beans enables scalability. Because they do not encapsulate data there is no risk to loose information when a node crashes. Thus, availability is no problem.
Clustering a Stateful Session Bean
Clustering stateful session beans is more complex than clustering a stateless session bean, because the state of the components must be managed between the cluster nodes. The application server does this, but a developer should be aware of the added complexity which can reduce scalability.
Configuring the JBoss to cluster a stateful session bean is the same as for stateless session beans: Annotate the bean-class with @org.jboss.ejb3.annotation.Clustered or add the corresponding tags in the xml deployment descriptor. In the default configuration of the AS 7, a clustered stateful session bean is replicated with other cluster nodes after every call. So, if one cluster node dies, the state of session is still available. State replication of a stateful session bean means, that the bean is passivated after every invocation and than stored to the ejb cache-container. With the next invocation the bean will be activated again. By default the other nodes get the new state through the Infinispan cache asynchronously.
@Stateful
@SessionScoped
@Clustered
@Named
public class ClusteredStatefulBean {
private final static Logger LOG =
Logger.getLogger(ClusteredStatefulBean.class.getName());
private int counter;
public int getCounterValue() {
LOG.info("invoke getCounter()");
return counter++;
}
@PrePassivate
public void passivate() {
LOG.info("passivate ejb component: " + this);
}
@PostActivate
public void activate() {
LOG.info("activate ejb component: " + this);
}
}
The example stateful session bean is a CDI component associated to the HTTP session. The bean has a counter which is increased every time it is read by the getter-method getCounterValue(). With the counter you can track how often the bean was invoked during the session.
Enable HTTP Session replication
Replication of the HTTP session ensure that the sessions of the clients will be available on all cluster nodes. That means a session object is replicated to other cluster-nodes. This is internally done by storing the session objects into an Infinispan cache-container. So, if one node dies the HTTP session of the client can be continued on another node. For this, an external mechanism is required to handle failover and load balancing transparent to the client side, the browser. mod_cluster can do that out-of-the-box and we cover this in a second.
Configuring the JBoss AS to replicate the HTTP session is done by adding the <distributable/> element to the web.xml:
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
version="3.0">
<distributable/>
</web-app>
Setting up the JBoss AS 7 instances
To see our example application working, we will need to set up a cluster. We will set up a vertical scalable cluster with two JBoss AS 7 instances located on the same machine.
We need to download the EAP 6 version, because we ran into several known bugs related to the clustering capabilities of the current community release 7.1.1.Final. These bugs are already fixed in the development branch. So you could also try the latest nightly build or download the JBoss 7.1.2.Final tag from github and run the build.sh script.
After you downloaded the JBoss AS, extract the ZIP archive in two different directories, for example jb1 and jb2. For our cluster environment, we run the JBoss instances in standalone mode. This mode is similar to the previous versions of the JBoss AS. Now we can deploy our example-application. So, copy the Web-archive to both jb1/standalone/deployments/ and jb2/standalone/deployments/ directories.
Before we start the two JBoss AS instances let us have a look at the start commands:
$./jb1/bin/standalone.sh -Djboss.node.name=jb1 \
--server-config=standalone-ha.xml
$./jb2/bin/standalone.sh -Djboss.node.name=jb2 \
--server-config=standalone-ha.xml \
-Djboss.socket.binding.port-offset=100
The first parameter will set the property jboss.node.name to jb1 or respectively jb2. Usually this value is automatically set to the hostname of the machine on which the application server is running. But as we are running two JBoss instances on the same machine this will lead to a name conflict in the cluster. The second parameter --server-config tells the JBoss not to use the standard configuration file. If you look into standalone/configuration/ directory you will see a bunch of configuration files for different purposes including our standalone-ha.xml. The standalone-ha.xml contains all basic configuration for high availability.
Because two JBoss AS 7 instances run on one machine, we need to avoid port conflicts. With the third parameter from the start-command of the second JBoss AS a port-offset of 100 will be activated. The obvious influence of that option is that if for example the port for the management-console is configured to 9990 it will really be available on port 9990+100=10090.
If you want to use different IP socket bindings instead of port-offsets, make sure to use the same multicast group.
Now let us start the first node jb1 with the above command. We will see in the log file that the cluster subsystems are activated and currently one cluster member is registered:
...
20:52:59,458 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 33) JBAS010280: Activating Infinispan subsystem.
20:52:59,459 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 37) JBAS010260: Activating JGroups subsystem.
...
20:52:59,643 INFO [org.jboss.modcluster.ModClusterService] (ModClusterService lifecycle - 1) Initializing mod_cluster 1.2.1.Final-redhat-1
...
20:53:03,260 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started remote-connector-client-mappings cache from ejb container
20:53:03,260 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started default-host/cluster cache from web container
20:53:03,260 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started repl cache from web container
20:53:03,260 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started repl cache from ejb container
20:53:03,267 INFO [org.jboss.as.clustering] (MSC service thread 1-2) JBAS010238: Number of cluster members: 1
20:53:03,267 INFO [org.jboss.as.clustering] (MSC service thread 1-11) JBAS010238: Number of cluster members: 1
...
Let us start the second JBoss instance jb2. In the log file, we see similar messages but now there are two registered cluster members.
...
21:18:27,075 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 33) JBAS010280: Activating Infinispan subsystem.
21:18:27,081 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 37) JBAS010260: Activating JGroups subsystem.
21:18:27,262 INFO [org.jboss.modcluster.ModClusterService] (ModClusterService lifecycle - 1) Initializing mod_cluster 1.2.1.Final-redhat-1
21:18:27,273 INFO [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ModClusterService lifecycle - 1) Listening to proxy advertisements on 224.0.1.105:23,364
21:18:28,156 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started repl cache from web container
21:18:28,156 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started default-host/cluster cache from web container
21:18:28,157 INFO [org.jboss.as.clustering.infinispan] (CacheService lifecycle - 1) JBAS010281: Started repl cache from ejb container
21:18:28,165 INFO [org.jboss.as.clustering] (MSC service thread 1-13) JBAS010238: Number of cluster members: 2
21:18:28,165 INFO [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 2
...
If we have a second look at the log file from cluster node jb1, we see also jb2 joined successfully our cluster!
...
21:18:27,844 INFO [org.jboss.as.clustering] (Incoming-1,null) JBAS010225: New cluster view for partition ejb (id: 1, delta: 1, merge: false) : [jb1/ejb, jb2/ejb]
21:18:27,844 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,null) ISPN000094: Received new cluster view: [jb1/ejb|1] [jb1/ejb, jb/ejb]
21:18:27,849 INFO [org.jboss.as.clustering] (Incoming-3,null) JBAS010225: New cluster view for partition web (id: 1, delta: 1, merge: false) : [jb1/web, jb2/web]
21:18:27,849 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,null) ISPN000094: Received new cluster view: [jb1/web|1] [jb1/web, jb2/web]
...
Configure Apache httpd and mod_cluster
Another thing we will need is a load balancer, which can also manage failover, if a cluster node goes down. mod_cluster is such a balancer. First, you will need to install an Apache httpd server and then install and configure the mod_cluster modules.
If you have already installed a recent version of the httpd server, you need only to downlod the mod_cluster dynamic libraries bundle and copy them into the httpd modules directory. Otherwise you can also download a pre-configured version of the httpd server with mod_cluster for your environment from the mod_cluster download site.
To configure the Apache httpd server, open the main configuration file (something like /opt/jboss/httpd/httpd.conf) and add the following to the end of the httpd.conf file. Do not forget to adapt the path to the modules accordingly to your system.
LoadModule proxy_module /opt/jboss/httpd/lib/httpd/modules/mod_proxy.so
LoadModule proxy_ajp_module /opt/jboss/httpd/lib/httpd/modules/mod_proxy_ajp.so
# load the mod_cluster modules
LoadModule slotmem_module /opt/jboss/httpd/lib/httpd/modules/mod_slotmem.so
LoadModule manager_module /opt/jboss/httpd/lib/httpd/modules/mod_manager.so
LoadModule proxy_cluster_module /opt/jboss/httpd/lib/httpd/modules/mod_proxy_cluster.so
LoadModule advertise_module /opt/jboss/httpd/lib/httpd/modules/mod_advertise.so
# MOD_CLUSTER_ADDS
# Adjust to you hostname and subnet.
<IfModule manager_module>
Listen 127.0.0.1:6666
ManagerBalancerName mycluster
<VirtualHost 127.0.0.1:6666>
<Location />
Order deny,allow
Deny from all
Allow from 127.0.0
</Location>
KeepAliveTimeout 300
MaxKeepAliveRequests 0
#ServerAdvertise on http://@IP@:6666
AdvertiseFrequency 5
#AdvertiseSecurityKey secret
#AdvertiseGroup @ADVIP@:23364
EnableMCPMReceive
<Location /mod_cluster_manager>
SetHandler mod_cluster-manager
Order deny,allow
Deny from all
Allow from 127.0.0
</Location>
</VirtualHost>
</IfModule>
Now start your Apache httpd with the following command: ./opt/jboss/httpd/sbin/apachectl start
You will see one interesting output line in both log files of the cluster nodes:
...
[org.jboss.modcluster.ModClusterService] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Engine [jboss.web] will use jvmRoute: 4e6189af-0502-3305-8ff3-fad7fee8b516
...
This means that the modcluster subsystem on the JBoss side has successfully registered with mod_cluster on the Apache side. If you open http://127.0.0.1:6666/mod_cluster-manager with your browser you will see some information about the cluster nodes.
Now feel free to play with the cluster you just built. Access our example-application through http://127.0.0.1/cluster.
You see which node is processing the requests – kill that node (ctrl-c in the console window of that node) wait a second and hit the refresh button of your browser, the request should now be processed by the other node and much more interesting you neither get a new session nor a new stateful session bean. It just looks like nothing happened except that the request is handled by another cluster-node. Feel free to play around a little bit with your cluster.
Summary and Prospects
In this post we used the EAP 6 version, which is completely stable and fully usable in relation to the clustering capability. With the current community release 7.1.1.Final we ran into several known bugs. All these bugs have been fixed in the 7.1.2.Final tag. But for production environments we recommend the supported and quality-assured EAP version.
As we have shown it is really easy to configure a simple Java EE application to run in a clustered environment. In this post we did not cover message-driven-beans or the second-level cache for JPA entities, but we plan to do this in later posts.
The next post covers the management of a clustered environment with the new domain mode of the JBoss AS 7.
Feel free to contact us for any questions or feedback.
immanuel.sims (at) akquinet.de
heinz.wilming (at) akquinet.de