Jboss Classloader Isolation Problem

Introduction:

Default behavior of Jboss application server is to let all deployed applications and web applications, free access to all classes/jars of other applications in the same deploy directory. This is in addition to the libraries/jars present in the <server instance>/lib directory. Actually at runtime the libraries in the <server instance>/lib take precedence over the jars defined in the ear application and the web application. The immediate consequence of this is that for some jar abc.jar present in your ear/war as well as in /lib, the <server instance>/lib takes precedence over your application jar file. All this happens transparently and you cannot see much about this in log files, but sometimes I have found DEBUG level logging in Jboss itself help. The actual pain comes in when your jar file (abc.jar which is present in <server instance>/lib also) depends on another jar (xyz.jar) which is present only in your ear/war and not in the <server instance>/lib. Standard Java enterprise edition rules don’t allow parent application server class loaders to access classes inside apps. This means that the class loader which is trying to load a class ABC.class present in abc.jar which is lying in /lib directory cannot load the class XYZ.class present in xyz.jar lying inside your ear/war, unless the class loader is itself an ear application class loader or a web application class loader bound to your application ear/war. I know this sounds very confusing, so let’s have a quick look at the basics of the Jboss ClassLoader hierarchy, straight out of Jboss wiki/manual, to understand how class loader delegation happens.

1. Brief Outline of Jboss ClassLoader hierarchy:

1.1. The core JBoss class loading components

The central component is the org.jboss.mx.loading.UnifiedClassLoader3 (UCL) class loader. This is an extension of the standard java.net.URLClassLoader that overrides the standard parent delegation model to use a shared repository of classes and resources. This shared repository can be understood as being a classpath for that specific UCL. This shared repository is the org.jboss.mx.loading.UnifiedLoaderRepository3. Every UCL is associated with a single UnifiedLoaderRepository3, and a UnifiedLoaderRepository3 typically has many UCLs. AThis is important to understand as this means that many UCL instances can share the same class path as provided by the associated UnifiedLoaderRepository3. A UCL may have multiple URLs associated with it for class and resource loading. Deployers use the top-level deployment’s UCL as a shared class loader and all deployment archives are assigned to this class loader.

When a UCL is asked to load a class, it first looks to the repository cache it is associated with to see if the class has already been loaded. Only if the class does not exist in the repository will it be loaded into the repository by the UCL. By default, there is a single UnifiedLoaderRepository3 shared across all UCL instances. This means the UCLs form a single flat class loader namespace. The complete sequence of steps that occur when a UnfiedClassLoader3.loadClass(String, boolean) method is called is:

a). Check the UnifiedLoaderRepository3 classes cache associated with the UnifiedClassLoader3. If the class is found in the cache it is returned.

b). Else, ask the UnfiedClassLoader3 if it can load the class. This is essentially a call to the super class’s URLClassLoader.loadClass(String, boolean) method to see if the class is among the URLs associated with the class loader, or visible to the parent class loader. If the class is found it is placed into the repository class’s cache and returned.

c). Else, the repository is queried for all UCLs that are capable of providing the class based on the repository package name to UCL map. When a UCL is added to a repository an association between the package names available in the URLs associated with the UCL is made, and a mapping from package names to the UCLs with classes in the package is updated. This allows for a quick determination of which UCLs are capable of loading the class. The UCLs are then queried for the requested class in the order in which the UCLs were added to the repository. If a UCL is found that can load the class it is returned, else a java.lang.ClassNotFoundException is thrown.

d). Sometimes there can be Loading constraints validate type expectations in the context of class loader scopes to ensure that a class X is consistently the same class when multiple class loaders are involved. This is important because Java allows for user defined class loaders. Linkage errors are essentially an extension of the class cast exception that is enforced by the VM when classes are loaded and used.

1.2. The Complete Class Loading Model

clip_image002

The following points apply to this figure:

a) System ClassLoaders: The System ClassLoaders node refers to either the thread context class loader (TCL) of the VM main thread or of the thread of the application that is loading the JBoss server if it is embedded.

b) ServerLoader: The ServerLoader node refers to the a URLClassLoader that delegates to the System ClassLoaders and contains the following boot URLs:

  • All URLs referenced via the jboss.boot.library.list system property. These are path specifications relative to the libraryURL defined by the jboss.lib.url property. If there is no jboss.lib.url property specified, it defaults to jboss.home.url + /lib/. If there is no jboss.boot.library property specified, it defaults to jaxp.jar, log4j-boot.jar, jboss-common.jar, and jboss-system.jar.
  • The JAXP JAR which is either crimson.jar or xerces.jar depending on the -j option to the Main entry point. The default is crimson.jar.
  • The JBoss JMX jar and GNU regex jar, jboss-jmx.jar and gnu-regexp.jar.
  • Oswego concurrency classes JAR, concurrent.jar
  • Any JARs specified as libraries via -L command line options
  • Any other JARs or directories specified via -C command line options

c) Server: The Server node represents a collection of UCLs created by the org.jboss.system.server. Server interface implementation. The default implementation creates UCLs for the patchDir entries as well as the server conf directory. The last UCL created is set as the JBoss main thread context class loader. This will be combined into a single UCL now that multiple URLs per UCL are supported.

d) All UnifiedClassLoader3s: The All UnifiedClassLoader3 node represents the UCLs created by deployers. This covers EARs, jars, WARs, SARs and directories seen by the deployment scanner as well as JARs referenced by their manifests and any nested deployment units they may contain. This is a flat namespace and there should not be multiple instances of a class in different deployment JARs. If there are, only the first loaded will be used and the results may not be as expected. Use this mechanism if you need to deploy multiple versions of a class in a given JBoss server.

e) EJB DynClassLoader: The EJB DynClassLoader node is a subclass of URLClassLoader that is used to provide RMI dynamic class loading via the simple HTTP WebService. It specifies an empty URL[] and delegates to the TCL as its parent class loader. If the WebService is configured to allow system level classes to be loaded, all classes in the UnifiedLoaderRepository3 as well as the system classpath are available via HTTP.

f) EJB ENCLoader: The EJB ENCLoader node is a URLClassLoader that exists only to provide a unique context for an EJB deployment’s java:comp JNDI context. It specifies an empty URL[] and delegates to the EJB DynClassLoader as its parent class loader.

g) Web ENCLoader: The Web ENCLoader node is a URLClassLoader that exists only to provide a unique context for a web deployment’s java:comp JNDI context. It specifies an empty URL[] and delegates to the TCL as its parent class loader.

h) WAR Loader: The WAR Loader is a servlet container specific classloader that delegates to the Web ENCLoader as its parent class loader. The default behavior is to load from its parent class loader and then the WAR WEB-INF classes and lib directories. If the servlet 2.3 class loading model is enabled it will first load from the WEB-INF directories and then the parent class loader.

So, now that we know a bit about how class loaders are designed around in Jboss, let’s see how we can fix the problems we face.

2. Configuring Classloader repository isolation:

The obvious solution to the problem is to segregate the class loader repository from the main application server libs/jars. This means that we make the associated UCL load our application jars/libraries first then the one present in <server instance>/lib.

If our application needs to have its own version of a library, we will have to configure classloader repository scoping through the xml file. There are two levels of scoping,

  • Isolation from other deployments, and
  • Isolation that overrides the loading of JBoss server classes.

There are two ways to fix this problem:

  • Global Isolation
  • Application Level Isolation

2.1. Global Isolation

Using Global class loader scope isolation we make all the ears deployed on the respective JBoss Server instance isolated from the in VM call optimization. This essentially means that all applications will be allowed to have their own scoped class loaders to isolate their classes from other deployments and server libs.

To enable global isolation, edit the ear-deployer.xml file present in the <server instance>/deploy directory. This file looks something like the following:

<server>
  <mbean code="org.jboss.deployment.EARDeployer"
                 name="jboss.j2ee:service=EARDeployer">
    <attribute name="Isolated">false</attribute>      
    <attribute name="CallByValue">false</attribute>
    <attribute name="EnablelibDirectoryByDefault">true</attribute>
   </mbean>
</server>

Change “Isolated” attribute value from “false” to “true”, and restart your application server. Now all your applications deployed will have their own scoped classloaders containing only the jars from <server instance>/lib and the application itself.

2.2. Application Level Isolation

Another more powerful approach is to have per application level control over deployments, instead of a global resolution. In the jboss deployment descriptor files, it is possible to specify which classloader to use. The file is not the same for all archives but varies in name as shown in the table below:

Archive Type

File Name and Location

Root Element

*-ejb.jar

<ejb-jar>/META-INF/jboss.xml

<jboss>

*.ear

<ear>/META-INF/jboss-app.xml

<jboss-app>

*.war

<war>/WEB-INF/jboss-web.xml

<jboss-web>

You just need to add an element <loader-repository> with the name of the classloader repository to use to allows the server bootstrap to create a scoped classloader repository namespace for your deployed application. With nested modules, only the top level file may specify class loader scoping. So in our ear file containing wars, only scoping specified in the ear’s META-INF/jboss-app.xml is used and sufficient.

The implications of this approach are that, all application referring to the same ‘loader-repository’ will share the same context classloader. And the will access to the classes of each other. But most important, this classloader will be isolated from all other application, even if the ‘Isolated‘ parameter seen in ear-deployer.xml is set to false.

This solves the “Isolation from other deployments” problem.

Another problem still remains which I have highlighted in the Introduction section above, about the same jar files being located in your application as well as in the <server instance>/lib directory. This is technically alled as “isolation with Overriding Server Classes” problem.

To solve this problem we need to disable we need to add the following to our jboss deployment descriptor files:

<loader-repository-config>
     java2ParentDelegation=false 
</loader-repository-config>

This fixes all our problems comfortably.

2.2.1. Some Examples

1. For ejb3 archives (-ejb.jar): add in META-INF/jboss.xml

<?xml version="1.0" encoding="UTF-8" ?>
<jboss>
    <loader-repository>
        com.myapplication.name:loader=SomeUniqueLoaderName
       <loader-repository-config>
            java2ParentDelegation=false
       </loader-repository-config>
    </loader-repository>
</jboss>

2. For ear archives: add in META-INF/jboss-app.xml

<?xml version="1.0" encoding="UTF-8" ?>
<jboss-app>
    <loader-repository>
        com.myapplication.name:loader=SomeUniqueLoaderName
       <loader-repository-config>
            java2ParentDelegation=false
       </loader-repository-config>
    </loader-repository>
</jboss-app>

3. For war archives: add in WEB-INF/jboss-web.xml

<?xml version="1.0" encoding="UTF-8" ?>
<jboss-web>
    <loader-repository>
        com.myapplication.name:loader=SomeUniqueLoaderName
       <loader-repository-config>
            java2ParentDelegation=false
       </loader-repository-config>
    </loader-repository>
</jboss-web>

3. Some of the common libraries that will give you this pain are:

activation.jar, antlr.jar, cglib.jar, dom4j.jar, commons-codec.jar, commons-collections.jar, commons-logging.jar, hibernate3.jar, hibernate-entitymanager.jar, javassist.jar, log4j.jar, mail.jar, quartz.jar

Advertisements
%d bloggers like this: