Failover Deployments

Karaf provides failover capability using either a simple lock file system or a JDBC locking mechanism. In both cases, a container-level lock system allows bundles to be preloaded into the slave Karaf instance in order to provide faster failover performance.

Simple lock file

The simple lock file mechanism is intended for failover configurations where instances reside on the same host machine.

To use this feature, edit the $KARAF_HOME/etc/system.properties file as follows on each system in the master/slave setup:

karaf.lock=true
karaf.lock.class=org.apache.karaf.main.SimpleFileLock
karaf.lock.dir=<PathToLockFileDirectory>
karaf.lock.delay=10

Note: Ensure that the karaf.lock.dir property points to the same directory for both the master and slave instance, so that the slave can acquire the lock only when the master releases it.

JDBC locking

The JDBC locking mechanism is intended for failover configurations where instances exist on separate machines. In this deployment, the master instance holds a lock on a Karaf locking table hosted on a database. If the master loses the lock, a waiting slave process gains access to the locking table and fully starts its container.

To use this feature, do the following on each system in the master/slave setup:

  • Update the classpath to include the JDBC driver

  • Update the $KARAF_HOME/bin/karaf script to have a unique JMX remote port set if instances reside on the same host

  • Update the $KARAF_HOME/etc/system.properties file as follows:

karaf.lock=true
karaf.lock.class=org.apache.karaf.main.DefaultJDBCLock
karaf.lock.level=50
karaf.lock.delay=10
karaf.lock.jdbc.url=jdbc:derby://dbserver:1527/sample
karaf.lock.jdbc.driver=org.apache.derby.jdbc.ClientDriver
karaf.lock.jdbc.user=user
karaf.lock.jdbc.password=password
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=karaf
karaf.lock.jdbc.timeout=30

Note:

  • This process will fail if a JDBC driver is not on the classpath.

  • The "sample" database referred to above will be created if it does not exist.

  • The first Karaf instance to acquire the locking table is the master instance.

  • If the connection to the database is lost, the master instance tries to gracefully shutdown, allowing a slave instance to become master when the database service is restored. The former master will require a manual restart.

JDBC locking on Oracle

If you are using Oracle as your database for JDBC locking, the karaf.lock.class property in the $KARAF_HOME/etc/system.properties file must point to org.apache.karaf.main.OracleJDBCLock.

Otherwise, configure the system.properties file as normal for your setup, for example:

karaf.lock=true
karaf.lock.class=org.apache.karaf.main.OracleJDBCLock
karaf.lock.jdbc.url=jdbc:oracle:thin:@hostname:1521:XE
karaf.lock.jdbc.driver=oracle.jdbc.OracleDriver
karaf.lock.jdbc.user=user
karaf.lock.jdbc.password=password
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=karaf
karaf.lock.jdbc.timeout=30

As with the default JDBC locking setup, the Oracle JDBC driver JAR file must be in your classpath. You can ensure this by copying the ojdbc14.jar into Karaf's lib folder before starting Karaf.

Note: The karaf.lock.jdbc.url requires an active SID, which means you must manually create a database instance before using this particular lock.

Derby

The same rules apply when using derby. Make sure you have the driver jar file in the Karaf lib folder before starting Karaf.

Then make you update the properties in $KARAF_HOME/etc/system.properties to look something like this example:

karaf.lock=true
karaf.lock.class=org.apache.karaf.main.DerbyJDBCLock
karaf.lock.jdbc.url=jdbc:derby://127.0.0.1:1527/dbname
karaf.lock.jdbc.driver=org.apache.derby.jdbc.ClientDriver
karaf.lock.jdbc.user=user
karaf.lock.jdbc.password=password
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=karaf
karaf.lock.jdbc.timeout=30

MySQL

Make sure you have the MySQL driver jar file in the Karaf lib folder before starting Karaf.

NOTE: for 2.2.x, 2.3.x you need to rename the MySQL Driver jar to prefix with 'karaf-' in order for karaf to pick it up, otherwise you will see karaf just hang on startup and the log will show you that it could not find the driver.

Then make you update the properties in $KARAF_HOME/etc/system.properties to look something like this example:

karaf.lock=true
karaf.lock.class=org.apache.karaf.main.MySQLJDBCLock
karaf.lock.jdbc.url=jdbc:mysql://127.0.0.1:3306/dbname
karaf.lock.jdbc.driver=com.mysql.jdbc.Driver
karaf.lock.jdbc.user=user
karaf.lock.jdbc.password=password
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=karaf
karaf.lock.jdbc.timeout=30

PostgreSQL

Make sure you have the PostgreSQL driver jar file in the Karaf lib folder before starting Karaf.

NOTE: for 2.2.x, 2.3.x you need to rename the PostgreSQL Driver jar to prefix with 'karaf-' in order for karaf to pick it up, otherwise you will see karaf just hang on startup and the log will show you that it could not find the driver.

Then make you update the properties in $KARAF_HOME/etc/system.properties to look something like this example:

karaf.lock=true
karaf.lock.class=org.apache.karaf.main.PostgreSQLJDBCLock
karaf.lock.jdbc.url=jdbc:postgresql://127.0.0.1:5432/dbname
karaf.lock.jdbc.driver=org.postgresql.Driver
karaf.lock.jdbc.user=user
karaf.lock.jdbc.password=password
karaf.lock.jdbc.table=KARAF_LOCK
karaf.lock.jdbc.clustername=karaf
karaf.lock.jdbc.timeout=0

Container-level locking

Container-level locking allows bundles to be preloaded into the slave kernel instance in order to provide faster failover performance. Container-level locking is supported in both the simple file and JDBC locking mechanisms.

To implement container-level locking, add the following to the $KARAF_HOME/etc/system.properties file on each system in the master/slave setup:

karaf.lock=true
karaf.lock.level=50
karaf.lock.delay=10

The karaf.lock.level property tells the Karaf instance how far into the boot process to bring the OSGi container. All bundles with an ID equal or lower to this start level will be started in that Karaf instance.

Bundle start levels are specified in $KARAF_HOME/etc/startup.properties, in the format jar.name=level. The core system bundles have levels below 50, where user bundles have levels greater than 50.

Level Behavior
1 A 'cold' standby instance. Core bundles are not loaded into container. Slaves will wait until lock acquired to start server.
<50 A 'hot' standby instance. Core bundles are loaded into the container. Slaves will wait until lock acquired to start user level bundles. The console will be accessible for each slave instance at this level.
>50 This setting is not recommended as user bundles will end up being started.

Note: When using a 'hot' spare on the same host you need to set the JMX remote port to a unique value to avoid bind conflicts. You can edit the Karaf start script to include the following:

DEFAULT_JAVA_OPTS="-server $DEFAULT_JAVA_OPTS -Dcom.sun.management.jmxremote.port=1100 -Dcom.sun.management.jmxremote.authenticate=false"