Configure Broker Fail-over in a WSO2 Message Broker and WSO2 ESB setup

What is Broker fail-over?

High availability (HA) is an important aspect associated with any server implementation that ensures a certain degree of operational continuity when unplanned downtime events impact parts of a system. JMS (Java Message Service) 1.1 does not define such instead most of the time they are vendor specific.

The idea behind fail-over is stopping single-point-of-failure in a system. As the broker is the middle man storing and forwarding the messages, if that server goes down entire message flowing of the system will go down no matter what other servers and functions are involved. Thus in order to make a robust messaging system it is mandatory to have a fail-over mechanism.

That is there are a few instances of Message Broker servers up and running in the system, and system generally uses one broker. If that broker goes down it automatically switch to the second broker and continue messaging. If the second fails it will try next and so on. Thus as a whole system will not have a downtime.

Fail-over scenario with WSO2 Message Broker and WSO2 ESB

Setting Up MB Instances In a Single Machine

Download MB 2.1.0 from here and extract it into a folder.
Make exact copy of MB to three places and rename them as MB1, MB2 and MB3. Those will be 3 separate MB instances.

Navigate to MB1/repository/conf/carbon.xml file and define an offset 1.

    
        <Ports>
        <!-- Ports offset. This entry will set the value of the ports defined below to
         the define value + Offset.
         e.g. Offset=2 and HTTPS port=9443 will set the effective HTTPS port to 9445
         -->
         <Offset>0</Offset>

Navigate to MB2/repository/conf/carbon.xml file and define an offset 2 as in the previous step.
Navigate to MB3/repository/conf/carbon.xml file and define an offset 3 as in the previous step.
Now our 3 MB instances are ready. Go to /bin folder of each MBx folder and start servers by typing in console wso2sever.sh (or wso2server.bat in Windows).
Above instances will start on ports 5673,5674 and 5675 respectively.

Note that in the above setup it will start 3 separate MB servers along with Cassandra servers dedicated for each. Those Cassandra servers will run in ports 9160, 9161 and 9162 respectively.

Setting Up MB Instances In different Machines

Download MB 2.1.0 from here and extract it into a folder.
Make exact copy of that folder in three machines. We will say those are MB1 in machine with IP1, MB2 in machine with IP2 and MB3 in machine with IP3.
Start all above instances by navigating to /bin folder and typing in console wso2sever.sh (or wso2server.bat in Windows). All severs will start with port offset 0, which is 5672.

Setting Up WSO2 ESB Instance

We will start ESB in default port (port offset 0). We can do that as we started all other servers with different port offsets in the singe machine setup. If you are using different machines use another machine to start WSO2 ESB.

For this you need to follow the WSO2 article on "Configuring WSO2 ESB with WSO2 Message Broker".
There is only a single difference to enable fail-over across the three brokers we have setup. That is when specifying

```
connectionfactory.ConnectionFactory
```

connectionfactory.QueueConnectionFactory

connectionfactory.TopicConnectionFactory

For an example, if you hope to use single machine MB setup described above

connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://localhost:5673 retries='5'&connectdelay='50';tcp://localhost:5674?retries='5'&connectdelay='50';tcp://localhost:5675?retries='5'&connectdelay='50''

If you hope to use the broker setup made across several machines as described above

connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?failover='roundrobin'&cyclecount='2'&brokerlist='tcp://IP1:5672 retries='5'&connectdelay='50';tcp://IP2:5672?retries='5'&connectdelay='50';tcp://IP3:5672?retries='5'&connectdelay='50''

We will look into the parameters in the connection url above.

Brokerlist option

brokerlist='<broker url>[;<broker url>]'

The broker list defines the various brokers that can be used for this connection. A minimum of one broker url is required additional URLs are semi-colon(';') delimited.

Broker URL format

<transport>://<host>[:<port>][?<option>='<value>'[&<option>='<value>']]

Option	Default	Description
retries	1	The number of times to retry connection to this Broker
ssl	false	Use ssl on the connection
connecttimeout	30000	How long in (milliseconds) to wait for the connection to succeed
connectdelay	none	How long in (milliseconds) to wait before attempting to reconnect

Brokerlist failover option

failover='<method>[?<options>]'

Method	Description
singlebroker	This will only use the first broker in the list.
roundrobin	This method tries each broker in turn.
nofailover	[New in 0.5] This method disables all retry and failover logic.

The current defaults are naturally to use the 'singlebroker' when only one broker is present and the 'roundrobin' method with multiple brokers. The '''method''' value in the URL may also be any valid class on the classpath that implements the FailoverMethod interface.

Options

Option	Default	Description
cyclecount	1	The number of times to loop through the list of available brokers before failure.

Testing setup

When sending messages (you can configure SOAP UI to send messages continuously one after the other) to a queue using setup described in above article, kill MB1. You will notice fail-over happens at ESB to the second broker i.e MB2. Logs will be there at ESB side indicating fail-over occurred.
If some previous messages were there at MB1, they will not be delivered.

Most of the time solution to the above and many other problems is broker clustering. Read "Why Broker Fail-over is not Enough? Broker Clustering With WSO2 Message Broker" to know the advantages of clustering and WSO2 Message Broker's support on clustering.

Hasitha's Tech Blessings