WSO2 Message Broker Clustering - Time Sync

WSO2 Message Broker supports clustering. This means, it is a scalable broker which supports clustering. Unlike many broker deployments provided by other vendors, it can be clustered in Active-Active way. When WSO2 MB works as a cluster it has an internal message distribution algorithm to avoid "broker-to-broker communication" and "remote get problem".

This internal algorithm needs all MB nodes to be in time sync to operate without flows. If nodes are not time sync, messages persisted to DB (message store) can remain there without being routed to subscribers. So time syncing is a MUST when setting up a message broker cluster. Also note that time sync should be rather accurate - most of the time to 10 ms range (depends on latency between nodes)

 Following article describes how to setup time syncing on linux servers using a remote ntp time server. You can use the internal ntp time server to time sync the nodes if you have a ntp server at your infrastructure.

Install ntpdate package

RedHat Enterprise Linux, CentOS, Oracle Linux:

# sudo yum install ntpdate

Debian, Ubuntu:

# sudo apt-get install ntpdate

Create a script to update time

At /home/user create a file called with the following content

sudo ntpdate >>  /tmp/timeSyncLog.log

What this command will do is sync the time of the local machine with time of time server cluster. The project is a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients. The pool is being used by millions or tens of millions of systems around the world. It's the default "time server" for most of the major Linux distributions and many networked appliances.

The logs of the time syncing will be routed to the file /tmp/timeSyncLog.log. 

Schedule a cron job on linux to time sync machine 

Every computer calculates time locally according to a clock. This clock may have slight drifting that vary from one machine to another. Along with time this difference may cause a considerable time gap between servers setup. 

To avoid this we need to run above script time to time in every server machine and make sure that they operate in time sync manner. All servers will sync to the time received by "". 

To schedule this cron job under super user type below command

$ crontab -e

A file will be open and you need to place the cron expression there. Syntax of crontab is as below

$ 1 2 3 4 5 /root/


1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])
* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)

For example, if you need to run time sync every minute (to verify cron is working), place the cron expression as below

* * * * * sh /home/user/

For run once a hour place it as

0 * * * * sh /home/user/

Make sure cron runs 

Schedule it to run every one minute as above, and tail the /tmp/timeSyncLog.log file. You will see entries as below. 

30 Oct 12:32:07 ntpdate[17848]: adjust time server offset -0.045923 sec
30 Oct 12:33:09 ntpdate[17894]: adjust time server offset 0.060208 sec
30 Oct 12:34:08 ntpdate[17909]: adjust time server offset 0.033788 sec
30 Oct 12:35:08 ntpdate[17926]: adjust time server offset 0.002323 sec
30 Oct 12:36:07 ntpdate[17939]: adjust time server offset -0.075001 sec

After that make it run once every 3 hours or so. It will make sure all nodes will be in sync all the time. 

To future

 In next versions of Message Broker releases we plan to get rid of this time sync requirement using a different generation mechanism

Hasitha Hiranya

No comments:

Post a Comment