WSO2 Message Broker supports clustering. This means, it is a scalable broker which supports clustering. Unlike many broker deployments provided by other vendors, it can be clustered in Active-Active way. When WSO2 MB works as a cluster it has an internal message distribution algorithm to avoid "broker-to-broker communication" and "remote get problem".
This internal algorithm needs all MB nodes to be in time sync to operate without flows. If nodes are not time sync, messages persisted to DB (message store) can remain there without being routed to subscribers. So time syncing is a MUST when setting up a message broker cluster. Also note that time sync should be rather accurate - most of the time to 10 ms range (depends on latency between nodes)
Following article describes how to setup time syncing on linux servers using a remote ntp time server. You can use the internal ntp time server to time sync the nodes if you have a ntp server at your infrastructure.
# sudo yum install ntpdate
Debian, Ubuntu:
# sudo apt-get install ntpdate
What this command will do is sync the time of the local machine with time of pool.ntp.org time server cluster. The pool.ntp.org project is a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients. The pool is being used by millions or tens of millions of systems around the world. It's the default "time server" for most of the major Linux distributions and many networked appliances.
After that make it run once every 3 hours or so. It will make sure all nodes will be in sync all the time.
This internal algorithm needs all MB nodes to be in time sync to operate without flows. If nodes are not time sync, messages persisted to DB (message store) can remain there without being routed to subscribers. So time syncing is a MUST when setting up a message broker cluster. Also note that time sync should be rather accurate - most of the time to 10 ms range (depends on latency between nodes)
Following article describes how to setup time syncing on linux servers using a remote ntp time server. You can use the internal ntp time server to time sync the nodes if you have a ntp server at your infrastructure.
Install ntpdate package
RedHat Enterprise Linux, CentOS, Oracle Linux:# sudo yum install ntpdate
Debian, Ubuntu:
# sudo apt-get install ntpdate
Create a script to update time
At /home/user create a file called timesync.sh with the following content
sudo ntpdate pool.ntp.org >> /tmp/timeSyncLog.log
What this command will do is sync the time of the local machine with time of pool.ntp.org time server cluster. The pool.ntp.org project is a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients. The pool is being used by millions or tens of millions of systems around the world. It's the default "time server" for most of the major Linux distributions and many networked appliances.
The logs of the time syncing will be routed to the file /tmp/timeSyncLog.log.
Schedule a cron job on linux to time sync machine
Every computer calculates time locally according to a clock. This clock may have slight drifting that vary from one machine to another. Along with time this difference may cause a considerable time gap between servers setup.
To avoid this we need to run above script time to time in every server machine and make sure that they operate in time sync manner. All servers will sync to the time received by "pool.ntp.org".
To schedule this cron job under super user type below command
$ crontab -e
A file will be open and you need to place the cron expression there. Syntax of crontab is as below
$ 1 2 3 4 5 /root/backup.sh
where
1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])
* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)
For example, if you need to run time sync every minute (to verify cron is working), place the cron expression as below
* * * * * sh /home/user/timesync.sh
For run once a hour place it as
0 * * * * sh /home/user/timesync.sh
$ crontab -e
A file will be open and you need to place the cron expression there. Syntax of crontab is as below
$ 1 2 3 4 5 /root/backup.sh
where
1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])
* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)
For example, if you need to run time sync every minute (to verify cron is working), place the cron expression as below
* * * * * sh /home/user/timesync.sh
For run once a hour place it as
0 * * * * sh /home/user/timesync.sh
Make sure cron runs
Schedule it to run every one minute as above, and tail the /tmp/timeSyncLog.log file. You will see entries as below.
30 Oct 12:32:07 ntpdate[17848]: adjust time server 192.248.1.162 offset -0.045923 sec
30 Oct 12:33:09 ntpdate[17894]: adjust time server 46.101.52.119 offset 0.060208 sec
30 Oct 12:34:08 ntpdate[17909]: adjust time server 129.250.35.251 offset 0.033788 sec
30 Oct 12:35:08 ntpdate[17926]: adjust time server 129.250.35.251 offset 0.002323 sec
30 Oct 12:36:07 ntpdate[17939]: adjust time server 192.248.1.162 offset -0.075001 sec
After that make it run once every 3 hours or so. It will make sure all nodes will be in sync all the time.
To future
In next versions of Message Broker releases we plan to get rid of this time sync requirement using a different generation mechanism
