Beware When Clustering WSO2 Message Broker


I though of putting this post to avoid many problems that might occur due to some small thing deployers do not tend to think.

I suppose you already understand that Apache Cassandra is time stamp based. It identifies the latest record by timestamps. Which means always  newer timestamps always win.


CASSANDRA'S LAST WRITE POLICY IS BASED ON TIMESTAMP, AND MORE INTERESTINGLY, THIS IS CLIENT TIMESTAMP..!!!




This sentence is very powerful as it might be the actual cause for lot of problems in a multi-node Message Broker cluster. Say that we have a two node MB cluster and we do following

1. Say we have a queue subscription for node 1. We kill it. We tell cassandra to decrement subscription count.
2. Now we do the subscription for same queue from node 2.  This increments the subscription count.

Now you understand if mb1 node is not time synced with mb2 node, decremented value could be considered the latest record and for MB this jumbles everything.

So TIME SYNCING BETWEEN MB NODES AND CASSANDRA NODES IS A MUST.

Use following commands for time syncing machines.


You Need administrator permissions

$sudo apt-get install ntpdate;
$ntpdate pool.ntp.org

References:

1. http://technotes.tostaky.biz/2010/11/timestamps-in-cassandra.html
2. http://www.datastax.com/docs/0.8/dml/data_consistency

Hasitha Hiranya

No comments:

Post a Comment

Instagram