Increasing Message Broker Performance & Tunining

WSO2 Message Broker  is a open source product developed by WSO2 to achieve JMS and MQTT messaging. WSO2 MB can be scaled vertically as well as horizontally until you get the performance you need. Nevertheless, this performance is limited by nature.

In a typical WSO2 MB cluster all broker nodes are pointed to a single database server. Thus, it is understandable that, performance of the whole message broker cluster is upper bounded by the performance of the Database server. Unfortunately, we do not have the scalability factor when come to the database. After message writing/reading/deleting comes to a threshold, the limit of the database performance is reached. Then the performance get divided between the subscribers and queues.

Having said that, let us discuss what the the ways to improve performance of a Message Broker cluster so that it get maximum of the provided resources.

Increase DB Performance

It is apparent that increasing the database performance has a direct impact to the performance of the whole MB cluster. Following steps can be taken to increase database performance.

  1. See if mySQL database get loaded (load average). If machine is loaded that means DB is tuned. Otherwise you need to tune the database. 
  2. Deploy mySQL on a SSD storage based server. Follow best deployment practices to get maximum performance of mySQL (consult DB admin).

Broker Level Performance

Apart from that, there is a few points where we can achive some performance at broker level.
  1.  if message size is around 1KB or larger, enable message compression. This reduces the amount of data being transferred in the wire (broker -->> DB), hence some performance can be achieved. Please refer here for detail.
  2. has "flow control" get triggered, when publishing messages? This can be inspected using following logs at server side
    • "Global buffer based flow control enabled"
    •  "Flow control enabled for channel [ ID:"
    If these can be seen in logs, that means server blocks client sending messages time-to-time. If DB server performs faster, this frequency of flow control triggering will get automatically lower. Or else you can try to increase flow control thresholds and push burden to the server side. Please refer here for flow control tunings. 
  3. Increase parallel writer threads. This makes message publishing faster. Edit [MB_HOME]/repository/conf/broker.xml file to reflect value under following tag (default=1) [parallelMessageWriters]
  4. Try increasing and decreasing message slot sizes. This controls how many messages are included to a Slot in message broker. Depending on that, how many thrift calls done from node to coordinator node, how many messages are received from DB as a chunk etc will be decided. You need to find the balance.
    Edit [MB_HOME]/repository/conf/broker.xml file to reflect value under following tag (default=1000) [windowSize]
  5. Try enabling message caching. When this is enabled message is persisted, but kept in memory as well. When delivering back message is read from memory saving DB reads. To enable this Edit [MB_HOME]/repository/conf/broker.xml to reflect value under following tag [cache] under [persistence]. Note that this is a local cache only. You need to read message from same node you published to. Then only this feature is effective. Refer here for more details on broker level tuning parameters

Deployment Level Performance

Apart from above there are deployment level tuning that can be done to smooth out message distribution

  1. Horizontally scale with more MB nodes if you have many queues and resources of individual nodes seems not sufficient (i.e high load average)
  2. If there are identified queues having a heavy usage, try localizing them to a single node. Make subscribers to nodes so that a particular node is dedicated for a queue. 
  3. Use a strong network (specially between MB nodes and the database server)
  4. First node in the cluster automatically becomes coordinator. Try to avoid traffic to this node and let it do the node coordination only.

Hasitha Hiranya

No comments:

Post a Comment