Hasitha's Tech Blessings

Came across this good resource on fit falls in distributed computing. It is extremely important that architects understand them before designing a software that runs across nodes, cumulatively achieving the defined work.



Following are the pit-falls discussed

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous. 

WSO2 Message Broker  is a open source product developed by WSO2 to achieve JMS and MQTT messaging. WSO2 MB can be scaled vertically as well as horizontally until you get the performance you need. Nevertheless, this performance is limited by nature.




In a typical WSO2 MB cluster all broker nodes are pointed to a single database server. Thus, it is understandable that, performance of the whole message broker cluster is upper bounded by the performance of the Database server. Unfortunately, we do not have the scalability factor when come to the database. After message writing/reading/deleting comes to a threshold, the limit of the database performance is reached. Then the performance get divided between the subscribers and queues.

Having said that, let us discuss what the the ways to improve performance of a Message Broker cluster so that it get maximum of the provided resources.

Increase DB Performance

Deployment Level Performance

When WSO2 ESB and WSO2 MB integrated together, we need to specify the connection  url at ESB side, so that it can connect to Message Broker. For that we use following configuration file.

[ESB_HOME]/repository/conf/jndi.properties

Issue here is this connection url contains credentials to connect to broker. Hence, the server admins would not like to specify it pain text. To get over with this concern and encrypt it, we can use encryption.



We cannot use cipertool to automate encryption process for the selected elements in the jndi.properties file, because we can only specify Xpath notation here, but still we can use the manual process.

# register some connection factories
# connectionfactory.[jndiname] = [ConnectionURL]
connectionfactory.QueueConnectionFactory = amqp://admin:admin@clientID/carbon?brokerlist
='tcp://localhost:5672'

# register some queues in JNDI using the form
# queue.[jndiName] = [physicalName]
queue.MyQueue = example.MyQueue

# register some topics in JNDI using the form
# topic.[jndiName] = [physicalName]
topic.MyTopic = example.MyTopic

1. Go to the [ESB_home]/bin and execute the following command to generate the encrypted value for the clear text  password.

   sh ciphertool.sh
   

2. It will prompt following  console for input value.  Answer: wso2carbon

   [Please Enter Primary KeyStore Password of Carbon Server : ]

3. Then it will appear second console for following input value. (Answer: According to our property file, the plain text is "amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5672'".) 
   

   Encryption is done Successfully
   Encrypted value is :cpw74SGeBNgAVpryqj5/xshSyW5BDW9d1UW0xMZDxVeoa6xS6CFtU
   +ESoR9jRjyA1JRHutZ4SfzfSgSzy2GQJ/2jQIw70IeT5EQEAR8XLGaqlsE5IlNoe9dhyLiPXEPRGq4k/BgU
   QDYiBg0nU7wRsR8YXrvf+ak8ulX2yGv0Sf8=
   

4. Open the cipher-text.properties file, which is under [ESB_home]/repository/conf/security and add the following entry.

   connectionfactory.QueueConnectionFactory=cpw74SGeBNgAVpryqj5/xshSyW5BDW9d1UW0xMZ
   DxVeoa6RjyA1JRHutZ4SfzfSgSzy2GQJ/2jQIw70IeT5EQEAR8XLGaqlsE5IlNoe9dhyLiPXEPRGq4k/BgUQD
   YiBg0nU7wRsR8YXrvf+ak8ulX2yGv0Sf8=
   

5. Open the [ESB_home]/repository/conf/jndi.properties file and update the key/value of connectionfactory field.

   connectionfactory.QueueConnectionFactory=secretAlias:connectionfactory.QueueConnectionFactory
   

In Apache Tomcat or WSO2 Application Server  you can configure a data source to connect to a database. When you consider WSO2 Middleware, any server is connected to a database using a datasource.

Sometimes the database instances has to be shut down for maintenance. Consider a situation where in a large deployment of WSO2 servers, database server is restarted. When database server is back, when servers connect back to the server you might see following error log

Caused by: java.sql.SQLRecoverableException: Closed Connection
        at oracle.jdbc.driver.PhysicalConnection.setAutoCommit(PhysicalConnection.java:3763)
        at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126)
        at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)
        at org.wso2.carbon.ndatasource.rdbms.ConnectionRollbackOnReturnInterceptor.invoke(ConnectionRollbackOnReturnInterceptor.java:51)
        at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)
        at org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:71)
        at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)
        at org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.invoke(ConnectionState.java:153)
        at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)
        at org.apache.tomcat.jdbc.pool.TrapException.invoke(TrapException.java:41)
        at org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109)
        at org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:80)
        at com.sun.proxy.$Proxy14.setAutoCommit(Unknown Source)
        at org.wso2.carbon.identity.core.persistence.JDBCPersistenceManager.getDBConnection(JDBCPersistenceManager.java:137)

        ... 51 more

1. Test On Borrow - (boolean) Used to indicate if objects will be validated before borrowed from the pool. If the object fails to validate, it will be dropped from the pool, and we will attempt to borrow another. NOTE - for a true value to have any effect, the validationQuery parameter must be set to a non-null string. In order to have a more efficient validation, see validationInterval. Default value is false.
2. Validation Query - (String) The SQL query used to validate connections from this pool before returning them to the caller. If specified, this query does not have to return any data, it just can't throw a SQLException. The default value is null. Example values are SELECT 1(mysql), select 1 from dual(oracle), SELECT 1(MS Sql Server).
3. Validation Interval - (long) avoid excess validation, only run validation at most at this frequency - time in milliseconds. If a connection is due for validation, but has been validated previously within this interval, it will not be validated again. The default value is 30000 (30 seconds).

<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1 FROM DUAL</validationQuery>
<validationInterval>30000</validationInterval>