Let's assume you are using a topic to send messages. There are several components subscribed to this topic that are processing the messages in different fashion (for example to process jobs A and B).
When a message M is sent to the topic, all consumers subscribed to the topic receive it:
The consumer CB1 is subscribed to the topic. Every time, it receives a message, it processes the job B. Suppose, unfortunately, this job is very time-consuming and CPU and memory sensitive so that we would like to scale it out by having several consumers for the job B running on different machines.
We CAN NOT have several consumers on the topic for this job, otherwise they would all receive copies of the same message and process it several times. What we need is a set of consumers that receives a single message from the topic amongst them.
Thus it is understandable that this solution is not a very sound one.
When this configuration is enabled, WSO2 MB allows shared topic subscriptions as depicted in JMS 2.0 and when disabled it adheres to JMS 1.1.
Achieving shared subscription behavior is tricky when come to distributed brokering. WSO2 MB is a broker that works in active-active mode in a distributed manner. Thus this feature is carefully implemented considering all possibilities
With WSO2 MB's shared topic subscription feature enabled, following behavior is supported.
Create a three node MB 3.x.x cluster following the documentation here. Use Apache Jmeter to create four durable topic subscriptions to topic "myTopic" using subscription ID "mySub1" as below. Publish to topic "myTopic" by all 3 nodes in cluster
Our expectation of the test
(x1 + x2 + x3 ) = (y1 + y2 + y3 + y4)
Publisher
Subscriber
Log into Management Console of MB and navigate to main >> Manage >> Subscriptions >> Topic subscription list. You will see all four subscriptions listed there. Note the owned node. We use customer defined node ID (using broker.xml) or if not configured [localMememberHost: localMememberPort] combination to indicate the node subscription is actually created.
Please note following.
Now close all subscriptions made to broker cluster. Now if you navigate to above page, you will see a single entry for durable topic subscription of that subscription ID indicating that there is an inactive subscription with pending messages to deliver.
If you change the subscribed nodes in the next time you subscribe (you can even bring in a new MB node MB4 and subscribe), no matter, they will be delivered to all subscriptions appropriately. If you sum up messages you sent to the cluster and you received at the end, they will be exactly equal.
If you now unsubscribe all messages will be gone.
When using shared subscriptions, until all subscriptions are inactive, you cannot unsubscribe. Basically WSO2 MB does not treat subscription is inactive until all subscriptions by that subscription ID is closed in whole MB cluster.
WSO2 MB does not suffer from "remote get problem" like many other brokers. We have a distributed algorithm that share message chunks across cluster nodes without causing any loss or duplication. This exercise alone shows how powerful and scalable WSO2 Message Broker is.
WSO2 ESB can work as publisher and consumer in a real world use case.
When a message M is sent to the topic, all consumers subscribed to the topic receive it:
The consumer CB1 is subscribed to the topic. Every time, it receives a message, it processes the job B. Suppose, unfortunately, this job is very time-consuming and CPU and memory sensitive so that we would like to scale it out by having several consumers for the job B running on different machines.
We CAN NOT have several consumers on the topic for this job, otherwise they would all receive copies of the same message and process it several times. What we need is a set of consumers that receives a single message from the topic amongst them.
External Solution
As a work around to this problem what we can do is, introduce a queue.
Whenever consumer CB1 receives a message, it routes the message to a queue in same Message Broker. Let us name that queue as CB1 queue. Now, you can point multiple subscribers to that queue. Then messages received through CB1 will be shared by multiple queue subscribers. We use the queue behavior in this workaround as queues only send a particular message to a single subscriber no matter how many subscribers consuming the queue. We can assume messages will be received by multiple subscribers in round-robin fashion.
Here note that an external application (i.e JMS), should listen to the topic messages and route it into the queue.
Even if this solution looks promising, it has its own drawbacks.
- There is an additional hop to receive for messages from topic and route it to queue.
- Owing to above additional hop, performance will go down.
- External application is a single point of failure to the system. If it malfunctioned or went down whole message processing is blocked.
- External application cannot be scaled. All messages MUST flow through that. It throttles the message flow even if we have multiple consumers for the queue.
- Reliably transfer message from topic to queue. We need to do it in transactional manner with guarantee no message will be lost without placing on the queue.
Solution provided by JMS 2.0 - shared subscriptions
With JMS 2.0 Oracle has introduced "shared subscription" concept.
A non-durable shared subscription is used by a client which needs to be able to share the work of receiving messages from a topic subscription amongst multiple consumers. A non-durable shared subscription may therefore have more than one consumer. Each message from the subscription will be delivered to only one of the consumers on that subscription.
This also applies to durable shared topic subscriptions. According to JMS 1.1 specification , only ONE subscriber can exist on the broker by same client ID. If we try to register a second subscriber with same "subscription ID", there will an exception and connection will not be allowed. But, as you can see, this behavior prevents subscriber scaling.
With JMS 2.0 following changes to the above behavior is expected.
- Attempts by multiple connections to use the same client id do not result in an exception, provided that the connections are from different instances in the cluster.
- Two or more subscriptions on the same topic with the same client id and (if the subscription is durable) the same durable subscription name are considered "shared"; that is, they are treated as a single subscription, with each message being sent to only one of the participating subscriptions.
WSO2 Message Broker - 3.x.x - shared subscription feature
WSO2 MB does not support JMS 2.0 yet. Instead, we enable this feature by a configuration.
At [MB_HOME]/repository/conf/broker.xml note the following configuration
<broker> <transports> <amqp enabled="true"> <allowSharedTopicSubscriptions>true</allowSharedTopicSubscriptions> </amqp> </transports> </broker>
When this configuration is enabled, WSO2 MB allows shared topic subscriptions as depicted in JMS 2.0 and when disabled it adheres to JMS 1.1.
Achieving shared subscription behavior is tricky when come to distributed brokering. WSO2 MB is a broker that works in active-active mode in a distributed manner. Thus this feature is carefully implemented considering all possibilities
With WSO2 MB's shared topic subscription feature enabled, following behavior is supported.
- You can connect multiple durable topic subscriptions to same node in broker cluster with same subscription ID.
- You can connect multiple durable topic subscriptions to different nodes in broker cluster with same subscription ID.
- You can subscribe mixing above two ways.
- You can publish to the topic you subscribed from using a publisher connection to any node in broker cluster.
- You cannot create a durable topic subscription by sub ID "x" to topic "foo", and another durable topic subscription by same sub ID "x" to topic "bar". It is not allowed. New connection will be terminated with an exception to client application.
- Hierarchical topic subscriptions are supported in clustered mode with shared durable topic subscriptions.
- Wildcard topic subscriptions are supported in clustered mode with shared durable topic subscriptions.
- If you unsubscribe one shared topic subscriber, it WILL affect all other durable topic subscriptions with same subscription ID (internally all share the same queue, during unsubscribe we remove it).
- Dead letter channel, routing messages back to durable topic subscriber are supported across cluster without any flow.
- You can view durable topic subscriptions and view/browse its messages via Management console of WSO2 MB. I will describe how management console behaves with an example below.
Demo and management console behavior
Setup
Create a three node MB 3.x.x cluster following the documentation here. Use Apache Jmeter to create four durable topic subscriptions to topic "myTopic" using subscription ID "mySub1" as below. Publish to topic "myTopic" by all 3 nodes in cluster
Our expectation of the test
(x1 + x2 + x3 ) = (y1 + y2 + y3 + y4)
Publisher
Subscriber
Observations
Log into Management Console of MB and navigate to main >> Manage >> Subscriptions >> Topic subscription list. You will see all four subscriptions listed there. Note the owned node. We use customer defined node ID (using broker.xml) or if not configured [localMememberHost: localMememberPort] combination to indicate the node subscription is actually created.
Please note following.
- You will see a uniform view of subscriptions across the cluster. That means no matter to which Management Console you logged in (MB1, MB2 or MB3) you will see same subscription list.
- Number of Messages Delivery Pending means number of total messages yet to deliver by all shared subscriptions as a whole. We do not divide the number of messages among shared durable topic subscriptions when rendering the UI.
Now close all subscriptions made to broker cluster. Now if you navigate to above page, you will see a single entry for durable topic subscription of that subscription ID indicating that there is an inactive subscription with pending messages to deliver.
If you change the subscribed nodes in the next time you subscribe (you can even bring in a new MB node MB4 and subscribe), no matter, they will be delivered to all subscriptions appropriately. If you sum up messages you sent to the cluster and you received at the end, they will be exactly equal.
If you now unsubscribe all messages will be gone.
When using shared subscriptions, until all subscriptions are inactive, you cannot unsubscribe. Basically WSO2 MB does not treat subscription is inactive until all subscriptions by that subscription ID is closed in whole MB cluster.
Conclusion
WSO2 MB does not suffer from "remote get problem" like many other brokers. We have a distributed algorithm that share message chunks across cluster nodes without causing any loss or duplication. This exercise alone shows how powerful and scalable WSO2 Message Broker is.
WSO2 ESB can work as publisher and consumer in a real world use case.
Way Forward
With next future versions of MB we will support JMS 2.0. Until then, we have implemented this powerful feature to enable shared durable topic subscriptions. You do not need to worry about scaling the durable topic subscribers or worry about single point of failure from consumer side. Just enable Shared Durable Topic Subscriptions in WSO2 Message Broker!
