Broker Concepts In webMethods

WebMethods Integration Server(IS) is mainly used to support the integration of diverse services and communication between systems.

But to do the same we need multiple Integration servers to do the complete job (Not compulsory, depends on business requirements 

and server load). 

And this is the area where Broker will come into the picture, which acts as "message backbone" which helps the IS's to communicate with each other using a model called as "Publish and Subscribe Model". 

We will look into more details about this model in another section.

When the IS's received the documents from the external applications, the IS publishes the document to the the respective configured Broker which in turn the configured subscriber(another IS or another application) will subscribe those documents for the further process.

So this inturn indicates that the IS interactions are in de-coupled format, meaning they do not interact directly. 

Instead, the interactions will happen through Broker.

These are the components that interact with the Broker:

1. webMethods IS

2. webMethods JMS with ASF Support

3. webMethods Task Engine

4. webMethods Logging Utility

5. webMethods Process Engine

A webMethods Broker environment mainly consists of following main components.

1. Broker Server: This is a runtime components which mainly used for the Publishers and Subscribers to Interact.
It also manages memory and disk resources for all the Brokers that reside on it.

By default, port 6849 is the Broker Server base port. 

It uses the default port 6849 for non-SSL requests and ports 6848 and 6847 for SSL requests.

2. Brokers: This resides on Broker Server.

When a client connects to Broker Server, the client specifies the Broker with which it wants to interact.

3. Broker User Interface: This is a administarting component, which we mainly used to monitor Broker Health, log size etc. This is running on My webMethods Server.

4. Broker Monitor: This will install automatically when we install Broker Server.

As in the name describes that, the Broker Monitor will mainly monitor all the Broker servers status.Broker Monitor continually checks the state of the Broker Server and automatically attempts to re-start it if it stops running.The Broker Monitor default port 6850.The Broker Monitor configuration file (awbrokermon.cfg) points to the Broker Servers that reside on the host machine.So when BrokerMonitor starts the broker server, it will capture the respective Broker Server PID and using this PID it will monitor the status of the respective Broker servers.

 Diagram: Broker Monitor-->Broker Servers(Multiple with it's own port)-->Brokers(Multiple)

  

A webMethods Broker environment has only one Broker Monitor even if it hosts multiple Broker Servers. Broker Monitor monitors all of the Broker Servers running in the webMethods Broker environment. 

It will automatically attempt to restart any Broker Server that stops running in its environment.

Broker encompasses the following objects

1. Document types: which identify the kinds of documents that the Broker’s clients can exchange.

2. Client Groups: which define specific properties and permissions that Broker applies to clients.

3. Client State Objects: which maintain information about the individual clients that use

the Broker.

Note: When we install Broker Server, the server will be installed with a "default" Broker, which we will be mainly used to communicate with applications which not mentioned with to which broker it needs to communicate. We can install required number of Brokers apart from this "default" broker.

How Integration Server communicate with Broker?

The Integration Server uses "dispatcher.cnf" config files to communicate with Broker. 

This config will store all the information about the Broker to which IS wants to communicates like broker server name, broker server port, status of the Broker. 

The dispatcher will also take care of the connection pool monitoring which helps to connectivity between IS and Broker.

How the Storage will be handling in the Brokers?

The broker Storage will be handling through the concept called "Queue". 

When the broker receives the documents from the IS, the Broker will place the documents that matches the client's subscription in the client's queue.

Each client has it's own queue.

The Queue will be either in "Locked" or "Unlocked" state. 

During the "Lock" state all the published documents will be keep piling up in the queue which we can be used to verify the documents structure by downloading the same. 

And during the "Unlock" state the documents will be subcribed by the respective subscribing clients.

Note: When there are no subscriptions for the respective documents, then the documents will be moved to queue called "DeadLetter" queue.

Note: We use another concept of queue called "Forward queue", this is mainly used in the Territories to store the documents between the brokers configured in territories.

What about StorageType of Documents?

Documents published to Broker can be stored as volatile documents or guaranteed documents. 

Volatile documents are stored only in memory and are lost if the host machine loses power or the Broker Server is otherwise re-started. 

Guaranteed documents are persisted to disk so that they can be recovered in case of a power failure or a restart.

Now we will see what is the combination of Queue+Storage i.e, QueueStorage refers to?

QueueStorage in the Broker Servers refers to fileStorage used to persist the guaranteed documents and non-volatile Broker meta-data. Queue Storage consists of two types of files LogFile and StorageFile.

LogFile: which is a fixed-sized file that Broker can write data to quickly when each documents through broker.

StorageFiles: To which the files/data will stored in batch wise after the LogFile size becomes full.This will happen automatically through a process called "logged-commit" process

We can configure QueueStorage either in Combined mode or Separate Mode. In combined mode, Broker meta-data and run-time data are stored in the same log and storage files. In Separate mode,the broker uses two log files and two storage files.It stores meta-data in one log/storage pair and run-time data in the other log/storage pair.

How can we sync multiple Brokers ?

We can sync the multiple brokers using the concept of Territory.

Brokers that form a territory function as a single logical Broker. Clients that connect to one Broker in a territory can automatically exchange documents with clients connected to any of the Brokers in the territory.

When Brokers operate in a territory, they maintain the same set of document types and client group definitions.

How multiple territories will communicate?

Territories will communicate through a bridge called "Gateway".

A gateway enables clients in one territory to receive documents that are published in another territory.

Procedure for Start and Stop of the Broker components:

We can achieve the Start and Stop mechanism of the Broker Components using either Broker user Interface through MWS or by with the help of UNIX commands(On UNIX Platform) or Windows Procedures (On Microsoft Platform)  as described below.

--> Broker Monitor:

To start Broker Monitor manually on Windows

1 From the Start menu, open Services as follows:

Settings > Control Panel > Administrative Tools > Services

2 Right-click the webMethods Broker Monitor service and click Start.The status of the Broker Monitor switches to “Started” and Broker Monitor starts the Broker Servers that reside on the machine.

To configure Broker Monitor to start automatically on Unix

1 Navigate to the directory that contains the Broker Monitor script.

2 Run the following command:

./S45_broker71 start

Where S45_broker71 specifies the file name of the start up script.

3 To confirm that Broker Monitor is running, enter the following command:

ps -ef | grep aw

If Broker Monitor is running, “awbrokermon” will appear in the process list.

To stop Broker Monitor on Windows

1 From the Start menu, open Services as follows:

Settings > Control Panel > Administrative Tools > Services

2 Right-click the webMethods Broker Monitor service and click Stop.

3 Click Yes to confirm that you want to stop the Broker Servers that Broker Monitor is monitoring.

To stop Broker Monitor on Unix

1 Navigate to the directory where the Broker Monitor script file resides.

2 Run the following command:

./S45_broker71 stop

Where S45_broker71 specifies the file name of the stop script.

--> Broker Server

To manually start a Broker Server using My webMethods

1 In My webMethods: Messaging > Broker Servers > Servers.

2 In the Broker Servers List, select the check box beside the Broker Server that you want to

start.

3 Click Restart.

To manually start a Broker Server from the command line

1 On the machine where Broker Server is installed, navigate to the following directory:

webMethods/Broker/bin

2 Run the following command:

broker_start hostName:portNum

To manually start a Broker Server from the Windows control panel

1 From the Start menu in Windows, open Services as follows:

Settings > Control Panel > Administrative Tools > Services

2 Right-click the webMethods Broker Server service that you want to start and click Start.

To stop a Broker Server using My webMethods

1 In My webMethods: Messaging > Broker Servers > Servers.

2 In the Broker Servers List, select the check box beside the Broker Server that you want to

stop.

3 Click Stop.

To stop a Broker Server from the command line (using broker_stop)

1 On the machine where Broker Server is installed, navigate to the following directory:

webMethods/Broker/bin

2 Run the following command:

broker_stop hostName:portNum idParams

To stop a Broker Server from the Windows control panel

1 From the Start menu, open Services as follows:

Settings > Control Panel > Administrative Tools > Services

2 Right-click the webMethods Broker Server service that you want to stop and click Stop.

Now we will see few more details of the Broker Objects which we have mentioned before.

Document Types:

As we know that, the broker communicates between Publisher and subscriber through documents. But how will the Broker knows, whether to accept or discard the incoming document when it is notified with the incoming document?. To achieve this the Broker uses the "Document Type" as filter technique, which mean that when the Broker receives the document the Broker will check whether the respective Document type is configured in the broker to accept the Document if it is then it will accept by the respective broker.

Each document is an instance of a document type. A document type is a schema-like definition that describes the structure of a document that publishers and subscribers can exchange using the Broker. A document type has a unique name, a structure that consists of named fields, and a set of properties that determines how the Broker handles instances of that document type at run time.

We can create these Document Types either in the following ways.

1. We can use Developer to define a document type and mark it as publishable.

2. Using Broker API's.

3. A developer can add a JMS topic as a Broker document type through My webMethods.

DocumentType Name appears in MWS as below.

folderName::subFolderName::documentTypeName

Example:Type::Hardware::PO::receivePO

Note: When we create document type is created through webMethods Developer, the Broker document type name includes the preface “wm::is::”.

Important Properties of Document Type:

1. Time to live specifies how long Broker maintains instances of this document type once

they are published.

2. Validation indicates whether or not Broker validates instances of this document type

and if so, whether or not the document can contain fields that are not defined in the

document type.

3. Storage type determines whether instances of this document type are stored in memory

or on disk.

To view documents types on a Broker

In My webMethods: Messaging > Broker Servers > Document Types.

We can modify the following Document Type Properties using My webMethods.

ion an

Document type description

Time to live value

Validation level

Note: If a client publishes a document for which there is not an associated document type on the Broker, Broker returns an error message to the client and publication fails.

To Delete document Types on a Broker

1. In My webMethods: Messaging > Broker Servers > Document Types

2. In the Document Type List, select the check box next to the name of the document types that you want to delete. If the document type does not appear in the list, use the Search tab to locate it.

3. Click Delete. My webMethods deletes the selected document type.

Client Groups:

Each Client in the Broker belongs to one Client Group, where we can use to define a particular classes of a client like Administrators,Integration servers processes.Below are the settings which we can do using Client Group Object.

a. Client Characteristics Like Queue Storage Type

b. Security-Labeled: Which includes ACL, encryption etc.

c. Access To Document Types:specified by lists of the document types available to the client group’s members.

d. Can-publish types:List the document types that the client group’s Broker clients are permitted to publish (or deliver).

e. Can-subscribe types: List all document types that Broker clients in the client group can subscribe to (or receive deliveries from).

f. Log publish types: List all the document types that Broker clients in the client group can log when published.

g. Log acknowledge types: Display the number of document types that Broker clients in the client group can log when received.

Another important property which we can assign using client Group to each client is "Queue Storage Type" and "Client LifeCycle".

Queue Storage Type:

You assign a queue storage type when you create a client group, and the value cannot be changed. You can set the storage type to volatile or guaranteed.

1. Volatile. Documents are stored only in local memory and are lost if the Broker Host loses power or the Broker Server is re-started.

2. Guaranteed. Documents are persisted to disk and can be recovered in case of a power failure or restart.

Client LifeCycle

Explicit destroy. Maintains the Broker client’s state object when the client disconnects.The client continues to receive documents in its queue, even though it is not actively connected. Use this setting for applications that need to maintain state information in the Broker between connections, e.g., if a network or system failure occurs.For example, a webMethods Integration Server uses the explicit destroy lifecycle so that documents that update the database are not lost if the Integration Server is not running. When the Integration Server is not running, the Broker queues documents for it; the Integration Server retrieves the documents when it restarts.

Destroy on disconnect. Broker automatically destroys client state object information whenever the connection between the client and Broker is terminated, thus losing all knowledge of the client. With this setting, the state of a Broker client exists for the duration of the client’s connection to the Broker.

Clients:

As mentioned above A client is an object on the Broker that represents the connection between a client program and the Broker.

A client program publishes documents to a Broker Server using its client. The Broker Server’s publishing engine routes documents published by a client to other clients that subscribe to those documents.A client program subscribes to documents by registering document type subscriptions with its client. Broker places the documents to which a client subscribes in the client’s queue. A client program then retrieves the documents from its client queue.

If multiple client programs can share the same client if the client was configured as a shared state client when it was created. Sharing a client allows multiple client programs, each using its own session, to process documents from a single client queue in parallel on a first-come, first-served basis.

Note: If you disconnect the last session to a client with a destroy on disconnect lifecycle, Broker deletes the client.

If you disconnect the last session for a client with an explicit destroy lifecycle, the client remains. The client program can reconnect to the client at a future time.

Note: Clients cannot be deleted when their queues are locked.

Clients can be copied only when the client group to which it belongs is present in the target server. Client groups can be copied only when all the documents in its can subscribe and can publish list are present in the target broker.

In our previous discussion we have covered the concepts of Territory and Gateways, now we will how to create and delete the same in MWS.

Territory:

To create a territory

1 In My webMethods: Messaging > Broker Territories > Brokers

2 Click Add Territory.

3 In Territory Name, type a name for the territory.

4 Use the following steps to select one of the Brokers that will participate in this

territory:

  a In the Broker Server field, select the Broker Server that hosts the Broker.

  b Use the Selected Brokers controls to specify the Brokers that you want to add.

5 Click OK.

How to add Brokers to Territory?

1 In My webMethods: Messaging > Broker Territories > Brokers

2 In the Territory Brokers List, click Add Broker.

3 From the Territory to Join list on the Add Territory Broker page, select the territory to

which you want to add a Broker.

4 Use the following steps to select the Brokers to add to the territory.

a In the Broker Server field, select the Broker Server that hosts the Broker(s) that you

want to add.

b Use the Selected Brokers controls to specify the Brokers that you want to add to

territory.

5 Click Join

How to Remove Broker from Territory?

1 In My webMethods: Messaging > Broker Territories > Brokers

2 In the Territory Brokers List, select the check box beside the Broker that you want to

remove from a territory.

3 Click Leave Territory.

To Delete a Territory

You can dissolve a territory by simply removing every Broker from the territory. When the last Broker has been removed from the territory, the territory will cease to exist.

Gateway:

A gateway connection enables two territories to exchange documents. Unlike territories, whose constituent Brokers automatically share their document types and client groups, a gateway connection requires an administrator to explicitly specify which document types the territories are permitted to send and receive. Moreover, the territories never share client group information.

A gateway performs functions similar to those performed by a Remote Broker object in a

territory. 

1. Establishes and maintains a connection with its counterpart in the remote territory.

2. Maintains the subscription list and forward queue for the remote territory.

3. Retrieves documents for the local territory from the remote gateway.

Note: Unlike a Remote Broker object, however, a gateway requires an administrator to explicitly specify which document types it can “forward to” and “receive from” the remote territory.

How the Gateway Name looks like ?

localTerritory > remoteTerritory

Steps to take before configuring the gateway betwen territories.

1. Identify the list of document types that each territory needs to receive from the other

2. Identify the two Brokers that will host the gateways

3. Determine what type of authentication and encryption is required between the territories

4. Create and configure each gateway in the gateway pair.

How to define a gateway pair?

1 In My webMethods: Messaging > Broker Territories > Gateways.

2 Click Add.

3 On the Create Gateway Pair tab, select the two Brokers that will form the gateway connection.

4 Click Create.

5 Then we need to select the Document Types which need to flow through the Gateway Connection.

Gateway Filters:

The Broker uses the filter string to determine which documents match your criteria and allows only those documents that match the filter string to pass through the gateway.

How to delete a gateway connection?

1 In My webMethods: Messaging > Broker Territories > Gateways.

2 In the Territory Gateways List, select the checkbox beside the gateway(s) that you want to delete.

3 Click Delete.

How to Pause and resume a Document Flow in Gateway?

To Pause:

1 In My webMethods: Messaging > Broker Territories > Gateways

2 Click Details for the gateway whose outbound flow of documents you want to pause.

3 On the Configuration tab, click Pause Forwarding.

To Resume:

1 In My webMethods: Messaging > Broker Territories > Gateways.

2 Click Details for the gateway whose outbound flow of documents you want to restore.

3 On the Configuration tab, click Resume Forwarding.

Note:It does not prevent that gateway from receiving documents from the remote territory. To halt the flow of inbound documents, you must pause the gateway in the remote territory, too.

THIS completes all the high level picture of the wM Broker Tool.

Other Info:

What are Infosets?

Infosets contain client application-specific information stored in the format of a Broker document. The infoset information is based on the application that created the client (for example, both JMS and Test Client store data in the infoset field), and the application uses the infoset to store specific configuration information for its own use. 

Can we copy document Type between Brokers?

You can copy document types between any Brokers to which you have access in one of two ways:

1. Copy and paste.

2. Export and Import. When you use the export-import functionality, My webMethods places the exported document type information in a file. The administrator for the destination Broker then imports the file. To use the export-import functionality, you must have administrator access. For more information about using the export-import

functionality

What is Can-Publish and Can-Subscribe?

This feature is w.r.t to document types, which we will select when the respective document wants to flow between two brokers within territories.

What is Allow-Forward and Allow-Receive ?

This feature is w.r.t to document types, which we will select when the respective document wants to flow between two Territories through Gateway.

What is Remote Broker Object?

Each Broker in a territory maintains a Remote Broker object for each of the Brokers to which it is connected. This object acts as the server-side state object for the Broker at the other end of the connection.