Back in May 2009, I mentioned that I had an article published in GroovyMag.
GroovyMag is very nice in that it allows the author to republish their work after a ‘decent’ interval has passed (either 60 or 90 days, can’t quite remember at the moment). That period is well and truly passed for this article so here it is in its full resplendency. Enjoy!
[edit]
Forgot to make the source available when I first republished this article. Silly me. All fixed now!
Solving the Enterprise Integration Puzzle
Getting Started with Spring Integration
Groovy is frequently promoted as “the scripting language for the JVM.” Grails is often described as being “great for smaller, quick-to-build web applications.” Neither of these pieces of perceived wisdom really gives the Groovy/Grails combination its due: a powerful tool in the Systems Integrator’s armory. This article will show how Groovy and Grails–in conjunction with SpringSource’s up-and-coming “Spring Integration” project–can make complex systems integration tasks as easy as building a standalone web application.
A Brief Overview of Spring Integration
Systems integration is frequently likened to attempting to piece together a badly made jigsaw puzzle. The end goal is often quite fuzzy; there may be several potential solutions to consider; there may be many conflicting approaches to evaluate; and when all is said and done, sometimes all that can be done is to say “To heck with the Grand Plan!” and ‘adapt’ the puzzle using a pair of nice sharp scissors.
Spring Integration (SI) is a recent project within the Spring stable that provides numerous tools to make the systems integration task less frustrating. SI builds on several existing Spring stablemates, but provides a unified way of configuring these components and linking them together. According to SI, all message endpoints (which act with and on behalf of application components) interact via messages sent over various flavors of messaging channels, perhaps undergoing various transformations along the way. In addition, external resources (files, JMS-based queueing systems, SOAP-based WebServices, etc.) interact with these channels via simple-to-configure adapters.
SI imposes very little onto the developer; almost everything in the system can be implemented using simple POG/JOs (Plain Old Groovy/Java Objects) and configuration is typically via XML or Annotations.
SI provides out-of-the-box support for many of the patterns defined in Hohpe and Woolfe’s Enterprise Integration Patterns book and associated website and–as this article will show–is quite “Groovy friendly.”
A Small Example Application
In this article, I am going to take a very simple systems integration task and build a small example application. The application is a small Grails application that performs a simple function: given a customer number, it calculates the shipping cost for a single widget.
To show how Spring Integration can be used, I am assuming that a customer will enter their customer ID into a Grails-originated web page, which is then routed to one instance of a set of JMS servers. The associated address data is then retrieved, the city field extracted and then passed on to a SOAP-based WebService that calculates the cost of shipping widgets from Brisbane to the customer’s home city. To add a smidgen of realism, I have built the various services to be completely unrelated; in particular, they do not reference any form of standardized messaging schema. The integration pipeline needs to handle various simple message types, including strings, CSV-formatted data records, XML-formatted strings and floats.
Figure 1 provides a high-level view of the application.
Figure 1: The processing flow and major components comprising the example application
The various steps highlighted in this flow are:
- The customer ID (a simple string) is entered into the Grails web application and passed off to a generic SI gateway. The gateway places the customer ID on an internal channel that leads to a message router.
- The router retrieves the message from its inbound channel and determines which available channel to use to ensure that the message is handled by the appropriate JMS server instance.
- The message is then handled by a SI JMS “channel adapter,” which deals with the mechanics of working with the underlying external JMS server.
- The nominated JMS server instance produces a record (a CSV-formatted string) that is placed onto a shared reply channel.
- The CSV record is removed from the shared channel and transformed. The result of the transformation is an XML-formatted string which is again placed on a ‘downstream’ queue.
- The WebService channel adaptor removes the string from its request channel and invokes the external SOAP WebService for further processing.
- The WebService response (a single float) is then passed back to the originating messaging gateway, which returns it to the Grails controller.
This is a very simple system and it may seem that this is overkill. There is a payoff, however: SI makes it easy to plug in disparate (often legacy) systems and thus produces an expandable framework that is capable of being built out to cover future, more complex, situations.
The Pieces of the Jigsaw Puzzle
Before starting out on the integration exercise proper, it is worth taking a quick look at the various pieces of the jigsaw puzzle that are going to be joined together.
You will quickly see that all the pieces are exceptionally simplistic. I haven’t even tried to make them correspond to a multitude of real-life issues such as error handling, logging, configuration, etc., which have been almost completely ignored. This is by design: for the purposes of this article, I want to focus mostly on SI, not on the various services.
The Grails Web Application
Figure 2 shows the Grails web interface “in action.” It is very simple, with only a single form and a single results page.
Figure 2: The deployed Grails web application
The ‘meat’ of the application (as far as this article is concerned) lies in the single Grails controller class shown in Listing 1.
public class SpringIntegrationController {
def siGateway
def index = { }
def submit = {
long start = System.currentTimeMillis()
def msg = params.custid
siGateway.send(msg)
def got = siGateway.receive()
flash.message =
"Query took: ${System.currentTimeMillis() - start} millis."
[shippingCost: got]
}
}
Listing 1: The Grails controller class
There are two points of interest in this simple class.
The siGateway instance injected from the Spring application context constitutes the sole point of contact between the Grails web application and the SI framework. (In theory, SI allows for an even less intrusive interface than this, based around dynamically constructed proxies, but a bug surfaced as I started writing the application…no matter, the hard way is not too onerous to use and I am sure that the SI guys will soon squash the bug.)
The submit closure is responsible for handling the POSTed form data and for driving the injected siGateway instance through its paces. It is pretty straightforward: all the action is happening over in “SI-land.”
The JMS Application
For the example application, I have written a very simple JMS service (using Apache ActiveMQ as the messaging provider), as shown in Listing 2.
import javax.jms.Session
import org.apache.activemq.ActiveMQConnectionFactory
public class AMQService
{
private static factory =
new ActiveMQConnectionFactory("tcp://localhost:61616")
public static void main(String[] args) throws Exception
{
def cKey = args[0]
def cFile = args[1]
println 'Starting...'
println "Config key: $cKey"
println "Config file: $cFile"
def config =
new ConfigSlurper(cKey).parse(new File(cFile).toURL())
def database = config.database
println "Database: $database"
def inQ = config.inQ
println "Incoming Queue: $inQ"
def replyTopic = config.replyTopic
println "Reply Topic: $replyTopic"
def qConn = factory.createQueueConnection()
def qSession =
qConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE)
def consumer =
qSession.createConsumer(qSession.createQueue(inQ))
def tConn = factory.createTopicConnection()
def tSession =
qConn.createTopicSession(false, Session.AUTO_ACKNOWLEDGE)
def producer =
tSession.createProducer(tSession.createTopic(replyTopic))
qConn.start()
tConn.start()
for ( ; ; )
process(database,
tSession.createTextMessage(),
consumer,
producer)
}
private static process(database, replyMessage, consumer, producer)
{
try
{
def textMessage = consumer.receive() // indefinite blocking
replyMessage.with {
setText(getResponse(database, textMessage.getText()))
setJMSCorrelationID(textMessage.getJMSCorrelationID())
}
producer.send(replyMessage)
}
catch (Throwable t)
{
t.printStackTrace()
}
}
private static getResponse(database, inMsg)
{
def resp = database[inMsg]
println "Request: '$inMsg'; Response: '$resp'"
resp
}
}
Listing 2: The JMS server
This is a standard JMS service; no need to elaborate.
One point to note (that is not directly SI-related, but of interest nonetheless) is that the service is written to allow multiple instances to run at the same time, dealing with a partitioned data space. For the purposes of the example application, I have configured two instances. One instance is prepared to deal with customer IDs whose first character is in the range a-l and the second will deal with the rest of the alphabet.
The service accepts input via a uniquely-designated input queue, but sends its response to a shared topic. This structure simplifies things by ensuring that any downstream component merely has to handle a single publish/subscribe channel (which can carry data from multiple sources), rather than trying to deal with a multitude of point-to-point links.
Each server instance is driven by a pair of application parameters, defining:
- The location of the configuration file to use
- The appropriate environment section in the designated configuration file
The configuration file supplies:
- The name of the input queue to use
- The name of the output topic to receive the responses
- The actual static database to use (I did warn that these applications were extremely simple!)
The service makes use of ConfigSlurper’s ability to deal with different environment sections and so I can define a single configuration file (see Listing 3) to configure the different instances uniquely.
replyTopic="SI.REPLY.TOPIC"
replyTopic="SI.REPLY.TOPIC"
environments {
al {
inQ="SI.AL.REQUEST"
database=['b9876':'123 Nowhere St,Brisbane',
'f0234':'42 Imaginary Place,Darwin']
}
mz {
inQ="SI.MZ.REQUEST"
database=['x5555':'123 Fake St,Hobart',
'w8888':'000 Talkfest Lane,Canberra']
}
}
Listing 3: JMS server configuration file
This service accepts a string, but returns a CSV-formatted record.
The WebService
The final part in the jigsaw puzzle is a SOAP WebService built on top of the facilities provided by GroovyWS.
The service’s raison-d’être is to retrieve the cost of shipping n widgets between major cities in Australia; see Listing 4 and Listing 5.
public class ShippingCostWebService {
static costs = [
Brisbane: [ Brisbane:0.0F,
Darwin:44.44F,
Canberra:22.22F,
Hobart:88.88F ],
Darwin: [ Brisbane:44.44F,
Darwin:0.0F,
Canberra: 33.33F,
Hobart: 99.99F ],
Canberra: [ Brisbane:22.22F,
Darwin:33.33F,
Canberra: 0.0F,
Hobart: 77.77F ],
Hobart: [ Brisbane:88.88F,
Darwin:99.99F,
Canberra: 77.77F,
Hobart: 0.0F ],
]
Float calculateShippingCost(String fromLoc,
String toLoc,
Integer nItem) {
def cost = costs[fromLoc][toLoc] * nItem
println "calculateShippingCost($fromLoc,$toLoc,$nItem) => $cost"
cost
}
}
Listing 4: The shipping cost WebService
println 'Starting...'
def ws = new groovyx.net.ws.WSServer()
ws.setNode("ShippingCostWebService",
"http://localhost:6980/ShippingCostWebService")
ws.start()
println '...Started.'
Listing 5: The GroovyWS WebService server harness
I can’t lie! This is just an old WebService that I had lying around that I created for a conference presentation, and which I have adopted for this application. It doesn’t have a customized API and isn’t necessarily a perfect fit to the task. In the systems integration world, improvisation is often required to squeeze data out of the numerous ‘sub-optimal’ nooks and crannies that are found all over an enterprise.
Joining the Pieces Together
Now that you are familiar with the basic componentry, it is time to look at SI proper.
As with any Spring project, SI is configured separately to the actual code in an application. Since this application is based on Grails, I have chosen to use the nice resources.groovy Grails Spring Beans DSL configuration file, rather than the ‘traditional’ XML-based configuration.
Listing 6 shows the complete configuration.
beans = {
xmlns si:"http://www.springframework.org/schema/integration"
xmlns jms:"http://www.springframework.org/schema/integration/jms"
xmlns stream:"http://www.springframework.org/schema/integration/stream"
xmlns ws:"http://www.springframework.org/schema/integration/ws"
// SI componentry and plumbing
si {
poller(default: true) {
"interval-trigger"(interval: 1, "time-unit": "SECONDS")
}
router("input-channel": "routerChannel",
ref: "highLowRouter",
method: "route")
transformer("input-channel": "InboundTopic",
"output-channel": "TransformedInbound",
ref: "csvStringTransformer",
method: "transform")
channel(id: "routerChannel")
channel(id: "ALOutboundChannel")
channel(id: "MZOutboundChannel")
channel(id: "RoutingRejectChannel") {
queue(capacity: "256")
}
channel(id: "TransformedInbound") {
queue(capacity: "16")
}
channel(id: "InboundTopic") {
queue(capacity: "16")
}
channel(id: "ShippingCostChannel",
dataType: "java.lang.Float") {
queue(capacity: "16")
}
}
stream {
"stderr-channel-adapter"(channel:"errorChannel",
"append-newline": true)
}
// WebService stuff
ws {
"outbound-gateway"("request-channel": "TransformedInbound",
"reply-channel": "ShippingCostChannel",
uri: "http://localhost:6980/ShippingCostWebService")
}
// AMQ/JMS stuff
connectionFactory(org.apache.activemq.pool.PooledConnectionFactory) { bean ->
bean.destroyMethod = "stop"
connectionFactory = { org.apache.activemq.ActiveMQConnectionFactory cf ->
brokerURL = "tcp://localhost:61616"
}
}
"AL.REQUEST"(org.apache.activemq.command.ActiveMQQueue, "SI.AL.REQUEST")
"MZ.REQUEST"(org.apache.activemq.command.ActiveMQQueue, "SI.MZ.REQUEST")
"REPLY"(org.apache.activemq.command.ActiveMQTopic, "SI.REPLY.TOPIC")
jms {
"outbound-channel-adapter"(channel: "ALOutboundChannel",
destination: "AL.REQUEST")
"outbound-channel-adapter"(channel: "MZOutboundChannel",
destination: "MZ.REQUEST")
"message-driven-channel-adapter"(channel: "InboundTopic",
destination: "REPLY")
}
// general componentry
siGateway(org.springframework.integration.gateway.SimpleMessagingGateway) {
requestChannel = ref("routerChannel")
replyChannel = ref("ShippingCostChannel")
replyTimeout = "10000"
}
highLowRouter(au.com.transentia.si.LookupRouter)
csvStringTransformer(au.com.transentia.si.CsvTransformer) {
homeBase = 'Brisbane'
}
}
Listing 6: The Spring Integration configuration
As you read the following discussion, you may find it useful to refer back to Figure 1.
The configuration is structured into various sections, according to the needs of SI and of the various resources that are being integrated.
The si section (more precisely, those elements of SI configured via the XML namespace allocated the si prefix in this document) configures the various aspects of SI itself. This includes establishing the various channels that interlink the components, defining a few application-specific components (such as the router and transformer), and putting in place a default polling schedule for those parts of SI that need to poll.
It is worth examining the router and transformer components here. I have mentioned that the JMS services deal with a partitioned data space and that traffic is directed to an individual service based on the actual data message being processed. SI allows for the definition of an application-specific router to perform this type of task, in this case the method route of the highLowRouter instance. Listing 7 shows the code for the router.
package au.com.transentia.si
public class LookupRouter {
public String route(String msg) {
switch(msg) {
case ~/(?i:[a-l].*)/: return 'ALOutboundChannel'
case ~/(?i:[m-z].*)/: return 'MZOutboundChannel'
default: return 'reject'
}
}
}
Listing 7: The router class
The router examines the incoming message data and determines the appropriate outbound path for the message. Groovy’s wonderfully versatile switch statement, combined with the ability to do case-independent pattern matching, makes for beautifully minimalistic code.
The transformer is equally simple, as Listing 8 shows.
package au.com.transentia.si
public class CsvTransformer {
def homeBase
public String transform(String csv) {
def dest = csv.tokenize(',')[1]
"""
$homeBase
$dest
1
"""
}
}
Listing 8: The transformer class
This code is presented with an input CSV-formatted record, extracts the relevant field, and passes an XML-formatted string onwards.
Recall that the WebService’s API is not custom-built for the purpose of this application; for this use, arg2 (the nItems parameter) is always fixed at 1.
The stream section of the configuration merely allocates a generic channel listener so that the standard errorChannel is not silently ignored; this configuration will ensure that errors/exceptions, etc., are logged.
The ws section deals–unsurprisingly–with WebService integration. The section configures an SI outbound gateway that is capable of receiving a request on a specified channel, invoking the configured WebService and finally sending the result to the appropriate response channel. All this is specified declaratively: the actual mechanism of dealing with the WebService (retrieving WSDL documents, generating proxies and handling XML-based request/response messages) is completely hidden away.
The remaining configuration concerns SI’s JMS adapter and ActiveMQ integration.
The siGateway has been discussed previously.
By default, all SI’s JMS adapters reference a bean with the standard name of connectionFactory. It is the responsibility of this bean to ‘vend’ connections to the external JMS system. In this case, a connection pool to a local Apache ActiveMQ server running at the URL “tcp://localhost:61616″ is being created. Since the external connections are pooled for efficiency, by specifying destroyMethod=’stop’ we ensure that they are simply inactivated after each (re)use and are never actually discarded.
Note: I found the actual syntax used to declare the connectionFactory a little tricky. I had to ask for help on the excellent user@grails.codehaus.org mailing list to get it right! Thanks to the generous souls who pointed me in the right direction.
Following the definition of the the connectionFactory, we create two instances of ActiveMQ queues and an instance of an ActiveMQ topic. It is these that carry the messages to/from the external JMS resources.
The jms section configures two JMS-specific outbound channel adapters and one inbound channel adapter. As the names suggest, these are concerned with driving (or being driven by) the ActiveMQ instances declared above. The configured message-driven-channel-adapter instance defines a component that will be asynchronously “actively invoked” to handle an incoming message. It is possible to configure a polling-oriented component, but this is generally less flexible, and may waste CPU, and increase the latency in the system.
As with the WebService gateway, everything is declarative.
Admiring the Finished Puzzle
SI has enabled a deceptively simple solution to a fairly complex problem. There are no messy tracts of confusing mechanistic code; the system is essentially defined all in a single configuration; the individual components are simple and (generally) reusable; and the groundwork for coping with change set with very little effort.
SI is a simple but powerful toolkit; the example application I have discussed here has really just touched the surface of what is possible. SI provides much more, including facilities for secure channels, inbound/outbound mail handling, event handling, RMI integration and much better XML handling than I have touched upon here.
Remember that these are still early days for SI: for example, there are no debuggers or GUI editors as found in the big “name brand” systems integration tools. In time these tools may come but for now remember that SI is orders of magnitude simpler, and the SI/Groovy/Grails triumvirate makes the developer many times more productive than some other tools I have used.
I hope that I have stimulated your interest in SI and shown that a Grails application need not be limited to serving up “little web apps” but can actually service the needs of larger-scale systems integration tasks.
I hope that I have also been able to show that when the time comes to get out the scissors, Groovy is more than equal to the task of “slicing and dicing” until the pieces of the systems integration jigsaw puzzle fit nicely together.
Learn More
- Gregor Hohpe and Bobby Woolf, Enterprise Integration Patterns : Designing, Building, and Deploying Messaging Solutions (ISBN 0321200683), also http://www.enterpriseintegrationpatterns.com/
- Spring Integration Home, http://www.springsource.org/spring-integration
- Apche ActiveMQ, http://activemq.apache.org
- GroovyWS, http://docs.codehaus.org/display/GROOVY/GroovyWS. The exemplar application uses the 0.5 snapshot jars from http://snapshots.dist.codehaus.org/groovy/distributions/groovyws/
Bob Brown is the director and owner of Transentia Pty. Ltd.. Based in beautiful Brisbane, Australia, Bob is a specialist in Enterprise Java and has found his niche identifying and applying leading-edge technologies and techniques to customers problems.
Watch out for my other Groovymag articles…coming soon to a browser near you.
