Take any integration project and you have multiple applications
talking over multiple transports on multiple platforms. As you can imagine, in
large enterprise applications this can get complex very fast. Much of the
complexity stems from two issues:
Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and DSLs. There is of course a lot more going on here under the hood but we'll leave that for another discussion.
Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 13 essential components. There are over 80 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queue aQueue and send them to a file system directory "/tmp", you could use URIs like "jms:aQueue" and "file:/tmp".
Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports over 40 patterns from the EIP book and many other useful Processors.
To wire Processors and Endpoints together, Camel defines multiple DSLs in regular programming languages such as Java, Scala and Groovy. It also allows routing rules to be specified in XML. Here are some DSL examples using different languages and staying functionally equivalent:
In all the above examples we define a routing rule that will load files in the “/tmp” directory into memory, create a new JMS message with the file contents, and send that message to a JMS queue named aQueue.
So we have several patterns in use here.
1. There are two Message Endpoints; one for FTP connectivity and another for HTTP.
2. Messages from these endpoints are fed into the incomingOrders Message Channel
3. The messages are consumed from the incomingOrders Message Channel and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file.
4. Both Message Translators convert the message content into a POJO, which is fed into the orders Message Channel.
The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer.
While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel entirely from Spring. In fact, the entire solution (except for the domain POJO and Maven build script) fits neatly into a single Spring XML file. This is because we are implementing our routing rules using the Spring XML configuration rather than one of Camel’s other DSLs. There are pros and cons to each of course but in this case he mainly wanted to show off some cool tooling that currently works with the XML based routing rules only. The Spring XML file is shown in Listing 2 below.
Listing 2: Complete Spring XML file that configures an embedded ActiveMQ broker and initializes a Camel Context with three routes.
Listing 3: Incoming message formats; XML on the left, CSV on the right.
At this point, successfully normalized messages are sent to the orders queue for processing by some other application at the Rider Auto Parts business.
1. dealing with the specifics of applications
and transports, and
2. coming up with good solutions to
integration problems.
Making your applications speak transports and APIs is relatively easy
on its own. I'm sure everyone knows how to send JMS messages to their broker of
choice; though it still requires in depth knowledge of the JMS specification,
which many developers may not have. On top of that, what happens when you want
to route that JMS message to another application? You then have to take care of
mapping the JMS message to the application plus handle any new concepts related
to the application. Add a dozen other applications into the mix and you've got
quite a headache on your hands.
Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations.
Apache Camel was created with the intention of addressing these two issues. In this article he'll first give you a quick overview of what Camel is. He’ll then guide you through how Camel solves a sample integration problem. Finally He’ll show off some new graphical tooling that makes it even easier to solve integration problems with Camel.
Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations.
Apache Camel was created with the intention of addressing these two issues. In this article he'll first give you a quick overview of what Camel is. He’ll then guide you through how Camel solves a sample integration problem. Finally He’ll show off some new graphical tooling that makes it even easier to solve integration problems with Camel.
What is Camel?
Apache Camel is an open source Java framework that focuses on making
integration easier and more accessible to developers. It does this by
providing:
• concrete implementations of all the widely
used EIPs
• connectivity to a great variety of
transports and APIs
•
easy
to use Domain Specific Languages (DSLs) to wire EIPs and transports together
Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and DSLs. There is of course a lot more going on here under the hood but we'll leave that for another discussion.
Figure
1: High level view of Camel's architecture (from Camel in
Action).
Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 13 essential components. There are over 80 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queue aQueue and send them to a file system directory "/tmp", you could use URIs like "jms:aQueue" and "file:/tmp".
Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports over 40 patterns from the EIP book and many other useful Processors.
To wire Processors and Endpoints together, Camel defines multiple DSLs in regular programming languages such as Java, Scala and Groovy. It also allows routing rules to be specified in XML. Here are some DSL examples using different languages and staying functionally equivalent:
•
Java
DSL
1.from ("file:/tmp").to("jms:aQueue");
•
Spring
DSL
1.<route>2.<from uri="file:/tmp"/>3.<to uri="jms:aQueue"/>4.</route>
•
Scala
DSL
1.from "file:/tmp" -> "jms:aQueue"In all the above examples we define a routing rule that will load files in the “/tmp” directory into memory, create a new JMS message with the file contents, and send that message to a JMS queue named aQueue.
These are the concepts that Camel was
built upon. Since then many other interesting features have been added. He
recommends reading Camel in Action,
by Claus Ibsen and myself to really get the full picture of what Camel
can do. But, to get you started, some of these extra features include:
• Pluggable data formats and type converters
for easy message transformation between CSV, EDI, Flatpack, HL7, JAXB, JSON,
XmlBeans, XStream, Zip, etc.
• Pluggable languages to create expressions
or predicates for use in the DSL. Some of these languages include: EL, JXPath,
Mvel, OGNL, BeanShell, JavaScript, Groovy, Python, PHP, Ruby, SQL, XPath,
XQuery, etc.
• Support for the integration of beans and
POJOs in various places in Camel.
• Excellent support for testing distributed
and asynchronous systems using a messaging approach
•
and
much more...
Introducing Rider Auto Parts
The example in this article is based on a fictional motorcycle parts
business used throughout the Camel in
Action book. The company, named Rider Auto Parts, supplies parts to
motorcycle manufacturers. Over the years, they’ve changed the way they receive
orders several times. Initially, orders were placed by uploading
comma-separated value (CSV) files to an FTP server. The message format was
later changed to XML. Currently they provide a website through which orders are
submitted as XML messages over HTTP.
Rider Auto Parts asks new customers to use the web interface to place
orders, but because of service level agreements (SLAs) with existing customers,
they must keep all the old message formats and interfaces up and running. All
of these messages are converted to an internal Plain Old Java Object (POJO)
format before processing. A high level view of the order processing system is
shown in Figure 2.
Figure
2: High level view of order processing at Rider Auto Parts (from Camel in
Action).
We need to find
out the best way to implement the “Rider order frontend” in Figure 2 above.
Let’s first see how this looks using notation from the Enterprise Integration Patterns book.
Solution using EIPs
Rider Auto Parts faces a pretty common problem; over years
of operation businesses acquire software baggage in the form of transports/data
formats that are popular at the time. Using patterns from the EIP book we can
envision the solution as something like Figure 3.
Figure
3: This shows the solution to Rider Auto Parts integration problem using
notation from the Enterprise Integration Patterns book.
So we have several patterns in use here.
1. There are two Message Endpoints; one for FTP connectivity and another for HTTP.
2. Messages from these endpoints are fed into the incomingOrders Message Channel
3. The messages are consumed from the incomingOrders Message Channel and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file.
4. Both Message Translators convert the message content into a POJO, which is fed into the orders Message Channel.
The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer.
Implementation using Camel
As mentioned
before, Camel has a small core set of components included by default. The rest
of the components exist as separate modules. In applications that require many
types of connectivity it is useful to figure out what Camel modules to include.
Listing 1 shows the dependencies using Apache Maven for the Camel
implementation of the Rider Auto Parts example. Of course, you don't need to
use Apache Maven for dependencies - it is just the easiest way to rapidly add
new dependencies to your applications. The list of dependencies includes
support for core Camel, ActiveMQ, JAXB marshaling, CSV marshaling, and HTTP. To
make the example easier to try out, He've opted to use the File endpoint instead
of the FTP. If we were using the FTP endpoint we would need to add a dependency
on the camel-ftp module as well.
Listing 1: Maven dependencies for the Camel implementation
Listing 1: Maven dependencies for the Camel implementation
01.<dependencies>02.<!-- Core Camel -->03.<dependency>04.<groupId>org.apache.camel</groupId>05.<artifactId>camel-core</artifactId>06.<version>${camel-version}</version>07.</dependency>08.<dependency>09.<groupId>org.apache.camel</groupId>10.<artifactId>camel-spring</artifactId>11.<version>${camel-version}</version>12.</dependency>13. 14.<!-- Embedded ActiveMQ broker -->15.<dependency>16.<groupId>org.apache.activemq</groupId>17.<artifactId>activemq-core</artifactId>18.<version>${activemq-version}</version>19.</dependency>20.<dependency>21.<groupId>org.apache.xbean</groupId>22.<artifactId>xbean-spring</artifactId>23.<version>${xbean-spring-version}</version>24.</dependency>25. 26.<!-- ActiveMQ connectivity for Camel --> 27.<dependency>28.<groupId>org.apache.activemq</groupId>29.<artifactId>activemq-camel</artifactId>30.<version>${activemq-version}</version>31.</dependency>32. 33.<!-- Add support for JAXB marshaling -->34.<dependency>35.<groupId>org.apache.camel</groupId>36.<artifactId>camel-jaxb</artifactId>37.<version>${camel-version}</version>38.</dependency>39. 40.<!-- Add support for CSV marshaling --> 41.<dependency>42.<groupId>org.apache.camel</groupId>43.<artifactId>camel-bindy</artifactId>44.<version>${camel-version}</version>45.</dependency> 46. 47.<!-- Add support for HTTP -->48.<dependency>49.<groupId>org.apache.camel</groupId>50.<artifactId>camel-jetty</artifactId>51.<version>${camel-version}</version>52.</dependency>53. 54.</dependencies>While it is perfectly legitimate to use Camel as a standalone Java application, it is often useful to embed it in a container. In this case, we will be loading Camel entirely from Spring. In fact, the entire solution (except for the domain POJO and Maven build script) fits neatly into a single Spring XML file. This is because we are implementing our routing rules using the Spring XML configuration rather than one of Camel’s other DSLs. There are pros and cons to each of course but in this case he mainly wanted to show off some cool tooling that currently works with the XML based routing rules only. The Spring XML file is shown in Listing 2 below.
Listing 2: Complete Spring XML file that configures an embedded ActiveMQ broker and initializes a Camel Context with three routes.
01.<beans xmlns="http://www.springframework.org/schema/beans"02.xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"03.xmlns:amq="http://activemq.apache.org/schema/core"04.xsi:schemaLocation="11. 12.<amq:broker brokerName="localhost" persistent="false" useJmx="false"/>13. 14.<bean id="jms" class="org.apache.activemq.camel.component.ActiveMQComponent">15.<property name="brokerURL" value="vm://localhost" />16.</bean>17. 18.<camelContext xmlns="http://camel.apache.org/schema/spring">19.<route id="FileToJMS">20.<from uri="file:target/placeorder" />21.<to uri="jms:incomingOrders" />22.</route>23. 24.<route id="HTTPtoJMS">25.<from uri="jetty:http://0.0.0.0:8888/placeorder" />26.<inOnly uri="jms:incomingOrders" />27.<transform>28.<constant>OK</constant>29.</transform>30.</route>31. 32.<route id="NormalizeMessageData">33.<from uri="jms:incomingOrders" />34.<convertBodyTo type="java.lang.String" />35.<choice>36.<when>37.<simple>${body} contains '?xml'</simple>38.<unmarshal>39.<jaxb contextPath="org.fusesource.camel" />40.</unmarshal>41.<to uri="jms:orders" />42.</when>43.<otherwise>44.<unmarshal>45.<bindy packages="org.fusesource.camel" type="Csv" />46.</unmarshal>47.<to uri="jms:orders" />48.</otherwise>49.</choice> 50.</route>51.</camelContext>52.</beans>
In this file we
first start an embedded Apache ActiveMQ broker and connect Camel to it. Next
comes the camelContext element where
we define our routing rules. Looking back at Figure 3, we need to receive
orders from an FTP (substituted with File in this example) and HTTP endpoint,
formatted as shown in Listing 3. In the Spring XML configuration we can specify
these incoming endpoints with two from
elements. The “FileToJMS” and “HTTPtoJMS” routes start off with these from elements. Both from elements are connected to a producer (to and inOnly elements)
using the “jms:incomingOrders"
URI, which will send the messages to a queue named incomingOrders on the ActiveMQ broker.
Listing 3: Incoming message formats; XML on the left, CSV on the right.
<?xml
version="1.0" encoding="UTF-8"?>
<order name="motor" amount="1"/> |
"name",
"amount"
"brake pad", "2" |
In the case of
the HTTP endpoint, there are a couple of extra things to mention. First off the
HTTP client will be expecting a response from the application so we have to
handle that. In Camel, we have full control over what the client gets back from
the HTTP endpoint. Each response is determined by the last method in our
current route definition. In our case we use the transform method to set the
response to the constant string "OK". Since we handle the response
ourselves, we don’t want any response to come from the JMS incomingOrders queue. To send to this queue in a fire-and-forget
fashion we use the inOnly element
rather than the to element.
The “NormalizeMessageData” route in Listing 2 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First we specify that we want to consume messages from the incomingOrders queue on the ActiveMQ broker. The content based routing of the messages is done with the choice, when, and otherwise elements. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message data format we have we use a Simple expression which checks the message body for the “<?xml” start tag. Simple is an expression language built in to Camel. Of course, this is demonstration code only; for production cases you would want to add more thorough checking of content types.
To unmarshal the XML and CSV payloads into an Order object, we use Camel’s data formats. Listing 2 shows how these data formats are defined within the unmarshal elements. As shown in Listing 5, the Order object has JAXB and Camel Bindy annotations to describe the mapping to XML and CSV.
Listing 5: The Order domain class with JAXB/Bindy annotations for easy mapping to and from XML/CSV.
The “NormalizeMessageData” route in Listing 2 specifies the Normalizer, complete with Content-Based Router and two Message Translators. First we specify that we want to consume messages from the incomingOrders queue on the ActiveMQ broker. The content based routing of the messages is done with the choice, when, and otherwise elements. In our case, we want to send CSV messages to one Message Translator and XML messages to another. To check what type of message data format we have we use a Simple expression which checks the message body for the “<?xml” start tag. Simple is an expression language built in to Camel. Of course, this is demonstration code only; for production cases you would want to add more thorough checking of content types.
To unmarshal the XML and CSV payloads into an Order object, we use Camel’s data formats. Listing 2 shows how these data formats are defined within the unmarshal elements. As shown in Listing 5, the Order object has JAXB and Camel Bindy annotations to describe the mapping to XML and CSV.
Listing 5: The Order domain class with JAXB/Bindy annotations for easy mapping to and from XML/CSV.
01.@XmlRootElement02.@XmlAccessorType(XmlAccessType.FIELD)03.@CsvRecord(separator = ",", skipFirstLine = true)04.public class Order implements Serializable {05.@XmlAttribute06.@DataField(pos = 1)07.private String name;08. 09.@XmlAttribute10.@DataField(pos = 2)11.private int amount;12. 13.public Order() {14.}15. 16.public Order(String name, int amount) {17.this.name = name;18.this.amount = amount;19.}20.}At this point, successfully normalized messages are sent to the orders queue for processing by some other application at the Rider Auto Parts business.
Implementation using Fuse IDE for Camel
As we’ve seen,
Apache Camel certainly makes it easy to create integrations. The DSLs in
various languages are designed to be concise, powerful and easy to read – all
with the goal of making you, the developer, more productive. With improving
productivity in mind, FuseSource has created an IDE for Camel development called
Fuse IDE for Camel.
Fuse IDE is an
Eclipse-based tool that allows you to drag and drop enterprise integration
pattern (EIP) icons onto a canvas and connect them together to form routing
rules. The backend of the IDE generates Camel routing rules in Spring XML for
you so you don’t ever have to learn the specifics of Camel’s DSLs. Essentially
you can create a picture like Figure 3 and have Fuse IDE generate the Camel
application for you.
Figure 4 shows what the Fuse IDE designer looks like when we loaded our example configuration from Listing 2.
Figure 4 shows what the Fuse IDE designer looks like when we loaded our example configuration from Listing 2.
Figure
4: Spring XML file in Listing 2 loaded into Fuse IDE’s designer.
From the
designer view we can see the graphical representation of each of the routes we
created in Spring XML beforehand. From this point we can change how the EIP
icons are connected, endpoint URI properties, add additional steps to the
routes, and so on. Also, as of version 1.0, Fuse IDE will support all EIPs and components available in
Apache Camel – so there is no waiting for version 2.0 to use your favorite
Camel features.
In this article
we started first by creating the Spring XML configuration by hand. This should
not mislead you however - there is no requirement to do this. In fact, since
Fuse IDE supports 1-to-1 round tripping between Camel Spring XML configuration
and the designer view, you can use whichever you prefer. Figure 5 shows the
source view of the Spring XML file. Any changes made to the routes in the source
view will be matched when you switch back to the designer view.
Figure
5: Source view in Fuse IDE.
When you have
finished constructing your routes, you can then take advantage of another neat
feature included in Fuse IDE: that is the ability to generate a JUnit test case
from any Camel Spring XML file. Once you are satisfied with your testing, you
can then debug in a local Spring container or deploy to your container of
choice – all from within the IDE. Of course, much more is planned for Fuse IDE
in the future so Camel development will only get easier.
Summary
In this article he've shown two common
problems that an integration developer may face: dealing with the specifics of
applications and transports, and coming up with good solutions to integration
problems. The Apache Camel project provides a nice answer to both of these
problems. As the example has shown, solving integration problems with Camel is
straight forward and results in very concise route definitions. Building on
that we saw how Fuse IDE for Camel makes Camel development even easier.
No comments:
Post a Comment