AMQP (Advanced Message Queuing Protocol) was born out of my own experience and frustrations in developing front- and back-office processing systems at investment banks. It seemed to me that we were living in integration Groundhog Day - the same problems of connecting systems together would crop up with depressing regularity. Each time the same discussions about which products to use would happen, and each time the architecture of some system would be curtailed to allow for the fact that the chosen middleware was reassuringly expensive.
From 1996 through to 2003 I was waiting for the solution to this obvious requirement to materialize as a standard, and thereby become a commodity. But that failed to happen, and I grew tired of waiting.
Consequently, AMQP was created and had its first mission-critical deployment to production in mid-2006. That project paid for itself with its first deployment, serves 2,000 users, and processes 300 million messages per day.
This article sets out the motivations, capabilities, and credentials of AMQP and offers it as a practical solution for a standards-based messaging infrastructure.
AMQP is a binary wire protocol and well-defined set of behaviors for transmitting application messages between systems using a combination of store-and-forward, publish-and-subscribe, and other techniques.1 I use the term application messages to distinguish AMQP from instant messaging or other forms of end-user messaging. AMQP addresses the scenario where there is likely to be some economic impact if a message is lost, does not arrive in a timely manner, or is improperly processed.
The protocol is designed to be usable from different programming environments, operating systems, and hardware devices, as well as making high-performance implementations possible on various network transports including TCP, SCTP (Stream Control Transmission Protocol), and InfiniBand.
Every major investment bank on Wall Street has at some point built its own messaging middleware. Many have either faded away or spun off to become commercial proprietary solutions.
Why do they build their own middleware? The financial services industry has some of the most demanding needs for messaging both in guaranteed delivery and in publish-subscribe. Demands often exceed the capabilities of currently available software, and there is no shortage of technology expertise in banks. Building one's own middleware is therefore a credible approach.
Banks are looking for high-performance service buses from which to hang their system architectures. Web services are not fitting the bill because they are too compute- and bandwidth-intensive per unit of work.
The growth of automated trading is also igniting interest in improving middleware. Banks are still pushing the envelope with market data events exceeding 500,000 per second at source, according to the Options Price Reporting Authority. Processing that flood of information and executing timely business transactions off the back of it is challenging. Market data volumes exacerbate transaction-processing volumes.
Given the clear need, why do many internal efforts not endure? Despite their technical abilities, banks are not software houses; messaging middleware is complex software, and it is difficult for a bank to focus the level of intellect and talent on the problem for a long time.
Banks have managed to work together in creating open technical standards where such standards are absolutely necessary to doing business; the FIX (Financial Information Exchange) protocol, FAST (FIX Adapted for Streaming), FpML (Financial products Markup Language), and SWIFT (Society for Worldwide Interbank Financial Telecommunication) are all good examples.
In 2003 I embarked on a quest to standardize MOM (message-oriented middleware) technology outside my firm, so we could pick it up inside the firm and use it between firms.
This had to be an industry initiative. Home-grown middleware could not thrive in the small market available within a host organization, even the largest host.
It is also notable that pervasive networking standards such as Ethernet, the Internet Protocol, e-mail, and the Web share some traits. They are all royalty-free and unencumbered by patents, they are all publicly specified, and they all shipped with a useful early implementation for free. The combination of freedom and usefulness drives their adoption when predicated on fitness for purpose.
To succeed, AMQP needed to adopt these same characteristics:
Because the project was sponsored by a bank, it also had to "wash its own face," as they say. This was not a research project. Through sheer good luck, there was a need to refresh some large systems with very specific requirements. This provided a tangible return for AMQP investment, so I was able to convince a forward-looking CIO that AMQP was the way to go.
When the shape of AMQP had been worked out between JPMorgan and iMatix, and the basics of the specification forged in the heat of an initial implementation, the time was right to extend the partnership and encourage others to bring their talents to the specification and share ownership of AMQP's future.
The heart of openness and trust in any group effort is effective governance. Expanding the group required a new contractual framework and a plan for the end game where AMQP could become a standard. Red Hat took the lead in establishing the legal framework for the standard; it, too, understood the issues in managing open intellectual property. The key part of doing this is to ensure that everyone contributing has the authority to do so and that there is a paper trail from every potential owner of IP through to the group effort, and that the intent to share is clear even in draft revisions of specifications. The result was a contract that clearly committed the members of the working group to promote unrestricted open middleware through AMQP.
The members of the working group have granted licenses to the necessary parts of their patent portfolios to anyone who wants to implement AMQP. You can see the license grant in the specification itself. This level of contribution shows the commitment of the group to open middleware. The AMQP Working Group's Web site is http://www.amqp.org.
The AMQP Working Group is quite unique in technology standards work because of the heavy involvement of users. JPMorgan, Credit Suisse, TWIST, and to some degree Cisco are more end users than developers. This balance leads to a group of people interested in solving the problem, not pandering to technology agendas or product agendas.
From the beginning, AMQP's design objective was to define enough MOM semantics (see figure 1) to meet the needs of most commercial computing systems and to do so in an efficient manner that could ultimately be embedded into the network infrastructure. It's not just for banks.
AMQP encompasses the domains of store-and-forward messaging, publish-and-subscribe messaging, and file transfer. It incorporates common patterns to ease the traversal of firewalls while retaining security, and to permit network QoS. To ease adoption and migration, AMQP is also designed to encompass JMS (Java Message Service) semantics. JMS is a hugely popular API for Java programmers and cannot be ignored. AMQP goes further, however, and includes additional semantics not found in JMS that members of the working group have found useful in delivering large, robust systems over the decades. Interestingly, AMQP does not itself specify the API a developer uses, though it is likely that will happen in the future.
An example feature not found in JMS is AMQP's Mandatory Delivery Mode, in which a client can use AMQP to request services from a pool of servers connected to AMQP broker queues. The AMQP broker can load-balance requests among the services subscribed to a request queue, and the number of processes providing a service can dynamically grow and shrink with no impact on the client(s). If the service pool shrinks to zero, however, the client can be informed by AMQP using the Mandatory Delivery Mode since that may be an operational error for the application.
AMQP also specifies a small wire-level type system for message properties, enabling them to be read efficiently by many programming languages, as well as by the MOM servers themselves for filtering and routing purposes. Thus, not only can a Python client read headers set in Java servers, but different vendors can relay messages between their implementations seamlessly. The type system, however, suffers the usual problem in that object types cannot be transported in the headers; what would a Cobol program do with a Smalltalk object?
Finally, AMQP draws heavily on the heritage of IETF open standards. It tries not to reinvent existing concepts. Early versions of the AMQP wire protocol were influenced by SMTP,3 MIME,4 HTTP-NG,5 NFSv4,6 SCTP,7 BEEP,8 and the writings of Marshal Rose,9 an IETF veteran.
AMQP is split into two main areas: transport model and queuing model. AMQP is unusual in that it thoroughly specifies the semantics of the services it provides within the queuing model; since applications have a very intimate relationship with their middleware, this needs to be well defined or interoperability cannot be achieved. In this respect, AMQP semantics are more tightly defined than JMS semantics.
As stated, AMQP's transport is a binary protocol using network byte ordering. We wanted to make it easy to embed AMQP inside the ASICs (application-specific integrated circuits) of network elements, by design. With free tools such as Wireshark, it is not necessary to use XML for technical infrastructure layers that only specialists will see. XML has been used in the likes of BEEP and XMPP: in the case of BEEP it complicates the protocol; in the case of XMPP it is limited to being carried on a stream-oriented transport. AMQP aims to be high performance and flexible, to be hardware friendly rather than human friendly. The protocol specification itself, however, is written in XML so implementers can code-generate large portions of their implementations; this makes it easier for vendors to support the technology.
The transport model itself can reuse different underlying transports. The first is TCP/IP, but by adopting SCTP, we can obtain better parallelism for messages (SCTP removes the byte-stream head-of-line blocking problem imposed by TCP). There are also planned mappings to UDP to support AMQP over multicast, and bindings to InfiniBand are planned. InfiniBand's performance is generating a lot of interest at banks, and AMQP would make it very accessible to developers.
TCP/IP, however, is expected to be the default choice for most end users for best interoperability. With these options emerging, it is important for AMQP to establish a useful functional default set of capabilities that all implementations must adhere to or suffer the lowest-common-denominator problem that plagued protocols such as early versions of NFS (many servers did not implement file locking). Hopefully, a compliance testing kit will address this issue.
Messages in AMQP are self-contained and long-lived, and their content is immutable and opaque. The content of messages is essentially unlimited in size; 4GB messages are supported just as easily as 4KB messages. Messages have headers that AMQP can read and use to help in routing.
You can liken this to a postal service: a message is the envelope, the headers are information written on the envelope and visible to the mail carrier, who may add various postmarks to the envelope to help deliver the message. The valuable content is within the envelope, hidden from and not modified by the carrier. The analogy holds quite well, except that it is possible for AMQP to make unlimited copies of the messages to deliver if required.
Queues are the core concept in AMQP. Every message always ends up in a queue, even if it is an in-memory private queue feeding a client directly. To extend the postal analogy, queues are mailboxes at the final destination or intermediate holding areas in the sorting office.
Queues can store messages in memory or on disk. They can search and reorder messages, and they may participate in transactions. The administrator can configure the service levels they expect from the queues with regard to latency, durability, availability, etc. These are all aspects of implementation and not defined by AMQP. This is one way commercial implementations can differentiate themselves while remaining AMQP-compliant and interoperable.
Exchanges are the delivery service for messages. In the postal analogy, exchanges provide sorting and delivery services. In the AMQP model, selecting a different carrier is how different ways of delivering the message are selected. The exchange used by a publish operation determines if the delivery will be direct or publish-and-subscribe, for example. The exchange concept is how AMQP brings together and abstracts different middleware delivery models. It is also the main extension point in the protocol.
A client chooses the exchange used to deliver each message as it is published. The exchange looks at the information in the headers of a message and selects where they should be transferred to. This is how AMQP brings the various messaging idioms together - clients can select which exchange should route their messages.
Several exchanges must be supported by a compliant AMQP implementation:
Throughout this process, exchanges never store messages, but they do retain binding parameters supplied to them by the clients using them. These bindings are the arguments to the exchange routing functions that enable the selection of one or more queues.
The arguments supplied to exchanges to enable the routing of messages are known as bindings (see figure 2). Bindings vary depending on the nature of the exchange; the direct exchange requires less binding information than the headers exchange. Notably, it is not always clear which entity should provide the binding information for a particular messaging interaction. In the direct exchange, the sender is providing the association between a routing key and the desired destination queue. This is the origin of the "destination" addressing idiom so common to JMS and other queuing products.
In the topic exchange, it is the receiving client that provides the binding information, specifying that when the topic exchange sees a message that matches any given client(s') binding(s), the message should be delivered to all of them.
AMQP has no concept of a "destination," since it does not make sense for consumer-driven messaging. It would limit its abstract routing capabilities. The concept of bindings and the convention of using a routing key as the default addressing information overcome the artificial divisions that have existed in many messaging products.
A standard is nothing without implementations. AMQP has several available now. The first implementation is iMatix OpenAMQ (openamq.org), which is a C implementation of the server in production use at JPMorgan.
Apache's Qpid project has entered incubation (incubator.apache.org/projects/qpid.html). It will be Apache's multilanguage implementation of AMQP, with servers available in C++ and Java, and clients available in C, C++, Java JMS, Python, and Ruby on Rails, with more to follow. Qpid has a very active development community and is making rapid progress toward its M2 release. Also, Qpid is being used as the basis for several commercial offerings, notably from IONA and Red Hat.
Most recently, and intriguingly, Rabbit Technologies has developed an implementation on the Erlang/OTP (Open Telecom Platform), building on that language's strong support for parallelism and networking.
Of course, you should be able to mix and match client and server components from any vendor product - Qpid's Java client talking to RabbitMQ's Erlang server, for example.
The best way to learn more about AMQP is to visit amqp.org, where you can download the specification, try one of the free implementations, or even build your own. Both the protocol working group and the implementation groups are open to review and feedback; we want this protocol to be useful, successful, and inclusive.
In addition to being used to support organizations' internal messaging needs, AMQP is useful as a standard transport for other standards; after all, open standards should be built using other open standards.
Many business messaging standards describe business transactions, but there are only a relatively small number of transport options. HTTP is often used simply because it is perceived to be generally available, and firewalls are often configured to let HTTP traffic pass. HTTP offers only basic push messaging semantics, however; it does not enable any kind of call-back or event notification, and it requires the applications that use it to build their own message store, reliability, and access control facilities on top of it. EDI AS2 and WS-RM are examples of standards layered on HTTP; but these describe only message transport, not the semantics of how a message is delivered to an application. This leaves the developer with a partial solution and more problems to solve.
By offering AMQP as a standard for business messaging, we can make the full richness of messaging middleware available to other standards. Using AMQP servers, we can remove the burden of providing availability and reliability from end-user applications, making them simpler, cheaper, and more functional. The only requirement is that an AMQP server is run and that TCP/IP port 5672 be opened in the firewall. Given that any commercial activity involves lawyers and contracts, opening a port is a small price to pay to gain rich messaging functionality.
The TWIST standards organization, which promotes standards around financial supply chain management, is a member of the AMQP working group and promotes AMQP as its preferred standard for transporting business messages.
The FIX community is also looking at AMQP as a possible transport layer for FIX 5. FIX is a popular trading protocol that has recently been extended to support efficient delivery of market data; the current session layer, however, cannot provide publish-subscribe or scalable guaranteed delivery. AMQP offers FIX users the chance to get the features they need, while remaining open.
SOA (service-oriented architecture) is a technique for building large systems out of highly cohesive services loosely coupled to provide business processes. SOA is not a new concept. It was well known in the mid-'90s and earlier, but in its latest incarnation it is being pitched as Web service-oriented architecture. The architectural pattern needs neither HTTP nor XML.
One of the most fundamental parts of SOA is the communications mechanism that links the services. The term EMB (enterprise message bus) is closely linked to SOA.
Traditional deployments of EMBs have used proprietary technology, but enterprises would rather have standards-based solutions that are open, as well as the ability to choose between and switch suppliers, and the improvements that competition brings. AMQP, therefore, represents an ideal choice for an EMB. It allows a clean migration away from proprietary protocols and provides an avenue into other standards, including Web services.
Web services has four basic parts: service description, XML message content, service discovery, and transport. The transport is commonly presumed to be HTTP, but it does not have to be. Enterprises often use XML over messaging middleware as the transport for all the benefits that brings. Having done this, enterprises find they have created the problem they wanted to avoid: running an open architecture over a proprietary transport. Combining Web services with AMQP as a transport gives the richness an enterprise needs with the openness it craves in its core architecture.
Use of AMQP in SOA is just beginning, and you don't need anything other than AMQP to do it. I am already involved in a project to migrate an existing mission-critical EMB from a proprietary middleware to AMQP, so we can cost-effectively scale the bus to many more systems.
AMQP is a complete middleware protocol. It is not a lowest-common-denominator solution, and the only political design constraint is its explicit support for JMS semantics. Obviously, software that implements the AMQP specification will be able to interoperate, even where that software is from different suppliers. That means that the JMS client from product A would be able to talk to the C++ server from product B and send messages to a Ruby client in product C. Of course, in the real world there will be some teething difficulties, and the AMQP working group is starting to focus on how to create and manage a protocol compliance test suite to mitigate that risk.
There are many middleware products, however, each with its internal architecture. The AMQP Working Group encourages middleware vendors to implement AMQP in their products, but it will be nontrivial to get a good semantic match, and some vendors may be reluctant to support interoperability for commercial reasons.
In the meantime, there will be a need for bridging from the installed base of proprietary products. The easiest way to do this may be to use one of the many commercial or open source EAI (enterprise application integration) packages, but we expect several AMQP products to include native bridges to the most common proprietary middleware soon.
The most natural way for an organization to adopt AMQP is to deploy it opportunistically or as part of a strategy to move to standards-based EMB. This approach can enable an organization to benefit from competitively priced or open source solutions and gain experience with the protocol and products that support it. This is the approach we took: using AMQP in isolated systems and then branching into core EMB systems.
Over the course of a few years, much of your messaging may become AMQP-enabled by following this approach. Your current middleware supplier may adopt AMQP and enable your move to a standards-based model in that way. It is likely that AMQP will eventually be provided as a core service by your network infrastructure, in hardware.
On the other hand, if your company is embarking on SOA (or other bus-based architecture), we suggest that you seriously consider an en-masse deployment of AMQP as the backbone of SOA. Doing so may position you to benefit from competition among suppliers of AMQP-compatible middleware so you may achieve the levels of support and qualities of service you need from a suitable supplier. Deploying AMQP may also mitigate the worrisome issue of vendor lock-in or supplier failure for the critical bus component of an SOA, which is a long-term investment for your company. This same thinking applies where open protocols for e-business are being deployed between trading partners over the Internet or leased lines. Of course, whatever strategy you choose is yours alone and must be determined by your circumstances.
After two decades of proprietary messaging middleware, a credible standards-based alternative exists at last. The AMQP Working Group is rapidly evolving the protocol and hopes to reach version 1.0 during 2008, but implementations are available today that are both useful and proven in real-world deployments.
AMQP lets more applications be enterprise-grade without the costs associated with that label. It provides a capable messaging backbone that can be a springboard for new innovations and new kinds of applications.
R. JOHN O'HARA is a senior architect and distinguished engineer at JPMorgan. He drove the creation of AMQP and was an early contributor to the FpML financial markets messaging standard. He is chairman of the AMQP Working Group. His interests include data center virtualization, creative uses of open source for commercial data processing, event and data replication middleware, and making software architecture "simple and intuitive." O'Hara holds an honors degree in electronic engineering and computer science from Aston University, England.
Originally published in Queue vol. 5, no. 4—
see this item in the ACM Digital Library
Ben Maurer - Fail at Scale
Reliability in the face of rapid change
Aiman Erbad, Charles Krasic - Sender-side Buffers and the Case for Multimedia Adaptation
A proposal to improve the performance and availability of streaming video and other time-sensitive media
Ian Foster, Savas Parastatidis, Paul Watson, Mark McKeown - How Do I Model State? Let Me Count the Ways
A study of the technology and sociology of Web services specifications
Steve Souders - High Performance Web Sites
Google Maps, Yahoo! Mail, Facebook, MySpace, YouTube, and Amazon are examples of Web sites built to scale. They access petabytes of data sending terabits per second to millions of users worldwide. The magnitude is awe-inspiring. Users view these large-scale Web sites from a narrower perspective. The typical user has megabytes of data that are downloaded at a few hundred kilobits per second. Users are not so interested in the massive number of requests per second being served; they care more about their individual requests. As they use these Web applications, they inevitably ask the same question: "Why is this site so slow?"