Over the last decade, businesses and governments have been giving increasing attention to business processes—to their description, automation, and management. This interest grows out of the need to streamline business operations, consolidate organizations, and save costs, reflecting the fact that the process is the basic unit of business value within an organization.
The design and automation of business processes even warrants its own field of study, known as BPM (business process management). A quote from IBM Systems Journal sums it up nicely: “BPM technology provides not only the tools and infrastructure to define, simulate, and analyze business process models, but also the tools to implement business processes in such a way that the execution of the resulting software artifacts can be managed from a business process perspective.”1
The level of interest and the concomitant marketing hyperbole around BPM has reached a crescendo. As an example, Gartner announced that BPM “wins the ‘Triple Crown’ of saving money, saving time, and adding value.” Is this promise being fulfilled? BPM does have the potential to deliver significant value, but there are missing elements that limit its effectiveness. Although BPM technologies are becoming more mature, many software developers and business analysts still find themselves asking the basics: What is it? And why should I care?
This article provides a high-level overview of BPM and where it is today and touches on some of the core technologies and standards. The focus is on four specific challenges facing BPM, which are aligned with the four phases of the application development life cycle:2
In each phase we explain the problem and explore its business and technical consequences.
Business process management is an old discipline that allows you to model the organizational structure, define the business processes, and show the interactions between them. It is traditionally taught in business schools and is put into practice with varying degrees of success. In the 1990s the work of Michael Hammer and James Champy,3 authors of the widely read Reengineering the Corporation, focused attention on business processes, both as a root cause of inefficiency and as the source of potential competitive advantage. They advised a deep and radical redesign of the business to root out waste and increase the focus on the customer. That was 10 years ago. What is different today is the novel use of computing technology to drive the analysis and automation of business processes.
In Beyond Engineering, a follow-up to Reengineering the Corporation, Hammer defines a business process as “a complete end-to-end set of activities that together create value for the customer.”4 The notion of “customer” in this context refers to the recipient of the value provided, not necessarily a paying customer in a commercial transaction.
Innovations in technology such as XML, Web services, component-based development, and message-oriented middleware have fueled the current interest in BPM. Vendors have developed BPMSs that provide the fine-grained integration of systems and data needed to automate business processes. The BPMS links people and systems, manages information access and transformation, handles exceptions, and orchestrates the flow of the process.
Organizations are looking to BPM to help solve the following kinds of problems:
In order to improve business processes, you must begin at the beginning. Typically, business analysts, needing to discover the current state of things, try to visually represent the various processes of the business. The approach taken by many BPM products fails to address this challenge adequately. They employ a drawing metaphor in which an analyst or developer sketches the process using a palette of standard icons. This assumes the existing process is known in advance. Most large organizations, however, simply do not know their end-to-end processes accurately or in detail. Their process knowledge is tacit and decentralized—not explicit and centralized.
How can you discover how a business operates? Classical manufacturing processes have been analyzed extensively in quantitative and qualitative terms. Discovering general business processes is somewhat less straightforward. You can adopt either a top-down or bottom-up approach. The top-down discovery approach typically begins with the organization chart. It lists the responsibilities of each department in the organization and identifies the high-level processes that support these responsibilities. These processes are then decomposed into lower-level processes, which are decomposed further, until the lowest level is reached. The advantage of this approach is that it provides a broad, organizational perspective. Its disadvantage is a lack of detail and a questionable degree of accuracy.
The bottom-up approach begins by interviewing employees about the day-to-day activities they perform and attempts to integrate this local information into coherent end-to-end processes. This approach can be extremely accurate but you can easily get lost in the details, failing to see the forest for the trees. Some hybrid top-down/bottom-up approaches seek to achieve the advantages of both methods.
Since no two organizations are exactly alike in how they operate, different discovery methods may be appropriate in different businesses. Further, a single capture of business processes is likely to be woefully inadequate. As is so often the case, later discoveries inform earlier ones, and an iterative discovery methodology that continually enhances and updates the processes may yield better results. Regardless of the methodology followed, there is of course the key requirement that senior management support and drive the business-process discovery. Without this support, the chance of success is minimal.
Business Consequences. As awareness of the importance of business processes grows, many companies are attempting to capture their current-state business processes. They form workshops of business users, sketch the processes, and try to achieve consensus within the team. Unfortunately, they have no way to ensure the accuracy or completeness of the resulting diagrams. Often the underlying complexity of existing business processes is oversimplified by such workshops. Much of the important meta-data about the processes, such as cost, cycle time, and information flow, cannot be easily fit into Visio diagrams. Moreover, the information contained in the processes cannot be analyzed effectively in these diagrams.
Technical Consequences. Accurate capture of current processes is a prerequisite for defining the structure and rules of the desired future process. Without this level of understanding, the developer may produce a suboptimal solution with the BPMS tool or, worse, a solution to the wrong problem. BPMS tools were designed to automate processes, not to analyze business. The process automation implemented using a BPMS may therefore fall short of the true goals of the business. Why not ask the business analysts to use the BPMS products to discover and analyze business processes? This would seem to solve this problem. Unfortunately, most BPMS products have been designed for developers, not for business analysts. They provide few if any capabilities for process discovery, defining meta-data, simulation, or analyzing process costs or cycle times.
The ultimate purpose of BPM is to improve and optimize the operations of an organization. Thus, its scope necessarily comprises not a single process but all processes across the organization. Unfortunately, BPMS products are generally designed to work on one process at a time. The developer draws the process, adds implementation constructs, and then executes the process. There is no way in these tools to look across multiple processes, examine process interconnections, make comparisons, or perform analysis.
It would be helpful to have a “process laboratory” of sorts, where business analysts could collaborate, exploring the process space, testing ideas, measuring, analyzing, and comparing processes, and generally performing business thought experiments. This laboratory would give analysts the tools to design new processes, view the processes from multiple points of view, extract analytical reports across processes, generate system requirements, and perform simulation and “what if” experiments. It would allow them to create centralized reusable processes that could be invoked by processes in different operating units.
Obviously, one key element in such a laboratory is a language to express ideas. This “process language” would be used to define the basic entities of business processes (processes, participants, activities, links, etc.) and the rules for their operation and interaction. A standard process language would allow customers to use products from different vendors for defining and implementing business processes. Processes defined in one product could be executed on another product.
Over the past three years, various groups have made numerous attempts to define standards for Web services and business processes. The relevant organizations are the Workflow Management Coalition (WfMC), Business Process Management Initiative (BPMI), World Wide Web Consortium (W3C), and OASIS.5
WfMC promotes the use of workflow by helping to establish standards for terminology, interoperability, and connectivity between workflow products. It has created xpdl, an XML-based definition for business processes.
The mission of BPMI is to promote and develop business process management by establishing standards for process design, deployment, execution, control, and optimization. BPMI has developed three standards: Business Processing Modeling Language, Business Process Modeling Notation, and Business Process Query Language.
Founded by Tim Berners-Lee, W3C has published the key standards for XML, HTTP, HTML, SOAP (Simple Object Access Protocol), and Web services. All W3C standards must be royalty free.
OASIS (Organization for the Advancement of Structured Information Standards) is a not-for-profit, global consortium that drives the development, convergence, and adoption of e-business standards. OASIS has defined standards for e-business applications (ebXML) and is working on other high-level Web services standards. In contrast to the W3C, standards published by OASIS may have royalties attached to them.
An important XML-based language for defining business processes is the Business Process Execution Language for Web Services (BPEL4WS or simply BPEL6), a language created by Microsoft, IBM, and BEA Systems. It grew out of work done by Microsoft (XLANG) and by IBM (Web Services Flow Language). Microsoft’s approach was inspired by structured programming, whereas IBM’s approach viewed a process as a directed graph. BPEL attempts to integrate these two viewpoints. The OASIS Web Services Business Process Execution Language Technical Committee is charted to continue work on the business process language originally published in August 2002.
The underlying assumption behind the BPEL specification is that business processes will be composed of a series of interacting Web services. Since WSDL (Web Services Description Language) is the natural language for describing Web services, the BPEL designers chose to make BPEL an extension of WSDL. Each activity in a BPEL process is implemented by a Web service, which is defined by its port types, operations, and messages.
In the conceptual scheme of BPEL, a business process is composed of a central process engine (see figure 1) that interacts with a set of business partners. Each Web-service operation is performed either by one of the partners or by the central process engine itself. The process engine communicates with its partners by exchanging messages.The process engine and partner send messages across a communications channel called a “service link” (see figure 2). To make this more concrete, let us consider a travel agency interacting with an airline. The service link is a bilateral contract between the travel agency process and the airline that defines the services each offers to the other.
The process and the partner play different roles across the service link. The process, in its role, exposes a WSDL port type—that is, a set of operations that it agrees to offer. Similarly, the partner, in its role, exposes another set of operations. In our example, when the travel agency process invokes the airline’s ticket-order operation, it is actually using a service link, which we have called the “buyerLink.” The role of the travel agency process in buyerLink is ticket requester and its port type is itinerary. The role of the airline partner is ticket service and its port type is ticket order.
Having defined this conceptual framework for partner interaction, BPEL goes on to specify the usual building blocks of processes: activities, flows, links, data containers, and assignment operations. These are defined in terms of their XML structure, but without a graphical model for representing them. The influence of Microsoft and IBM is apparent in these definitions. For example, if you want to specify that activity 1 must precede activity 2, you have two ways to do so. One way is to use the structured activity called “sequence.” The other way is to connect these activities through a link within a flow. The first approach is inherited from Microsoft’s XLANG, and the second is inherited from IBM’s WSDL. The BPEL specification provides no hints as to when to use one technique or the other.
BPEL also addresses technical constructs required for proper execution of the business process. These include correlation, fault handling, and compensation. Correlation is the technique used to create associations between process instances by using the data fields as identifiers. For example, a field “order number” might be used to correlate a purchase order and a purchase-order acknowledgment. Fault handling and compensation specify the procedures to be followed when an error occurs in the process. For long-running processes, the idea is to “undo” a complex series of activities with a compensating series of activities.
In BPEL, each process is an assemblage of Web services, but the process is itself a large-scale Web service. This fractal-like approach enables unlimited composition and decomposition of Web services.
BPEL is a significant achievement but has several weaknesses that limit its widespread adoption:
The major competitor to BPEL as a process language based on Web services is BPML (Business Process Management Language). Both BPEL and BPML are ultimately based on the π calculus, but the Business Process Management Initiative introduced BPML two years earlier than BPEL. BPEL has actually incorporated many BPML concepts and, with the support of Microsoft and IBM, it has the advantage of industry momentum. Microsoft, IBM, and BEA Systems will likely introduce BPEL in their products over the next year, perhaps with some proprietary language extensions and some tools to aid in process design.
Business Consequences. The goal is to translate the process information gathered in discovery into a standard process language, such as BPEL. But business analysts can’t use BPEL, at least not in its current form. Given a BPEL process definition, you will find it nearly impossible to disentangle the business logic from the details of the implementation. The business semantics are obscured by the technical details required for execution. What is lacking is a way to layer BPEL, to filter out such technical constructs as fault handling, correlation, and transaction management. It should be possible for a business analyst to use a process language to design the business logic. Once the business logic is mapped out, a developer can insert the technical constructs.
Business analysts generally prefer a visual way to design processes. They are intimidated by code, which tends to obscure the business issues. Also, nondevelopers need to see processes from multiple perspectives, to determine inter-process dependencies, and to extract system requirements from processes. Business analysts would also likely benefit from a way to design processes from the top-down, beginning with high-level processes and refining them to low-level processes. This lies outside the scope of BPEL. Because BPEL has no graphical rendering, the business analyst would have to code XML. This is not likely to happen.
Technical Consequences. In many cases the business analysts will design the business process using an analysis tool. Once this is done, how will the process then be automated? It would be desirable to export the process to the BPMS for execution, perhaps using BPEL as the lingua franca. If BPEL is not used by the business process analysis tool as well as by the BPMS, then a custom mapping will be required to translate between the XML dialects of the two products.
In its current form, BPEL will be of limited use since it is designed for processes that are implemented using Web services. It is not usable for “legacy processes” that comprise package applications, custom applications, or manual activities.7 This excludes 95 percent of all existing business processes.
A key goal of business-process design is to define and communicate requirements. For example, suppose a new system will be used in multiple processes, supporting different users and systems performing different functions. Ideally, the business processes are defined in such a way that the functional requirements can simply be extracted from the process definitions. If our new system performs 25 activities across 17 processes, then these activities can be summarized by an appropriate query. This procedure is precisely analogous to extracting data from a relational database. Unfortunately, there is currently no searchable knowledge base of BPEL processes. Moreover, since BPEL has no notion of “participant” or “actor” to identify the proposed system, it will not be able to support this important goal of the process laboratory.
A successful development project is the result of many favorable conditions, one of the most important being close collaboration between business analysts, who define the needs of the business, and developers, who implement systems that meet these needs. Business analysts and technical developers, however, are driven by different goals, speak different languages, and tend to work at different levels of precision. In the domain of BPM this gap is manifested by automated business processes whose execution does not match the original business requirements.
Business analysts typically communicate business needs in the form of requirements documents to the developers, who may interpret the needs rather differently. Using a BPMS tool, the developers implement the automated solution as they understand it. Why not have the business analysts define the business process using the BPMS tools? This is not realistic since BPMS products are generally intended for use by developers, not by business analysts. They enable developers to create technical constructs, not business requirements. Tools based on UML (Unified Modeling Language) are sometimes suggested as an alternative. UML is well suited for developers who need to design class diagrams and lay out the interactions between method calls. The UML suite of diagrams is, however, not so well suited for business analysts working with business processes.
Business Consequences. The business consequences of the gap between business analysts and developers are increased risk of failure and longer lead times for development. In its research into the success rate of application development projects, the Standish Group discovered that most software development projects are cancelled before they are completed.8 It also found that fewer than 20 percent of projects are completed on time and on budget. Of those projects that do deliver on time and budget, half fail to deliver the projected functionality. The Standish Group notes that a key cause of such failures is the lack of clarity in capturing and communicating user requirements.
Technical Consequences. The classical requirements document leaves considerable room for interpretation. It rarely provides the IT level of precision needed by developers, who need to know each activity that must be performed by the system, the step-by-step control logic of the enveloping business process, the specific data required at each step of the process, and the business rules that govern changes to the data. This lack of precision may lead to misunderstandings between the analyst and the development team. Software misses the mark, making scrap and rework inevitable. Redefining requirements, redesigning, and recoding midstream are expensive and time-consuming.
What is missing is a seamless way to integrate design and development. Business analysts should create process designs at the business level within their process laboratory and then export these designs in XML form to the BPMS. Developers will then refine the design, adding the technical constructs needed for implementation. This eliminates any ambiguity about the requirements and business need.
The whole point of automating business processes is to improve operations—in cost, time, or quality. Once a process has been developed and deployed, how can we know if it is meeting the intended goals? We know how to instrument IT systems and monitor them with a high degree of precision. These statistics, however, do not generally provide a business-process context around this information. The challenge is to aggregate and present execution data at the business-process level. Gartner coined the term business activity monitoring (BAM) for this capability. It defines BAM as providing “realtime access to critical business performance indicators to improve the speed and effectiveness of business operations.”9
Business Consequences. Without BAM, operational managers have no way of determining whether the processes for which they are responsible are meeting their objectives. For example, they will not be aware that the cost of the order fulfillment process in the Cincinnati region has increased 20 percent above average, or that the time required for handling new benefits claims declined by 10 percent in the second quarter, or that the outage optimization process for steam turbine generators is in trouble. Lacking this information, executives have no way to determine which action to take. A way to aggregate execution statistics in process context would help the business manager better manage these types of exceptions.
Over the past year, several companies have developed BAM products, but in many cases they are discrete event monitors that lack overall process context. To achieve true process context, you must link individual activities into a process to provide information on what is done in that process and by whom. For example, it is necessary to evaluate the cost of each process instance. Moreover, it is necessary to aggregate process instance information based on various criteria. For example, you can group process instances by geography, customer, or organization. Finally you need to “chain” processes that logically belong together, such as an order process and an invoice process. All this information should be summarized and presented on an executive dashboard on the enterprise portal.
Technical Consequences. At some level, most companies recognize the importance of BAM but have no effective way to collect, aggregate, and analyze execution statistics. Often it is done in an ad hoc manner, in which reports from legacy systems are combined into a data warehouse from which summary reports can be generated. It is possible to determine quite precisely the utilization of each disk drive, server, and network component in the IT environment. From such statistics, however, you will not know which resources need to be expanded, consolidated, or upgraded to support the business objectives. For example, you simply may not know whether or not increasing the capacity of a specific disk will affect the order-fulfillment-cycle time.
With BAM, we come full cycle. The results of process monitoring will enable the rediscovery and redesign of business processes. Executives will know about hotspots that demand their immediate attention. In the longer term, the execution will keep pace with the business needs and the process designs. Figure 3 summarizes this life-cycle concept.
Any enterprise—corporation, government, or nonprofit organization—can be viewed as the sum of its business processes. Each process delivers value to customers, suppliers, employees, or other stakeholders. BPM, the discipline for enabling and automating business processes, is in a period of rapid growth and will fundamentally change the way computing power is applied in organizations. Whereas BPM has already delivered considerable value in many companies, the components of the full BPM solution are still evolving and are the subject of ongoing research and development. One noteworthy advance has been BPEL, an XML-based language for describing business processes composed of Web services.
This article has focused on four immediate challenges:
We’ve made significant progress in BPM over the past three years. We expect many of these challenges to be addressed in the next three years. Others may prove less tractable and will take a little longer to solve. Once this happens, for the first time we will have a complete, closed-loop approach to business processes: from conception to execution and back. This will give executives the ability to design their business processes, automate them, and determine quantitatively how well they are doing against plan. With this information, they can then redesign or optimize the processes.
Gradually, the technologies and techniques described here will change the way businesses and governmental agencies apply technology. In the words of Howard Smith, “Third-wave business process management methods and systems will utterly transform the way companies conceive, build, and operate automated systems.”10
1. Leyman, F., Roller, D., and Schmidt, M.-T. Web services and business process management. IBM Systems Journal 41, 2 (2002) 198.
2. CSC’s Catalyst 4D development methodology.
3. Hammer, M., and Champy, J. Reengineering the Corporation. New York: HarperCollins, 1993.
4. Hammer, M. Beyond Reengineering. New York: HarperBusiness, 1996.
5. Workflow Management Coalition: see http://www.wfmc.org; Business Process Management Initiative: see http://www.bpmi.org; World Wide Web Consortium: see http://www.w3.org; OASIS: see http://www.oasis-open.org.
6. Business Process Execution Language for Web Services, Version 1.0. IBM: see http://www.ibm.com/developerworks/library/ws-bpel/.
7. Unless they have been previously wrapped as Web services.
8. The Standish Group. The Chaos Report (1994), http://www.standishgroup.com/sample_research/chaos_1994_1.php.
9. McCoy, D. Business activity monitoring (BAM)—deeper meanings, Business Integration Journal (Aug. 2003), http://www.bijonline.com/Article.asp?ArticleID=755.
10. Smith, H., and Fingar, P. Business Process Management: The Third Wave. Tampa: Meghan-Kiffer Press, 2002.
LAURY VERNER (firstname.lastname@example.org) is chief technology officer of ProActivity, a Boston-based company that develops software for business process analysis and design. Prior to that, Verner was a partner at Computer Sciences Corporation Consulting, specializing in application architecture. Verner has a Ph.D. from the Johns Hopkins University and a B.A. from the University of Pennsylvania.
Originally published in Queue vol. 2, no. 1—
see this item in the ACM Digital Library
Eric Bouwers, Joost Visser, Arie Van Deursen - Getting What You Measure
Four common pitfalls in using software metrics for project management
Tracy Ragan - Keeping Score in the IT Compliance Game
Achieving developer acceptance of standardized procedures for managing applications from development to release is one of the largest hurdles facing organizations today. Establishing a standardized development-to-release workflow, often referred to as the ALM (application lifecycle management) process, is particularly critical for organizations in their efforts to meet tough IT compliance mandates. This is much easier said than done, as different development teams have created their own unique procedures that are undocumented, unclear, and nontraceable.
Duncan Johnston-Watt - Under New Management
In an increasingly competitive global environment, enterprises are under extreme pressure to reduce operating costs. At the same time they must have the agility to respond to business opportunities offered by volatile markets.
Derek Miers - Best Practice (BPM)
Just as BPM (business process management) technology is markedly different from conventional approaches to application support, the methodology of BPM development is markedly different from traditional software implementation techniques. With CPI (continuous process improvement) as the core discipline of BPM, the models that drive work through the company evolve constantly. Indeed, recent studies suggest that companies fine-tune their BPM-based applications at least once a quarter (and sometimes as often as eight times per year). The point is that there is no such thing as a “finished” process; it takes multiple iterations to produce highly effective solutions.