January/February issue of acmqueue

The January/February issue of acmqueue is out now

Quality Assurance

  Download PDF version of this article PDF

Model-based Testing: Where Does It Stand?

MBT has positive effects on efficiency and effectiveness, even if it only partially fulfills high expectations.

Robert V. Binder, Bruno Legeard, and Anne Kramer

You have probably heard about MBT (model-based testing), but like many software-engineering professionals who have not used MBT, you might be curious about others' experience with this test-design method.

From mid-June 2014 to early August 2014, we conducted a survey to learn how MBT users view its efficiency and effectiveness. The 2014 MBT User Survey, a follow-up to a similar 2012 survey (http://robertvbinder.com/real-users-of-model-based-testing/), was open to all those who have evaluated or used any MBT approach. Its 32 questions included some from a survey distributed at the 2013 User Conference on Advanced Automated Testing. Some questions focused on the efficiency and effectiveness of MBT, providing the figures that managers are most interested in. Other questions were more technical and sought to validate a common MBT classification scheme. A common classification scheme could help users understand both the general diversity and specific approaches.

The 2014 survey provides a realistic picture of the current state of MBT practice. This article presents some highlights of the survey findings. The complete results are available at http://model-based-testing.info/2014/12/09/2014-mbt-user-survey-results/.


Survey Respondents

A large number of people received the 2014 MBT User Survey call for participation, in both Europe and North America. Additionally, it was posted to various social-networking groups and software-testing forums. Several tool providers helped distribute the call. Last but not least, ETSI (European Telecommunications Standards Institute) supported the initiative by informing all participants at the User Conference on Advanced Automated Testing.

Exactly 100 MBT practitioners responded by the closing date. Not all participants answered every question; the number of answers is indicated if considerably below 100. Percentages for these partial response sets are based on the actual number of responses for a particular question.

The large majority of the respondents (86 percent) were from businesses. The remaining 14 percent represented research, government, and nonprofit organizations. The organizations ranged in size from three to 300,000 employees (figure 1).

Model-based Testing: Where Does It Stand? Survey Participants Come from Organizations of All Sizes

As shown in figure 2, almost half of the respondents had moved from evaluation and pilot to rollout or generalized use. On average, respondents had three years of experience with MBT. In fact, the answers ranged from zero (meaning "just started") to 34 years.

Model-based Testing: Where Does It Stand? 48% of the Respondents Routinely Use MBT, and 52% Are Still in the Evaluation or Trial Phase

To get an impression of how important MBT is compared to other test-design techniques, the survey asked for the percentage of total testing effort spent on MBT, hand-coded test automation, and manual test design. Each of the three test-design methods represented approximately one-third of the total testing effort. Thus, MBT is not a marginal phenomenon. For those who use it, its importance is comparable to other kinds of test automation.

Nearly 40 percent of the respondents came from the embedded domain. Enterprise IT accounted for another 30 percent, and Web applications for roughly 20 percent. Other application domains for the system under test were software infrastructure, communications, gaming, and even education. The exact distribution is given in figure 3.

Model-based Testing: Where Does It Stand? Various Application Domains Represented

The role of external MBT consultants turned out to be smaller than expected. Although we initially speculated that MBT was driven mainly from the outside, the survey showed a different picture. A majority (60 percent) of those who answered the survey were in-house MBT professionals. Only 18 percent of respondents were external MBT consultants, and 22 percent were researchers in the MBT application area (figure 4).

Model-based Testing: Where Does It Stand? 3 in 5 Respondents are In-house MBT Professionals


Does MBT Work as Expected?

In our projects, we have observed that the expectations regarding MBT are usually very, if not extremely, high. "Does MBT fulfill expectations?" The MBT user survey asked whether the expectations in five different categories are being met: testing is becoming cheaper; testing is better; the models are helping manage the complexity of the system with regard to testing; the test design is starting earlier; and, finally, models are improving communication among stakeholders.

Figure 5 shows the number of responses and the degree of satisfaction for each of the five categories. The answers reflect a slight disenchantment. MBT does not completely fulfill the extremely high expectations in terms of cost reduction, quality improvement, and complexity, but, still, more than half of the respondents were partially or completely satisfied (indicated by the green bars in figure 5). For the two remaining categories, MBT exceeds expectations. Using models for testing purposes definitely improves communication among stakeholders and helps initiate test design earlier.

Model-based Testing: Where Does It Stand? Comparison Between Expectations and Observed Satisfaction Level

Overall, the respondents viewed MBT as a useful technology: 64 percent found it moderately or extremely effective, whereas only 13 percent rated the method as ineffective (figure 6). More than 70 percent of the respondents stated that it is very likely or extremely likely that they will continue with the method. Only one respondent out of 73 rejected this idea. Participants self-selected, however, so we cannot exclude a slight bias of positive attitude toward MBT.

Model-based Testing: Where Does It Stand? Nearly All Participants Rate MBT as Being Effective (to Different Degrees)

To obtain more quantitative figures on effectiveness and efficiency, the survey asked three rather challenging questions:

• To what degree does MBT reduce or increase the number of escaped bugs—that is, the number of bugs nobody would have found before delivery?

• To what degree does MBT reduce or increase testing costs?

• To what degree does MBT reduce or increase testing duration?

Obviously, those questions were difficult to answer, and it was impossible to deduce statistically relevant information from the 21 answers obtained in the survey. Only one respondent clearly answered in the negative regarding the number of escaped bugs. All others provided positive figures, and two answers were illogical.

On the cost side, the survey asked respondents how many hours it took to become a proficient MBT user. The answers varied from zero to 2,000 hours of skill development, with a median of two work weeks (80 hours).


What Kind of Testing is Model-Based?

Model-based testing is used at all stages of software development, most often for integration and system testing (figure 7). Half of the survey respondents reported model-based integration testing, and about three-quarters reported conducting model-based system testing. Nearly all respondents used models for functional testing. MBT played a subordinate role in performance, usability, and security testing, with less than 20 percent usage for each.

Model-based Testing: Where Does It Stand? MBT Plays an Important Role at All Test Levels

Only one participant found it difficult to fit MBT into an agile approach, while 44 percent reported using MBT in an agile development process, with 25 percent at an advanced stage (rollout or generalized use).

There was a clear preference regarding model focus: 59 percent of the MBT models (any model used for testing purposes is referred to here as an MBT model) used by survey respondents focused on behavioral aspects only, while 35 percent had both structural and behavioral aspects. Purely statistical models played a minor role (6 percent). The trend was even more pronounced for the notation type. Graphical notations prevailed with 81 percent; only 14 percent were purely textual MBT models—that is, they used no graphical elements. All kinds of combinations were used, however. One respondent put it very clearly: "We use more than one model per system under test."


Test Modeling Independent of Design Modeling

Note that more than 40 percent of the respondents did not use modeling in other development phases. Only eight participants stated that they reuse models from analysis or design without any modification. Figure 8 shows the varying degrees of reuse. Twelve participants stated that they wrote completely different models for testing purposes. The others more or less adapted the existing models to testing needs. This definitely showed that MBT may be used even when other development aspects are not model-based. This result was contrary to the oft-voiced opinion that model-based testing can be used only when modeling is also used for requirements and design.

Model-based Testing: Where Does It Stand? Reusing Models from Other Development Phases Has Its Limits


Model-Based Testing Compatible with Manual Testing

The 2014 MBT User Survey also showed that automated test execution is not the only way that model-based tests are applied. When asked about generated artifacts, the large majority mentioned test scripts for automated test execution, but more than half of the respondents also generated test cases for manual execution from MBT models (see figure 9). One-third of the respondents executed their test cases manually. Further, artifact generation did not even have to be tool-based: 12 percent obtained the test cases manually from the model; 36 percent at least partly used a tool; and 53 percent have established a fully automated test-case generation process.

Model-based Testing: Where Does It Stand? MBT Is More than Test-Case Generation for Automated Execution



One hundred participants shared their experience in the 2014 MBT User Survey and provided valuable input for this analysis. Although the survey was not broadly representative, it provided a profile of active MBT usage over a wide range of environments and organizations. For these users, MBT has had positive effects on efficiency and effectiveness, even if it only partially fulfills some extremely high expectations. The large majority said they intend to continue using models for testing purposes.

Regarding the common classification scheme, the responses confirmed the diversity of MBT approaches. No two answers were alike. This could put an end to the discussion of whether an MBT model may be classified as a system model, test model, or environment model. It cannot. Any model used for testing purposes is an MBT model. Usually, it focuses on all three aspects in varying degrees. Classifying this aspect appears to be an idle task. Some of the technical questions did not render useful information. Apparently, the notion of "degree of abstraction" of an MBT model is too abstract in itself. It seems to be difficult to classify an MBT model as either "very abstract" or "very detailed."

The work is not over. We are still searching for correlations and trends. If you have specific questions or ideas regarding MBT in general and the survey in particular, please contact us.



Robert V. Binder (rvbinder@sysverif.com) is a high-assurance entrepreneur and president of System Verification Associates. He has developed hundreds of application systems and advanced automated testing solutions. As test process architect for Microsoft's Open Protocol Initiative, he led the application of model-based testing to all of Microsoft's server-side APIs. He is the author of the definitive Testing Object-Oriented Systems: Models, Patterns, and Tools and two other books.

Bruno Legeard(bruno.legeard@femto-st.fr), professor at the University of Franche-Comté and cofounder and senior scientist at Smartesting, is internationally recognized as an expert and a well-known speaker in the model-based testing field. He has given presentations and keynotes at numerous testing and software-engineering conferences. Bruno is coauthor of the first industry-oriented book on model-based testing, Practical Model-Based Testing: A Tools Approach.

Anne Kramer (anne.kramer@seppmed.de) holds a Ph.D in physics and works as project manager and senior process consultant at sepp.med gmbh, a service provider specializing in IT solutions with integrated quality assurance in complex, safety-relevant domains. She teaches various certified training programs including the iSQI (International Software Quality Institute) Certified Model-Based Tester. Along with Bruno Legeard, she is an active member of the author group that establishes the ISTQB (International Software Testing Qualifications Board) Model-Based Tester Essential training scheme.

© 2014 ACM 1542-7730/14/1200 $10.00


Originally published in Queue vol. 13, no. 1
see this item in the ACM Digital Library



Robert Guo - MongoDB's JavaScript Fuzzer
The fuzzer is for those edge cases that your testing didn't catch.

Terry Coatta, Michael Donat, Jafar Husain - Automated QA Testing at EA: Driven by Events
A discussion with Michael Donat, Jafar Husain, and Terry Coatta

James Roche - Adopting DevOps Practices in Quality Assurance
Merging the art and science of software development

Neil Mitchell - Leaking Space
Eliminating memory hogs


(newest first)

govind | Mon, 23 Jan 2017 16:12:28 UTC

is MBT going to integrate with open source tools , like selenium

Sandy | Thu, 19 Mar 2015 18:53:47 UTC

Thank you Bob! Sorry for comming back to the issue with such a delay! It is high time I poked my nose into the add. resources you offer here. I understood that it needs a deeper dive into test design - not only one aspect like TDD. I think I should also continue reading "The art of softwaretesting" by Glenford J. Myers, Corey Sandler, Tom Badgett to gain a broader notion of test design. As you mention the importance of separation of logical from physical details in modeling - and the Adaptor Pattern as a framework - is there as well a good intro resource re. this Pattern? I'm looking forward to ongoing topics in the area of modeling and testing from ACM. Thank you!

Bob Binder | Thu, 12 Mar 2015 18:32:16 UTC

Hi Sandy - great questions - here are our answers.

1. What do you think about my idea on making TDD a partner of MBT? Bruno Yes, good idea. But, more a partner of ATDD. Bob MBT may be used to generate tests at any level, from a class interface to a Web 2.0 application UI. So, yes, you can practice Test Drive Development (TDD) with MBT. However, TDD is a weak approach to testing, so you will need to think more deeply about test design if you apply MBT when following a TDD approach. The same holds for BDD and ATDD. Please see http://www.infoworld.com/article/2892021/development-tools/saving-tdd-from-itself.html

2. Braking the model into re-usable components (for other dev. phases)? Bruno: Yes, braking the model in re-usable components is a good practice, but not for other dev. phases, more for other testing projects, because the MBT model is really testing oriented.

Bob: I agree with Bruno. The rationale for the parts and structure of a test model are somewhat different than for other system models. For example, test models are often organized around test purposes, which are not always the same as the functional hierarchy or interface structure of the system under test. It is also important to separate the "logical" aspects of the test model from the "physical" details of interfaces. The Adapter Pattern is typically used for this.

3. If we depart from (the codex of) clean code could it be helpful to establish a clean modeling code to foster the gains of MBT? Bruno: Yes, for sure. We have to establish that. MBT clean modeling ! Bob: Agree. Keith Stobies talk on this has some excellent advice for good test modeling practices. http://www.stickyminds.com/sites/default/files/presentation/file/2013/08STRWR_T2.pdf

4. And again: How much time do we and our stakeholders need to be able to harvest the entire MBT- benefits on the whole engineering lifecycle (what might 80hrs as a median mean in terms of proficiency level across the lifecycle?)?

Bruno: The learning curve to be proficient depends of many parameters (background of the team, tooling, commitment of the stakeholders, &). So 80 hrs is an average consistent with what we can see in organizations that deploys MBT.

Bob: 80 hours (about two working weeks) is often adequate to learn the basic usage of an MBT tool, but mastering a tool and its modeling concepts takes more time.

Our study didn't directly answer the question of benefits versus life-cycle. However, modeling almost always results in more discussion and deeper understanding of the system under test. This usually has immediate positive effects. Other studies found that, overall, MBT produce test suites in about half the time it takes to program equivalent test suites by hand (for details, see http://queue.acm.org/detail.cfm?id=1996412)

Sandy Flath | Thu, 12 Mar 2015 09:15:06 UTC

One critical surveyee points at the fact that MBT gets tricky when the iteration of functionality results in big changes. Id claim here: MBT has to go hand in hand with the iteration of functionality and maybe Test Driven Design (TDD) could be a good partner of MBT in that sense.

What do you think about making TDD a partner of MBT?

The usefulness of MBT seems to be proven - at least among the practitioners surveyed - it plays a major role in functional testing  and this even across all test levels (unit, integration, system and acceptance). Unfortunately, an effortless adaptability of the model into other development phases is not given. Do you think it is possible braking the model into re-usable components (for other dev. phases)?

I can make out some typical pain points mentioned by the rather critical respondents. Those comprise sth. I would call The art of taking and being given enough time to focus on quality that pays off. Components of  time for quality ¬ are: Communication with stakeholders across the engineering lifecycle. Plus: The modelling against real world problems which means nothing else than continuous reasoning over the impact of functionality changes, adaptations  a sense for clean modeling. Akin to clean coding, clean modeling might be the right attitude.

If we depart from (the codex of) clean code could it be helpful to establish a clean modeling code to foster the gains of MBT?

Now which time is needed to become proficient in MBT? The answer to that question is  sorry for being that frank: Hilarious  the expectations among the interviewees range from 0 to 2,000 hrs. which lets the authors conclude to a 80hrs (two work weeks median). A figure that must be further explained in terms of proficiency levels and the according skills on those levels. Maybe we could approach the time frame from a different angle. Let me tweak for a moment the question on MBT-proficiency:

How much time do we and our stakeholders need to be able to harvest the entire MBT- benefits on the whole engineering lifecycle?

Leave this field empty

Post a Comment:

© 2017 ACM, Inc. All Rights Reserved.