Model-based Testing: Where Does It Stand?

MBT has positive effects on efficiency and effectiveness, even if it only partially fulfills high expectations.

Robert V. Binder, Bruno Legeard, and Anne Kramer

You have probably heard about MBT (model-based testing), but like many software-engineering professionals who have not used MBT, you might be curious about others' experience with this test-design method.

From mid-June 2014 to early August 2014, we conducted a survey to learn how MBT users view its efficiency and effectiveness. The 2014 MBT User Survey, a follow-up to a similar 2012 survey (http://robertvbinder.com/real-users-of-model-based-testing/), was open to all those who have evaluated or used any MBT approach. Its 32 questions included some from a survey distributed at the 2013 User Conference on Advanced Automated Testing. Some questions focused on the efficiency and effectiveness of MBT, providing the figures that managers are most interested in. Other questions were more technical and sought to validate a common MBT classification scheme. A common classification scheme could help users understand both the general diversity and specific approaches.

The 2014 survey provides a realistic picture of the current state of MBT practice. This article presents some highlights of the survey findings. The complete results are available at http://model-based-testing.info/2014/12/09/2014-mbt-user-survey-results/.

Survey Respondents

A large number of people received the 2014 MBT User Survey call for participation, in both Europe and North America. Additionally, it was posted to various social-networking groups and software-testing forums. Several tool providers helped distribute the call. Last but not least, ETSI (European Telecommunications Standards Institute) supported the initiative by informing all participants at the User Conference on Advanced Automated Testing.

Exactly 100 MBT practitioners responded by the closing date. Not all participants answered every question; the number of answers is indicated if considerably below 100. Percentages for these partial response sets are based on the actual number of responses for a particular question.

The large majority of the respondents (86 percent) were from businesses. The remaining 14 percent represented research, government, and nonprofit organizations. The organizations ranged in size from three to 300,000 employees (figure 1).

Model-based Testing: Where Does It Stand? Survey Participants Come from Organizations of All Sizes

As shown in figure 2, almost half of the respondents had moved from evaluation and pilot to rollout or generalized use. On average, respondents had three years of experience with MBT. In fact, the answers ranged from zero (meaning "just started") to 34 years.

Model-based Testing: Where Does It Stand? 48% of the Respondents Routinely Use MBT, and 52% Are Still in the Evaluation or Trial Phase

To get an impression of how important MBT is compared to other test-design techniques, the survey asked for the percentage of total testing effort spent on MBT, hand-coded test automation, and manual test design. Each of the three test-design methods represented approximately one-third of the total testing effort. Thus, MBT is not a marginal phenomenon. For those who use it, its importance is comparable to other kinds of test automation.

Nearly 40 percent of the respondents came from the embedded domain. Enterprise IT accounted for another 30 percent, and Web applications for roughly 20 percent. Other application domains for the system under test were software infrastructure, communications, gaming, and even education. The exact distribution is given in figure 3.

Model-based Testing: Where Does It Stand? Various Application Domains Represented

The role of external MBT consultants turned out to be smaller than expected. Although we initially speculated that MBT was driven mainly from the outside, the survey showed a different picture. A majority (60 percent) of those who answered the survey were in-house MBT professionals. Only 18 percent of respondents were external MBT consultants, and 22 percent were researchers in the MBT application area (figure 4).

Model-based Testing: Where Does It Stand? 3 in 5 Respondents are In-house MBT Professionals

Does MBT Work as Expected?

In our projects, we have observed that the expectations regarding MBT are usually very, if not extremely, high. "Does MBT fulfill expectations?" The MBT user survey asked whether the expectations in five different categories are being met: testing is becoming cheaper; testing is better; the models are helping manage the complexity of the system with regard to testing; the test design is starting earlier; and, finally, models are improving communication among stakeholders.

Figure 5 shows the number of responses and the degree of satisfaction for each of the five categories. The answers reflect a slight disenchantment. MBT does not completely fulfill the extremely high expectations in terms of cost reduction, quality improvement, and complexity, but, still, more than half of the respondents were partially or completely satisfied (indicated by the green bars in figure 5). For the two remaining categories, MBT exceeds expectations. Using models for testing purposes definitely improves communication among stakeholders and helps initiate test design earlier.

Model-based Testing: Where Does It Stand? Comparison Between Expectations and Observed Satisfaction Level

Overall, the respondents viewed MBT as a useful technology: 64 percent found it moderately or extremely effective, whereas only 13 percent rated the method as ineffective (figure 6). More than 70 percent of the respondents stated that it is very likely or extremely likely that they will continue with the method. Only one respondent out of 73 rejected this idea. Participants self-selected, however, so we cannot exclude a slight bias of positive attitude toward MBT.

Model-based Testing: Where Does It Stand? Nearly All Participants Rate MBT as Being Effective (to Different Degrees)

To obtain more quantitative figures on effectiveness and efficiency, the survey asked three rather challenging questions:

• To what degree does MBT reduce or increase the number of escaped bugs—that is, the number of bugs nobody would have found before delivery?

• To what degree does MBT reduce or increase testing costs?

• To what degree does MBT reduce or increase testing duration?

Obviously, those questions were difficult to answer, and it was impossible to deduce statistically relevant information from the 21 answers obtained in the survey. Only one respondent clearly answered in the negative regarding the number of escaped bugs. All others provided positive figures, and two answers were illogical.

On the cost side, the survey asked respondents how many hours it took to become a proficient MBT user. The answers varied from zero to 2,000 hours of skill development, with a median of two work weeks (80 hours).

What Kind of Testing is Model-Based?

Model-based testing is used at all stages of software development, most often for integration and system testing (figure 7). Half of the survey respondents reported model-based integration testing, and about three-quarters reported conducting model-based system testing. Nearly all respondents used models for functional testing. MBT played a subordinate role in performance, usability, and security testing, with less than 20 percent usage for each.

Model-based Testing: Where Does It Stand? MBT Plays an Important Role at All Test Levels

Only one participant found it difficult to fit MBT into an agile approach, while 44 percent reported using MBT in an agile development process, with 25 percent at an advanced stage (rollout or generalized use).

There was a clear preference regarding model focus: 59 percent of the MBT models (any model used for testing purposes is referred to here as an MBT model) used by survey respondents focused on behavioral aspects only, while 35 percent had both structural and behavioral aspects. Purely statistical models played a minor role (6 percent). The trend was even more pronounced for the notation type. Graphical notations prevailed with 81 percent; only 14 percent were purely textual MBT models—that is, they used no graphical elements. All kinds of combinations were used, however. One respondent put it very clearly: "We use more than one model per system under test."

Test Modeling Independent of Design Modeling

Note that more than 40 percent of the respondents did not use modeling in other development phases. Only eight participants stated that they reuse models from analysis or design without any modification. Figure 8 shows the varying degrees of reuse. Twelve participants stated that they wrote completely different models for testing purposes. The others more or less adapted the existing models to testing needs. This definitely showed that MBT may be used even when other development aspects are not model-based. This result was contrary to the oft-voiced opinion that model-based testing can be used only when modeling is also used for requirements and design.

Model-based Testing: Where Does It Stand? Reusing Models from Other Development Phases Has Its Limits

Model-Based Testing Compatible with Manual Testing

The 2014 MBT User Survey also showed that automated test execution is not the only way that model-based tests are applied. When asked about generated artifacts, the large majority mentioned test scripts for automated test execution, but more than half of the respondents also generated test cases for manual execution from MBT models (see figure 9). One-third of the respondents executed their test cases manually. Further, artifact generation did not even have to be tool-based: 12 percent obtained the test cases manually from the model; 36 percent at least partly used a tool; and 53 percent have established a fully automated test-case generation process.

Model-based Testing: Where Does It Stand? MBT Is More than Test-Case Generation for Automated Execution

Conclusion

One hundred participants shared their experience in the 2014 MBT User Survey and provided valuable input for this analysis. Although the survey was not broadly representative, it provided a profile of active MBT usage over a wide range of environments and organizations. For these users, MBT has had positive effects on efficiency and effectiveness, even if it only partially fulfills some extremely high expectations. The large majority said they intend to continue using models for testing purposes.

Regarding the common classification scheme, the responses confirmed the diversity of MBT approaches. No two answers were alike. This could put an end to the discussion of whether an MBT model may be classified as a system model, test model, or environment model. It cannot. Any model used for testing purposes is an MBT model. Usually, it focuses on all three aspects in varying degrees. Classifying this aspect appears to be an idle task. Some of the technical questions did not render useful information. Apparently, the notion of "degree of abstraction" of an MBT model is too abstract in itself. It seems to be difficult to classify an MBT model as either "very abstract" or "very detailed."

The work is not over. We are still searching for correlations and trends. If you have specific questions or ideas regarding MBT in general and the survey in particular, please contact us.

LOVE IT, HATE IT? LET US KNOW

[email protected]

Robert V. Binder ([email protected]) is a high-assurance entrepreneur and president of System Verification Associates. He has developed hundreds of application systems and advanced automated testing solutions. As test process architect for Microsoft's Open Protocol Initiative, he led the application of model-based testing to all of Microsoft's server-side APIs. He is the author of the definitive Testing Object-Oriented Systems: Models, Patterns, and Tools and two other books.

Bruno Legeard([email protected]), professor at the University of Franche-Comté and cofounder and senior scientist at Smartesting, is internationally recognized as an expert and a well-known speaker in the model-based testing field. He has given presentations and keynotes at numerous testing and software-engineering conferences. Bruno is coauthor of the first industry-oriented book on model-based testing, Practical Model-Based Testing: A Tools Approach.

Anne Kramer ([email protected]) holds a Ph.D in physics and works as project manager and senior process consultant at sepp.med gmbh, a service provider specializing in IT solutions with integrated quality assurance in complex, safety-relevant domains. She teaches various certified training programs including the iSQI (International Software Quality Institute) Certified Model-Based Tester. Along with Bruno Legeard, she is an active member of the author group that establishes the ISTQB (International Software Testing Qualifications Board) Model-Based Tester Essential training scheme.

Originally published in Queue vol. 13, no. 1—
Comment on this article in the ACM Digital Library