Quality Assurance

Vol. 3 No. 1 – February 2005

Quality Assurance

Interviews

A Conversation with Tim Bray

Tim Bray's Waterloo was no crushing defeat, but rather the beginning of his success as one of the conquerors of search engine technology and XML. In 1986, after working in software at DEC and GTE, he took a job at the University of Waterloo in Ontario, Canada, where he managed the New Oxford English Dictionary Project, an ambitious research endeavor to bring the venerable Oxford English Dictionary into the computer age.

A Conversation with Tim Bray

Searching for ways to tame the world’s vast stores of information.

Tim Bray’s Waterloo was no crushing defeat, but rather the beginning of his success as one of the conquerors of search engine technology and XML. In 1986, after working in software at DEC and GTE, he took a job at the University of Waterloo in Ontario, Canada, where he managed the New Oxford English Dictionary Project, an ambitious research endeavor to bring the venerable Oxford English Dictionary into the computer age.

Using the technology developed at Waterloo, Bray then founded Open Text Corporation and developed one of the first successful search engines. That experience led to his invitation to be one of the editors of the World Wide Web Consortium’s XML specification. He later founded a visualization software company called Antarctica Systems. He joined Sun Microsystems in 2004 as director of Web technologies.

Articles

A Passage to India

Most American IT employees take a dim view of offshore outsourcing. It's considered unpatriotic and it drains valuable intellectual capital and jobs from the United States to destinations such as India or China. Online discussion forums on sites such as isyourjobgoingoffshore.com are headlined with titles such as "How will you cope?" and "Is your career in danger?" A cover story in BusinessWeek magazine a couple of years ago summed up the angst most people suffer when faced with offshoring: "Is your job next?"

A passage to India

Pitfalls that the outsourcing vendor forgot to mention

MARK KOBAYASHI-HILLARY, OFFSHORE ADVISORY SERVICES

Most American IT employees take a dim view of offshore outsourcing. It’s considered unpatriotic and it drains valuable intellectual capital and jobs from the United States to destinations such as India or China. Online discussion forums on sites such as isyourjobgoingoffshore.com are headlined with titles such as “How will you cope?” and “Is your career in danger?” A cover story in BusinessWeek magazine a couple of years ago summed up the angst most people suffer when faced with offshoring: “Is your job next?”

This article, however, is not designed to make the economic case for or against offshore outsourcing. Economists across the world are debating this issue and producing evidence on both sides of the macro-economic fence. What cannot be disputed is that the American IT industry is changing. According to the U.S. Bureau of Labor Statistics, in 1983, computer programmers accounted for 62 percent of all IT jobs in the U.S. with the remainder being computer engineers, analysts, and scientists. By 2002 this ratio had almost reversed, with programmers accounting for just 26 percent of IT jobs.

by Mark Kobayashi-Hillary

Kode Vicious

Kode Vicious Unleashed

Dear KV, My officemate writes methods that are 1,000 lines long and claims they are easier to understand than if they were broken down into a smaller set of methods. How can we convince him his code is a maintenance nightmare?

Kode Vicious Unleashed

Koding konundrums driving you nuts? Ko-workers making you krazy? Not to worry, Kode Vicious has you covered—answering your questions, solving your problems, and just making the world a better place.

Dear KV,

My officemate writes methods that are 1,000 lines long and claims they are easier to understand than if they were broken down into a smaller set of methods. How can we convince him his code is a maintenance nightmare?

by George Neville-Neil

Articles

Orchestrating an Automated Test Lab

Networking and the Internet are encouraging increasing levels of interaction and collaboration between people and their software. Whether users are playing games or composing legal documents, their applications need to manage the complex interleaving of actions from multiple machines over potentially unreliable connections. As an example, Silicon Chalk is a distributed application designed to enhance the in-class experience of instructors and students. Its distributed nature requires that we test with multiple machines. Manual testing is too tedious, expensive, and inconsistent to be effective. While automating our testing, however, we have found it very labor intensive to maintain a set of scripts describing each machine's portion of a given test. Maintainability suffers because the test description is spread over several files.

Orchestrating an Automated Test Lab

Composing a score can help us manage the complexity of testing distributed apps.

MICHAEL DONAT, SILICON CHALK

Networking and the Internet are encouraging increasing levels of interaction and collaboration between people and their software. Whether users are playing games or composing legal documents, their applications need to manage the complex interleaving of actions from multiple machines over potentially unreliable connections. As an example, Silicon Chalk is a distributed application designed to enhance the in-class experience of instructors and students. Its distributed nature requires that we test with multiple machines. Manual testing is too tedious, expensive, and inconsistent to be effective. While automating our testing, however, we have found it very labor intensive to maintain a set of scripts describing each machine’s portion of a given test. Maintainability suffers because the test description is spread over several files.

Experience has convinced me that testing distributed software requires an automated test lab, where each test description specifies behavior over multiple test-lab components.

by Michael Donat

Quality Assurance: Much More than Testing

Quality assurance isn't just testing, or analysis, or wishful thinking. Although it can be boring, difficult, and tedious, QA is nonetheless essential.

Quality Assurance: Much More than Testing

Good QA is not only about technology, but also methods and approaches.

STUART FELDMAN, IBM RESEARCH

Quality assurance isn’t just testing, or analysis, or wishful thinking. Although it can be boring, difficult, and tedious, QA is nonetheless essential.

Ensuring that a system will work when delivered requires much planning and discipline. Convincing others that the system will function properly requires even more careful and thoughtful effort. QA is performed through all stages of the project, not just slapped on at the end. It is a way of life.

by Stuart Feldman

Sifting Through the Software Sandbox: SCM Meets QA

Thanks to modern SCM (software configuration management) systems, when developers work on a codeline they leave behind a trail of clues that can reveal what parts of the code have been modified, when, how, and by whom. From the perspective of QA (quality assurance) and test engineers, is this all just "data," or is there useful information that can improve the test coverage and overall quality of a product?

Sifting through the Software Sandbox: SCM meets QA

Source control—it’s not just for tracking changes anymore.

WILLIAM W. WHITE, PERFORCE SOFTWARE

Thanks to modern SCM (software configuration management) systems, when developers work on a codeline they leave behind a trail of clues that can reveal what parts of the code have been modified, when, how, and by whom. From the perspective of QA (quality assurance) and test engineers, is this all just “data,” or is there useful information that can improve the test coverage and overall quality of a product?

TESTING AS USUAL

Test and QA engineers have a variety of tools at their disposal when they set out to assess a software product. They can use functional specifications and user manuals to map out a test plan; design specifications and interviews with developers can provide the information needed to develop functional tests; scripting tools can be used to automate the execution of some of those tests; and code coverage tools can be deployed to identify gaps in the testing procedures. After the test plan has been written and the basic test suite developed and executed, however, much work remains to be done.

by William W. White

Too Darned Big to Test

The increasing size and complexity of software, coupled with concurrency and distributed systems, has made apparent the ineffectiveness of using only handcrafted tests. The misuse of code coverage and avoidance of random testing has exacerbated the problem. We must start again, beginning with good design (including dependency analysis), good static checking (including model property checking), and good unit testing (including good input selection). Code coverage can help select and prioritize tests to make you more efficient, as can the all-pairs technique for controlling the number of configurations. Finally, testers can use models to generate test coverage and good stochastic tests, and to act as test oracles.

TOO DARNED BIG TO TEST

Testing large systems is a daunting task, but there are steps we can take to ease the pain.

KEITH STOBIE, MICROSOFT

The increasing size and complexity of software, coupled with concurrency and distributed systems, has made apparent the ineffectiveness of using only handcrafted tests. The misuse of code coverage and avoidance of random testing has exacerbated the problem. We must start again, beginning with good design (including dependency analysis), good static checking (including model property checking), and good unit testing (including good input selection). Code coverage can help select and prioritize tests to make you more efficient, as can the all-pairs technique for controlling the number of configurations. Finally, testers can use models to generate test coverage and good stochastic tests, and to act as test oracles.

HANDCRAFTED TESTS OUTPACED BY HARDWARE AND SOFTWARE

Hardware advances have followed Moore’s law for years, giving the capability for running ever-more complex software on the same cost platform. Developers have taken advantage of this by raising their level of abstraction from assembly to first-generation compiled languages to managed code such as C# and Java, and rapid application development environments such as Visual Basic. Although the number of lines of code per day per programmer appears to remain relatively fixed, the power of each line of code has increased, allowing complex systems to be built. Moore’s law provides double the computing power every 18 months and software code size tends to double every seven years, but software testing does not appear to be keeping pace.

by Keith Stobie

Curmudgeon

Traipsing Through the QA Tools Desert

The Jeremiahs of the software world are out there lamenting, "Software is buggy and insecure!" Like the biblical prophet who bemoaned the wickedness of his people, these malcontents tell us we must repent and change our ways. But as someone involved in building commercial software, I'm thinking to myself, "I don't need to repent. I do care about software quality." Even so, I know that I have transgressed. I have shipped software that has bugs in it. Why did I do it? Why can't I ship perfect software all the time?

Traipsing through the QA Tools Desert

Who’s really to blame for buggy code?

Terry Coatta, Silicon Chalk

The Jeremiahs of the software world are out there lamenting, “Software is buggy and insecure!” Like the biblical prophet who bemoaned the wickedness of his people, these malcontents tell us we must repent and change our ways. But as someone involved in building commercial software, I’m thinking to myself, “I don’t need to repent. I do care about software quality.” Even so, I know that I have transgressed. I have shipped software that has bugs in it. Why did I do it? Why can’t I ship perfect software all the time?

Like anything in life, the reasons are complex, but a big factor is just how hard it is to do QA (quality assurance). You can spend days or even weeks looking for a single bug, and eventually you get to the point where it doesn’t make sense to hold up shipping the product looking for a bug that it seems you may never find.

by Terry Coatta