Contact information: email@example.com, www.testing.com; firstname.lastname@example.org, www.satisfice.com;
Copyright 2000 Brian Marick, James Bach, and Cem Kaner. All Rights Reserved. Permission granted
to copy for personal use.
Some images copyright www.arttoday.com.
A Manager’s Guide to Evaluating Test Suites
Brian Marick, Testing Foundations
James Bach, Satisfice Inc.
Cem Kaner, kaner.com
Whatever else testers do, they run tests. They execute programs and judge
whether the results produced are correct or not. Collections of tests – test
suites – are expensive to create. It’s fair to ask how good any given suite
is. This paper describes ways we would go about answering that question.
There are two broad traditions of testing. One holds that the purpose of testing is to justify
confidence in the program. Reliability modeling [Lyu96] is in that tradition. The other
tradition focuses on producing valid bug reports. We will not argue here which tradition is
correct. We merely state that we belong to the second tradition. This paper is addressed to
people like us.
Evaluating a test suite one time
In this section, we’ll assume that you have been presented with a test suite, the tester who
created it, the bugs it found, and all relevant documentation. We’ll present steps you could
use to evaluate the suite. When detailed discussions of topics are readily available online,
we only sketch them here.
Step 1: What is the suite’s purpose?
Suppose you have two test suites, Suite1 and Suite2. Is Suite1 better if it finds more bugs?
Not necessarily. Let’s explore the issue by way of an example.
Suppose the product has five subsystems. For whatever reason, the testing team devotes
almost all its effort to subsystem 1 and files 100 bug reports. The development team
works diligently and fixes them all. The testing team can’t find any bugs in the fixed
subsystem, so the product is released. After release, we discover that the testing