Software Testing - Re: Defect classification and Severity for the bugs found in TCDs

Author

Re: Defect classification and Severity for the bugs found in TCDs

H. S. Lahman

2007-02-28, 7:15 pm

Responding to SUHAS...

> Need some help regarding how do we classify the defects found in TCD
> and how can we assign the severity for the defects found in TCDs.

Classifying defects and ranking their severity are orthogonal. Figure
out what the defects are first and then worry about assigning severity.

For a classification I would start with the fact that all test cases
have five basic elements:

test harness infrastructure. That's all the supporting stuff shared
across test cases (e.g., doing differences vs. Gold Files).

input data. Every test case has input data to be used as stimuli that is
unique to the test case. Depending on the degree of automation, there
can be specifications of things like what to test. IOW, it includes
anything that is specified in external data that the test harness reads.

software state initialization. This is whatever setup the test case
requires to get the software into a proper state for the test.

execution mechanics. This is infrastructure needed to actually initiate
the test, exercise the software, and extract the results. If one needs
to stub return values for unit testing, it would be covered here because
one is emulating the run-time execution.

results data. This is the specification of expected results.

[These categories apply to an individual test case. One can argue that
there is another category for the entire test suite. Then the notion of
missing an entire test case becomes important. One also needs to deal
with things like sequencing if efficient setups require that the test
cases be run in a particular order.]

How far one wants to drill down into each classification will depend on
the goals of process improvement and the particular testing environment.
Perhaps more important, it will depend on the shop's history of testing.

Generally one wants to start simple with broad categories of defects. As
one monitors the process it will become clear that defects in certain
categories have substantially higher frequency or that more detail is
needed to do process improvement analysis. Then one can further
subdivide defect categories based on real need.

FWIW, I find bi-value defect classification to be more useful. That is,
one has a basic categorization but applies a qualifier to it. For
example, most data defects can be qualified with causative clauses like
'incorrect', 'missing', 'incomplete', 'out of order', 'misinterpreted',
etc.. So if you start with just "input data" and "results data" as the
defects for both those categories above, one can probably qualify both
with the same set of qualifiers.

This gives one combinatorial classification for a spartan set of
definitions that everyone needs to learn. In the example one needs to
understand 7 definitions rather than 10 unique defect classes. This
becomes important when the number of categories to which the qualifies
apply increases (e.g., when one needs to break up "input data" into
categories like "test parameters", "state initializers", "unit
selectors", and "stub identifiers").

As far as severity is concerned, I would have to be convinced there was
value to be achieved that would be worth the trouble of tracking it.
Clearly failing to execute a test case is a potentially Big Problem.
Similarly, typos in the test description are a pretty Small Problem.
Anything in between, though, seems to be pretty much the same. All such
defects can potentially lead to either false passes or false failures.
Either way exactly the same amount diagnostic effort and rework is going
to be done.

One can argue that any false pass is a Big Problem because it implies a
field escape while false failures are Small Problems because they are
always fixed internally without touching the software. The problem is:
how does one know about false positives? Simply passing is not a problem
in itself. It only becomes a problem if the software is actually broken
and you aren't going to know about that until some irate customer is on
the phone.

So rather than trying to classify defects a priori with severity, I
would rather track the defects associated with escapes that should have
been detected. I would then use that data to try to prevent the Fat
Rabbit defects (highest frequency) in that collection by changing the
test process.

*************
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH

James Dennett

2007-03-01, 4:24 am

Phlip wrote:
> H. S. Lahman wrote:
>
>
> Tip: Severity belongs to developers, and Priority belongs to the business
> side.

Severity and priority are both on the business side,
if you're doing things right. Developers don't get
to choose how much something matters to the business,
nor which order the business wants things in.

Severity is how severe something is; that is, how much
of a problem it is (whether that be a known dollar amount,
a rough estimate of cost to reputation, or an expected
cost based on risk analysis).

Priority is to do with timing of a resolution.

Developers should have input into the determination
of severity, through understanding and explaining
the scope/possible consequences of a defect.

They will also have input into estimates of the cost
of *fixing* the defect.

Based on these determinations (of scope and cost to
fix), the business should make its decision on the
severity and priority (severity being based on scope,
and priority on a combination of severity and cost
to fix).

-- James

Sponsored Links