Home > Archive > Software Testing > September 2007 > "Lean" test scripts
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
"Lean" test scripts
|
|
| Ali Chambers 2007-08-30, 9:04 pm |
| Hi,
I am responsible at my organisation for reviewing test scripts for a
software product in the Healthcare Arena. I have noticed a lot of
repetition in the scripts.
Can anyone point me to/describe a way/methodology of creating test
scripts that avoids repetition? I have heard of "equivalence
partitioning", but it is a bit too technical for me! I can then
summarise the method for the people who write the scripts.
The scripts describe how the user will interact with the software in a
Hospital, eg:
- User logs in
- User registers patient on the system (do X, Y, Z on the system)
- User books in surgery, etc...
Thanks,
Alex
| |
|
| Ali Chambers wrote:
> I am responsible at my organisation for reviewing test scripts for a
> software product in the Healthcare Arena. I have noticed a lot of
> repetition in the scripts.
If these tests are manual, automate them.
Next, it sounds like someone created the test cases by cloning and modifying
a previous test. That means all the duplicated activities in each case
should be extracted out and placed in common repositories. Place common
behavior into common setup() and teardown() methods, to run before and
after each case, respectively. Place common data into light database
records. Some call those "fixtures", but the correct term is "resources".
The result should be each test case only contains the details that differ
from other cases.
--
Phlip
http://www.oreilly.com/catalog/9780596510657/
^ assert_xpath
http://tinyurl.com/23tlu5 <-- assert_raise_message
| |
| Ali Chambers 2007-08-30, 9:04 pm |
| Thanks for your reply. I shall do that.
Are there other methods out there that I am missing - eg: ways to
eliminate obvious things first, rather than testing everything??? I
feel I am clutching at straws a little, so any more pointers welcome.
AC
| |
| Loda_Schitt 2007-08-30, 9:04 pm |
| Combination testing and load testing should quickly point out major
flaws in the product
On Thu, 30 Aug 2007 15:55:32 -0700, Ali Chambers
<info@alexchambers.co.uk> wrote:
>Thanks for your reply. I shall do that.
>
>Are there other methods out there that I am missing - eg: ways to
>eliminate obvious things first, rather than testing everything??? I
>feel I am clutching at straws a little, so any more pointers welcome.
>
>AC
| |
| S Perryman 2007-08-31, 6:14 am |
| "Ali Chambers" <info@alexchambers.co.uk> wrote in message
news:1188491886.489437.79550@r34g2000hsd.googlegroups.com...
> I am responsible at my organisation for reviewing test scripts for a
> software product in the Healthcare Arena. I have noticed a lot of
> repetition in the scripts.
> Can anyone point me to/describe a way/methodology of creating test
> scripts that avoids repetition?
Factor the common behaviour into a common infrastructure, which all
the repetitive tests invoke (parametrically etc) .
> I have heard of "equivalence
> partitioning", but it is a bit too technical for me! I can then
> summarise the method for the people who write the scripts.
Equivalence class partitioning is very simple.
It is for many the "1 + 1 = 2" of testing and one expects good/
experienced testers to be using it.
The basic concept is this :
There is a set (class) of input values that invoke the same behaviour
and/or result in the same form of outputs. These sets are known as
the equivalence classes (EC) . The theory goes thus (forall e IN EC) :
- If an IUT behaves correctly for e, it will behave correctly for every
value in EC
- If an IUT produces a defect when given e, it will produce the same
defect for any value in EC
- If an IUT is defective but e does not expose the defect, no other
value in EC will either.
A quick example : integer addition.
One equivalence class is the input pairs (X,Y) where X/Y are both
positive. Another is where X/Y are both negative. And so on.
FWIW, the situation you describe suggests that you might apply
ECP. One symptom of poor testing envs, is one that proclaims to have
"lots" of tests, but when you apply ECP to these tests you find that
actually the tests as a whole are testing hardly anything (ie they are
the equivalent of src code that has a high LOC because of blank lines
etc) because the inputs are part of the same equivalence classes.
Regards,
Steven Perryman
| |
| Michael Bolton 2007-09-01, 10:08 pm |
| Hi, Alex...
I'm answering off the list because I'd prefer to keep our conversation
private.
> I am responsible at my organisation for reviewing test scripts for a
> software product in the Healthcare Arena. I have noticed a lot of
> repetition in the scripts.
I see a problem with repetition, but I also see big problems with
scripts.
* They prescribe the activities that a tester will perform to some
degree of precision. If the tester wouldn't know what to do
otherwise, or might know what to do and forget, the script might be
helpful.
* They provide some sense of reproducibility, so that if the test is
interesting in some way, we can repeat the steps that it prescribes.
* If you can demonstrate that they were followed, you can demonstrate
that something happened.
* They may provide a set of steps for developers to follow, such that
the developers can test the code before releasing it to testing (this
never happens, but we can dream).
* The script might help to identify a pass/fail decision rule.
* A set of scripts might help to co-ordinate the activities of
multiple people.
* A sufficiently simple set of actions, with a sufficiently clear
decision rule, can be automated. The automation can then be
programmed to incorporate a good deal of variation at high speed.
Here are some bad things about scripts:
* They're very expensive to produce.
* They typically take an experienced and skilled tester away from the
testing task an into a documentation task.
* Only a fraction of what an experienced and skilled tester thinks and
does can be captured in a script.
* Scripts tend to be produced at the beginning of the project, when we
know less about the product than we'll ever know.
* They go out of date quickly and easily; when the program changes,
the script(s) have to change too.
* The precision that they provide is often illusory. For example:
"1. Start the program." How many ways are there to do that?--and
that's just the first line.
* While providing step-by-step instructions to follow, they tend not
to provide a motivation for the test. In many ways, it's the
motivation that's the important part.
* By focusing the tester on specific observations, they tend to
promote inattentional blindness, which is a well-known phenomenon in
cognitive psychology. If you're paying close attention to something,
you tend not to be open to other observations. That is, a test tends
to orient the tester towards answering the question, "Did this
assertion pass or fail?" rather than the much more important question,
"Is there a problem here?"
* They tend to be oblivious to the idea that testers really can think
for themselves.
* They constrain the variability that allows us to cover more of the
program.
* Like use cases themselves, they tend to be straightforward, single,
happy paths through a program or task, where real users do all kinds
of things that aren't like that. People change their minds, branch
and backtrack, forget certain steps (and experience errors and error
messages), do things out of order, pause and go for coffee...
* If the program behaves in an unexpected way at some point earlier in
the script, steps that follow may be invalidated.
* They promote repetition, where variation might be a better idea if
the object of the exercise is to reveal new bugs.
* They tend to inhibit the development of skill, where supervision,
mentoring, and rapid feedback tend to be more conducive to learning.
* It's relatively easy for a tester to say, "Yeah, I did that" while
pointing to a script, where a set of notes taken on the fly is
somewhat harder to forge.
* When used to guide humans, they tend to automate the behaviour of
humans--but people make rotten machines. If you have a task that's
simple enough to be expressed in a script, maybe it can be run (with
greater speed, precision, variation, and evaluation) by a machine.
> Can anyone point me to/describe a way/methodology of creating test
> scripts that avoids repetition? I have heard of "equivalence
> partitioning", but it is a bit too technical for me!
If equivalence partitioning is really too technical for you, I'd be
concerned that your understanding of testing puts the quality of your
evaluation of the scripts into some question.
In fact, I doubt that the idea is too technical for you; it's not
terribly hard to understand; equivalence partitioning is essentially
the same thing as classification. The notion is that for any given
set of things, there are subsets whose elements we can consider to be
sufficiently similar based on some criterion. Suppose that our
program classifies people as children (those who have not had an 18th
birthday), adults (who are at least 18 years old and not more than 64
years old), and seniors (who are 65 years old or more). For some
purpose, we might (and I emphasize /might/) expect the program to
treat all children in the same way, all adults in a different way, and
all seniors in a third way. The default assumption is that any
element of one set will be treated in the same was as the other
elements of the same set, so we further assume that a test using one
element of that set will expose a problem that will happen for all of
the elements of that set. This is an interesting idea, but it is a /
heuristic/--that is, it's a plausible and even powerful method for
solving the problem of too many tests and too little time, but like
all heuristics, it's fallible. The principal vulnerability to the
heuristic is that, just as the developer misclassify sets or their
members, we may misclassify them too, in the same way or in different
ways.
The lack of knowledge about testing is a solvable problem, of course:
you can study and learn. I'd recommend a couple of books (A
Practitioner's Guide to Software Testing, by Lee Copeland; Lessons
Learned in Software Testing by Kaner, Bach, and Pettichord). Check
out some of the articles on James Bach's site (http://
www.satisifice.com) or on my site (http://www.developsense.com/
articles).
>I can then summarise the method for the people who write the scripts. The scripts describe how the user will interact with the software in a Hospital, eg:
>
> - User logs in
> - User registers patient on the system (do X, Y, Z on the system)
> - User books in surgery, etc...
Rather than writing scripts, consider writing charters; sets of
missions for the testers to accomplish in 90-minute-or-so sessions.
When there's a specific conditition to test, make that explicit ("log
in using an administrator account and create a new customer. Test the
pre-existing condition logic according the the tables in standard R.
47. In addition to general coverage, make sure to include tests for
customers in the mining and railroading industries. Also test for
completed and cancelled surgeries.")
Testing without scripts is very powerful, more powerful than testing
with scripts. Have a look at Session-Based Test Management at
http://www.satisfice.com/sbtm.
---Michael B.
| |
| Loda_Schitt 2007-09-01, 10:08 pm |
| Beautiful answer michael!. I've been looking for something to justify
exploratory testing and I absolutely hate test scripts
btw do you believe it's possible to do representative class
partitioning without any form of gray box understanding of code?
Thanks and I also enjoy your music
On Sat, 01 Sep 2007 09:59:53 -0700, Michael Bolton
<michael.a.bolton@gmail.com> wrote:
>Hi, Alex...
>
>I'm answering off the list because I'd prefer to keep our conversation
>private.
>
>
>I see a problem with repetition, but I also see big problems with
>scripts.
>
>* They prescribe the activities that a tester will perform to some
>degree of precision. If the tester wouldn't know what to do
>otherwise, or might know what to do and forget, the script might be
>helpful.
>* They provide some sense of reproducibility, so that if the test is
>interesting in some way, we can repeat the steps that it prescribes.
>* If you can demonstrate that they were followed, you can demonstrate
>that something happened.
>* They may provide a set of steps for developers to follow, such that
>the developers can test the code before releasing it to testing (this
>never happens, but we can dream).
>* The script might help to identify a pass/fail decision rule.
>* A set of scripts might help to co-ordinate the activities of
>multiple people.
>* A sufficiently simple set of actions, with a sufficiently clear
>decision rule, can be automated. The automation can then be
>programmed to incorporate a good deal of variation at high speed.
>
>Here are some bad things about scripts:
>
>* They're very expensive to produce.
>* They typically take an experienced and skilled tester away from the
>testing task an into a documentation task.
>* Only a fraction of what an experienced and skilled tester thinks and
>does can be captured in a script.
>* Scripts tend to be produced at the beginning of the project, when we
>know less about the product than we'll ever know.
>* They go out of date quickly and easily; when the program changes,
>the script(s) have to change too.
>* The precision that they provide is often illusory. For example:
>"1. Start the program." How many ways are there to do that?--and
>that's just the first line.
>* While providing step-by-step instructions to follow, they tend not
>to provide a motivation for the test. In many ways, it's the
>motivation that's the important part.
>* By focusing the tester on specific observations, they tend to
>promote inattentional blindness, which is a well-known phenomenon in
>cognitive psychology. If you're paying close attention to something,
>you tend not to be open to other observations. That is, a test tends
>to orient the tester towards answering the question, "Did this
>assertion pass or fail?" rather than the much more important question,
>"Is there a problem here?"
>* They tend to be oblivious to the idea that testers really can think
>for themselves.
>* They constrain the variability that allows us to cover more of the
>program.
>* Like use cases themselves, they tend to be straightforward, single,
>happy paths through a program or task, where real users do all kinds
>of things that aren't like that. People change their minds, branch
>and backtrack, forget certain steps (and experience errors and error
>messages), do things out of order, pause and go for coffee...
>* If the program behaves in an unexpected way at some point earlier in
>the script, steps that follow may be invalidated.
>* They promote repetition, where variation might be a better idea if
>the object of the exercise is to reveal new bugs.
>* They tend to inhibit the development of skill, where supervision,
>mentoring, and rapid feedback tend to be more conducive to learning.
>* It's relatively easy for a tester to say, "Yeah, I did that" while
>pointing to a script, where a set of notes taken on the fly is
>somewhat harder to forge.
>* When used to guide humans, they tend to automate the behaviour of
>humans--but people make rotten machines. If you have a task that's
>simple enough to be expressed in a script, maybe it can be run (with
>greater speed, precision, variation, and evaluation) by a machine.
>
>
>If equivalence partitioning is really too technical for you, I'd be
>concerned that your understanding of testing puts the quality of your
>evaluation of the scripts into some question.
>
>In fact, I doubt that the idea is too technical for you; it's not
>terribly hard to understand; equivalence partitioning is essentially
>the same thing as classification. The notion is that for any given
>set of things, there are subsets whose elements we can consider to be
>sufficiently similar based on some criterion. Suppose that our
>program classifies people as children (those who have not had an 18th
>birthday), adults (who are at least 18 years old and not more than 64
>years old), and seniors (who are 65 years old or more). For some
>purpose, we might (and I emphasize /might/) expect the program to
>treat all children in the same way, all adults in a different way, and
>all seniors in a third way. The default assumption is that any
>element of one set will be treated in the same was as the other
>elements of the same set, so we further assume that a test using one
>element of that set will expose a problem that will happen for all of
>the elements of that set. This is an interesting idea, but it is a /
>heuristic/--that is, it's a plausible and even powerful method for
>solving the problem of too many tests and too little time, but like
>all heuristics, it's fallible. The principal vulnerability to the
>heuristic is that, just as the developer misclassify sets or their
>members, we may misclassify them too, in the same way or in different
>ways.
>
>The lack of knowledge about testing is a solvable problem, of course:
>you can study and learn. I'd recommend a couple of books (A
>Practitioner's Guide to Software Testing, by Lee Copeland; Lessons
>Learned in Software Testing by Kaner, Bach, and Pettichord). Check
>out some of the articles on James Bach's site (http://
>www.satisifice.com) or on my site (http://www.developsense.com/
>articles).
>
>
>Rather than writing scripts, consider writing charters; sets of
>missions for the testers to accomplish in 90-minute-or-so sessions.
>When there's a specific conditition to test, make that explicit ("log
>in using an administrator account and create a new customer. Test the
>pre-existing condition logic according the the tables in standard R.
>47. In addition to general coverage, make sure to include tests for
>customers in the mining and railroading industries. Also test for
>completed and cancelled surgeries.")
>
>Testing without scripts is very powerful, more powerful than testing
>with scripts. Have a look at Session-Based Test Management at
>http://www.satisfice.com/sbtm.
>
>---Michael B.
| |
| Vladimir Trushkin 2007-09-03, 4:30 am |
| On Sep 1, 7:59 pm, Michael Bolton <michael.a.bol...@gmail.com> wrote:
> Hi, Alex...
>
> I'm answering off the list because I'd prefer to keep our conversation
> private.
>
>
> I see a problem with repetition, but I also see big problems with
> scripts.
In contrary, scripts are a wise choice if:
-- You think that the way testing is performed is as important as the
checkpoint claimed in the test.
-- You believe that review of test cases is not just a formal process
but a tool that helps removing badness.
-- You want to learn from each other in how to build effective tests.
-- You rely on building an effective organization when a couple of
experts may supply with scripts 10-20 novices.
-- You work on an evolving product and plan on using same tests often
for the regression testing purpose.
-- You have an automation engineer working next to the manual tester.
-- You work to the plans and want to have reliable estimates before
the work is done, not after.
If you are working in a [somewhat] stable environment then scripting
is not a waste of time but a thing that helps keeping the process (and
the result) under control. Thinking ahead is not always a bad thing.
----
Best Wishes,
Vladimir
|
|
|
|
|