Extreme Programming - Brute force testing

Author

Brute force testing

John B

2005-08-04, 10:01 pm

Ok, please bear with me as I am (very) new to this agile/xp/tdd thing
and am trying to get my head around it.

I have finally started a test TDD project.
I chose to do a bowling game, complete with frames, strikes, spares, etc..

Now, obviously if you throw a strike, the rest of the frame will not be
thrown, and the next frame should become valid.
Currently I have a Throw method which generates a random int between 0
and 10.
I have no way of "forcing" a strike so I loop until I get a strike,
calling my [Setup] method at the beginning of the loop.
This works but slows the tests "way" down.
I could re-design the game to refactor out the number generation to a
separate method which I could then subclass and override in a testing
class to return a strike or a spare as the case called for.

What would be the preferred way or am I way off base?

Cheers
JB

Paul Sinnett

2005-08-04, 10:01 pm

John B wrote:
> I could re-design the game to refactor out the number generation to a
> separate method which I could then subclass and override in a testing
> class to return a strike or a spare as the case called for.
>
> What would be the preferred way or am I way off base?

Usually, bowling game examples take the number as input to the
function: ie. Throw(5) or Throw(10). The program takes these throws as
input and returns the total score as output. This is an interesting
variant, assuming I am understanding you correctly, in that you want
the program to generate randomly created throws as well as score the
result. Is that correct?

Your suggested approach seems reasonable as you describe it, but it's
difficult to know without some code to look at.

John B

2005-08-04, 10:01 pm

Paul Sinnett wrote:
> John B wrote:
>
>
>
> Usually, bowling game examples take the number as input to the
> function: ie. Throw(5) or Throw(10). The program takes these throws as
> input and returns the total score as output. This is an interesting
> variant, assuming I am understanding you correctly, in that you want
> the program to generate randomly created throws as well as score the
> result. Is that correct?
>
Yes, this is correct.
I did consider the Throw(value) idea but wanted it to be contained.
I also wanted (erroneously maybe) to go with my first instinct instead
of taking what I sort of saw as a compromise.

> Your suggested approach seems reasonable as you describe it, but it's
> difficult to know without some code to look at.
>
I implemented it the way I described it by isolating a
GenerateThrowValue method and subclassing it in a test class, and adding
ThrowStrike and ThrowSpare methods which set a class field which
indicates the next throw value.

This works well and brought the time taken down from 0.12s to around
0.06s so it accomplished what I wanted.

However, another question.
I noticed when I was running tests in the red, the values I was getting
for each throw varied between 0 and 9.
When I investigated, I found that M$'s Random.Next(minValue, maxValue)
returns an integer greater than or equal to minValue but less than
maxValue, so I had to use 11 as the maxValue to include 10 in the random
throws.
This led me to believe that I should test to make sure that "each and
every" value was represnted.
Of course in this case I could not "simulate" a value like I did for the
strikes and spares, but would have to do it "real".
The only way I could think of to actually do this was the "brute force"
method, where I would just continue to throw (or call
GenerateThrowValue) until I had covered 0-10.

Would this be "acceptable"?

Apparently it takes 6 months to get the hang of TDD, this poor ng is
going to be swamped by me :)

Thanks for the help
JB

Paul Sinnett

2005-08-04, 10:01 pm

John B wrote:
> However, another question.
> I noticed when I was running tests in the red, the values I was getting
> for each throw varied between 0 and 9.
> When I investigated, I found that M$'s Random.Next(minValue, maxValue)
> returns an integer greater than or equal to minValue but less than
> maxValue, so I had to use 11 as the maxValue to include 10 in the random
> throws.
> This led me to believe that I should test to make sure that "each and
> every" value was represnted.
> Of course in this case I could not "simulate" a value like I did for the
> strikes and spares, but would have to do it "real".
> The only way I could think of to actually do this was the "brute force"
> method, where I would just continue to throw (or call
> GenerateThrowValue) until I had covered 0-10.
>
> Would this be "acceptable"?

While it would probably work in this case it doesn't seem practical.
You could not tell, for example, that the test would ever fail. You
might want to run a test like this if you were creating a random number
generating function. I would envision taking a statistical sample of
the output and checking that the frequency for each number was within
acceptable limits.

However, here I would trust MS to have worked out something acceptable.
If I was feeling paranoid about that I might write a small separate
test program to verify.

> Apparently it takes 6 months to get the hang of TDD, this poor ng is
> going to be swamped by me :)

I seriously doubt it would take anybody that long. You seem to be well
on your way already.

John Roth

2005-08-05, 5:01 pm

"John B" <jbngspam@yahoo.com> wrote in message
news:42f2a856$0$18647$14726298@news.sunsite.dk...
> Ok, please bear with me as I am (very) new to this agile/xp/tdd thing and
> am trying to get my head around it.
>
> I have finally started a test TDD project.
> I chose to do a bowling game, complete with frames, strikes, spares, etc..
>
> Now, obviously if you throw a strike, the rest of the frame will not be
> thrown, and the next frame should become valid.
> Currently I have a Throw method which generates a random int between 0 and
> 10.
> I have no way of "forcing" a strike so I loop until I get a strike,
> calling my [Setup] method at the beginning of the loop.
> This works but slows the tests "way" down.
> I could re-design the game to refactor out the number generation to a
> separate method which I could then subclass and override in a testing
> class to return a strike or a spare as the case called for.
>
> What would be the preferred way or am I way off base?

Random input won't help you unless you have a separate
oracle that can specify the output you expect. That's
overkill for this project (although there are projects where
it's a real good idea.)

Randomly generated values without an oracle are
only useful for pressure testing. Depending on the
situation, this may be a good thing.

The basic notion is that your test should be an
example of input, together with the expected output. Choosing
appropriate examples is a bit of an art that improves with
practice.

Try an input that's all strikes, then one that's all open frames,
then one that's all spares, then a mixture. By that point your
program should be working perfectly.

John Roth

>
> Cheers
> JB

paul campbell

2005-08-06, 9:01 am

On Fri, 05 Aug 2005 09:48:04 +1000, John B <jbngspam@yahoo.com> wrote:

> Ok, please bear with me as I am (very) new to this agile/xp/tdd thing
> and am trying to get my head around it.
>
> I have finally started a test TDD project.
> I chose to do a bowling game, complete with frames, strikes, spares,
> etc..
>
> Now, obviously if you throw a strike, the rest of the frame will not be
> thrown, and the next frame should become valid.
> Currently I have a Throw method which generates a random int between 0
> and 10.

IME brute force testing almost always wrong in TDD. With random
input you often have n input values driving the same code path. TDD tests
should be deterministic and have a near 1:1 mapping to code paths.

Often the use of random input to a test is effectively a substitute for
exhaustive testing over a range of values, which is itself the wrong thing
to do in TDD because you should only be interested in boundary values that
force out extra code paths.

> I have no way of "forcing" a strike so I loop until I get a strike,
> calling my [Setup] method at the beginning of the loop.
> This works but slows the tests "way" down.
> I could re-design the game to refactor out the number generation to a
> separate method which I could then subclass and override in a testing
> class to return a strike or a spare as the case called for.

Your design lacks cohesion. The throw point generation should be seperate
from the score tallying (single responsibility principal). This will
typically
manifest itself as some interface which yields a number being passed as an
initialisation or construction parameter on the score keeper, and which
also
serves as a covenient mocking point.

If you dont want to make an implementation dependency visible in the
classes
interface then yes there are several nefarious means of mocking including
overriding protected getters, introspection, and my own personal favorite:
dissabled initialiser.

A dissabled public initialiser looks like this:

class MyImplementationDependentClass {

MyImplementationDependentClass () {
this.thingie =
FactoryThatIDontWantToBeVisibleInMyClass
Interface.thingie();
}

public void aGenuineBusinessMethod() {
// do real business stuff that depends on thingie
}

// package/public scope as needed
setMockThingie (DependedOnThingie thingie) {
this.thingie = thingie;
throw new UnsupportedOperationException("for test use only !");
}
}

In a test you do this:

MyImplementationDependentClass m = new MyImplementationDependentClass();
Thingie t = new MockThingie();
try {
m.setMockThingie(t);
} catch (UnsupportedOperationException) {
// ignore
}

// test body here
.... aGenuineBusinessMethod()

Paul C.

Sponsored Links