Software Testing - Where to Put C++ Test Code

Author

Where to Put C++ Test Code

cslowinski@gmail.com

2007-07-13, 8:15 am

Hi,

I was chatting with a colleague about the merits of putting C++ unit
test code in a separate directory from production code. He felt that
it was more appropriate to put test code in the same file as the
production code (guarded by preprocessor definitions so it could be
removed during a production compile). His reasoning was twofold:

1. If a developer wanted to include some subset of a library's source
for use in another project, the developer would not have to sift
through test directories to figure out what the corresponding subset
of test code to take too.

2. During compilation or linking (depending on what form the library
is in), compiler definitions could change the behaviour of the code.
Therefore, by keeping the tests in the same compilation unit, they
will be placed in the same object file and be subjected to identical
conditions as the production code. When the tests are run, they will
include not only the tests for the application itself, but the
libraries too, proving the libraries check out in this "new"
environment.

I'm hesitant to place test and production code together for a number
of reasons, one being that editing test or production code can have
inadvertent effects on the other. For example, refactoring some test
code using search and replace and -- whoops -- production code was
changed in a subtle but destructive way, causing anything from no
effect to incovenience to catastrophic bugs.

I was wondering what people's thoughts on the subject are.

Thanks,

Chris

adam

2007-07-13, 7:17 pm

On Jul 13, 8:57 am, cslowin...@gmail.com wrote:
> Hi,
>
> I was chatting with a colleague about the merits of putting C++ unit
> test code in a separate directory from production code. He felt that
> it was more appropriate to put test code in the same file as the
> production code (guarded by preprocessor definitions so it could be
> removed during a production compile). His reasoning was twofold:
>
> 1. If a developer wanted to include some subset of a library's source
> for use in another project, the developer would not have to sift
> through test directories to figure out what the corresponding subset
> of test code to take too.
>
> 2. During compilation or linking (depending on what form the library
> is in), compiler definitions could change the behaviour of the code.
> Therefore, by keeping the tests in the same compilation unit, they
> will be placed in the same object file and be subjected to identical
> conditions as the production code. When the tests are run, they will
> include not only the tests for the application itself, but the
> libraries too, proving the libraries check out in this "new"
> environment.
>
> I'm hesitant to place test and production code together for a number
> of reasons, one being that editing test or production code can have
> inadvertent effects on the other. For example, refactoring some test
> code using search and replace and -- whoops -- production code was
> changed in a subtle but destructive way, causing anything from no
> effect to incovenience to catastrophic bugs.
>
> I was wondering what people's thoughts on the subject are.
>
> Thanks,
>
> Chris

If you have not read through http://xunitpatterns.com/Organization.html
(and the other sections) you might want to. The answer as to what is
'right' is dependent on a lot of other factors which he helps explain.
The site is 'out of date' in that the final review changes did not
make it back to the site but are in the final print book.

-adam
http://www.ninjatactics.com/blog

H. S. Lahman

2007-07-13, 7:17 pm

Responding to slowinski...

> I was chatting with a colleague about the merits of putting C++ unit
> test code in a separate directory from production code. He felt that
> it was more appropriate to put test code in the same file as the
> production code (guarded by preprocessor definitions so it could be
> removed during a production compile). His reasoning was twofold:
>
> 1. If a developer wanted to include some subset of a library's source
> for use in another project, the developer would not have to sift
> through test directories to figure out what the corresponding subset
> of test code to take too.

I don't find this very convincing. If the developer can find reusable
library code, the developer should be able to use the same system to
find particular unit test cases.

> 2. During compilation or linking (depending on what form the library
> is in), compiler definitions could change the behaviour of the code.
> Therefore, by keeping the tests in the same compilation unit, they
> will be placed in the same object file and be subjected to identical
> conditions as the production code. When the tests are run, they will
> include not only the tests for the application itself, but the
> libraries too, proving the libraries check out in this "new"
> environment.

The other side of that coin is that such linking may just ensure that
the tests and the code are consistent with each other but not with the
production context. Actually, I think this is a strong reason for
separating the test code from the UUT code because it increases the
probability that a synchronization inconsistency in the UUT code can be
detected.

That is, there is a certain production environment that should be the
"current" one. The UUT code may or may not be synchronized with that
environment. One needs to maximize the chances that an inconsistency
will be detected. The best way to do that is to synchronize the test
code separately. Then if a synchronization problem arises during test
execution one can work from there to figure out which one is wrong.

Actually, synchronization is probably more important at the header file
and build file level than for compiler switches and pragmas. If the UUT
code was compiled against the wrong headers, in the wrong directory,
with the wrong library, or whatever, one wants to be able to detect
that. The test is unlikely to detect such synchronization problems
directly, but there is at least a chance that the problem will be
manifested if the the test code is built separately.

> I'm hesitant to place test and production code together for a number
> of reasons, one being that editing test or production code can have
> inadvertent effects on the other. For example, refactoring some test
> code using search and replace and -- whoops -- production code was
> changed in a subtle but destructive way, causing anything from no
> effect to incovenience to catastrophic bugs.

I very strongly agree with you. Testing is a quite different problem
than the one being solved for the customer in the UUT. When modifying
test cases one does not want to be touching anything remotely resembling
production code. Side effects are the bane of software maintenance so
one needs to isolate them as much as possible. Keeping and maintaining
test code separately from production code is the best way to do that.

Ask your colleague what happens when the UUT code is reused in different
applications and the requirements for one application are more stringent
than for the other (e.g., the data domain of an input is more limited in
one context than in the other). As a contrived example of the problem
consider:

UUTMethod (int x)
{
ASSERT (x <= Context::maxIntValue);
....
}

How does a single automated unit test embedded in UUTMethod's file know
which values of 'x' should work and which should throw an exception for
a particular application context? One can white box it and reference
maxIntValue in the test code to do the right thing. However, that is a
very slippery slope.

There is a big difference between using the design to determine what
should be tested and implementing the test itself with design
mechanisms. In the first case one is tailoring the test cases to
potential vulnerabilities of the design and the notion of what is
correct is still rooted in requirements. In the second case one is using
the design itself to determine what is correct and what is not. That
risks a Type II error.

*************
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH

Sponsored Links