Extreme Programming - Busting the myth "You can't test your own code"

Author

Busting the myth "You can't test your own code"

sirgilligan

2004-03-18, 6:54 pm

I am working on some ideas of an agile approach for large teams. One
of the problems I have experienced in a large team working on a large
project is that integration testing is outside of the methodology.

In my Maverick Development I recommend that integration testing is
done by the developers, not by an outside team. Here is a snippet from
that document.

----------

Testing

Like many other agile approaches, unit tests are part of the method.
The unit test framework will include the ability to generate valid
instances of the objects that are tested. These object factories will
be used by the unit tests as well as the integration tests. If there
is any requirement from the "second dimension" that was not fulfilled
by the object definition it can be placed into the testing framework.

In Maverick Agile Development, integration testing is not an after
thought or an add-on. The developers write the integration tests just
as they do the unit tests. Why? Because they know how things are
created and used. An external team will use most of its time trying to
discover how to properly create objects and initiate some process in
the system. External testing teams create the need for documentation,
meetings, and other distractions for development work.

Maverick Agile Development uses pair programming for integration
issues. When one method calls into another part of the system, the
developers of each part of the system pair with the each other to make
sure the system is called correctly. They review the call, the return
values, and the exceptions and refactor accordingly.

-----------

Recently I was challenged with the old statement, "You can't test your
own code! Everyone knows that." The Maverick Model says, "Question
everything" and I say do it until you understand the reasons for the
statement. This is like distillation, or another way of saying it,
"Getting to the meat"

So, I say, "Bull. You can test your own code. The reason why we
haven't done it well before is due to social reasons, not capability
reasons."

Now please note I am doing a brain storm here, and dumping out
everything that I can think of from my past. (I started coding in 1987
on small DOS product, Graduated in 1992 with MS in CS and first "real"
job was that same year. My first computer was a Mac SE using
Lightspeed's Object C and Object Pascal. Structured languages were a
second to me. I took to the single inheritence langauges like a fish
to water. I loved C++ and bought a compiler in 1989. I love Java now
and 90% of my code is currently Java).

Reasons why the myth "You can't test your own code" exists.
1) Management used a manufacturing model for development. Developers
have expertise in writing code. Developers are expensive. Testing can
be done in a cost effective way by other people. The birth of the
testing department! Once a department is created it will evolve to
survive and during that evolution any amount of infrastructure needed
to support the "creature" will be developed.

2) Programmers are not fully trained as what I refer to as Computer
Scientists. They write code that does not have pre and post
conditions. Where invariants are not used. Where knowledge of the
built-in types are not understood (such as signed and unsigned). Where
the knowledge of type conversions, copy constructors, and method
signatures are not understood. If programmers don't understand these
things and many more, they cannot test their code because they do not
recognize where assumptions are made and where potential problems and
bugs may exist. As I think about this, I realize that the tester for
this programmer has to be a senior engineer so that he can write tests
that expose the requirements, assumptions, and states of the code.

3) You can't write your own tests because you will do the same things
in the same fashion. This "feels" like it was based on the idea that
you can't proof read your own writing. Well, there is truth that we do
things in a similar fashion. Is that enough to say you can not test
your own code? I think you can test your own code but you have to step
back and look at the code and then attack it.

So, I ask, what other ideas do you come up with on why the statement
"you can't test your own code" exists?

I can think of reasons why programmers shouldn't test their own code.
These reasons can all be corrected.

1) Programmers will not log bugs against themselves. Why? Because the
bug count is viewed in a negative connotation. They will fix the bugs
they encounter but will not log the bug to defect tracking database.

2) Programmers will not take time to properly test their code. As long
as the code is working it is easy to hide the fact that it is not
thoroughly tested. If metrics for feature completion does not measure
testing as work, then testing will be neglected. The industry is so
cut-throat and lay-offs come so quickly programmers will do what ever
it takes to "look" good to management. If management doesn't
understand all of the nuances of developing software they will measure
traditional things such as features completed. If the programmer isn't
really rewarded for testing their software then they will not test the
software well. Now we must resist the idea that the developer is too
expensive to test their own software. I think that is a myth as well.

3) The tests for the code verify the code, but the code isn't valid
and thus the tests are invalid as well. A single programmer can easily
misinterpret the requirements. The code he develops can be bug free
and the tests can be very good and cover the code and the conditions.
But the code does not "answer" the right question! Agile methods
already address this issue. Continuous integration, pair programming,
short interations, and other methods address this.

So, can we bust this myth or not?????

Sincerely,
Geoff

Steve Jorgensen

2004-03-18, 6:54 pm

On 16 Mar 2004 13:05:18 -0800, sirgilligan@yahoo.com (sirgilligan) wrote:

....
>So, I ask, what other ideas do you come up with on why the statement
>"you can't test your own code" exists?

I'm no expert, but I think the following is a pretty good regurgitation of
ideas I've picked up from others here.

There is a very good reason the statement exists, and that reason should be
distinguished from the kinds of testing described in Extreme Programming.
Programmer, Test-First tests produce much more reliable code, and that's good
enough by itself for a lot of things. If you want more certainty of the
quality and reliability of the code, it makes perfect sense to have someone
else write test-after tests as well, they just won't find as many problems as
they would if the code had not been written TDD in the first place.

If you want more certainty, by all means, do what it takes to have it.

Jason

2004-03-18, 6:54 pm

sirgilligan wrote:

> I am working on some ideas of an agile approach for large teams. One
> of the problems I have experienced in a large team working on a large
> project is that integration testing is outside of the methodology.

> In my Maverick Development I recommend that integration testing is
> done by the developers, not by an outside team. Here is a snippet from
> that document.

Tests can serve many roles. For example, validation tests can be used
during continuous integration to verify that previously passing validation
tests have not been broken and the validation test being worked on passes
even after integrating the latest changes with the rest of the system. So,
validation tests are also integration tests.

But, IMHO validation tests serve a much more important role. They capture
the communication between the customer (the one defining requirements) and
the development team (the one's implementing the requirements). Without
validation tests, the developers don't know when they are really done, or
what done even means.

> ----------

> Testing

> Like many other agile approaches, unit tests are part of the method.
> The unit test framework will include the ability to generate valid
> instances of the objects that are tested. These object factories will
> be used by the unit tests as well as the integration tests. If there
> is any requirement from the "second dimension" that was not fulfilled
> by the object definition it can be placed into the testing framework.

> In Maverick Agile Development, integration testing is not an after
> thought or an add-on. The developers write the integration tests just
> as they do the unit tests. Why? Because they know how things are
> created and used. An external team will use most of its time trying to
> discover how to properly create objects and initiate some process in
> the system. External testing teams create the need for documentation,
> meetings, and other distractions for development work.

Validation (Integration) Tests are not Unit Tests. Unit Tests are closely
related to the objects they are testing. Validation Tests should not be
closely coupled to the various internal objects of a module or system
being developed. The only objects a Validation Test would be coupled with
would be some test fixture code necessary to implement the test itself.
The business objects should probably be separate.

> Maverick Agile Development uses pair programming for integration
> issues. When one method calls into another part of the system, the
> developers of each part of the system pair with the each other to make
> sure the system is called correctly. They review the call, the return
> values, and the exceptions and refactor accordingly.

> -----------

> Recently I was challenged with the old statement, "You can't test your
> own code! Everyone knows that." The Maverick Model says, "Question
> everything" and I say do it until you understand the reasons for the
> statement. This is like distillation, or another way of saying it,
> "Getting to the meat"

You are quoting a saying that's been around for quite some time. It's flat
out untrue. Developers can and should write their own Unit Tests. This is
critical in TDD.

However, if developers define their own Validation Tests, it is critical
to have the customer sign off on them. This is difficult at best and often
leads to the team just guessing at what is required. At worst, you may
wind up with no Validation Tests at all, or almost meaningless ones.
Again, the customer must be involved, because the customer should be
deciding what done means.

From this perspective, the tests define what the system does (and
conversely what it is not expected to do - by tests that are not defined).
Some people like to refer to things that define what a system does and
doesn't do as "documentation". But I digress.

> So, I say, "Bull. You can test your own code. The reason why we
> haven't done it well before is due to social reasons, not capability
> reasons."

I would submit that until xUnit came along, developers flat out avoided
writing tests at almost all costs.

> Now please note I am doing a brain storm here, and dumping out
> everything that I can think of from my past. (I started coding in 1987
> on small DOS product, Graduated in 1992 with MS in CS and first "real"
> job was that same year. My first computer was a Mac SE using
> Lightspeed's Object C and Object Pascal. Structured languages were a
> second to me. I took to the single inheritence langauges like a fish
> to water. I loved C++ and bought a compiler in 1989. I love Java now
> and 90% of my code is currently Java).

> Reasons why the myth "You can't test your own code" exists.
> 1) Management used a manufacturing model for development. Developers
> have expertise in writing code. Developers are expensive. Testing can
> be done in a cost effective way by other people. The birth of the
> testing department! Once a department is created it will evolve to
> survive and during that evolution any amount of infrastructure needed
> to support the "creature" will be developed.

I've worked with managers that previously seemed to think that writing
tests would be unnecessary if developers would just write bug free code.
This is why the project would put off the tests until the end. Things that
are put off are things that are perceived as not important. Man managers
probably now realize that developers don't just sit down and write bug
free code, but probably don't know why.

> 2) Programmers are not fully trained as what I refer to as Computer
> Scientists. They write code that does not have pre and post
> conditions. Where invariants are not used. Where knowledge of the
> built-in types are not understood (such as signed and unsigned). Where
> the knowledge of type conversions, copy constructors, and method
> signatures are not understood. If programmers don't understand these
> things and many more, they cannot test their code because they do not
> recognize where assumptions are made and where potential problems and
> bugs may exist. As I think about this, I realize that the tester for
> this programmer has to be a senior engineer so that he can write tests
> that expose the requirements, assumptions, and states of the code.

You are no longer talking about Validation or Integration Tests. You are
talking about Unit, Regression, and Lint-type tests.

> 3) You can't write your own tests because you will do the same things
> in the same fashion. This "feels" like it was based on the idea that
> you can't proof read your own writing. Well, there is truth that we do
> things in a similar fashion. Is that enough to say you can not test
> your own code? I think you can test your own code but you have to step
> back and look at the code and then attack it.

You can test your own code very well if you practice TDD. The result is
that you get to see your interfaces in use before you code them. That is
the best time to correct your interface. You don't have to change the
code, because you haven't written it yet.

> So, I ask, what other ideas do you come up with on why the statement
> "you can't test your own code" exists?

<snip>

> So, can we bust this myth or not?????
It has already been done. See TDD.

> Sincerely,
> Geoff

sirgilligan

2004-03-18, 6:54 pm

nocksj@sourcextreme.com (Jason) wrote in message news:<485a6$40580275$cff54506$1691@dcanet.allthenewsgroups.com>...
> sirgilligan wrote:
>
>

Good to hear that. How is the "word" getting spread? It was yesterday
that our Product Manager said, "you can't test your own code. We can't
have that."

How is TDD turning Managers into Thinkers, chiefs instead of cooks,
etc?

Is TDD addressing these issues where you have Developers and Managers?

Thanks,
Geoff

sirgilligan

2004-03-18, 6:54 pm

Steve Jorgensen <nospam@nospam.nospam> wrote in message
> If you want more certainty, by all means, do what it takes to have it.

I agree 100%. Do what ever it takes to achieve the quality that is
required for the product.

Black box testing can be done by someone other than the developer. In
fact, when the product ships, it is really being black-box tested by
all of the users, and they are paying you to do it!

If the test is a white-box type, regardless of being unit, component,
regression, performance, etc., then where are the lines drawn on who
can most correctly test the code?

I think agile methods have shown that Unit tests can be done by the
developer.
I propose that integration tests can be done by the developer.
Currently our dev process has a seperate team write the integration
tests and the component tests and the performance tests... basically
everything but the unit tests.

Here are some real problems with having another team write the
integration tests.

1) They have no power to have code that is being tested changed to
facilitate the tests. I know, everyone is going to say, "You can't
change the code to facilitate tests, it changes the code, introduces
bugs, and is intrusive." I would point out that agile methods have
started to bust that myth as well. The developers write their unit
tests, and if there is some behavior needed by those tests they
include that behavior and write tests to test the new behavior.

2) The integration test team has to learn the context and the language
of at least two systems. If there is an abstraction layer that
translates between the two systems then there is a third piece the
test engineer has to learn. To be able to test, you have to know the
requirements of the system and test for that, thus you have to know as
much about the requirements and domain of the system as the developer
that wrote the solution. To learn this you have to studying the
requirements (user stories, use cases, whatever they are) and you have
to get into the "mind" of the implementation. This requires
documentation of the code at a level that is not necessarily part of
an agile method. This requires pulling the developer at times to
gather information.

I have proposed that either the developers do the integration tests (I
have not got to the issues of component tests and performance
tests...) or that the integration test engineers become customers of
the product. I just started reading Domain Driven Design yesterday and
I was pleased to see that there is the idea of customer/supplier in
the seperations when a unified context is not possible. In his terms
it seems that integration testing is a conformist at best and all of
the problems with being at the mercy of another are present.

Thanks for your comments,
Geoff

Jason

2004-03-18, 6:54 pm

sirgilligan wrote:

> nocksj@sourcextreme.com (Jason) wrote in message
news:<485a6$40580275$cff54506$1691@dcanet.allthenewsgroups.com>...
[color=darkred]
> Good to hear that. How is the "word" getting spread? It was yesterday
> that our Product Manager said, "you can't test your own code. We can't
> have that."

Well, for starters, the newsgroup you are currently interacting with is
one way to spread "the word". How did you hear about TDD or XP?

Convincing people to try something new is ultimately a salesmanship issue.
Unless a person discovers these concepts on his or her own, you need to
convince that person if you want a set of ideas to be embraced.

IMHO, the TDD practices of Simple Design, Test First, and Refactoring are
critical practices in XP. So, if you are doing XP, you should already be
doing TDD, or have a strongly compelling reason for doing otherwise (and
limit it to where the compulsion is present).

I am not an evangelist. XP and TDD are not for everyone. If you are
interested in XP practices, particularly TDD, then the thoughts presented
will ideally be of some help to you.

> How is TDD turning Managers into Thinkers, chiefs instead of cooks,
> etc?

Out of Kent Beck's TDD book:
<I>Clean code that works</I>, in Ron Jeffries' pithy phrase is the goal of
Test-Driven Development.

TDD is a set of practices, it does not change people, only how they
approach software development, and only if they choose to adopt that
particular set of practices.

If someone does not wish to think, then you might not want to work with
that person. Typically, however, it is not that someone does not wish to
think, it is that they do not want to try something new. Again, this is
really a sales issue. You need to sell them on considering your ideas or
the ideas of others.

> Is TDD addressing these issues where you have Developers and Managers?

If you mean me personally, I generally serve both roles. But, I assume
that I am an exception.

In general, getting a team of many different types of people to work
together is the focus of the manager. If the manager is going to stick to
some other approach no matter what you have to say, then you probably will
not be able to sell them on the merits of any other approach. That has
little to do with whatever approach you decide to sell them on and more to
do with the nature of human behavior (of which I am most definitely not an
expert).

Probably the vast majority of teams that adopt (not adapt) XP and TDD have
worked through issues between Developers, Managers, and Customers all
gaining acceptance.

> Thanks,
> Geoff

Cheers,
Jason Nocks

Sponsored Links