Home > Archive > Extreme Programming > March 2005 > are unit test and acceptance test enough ? how to integrate test in code ?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
are unit test and acceptance test enough ? how to integrate test in code ?
|
|
| emmanuel.caradec@gmail.com 2005-03-10, 8:56 am |
| There seem to be a lot of different kind of testing but xp seem to
focus on unit test and acceptance test. As I understand them :
- Unit test check that the logic of the code work
- Acceptance test check that the logic of the code match the customer
needs
In my case those tests are not able to cover all the testing that need
to be done.
Some of the bugs I am faced with can be found by testing the
application as a whole. I come up with a method that may have some
interest. A testing dll is loaded by the application that replace and
regulate the application event queue. The code can register some object
and states in the dll if it is loaded. I like this method because it
allow to test the released code at the binary level.
This way I can detect bug that occur because of the use of resource
with wrong number, bad COM registration, conflict with previous
version, etc...
May be my difficulties are related to the fact that I work on a c++/mfc
oriented project.
Do people doing xp need to test application as a whole to guaranty
behaviour or is it covered by UnitTest and AcceptanceTest only ? Or is
there another way ?
There is a lot of papers of how to make tables, use word to make
test... But I didn't find anything on how to integrate these tests in a
executable that has no external interface (com,script). How do people
do that ?
Emmanuel
| |
| John Roth 2005-03-10, 8:56 am |
| <emmanuel.caradec@gmail.com> wrote in message
news:1110449115.036571.270330@g14g2000cwa.googlegroups.com...
> There seem to be a lot of different kind of testing but xp seem to
> focus on unit test and acceptance test. As I understand them :
> - Unit test check that the logic of the code work
> - Acceptance test check that the logic of the code match the customer
> needs
> In my case those tests are not able to cover all the testing that need
> to be done.
The published descriptions of XP focus on what you do differently
from what any reasonably competent developer would expect.
So yes, there are a lot of kinds of testing that don't fit into the
"acceptance test, unit test" paradigm.
There's also a misconception of how XP uses the term "test."
In XP, a unit test specifies some aspect of how the code is
supposed to work, and also serves as a regression test to insure
that it continues to satisfy the specification. As a specification,
it's written first.
An acceptance test specifies some aspect of how the application
is supposed to work.
> Some of the bugs I am faced with can be found by testing the
> application as a whole. I come up with a method that may have some
> interest. A testing dll is loaded by the application that replace and
> regulate the application event queue. The code can register some object
> and states in the dll if it is loaded. I like this method because it
> allow to test the released code at the binary level.
> This way I can detect bug that occur because of the use of resource
> with wrong number, bad COM registration, conflict with previous
> version, etc...
> May be my difficulties are related to the fact that I work on a c++/mfc
> oriented project.
Every environment presents its own unique challenges.
> Do people doing xp need to test application as a whole to guaranty
> behaviour or is it covered by UnitTest and AcceptanceTest only ? Or is
> there another way ?
Acceptance tests normally assume you're testing the entire
application, since they're for user visible behavior.
What you're doing is verifying that the application was built
properly. In a properly constructed XP environment, that's
guraranteed by the automated build process: it should be
impossible for a developer to build the product with the
wrong components.
If your integration and build procedures do what's needed
to produce a correct build each and every time, a separate
verification step has got a relatively low payback, and could
actually cause more difficulty from false positives.
> There is a lot of papers of how to make tables, use word to make
> test... But I didn't find anything on how to integrate these tests in a
> executable that has no external interface (com,script). How do people
> do that ?
Mostly, we don't.
John Roth
>
> Emmanuel
>
| |
|
| emmanuel.caradec wrote:
> Some of the bugs I am faced with can be found by testing the
> application as a whole. I come up with a method that may have some
> interest. A testing dll is loaded by the application that replace and
> regulate the application event queue. The code can register some object
> and states in the dll if it is loaded. I like this method because it
> allow to test the released code at the binary level.
This is why the XP literature recommends, where possible...
- daily deployment into a reference situation
- w ly releases
- an onsite customer
XP does not recommend a titular Test Engineer role, because that role should
either emerge from team interaction, or be appointed by special business
needs.
To deploy daily, you must write build scripts that integrate and test. To
release wly, your onsite customer must track progress daily, and must be
able to take delivery (even only internally) of a working product with new
features, every w.
The goal is to force as many issues to surface, as early and often as
possible. Even issues that no programming or testing book ever though of.
> There is a lot of papers of how to make tables, use word to make
> test...
Those papers are using elaborate words to describe a very simple concept:
Your program has at least two skins. One for the end users, and the other
for your onsite customer. In that role, the skin needs a simple system to
store tables of inputs and expected outputs, and needs a system to run these
tests interactively and in batches.
> But I didn't find anything on how to integrate these tests in a
> executable that has no external interface (com,script). How do people
> do that ?
Because your customer is onsite, this new skin needn't be productized. It
can use only a light installer that makes assumption about your environment.
However, this expedience comes with a price. If your app has no pluggable
GUI layer, you must build a very light one.
--
Phlip
http://industrialxp.org/community/b...tUserInterfaces
| |
| Robert C. Martin 2005-03-10, 3:59 pm |
| On 10 Mar 2005 02:05:15 -0800, emmanuel.caradec@gmail.com wrote:
>There seem to be a lot of different kind of testing but xp seem to
>focus on unit test and acceptance test. As I understand them :
>- Unit test check that the logic of the code work
Unit tests *specify* how the logic of the code should work. The
distinction between "check" and "specify" is important. In XP, the
unit tests are written before the code that they specify. The unit
tests are very low level and very details specifications of the
behavior of a module.
>- Acceptance test check that the logic of the code match the customer
>needs
Again, it's a *specification*, not a check. The acceptance tests are
written first and act as the true requirements.
>
>Do people doing xp need to test application as a whole to guaranty
>behaviour or is it covered by UnitTest and AcceptanceTest only ? Or is
>there another way ?
Whole system testing is part of any good software development team, XP
or otherwise. XPers tend to automate as much of this as they can.
>
>There is a lot of papers of how to make tables, use word to make
>test... But I didn't find anything on how to integrate these tests in a
>executable that has no external interface (com,script). How do people
>do that ?
By using Fixtures that bind the tests to the application. See the
documentation on FIT and FitNesse (www.fitnesse.org) to see how this
is done.
-----
Robert C. Martin (Uncle Bob) | email: unclebob@objectmentor.com
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716
"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo
| |
| Gilligan 2005-03-11, 3:57 am |
| Emmanuel,
I had to refresh my mind on MFC, it has been two years.
So you replace the event queue, I guess you subtitute in your own
CWinThread:PumpMessage or AfxPreTranslateMessage or even
DispatchMessage or something similar. I am definitely rusty.
This replaced Message queue can register objects and states, do you
mean it can record objects and their states so that you can see what
the system had done?
I am  on how you detect the use of a resource with the wrong
number, is it because the resource is not found? If you know what the
correct resource id is in the test dll then why doesn't the real
application know the correct dll?
As for the difficulty in testing because you have no external
interface, do you mean, because you are in the MFC framework that it is
not possible to extract a class out of the system and test it
independently because it can not exist independently? I know that MFC
applications are tightly coupled and much of the code is generated by
the designer.
I am not an XP expert, just a novice, and I have read the remarks by
Mr. Roth, Mr. Martin, and Phlip and they seem to conform with
statements I have heard before and from my experiences on an Agile (not
XP, but similar) team.
The Unit tests they refer to are also known as programmer's tests.
Programmer's tests do not necessarily test in isolation like the
traditional definition of unit tests. These programmer's tests are
exactly how they describe, a specification approach / a design step,
and the test in the isolation of minimal changes.
As for testing MFC apps, isn't there something from Mercury, the makers
of WinRunner?
As for integration testing and agile methods, I have had great
difficulty in this area.
See the current discussion on this other group:
http://groups.yahoo.com/group/maver...ment/message/77
As for terms and definitions of testing, I just finished a paper to
help my friends and me talk about testing in the same terms. The XP
community may or may not agree with my defintions.
http://home.att.net/~geoffrey.slink...k/UnitTest.html
I would be glad to help if I could understand better what your dll is
really doing and how it is verifying things.
Geoff
| |
| emmanuel.caradec@gmail.com 2005-03-20, 8:56 pm |
| > If your integration and build procedures do what's needed
> to produce a correct build each and every time, a separate
> verification step has got a relatively low payback, and could
> actually cause more difficulty from false positives.
Some horribles errors occur because of stupid mistakes like forgetting
to increase the version number of a COM library causing the installer
to not upgrade the component. I do not understand how build procedure
could avoid that.
> Unit tests *specify* how the logic of the code should work. The
> distinction between "check" and "specify" is important.
I known that tests beeing specifications is a delicate subject. However
I feel what is important is the coupling between specification and
testing. There are easier way to specify an application like a pencil,
a pen but they are probably less efficient. Telling that the tests are
the specification seem like a shortcut to me.
> By using Fixtures that bind the tests to the application. See the
> documentation on FIT and FitNesse (www.fitnesse.org) to see how this
> is done.
I have looked at Fitnesse again and this is really what I was looking
for. Thank you.
I am missing of the basic brick to write my fixtures that bind the
tests. I was missing a gui layer. I cannot create a whole application
in the test at each time because mfc does let me do that. So I just let
it running normally and drive its network and dialog access. I had some
difficulties about where begin the gui layer. I ended with an C++
interface that activate the dialog input :
getDlg()->getButton("OK")->press();
getDlg()->getSliderCtl("SPEED")->GetPos();
I currently keep the mfc object and activate them by sending WM_*.
> This replaced Message queue can register objects and states, do you
> mean it can record objects and their states so that you can see what
> the system had done?
Yes, but I was missing an adequate architecture for driving my dialog
and query states of the application.
> I am on how you detect the use of a resource with the wrong
> number, is it because the resource is not found? If you know what the
> correct resource id is in the test dll then why doesn't the real
> application know the correct dll?
I do not detect wrong resource. I only detect incoherent behaviour that
can be caused by incorrect resources.
Emmanuel
| |
|
| emmanuel.caradec wrote:
[color=darkred]
> Some horribles errors occur because of stupid mistakes like forgetting
> to increase the version number of a COM library causing the installer
> to not upgrade the component. I do not understand how build procedure
> could avoid that.
Test the script that generates the installer. (That's not idle speculation.
I did it, to defeat that particular issue, among others.)
[color=darkred]
> I known that tests beeing specifications is a delicate subject. However
> I feel what is important is the coupling between specification and
> testing. There are easier way to specify an application like a pencil,
> a pen but they are probably less efficient. Telling that the tests are
> the specification seem like a shortcut to me.
From the point of view of the code, the tests are the ultimate
specification. The code can't survive unless it passes.
From the point of view of humans, tests can be a literate form of
communicating requirements. You get that with practice, and endless
refactoring of the tests. This refactoring leads to clean decoupling on both
sides, so the code becomes not only Designed For Testing, but also Designed
For Documentation.
> I have looked at Fitnesse again and this is really what I was looking
> for. Thank you.
> I am missing of the basic brick to write my fixtures that bind the
> tests. I was missing a gui layer.
This paper...
http://flea.sourceforge.net/gameTestServer.pdf
....describes a simpler fixture system than FIT uses.
> I cannot create a whole application
> in the test at each time because mfc does let me do that.
Actually, it probably does. However, I have never bothered to research how
to do it.
Decoupling MFC from itself is super hard when doing it the wrong way loses
you tool support. So long as that dratted Wizard is part of MFC, you might
as well use what you paid for. So if there were a right way to do it, only
an MFC guru could figure it out.
Instead, read /Working Effectively with Legacy Code/ by Mike Feathers, and
then use the /in-vivo/ testing pattern, like this:
http://www.xpsd.org/cgi-bin/wiki?TfuiForMfc
That introduces the test rig just after the Application stuff initializes,
and before the main window. That code is in a tractable location, so you can
create parts of the windows that you need.
> So I just let
> it running normally and drive its network and dialog access.
Then insert the test runner just after the main window raises, and after
these things are warmed up. However, any effort to decouple them will pay
you back a thousand times.
> I had some
> difficulties about where begin the gui layer. I ended with an C++
> interface that activate the dialog input :
> getDlg()->getButton("OK")->press();
> getDlg()->getSliderCtl("SPEED")->GetPos();
> I currently keep the mfc object and activate them by sending WM_*.
Why don't you just use SendDlgItemMessage(dlg, IDOK, BM_CLICK); ?
I don't understand how the strings help. Don't your tests have access to
resource.h?
[color=darkred]
> Yes, but I was missing an adequate architecture for driving my dialog
> and query states of the application.
You seem to have grown an adequate one. My paper TfuiForMfc shows how to
make that easier by writing a temporary test-side method called reveal().
You temporarily display your window or dialog, to check if it behaves like
your tests say it does.
--
Phlip
http://industrialxp.org/community/b...tUserInterfaces
| |
| emmanuel.caradec@gmail.com 2005-03-20, 8:56 pm |
| > Test the script that generates the installer. (That's not idle
speculation.
> I did it, to defeat that particular issue, among others.)
The problem is not in the installer. Its behaviour is correct. The data
given to it are incorrect. I may detect the bad versionning by
comparing byte to byte with the previous version. Are you suggesting
that ?
> From the point of view of humans, tests can be a literate form of
> communicating requirements. You get that with practice, and endless
> refactoring of the tests.
Ok I'll keep trying.
> This paper...
> http://flea.sourceforge.net/gameTestServer.pdf
> ...describes a simpler fixture system than FIT uses.
Command line testing seem so easy and simple :)
> Decoupling MFC from itself is super hard when doing it the wrong way
loses
> you tool support. So long as that dratted Wizard is part of MFC, you
might
> as well use what you paid for.
I did try to change the linker entry point to point somewhere else that
in the mfc dll... Doesnt work. I need a mfc guru :)
> Instead, read /Working Effectively with Legacy Code/ by Mike
Feathers, and
> then use the /in-vivo/ testing pattern.
In-vivo testing look similar to what I'm doing now. The book is
definitly on my future reading list. I found this in
http://www.xpsd.org/cgi-bin/wiki?TfuiForMfc
CCheckableGroupboxDemoDlg dlg;
m_pMainWnd = &dlg;
#ifdef DEBUG
runAllTests();
#endif
int nResponse = dlg.DoModal();
Actually I did something really similar:
CCheckableGroupboxDemoDlg dlg;
m_pMainWnd = &dlg;
HINSTANCE hTest = LoadLibrary("test.dll")
if(hTest)
{
(void (*runAllTests)()) = (void
(*)())GetProcAddress(hTest,"runAllTests");
runAllTests();
}
int nResponse = dlg.DoModal();
> I don't understand how the strings help
The application has skin. I already has the strings. I just put them
inadvertantly in my example because they seem natural.
Resource id are probably the way to go in usual case.
> My paper TfuiForMfc shows how to
> make that easier by writing a temporary test-side method called
reveal().
I'm doing that often in unit test.
Emmanuel
| |
|
| emmanuel.caradec wrote:
[color=darkred]
> The problem is not in the installer. Its behaviour is correct. The data
> given to it are incorrect. I may detect the bad versionning by
> comparing byte to byte with the previous version. Are you suggesting
> that ?
One way to create an MSI file is with DevEnv.exe (Visual Studio 7) and
VDPROJ files. They are written in a screwey text format, similar to the old
INI profile format.
I used TDD to write a parser for that format. This permited build scripts to
unambiguously stamp the correct GUID into those files.
> HINSTANCE hTest = LoadLibrary("test.dll")
> if(hTest)
> {
> (void (*runAllTests)()) = (void
> (*)())GetProcAddress(hTest,"runAllTests");
> runAllTests();
> }
>
> int nResponse = dlg.DoModal();
I hope that means you can run the tests after hitting Run in the editor?
(Formerly <F5>.)
[color=darkred]
> The application has skin. I already has the strings. I just put them
> inadvertantly in my example because they seem natural.
>
> Resource id are probably the way to go in usual case.
The strings are perfect; they are a Command Language.
--
Phlip
http://industrialxp.org/community/b...tUserInterfaces
| |
|
|
|
|
|