Extreme Programming - How to do there Customer Acceptance Tests?

Author

How to do there Customer Acceptance Tests?

Steve Jorgensen

2004-09-23, 3:57 pm

Hi all,

I've been working with a team on w

ends to to a "real" XP project, and then
I read more about XP, and I see that Customer Acceptance Tests are a piece
that we just don't have yet. Reading Ron Jeffries' book "Extreme Programming
Adventures in C#", I see that these acceptance tests need not be rocket
science to implement.

On our project, though, I'm having trouble seeing how we would write
end-to-end customer acceptance tests.

Basically, the issue is that we're writing a system to do Mixed Code
Generation on VB/VBA code modules, and we're relying to a large extent on the
VB IDE itself to tell us some harder things about the code (like decide where
a procedure starts) so we don't have to reinvent that wheel and try to guess
all the test cases to see if our code correctly emulates what's already
available.

So, for the customer acceptance test, the only ways I can think of to do a
reliable end-to-end test are for the customer to supply actual Office
documents with embedded VBA code, or for our system to use customer test input
to generate and save such documents, so they can be run through the tests. It
seems like it would be a bad idea to mock the VB IDE in a customer acceptance
test.

Does anyone have any input for me on this?

Thanks,

Steve Jorgensen

Steve Jorgensen

2004-09-23, 3:57 pm

Oops - that should be "How to do these Customer Acceptance Tests?

Ronald E Jeffries

2004-09-24, 3:56 am

On Thu, 23 Sep 2004 15:28:33 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>Does anyone have any input for me on this?

Steve, I don't understand yet what the program does.

Tell us some of the Stories, perhaps?

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

Steve Jorgensen

2004-09-24, 8:57 am

On Thu, 23 Sep 2004 22:27:03 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Thu, 23 Sep 2004 15:28:33 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>Steve, I don't understand yet what the program does.
>
>Tell us some of the Stories, perhaps?

OK, first a broad overview. This system uses macros embedded in comments in
the code, and expansion point tags also embedded in the code to insert the
expanded macros at the expansion points.

We have a story called "Context Parameters". In this story, we must allow
parameters to be embedded in the macro definition that will be replaced with
contextual information about the code context at the insertion point, e.g.
<:ProcedureName:> or <:ProcedureType:>. This is required because VB/VBA
include the procedure type in the procedre exit statement ("Exit Sub" or "Exit
Function")

So the issue I have is that we have chosen to implement this by asking the
VBIDE. This allows us to not reinvent that wheel and to never have to worry
about whether our code's logic is consistent with Microsoft's logic. That
means, though, that our system can't fully process anything that isn't a
bonifide module opened in the VB IDE.

And that gets back to my question of how I should structure a Customer
Acceptance Test. The ideas I have are to either (a) require the user to
provide actual VB and/or VBA projects in a directory as input or (b) to have
the acceptance testing framework automatically generate these from textual
command input in order to test processing of them.

Ronald E Jeffries

2004-09-24, 8:57 am

On Fri, 24 Sep 2004 08:15:34 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>On Thu, 23 Sep 2004 22:27:03 -0400, Ronald E Jeffries <ronjeffries@acm.org>
>wrote:
>
>
>OK, first a broad overview. This system uses macros embedded in comments in
>the code, and expansion point tags also embedded in the code to insert the
>expanded macros at the expansion points.
>
>We have a story called "Context Parameters". In this story, we must allow
>parameters to be embedded in the macro definition that will be replaced with
>contextual information about the code context at the insertion point, e.g.
><:ProcedureName:> or <:ProcedureType:>. This is required because VB/VBA
>include the procedure type in the procedre exit statement ("Exit Sub" or "Exit
>Function")
>
>So the issue I have is that we have chosen to implement this by asking the
>VBIDE. This allows us to not reinvent that wheel and to never have to worry
>about whether our code's logic is consistent with Microsoft's logic. That
>means, though, that our system can't fully process anything that isn't a
>bonifide module opened in the VB IDE.
>
>And that gets back to my question of how I should structure a Customer
>Acceptance Test. The ideas I have are to either (a) require the user to
>provide actual VB and/or VBA projects in a directory as input or (b) to have
>the acceptance testing framework automatically generate these from textual
>command input in order to test processing of them.

Are you saying that the USER will type macros into some text, and
something will happen? Is this a program for programmers, or word
processor users, or what?

Who uses this program, why do they use it, and please give an example
of how they use it and what happens when they do?

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

Steve Jorgensen

2004-09-24, 4:00 pm

On Fri, 24 Sep 2004 08:21:13 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Fri, 24 Sep 2004 08:15:34 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>Are you saying that the USER will type macros into some text, and
>something will happen? Is this a program for programmers, or word
>processor users, or what?
>
>Who uses this program, why do they use it, and please give an example
>of how they use it and what happens when they do?

Ah. Yes, the user is a VB/VBA programmer. It is for the purpose of
automating generation of duplicated code that cannot be factored out in VB/VBA
such as error handling blocks.

Let's say that somewhere in some module in the project, the programmer types
the following...

'/<MacroDef name="Test1">
'/ Debug.Print "Test1.1"
'/ Debug.Print "Test1.2"
'/ Exit <:ProcType:>
'/</MacroDef>

Then, elsewhere in a procedure in another module, the programmer types

Public Sub Test()
'/<Macro name="Test1"/>
End Sub

The we run the metacode processor on it, the code containing the Macro tag now
looks like this...

Public Sub Test()
'/<Macro name="Test1">
Debug.Print "Test1.1"
Debug.Print "Test1.2"
Exit Sub
'/</Macro>
End Sub

If we change the macro, and re-run the processor, the previous expansions are
replaced according to the new definitions, which is possible because both the
start and end of the expanded macros are identified with tags.

Ronald E Jeffries

2004-09-25, 8:56 pm

On Fri, 24 Sep 2004 15:23:49 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:
[color=darkred]

I think I understand what you're up to, and I think I agree with your
basic plan. At a guess, I'd be building the tests up, maybe like this:

1. Some program samples in xUnit, just as text strings. (Hmm, but you
said you were using some of the VBIDE features for help. Well, I still
might do it with text.)

2. Maybe some of the source in text files, running thru the interface
in 1.

3. Then some real programs.

Each would have matching input and output "files".

Offhand, nothing much more clever comes to mind. If there's access to
the compiler's internal trees and such, I might be unit testing at
that level, but it seems that the acceptance tests want to look like
code.

Sounds like you're on the right track as far as I can tell from here.
Wish we could sit down and brainstorm more options, but I'm sure
you'll stay open to things to make it easier and better.

Regards,

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

Steve Jorgensen

2004-09-26, 3:55 am

On Sat, 25 Sep 2004 18:42:29 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Fri, 24 Sep 2004 15:23:49 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>I think I understand what you're up to, and I think I agree with your
>basic plan. At a guess, I'd be building the tests up, maybe like this:
>
>1. Some program samples in xUnit, just as text strings. (Hmm, but you
>said you were using some of the VBIDE features for help. Well, I still
>might do it with text.)

xUnit (in this case, VB Lite Unit) should not be involved in customer tests,
should it?

>2. Maybe some of the source in text files, running thru the interface
>in 1.
>
>3. Then some real programs.
>
>Each would have matching input and output "files".

That pretty much describes how our programmer tests look now except that the
"text files" are records in an Access database table. We in fact, have tests
that process text in strings embedded in the unit test code, process text read
from records in a table, and process sample code projects by copying them,
running the process, then checking then against expected result data in the
table records.

>Offhand, nothing much more clever comes to mind. If there's access to
>the compiler's internal trees and such, I might be unit testing at
>that level, but it seems that the acceptance tests want to look like
>code.
>
>Sounds like you're on the right track as far as I can tell from here.
>Wish we could sit down and brainstorm more options, but I'm sure
>you'll stay open to things to make it easier and better.

Thanks - I guess the things I'm most concerned about are...

1. What should we have the customer (well, cutomer hat) supply for sample
input/output: blocks of text in a file, actual code projects, both, or
something else?

2. How should the test results be indicated to the customer?

3. Making sure tests are as end-to-end as possible. I -think- we're doing
pretty well on this one from the Programmer Test side because there's very
little code directly in the UI layer, and the only time we've had a bug creep
into the UI without breaking unit tests, it turned out a test was simply
missing. Added the test - hole plugged.

>
>Regards,

John Roth

2004-09-26, 3:56 pm

"Steve Jorgensen" <nospam@nospam.nospam> wrote in message
news:gdbcl0572ba7rfc8utuh7pceufsnattgd4@
4ax.com...
> On Sat, 25 Sep 2004 18:42:29 -0400, Ronald E Jeffries
> <ronjeffries@acm.org>
> wrote:
>
>
> xUnit (in this case, VB Lite Unit) should not be involved in customer
> tests,
> should it?

It depends on the customer. If your customers are technically
inclined, then xUnit may be perfectly adequate (and it's one less
tool to deal with), If they're not, then you need a tool (such as
FIT) that is adapted to speaking the language the customer is
comfortable with.

We tend to recommend FIT, and downplay xUnit, because
the non-technical customers are in the overwhelming
majority.

John Roth

Ronald E Jeffries

2004-09-28, 3:59 am

On Sun, 26 Sep 2004 03:07:17 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>
>xUnit (in this case, VB Lite Unit) should not be involved in customer tests,
>should it?

It /could/ be. We build a customer framework on top of sUnit, for
example. But I'd not think it useful to use xUnit directly.

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

Ronald E Jeffries

2004-09-28, 3:59 am

On Sun, 26 Sep 2004 03:07:17 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>Thanks - I guess the things I'm most concerned about are...
>
>1. What should we have the customer (well, cutomer hat) supply for sample
>input/output: blocks of text in a file, actual code projects, both, or
>something else?

At a guess, small things will be easier and better-directed, and some
large things will be desirable to make sure that there aren't
bigness-related problems.
>
>2. How should the test results be indicated to the customer?

Ideally they specify the test, specify the right answer, push the
button whenever they care to, and see happy faces pop up when things
work.
>
>3. Making sure tests are as end-to-end as possible. I -think- we're doing
>pretty well on this one from the Programmer Test side because there's very
>little code directly in the UI layer, and the only time we've had a bug creep
>into the UI without breaking unit tests, it turned out a test was simply
>missing. Added the test - hole plugged.

Yep, that's the trick.

The thing is to adapt to what happens. The up front answers won't be
write. End-to-end is always further than you think.

I'd say carry on, and pay attention. If anything specific comes up
that you can't figure out, ask.

Regards,

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

Steve Jorgensen

2004-09-28, 3:59 am

On Mon, 27 Sep 2004 23:06:06 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Sun, 26 Sep 2004 03:07:17 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>At a guess, small things will be easier and better-directed, and some
>large things will be desirable to make sure that there aren't
>bigness-related problems.

So, I think you're saying to do both.

>
>Ideally they specify the test, specify the right answer, push the
>button whenever they care to, and see happy faces pop up when things
>work.

Well, to be more specific, how should the test results be tied back to the
test cases, so it's clear at a glance what passed and what failed?

A fit-style system would be

, perhaps using file names as test input for
the non-plain-text tests, but I don't know how easy it would be to hook FIT
into VB/VBA, and I suspect that's not the simplest thing that would work here.
I guess I just haven't envisioned what -is- the simplest thing, then.

Since the team knows VB/VBA better than anyone should ever want to, perhaps,
use an Excel document, and color the cells, or am I still thinking bigger than
I need to?

Sponsored Links