How to do there Customer Acceptance Tests?

Hi all,

I've been working with a team on wends to to a "real" XP project, and the
n
I read more about XP, and I see that Customer Acceptance Tests are a piece
that we just don't have yet.  Reading Ron Jeffries' book "Extreme Programmin
g
Adventures in C#", I see that these acceptance tests need not be rocket
science to implement.

On our project, though, I'm having trouble seeing how we would write
end-to-end customer acceptance tests.

Basically, the issue is that we're writing a system to do Mixed Code
Generation on VB/VBA code modules, and we're relying to a large extent on th
e
VB IDE itself to tell us some harder things about the code (like decide wher
e
a procedure starts) so we don't have to reinvent that wheel and try to guess
all the test cases to see if our code correctly emulates what's already
available.

So, for the customer acceptance test, the only ways I can think of to do a
reliable end-to-end test are for the customer to supply actual Office
documents with embedded VBA code, or for our system to use customer test inp
ut
to generate and save such documents, so they can be run through the tests.  
It
seems like it would be a bad idea to mock the VB IDE in a customer acceptanc
e
test.

Does anyone have any input for me on this?

Thanks,

Steve Jorgensen

Oops - that should be "How to do these Customer Acceptance Tests?

On Thu, 23 Sep 2004 15:28:33 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>Does anyone have any input for me on this?

Steve, I don't understand yet what the program does.

Tell us some of the Stories, perhaps?

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

On Thu, 23 Sep 2004 22:27:03 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Thu, 23 Sep 2004 15:28:33 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
> 
>
>Steve, I don't understand yet what the program does.
>
>Tell us some of the Stories, perhaps?

OK, first a broad overview.  This system uses macros embedded in comments in
the code, and expansion point tags also embedded in the code to insert the
expanded macros at the expansion points.

We have a story called "Context Parameters".  In this story, we must allow
parameters to be embedded in the macro definition that will be replaced with
contextual information about the code context at the insertion point, e.g.
<:ProcedureName:> or <:ProcedureType:>.  This is required because VB/VBA
include the procedure type in the procedre exit statement ("Exit Sub" or "Ex
it
Function")

So the issue I have is that we have chosen to implement this by asking the
VBIDE.  This allows us to not reinvent that wheel and to never have to worry
about whether our code's logic is consistent with Microsoft's logic.  That
means, though, that our system can't fully process anything that isn't a
bonifide module opened in the VB IDE.

And that gets back to my question of how I should structure a Customer
Acceptance Test.  The ideas I have are to either (a) require the user to
provide actual VB and/or VBA projects in a directory as input or (b) to have
the acceptance testing framework automatically generate these from textual
command input in order to test processing of them.

On Fri, 24 Sep 2004 08:15:34 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>On Thu, 23 Sep 2004 22:27:03 -0400, Ronald E Jeffries <ronjeffries@acm.org>
>wrote:
> 
>
>OK, first a broad overview.  This system uses macros embedded in comments i
n
>the code, and expansion point tags also embedded in the code to insert the
>expanded macros at the expansion points.
>
>We have a story called "Context Parameters".  In this story, we must allow
>parameters to be embedded in the macro definition that will be replaced wit
h
>contextual information about the code context at the insertion point, e.g.
><:ProcedureName:> or <:ProcedureType:>.  This is required because VB/VBA
>include the procedure type in the procedre exit statement ("Exit Sub" or "E
xit
>Function")
>
>So the issue I have is that we have chosen to implement this by asking the
>VBIDE.  This allows us to not reinvent that wheel and to never have to worr
y
>about whether our code's logic is consistent with Microsoft's logic.  That
>means, though, that our system can't fully process anything that isn't a
>bonifide module opened in the VB IDE.
>
>And that gets back to my question of how I should structure a Customer
>Acceptance Test.  The ideas I have are to either (a) require the user to
>provide actual VB and/or VBA projects in a directory as input or (b) to hav
e
>the acceptance testing framework automatically generate these from textual
>command input in order to test processing of them.

Are you saying that the USER will type macros into some text, and
something will happen? Is this a program for programmers, or word
processor users, or what?

Who uses this program, why do they use it, and please give an example
of how they use it and what happens when they do?

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

On Fri, 24 Sep 2004 08:21:13 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Fri, 24 Sep 2004 08:15:34 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
> 
>
>Are you saying that the USER will type macros into some text, and
>something will happen? Is this a program for programmers, or word
>processor users, or what?
>
>Who uses this program, why do they use it, and please give an example
>of how they use it and what happens when they do?


Ah.  Yes, the user is a VB/VBA programmer.  It is for the purpose of
automating generation of duplicated code that cannot be factored out in VB/V
BA
such as error handling blocks.

Let's say that somewhere in some module in the project, the programmer types
the following...

'/<MacroDef name="Test1">
'/    Debug.Print "Test1.1"
'/    Debug.Print "Test1.2"
'/    Exit <:ProcType:>
'/</MacroDef>

Then, elsewhere in a procedure in another module, the programmer types

Public Sub Test()
'/<Macro name="Test1"/>
End Sub

The we run the metacode processor on it, the code containing the Macro tag n
ow
looks like this...

Public Sub Test()
'/<Macro name="Test1">
Debug.Print "Test1.1"
Debug.Print "Test1.2"
Exit Sub
'/</Macro>
End Sub

If we change the macro, and re-run the processor, the previous expansions ar
e
replaced according to the new definitions, which is possible because both th
e
start and end of the expanded macros are identified with tags.

On Fri, 24 Sep 2004 15:23:49 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:
 

I think I understand what you're up to, and I think I agree with your
basic plan. At a guess, I'd be building the tests up, maybe like this:

1. Some program samples in xUnit, just as text strings. (Hmm, but you
said you were using some of the VBIDE features for help. Well, I still
might do it with text.)

2. Maybe some of the source in text files, running thru the interface
in 1.

3. Then some real programs.

Each would have matching input and output "files".

Offhand, nothing much more clever comes to mind. If there's access to
the compiler's internal trees and such, I might be unit testing at
that level, but it seems that the acceptance tests want to look like
code.

Sounds like you're on the right track as far as I can tell from here.
Wish we could sit down and brainstorm more options, but I'm sure
you'll stay open to things to make it easier and better.

Regards,

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

On Sat, 25 Sep 2004 18:42:29 -0400, Ronald E Jeffries <ronjeffries@acm.org>
wrote:

>On Fri, 24 Sep 2004 15:23:49 GMT, Steve Jorgensen
><nospam@nospam.nospam> wrote:
> 
>
>I think I understand what you're up to, and I think I agree with your
>basic plan. At a guess, I'd be building the tests up, maybe like this:
>
>1. Some program samples in xUnit, just as text strings. (Hmm, but you
>said you were using some of the VBIDE features for help. Well, I still
>might do it with text.)

xUnit (in this case, VB Lite Unit) should not be involved in customer tests,
should it?

>2. Maybe some of the source in text files, running thru the interface
>in 1.
>
>3. Then some real programs.
>
>Each would have matching input and output "files".

That pretty much describes how our programmer tests look now except that the
"text files" are records in an Access database table.  We in fact, have test
s
that process text in strings embedded in the unit test code, process text re
ad
from records in a table, and process sample code projects by copying them,
running the process, then checking then against expected result data in the
table records.

>Offhand, nothing much more clever comes to mind. If there's access to
>the compiler's internal trees and such, I might be unit testing at
>that level, but it seems that the acceptance tests want to look like
>code.
>
>Sounds like you're on the right track as far as I can tell from here.
>Wish we could sit down and brainstorm more options, but I'm sure
>you'll stay open to things to make it easier and better.

Thanks - I guess the things I'm most concerned about are...

1. What should we have the customer (well, cutomer hat) supply for sample
input/output: blocks of text in a file, actual code projects, both, or
something else?

2. How should the test results be indicated to the customer?

3. Making sure tests are as end-to-end as possible.  I -think- we're doing
pretty well on this one from the Programmer Test side because there's very
little code directly in the UI layer, and the only time we've had a bug cree
p
into the UI without breaking unit tests, it turned out a test was simply
missing.  Added the test - hole plugged.

>
>Regards,

"Steve Jorgensen" <nospam@nospam.nospam> wrote in message
 news:gdbcl0572ba7rfc8utuh7pceufsnattgd4@
4ax.com...
> On Sat, 25 Sep 2004 18:42:29 -0400, Ronald E Jeffries
> <ronjeffries@acm.org>
> wrote:
>
>
> xUnit (in this case, VB Lite Unit) should not be involved in customer
> tests,
> should it?

It depends on the customer. If your customers are technically
inclined, then xUnit may be perfectly adequate (and it's one less
tool to deal with), If they're not, then you need a tool (such as
FIT) that is adapted to speaking the language the customer is
comfortable with.

We tend to recommend FIT, and downplay xUnit, because
the non-technical customers are in the overwhelming
majority.

John Roth

On Sun, 26 Sep 2004 03:07:17 GMT, Steve Jorgensen
<nospam@nospam.nospam> wrote:
 
>
>xUnit (in this case, VB Lite Unit) should not be involved in customer tests
,
>should it?

It /could/ be. We build a customer framework on top of sUnit, for
example. But I'd not think it useful to use xUnit directly.

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.