Unit and Acceptance tests for report generation

I'm going to be doing some work with jasperreports soon, I'm wondering if
anyone else here has dealt with that or something similar in an Agile
development environment.  How do you implement acceptance tests and unit tes
ts
for the reporting stories?

Thanks,

Steve J.

On Fri, 15 Apr 2005 19:35:16 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>I'm going to be doing some work with jasperreports soon, I'm wondering if
>anyone else here has dealt with that or something similar in an Agile
>development environment.  How do you implement acceptance tests and unit te
sts
>for the reporting stories?

Reports are formatted data structures.  The first step in writing both
unit tests and acceptance tests is to create the data structures that
represent the reports.  Then you can test that the data is being
located properly, and the data structure if being filled in properly.

Then you can test the formatting simply by loading canned data
structures and ensuring those data structures are being formatted
correctly into the report.


-----
Robert C. Martin (Uncle Bob)  | email: unclebob@objectmentor.com
Object Mentor Inc.            | blog:  www.butunclebob.com
The Agile Transition Experts  | web:   www.objectmentor.com
800-338-6716


"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo

On Sat, 16 Apr 2005 09:12:45 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:

>On Fri, 15 Apr 2005 19:35:16 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
> 
>
>Reports are formatted data structures.  The first step in writing both
>unit tests and acceptance tests is to create the data structures that
>represent the reports.  Then you can test that the data is being
>located properly, and the data structure if being filled in properly.
>
>Then you can test the formatting simply by loading canned data
>structures and ensuring those data structures are being formatted
>correctly into the report.

So to make sure I understand - it sounds like you're not recommending that I
test anything coming out of the JasperReports engine at all, just the data
structure feeding into it?

The problem with trying to test the data coming out of JasperReports would b
e
that there's more than one "right" way to produce, say, PDF output that will
look a certain way when printed.  I can't invent a pdf file in advance that
should be the same bytes JasperReports should generate.  If I don't test tha
t,
though, I'm not testing anything about the report design or how that interac
ts
with a set of data coming into it.

Steve Jorgensen wrote:

> So to make sure I understand - it sounds like you're not recommending that
I
> test anything coming out of the JasperReports engine at all, just the data
> structure feeding into it?

Do you plan to refactor the JasperReports engine?

> The problem with trying to test the data coming out of JasperReports would
be
> that there's more than one "right" way to produce, say, PDF output that
will
> look a certain way when printed.  I can't invent a pdf file in advance
that
> should be the same bytes JasperReports should generate.  If I don't test
that,
> though, I'm not testing anything about the report design or how that
interacts
> with a set of data coming into it.

If you needed to test that level, you can find a way to parse and spot-check
the PDF. Don't test everything.

If the output were XML, I could recommend XPath to query in and find
specific data, within the correct context. Adjust such queries to work
despite refactors and new features that move graphic commands around.

If no XPath is available for PDF, uh, keep going with tests on the data
until you encounter a bug, and the nature of the bug will tell you what,
down inside JasperReports, to test.

--
Phlip
http://industrialxp.org/community/b...tUserInterfaces

On Sun, 17 Apr 2005 03:08:11 GMT, "Phlip" <phlip_cpp@yahoo.com> wrote:

>Steve Jorgensen wrote:
> 
>I 
>
>Do you plan to refactor the JasperReports engine?
> 
>be 
>will 
>that 
>that, 
>interacts 
>
>If you needed to test that level, you can find a way to parse and spot-chec
k
>the PDF. Don't test everything.
>
>If the output were XML, I could recommend XPath to query in and find
>specific data, within the correct context. Adjust such queries to work
>despite refactors and new features that move graphic commands around.
>
>If no XPath is available for PDF, uh, keep going with tests on the data
>until you encounter a bug, and the nature of the bug will tell you what,
>down inside JasperReports, to test.

I think I see what you're saying and it sounds reasonable.  It's not really
JasperReports that I'm concerned about testing though, it's our report desig
ns
that will be created by me and my team to describe how data will be presente
d.

I like your idea about just testing specific things that should be true abou
t
the output.  I think I might be able to find a tool that can automate a text
search of a PDF.

Steve Jorgensen wrote:

> It's not
> really JasperReports that I'm concerned about testing though, it's
> our report designs that will be created by me and my team to describe
> how data will be presented.

Well, in my experience, JasperReports behavior isn't very stable between
versions - when you update to a new version, you might well need to update
your design definitions, too.

It's hard to test for specifics, though, because the look of a PDF is
composed of so many things.

What I'd probably do if I were concerned enough is using the Gold Master
technique: check the result manually *once* and after that automatically
compare the test result to the known good PDF file. Then, when the output of
JasperReports changes, you need to check manually again wether the result is
still acceptable and update the design and/or the Gold Master.

Hope this helps,

Ilja

On Sat, 16 Apr 2005 14:18:52 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>On Sat, 16 Apr 2005 09:12:45 -0500, Robert C. Martin
><unclebob@objectmentor.com> wrote:
> 
>
>So to make sure I understand - it sounds like you're not recommending that 
I
>test anything coming out of the JasperReports engine at all, just the data
>structure feeding into it?

You should certainly do the latter.  However, you should probably do
some form of the former.  i.e. you should test what comes out of
JasperReports.  However, I would test that using canned data
structures so that you have everything under complete control, and
that the number of tests that check formatting is very small.

Tying tests to formatting is an insidious form of coupling.  If you
test your business rules (i.e. the data that you are reporting) by
checking the output of the report, then you are coupling business
rules to format.  When the format changes, the tests must change.
This is *terrible*.  What you want is to decouple the tests of the
business rules from the tests of the formatting; and to severely limit
the number of tests that actually check formatting.  That way, you are
free to change the format of a report without having to change a vast
suite of tests.

>The problem with trying to test the data coming out of JasperReports would 
be
>that there's more than one "right" way to produce, say, PDF output that wil
l
>look a certain way when printed.  I can't invent a pdf file in advance that
>should be the same bytes JasperReports should generate.  If I don't test th
at,
>though, I'm not testing anything about the report design or how that intera
cts
>with a set of data coming into it.

Create a PDF from canned data using JasperReports.  Eyeball it for
correctness.  Then use it as the standard of comparison thereafter.
Write a test that generates the PDF from the same canned data and
compare the output file to the standard PDF.

This is not ideal, but at least it gives you a repeatable test that
will tell you if anyone has been fiddling with the report generation
code, or whether a new version of JasperReports broke something you
depended upon.

If you thoroughly trust JasperReports, then don't bother testing the
output at all.  Eyeball the results, and just test the incoming data
structures.

Does JasperReports generate formats other than PDF that might be more
amenable to testing?  e.g. text files, or HTML?  If so, you can run
your tests in those formats, and eyeball the PDF generation.  At least
this way you'd be testing some portion of Jasper Reports, even if you
weren't testing the entire pathway.


-----
Robert C. Martin (Uncle Bob)  | email: unclebob@objectmentor.com
Object Mentor Inc.            | blog:  www.butunclebob.com
The Agile Transition Experts  | web:   www.objectmentor.com
800-338-6716


"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo

On Sun, 17 Apr 2005 10:58:09 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:

>On Sat, 16 Apr 2005 14:18:52 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
> 
>
>You should certainly do the latter.  However, you should probably do
>some form of the former.  i.e. you should test what comes out of
>JasperReports.  However, I would test that using canned data
>structures so that you have everything under complete control, and
>that the number of tests that check formatting is very small.
>
>Tying tests to formatting is an insidious form of coupling.  If you
>test your business rules (i.e. the data that you are reporting) by
>checking the output of the report, then you are coupling business
>rules to format.  When the format changes, the tests must change.
>This is *terrible*.  What you want is to decouple the tests of the
>business rules from the tests of the formatting; and to severely limit
>the number of tests that actually check formatting.  That way, you are
>free to change the format of a report without having to change a vast
>suite of tests.

That all makes sense at a unit testing level, but not at an acceptance testi
ng
level.  The picture I'm getting from what you're saying, though, is that if 
I
have a few trivial unit tests for report formatting, and I have both
acceptance and unit test for report data set content as would feed into JR,
then my coverage would be pretty decent.  Would you agree with that statemen
t.
 
>
>Create a PDF from canned data using JasperReports.  Eyeball it for
>correctness.  Then use it as the standard of comparison thereafter.
>Write a test that generates the PDF from the same canned data and
>compare the output file to the standard PDF.
>
>This is not ideal, but at least it gives you a repeatable test that
>will tell you if anyone has been fiddling with the report generation
>code, or whether a new version of JasperReports broke something you
>depended upon.
>
>If you thoroughly trust JasperReports, then don't bother testing the
>output at all.  Eyeball the results, and just test the incoming data
>structures.

I agree that would be something that's much better than nothing, but I've do
ne
some research now, and I think I might be able to search for content in pdf
using something like JPedal.  JPedal claims to be able to return the x/y
coordinates where text is found, I could write the test to see if the values
are within a sanity check rectangle.

>Does JasperReports generate formats other than PDF that might be more
>amenable to testing?  e.g. text files, or HTML?  If so, you can run
>your tests in those formats, and eyeball the PDF generation.  At least
>this way you'd be testing some portion of Jasper Reports, even if you
>weren't testing the entire pathway.

JasperReports does generate output in those formats, but I'm concerned that
most kinds of error I can imagine might affect PDF only, and PDF will be the
one most used in production.  For instance, when generating an HTML table, i
t
would make sense for a report generator to include all the characters in an
element's data, but in formatted PDF, characters may be truncated to fit int
o
the coordinate range of the element.  If JPedal works as advertised, that
might be my solution.

On Sun, 17 Apr 2005 12:26:20 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:

>On Sun, 17 Apr 2005 10:58:09 -0500, Robert C. Martin
><unclebob@objectmentor.com> wrote:
> 
>
>That all makes sense at a unit testing level, but not at an acceptance test
ing
>level.

I disagree!  Indeed, I was thinking more of the acceptance testing
level than I was the unit testing level.  I would divide my acceptance
tests into those that test the business rules (the data contained in
the reports) and those that test the format.  I would write the
business rule acceptance tests against the data structures, not
against the finally formatted reports.  (I'd use something like
FitNesse).  I would write the format acceptance tests against reports
generated with canned data.

>The picture I'm getting from what you're saying, though, is that if I
>have a few trivial unit tests for report formatting, and I have both
>acceptance and unit test for report data set content as would feed into JR,
>then my coverage would be pretty decent.  Would you agree with that statement.[/col
or]

Yes.  You'll have to eyeball the generated reports of course, but that
is appropriate.


-----
Robert C. Martin (Uncle Bob)  | email: unclebob@objectmentor.com
Object Mentor Inc.            | blog:  www.butunclebob.com
The Agile Transition Experts  | web:   www.objectmentor.com
800-338-6716


"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo

On Sun, 17 Apr 2005 21:38:20 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:

>On Sun, 17 Apr 2005 12:26:20 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
> 
>
>I disagree!  Indeed, I was thinking more of the acceptance testing
>level than I was the unit testing level.  I would divide my acceptance
>tests into those that test the business rules (the data contained in
>the reports) and those that test the format.  I would write the
>business rule acceptance tests against the data structures, not
>against the finally formatted reports.  (I'd use something like
>FitNesse).  I would write the format acceptance tests against reports
>generated with canned data.
> 
>
>Yes.  You'll have to eyeball the generated reports of course, but that
>is appropriate.

You seem to be both agreeing and disagreeing with my post.  When I suggest
having only unit tests for report formatting, not acceptance tests, that wou
ld
mean there would be no acceptance test for formatting.

I understand what you are saying about having some acceptance tests just for
formatting, but that doesn't match what I thought I understood about
accpetance testing.  A user would never input data at the report data source
level, so would it be appropriate for an acceptance test do so?

Another thing I just realized about all of the above, is that none of those
strategies tests end-to-end.  Is it OK that I test making data sets from the
app, and test making reports from data, but never test the process across th
at
entire path?