Home > Archive > Extreme Programming > April 2005 > Unit and Acceptance tests for report generation
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Unit and Acceptance tests for report generation
|
|
| Steve Jorgensen 2005-04-16, 3:57 am |
| I'm going to be doing some work with jasperreports soon, I'm wondering if
anyone else here has dealt with that or something similar in an Agile
development environment. How do you implement acceptance tests and unit tests
for the reporting stories?
Thanks,
Steve J.
| |
| Robert C. Martin 2005-04-16, 3:57 pm |
| On Fri, 15 Apr 2005 19:35:16 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:
>I'm going to be doing some work with jasperreports soon, I'm wondering if
>anyone else here has dealt with that or something similar in an Agile
>development environment. How do you implement acceptance tests and unit tests
>for the reporting stories?
Reports are formatted data structures. The first step in writing both
unit tests and acceptance tests is to create the data structures that
represent the reports. Then you can test that the data is being
located properly, and the data structure if being filled in properly.
Then you can test the formatting simply by loading canned data
structures and ensuring those data structures are being formatted
correctly into the report.
-----
Robert C. Martin (Uncle Bob) | email: unclebob@objectmentor.com
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716
"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo
| |
| Steve Jorgensen 2005-04-16, 8:56 pm |
| On Sat, 16 Apr 2005 09:12:45 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:
>On Fri, 15 Apr 2005 19:35:16 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>Reports are formatted data structures. The first step in writing both
>unit tests and acceptance tests is to create the data structures that
>represent the reports. Then you can test that the data is being
>located properly, and the data structure if being filled in properly.
>
>Then you can test the formatting simply by loading canned data
>structures and ensuring those data structures are being formatted
>correctly into the report.
So to make sure I understand - it sounds like you're not recommending that I
test anything coming out of the JasperReports engine at all, just the data
structure feeding into it?
The problem with trying to test the data coming out of JasperReports would be
that there's more than one "right" way to produce, say, PDF output that will
look a certain way when printed. I can't invent a pdf file in advance that
should be the same bytes JasperReports should generate. If I don't test that,
though, I'm not testing anything about the report design or how that interacts
with a set of data coming into it.
| |
|
| Steve Jorgensen wrote:
> So to make sure I understand - it sounds like you're not recommending that
I
> test anything coming out of the JasperReports engine at all, just the data
> structure feeding into it?
Do you plan to refactor the JasperReports engine?
> The problem with trying to test the data coming out of JasperReports would
be
> that there's more than one "right" way to produce, say, PDF output that
will
> look a certain way when printed. I can't invent a pdf file in advance
that
> should be the same bytes JasperReports should generate. If I don't test
that,
> though, I'm not testing anything about the report design or how that
interacts
> with a set of data coming into it.
If you needed to test that level, you can find a way to parse and spot-check
the PDF. Don't test everything.
If the output were XML, I could recommend XPath to query in and find
specific data, within the correct context. Adjust such queries to work
despite refactors and new features that move graphic commands around.
If no XPath is available for PDF, uh, keep going with tests on the data
until you encounter a bug, and the nature of the bug will tell you what,
down inside JasperReports, to test.
--
Phlip
http://industrialxp.org/community/b...tUserInterfaces
| |
| Steve Jorgensen 2005-04-17, 3:58 am |
| On Sun, 17 Apr 2005 03:08:11 GMT, "Phlip" <phlip_cpp@yahoo.com> wrote:
>Steve Jorgensen wrote:
>
>I
>
>Do you plan to refactor the JasperReports engine?
>
>be
>will
>that
>that,
>interacts
>
>If you needed to test that level, you can find a way to parse and spot-check
>the PDF. Don't test everything.
>
>If the output were XML, I could recommend XPath to query in and find
>specific data, within the correct context. Adjust such queries to work
>despite refactors and new features that move graphic commands around.
>
>If no XPath is available for PDF, uh, keep going with tests on the data
>until you encounter a bug, and the nature of the bug will tell you what,
>down inside JasperReports, to test.
I think I see what you're saying and it sounds reasonable. It's not really
JasperReports that I'm concerned about testing though, it's our report designs
that will be created by me and my team to describe how data will be presented.
I like your idea about just testing specific things that should be true about
the output. I think I might be able to find a tool that can automate a text
search of a PDF.
| |
| Ilja Preuß 2005-04-17, 8:57 am |
| Steve Jorgensen wrote:
> It's not
> really JasperReports that I'm concerned about testing though, it's
> our report designs that will be created by me and my team to describe
> how data will be presented.
Well, in my experience, JasperReports behavior isn't very stable between
versions - when you update to a new version, you might well need to update
your design definitions, too.
It's hard to test for specifics, though, because the look of a PDF is
composed of so many things.
What I'd probably do if I were concerned enough is using the Gold Master
technique: check the result manually *once* and after that automatically
compare the test result to the known good PDF file. Then, when the output of
JasperReports changes, you need to check manually again wether the result is
still acceptable and update the design and/or the Gold Master.
Hope this helps,
Ilja
| |
| Robert C. Martin 2005-04-17, 3:57 pm |
| On Sat, 16 Apr 2005 14:18:52 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:
>On Sat, 16 Apr 2005 09:12:45 -0500, Robert C. Martin
><unclebob@objectmentor.com> wrote:
>
>
>So to make sure I understand - it sounds like you're not recommending that I
>test anything coming out of the JasperReports engine at all, just the data
>structure feeding into it?
You should certainly do the latter. However, you should probably do
some form of the former. i.e. you should test what comes out of
JasperReports. However, I would test that using canned data
structures so that you have everything under complete control, and
that the number of tests that check formatting is very small.
Tying tests to formatting is an insidious form of coupling. If you
test your business rules (i.e. the data that you are reporting) by
checking the output of the report, then you are coupling business
rules to format. When the format changes, the tests must change.
This is *terrible*. What you want is to decouple the tests of the
business rules from the tests of the formatting; and to severely limit
the number of tests that actually check formatting. That way, you are
free to change the format of a report without having to change a vast
suite of tests.
>The problem with trying to test the data coming out of JasperReports would be
>that there's more than one "right" way to produce, say, PDF output that will
>look a certain way when printed. I can't invent a pdf file in advance that
>should be the same bytes JasperReports should generate. If I don't test that,
>though, I'm not testing anything about the report design or how that interacts
>with a set of data coming into it.
Create a PDF from canned data using JasperReports. Eyeball it for
correctness. Then use it as the standard of comparison thereafter.
Write a test that generates the PDF from the same canned data and
compare the output file to the standard PDF.
This is not ideal, but at least it gives you a repeatable test that
will tell you if anyone has been fiddling with the report generation
code, or whether a new version of JasperReports broke something you
depended upon.
If you thoroughly trust JasperReports, then don't bother testing the
output at all. Eyeball the results, and just test the incoming data
structures.
Does JasperReports generate formats other than PDF that might be more
amenable to testing? e.g. text files, or HTML? If so, you can run
your tests in those formats, and eyeball the PDF generation. At least
this way you'd be testing some portion of Jasper Reports, even if you
weren't testing the entire pathway.
-----
Robert C. Martin (Uncle Bob) | email: unclebob@objectmentor.com
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716
"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo
| |
| Steve Jorgensen 2005-04-17, 8:56 pm |
| On Sun, 17 Apr 2005 10:58:09 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:
>On Sat, 16 Apr 2005 14:18:52 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>You should certainly do the latter. However, you should probably do
>some form of the former. i.e. you should test what comes out of
>JasperReports. However, I would test that using canned data
>structures so that you have everything under complete control, and
>that the number of tests that check formatting is very small.
>
>Tying tests to formatting is an insidious form of coupling. If you
>test your business rules (i.e. the data that you are reporting) by
>checking the output of the report, then you are coupling business
>rules to format. When the format changes, the tests must change.
>This is *terrible*. What you want is to decouple the tests of the
>business rules from the tests of the formatting; and to severely limit
>the number of tests that actually check formatting. That way, you are
>free to change the format of a report without having to change a vast
>suite of tests.
That all makes sense at a unit testing level, but not at an acceptance testing
level. The picture I'm getting from what you're saying, though, is that if I
have a few trivial unit tests for report formatting, and I have both
acceptance and unit test for report data set content as would feed into JR,
then my coverage would be pretty decent. Would you agree with that statement.
>
>Create a PDF from canned data using JasperReports. Eyeball it for
>correctness. Then use it as the standard of comparison thereafter.
>Write a test that generates the PDF from the same canned data and
>compare the output file to the standard PDF.
>
>This is not ideal, but at least it gives you a repeatable test that
>will tell you if anyone has been fiddling with the report generation
>code, or whether a new version of JasperReports broke something you
>depended upon.
>
>If you thoroughly trust JasperReports, then don't bother testing the
>output at all. Eyeball the results, and just test the incoming data
>structures.
I agree that would be something that's much better than nothing, but I've done
some research now, and I think I might be able to search for content in pdf
using something like JPedal. JPedal claims to be able to return the x/y
coordinates where text is found, I could write the test to see if the values
are within a sanity check rectangle.
>Does JasperReports generate formats other than PDF that might be more
>amenable to testing? e.g. text files, or HTML? If so, you can run
>your tests in those formats, and eyeball the PDF generation. At least
>this way you'd be testing some portion of Jasper Reports, even if you
>weren't testing the entire pathway.
JasperReports does generate output in those formats, but I'm concerned that
most kinds of error I can imagine might affect PDF only, and PDF will be the
one most used in production. For instance, when generating an HTML table, it
would make sense for a report generator to include all the characters in an
element's data, but in formatted PDF, characters may be truncated to fit into
the coordinate range of the element. If JPedal works as advertised, that
might be my solution.
| |
| Robert C. Martin 2005-04-18, 3:56 am |
| On Sun, 17 Apr 2005 12:26:20 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:
>On Sun, 17 Apr 2005 10:58:09 -0500, Robert C. Martin
><unclebob@objectmentor.com> wrote:
>
>
>That all makes sense at a unit testing level, but not at an acceptance testing
>level.
I disagree! Indeed, I was thinking more of the acceptance testing
level than I was the unit testing level. I would divide my acceptance
tests into those that test the business rules (the data contained in
the reports) and those that test the format. I would write the
business rule acceptance tests against the data structures, not
against the finally formatted reports. (I'd use something like
FitNesse). I would write the format acceptance tests against reports
generated with canned data.
>The picture I'm getting from what you're saying, though, is that if I
>have a few trivial unit tests for report formatting, and I have both
>acceptance and unit test for report data set content as would feed into JR,
>then my coverage would be pretty decent. Would you agree with that statement.
Yes. You'll have to eyeball the generated reports of course, but that
is appropriate.
-----
Robert C. Martin (Uncle Bob) | email: unclebob@objectmentor.com
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716
"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo
| |
| Steve Jorgensen 2005-04-18, 3:56 am |
| On Sun, 17 Apr 2005 21:38:20 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:
>On Sun, 17 Apr 2005 12:26:20 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>I disagree! Indeed, I was thinking more of the acceptance testing
>level than I was the unit testing level. I would divide my acceptance
>tests into those that test the business rules (the data contained in
>the reports) and those that test the format. I would write the
>business rule acceptance tests against the data structures, not
>against the finally formatted reports. (I'd use something like
>FitNesse). I would write the format acceptance tests against reports
>generated with canned data.
>
>
>Yes. You'll have to eyeball the generated reports of course, but that
>is appropriate.
You seem to be both agreeing and disagreeing with my post. When I suggest
having only unit tests for report formatting, not acceptance tests, that would
mean there would be no acceptance test for formatting.
I understand what you are saying about having some acceptance tests just for
formatting, but that doesn't match what I thought I understood about
accpetance testing. A user would never input data at the report data source
level, so would it be appropriate for an acceptance test do so?
Another thing I just realized about all of the above, is that none of those
strategies tests end-to-end. Is it OK that I test making data sets from the
app, and test making reports from data, but never test the process across that
entire path?
| |
| Shane Mingins 2005-04-18, 3:56 am |
| "Steve Jorgensen" <nospam@nospam.nospam> wrote in message
news:rd8661hk9tmu7c2c2q1qbj4bdkg9heqovm@
4ax.com...
> I understand what you are saying about having some acceptance tests just
> for
> formatting, but that doesn't match what I thought I understood about
> accpetance testing. A user would never input data at the report data
> source
> level, so would it be appropriate for an acceptance test do so?
I am reading Testing Extreme Programming at the moment and have just
finished User Stories Applied. It would appear that a (customer) acceptance
test is *whatever* satisfies the customer that you have given them what they
want.
So if one test satifies that the data is correct and another satisfies that
the format is correct then is the customer satified?
>
> Another thing I just realized about all of the above, is that none of
> those
> strategies tests end-to-end.
OTOH .. Ron Jeffries says "end to end is futher than you think". Which
makes me think ...perhaps if the customer thinks you need to test end-to-end
you have two stories: the first with the split approach and estimates to
implement the tests; and the second with a full end to end test and
estimates ... they get to pick one.
Gotta catch a train .....
Shane
--
shanemingins@yahoo.com.duplication
remove duplication
--------------------------------------------------------------------------
"Our thinking was wrong - but our tests were not. Very interesting..." -
Ron Jeffries
| |
| Robert C. Martin 2005-04-20, 3:58 am |
| On Sun, 17 Apr 2005 20:02:48 -0700, Steve Jorgensen
<nospam@nospam.nospam> wrote:
>You seem to be both agreeing and disagreeing with my post. When I suggest
>having only unit tests for report formatting, not acceptance tests, that would
>mean there would be no acceptance test for formatting.
It is my policy never to admit that unit tests are sufficient, and
that acceptance tests can be manual. There are times when I am not
creative enough to imagine how to write an acceptance test, but that's
my failure, not a failure of the principle: "Everything has an
acceptance test and all acceptance tests are automated."
>I understand what you are saying about having some acceptance tests just for
>formatting, but that doesn't match what I thought I understood about
>accpetance testing. A user would never input data at the report data source
>level, so would it be appropriate for an acceptance test do so?
Sure, to test the formatting. Acceptance tests don't have to be
completely end to end. They can bracket modules or subsystems too.
>Another thing I just realized about all of the above, is that none of those
>strategies tests end-to-end. Is it OK that I test making data sets from the
>app, and test making reports from data, but never test the process across that
>entire path?
I often prefer end-to-end tests. However, I'm also a strong believer
in integration tests. Given the following modules:
A--->B--->C--->D
If I can test A->C, and B->D, I can be reasonably certain that A->D
works.
-----
Robert C. Martin (Uncle Bob) | email: unclebob@objectmentor.com
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716
"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo
| |
| Steve Jorgensen 2005-04-20, 3:58 am |
| On Tue, 19 Apr 2005 20:20:15 -0500, Robert C. Martin
<unclebob@objectmentor.com> wrote:
>On Sun, 17 Apr 2005 20:02:48 -0700, Steve Jorgensen
><nospam@nospam.nospam> wrote:
>
>
>It is my policy never to admit that unit tests are sufficient, and
>that acceptance tests can be manual. There are times when I am not
>creative enough to imagine how to write an acceptance test, but that's
>my failure, not a failure of the principle: "Everything has an
>acceptance test and all acceptance tests are automated."
>
>
>Sure, to test the formatting. Acceptance tests don't have to be
>completely end to end. They can bracket modules or subsystems too.
>
>
>I often prefer end-to-end tests. However, I'm also a strong believer
>in integration tests. Given the following modules:
>
> A--->B--->C--->D
>
>If I can test A->C, and B->D, I can be reasonably certain that A->D
>works.
OK, I think I follow. I very much appreciate your help.
|
|
|
|
|