Home > Archive > Extreme Programming > April 2006 > automate quality testing
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
automate quality testing
|
|
| Eric Lavigne 2006-04-25, 7:04 pm |
| I would like to add automated quality testing to my project, with
reports of test scores automatically generated after each commit,
preferably with historical graphs showing how scores have changed over
time.
>From http://en.wikipedia.org/wiki/Regression_testing
>Regression testing can not only be used for testing the correctness of a
>program, it is also often used to track the quality of its output. For instance
>in the design of a compiler, regression testing should track the code size,
>simulation time, and compilation time of the test suite.
The testing frameworks that I have seen so far focus only on pass/fail
tests. Continuous Integration tools frequently provide very nice
reporting regarding pass/fail, and occassionally on specific quality
metrics such as code size, testing time, and compilation time, but I
need to be able to define my own quality tests.
http://www.pragmaticautomation.com/...uto.cgi/Monitor
I am very new to automated testing (reading only so far), so hopefully
I have just overlooked some tool that does exactly what I need.
| |
|
| Eric Lavigne wrote:
> The testing frameworks that I have seen so far focus only on pass/fail
> tests.
Add a "high water mark" to test cases with variable qualities. Each time the
quality goes up, save the new high mark to a file. Each time the quality
goes below the mark, flunk the test.
I have a test case that, on Linux, shells to Valgrind and runs a sample of
our code through it, for example.
--
Phlip
[url]http://www.greencheese.org/Z Land[/url] <-- NOT a blog!!!
| |
| Eric Lavigne 2006-04-25, 7:04 pm |
|
>
> Add a "high water mark" to test cases with variable qualities. Each time the
> quality goes up, save the new high mark to a file. Each time the quality
> goes below the mark, flunk the test.
>
I'll take this route if I can't find any other options, but it seems
far from ideal. Let's take execution time as a simple example. I get
run time down to 10 seconds - great, change the high water mark to 12
seconds. A few days later, I add a new feature that improves accuracy
but adds a few seconds to execution. Time to reduce the testing
standards.
Besides, what about my nice graphs of performance improvements over
time that I can show to the boss :-) Actually, I'd like to look at them
myself too. Some things are absolute, can't let those unit tests fail,
but when it comes to tradeoffs in various quality metrics I just want
the computer to give enough feedback to help me make the call on my
own.
Valgrind looks like a nice tool. I'll give that a try next time I have
a C++ project.
Has anyone tried DamageControl? It is by the same developers as
CruiseControl (but not so Java focused), has been under development for
years, is seeing daily CVS commits... all these things make me suspect
that there's something there worth checking out. On the other hand,
there is no mailing list and the online docs are very sparse.
http://dev.buildpatterns.com/trac/wiki/DamageControl
I'll probably check this out myself in a couple ws when things slow
down at work, just install it and see if I can make it work. I'd
appreciate recommendations from people who have tried DamageControl or
other Continuous Integration tools.
| |
|
| Eric Lavigne wrote:
> Besides, what about my nice graphs of performance improvements over
> time that I can show to the boss :-) Actually, I'd like to look at them
> myself too. Some things are absolute, can't let those unit tests fail,
> but when it comes to tradeoffs in various quality metrics I just want
> the computer to give enough feedback to help me make the call on my
> own.
Then I don't understand the problem. It sounds like you could just write a
test case that records each test run's profile to an XML file. Then you
write an on-demand tool (such as a web page) that converts the file to a
graph. There's no tool support for all this because it's so easy, and is
orthogonal to the test runner's green bar.
--
Phlip
[url]http://www.greencheese.org/ZLand[/url] <-- NOT a blog!!!
| |
| Eric Lavigne 2006-04-25, 7:04 pm |
|
>
> Then I don't understand the problem. It sounds like you could just write a
> test case that records each test run's profile to an XML file. Then you
> write an on-demand tool (such as a web page) that converts the file to a
> graph.
That's what I'll do then (minus the XML, space separated values are
easier to parse). Thanks for the advice.
> There's no tool support for all this because it's so easy, and is
> orthogonal to the test runner's green bar.
>
It is easy to do just this. I was hoping for integration with other
testing tools, so that one report of project status would include all
of this information. That is not a big deal, though. I'll deal with
these issues separately for now, and when I have more experience with
the tools a solution should be easier to find. It may even turn out
that all the CI tools have hooks for this that I haven't noticed yet.
One step at a time. Right now my project doesn't even have a scriptable
build or automated testing, and version control is a set of backup
folders manually copied from the working directory each time I make a
change. I'm probably trying to think way too far ahead at this time,
but the managers make technical plans a year in advance (discussion
going on right now) and if I can't do the same then I have considerably
less input into the process.
Again, thanks for the input. It's good to know that a canned solution
isn't available, so I won't have any hesitation about wheel reinvention.
| |
|
| Eric Lavigne wrote:
> That's what I'll do then (minus the XML, space separated values are
> easier to parse). Thanks for the advice.
XML is easier to parse when you use XPath. Just write a little query inside,
almost as powerful as an SQL query.
Don't use space-separated values, because your job is not to write a new
database engine from scratch, yet your data must scale.
--
Phlip
[url]http://www.greencheese.org/ZLand[/url] <-- NOT a blog!!!
| |
| Eric Lavigne 2006-04-25, 7:04 pm |
|
>
> XML is easier to parse when you use XPath. Just write a little query inside,
> almost as powerful as an SQL query.
>
> Don't use space-separated values, because your job is not to write a new
> database engine from scratch, yet your data must scale.
>
I haven't used XML before, but I will look into XPath when I am ready
to write those tests. It doesn't sound like the time required to learn
would pay off for this particular use, but there's a good chance I will
find other uses for it.
| |
|
| Eric Lavigne wrote:
> I haven't used XML before, but I will look into XPath when I am ready
> to write those tests.
XPath is only like 2 lines of Ruby. Or 27 lines of Java...
--
Phlip
[url]http://www.greencheese.org/ZLand[/url] <-- NOT a blog!!!
| |
| Eric Lavigne 2006-04-25, 7:04 pm |
|
>
> XPath is only like 2 lines of Ruby. Or 27 lines of Java...
>
I assumed the XPath part would be roughly the same, with only small
syntactic changes to match the host language. Which of those conforms
to the XPath spec, Ruby or Java? Either way, I choose the two lines :-)
Then back to thousands of lines of Fortran 90 for the rest of the
project :-(
| |
|
| Eric Lavigne wrote:
> I assumed the XPath part would be roughly the same, with only small
> syntactic changes to match the host language. Which of those conforms
> to the XPath spec, Ruby or Java?
Both! You just say XPath.each(doc,
/my_scratch_database/my_test_run/@my_high_water_mark) or something.
The extra Java lines are all the excessive OO crap they force you to use
_around_ that statement.
(It's a topic for a different thread, but plenty of bake-offs between Java
and Ruby have shown the latter requires orders of magnitude fewer lines...)
--
Phlip
[url]http://www.greencheese.org/ZLand[/url] <-- NOT a blog!!!
| |
| Robert Atkins 2006-04-26, 4:00 am |
| On 2006-04-24, Eric Lavigne <lavigne.eric@gmail.com> wrote:
> Has anyone tried DamageControl? It is by the same developers as
Had a crack at it a while back and then it went into a re-write
from which it hasn't yet emerged. There is a mailing list but it
doesn't get much traffic aside from questions of the form "I just
downloaded the latest svn HEAD of DC and it won't work on my machine;
this is my output" followed by answers like "Upgrade to the latest
version of foo and frobnitz the bar option; if that doesn't work
wait for DC version blah" which never comes.
If you do get it running it's way sexier than Cruise though :-).
Cheers, Robert.
|
|
|
|
|