Home > Archive > Software Testing > March 2007 > Regression test - how much to test?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Regression test - how much to test?
|
|
|
| We are having an interesting discussion at work regarding regression test. I
always thought that a true regression test would be a test of some new added
funtionality and then a re-test of ALL testcases used in the initial test of
the software.
Now my collegues argues that the regression test is only re-testing part of
the initial test (not all) eg. if the new added functionality is evidently
not affecting some parts of the existing functionality then those parts are
not tested in the regression test.
What is your experience and thoughts on regression test?
Thanks
Lars
| |
| James Bond 007 2007-02-12, 7:19 pm |
|
"Lars" <osborne(nospam)2210@gmail.com> wrote in message =
news:45d0e8dd$0$8228$edfadb0f@dread14.news.tele.dk...
> We are having an interesting discussion at work regarding regression =
test. I=20
> always thought that a true regression test would be a test of some new =
added=20
> funtionality and then a re-test of ALL testcases used in the initial =
test of=20
> the software.
>=20
> Now my collegues argues that the regression test is only re-testing =
part of=20
> the initial test (not all) eg. if the new added functionality is =
evidently=20
> not affecting some parts of the existing functionality then those =
parts are=20
> not tested in the regression test.
>=20
> What is your experience and thoughts on regression test?
>=20
> Thanks
> Lars=20
The regression test does not need to include every test case from the =
initial test of the software, but should cover all features that were =
touched in any way by the new release and a sampling of some of the =
other areas just to make sure nothing else was affected by the changes. =
Just because someone said that the new functionality "evidently does not =
affect some part of the existing functionality" does not mean it should =
not be tested. I'm sure he is an excellent engineer, but don't trust =
him. To paraphrase from the movies, "Assumption is the mother of all =
screw ups". :) In other words, don't assume anything. Better to test =
it to be sure.
| |
|
| On Feb 12, 5:23 pm, "Lars" <osborne(nospam)2...@gmail.com> wrote:
> We are having an interesting discussion at work regarding regression test. I
> always thought that a true regression test would be a test of some new added
> funtionality and then a re-test of ALL testcases used in the initial test of
> the software.
>
> Now my collegues argues that the regression test is only re-testing part of
> the initial test (not all) eg. if the new added functionality is evidently
> not affecting some parts of the existing functionality then those parts are
> not tested in the regression test.
>
> What is your experience and thoughts on regression test?
>
> Thanks
> Lars
Hi,
Basically regression testing is re-testing of the modified
functionality part. but you must test the other areas which would
impact because of these modifications in the functionality.Retesting
all testcases is most time consuming and resource taking job.Anyway
you have to do the system testing before you go for live. So
everything will be tested at that time.
In our project we have 350 automated test scripts and 4 different
users. So we generally do the regression testing on those modules
where the new patches have been applied considering other impacted
modules aswell. Then we do the Systemtesting before the product go for
live.
Thanks
| |
| dumitru.corobceanu@gmail.com 2007-02-13, 4:23 am |
| On Feb 13, 12:28 am, "James Bond 007" <0...@hmss.uk.gov> wrote:
> "Lars" <osborne(nospam)2...@gmail.com> wrote in messagenews:45d0e8dd$0$8228$edfadb0f@dre
ad14.news.tele.dk...
>
>
>
>
> The regression test does not need to include every test case from the initial test of the software, but should cover all features that were touched in any way by the new release and a sampling of some of the other areas just to make sure nothing else wa
s affected by the changes. Just because someone said that the new functionality "evidently does not affect some part of the existing functionality" does not mean it should not be tested. I'm sure he is an excellent engineer, but don't trust him. To par
aphrase from the movies, "Assumption is the mother of all screw ups". :) In other words, don't assume anything. Better to test it to be sure.
I agree with 007. But I'd like to add that when testing those "some
tests" of the "unaffected" modules of the system make sure you pay
some attention to them. Not test just for FAILED/PASSED.
A test can pass, but some strange behaviour could reveal a hidden
issue.
| |
| H. S. Lahman 2007-02-13, 7:17 pm |
| Responding to Lars...
> We are having an interesting discussion at work regarding regression test. I
> always thought that a true regression test would be a test of some new added
> funtionality and then a re-test of ALL testcases used in the initial test of
> the software.
>
> Now my collegues argues that the regression test is only re-testing part of
> the initial test (not all) eg. if the new added functionality is evidently
> not affecting some parts of the existing functionality then those parts are
> not tested in the regression test.
Your colleague is correct. The goal is to make sure that changes have
not broken the existing functionality. The changes should have their
own, new test suite to ensure they were done correctly. When that suite
passes, they are added to the regression suite in anticipation of the
next round of changes.
*************
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH
| |
| Llarry 2007-02-13, 7:17 pm |
| It is better to over-test than to under-test.
Yes, in an ideal development world, all changes would be tightly
compartmentalized, and there would be no chance of unanticipated
consequences. In this case, once the specific changes were tested, a
minimal regression suite concentrated on the affected areas would suffice.
I don't know about you, but I am not in that world.
On the other hand, in an ideal world, you would always have sufficient time
and resources to run a fully complete regression suite every time. I don't
live in that world, either...
In the end, you have to come up with an amount and distribution of
regression testing that is as effective and feasible as possible. Time to
bring on the ol' Risk Assessment. Obviously, the areas where the changes
were made is the most risky, followed by any "unaffected", but critical
areas. Finally, a random scattering of other tests as a sanity check.
--
--Llarry Amrose
This was the sort of time where people were prone to say
'God only knows'. However, it was generally conceded
that if there was anything God didn't know, this would most likely be it.
"Lars" <osborne(nospam)2210@gmail.com> wrote in message
news:45d0e8dd$0$8228$edfadb0f@dread14.news.tele.dk...
> We are having an interesting discussion at work regarding regression test.
I
> always thought that a true regression test would be a test of some new
added
> funtionality and then a re-test of ALL testcases used in the initial test
of
> the software.
>
> Now my collegues argues that the regression test is only re-testing part
of
> the initial test (not all) eg. if the new added functionality is evidently
> not affecting some parts of the existing functionality then those parts
are
> not tested in the regression test.
>
> What is your experience and thoughts on regression test?
>
> Thanks
> Lars
>
>
| |
| ramy.saad@saad.de 2007-02-14, 8:17 am |
| > Your colleague is correct. The goal is to make sure that changes have
> not broken the existing functionality. The changes should have their
> own, new test suite to ensure they were done correctly. When that suite
> passes, they are added to the regression suite in anticipation of the
> next round of changes.
I totally agree with the opinion of your collegue and H.S. Lahman.
Usually a regression test means to test the main functionalities of a
system which should cover the most important business cases of your
customer. And this should be discussed with the customer in the
forefield. And if the new added/modified modules are part of these
business cases they should be included in the regression test as well.
Otherwise they can be tested for the first test-delivery as a module
test and not as a regression test.
Cheers,
Ramy
| |
| Michael Bolton 2007-02-15, 7:17 pm |
| On Feb 12, 5:23 pm, "Lars" <osborne(nospam)2...@gmail.com> wrote:
> We are having an interesting discussion at work regarding regression test. I
> always thought that a true regression test would be a test of some new added
> funtionality and then a re-test of ALL testcases used in the initial test of
> the software.
>
> Now my collegues argues that the regression test is only re-testing part of
> the initial test (not all) eg. if the new added functionality is evidently
> not affecting some parts of the existing functionality then those parts are
> not tested in the regression test.
>
> What is your experience and thoughts on regression test?
I am aware of two common but different definitions of "regression
test".
1) A repetition of a test that we've run before.
2) A test to make sure that quality hasn't worsened.
Note that these definitions are orthogonal; you can repeat a test
without the intention of making sure that quality hasn't worsened, and
you can run varied tests to make sure that quality hasn't worsened.
Note that there is no such thing as a totally repeated test. It's
never the same test as it was before. The machine is not running the
same set of instructions, not responding to the same interrupts,
doesn't retain the same stack (on either an application level or a
system level), doesn't have the same values in uninitialized data.
The idea of repeating a test is all about repeating the things that we
understand to be important and that we believe we can control about a
test.
If you repeat a test, and you notice something this time that you
didn't notice last time, is it a repeated test or is it a new test?
---Michael B.
| |
|
|
>The regression test does not need to include every test case from the
>initial test of the software, but should
>cover all features that were touched in any way by the new release and a
>sampling of some of the other areas
>just to make sure nothing else was affected by the changes. Just because
>someone said that the new
>functionality "evidently does not affect some part of the existing
>functionality" does not mean it should not be
> tested. I'm sure he is an excellent engineer, but don't trust him. To
> paraphrase from the movies, "Assumption
>is the mother of all screw ups". :) In other words, don't assume anything.
>Better to test it to be sure.
Yes, then the problem for me would be to determine which parts not to test
... thanks for your insight
Lars
| |
|
| > I agree with 007. But I'd like to add that when testing those "some
> tests" of the "unaffected" modules of the system make sure you pay
> some attention to them. Not test just for FAILED/PASSED.
> A test can pass, but some strange behaviour could reveal a hidden
> issue.
Hmm .. would that not mean that the original tests was not sufficient in the
first place? And if so then I would assume that those additional tests would
then be added to the total test for completeness sakes
Lars
| |
|
| > Basically regression testing is re-testing of the modified
> functionality part. but you must test the other areas which would
> impact because of these modifications in the functionality.Retesting
> all testcases is most time consuming and resource taking job.Anyway
> you have to do the system testing before you go for live. So
> everything will be tested at that time.
I see .. but would'nt regression testing also be a situation where some new
functionality is added, with no change to the existing system and then
testing the existing system to see if something failed .. plus testing the
new part ?
> In our project we have 350 automated test scripts and 4 different
> users. So we generally do the regression testing on those modules
> where the new patches have been applied considering other impacted
> modules aswell. Then we do the Systemtesting before the product go for
> live.
Ah, thanks.
| |
|
| >> Now my collegues argues that the regression test is only re-testing part
>
> Your colleague is correct. The goal is to make sure that changes have not
> broken the existing functionality. The changes should have their own, new
> test suite to ensure they were done correctly. When that suite passes,
> they are added to the regression suite in anticipation of the next round
> of changes.
>
Thanks H.S. .. so you would also argue that it is possible to make sure the
added functionality does not change anything in the existing system by only
testing part of the initial tests for that system (plus of course tesing the
new functionality) ?
Lars
| |
|
| > I totally agree with the opinion of your collegue and H.S. Lahman.
> Usually a regression test means to test the main functionalities of a
> system which should cover the most important business cases of your
> customer. And this should be discussed with the customer in the
> forefield. And if the new added/modified modules are part of these
> business cases they should be included in the regression test as well.
> Otherwise they can be tested for the first test-delivery as a module
> test and not as a regression test.
Thanks Ramy. I see
| |
|
| > It is better to over-test than to under-test.
>
> Yes, in an ideal development world, all changes would be tightly
> compartmentalized, and there would be no chance of unanticipated
> consequences. In this case, once the specific changes were tested, a
> minimal regression suite concentrated on the affected areas would suffice.
>
> I don't know about you, but I am not in that world.
>
> On the other hand, in an ideal world, you would always have sufficient
> time
> and resources to run a fully complete regression suite every time. I
> don't
> live in that world, either...
>
> In the end, you have to come up with an amount and distribution of
> regression testing that is as effective and feasible as possible. Time to
> bring on the ol' Risk Assessment. Obviously, the areas where the changes
> were made is the most risky, followed by any "unaffected", but critical
> areas. Finally, a random scattering of other tests as a sanity check.
>
Wise words Llarry Amrose.. a compromise it seems. You are right I do not
live in either of those worlds you mention :-) Risk based testing.
| |
|
| > I am aware of two common but different definitions of "regression
> test".
>
> 1) A repetition of a test that we've run before.
> 2) A test to make sure that quality hasn't worsened.
>
> Note that these definitions are orthogonal; you can repeat a test
> without the intention of making sure that quality hasn't worsened, and
> you can run varied tests to make sure that quality hasn't worsened.
Intesting definitions Michael ...
>
> Note that there is no such thing as a totally repeated test. It's
> never the same test as it was before. The machine is not running the
> same set of instructions, not responding to the same interrupts,
> doesn't retain the same stack (on either an application level or a
> system level), doesn't have the same values in uninitialized data.
You have a point there.
> The idea of repeating a test is all about repeating the things that we
> understand to be important and that we believe we can control about a
> test.
>
> If you repeat a test, and you notice something this time that you
> didn't notice last time, is it a repeated test or is it a new test?
>
I would say a repeated test .. and then comparing the outcome with a
specified expected result (even though I notice someting else also). I mean
other changes might not be important to the system or how the system behaves
or is percieved, might not matter if the stack is different, if the values
changes in uniitialized data etc. as long at the application/module or
whatever we are testing is still doing what we expect.
The idea of repeating tests being about repeating thing we understand to be
important and that we believe we can control sounds right to me. Thanks
Lars
| |
| H. S. Lahman 2007-02-17, 7:11 pm |
| Responding to Lars...
>
>
> Thanks H.S. .. so you would also argue that it is possible to make sure the
> added functionality does not change anything in the existing system by only
> testing part of the initial tests for that system (plus of course tesing the
> new functionality) ?
Actually, no. There was a typo. B-( My first sentence should have read
"... is incorrect." A regression suite tests /all/ of the previous
functionality (adjusted for changes to the previous functionality that
would cause the original tests to fail). Sorry about the confusion.
There are sometimes exceptions that require using only a partial
regression suite, as I indicated. But they are dictated by feasibility
in the development environment, not fault coverage. When doing that one
would clearly be selective about what part of the suite was actually run
based on its "closeness" of the target functionality to the changes. But
the entire regression suite should be run at least before release.
*************
There is nothing wrong with me that could
not be cured by a capful of Drano.
H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH
| |
| Raghu V. Hudli 2007-02-20, 4:15 am |
| There have been quite some views and discussion on this topic. Here is my 2
cents.
I agree with your colleagues that you do not need to run all test cases. You
need
to worry about regression in functionality when there is a change to a
module or
new features are added. The straight-foward brute-force thing to do is to
run all
test cases of the previous release with the new test cases for the change or
new
feature. But some parts of the application are not affected. Why would you
want
to test he authentication module when something in reporting changes?
Well, how do you determine what to retest? Usually one has two imporant
mapping structures or matricies - a traceability matrix to do help you trace
from requirements to modules to code and test cases and a module dependency
matrix, which is a simple matrix and as the name implies, it captures the
dependency
betwen modules. Using this matrix you determine which modules need to be
retested when there is a change or a new feature added. Dependencies are
captured
based on locgical dependencies, workflow dependencies, structural
dependencies, etc.
When you know the modules to be tested, you can determine which test cases
to
run. The test cases for other modules need not be run at all. Organziations
that have
a good requirements engineering process will have these two matrices and
update
them when changes occur or new features are added.
Now, after you have selected your test cases, you can do something better.
You
can rank the test cases in the order of their ability to find "hard"
defects - some
measure of goodness of test cases. It always makes sense to try the hard
cases
first (for regresssion) testing. You will know there is a problem sooner
than later,
which should be one of the goals of regression testing.
Here are some references:
1. S. Elbaum, et.al, "Test Case Prioritization: A Family of Empirical
Studies",
IEEE Transactions on Software Engineering, v.28
n.2, p.159-182, February 2002
2. D. Binkley, "Reducing the Cost of Regression Testing by Semantics
Guided Test Case
Selection" Proc. Of the International Conference on
Software Maintenance, pp 251-260
This will work ONLY if you have module dependency and traceability matrices.
Otherwise
you are better off running all test cases, because you would not know which
modules are
affected and which are not.
Hope this helps.
Raghu
"Lars" <osborne(nospam)2210@gmail.com> wrote in message
news:45d0e8dd$0$8228$edfadb0f@dread14.news.tele.dk...
> We are having an interesting discussion at work regarding regression test.
> I always thought that a true regression test would be a test of some new
> added funtionality and then a re-test of ALL testcases used in the initial
> test of the software.
>
> Now my collegues argues that the regression test is only re-testing part
> of the initial test (not all) eg. if the new added functionality is
> evidently not affecting some parts of the existing functionality then
> those parts are not tested in the regression test.
>
> What is your experience and thoughts on regression test?
>
> Thanks
> Lars
>
--
Posted via a free Usenet account from http://www.teranews.com
| |
| Vladimir Trushkin 2007-03-20, 7:19 pm |
| On Feb 13, 12:23 am, "Lars" <osborne(nospam)2...@gmail.com> wrote:
> We are having an interesting discussion at work regarding regression test. I
> always thought that a true regression test would be a test of some new added
> funtionality and then a re-test of ALL testcases used in the initial test of
> the software.
>
> Now my collegues argues that the regression test is only re-testing part of
> the initial test (not all) eg. if the new added functionality is evidently
> not affecting some parts of the existing functionality then those parts are
> not tested in the regression test.
>
> What is your experience and thoughts on regression test?
>
A agree with the opinion of your colleagues. Just imagine an extreme
condition when the only change in the software is the static text in a
message box. Does it require re-running all the tests? No. The actual
selection of tests may also depend on whether or not they are
automated. If you are lucky and have an automated suit that covers
100% of functionality you may not even spend your time analyzing the
changes and what regression it may cause. Just run your automated
tests and see the results next morning :)
----
Best Wishes,
Vladimir
|
|
|
|
|