Home > Archive > Software Testing > October 2006 > Reason(s) to Automate....
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Reason(s) to Automate....
|
|
| Shrinik 2006-10-15, 4:07 am |
| Hi,
As a part of study and research about software test automation (by
holistic Approach), I am collecting information about various reasons,
objectives and expectations about Test Automation. Help me with your
views, views that you have come accross about -
"Why a test needs to be automated"?
Use following definitions or meanings for the words used in above
question:
Test : A means of evaluation of a feature or group of features of a
software product - a smallest possible unit of Test execution, often
against a specification (where documented) and/or a perceived notion of
"Correctness" or "Meeting some one's expectations"
Automated or Automation: Using a computer program to do one or more of
activities like setup, execution, evaulation, test clean up, result
reporting etc related to a "test" (as defined above)
Shrini
Shrinik.blogspot.com
| |
| James Bond 007 2006-10-15, 7:15 pm |
|
"Shrinik" <Shrinik@gmail.com> wrote in message =
news:1160896951.962550.159850@i3g2000cwc.googlegroups.com...
> Hi,
>=20
> As a part of study and research about software test automation (by
> holistic Approach), I am collecting information about various reasons,
> objectives and expectations about Test Automation. Help me with your
> views, views that you have come accross about -
>=20
> "Why a test needs to be automated"?
>=20
> Use following definitions or meanings for the words used in above
> question:
>=20
> Test : A means of evaluation of a feature or group of features of a
> software product - a smallest possible unit of Test execution, often
> against a specification (where documented) and/or a perceived notion =
of
> "Correctness" or "Meeting some one's expectations"
>=20
> Automated or Automation: Using a computer program to do one or more of
> activities like setup, execution, evaulation, test clean up, result
> reporting etc related to a "test" (as defined above)
>=20
> Shrini
> Shrinik.blogspot.com
Automated testing is best suited for regression. That is, when test =
cases need to be run to verify that existing features have not been =
broken by new features or functionality added to a system. Regression =
tests are typically mundane and boring because they do the same thing =
every time and are expected to get the same result. Perfect for =
automation.
Many companies (my employer included) are trying to automate functional =
testing of new features and applications. I think it is a bad idea.
| |
|
| Shrinik wrote:
> "Why a test needs to be automated"?
Because if you can't, you don't know your platform, your code, and/or your
requirements. Hence you shouldn't code or deliver.
--
Phlip
[url]http://www.greencheese.us/Z Land[/url] <-- NOT a blog!!!
| |
| James Bond 007 2006-10-15, 7:15 pm |
|
"Phlip" <phlipcpp@yahoo.com> wrote in message =
news:49tYg.15164$6S3.3883@newssvr25.news.prodigy.net...
> Shrinik wrote:
>=20
>=20
> Because if you can't, you don't know your platform, your code, and/or =
your=20
> requirements. Hence you shouldn't code or deliver.
Philip, I agree with almost everything you post here, but I can't this =
time. A blanket statement such as yours is nonsense, and you know that =
it is a bunch of bull. Testing of new features and new applications =
should not be automated until the systems have been thoroughly tested =
manually. After that, automated regression testing makes good sense if =
it is possible and cost effective. Many systems just don't lend =
themselves to automated testing at all. =20
For example, I worked for a small company that developed touch screen =
communication console systems for dispatchers. They are the industry =
leader for such systems which are used by almost all the airlines and =
railroad companies in North America, as well as many utilities and E911 =
systems. We tried to find test automation software to verify that the =
audio paths existed properly when the screen features were used, such as =
answering calls, putting them on hold, muting them, monitoring them, =
forwarding them, transferring them, patching them, etc. We had =
automated call generators, and we could find plenty of software that =
would perform the screen actions, but nothing to verify the audio paths. =
It made no sense to automate the testing if we could not verify the =
presence of the audio. We found one company that could help us automate =
part of the audio testing, but their application was very expensive and =
would have required additional hardware. It was not cost effective to =
pursue a partial solution at such a high price. It didn't mean that we =
didn't know our platform, our code, or our requirements.
Another example: Lotus Notes applications. About 10 years ago I worked =
on a Lotus Notes application and we could not find any automated test =
software that worked with Lotus Notes. Again, that doesn't mean that we =
didn't know our platform, our code, or our requirements. It simply =
meant that there were no tools available to automate the testing we were =
doing.
| |
|
| James Bond 007 wrote:
> Philip, I agree with almost everything you post here, but I can't this
> time. A blanket statement such as yours is nonsense, and you know that it
> is a bunch of bull. Testing of new features and new applications should
> not be automated until the systems have been thoroughly tested manually.
> After that, automated regression testing makes good sense if it is
> possible and cost effective. Many systems just don't lend themselves to
> automated testing at all.
Divide the word "test" into two concepts, "constraints" and "attacks".
You describe attacks. Specifically, you describe the difference between
constraints and attacks. You can only attack something that already exists.
I describe using the constraint process as part of analyzing the problem
space. Tests drive development.
Among other benefits, that leads to the exact framework that the attacks
will need.
> Another example: Lotus Notes applications. About 10 years ago I worked on
> a Lotus Notes application and we could not find any automated test
> software that worked with Lotus Notes. Again, that doesn't mean that we
> didn't know our platform, our code, or our requirements. It simply meant
> that there were no tools available to automate the testing we were doing.
It means Lotus Notes sucks. If you can't constrain or attack something, you
can't really program it.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| James Bond 007 2006-10-15, 7:15 pm |
|
"Phlip" <phlipcpp@yahoo.com> wrote in message =
news:01zYg.113$T_1.7@newssvr14.news.prodigy.com...
> James Bond 007 wrote:
>=20
this=20[color=darkred]
that it=20[color=darkred]
should=20[color=darkred]
manually.=20[color=darkred]
to=20[color=darkred]
>=20
> Divide the word "test" into two concepts, "constraints" and "attacks".
Tests are not constraints. They explore the capabilities of the =
software in an attempt to verify, validate, and find bugs. Kind of like =
Star Trek. A good tester boldly goes where no tester has gone before to =
s out new bugs. Poor testers make excuses for not finding bugs, such =
as the constraints on their test environment, their lack of time, tools, =
or resources.
> You describe attacks. Specifically, you describe the difference =
between=20
> constraints and attacks. You can only attack something that already =
exists.
And you can only test something that already exists.
> I describe using the constraint process as part of analyzing the =
problem=20
> space. Tests drive development.
You describe the hypothetical; I describe reality. Test driven =
development is fine, but it still needs to be developed first before it =
can be tested. =20
worked on=20[color=darkred]
we=20[color=darkred]
meant=20[color=darkred]
doing.[color=darkred]
>=20
> It means Lotus Notes sucks. If you can't constrain or attack =
something, you=20
> can't really program it.
I agree that Lotus Notes sucks. It was a poor choice for the system we =
deployed. But once management decided to go with it, we had to make it =
work. I don't think we could have said, "We can't deploy or release =
this system because we can't find any automated softare that works with =
it".
| |
| Vladimir Trushkin 2006-10-16, 7:03 pm |
| Shrinik wrote:
> Hi,
>
> As a part of study and research about software test automation (by
> holistic Approach), I am collecting information about various reasons,
> objectives and expectations about Test Automation. Help me with your
> views, views that you have come accross about -
>
> "Why a test needs to be automated"?
>
> Use following definitions or meanings for the words used in above
> question:
>
> Test : A means of evaluation of a feature or group of features of a
> software product - a smallest possible unit of Test execution, often
> against a specification (where documented) and/or a perceived notion of
> "Correctness" or "Meeting some one's expectations"
Where did you get such a definition of testing? IMO it is wrong at this
"and/or a perceived notion of "Correctness" or "Meeting some one's
expectations".
You cannot measure "correctness" especially if it's percieved. Every
project shall strive for having formal quality criteria. And the only
one reliable out there [that I know] is "conformance to requirements".
Now back to the topic...
> "Why a test needs to be automated"?
Because we do not want to run it by hand (or manually). Tests are much
easier to design and implement manually (in my world this is at least 2
times cheaper). Manual tests in contrary are more costly to execute. If
you are going to run a test just once this is not worthwhile of being
automated. But this is rarely a case because:
1. We need to re-run failed tests when a defect is fixed.
2. We need to re-run tests for regression testing.
3. We would like to re-run tests for final acceptance (regression)
testing on a release candidate before chipping the product.
4? Several additional runs to respond to risks (special suits ay be
composed to do additional tests in the area of high risk or after sever
low level design change).
This would mean at least 3 runs of tests. And this would means that
atuomation pays for itself, for the investment we have made at the
start of the project.
Also we would need automated tests to support Configuration testing.
Not saying that testing often helps to keep project under control and
find regression issues along the way, not all at once at the back end.
Depending on the complexity of the project automation may be less and
more costly. In some cases it requires 4-5 runs to pay back for
automated tests implementation. The cost depends on the efforts
required to automate one test and, as we learned, is greatly dependent
on:
1. Experience of the staff
2. Project structure
3. The way tests are designed
4. Tools
5. Availability of support from dev team (sometimes you may need
additional functions to access UI elements)
Automated tests are vulnerable to (at least those that are dependent on
UI):
1. Changes in UI
2. Changes in environment in which they are executed
So, automated tests shall be easy to adapt to changes in UI and be
ready to be run on different supported platform including different
localization setting, screen and color resolution.
Localization of automated tests is a whole diffrerent story. This is
not always as easy as running the translator on "string tables" or
connecting tests to resource maps. Sometimes making tests localized and
ready for different languages is a very complex task.
Hope this helps. Sorry for some inconsistency, I've got very short time
%)
----
Best Wishes,
Vladimir
| |
| Vladimir Trushkin 2006-10-16, 7:03 pm |
| Phlip wrote:
> James Bond 007 wrote:
>
>
> Divide the word "test" into two concepts, "constraints" and "attacks".
>
> You describe attacks. Specifically, you describe the difference between
> constraints and attacks. You can only attack something that already exists.
....as well as you can't test what does not exist. You can only design
and code your test. Then you just sit a wait until the function your
test is aimed to "attack" is there. You may of course run your test
1000 times to see that the function is not implemented but I would not
do that becuase it is silly. I would do it only in cases when I do not
know when it is going to available or if I am so sparse that I am
afraid to miss it.
----
Best Wishes,
Vladimir
| |
| Shrinik 2006-10-16, 7:03 pm |
| Philip/Jamebond07 ....
Please dont hijack the main topic Let us focus on the topic of
discussion. I am collecting various reasons,motivations and objectives
for test automation.
Here are few I (most of you hear) ...
1. This test does not require human to watch and check if test passed.
Hence would want computer to do it.
2. This test can not performed by a human hence would want computer to
do it.
3. This test needs to be repeated in all respects on 5 different
platforms - pretty boring and error prone. hence would like to
automate.
4. Making human to execute this test is waste of time and effort - I
would rather make machine to do this.
5. Sheer unwillingness - Dont want to do this mind less repitation -
hence automate
and so on ....
Please add your views and points ...
Shrini
Vladimir Trushkin wrote:
> Phlip wrote:
>
> ...as well as you can't test what does not exist. You can only design
> and code your test. Then you just sit a wait until the function your
> test is aimed to "attack" is there. You may of course run your test
> 1000 times to see that the function is not implemented but I would not
> do that becuase it is silly. I would do it only in cases when I do not
> know when it is going to available or if I am so sparse that I am
> afraid to miss it.
>
> ----
> Best Wishes,
> Vladimir
| |
| Shrinik 2006-10-16, 7:03 pm |
| >>>>>
Vladimir Wrote
Because we do not want to run it by hand (or manually).
Shrini: is this a willingness issue as in I dont want to execute this
test ? Is a test automated as no one wants execute that by hand?
Vladimir Trushkin wrote:
> Phlip wrote:
>
> ...as well as you can't test what does not exist. You can only design
> and code your test. Then you just sit a wait until the function your
> test is aimed to "attack" is there. You may of course run your test
> 1000 times to see that the function is not implemented but I would not
> do that becuase it is silly. I would do it only in cases when I do not
> know when it is going to available or if I am so sparse that I am
> afraid to miss it.
>
> ----
> Best Wishes,
> Vladimir
| |
|
| Shrinik wrote:
> Please dont hijack the main topic
Sorry. Tests should be easy to write. Hence...
> Here are few I (most of you hear) ...
> 1. This test does not require human to watch and check if test passed.
> Hence would want computer to do it.
Why should a human need to watch, if code could trap an intermediate value?
In my experience, the only place I haven't gone here is the last stage of
scripting an installer. Installing it onto target hardware. I would bet
someone has figured out how to automate that, too...
> 2. This test can not performed by a human hence would want computer to
> do it.
Now turn the question around - why can't a human do it? What's wrong with
this program's GUI?
Suggestion: The test requires split-second timing, which could occur
randomly in real life. So it must be automated.
> 3. This test needs to be repeated in all respects on 5 different
> platforms - pretty boring and error prone. hence would like to
> automate.
> 4. Making human to execute this test is waste of time and effort - I
> would rather make machine to do this.
> 5. Sheer unwillingness - Dont want to do this mind less repitation -
> hence automate
These are still the idea that human power manually testing must be bargained
against human power automating tests. Sometimes automating tests is easier
than manually running them, so the more you automate, the better you get at
it.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
|
| Someone wrote:
> Because we do not want to run it by hand (or manually).
If I'm a programmer, I want the shortest time possible between making the
smallest change, and getting confirmation I didn't break anything.
Anything manual testers could do, auto tests should do enough of it that the
test batch is ... automated.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| James Bond 007 2006-10-16, 7:03 pm |
| Shrini,
I think you've got the topic covered. We automate when
1) a human can't (physically or mentally) do it,
2) if we don't want a human to do it (maybe it's dangerous), or
3) if it needs to be repeated so many times that it is more cost
effective and less error prone for a computer to do it.
An example of the first case might be when we want to pump 1 million
random numbers into an application or if we need to stress test 1000
simultaneous hits on a web site. An example of the second case might
be some kind of electrical or chemical test that could expose a human
to electric shock or radioactivity. Most automated testing falls into
the third case. Regression testing is a good candidate for automation
because it needs to be repeated many times over months or years.
James Bond 007
Shrinik wrote:
> Philip/Jamebond07 ....
>
> Please dont hijack the main topic Let us focus on the topic of
> discussion. I am collecting various reasons,motivations and objectives
> for test automation.
>
> Here are few I (most of you hear) ...
> 1. This test does not require human to watch and check if test passed.
> Hence would want computer to do it.
> 2. This test can not performed by a human hence would want computer to
> do it.
> 3. This test needs to be repeated in all respects on 5 different
> platforms - pretty boring and error prone. hence would like to
> automate.
> 4. Making human to execute this test is waste of time and effort - I
> would rather make machine to do this.
> 5. Sheer unwillingness - Dont want to do this mind less repitation -
> hence automate
> and so on ....
>
> Please add your views and points ...
>
> Shrini
>
| |
|
| James Bond 007 wrote:
> I think you've got the topic covered. We automate when
The reason I keep derailing this topic is because I'm in the camp that
believes automation _attempts_ should be the default, and that failures to
automate are a form of analysis and correction. Therefor, don't look at the
test criteria and try to find an excuse to automate. Try to automate, and
learn from the attempt.
> 1) a human can't (physically or mentally) do it,
> 2) if we don't want a human to do it (maybe it's dangerous), or
> 3) if it needs to be repeated so many times that it is more cost
> effective and less error prone for a computer to do it.
3 is always needed, to help developers go faster.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Vladimir Trushkin 2006-10-17, 4:07 am |
| Shrinik wrote:
> Vladimir Wrote
> Because we do not want to run it by hand (or manually).
>
> Shrini: is this a willingness issue as in I dont want to execute this
> test ? Is a test automated as no one wants execute that by hand?
Sometimes you just can't test manually, like in case of concurrent or
dynamic load testing ... But the universal measure is money. We
automated if we think that this is more effective to invest in
automatomation because we will have benefit (ROI) in the following
execution.
----
Best Wishes,
Vladimir
|
|
|
|
|