Home > Archive > Software Testing > November 2006 > Software Testing Tools
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Software Testing Tools
|
|
| Jacky 2006-11-22, 10:04 pm |
| Hi,
I'm in a software testing field in a new company, and I am their 1st
software tester.
I am looking around for some kind of automation tool (scripting) to
test the applications. I have stumbled onto AutoIT; it's free and kind
of useful.
What is your opinion of AutoIT? And how about the professional
automated testing tools like Mercury Test Director/IBM Rational Robot?
I am going to download them and try it.
| |
| Jacky 2006-11-22, 10:04 pm |
| What is the difference between WinRunner and QuickTest Professional?
Both of them are from Mercury.
| |
|
|
| Phlip 2006-11-22, 10:04 pm |
| Jacky wrote:
> I am looking around for some kind of automation tool (scripting) to
> test the applications. I have stumbled onto AutoIT; it's free and kind
> of useful.
The most efficient way to write tests is in the same language as the tested
code.
(The _least_ efficient way is thru a screen-scraping tool.)
Look up the unit test rig for your team's main language. JUnit for Java,
NUnit for .NET, UnitTest++ is for C++, etc.
--
Phlip
[url]http://www.greencheese.us/Z Land[/url] <-- NOT a blog!!!
| |
|
| Hi Phlip,
You're right. screen-scrapping tool (AutoIT) is really tedious and I
need to pray that the interface don't change; though they have advanced
abilities to read windows/events/system text.
We are using Visual C++, but I'm the 1st QA guy here, and I don't think
they will like me to touch their codes (white box testing). They are
used to manual testing. I have no experience in writing unit test.
hmmmm, maybe i should go for quicktest pro, let me download and try.
| |
|
| http://en.wikipedia.org/wiki/Unit_testing
Guess this is the reason:
[Limitations of unit testing]
Unit-testing will not catch every error in the program. By definition,
it only tests the functionality of the units themselves. Therefore, it
will not catch integration errors, performance problems or any other
system-wide issues. In addition, it may not be easy to anticipate all
special cases of input the program unit under study may receive in
reality. Unit testing is only effective if it is used in conjunction
with other software testing activities.
It is unrealistic to test all possible input combinations for any
non-trivial piece of software. A unit test can only show the presence
of errors; it cannot show the absence of errors. Though these two
limitations apply to any form of software test.
| |
|
| Jacky wrote:
> Unit-testing will not catch every error in the program.
Eating a high-fiber diet will not protect you from all forms of cancer.
(Just a few of them.) Hence, eat candy, dairy, and red meat every day.
I am about to introduce you to one of our industry's biggest and most
pernicious quandaries. Those programmers who might not want you to introduce
UnitTest++; they probably spend all their days debugging.
Debugging will not catch every error in the program, either. However, it
will also slow them down, and _cause_ bugs. Specifically "churn", where
fixing bugs causes more bugs.
If they wrote developer-tests, at the same time as they wrote the tested
code, they would debug much less often, and for much shorter cycles. They
would then be able to finish their features faster, and you would have a
much easier time adding QA tests into their test rig.
(I met a team the other day who were writing a website without a staging
server. They could test on their own desktop, hit a button, and push live
code into the running website. No intermediate QA website; no big database
full of known bugs. This practice is called "Daily Deployment".)
> By definition,
> it only tests the functionality of the units themselves. Therefore, it
> will not catch integration errors, performance problems or any other
> system-wide issues.
Then don't use that definition. Pure "unit tests" test units in isolation.
The test cases I'm advocating can test units, or packages, or the whole. The
tests should cover every feature, and each feature could have aspects in
many program layers.
> In addition, it may not be easy to anticipate all
> special cases of input the program unit under study may receive in
> reality. Unit testing is only effective if it is used in conjunction
> with other software testing activities.
WikiPedia has a typo. Unit testing is most effective when used in
conjunction with other _automated_ software testing techniques.
> It is unrealistic to test all possible input combinations for any
> non-trivial piece of software. A unit test can only show the presence
> of errors; it cannot show the absence of errors. Though these two
> limitations apply to any form of software test.
Right. And without automated tests, a developer's only recourse is endless
debugging, to inspect code as it changes. Each debugging session has those
problems too, amplified many times by simple human limitations. I cannot
remember every danged feature in my program, when I debug it. I can barely
remember all the features of the function I'm working on!
None of that WikiPedia entry represents a reason not to unit test. Just well
known reasons not to go with _only_ unit testing for life-critical software.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Vladimir Trushkin 2006-11-23, 7:11 pm |
|
On Nov 23, 8:03 am, "Jacky" <jacky.newsgr...@gmail.com> wrote:
> Hi Phlip,
>
> You're right. screen-scrapping tool (AutoIT) is really tedious and I
> need to pray that the interface don't change; though they have advanced
> abilities to read windows/events/system text.
>
Their weaknesses are their strength. Using UI-based tools is tedious
and less reliable but it allows to sensitize the whole path in the
system starting from topmost UI, almost same way as the end users do.
This makes testing more like a system test, less artificial, and closer
to the reality.
----
Best Wishes,
Vladimir
| |
| Vladimir Trushkin 2006-11-23, 7:11 pm |
|
On Nov 23, 8:03 am, "Jacky" <jacky.newsgr...@gmail.com> wrote:
> Hi Phlip,
>
> You're right. screen-scrapping tool (AutoIT) is really tedious and I
> need to pray that the interface don't change; though they have advanced
> abilities to read windows/events/system text.
>
> We are using Visual C++, but I'm the 1st QA guy here, and I don't think
> they will like me to touch their codes (white box testing). They are
> used to manual testing. I have no experience in writing unit test.
> hmmmm, maybe i should go for quicktest pro, let me download and try.
I would recommend using one of UI-level tools for the system testing.
Let C++ be used for covering code structure and interrelations in Unit
and Integration testing on dev side. You need to focus on function in
your testing. A good system testing is abstraction from the design of
code. You simply don't care (to the reasonable extent of course) how it
works. You only need to know what it should do and examine it in all
possible equivalent cases that, you think, are appropriate. (Having
defined system requirements is a plus)
----
Best Wishes,
Vladimir
| |
| Lilburne 2006-11-23, 7:11 pm |
| Phlip wrote:
> Jacky wrote:
>
>
>
>
> The most efficient way to write tests is in the same language as the tested
> code.
>
Not always.
Our algorithmic engine is multi-staged. Errors in one stage will affect
the final output geometry. So we have the ability to output the
intermediate stages as if they were in themselves the final geometry. A
code change resulted in a particular c++ property class not being added
to the intermediates.
Here is the test:
debug toolpaths all 1
import model /devdisk/dmk/test_data/pmdata/dunns.tri
edit block reset
create tool ;
edit tool ; diameter 10.0
edit tool ; type ballnosed
create toolpath fred raster
edit toolpath fred calculate
debug utassert entity toolpath
'raster_machiner:parallel_before_filteri
ng_1' exists
debug toolpath 'raster_machiner:parallel_before_filteri
ng_1' parameter
'dmkMcParUnits' exists
It would have taken more than 10 lines to test that in C++.
Question for the experts is it a unit test, application test, or what?
| |
|
| Lilburne wrote:
[color=darkred]
> Not always.
That's the sound-bite; the best place to start.
> Our algorithmic engine is multi-staged.
E! He played the "algorithmic complexity" card! Run away! Run away!
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Wolfram Jahn 2006-11-23, 7:11 pm |
| Jacky wrote:
> What is the difference between WinRunner and QuickTest Professional?
> Both of them are from Mercury.
>
Winrunner is the older tool, Quicktest the newer one.
Winrunner supports some (older) development languages/environments which
QTP does not.
QTP is somewhat easier to use and more intuitive. IMHO you will get the
first useable results faster with QTP.
Main reason to stay with Winrunner is usually the installed base of
automated tests in a company.
When you're starting fresh, and do not need the special old
Winrunner-only solutions, then I recommend QTP.
Both tools are well-integrated with Quality Center, a very complete test
management solution which covers all you will need from requirements
over test planning and execution upto defect tracking.
Of course all these are not for free, not even cheap...
Wolfram
| |
| Jim Cochrane 2006-11-23, 7:11 pm |
| On 2006-11-23, Lilburne <godzilla@nospam.net> wrote:
> Phlip wrote:
>
>
> Not always.
>
> Our algorithmic engine is multi-staged. Errors in one stage will affect
> the final output geometry. So we have the ability to output the
> intermediate stages as if they were in themselves the final geometry. A
> code change resulted in a particular c++ property class not being added
> to the intermediates.
>
> Here is the test:
>
> debug toolpaths all 1
> import model /devdisk/dmk/test_data/pmdata/dunns.tri
> edit block reset
> create tool ;
> edit tool ; diameter 10.0
> edit tool ; type ballnosed
> create toolpath fred raster
> edit toolpath fred calculate
> debug utassert entity toolpath
> 'raster_machiner:parallel_before_filteri
ng_1' exists
> debug toolpath 'raster_machiner:parallel_before_filteri
ng_1' parameter
> 'dmkMcParUnits' exists
>
> It would have taken more than 10 lines to test that in C++.
>
> Question for the experts is it a unit test, application test, or what?
Looks to me like a component test; and if a component can be called a
unit according to your organization's glossary, a unit test.
--
| |
| Jim Cochrane 2006-11-23, 7:11 pm |
| On 2006-11-23, Vladimir Trushkin <Vladimir.Trushkin@gmail.com> wrote:
>
>
> On Nov 23, 8:03 am, "Jacky" <jacky.newsgr...@gmail.com> wrote:
>
> Their weaknesses are their strength. Using UI-based tools is tedious
> and less reliable but it allows to sensitize the whole path in the
> system starting from topmost UI, almost same way as the end users do.
> This makes testing more like a system test, less artificial, and closer
> to the reality.
Are you sure you don't mean something like:
This would be a system test - less artificial, and closer to the reality.
[i.e., it's not like a system test, it is a system test; being picky,
perhaps, but precision often cuts down on misunderstandings.]
--
| |
| Jim Cochrane 2006-11-23, 7:11 pm |
| On 2006-11-23, Vladimir Trushkin <Vladimir.Trushkin@gmail.com> wrote:
>
>
> On Nov 23, 8:03 am, "Jacky" <jacky.newsgr...@gmail.com> wrote:
>
> I would recommend using one of UI-level tools for the system testing.
> Let C++ be used for covering code structure and interrelations in Unit
> and Integration testing on dev side. You need to focus on function in
> your testing. A good system testing is abstraction from the design of
> code. You simply don't care (to the reasonable extent of course) how it
> works. You only need to know what it should do and examine it in all
> possible equivalent cases that, you think, are appropriate. (Having
> defined system requirements is a plus)
Phlip appears to disagree with you [re. his earlier post in this
thread.] From what I've learned so far (being fairly new to
professional testing), I'd say you are correct - that system testing
using a driver outside the system (whether manual or automatic) is needed
in addition to unit/integration tests for a quality product, in most
cases.
--
| |
| Jim Cochrane 2006-11-23, 7:11 pm |
| On 2006-11-23, Phlip <phlipcpp@yahoo.com> wrote:
> Jacky wrote:
>
>
> The most efficient way to write tests is in the same language as the tested
> code.
By 'efficient' are you also implying 'complete' - i.e., that a thorough
set of tests in the same language as the app. is adequate? Or would
you say that it is wise to also include tests driven from an 'external'
source (i.e., not in the same language - whether scripted on the back-end,
using an API for the front/UI end, manual, or whatever)?
--
| |
| Phlip 2006-11-23, 10:04 pm |
| Jim Cochrane wrote:
> Phlip appears to disagree with you [re. his earlier post in this
> thread.] From what I've learned so far (being fairly new to
> professional testing), I'd say you are correct - that system testing
> using a driver outside the system (whether manual or automatic) is needed
> in addition to unit/integration tests for a quality product, in most
> cases.
The distinction is between "outside" and "far from". A test rig that drives
a program thru one of its API interfaces is ideal, and is outside the units.
A test rig that tests back-end behavior thru the GUI is "far from" the
tested code. That reintroduces much of the fragility that testing was
supposed to address.
Next, if your tests can't access your back-end thru an API of some kind,
then you don't have "Design for Testing", and that's generally very bad,
too.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Phlip 2006-11-23, 10:04 pm |
| Jim Cochrane wrote:
> By 'efficient' are you also implying 'complete' - i.e., that a thorough
> set of tests in the same language as the app. is adequate?
I mean the most efficient team, rapidly producing healthy features, is the
one with lots of Design for Testing, and lots of tests of various types, in
various layers.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Jacky 2006-11-23, 10:04 pm |
| Phlip wrote:
> (I met a team the other day who were writing a website without a staging
> server. They could test on their own desktop, hit a button, and push live
> code into the running website. No intermediate QA website; no big database
> full of known bugs. This practice is called "Daily Deployment".)
haha! you're right on this one. They have Daily Deployment almost every
other day. I can't catch up with them.
I was thinking of using some functional test tools (GUI Testing -
Automation) to help me. I tried QuickTest Pro 9, and did a quick search
on the internet - the price is about USD$5000 and above. I don't think
my company is going to go for it.....
Looking for cheaper alternatives from wikipedia:
[ Windows Test Tools ]
TestDrive-Gold, code-free testing of GUI and browser applications.
TestPartner an automated functional and regression testing tool from
Compuware Corporation.
WinRunner a commercial GUI testing tool.
Badboy is an inexpensive automated testing tool targeted specifically
at testing of web applications.
NUnitForms is an open source NUnit extension for unit and acceptance
testing of Windows Forms applications.
Ranorex is a free GUI test and automation Library for C++, Python and
for all .NET languages.
I know unit testing is good and etc. I'm new to Automatic testing
(manual device testing before), and I can't implement this process by
myself. Still self-studying.....
| |
| Phlip 2006-11-23, 10:04 pm |
| Jacky wrote:
> haha! you're right on this one. They have Daily Deployment almost
> every other day. I can't catch up with them.
Daily Deployment to live customers?
It's _typical_ to not be able to keep up with the _daily builds_ or
_continuous integrations_ from developers. That's why they should work in
cycles of one w, where the team specifies what features to do during that
w, and then writes tests and code during the w. Catching up to the
daily builds is irrelevant.
I mean the developers could hit a button, pass all their automated tests,
and update a live site, with no QA, bosses, marketects, or anyone reviewing
the product. And nobody running the product on a special server that
simulated a website environment. This team was doing nothing but changing
code, passing those tests, hitting that button, and pulling the next feature
from the feature list.
> I was thinking of using some functional test tools (GUI Testing -
> Automation) to help me. I tried QuickTest Pro 9, and did a quick search
> on the internet - the price is about USD$5000 and above. I don't think
> my company is going to go for it.....
That is throwing good money after bad.
For the Web, use Watir, and for C++ use UnitTest++. Don't read WikiPedia
again for this topic.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Jacky 2006-11-23, 10:04 pm |
| Phlip wrote:
> Daily Deployment to live customers?
Yupe, it's kinda throw the application out in the market, and when
error bugs are submitted, developers will rush out fixes. Patches will
be throw to the live update server, and users will be notified when
they use the application.
> It's _typical_ to not be able to keep up with the _daily builds_ or
> _continuous integrations_ from developers. That's why they should work in
> cycles of one w, where the team specifies what features to do during that
> w, and then writes tests and code during the w. Catching up to the
> daily builds is irrelevant.
I think here......build = release........they could do minor build (not
full build) and sort of test it themselves, and after it works, they
would sort of release it. hmmmmm,
> I mean the developers could hit a button, pass all their automated tests,
> and update a live site, with no QA, bosses, marketects, or anyone reviewing
> the product. And nobody running the product on a special server that
> simulated a website environment. This team was doing nothing but changing
> code, passing those tests, hitting that button, and pulling the next feature
> from the feature list.
it's a windows application here, not a web application. There's no
formal testing here before I arrived. And I'm expected to help change
it....hmmmm,
> For the Web, use Watir, and for C++ use UnitTest++. Don't read WikiPedia
> again for this topic.
haha! ok, I will try.
| |
| Phlip 2006-11-23, 10:04 pm |
| Jacky wrote:
>
> Yupe, it's kinda throw the application out in the market, and when
> error bugs are submitted, developers will rush out fixes. Patches will
> be throw to the live update server, and users will be notified when
> they use the application.
I really doubt we discuss the same thing. "Daily Deployment" means each and
every day, a new version goes online, and real customers use it. I don't
mean haphazard releases, or daily selling of new builds. I mean all
registered users get the new version, every day.
> it's a windows application here, not a web application. There's no
> formal testing here before I arrived. And I'm expected to help change
> it....hmmmm,
Okay. What's the amount of time between the users (via marketing) request a
new feature, and that new feature goes online?
> haha!
WikiPedia refers to the situation in Iraq as an "insurgency", not a "civil
war". The USA's own puppet government in Iraq refers to it as the latter,
but I guess they don't have editing privileges!
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Lilburne 2006-11-24, 8:05 am |
| Phlip wrote:
> Lilburne wrote:
>
>
>
>
>
>
> That's the sound-bite; the best place to start.
>
>
>
>
> E! He played the "algorithmic complexity" card! Run away! Run away!
>
The idea is to reuse the app as its own mock object and test harness.
You'll notice that the debug command can interrogate internal data
structures, I can add a new command in a few minutes to call any of the
inbuilt analysis methods as part of a test.
C++ tests are great for testing small bits of functionality, isolated
classes and clusters of classes. But as you move up the call stack (as
it were) you need more and more infrastructure support and there comes a
point when you've almost linked the application anyway.
Most of us see the application as a graphical testing, debugging, and
development tool for CADCAM functionality. Often commands developed to
aid testing, end up being packaged as user functionality.
| |
| S Perryman 2006-11-24, 7:04 pm |
| "Phlip" <phlipcpp@yahoo.com> wrote in message
news:1Ur9h.2917$yf7.1561@newssvr21.news.prodigy.net...
> Jim Cochrane wrote:
[color=darkred]
> I mean the most efficient team, rapidly producing healthy features, is the
> one with lots of Design for Testing, and lots of tests of various types,
> in various layers.
Not if they're using XP as the development method.
1. XP = developing to pass a set of tests.
2. Developing to pass a set of tests != Design for Testing ***
*** Design for Testability is where a component has been designed
*from the outset* with the *intent of enabling particular tests to be
performed* . For example, DFT to allow context-free tests to be
executed (for random/exhaustive testing) .
Regards,
Steven Perryman
| |
| Dmitry Motevich 2006-11-24, 7:04 pm |
| Wolfram Jahn wrote:
> Winrunner is the older tool, Quicktest the newer one.
>
> QTP is somewhat easier to use and more intuitive. IMHO you will get the
> first useable results faster with QTP.
>
> When you're starting fresh, and do not need the special old
> Winrunner-only solutions, then I recommend QTP.
Yes, I agree, QTP (QuickTest Professional) is a powerful and excellent
functional testing tool.
You can join the following google-group to get more info about QTP:
http://groups.google.com/group/MercuryQTP
--
Dmitry Motevich
| |
| Jim Cochrane 2006-11-25, 4:14 am |
| On 2006-11-24, Phlip <phlipcpp@yahoo.com> wrote:
> Jim Cochrane wrote:
>
>
> The distinction is between "outside" and "far from". A test rig that drives
> a program thru one of its API interfaces is ideal, and is outside the units.
>
> A test rig that tests back-end behavior thru the GUI is "far from" the
> tested code. That reintroduces much of the fragility that testing was
> supposed to address.
OK, then perhaps you don't disagree with me (or with the person I was
responding to - forgot his name). Let me state my view in relation to
what you're saying and let's see if there are any points you disagree
with or wish to elaborate on.
First, I think we're assuming a well-designed system (meaning that
its design allows for efficient testing as well as enhancement and
maintainability - it's not 'brittle'). For most applications (assuming
a client/server model), this well-designed system will have a large
majority of its functionality implemented on the server. (If there
were no need for UI optimization, virtually all functionality would be
implemented in the server.) And the server will be designed with a clean
API for the client such that the UI is an entirely separate component -
it can be (relatively) easily replaced by another UI. Therefore,
the main focus of testing will be on the server, since it is the most
important component and will have the most complexity. Obviously, the
most efficient and reliable way to test the server is not with the GUI
but via the API, by writing test programs or scripts that use the API and
test for correct results. Any failures that occur during testing will
indicate defects in the server. (In comparison, the cause of failures
that occur when testing with the UI may be in the UI, the server [or
both?] and are likely to be harder to track down as a result.)
With this scheme, a thorough test suite will result in a rock-solid,
reliable server. However, in most cases tests will still need to be
executed using the UI, for at least three reasons:
- To test the correctness and completeness of the UI itself.
- To test that the resulting behavior of the system (client and
server) makes sense from the user's point of view. (Example: the
behavior of the system for a particular path is obviously wrong when
visualized with the UI. The bug is in the requirements spec., but
this would not be uncovered with an automated test using the API.)
- For completeness in testing the whole system. Since it is not
possible to test all possible states with the API tests (or with any
kind of test), testing with the UI could uncover defects in the
server that did not show up with the API tests. Since, with most
systems, the main view of the system for the customers will be via
the UI, these defects are important to uncover.
I think there is one difference between the API testing discussed above
and what (if I remember and understood correctly) you said in an earlier
post - that the most efficient tests are in the same language as the
implementation of the system. I don't think that's true here, since
an API does not need to be designed to require that the same language
be used. The server could be designed such that it is written in C++,
for example, while the tests that call the API could be written in Ruby,
Perl, Python, etc.
> Next, if your tests can't access your back-end thru an API of some kind,
> then you don't have "Design for Testing", and that's generally very bad,
> too.
Yes - the goal is to release a high-quality system in an efficient
manner. Designing a system that can only be thoroughly tested via the
UI creates a very large handicap that works against this goal.
--
| |
| Jim Cochrane 2006-11-25, 4:14 am |
| On 2006-11-24, S Perryman <a@a.net> wrote:
> "Phlip" <phlipcpp@yahoo.com> wrote in message
> news:1Ur9h.2917$yf7.1561@newssvr21.news.prodigy.net...
>
>
>
>
> Not if they're using XP as the development method.
>
> 1. XP = developing to pass a set of tests.
>
> 2. Developing to pass a set of tests != Design for Testing ***
>
> *** Design for Testability is where a component has been designed
> *from the outset* with the *intent of enabling particular tests to be
> performed* . For example, DFT to allow context-free tests to be
> executed (for random/exhaustive testing) .
Developing using extreme programming methods does not preclude designing
for testability. (Does it?)
--
| |
| JTC ^..^ 2006-11-25, 8:05 am |
| Jacky wrote:
> Hi,
>
> I'm in a software testing field in a new company, and I am their 1st
> software tester.
>
> I am looking around for some kind of automation tool (scripting) to
> test the applications. I have stumbled onto AutoIT; it's free and kind
> of useful.
>
> What is your opinion of AutoIT? And how about the professional
> automated testing tools like Mercury Test Director/IBM Rational Robot?
>
> I am going to download them and try it.
>
Why automation? What are you trying to gain from this.
I asked a similar question and I was recommended "Lessons Learned in
Software Testing" by Cem Kaner,James Bach,Bret Pettichord. It has been
my saviour, but has also prevented me from making the jump into automation.
--
Regards
JTC ^..^
| |
|
| Jim Cochrane wrote:
> Developing using extreme programming methods does not preclude designing
> for testability. (Does it?)
Perryman is an anti-XP troll; feel free to Google for this effect, and to
disregard his posts.
XP forces to you design (and analyze) carefully for testing.
Some XP literature talks about deliberately neglecting the features you plan
to implement later, and only focus on the ones you are writing now. That's a
good mind-game, and it helps keep code clean. It helps a good design emerge.
But you will always need tests, and that's where you should plan ahead more.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
|
| Jim Cochrane wrote:
> First, I think we're assuming a well-designed system (meaning that
> its design allows for efficient testing as well as enhancement and
> maintainability - it's not 'brittle').
To get there, you should design and redesign frequently (refactor) while
running all the tests. This exposes brittle parts early, when they are
cheaper to fix.
> For most applications (assuming
> a client/server model), this well-designed system will have a large
> majority of its functionality implemented on the server.
Why?
(I know I know - because we can control the server's hardware and software
better, so we don't get screwed when a user picks the wrong client!)
> (If there
> were no need for UI optimization, virtually all functionality would be
> implemented in the server.) And the server will be designed with a clean
> API for the client such that the UI is an entirely separate component -
> it can be (relatively) easily replaced by another UI.
Right. And you don't get that so easily until you actually replace things
with another UI. A back-end could avoid using any GUI Toolkit identifiers,
and still couple with the UI's assumptions. If you plan ahead, and add extra
code to make the UI pluggable, but never replug it, then you might set
yourself up for a nasty surprise. But the fix is coming...
> Therefore,
> the main focus of testing will be on the server, since it is the most
> important component and will have the most complexity.
And because these tests form the "second UI", and force that code to
decouple more.
You can get this by making the jump into automation as early as possible -
by planning ahead for testing.
> Obviously, the
> most efficient and reliable way to test the server is not with the GUI
> but via the API, by writing test programs or scripts that use the API and
> test for correct results. Any failures that occur during testing will
> indicate defects in the server.
Right. And each time you (manually) discover a difference in how the GUI
drives the server, you _record_ that difference as more back-end tests.
> (In comparison, the cause of failures
> that occur when testing with the UI may be in the UI, the server [or
> both?] and are likely to be harder to track down as a result.)
Yep. The goal is not to write tests forever, the goal is to help development
go faster. So such tests as fail should help us rapidly diagnose _why_ they
failed.
> With this scheme, a thorough test suite will result in a rock-solid,
> reliable server. However, in most cases tests will still need to be
> executed using the UI, for at least three reasons:
>
> - To test the correctness and completeness of the UI itself.
Yep. But not even these tests need a capture/playback tool. You just said
that the back-end should get a rich test-rig, written in (generally) the
same language as the tested code. The GUI is not exempt.
An advanced GUI, using fragile technology (think Google Maps) needs rich
tests for exactly the same reason - so you can develop fast. And most tests
should run inside that GUI Layer, and using its same languages.
A GUI that just pumps lowly data entry forms doesn't need so elaborate a
test rig...
> - To test that the resulting behavior of the system (client and
> server) makes sense from the user's point of view. (Example: the
> behavior of the system for a particular path is obviously wrong when
> visualized with the UI. The bug is in the requirements spec., but
> this would not be uncovered with an automated test using the API.)
That's a perfect reason to test the GUI, and to (sometimes) test the
back-end thru the GUI. The GUI might call API functions in an order
different from how the back-end tests call them.
Record that difference as more back-end tests.
And that's a good reason to have a test rig available for your lowly data
entry form. Because you are not testing the controls themselves, you can
easily test them by driving them in code. Put data-entry text into them and
call the same function as the Submit button would call.
> - For completeness in testing the whole system. Since it is not
> possible to test all possible states with the API tests (or with any
> kind of test), testing with the UI could uncover defects in the
> server that did not show up with the API tests. Since, with most
> systems, the main view of the system for the customers will be via
> the UI, these defects are important to uncover.
That's the role of manual testing - to find situations to record as
automated tests.
You can't test all possible states, but you can rapidly stack up a lot of
tests that cross-test several components in their edge conditions. Consider
turning on Google Maps and zooming in on the North Pole. What will it do
with Longitude Zero? etc.
> I think there is one difference between the API testing discussed above
> and what (if I remember and understood correctly) you said in an earlier
> post - that the most efficient tests are in the same language as the
> implementation of the system. I don't think that's true here, since
> an API does not need to be designed to require that the same language
> be used. The server could be designed such that it is written in C++,
> for example, while the tests that call the API could be written in Ruby,
> Perl, Python, etc.
Think of the Food Pyramid, with seeds at the bottom, fruits and veggies in
the middle, and a tip of animal products.
The Testing Food Pyramid starts at the bottom with a huge layer of Developer
Tests, written by developers, in the same language as their tested code.
That must use the same language because it's _under_ that API. It's "white
box" tests that reach directly into private details.
The next-smaller tier are the Acceptance, Functional, Integration, and
Customer Tests. They face the API, and are written in whatever system is
convenient and maintainable. Many should still be written in the target
language. More importantly, they should be literate (such as via
http://fitnesse.org ), so non-technicals can review them.
The tip of the pyramid is all the hard tests that professional testers
should work on. The salty, sweet, greasy, fun tests. The Soak Tests that
make the server shake and smoke. The Combinatoric Tests that put together
lots of edge cases. The Metric Tests that run Lint on the code, that profile
the code and that look for memory leaks.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| S Perryman 2006-11-25, 7:05 pm |
| Phlip wrote:
> Jim Cochrane wrote:
[color=darkred]
> Perryman is an anti-XP troll; feel free to Google for this effect, and to
> disregard his posts.
Phlip is Usenets' XP muppet.
Feel free to Google comp.object, comp.software-eng etc for examples of
his muppetry.
Said muppetry usually involves claims that he cannot actually support
with real-world evidence.
Sorry that comp.software.testing appears to have this infestation.
But eradication is usually possible with a challenge to provide
supporting evidence for any claim made.
Steven Perryman
| |
| S Perryman 2006-11-25, 7:05 pm |
| Jim Cochrane wrote:
> On 2006-11-24, S Perryman <a@a.net> wrote:
[color=darkred]
[color=darkred]
[color=darkred]
[color=darkred]
[color=darkred]
[color=darkred]
[color=darkred]
[color=darkred]
> Developing using extreme programming methods does not preclude designing
> for testability. (Does it?)
1. The claim is :
X : "The most efficient team, rapidly producing healthy features"
Y: "is the one with lots of Design for Testing"
X does not imply Y.
If it does, it should be easy for a certain individual to provide
suitable examples to support the claim.
In fact, if you look at many of the XP examples put on Usenet, most have
actually quite poor testability (controllability, observability etc) .
Regards,
Steven Perryman
| |
|
| > Think of the Food Pyramid, with seeds at the bottom, fruits and veggies in
> the middle, and a tip of animal products.
It seems I parallel-evolved this idea:
Grig Gheorghiu wrote:
> Interesting you mention this. I first heard about the testing "food
> pyramid"
> from Jason Huggins, the author of Selenium, in December 2005. Here's a
> picture of his back-of-the-envelope drawing:
>
> http://vallista.idyll.org/~grig/testing-pyramid-big.jpg
>
> And here's how Titus Brown and I presented the concept at our PyCon '06
> tutorial on Agile Development and Testing:
>
> http://agile.idyll.org/wiki/HandoutNotes
>
> As for seeing this stuff in the wild, I confess it's a very rare event,
> but
> the discussions on this list certainly contribute to its popularity. I
> think
> the size of each tier is proportional to the time spent on writing/running
> the tests in that tier.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
|
| Hi Jacky,
Exactly what kind of application is it that you want to test? You
mentioned that you are having a lot of builds passed to you. What
problems are you encountering, and what have you tried so far to deal
with them? What problem are you looking for automated test tools to
solve?
What interfaces does your application present to you? What kind of
audience is it going out to and how frequently? Which programming
languages do you know, and how well do you know them?
There may be (and usually are) many ways to tackle these kinds of
problems. More information will help us to help you.
Jared
http://www.quinert.com/blog/
On Nov 25, 9:24 pm, "JTC ^..^" <davenos...@jazzthecat.co.uk> wrote:
> Jacky wrote:
>
>
>
>
>
> I asked a similar question and I was recommended "Lessons Learned in
> Software Testing" by Cem Kaner,James Bach,Bret Pettichord. It has been
> my saviour, but has also prevented me from making the jump into automation.
>
> --
> Regards
> JTC ^..^
| |
| salinimano@gmail.com 2006-11-27, 7:11 pm |
| Hi Jacky
I am still studying software testing.From what i have known about QTP,
you can view the scripts in the form of icons also rather than the
normal scripting(TSL) language.
salu
Jacky wrote:
> What is the difference between WinRunner and QuickTest Professional?
> Both of them are from Mercury.
| |
| info@e-valid.com 2006-11-29, 7:13 pm |
|
Jacky wrote:
> Hi,
>
> I'm in a software testing field in a new company, and I am their 1st
> software tester.
>
> I am looking around for some kind of automation tool (scripting) to
> test the applications.
Assuming that the application is web based -- as most new
developments are -- you may also wish to consider the eValid
solution.
eValid puts the entire testing activity inside a full-featured
web browser, is very easy to use and very reliable.
You can download your evaluation copy of eValid V6 from:
http://www.e-Valid.com/Products/Dow...tml?status=FORM
Full details about the eValid web analysis and testing suite
can be found at:
http://www.e-Valid.com
|
|
|
|
|