| sirgilligan 2004-03-26, 11:51 pm |
| I just finished a new document for Maverick QA. I posted it to a test
group but since this is like a revolution or maybe evolution to XP I
want to share it.
Geoff
__________________________
Maverick Quality Assurance
for
Agile Development
By Geoffrey Slinker
Abstract
Agile development has made test driven development and unit tests come
from the arena of good practices to mainstream development. The
aggregation of customer driven requirements, customer driven order of
feature development, writing unit tests before code, continuous
builds, and short iterations work together to streamline and focus
development on the real tasks. Agile development works bests for small
teams and projects that are not large. The benefits of Agile
development are applied to large teams and large projects but
concessions are made to deal with issues such as the need for
increased communication and the problems with a large set of unit
tests that can take a significant amount of time to run. With large
teams on large projects the continuous build becomes less and less
continuous and becomes "build as often as possible". Agile development
has not been clearly understood in the area of integration testing.
Maverick Quality Assurance (MQA) considers the goals of quality
assurance (QA) upfront and proposes a method for full QA in an Agile
development process.
Goals of QA
QA processes provide balance between quality of product, cost of
product, features provided by the product, and release date of the
product. QA is fundamentally an aspect of what some refer to as "Big
upfront design", but is more accurately described as a predictive
process. Because it is predictive, it is not necessarily iterative but
more likely will be based on well defined phases such as a Waterfall
Methodology.
A very accurate definition of QA was found on a NASA web site.
Software Quality Assurance (SQA) is defined as a planned and
systematic approach to the evaluation of the quality of and adherence
to software product standards, processes, and procedures. SQA
includes the process of assuring that standards and procedures are
established and are followed throughout the software acquisition life
cycle. Compliance with agreed-upon standards and procedures is
evaluated through process monitoring, product evaluation, and audits.
Software development and control processes should include quality
assurance approval points, where an SQA evaluation of the product may
be done in relation to the applicable standards.
As with any Maverick approach the first thing we do with a methodology
or a process is to question why it exists and to drill down to the
reasons the method exists, the problems the method addresses, and the
goals or results the method was to supply. It has been said that
methodologies only represent the solutions to problems that someone
has faced. Hidden in the methodology are the complaints that the
process addresses. These complaints are at the core of the issues and
need to be clearly stated so that alternative solutions can be
correctly identified.
From investigation, deduction, education, and reason MQA identifies
the goals of QA as follows:
In the interest of the customer deliver a valid product and verify
that the product was built as required. In interest of the software
producer and the customer deliver this product within budget and on
time.
A valid product is defined as a product that performs the function for
which it was intended. This implies that if it is performing intended
function the software is error free.
A verified product is a product that was built in the way that was
proposed. This means the product is made of the components that were
agreed would compose the product. This includes the understanding that
it was developed in the programming language that was specified, uses
the data stores specified, or what ever pieces that were selected to
build the product.
QA methodologies try to insure the goals are met through a very
complete and sophisticated process of deliverables. The task of QA is
to help safeguard the interests of the customer and the company
developing the software. Because QA serves two masters it becomes a
difficult task to decide which party is represented at certain times.
If the customer is represented exclusively and the company is worked
until it is out of money then what has been accomplished? The goal of
delivering the product has been missed. This is just the tip if the
iceberg of all of the difficulties and issues facing QA teams.
MQA is an Agile QA process. Traditional QA processes set up standards
and compare the adherence to these standards through processes such as
audits. MQA sets up the standards that are the simplest to meet the
goals of QA and through iterative modifications to the QA process
emerges a process that addresses the issues that are faced by the
particular development task. MQA doesn't believe that software
development is so dangerous that the managing processes, procedures,
and the time to complete their requirements should be larger than the
effort to actually perform the development of the software.
In order to deliver on the goals of QA the QA department has to be an
impartial judge at times. The realization that people write software
and therefore software becomes subject to all of the human frailties
can not be ignored. Pressures of lay-offs can cause people to hide
issues that are considered as bad or of poor performance. Individuals
s ing advancement or recognition can put themselves before the team.
Management can hide the real state of the development process hoping
against hope to catch up lost time, effectively hoping that the
completion of work left to be done is not constrained by time itself.
MQA proposes that the computer is the ultimate impartial judge when it
comes to verifying and validating software. People are needed to
decide on schedules and features and such but once the features are
decided upon the computer can validate the feature set completely
using Unit tests, Components tests, and an automated test process with
a continuous build process. The computer can not be influenced by
gain, honor, or recognition. The computer will not be affected by the
drive to work, the pressures of work, or any of those things that
affect people and their performance.
NASA follows a very formal and rigorous process for QA. Given that
fact, let us remember the Mars Orbiter.
WASHINGTON (AP) -- For nine months, the Mars Climate Orbiter was
speeding through space and speaking to NASA in metric. But the
engineers on the ground were replying in non-metric English.
It was a mathematical mismatch that was not caught until after the
$125-million spacecraft, a key part of NASA's Mars exploration
program, was sent crashing too low and too fast into the Martian
atmosphere. The craft has not been heard from since.
"We were on the wrong trajectory and our system of checks and
balances did not allow us to recognize that," Edward Stone, director
of the Jet Propulsion Laboratory, said Wednesday.
The NASA centre in California was in charge of the Mars mission.
Noel Henners of Lockheed Martin Astronautics, the prime contractor
for the Mars craft, said at a news conference it was up to his
company's engineers to assure the metric systems used in one computer
program were compatible with the English system used in another
program. The simple conversion check was not done, he said.
"It was overlooked," Henners said.
The mathematical mismatch was identified within days after the
spacecraft was lost and the report released Wednesday confirmed the
problem.
The Mars Climate Orbiter was launched last Dec. 11 and spent nine
months coasting toward Mars.
Art Stephenson, director of the Marshall Spaceflight Center and head
of a NASA investigation team, said the spacecraft was not symmetrical
and pressure from the sun caused it to slowly twist or roll as it sped
along.
On board gyroscopes partially controlled the motion but eventually
rocket-firings were needed to stabilize the craft, he said. This
happened 12-14 times a w over the nine-month voyage.
Engineers on the ground calculated the size of the rocket-firing
using feet-per-second of thrust, a value based on the English measure
of feet and inches.
However, the spacecraft computer interpreted the instructions in
Newtons-per-second, a metric measure of thrust. The difference is 1.3
metres a second.
"Each time there was a burn (rocket-firing) the error built up," said
Stephenson.
As the spacecraft approached its rendezvous with Mars and the
engineers prepared for a final rocket-firing, there were indications
of something seriously wrong with the navigation but no corrective
action was taken, Stephenson said.
When the Mars Climate Orbiter did fire its rockets, the craft went
too low into the planet's atmosphere, instead of into a safe orbit.
Communication signals stopped when the craft passed behind Mars and
have not been heard since.
"We entered the Mars atmosphere at a much lower altitude (than
planned)," said Ed Weiler, NASA's chief scientist.
"It (the spacecraft) either burned up in the Martian atmosphere or
sped out (into space). We're not sure which happened."
Stephenson said the problem was not with the spacecraft but with the
engineers and the systems used to direct it.
"The spacecraft did everything we asked of it," said Stephenson.
He said the mathematical mismatch was "a little thing" that could
have been easily fixed if it had been detected.
"Sometimes the little things can come back and really make a
difference," he said.
Domain Knowledge
Domain knowledge is the knowledge and understanding of a specific area
or domain. Communication of domain knowledge is done in many and
various ways. These ways include verbal communication in a language
that those involved understand, use cases or stories, modeling
languages, documentation, and code.
MQA recognizes that communication is not perfect and if communication
can be avoided it should be. This does not mean to work in isolation.
It does mean to allow those with domain knowledge take responsibility
for any items that fall into that domain. Specifically MQA is
concerned with the full range of testing of software and that these
tests are done by those that know how to do it best instead of
external test teams.
Agile development has been adopted in many areas and as a part of that
unit testing has also been adopted. The idea that engineers can not
test their own code has been a myth that is hard to trace. I am sure
it addressed a problem and I will guess it was a problem where the
engineer's performance was measured by lines of code produced and not
by the quality of the code. Therefore when the engineer was asked to
test the code he did a poor job because he needed to get back to what
really matters, and what really matters is what management measures
and brings up in performance reviews.
The developers have the correct domain knowledge to write the unit
tests. They also have the correct knowledge to write the component
tests. External teams introduced to write component tests require the
transfer of domain knowledge. If the developer with the domain
knowledge doesn't give proper attention to the test engineers then it
will require a new method of transferring the domain knowledge and
that method is typically documentation. Now the developers are writing
documentation, and the production of this documentation is rarely
included as "real" work for the developer so the developer is anxious
to get back to what he is measured on which is writing code.
Contexts and Unification Methods
Large software is divided into contexts and these contexts are unified
by various means. Domain Driven Design by Eric Evans goes into great
detail on this subject and does an excellent job in describing the
issues worthy of consideration. Since a large software product is
usually divided into contexts for reasons such as abstraction and
resource allocation there will be boundaries, relationships, and
translation layers that must be dealt with.
Those that have the correct knowledge of these contexts and any
mapping layers or anticorruption layers should be the ones that write
the component tests. If an external team is assigned to write the
component tests then the problem of transferring domain knowledge
becomes the problem.
There are tell-tale signs of problems with external test teams failing
to work well in the varied contexts of the system. Statements like,
"That is an invalid tests", or "It took two ws for testing to write
that test when I (the developer) could have done it in a day", or "I
(test engineer) can't get my tests done because I can't figure out
what the system should be doing", or "I can not validate the data
object because there is no method or criteria that defines a valid or
invalid object."
Because of these problems the traditional QA process requires
increased transfer of domain knowledge. It is the most common solution
that addresses the problem without reorganizing teams and changes
roles.
Definition of Tests
The definition of test the types of tests can be found at
http://www.faqs.org/faqs/software-e...ection-14.html.
The following definitions are from a posting by Boris Beizer on
the topic of "integration testing" in the c.s.t. newsgroup.
The definitions of integration tests are after Leung and White.
Note that the definitions of unit, component, integration, and
integration testing are recursive:
Unit. The smallest compilable component. A unit typically is the
work of one programmer (At least in principle). As defined, it
does
not include any called sub-components (for procedural languages)
or
communicating components in general.
Unit Testing: in unit testing called components (or communicating
components) are replaced with stubs, simulators, or trusted
components. Calling components are replaced with drivers or
trusted
super-components. The unit is tested in isolation.
component: a unit is a component. The integration of one or more
components is a component.
Note: The reason for "one or more" as contrasted to "Two or
more" is to allow for components that call themselves
recursively.
component testing: the same as unit testing except that all stubs
and simulators are replaced with the real thing.
Two components (actually one or more) are said to be integrated
when:
a. They have been compiled, linked, and loaded together.
b. They have successfully passed the integration tests at
the
interface between them.
Thus, components A and B are integrated to create a new, larger,
component (A,B). Note that this does not conflict with the idea
of
incremental integration -- it just means that A is a big component
and B, the component added, is a small one.
Integration testing: carrying out integration tests.
Integration tests (After Leung and White) for procedural
languages.
This is easily generalized for OO languages by using the
equivalent
constructs for message passing. In the following, the word "call"
is to be understood in the most general sense of a data flow and
is
not restricted to just formal subroutine calls and returns -- for
example, passage of data through global data structures and/or the
use of pointers.
Let A and B be two components in which A calls B.
Let Ta be the component level tests of A
Let Tb be the component level tests of B
Tab The tests in A's suite that cause A to call B.
Tbsa The tests in B's suite for which it is possible to sensitize
A
-- the inputs are to A, not B.
Tbsa + Tab == the integration test suite (+ = union).
Note: Sensitize is a technical term. It means inputs that
will
cause a routine to go down a specified path. The inputs are
to
A. Not every input to A will cause A to traverse a path in
which B is called. Tbsa is the set of tests which do cause A
to
follow a path in which B is called. The outcome of the test
of
B may or may not be affected.
There have been variations on these definitions, but the key point
is
that it is pretty darn formal and there's a goodly hunk of testing
theory, especially as concerns integration testing, OO testing,
and
regression testing, based on them.
As to the difference between integration testing and system
testing.
System testing specifically goes after behaviors and bugs that are
properties of the entire system as distinct from properties
attributable to components (unless, of course, the component in
question is the entire system). Examples of system testing
issues:
resource loss bugs, throughput bugs, performance, security,
recovery,
transaction synchronization bugs (often misnamed "timing bugs").
These definitions of unit, component, and integration testing will be
used as the MQA definition of tests.
QA Roles in MQA
The roles of individuals in MQA are aligned with domain knowledge.
MQA requires that the former role of Manager/Director of Test
Engineering (the person that manages the traditional external test
team) become critical and essential to the software development
process in a continuous manner. In traditional QA the software
process' goal was to make the entire process predictable with distinct
deliverables and phases. The Director of Test Engineering would make
sure that all requirements were understood upfront and that a test
plan and criteria for acceptance was developed, reviewed, and accepted
very early in the software life cycle. Then such things as criteria to
determine if the software was ready to advance to the testing stage
were used to move the software from development to testing. This does
not meet the needs for Agile development.
The Director of Test Engineering becomes a Director of Requirements
Adherence (This title could be improved). This role is more critical
in that it requires the management of every software developer.
Maverick Agile Development for Large teams requires the development of
unit tests first. This good practice was widely introduced in the XP
methodology. The unit tests are must be driven by data or object
generators. MQA requires a correct implementation of a data/object
generator or factory. This component is necessary for the MQA
methodology to work. It is the Director's responsibility to see that
this generator is developed by those with the domain knowledge of the
objects. Also, a data validator is needed as well. Those that
understand the context of the data must be able to determine if an
object is valid. Because of the 2nd Dimension of programming described
in Maverick Agile Development there are certain behaviors that objects
must have because they exist in the computer real (2nd Dimension). One
of these behaviors is equivalency. In order to validate that an object
that is immutable did not mutate the object must be comparable to
another object. It is the Director's responsibility to see that the
data generator, data validator, and data equivalence verifier are in
place.
Unit tests call stubs or mock objects. Many developers want to skip
this step. The Director of Requirements Adherence is charged with
insuring these stubs and mock objects are created. Through the use of
the data generators the difficulty with the mock objects is
alleviated. Many unit tests just test the properties of the object
(the getters and the setters). This is an incomplete unit test and
therefore is not accepted as a unit test at all.
Since all tests are written before the code is developed the role of
the Director of Requirements Adherence is more significant still. A
traditional Director of Development has not been charged with writing
"good" tests. That has been the role of the testing department.
Developers may know the fundamentals of writing good tests but they
have not written tests often and they have not been evaluated on their
ability to write good tests. Therefore the Director of RA has to
insure that good tests are written.
Writing the unit tests will take a significant amount of time and it
will be a while before development begins on the other code. Since MQA
is proposed for iterative development methodologies, the beginning of
each new iteration that adds a new use case to the system will begin
with writing unit tests. As you can see the role of the Director of RA
is on going and very demanding. It raises the bar for the person in
the role. They have to be able to internalize changes and react
correctly to those changes.
If a modification is needed to an object then a test is written first
to validate the change when it is implemented. Thus the role of the
Director of RA is continuous and essential. Making sure the tests are
correct, that the data generation and other infrastructure are kept up
to date with the changes are difficult tasks.
Metrics
Metrics are necessary to report the many states of software
development. MQA requires that metrics and their gathering be kept
simple and to the point. Multiple reports that present the same data
in various ways should not be generated.
Customer acceptance can be done with a simple rating. A scale from one
to ten on satisfaction can capture the acceptance level. When the
product reaches a ten rating, the product is accepted. Metrics could
be broken down on factors such as on time delivery, feature
correctness, and other factors that determine the overall acceptance
of a software product.
Verification and Validation of the software for MQA should measure
that all unit tests are in place before coding, that the data
generators, validators, and equivalence verifiers are in place, and
that all tests are passing. Also, from the iteration goals, metrics
should be available on the features for that iteration and whether or
not they were produced.
Since the continuous build process runs all of the tests (unit and
component) there should be a metric from acceptance testing on any
bugs that were found that was not caught by the unit and component
tests. Any bug found at the acceptance test level should be converted
into a unit or component test and placed into the system. Also, the
Director of Requirements Adherence will determine why the test case
was missed and inspect the system to see if similar oversights exist
elsewhere. From this a new type of test will be introduced into the
required test suite for testing in applicable and similar domains.
Unit Tests
Unit tests are written before the code is developed. If a change in
existing code is needed a test is written first to show the code
adheres to the new requirement.
Unit tests test all of the behavior of the object. Any calls external
to the object will be mocked or stubbed and the data generation
facilities will be used for these stubs and mock objects.
The developers have the domain knowledge of the units and they write
the unit tests.
Component Tests
Component tests are the same as unit tests. They are built differently
in that the libraries (JAR files) that contain the mock objects and
stubs are replaced with the real components. This gives maximum reuse
of the unit tests. This should reduce the issues of "why didn't the
unit tests catch this?"
The product is integrated when all of the component tests pass.
The developers have the domain knowledge of the components and will
pair with the developers of the called components and write the
component tests.
System Tests
System tests are the component tests ran in a deployed environment.
The configuration and deployment team have the domain knowledge and
access to the deploy architecture.
Performance Tests
Performance tests are ran on the deployed environment. It is not
reasonable to make optimizations for any platform other than the
deploy configuration. Performance test harnesses may be needed to
provide timing and load tests. These harnesses should call and reuse
the component tests. Additional tests will be needed to simulate
network outages and other aspects.
The configuration, deployment, and development teams have the domain
knowledge to produce the performance tests.
Regression Tests
Regression tests are necessary to identify changes. The Director of
Requirements Adherence is responsible that unit tests are not changed
to perform an entirely new test when functionality changes but that an
entirely new test is written.
Tests of Third Party Components
The test of such things as database performance, network performance
and other third party and external systems is essential.
The configuration and deployment team as well as those that selected
the third part solutions know and understand the expectation of these
solutions. These individuals will be responsible for writing tests
that show these systems meet the requirements of the product. (The bar
has been raised for the configuration management teams in that member
of their team must know how to test the third party components of the
system to the standard that the third party components were expected
to meet).
Mutation Tests
Mutation tests are tests of the tests. Given a unit that passes all of
the tests for that unit the unit is modified in a way that is not
correct for the requirements. Then it is observed to see if the unit
tests fail as expected or if the unit tests do not catch the bug.
The domain knowledge of mutation testing is in development. The
Directory of Requirements Adherence is responsible for assuring
adequate mutation testing is occurring.
Acceptance Tests
Acceptance tests are done by the customer, by a test technician (using
white box and scripting approaches), and by beta testers. Any bugs are
logged and the first question to be asked is why this bug wasn't
caught by unit or component tests.
The domain knowledge for performing these tests is with the customer
and the test technician. Specific features will be known by the
developer that is responsible for that feature.
Trusted Data Generation
The data generation for MQA is essential. The data generation is
considered a trusted component. The data generated must meet the
requirements of the system. Faulty data generation invalidates all
tests associated with that data. The Director of Requirements
Adherence is responsible to make sure the data generation facility is
correct. This component has the status of a critical component like
the steering wheel of your car.
Data Validation and Equivalence Verification
In order to test the units there must be a way to validate the data
and compare the data with standards. If the result of a call into the
system produces an object it must be validated. The developer
responsible for the object must know the requirements for a valid
object and is responsible for providing a validator. As well, the
developer knows what is considered equivalent. Equivalence can be very
difficult and the problem of record linkage addresses on aspect of
equivalence. Trying to identify if two person objects are equivalent
may not be possible. A ranking or some solution must be developed in
order to accommodate the tests. If the matching of persons are part of
the system then the matching facilities would be used by the
Equivalence verifier and the status of trusted component would have to
be removed from the equivalence verifier. Reality is such that these
concessions would have to be made.
Conclusion
Maverick Quality Assurance (MQA) considers the goals of quality
assurance (QA) upfront and proposes a method for full QA in an Agile
development process.
Maverick Quality Assurance identifies the goals of the QA team to be
the delivery of a valid product that is verified to the customer. As
well the QA team is responsible to the employer to identify that the
product is on time and within budget.
By moving the component tests into the area of development where the
domain knowledge resides the problems with the transfer of domain
knowledge to an external integration test team is eliminated.
By adding a trusted data generator, validator, and equivalence
verifier to the other aspects of Agile development the methods of MQA
can be performed.
|