Extreme Programming - The 'Extreme Programming' Testing Trap

Author

The 'Extreme Programming' Testing Trap

Phlip

2004-05-04, 1:40 pm

http://itmanagement.earthweb.com/co...cle.php/1576121

January 29, 2003
By Linda G. Hayes

"But the scary part is that regression errors represent the highest risks of
all, because if a customer is already using a function and you break it, the
results can be catastrophic. If a new feature doesn't work it's
embarrassing, but it's new so the customer hasn't been using it and there is
less risk that production will grind to a halt.

"Let's say that you are delivering an application in 10 iterations of two
w

s each. In the first iteration, you have the same amount of code to
write as you have to test, so all is well. The problem is that for the
second iteration, and every iteration thereafter, testing has to cover all
of the cumulative functionality as well as what has been added, so that by
the tenth iteration there is at least 10X the amount of testing to be done.
In reality there is probably 100X because of the interactions among
functions, but no matter how you slice it you can't turn around the testing
as fast as you can turn around the code.

"The reason is that that adding or changing code can break other code. So,
you have to test everything the software used to do as well as any changes
or additions. Testers call this 'regression' testing, which means making
sure the software didn't lose functionality, or regress. I personally hate
the word regression because it sounds pejorative (repression, depression,
oppression), and because everyone seems to think it is an obscure obsession
unique to anal retentive testers who won't let you pry the code out of their
grasping fingers.

"Here's an example. An airline needed to implement a new fare discount plan
very quickly to meet competitive pressure. The fare eligibility rules needed
some changes, but like most discount fares, it was limited to a few seats
per flight -- functionality that had been around for ages. The developers
valiantly delivered the code in a matter of hours, and so everyone assumed
the testers could turn it around in the same amount of time. Unfortunately,
this meant the only thing that could be tested was whether the eligibility
rules worked; all existing functionality -- such as the limited seat
count -- was assumed to still work since it did before.

"What happened next cost the company $50 million. Yes, the fare was only
available when the rules were satisfied, and only until the seat count was
fulfilled. Unfortunately, the flag that was added to keep count on the seats
for this particular fare was never reset for each flight. The result? The
first flight that sold out the new fare was the last one -- all other
flights showed the fare as unavailable. Planes were taking off with minimal
passenger loads as customers were driven to competing airlines where they
could obtain the lower fare. Those empty seats, and the revenue they
represented, were gone forever.

"Maybe the worst part was that no one caught it until the airlines issued
their financial results analysis and it was suddenly clear that revenue for
that fare class was next to nothing. There were no error messages, no system
crashes, no indications of anything amiss. After all...it used to work."

What XP practice, if any, did they miss in the project Ms Hayes described?

And why is XP in the title of her report if the projects she describes don't
say they use XP?

--
Phlip
http://www.xpsd.org/cgi-bin/wiki?Te...tUserInterfaces

Ilja Preuß

2004-05-04, 1:40 pm

Phlip wrote:

> What XP practice, if any, did they miss in the project Ms Hayes
> described?

She gives the answer at the end of the article:

"So what is the answer? The most effective response is to incorporate test
automation."

> And why is XP in the title of her report if the projects she
> describes don't say they use XP?

Because she wanted to write about "short iteration development", but XP in
the title sells better? Just a guess...

Regards, Ilja

Scott Kinney

2004-05-04, 4:48 pm

Without getting too hung up on specific XP practices,
take the article at face value. A defect in an iteration
went into production and was not discovered for some time.

The author's basic conclusion is one that you've asserted yourself;
the need for strong automated testing tools.

Another conclusion; that regression testing becomes more
complex and more resource consuming as the number of
features delivered grows, is hardly surprising. It also supports
a point made in "Balancing Agility and Discipline"; that the percentage
of effort spent on refactoring and correcting defects increases with
the number of user requirements/stories.

A third conclusion, or perhaps it's really just a reminder, the
cost of a defect shoots up rapidly with the amount of time it
goes uncorrected.

Scott Kinney

"Phlip" <phlip_cpp@yahoo.com> wrote in message
news:IPOlc.594$mr2.474@newssvr16.news.prodigy.com...
> http://itmanagement.earthweb.com/co...cle.php/1576121
> What XP practice, if any, did they miss in the project Ms Hayes described?
>
> And why is XP in the title of her report if the projects she describes
don't
> say they use XP?
>
> --
> Phlip
> http://www.xpsd.org/cgi-bin/wiki?Te...tUserInterfaces
>
>

Sponsored Links