How do you make decisions that optimize software quality?

Hi there,

Thanks to all for your feedback on my previous question on unit
testing.

I want to step back and ask the group a broader question.  Imagine
you are the VP Engineering at a large software company with the
mandate that you must deliver software at the absolute lowest possible
total cost of development and deployment - that means, low cost to
engineer but with high enough quality that post-release deployment
costs are also low (limited bugs, etc).


1) What are the top 3-5 problems or issues be in your experience that
would strain my ability to be low cost AND high quality?

1a) How do people resolve these 3-5 problems?  Tools?  Software
lifecyle methodologies?  Which of these tools and methods can be
provably shown to mitigate those 3-5 problems or is it just
statistical/anecdotal suggestion?


2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
but could manifest as a problem) that would strain low cost/high
ability?

2a) How do people prevent those risks from arising?  Tools?
Methodologies?  Can the tools and methods be provably shown to
suppress the risks or is it statistical/anecdotal suggestion?


3) What are the top 3-5 decision tradeoffs that a VP Engineering has
to make to deliver software - i.e. not problems here, but just
decisions that have to be made between deliving in one way vs.
another?

3a) How do people determine the optimal decision for those tradeoffs?



Any input on these questions - even if only for a subsection would be
greatly appreciated.  I hope it actually spurns some good discussionn
because my guess is that different people have different opinions.

Andrew wrote:

>   I want to step back and ask the group a broader question.  Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

Release frequently and deploy as often as feasible, so that most of
development occurs within a maintenance context.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

Suppose your strategy were to sort features in business priority, and work
on the most important ones first. They might not lead immediately to a
deployable solution. They might require a few more iterations of
lower-priority work until

Suppose you address that issue by buying a legacy system that comes close to
covering your features. Now you must retrofit tests on it.

Suppose instead you address that issue by writing an entire tool from
scratch (test-first, natch).

The first and last option raise quality, because in-house test-first is
always better than OTS. (Jack of all trades, master of none, etc.)

The middle option lowers quality.

All three options delay feedback regarding external quality. How well does
our deployment channel work?

> 1a) How do people resolve these 3-5 problems?  Tools?  Software
> lifecyle methodologies?  Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

You are asking how to be a Software Engineer... ;-)

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

Unexpected and untested things. Tests should cover every detail from the
programmer's keyboard to the user's platform.

For example, consider these two user stories:

- the appliance shall use a dual-booting RAID

- install the appliance, turn it off, yank either RAID
cable, turn it on, and expect it to boot and work

Now suppose the first story goes long, because Fry's does not carry the RAID
card that supports our appliance and OS. That bumps the second story out of
the iteration.

Now suppose you fly to the customer's site, install the appliance, turn it
on, plant your hands on your hips with a big satisfied grin, and the RAID
cable falls out. The system doesn't boot, and you have to fly home, fix
everything, and fly back.

That's a true story: The one thing they didn't test was the first thing that
happened to them at deployment time.

Now consider this user story (also part of that true story):

- a form on a Web page backs up the server's database

To get that to pass, write a test that claws its way thru all these steps:
Create a sample database, put sample data in it, call production code to
back it up, zilch the database, restore it from the backup, and look for
that sample data.

On the customer site (the second time around), demonstrate the backup
system. Then accidentally erase the production database. You are saved: The
backup is tested, and all you need to do is restore it. (Then quickly
disable the backdoor you used to erase the database!)

> 2a) How do people prevent those risks from arising?  Tools?
> Methodologies?  Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

By trying to force them to surface. Engineers s ways to cause the next
problem they plan to solve.

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

I have heard of the acronym ROI, but folks use it somewhat mystifyingly...

> 3a) How do people determine the optimal decision for those tradeoffs?

Emergently. Follow simple rules that constrain decisions, and repeat all
aspects of a project over and over again. This cycle converges on an
optimum, without over-control.

--
Phlip
http://industrialxp.org/community/b...tUserInterfaces

Hello Andrew,

On Wed, 11 Aug 2004 19:41:31 UTC, analystresearch2002@yahoo.com (Andrew)
wrote:

> Hi there,
>
>   Thanks to all for your feedback on my previous question on unit
> testing.
>
>   I want to step back and ask the group a broader question.  Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

I would hope that as VP of Engineering, you would include as part of
your responsibilities, a periodic evaluation of your overall costs,
productivity, and conformance/deviations to prescribed standards.
This would help identify your costs, how they fluctuate, and which
measurements might be tunable.

While your responsibility would be very high level, the figures
gathered from teams or groups under you may have varied organizations.
You must also be careful to make changes that will achieve your goals
while not adversely affecting other parameters.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

What are your goals?  What is an acceptable cost for all the
development?  This should include design, implementation, verification,
validation, release preparation, and training.  What are your post
delivery costs?  These may include rework time, emergency customer
support, irate customers, and perhaps unexpected impact on
subsequent development or procedures.  When must you deliver products?
What classes of service do you offer?  For instance, many companies
provide "bronze", "gold", and "platinum" service to customers.
What preparations have you and will you make to accomplish those
service agreements?


Are there particular failures within your organization today?
If not, look closer.  I could name perhaps a dozen problems
that could be factors in a large organization.  The top 3-5
problems for your hypothetical organization must be determined
so that you will address the proper issues.

> 1a) How do people resolve these 3-5 problems?  Tools?  Software
> lifecyle methodologies?  Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?
>
>
> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

There are more risks than can easily be mentioned.  How good is
your current process and what are you and your teams/departments
doing to improve your overall goals?

> 2a) How do people prevent those risks from arising?  Tools?
> Methodologies?  Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?
>
>
> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

I'll hazard a guess that in many organizations your top problem
is that you were given the goal to unilaterally reduce costs in the
first place.  Your second problem would be the unexpected results
of changes that you actually can make.  Your third problem would be
the political ramifications and tradeoffs you had to make to accomplish
your goals.  (e.g. Reducing documentation effort under your group
may adversely affect another department who has similar or perhaps
conflicting goals.)

> 3a) How do people determine the optimal decision for those tradeoffs?

Any optimal configuration is likely to be for the moment and will
change.  Your next set of goals as VP may be different, but affect what
you just (hopefully) accomplished.

> Any input on these questions - even if only for a subsection would be
> greatly appreciated.  I hope it actually spurns some good discussionn
> because my guess is that different people have different opinions.

David

"Andrew" <analystresearch2002@yahoo.com> wrote in message
news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
> Hi there,
>
>   Thanks to all for your feedback on my previous question on unit
> testing.
>
>   I want to step back and ask the group a broader question.  Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

I think the first thing is to get an economic model, and since I'm
answering this from an XP perspective, it's going to be the simplest
economic model that could possibly be informative.

Post-deployment costs of defects start with lost productivity
due to the defects, which in turn cascades into lost sales, etc.
Those, in turn, go back to engineering costs in finding and fixing
the defects, which in turn displaces other activities that could
be used to enhance revenue.

In other words, I find it hard to believe that there is such a
thing as a field-reported defect where the economic loss in
the field is less than the economic cost of not letting it out in
the first place. Other people may differ on this, however.

The second limb of this is that code quality directly affects
the ease of adding additional features (and of repairing
defects) in an existing code base. Remember that I'm answering
this from an agile perspective: most of what is delivered will
consist of code that has been extensively modified in the
process of developing the product.

The metric here is velocity: how does the ease (that is to say,
the cost) of adding a feature change over time. If it's stable, then
you've got a code base whose quality is "good enough."

These two measures of quality are related, but not identical.
For may business applications, the second (quality as seen
by the developers) may exceed the quality as needed by
the end users. For life, safety or mission critical applications,
the reverse may be true.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?
>
> 1a) How do people resolve these 3-5 problems?  Tools?  Software
> lifecyle methodologies?  Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

XP teams can deliver substantial projects that have less than
one field reported defect per month. See, for example,

http://martinfowler.com/bliki/VeryLowDefectProject.html

By can I don't mean that they manage to do so as a routine
consequence of using XP! I mean that there are some real
world examples of projects that do it. Regardless of some
proponent's rhetoric, XP is not a magic bullet that cures
every software development ill under the sun. It is, combined
with experianced project management and a dedicated
development staff (and possibly a secret sauce) capable
of getting the job done.

There are several basic processes needed.

1. Executable (meaning repeatable) acceptance tests.
These tests must be accessable to non-technical people
on the customer side (including analysts, QA technicians,
and documentation specialists.) That means that they
can't be written in xUnit or, even worse, some variety
of GUI automation scripting language.

2. Every line of code the developers write must be
integrated that night. The integration and build system
must run all of the passing acceptance tests to date,
and all of the unit tests must run at 100%. The build
must finish with a properly packaged deployable. The
build system may run analysis code like code coverage
metrics; if so, unit test statement and branch coverage
must be at 100%, and acceptance tests coverage and
branch metrics must be fairly high.

3. QA must be done statistically. That is, it is the job
of QA to insure that the product meets quality standards,
not to discover specific defects. A failure to meet
quality standards means that the process needs to be
amended until it can produce a product that meets
quality standards.

The interesting thing about this list is that it's not completely
specific to XP. XP mandates the first two and strongly suggests
the third. Most shops doing standard software engineering
could put in 1 and 3 without seriously affecting their
work flow. Item 2 is not possible unless the shop shifts
to TDD rather than sequential design, code, test and
integrate phases on each module.

The other thing about this is that it doesn't mandate
a lot of expensive tools. The current practice on 1
involves the FIT and FitNesse packages, both of which
are GPL.

Code analyzers (like coverage analyzers) in item 2
may well cost a bit, but they're worth every cent both
from the standpoint of knowing where you're at and
from the standpoint of controlling cowboy coders.

Likewise, some tool support of item 3 may well
be appropriate.

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

The top risk is not paying enough attention to keeping
the code base sufficiently maintainable that you get a
flat velocity curve. Crap code costs money. So does
deteriorating design.

> 2a) How do people prevent those risks from arising?  Tools?
> Methodologies?  Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

At this point, there are projects that succeed with just about
every imaginable process, and projects that fail with the
exact same processses.

The only thing I know of that's helpful across the board is
to keep believable progress indicators, and the only way
I know of doing that is to keep WIP (work in progress)
to an absolute minimum. WIP is anything that's somewhere
between "detailed requirements analysis started" and
"successfully integrated, all acceptance tests pass."

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

The biggest factor with an XP team is that it's
capable of putting out a fairly predictable amount
of function per iteration. Once a team has geled,
it's not all that easy to add or subtract people, or
change process. Increasing the amount of function
per iteration is a slow process that requires a lot
of attention.

The lever that can be controlled is the actual functionality
that's worked on in any iteration going toward a formal
release. Pulling all of the stakeholders together for a
project is a problem that's fully worthy of a VP's time.

The other issue is that there will be times when the
product needs major restructuring work to maintain
design integrity, and that may not correspond to
actual deliverables. This can be deferred for a while,
but it cannot be ignored or the ability of the team
to deliver at a constant velocity will suffer.

> 3a) How do people determine the optimal decision for those tradeoffs?

I've never figured out how a marketing department comes
up with their estimates.

> Any input on these questions - even if only for a subsection would be
> greatly appreciated.  I hope it actually spurns some good discussionn
> because my guess is that different people have different opinions.

You're welcome.

John Roth

In article <43cf64b0.0408111141.4cc1a6dc@posting.google.com>,
analystresearch2002@yahoo.com (Andrew) wrote:

>   I want to step back and ask the group a broader question.  Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

Alarm bells start ringing here - "absolute lowest possible total cost"
emphasizes low cost twice.  This is means _cheap_.  That is ok - a first
crack proof of concept is usually better done really cheaply, because it
is going to be completely redone before delivery.  Two poorly written,
but working, demos are often better than one quality demo.

That said, if the folks at the top REALLY mean absolute bottom quality,
you may have some frights ahead.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

The tradeoff, with vague terms.  Get the stakeholders to commit to a
level of quality and a cost, and then you might have a prayer.  For
example, ask them how many ws of downtime a month they want, then
when they go ballistic, point out that they likely want fairly high
quality code, and thus it will take appropriate resources to create.
Get a commitment for the number of bugs (statistically) they are willing
to accept in the code base, and some idea of budget.  Then, you can
start planning.  Also ask which one is more flexible - budget, time, or
bug count.

Also, if lowest cost is a priority, they are unlikely to pay for top
tier talent.  Again, if the project does not justify it, you have
nothing to worry about, but if it does, then you have made a business
case for better employees.

Beware of vague specs and people who refuse to comment on intermediate
builds.  You do not always need an "on site customer", but you do need
someone who has approval and rejection authority for features, and you
need the key stakeholders to agree that said person can do that.
Nothing sucks more than hitting the end of a project and being told "the
VP of finance really wanted feature foo, and is thus unhappy with the
product and the high cost", when you gave them the choice of feature foo
or feature bar, and they chose foo.

Finally, lowest cost often implies cheaping out on the physical plant.
Get your developers in offices with doors that close, and get them out
of meetings.  This increases productivity and lowers costs over the long
run, though it may be hard to run by the hierarchy police.

Again, these are not always necessary to accomplish great things, but
they help, and thus they reduce the risk of problems.

> 1a) How do people resolve these 3-5 problems?  Tools?  Software
> lifecyle methodologies?  Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

Provable?  Don't go down that road, as that boosts your cost by orders
of magnitude.  For reasonable cost, you get statistical bug counts.
This is not a bad state, as you can adopt unit tests and continuous
integration to try and get a handle on bug counts, fix rates, and
reversion prevention.

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

If your bug counts are higher than you expect, or your feature set
changes after you have built your acceptence criteria and unit test
harness, you may well be shafted.


> 2a) How do people prevent those risks from arising?  Tools?
> Methodologies?  Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

Some risks happen - that is why they are risks.  By taking control, you
can manage risk, but then you have higher cost as a result.  Look at
DeMarco's work, and the people he references to see how you can  do some
of that management.  (Hint: you can drive the odds of a given failure
arbitrarily close to zero by spending enough and doing enough testing,
but cost goes up exponentially, because these techniques

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

The biggest single decision is who gets to work on the project, and how
you intend to run it.  Get buy-in on your high level decisions, and you
are more likely to make the proper low level decisions.

Scott

"Andrew" <analystresearch2002@yahoo.com> wrote in message
news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
> Hi there,
>
>   Thanks to all for your feedback on my previous question on unit
> testing.
>
>   I want to step back and ask the group a broader question.  Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).
>
>
> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

1: Institutional blindness: a typical problem in many institutions
is the inability to see actual differences between what we
-say- we do and what we actually do. Most companies
that make crap have a motto that says something like
"quality is job 1"
2: Unmeasurable goals. You have one above. "Lowest possible
cost, but high enough quality to keep post release costs low"
is unmeasurable. Accept that that is a -wish-, not a goal,
and use it as a guideline in setting measurable goals.
Measure current results, decide on an acceptable metric,
and measure future progress against that metric.
3: Authority and responsibility resting in separate hands.
Whoever is charged with making things work has to have
the authority to: (a) change the plan, (b) reject requests from
business people (c) go directly to highest management levels
at need (d) decide spending [how much may be up to
others; how it is spent must be up to the guy who has to
make things work] (e) the authority to slip the release
date; this will be the one you probably won't get.

> 1a) How do people resolve these 3-5 problems?  Tools?

No.

> Software lifecyle methodologies?

No.

> Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

No tools can solve the biggest problems in developing
quality sotware cheap. The biggest issues are -always-
people issues, never technology issues. For every one
project that fails due to a technology issue, there are
probably hundreds that fail due to people management
issues.

The absolute best thing any tool or mehodology can do
for you is let you see the people problems quickly
enough to step in.

Many of the things Agile mthodologies recommend are
directed at this problem. One value of peer programming
is that it identifies coder problems that show up in the
code more quickly than otherwise.

An advantage to frequent unit testing and frequent
releases is that they identify communication issues
quickly. They find cases where the code fulfils the
requirements but not the user needs.

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

1: slow reaction time to a problem. Problems will occur;
the earlier they are identified and addressed, the cheaper
it is, and the better the code quality.

> 2a) How do people prevent those risks from arising?  Tools?

No.

> Methodologies?

No.

> Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

Tools don't solve problems. People solve problems. The
good tools help people solve problems faster, easier, cheaper.

Ultimately, though, computers don't run the business;
people do. You can have the best technology, and the
company can still end up in the dumpster.

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

1: quality vs time to market. The best thing is to try to
accurately measure the cost vs. benefit. A typcial
problem is that a decision is made prior to the CB
analysis, and the CB analysis is subtly skewed toward
supporting the prior decision. Logic is used to justify
the decision, rather than to make the decision.

2: support costs vs time to market. See above. Put
numbers on time to market.

> 3a) How do people determine the optimal decision for those tradeoffs?

Mathematically. See a basic accounting textbook.
Typical examples used are the book publishing examples
for return on investment. Don't forget returns, exchanges,
competitive upgrades, other non-obvious support costs.

> Any input on these questions - even if only for a subsection would be
> greatly appreciated.  I hope it actually spurns some good discussionn
> because my guess is that different people have different opinions.

My apologies if my answers seem terse. But your
questions are geared toward finding a technological
solution, and there ain't no such animal. Ultimately
the best technology we have can only -support-
good management, never make up for its absence.


--
Ron Ruble
raffles2@att.net
Prefix email messages with "Newsgroup" to bypass spam filters

On Wed, 11 Aug 2004 20:20:39 +0000, Phlip wrote:
[liberal snippage]
> Suppose you address that issue by buying a legacy system that comes
> close to covering your features. Now you must retrofit tests on it.

Unfortunately, I get the impression that COTS solutions are hard to come
by, and harder to retrofit. I know several executives that have gone to
acquire products and/or divisions who narrowly escaped buying (hot) "air".
I think this may be related to a general difficulty of arbitrarily
breaking up systems and contracting out parts. Ostensibly that should be
easy, but in practice (inadequate and/or incomplete specs?) it's hard.

> Unexpected and untested things. Tests should cover every detail from the
> programmer's keyboard to the user's platform.

That's a pretty general statement, sort of "motherhood and apple pie", and
therefore (on its own) pretty useless. You cannot test everything, except
for very simple, highly constrained (sub)systems. Your anecdotes were more
useful to give examples of things that should be tested but are
overlooked. I think in practice, one is given (or has to fight for?) a
testing budget, and after some brainstorming one devises a practically
executable (within that budget) test plan. This should include the most
important (i.e. highest risk? by some measure) kinds of error conditions,
and probably also the most annoying (another measure?). BTW, I've seen
testing budgets cut as a project got into financial difficulty. Amazing?

--
Juhan Leemet
Logicognosis, Inc.

On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
> "Andrew" <analystresearch2002@yahoo.com> wrote in message
> news:43cf64b0.0408111141.4cc1a6dc@posting.google.com... 

Somewhat open ended and demanding question?

> There are several basic processes needed.
>
> 1. Executable (meaning repeatable) acceptance tests. These tests must be
> accessable to non-technical people on the customer side...

I would go further and say that these tests should be "defined" (perhaps
with assistance) by the customer and/or his representatives. In some/many
contractual arrangments that is mandated. Always a good idea.

> 2. Every line of code the developers write must be integrated that
> night. The integration and build system must run all of the passing
> acceptance tests to date, and all of the unit tests must run at 100%.
> The build must finish with a properly packaged deployable. The build
> system may run analysis code like code coverage metrics; if so, unit
> test statement and branch coverage must be at 100%, and acceptance tests
> coverage and branch metrics must be fairly high.

That sounds pretty arbitrary to me: esp. "every night". That's the popular
line, but any period would do, depending on the shop and the process, etc.

viz. the requirements of the built system: as a lawyer used to tell me
"what is the remedy?" Clearly it should be the goal of a built system
to pass all/most tests, etc., but suppose it does not work? (at all?) What
do you do about it? Probably a "jeopardy alert" to the project management
viz. schedule and cost overruns. What else? What can you do, but "reject"
the build contents and redo the build and test at a later time.

> 3. QA must be done statistically...

This will probably be a quality measured "after the fact"? Naively one
might think one can set a target like 99.9% (or some such?). A mature shop
with an establised and "calibrated" process might be able to "call their
shots" like that. More likely they know what their average process results
are, and they won't stray significantly. Isn't this related to CMM stuff?
If you're starting out with a new group/process it's going to be tough.
Maybe set a goal of no severe bugs found within a particulat time interval?

> The interesting thing about this list is that it's not completely
> specific to XP. XP mandates...

I would hesitate to presuppose any implementation method or methodology,
while trying to set objectives and targets. If they are chosen after goals
are set, and they achieve the goals, then that's great!

Also, a general comment on "bugs" and "issues". Some organizations and/or
customers are shocked at the numbers of bugs that are actually discovered
and/or recorded. I would suggest that even pretty good shops have some few
per KLOC (not popular metric, but that IS what people write!). Boehm
suggested IIRC that the measured injection rate for software development
was something like 50 errors per KLOC? Good developers and integrators
get most of them out, but some remain. That was some 20 years ago, but
like the saying goes "times may have changed, but people haven't". Does
anyone have any good numbers, from more recent studies? Wooly anyway?

I have found it useful to "manage expectations" of (naive?) project and
corporate management to mention that they should expect some at least
5/KLOC bugs/issues. (No way! is the usual response) If you don't have
tools and processes in place to manage that number (i.e. 1MLOC -> 5000+
"bugs" or "issues") you will get buried! Dunno what an average cost is,
but for smallish projects I have used about $500/bug for budgeting in the
past (i.e. 1MLOC -> $2.5M+ for bug/issue fixing). If you're not planning
or setting aside something similar (or more), there are likely going to be
some bitter tears (yours and/or customers)!

Like that old joke:
"You want good, cheap, fast? Pick any 2 and call me back!"

Sometimes organizations set themselves up for failure by arbitrarily and
"willfully" driving the software development process out of its
"compliance limits" (i.e. until something breaks). There's no free lunch.

--
Juhan Leemet
Logicognosis, Inc.

"Juhan Leemet" <juhan@logicognosis.com> wrote in message
news:pan.2004.08.12.18.20.57.825960@logicognosis.com...
> On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote: 
>
> Somewhat open ended and demanding question?
> 
>
> I would go further and say that these tests should be "defined" (perhaps
> with assistance) by the customer and/or his representatives. In some/many
> contractual arrangments that is mandated. Always a good idea.

That is, of course, what XP suggests is the best way of doing it.
 
>
> That sounds pretty arbitrary to me: esp. "every night". That's the popular
> line, but any period would do, depending on the shop and the process, etc.

Actually, any period will not do. There is a growing body
of evidence that the amount of trouble you have in integration
is super-linear in the amount of time code has stayed out without
being integrated. In other words, "integration Hell" is an easily
predictable consequence of leaving integration until the end
of the project.

On the other side, daily is "white book" XP. In reality,
XP today recommends that you integrate as soon as you've
gone though a full TDD cycle of 5 to 10 test/edit cycles and then
refactor to design goodness. That is, every 20 to 30 minutes.
I didn't want to be *quite* that extreme on this response,
though!

The experiance in XP shops is that they almost never have
integration problems, even though everyone is allowed
(indeed encouraged) to change anything, anywhere in the
code base that they need to for the current story.

>
> viz. the requirements of the built system: as a lawyer used to tell me
> "what is the remedy?" Clearly it should be the goal of a built system
> to pass all/most tests, etc., but suppose it does not work? (at all?) What
> do you do about it? Probably a "jeopardy alert" to the project management
> viz. schedule and cost overruns. What else? What can you do, but "reject"
> the build contents and redo the build and test at a later time.

If you're doing daily (or more frequent) integration / builds, then
"it doesn't work" isn't a crisis - except for the dummy that broke
the build. (The Microsoft principle - one of the few real crimes that
their developers can commit is to break the build.)

 

> This will probably be a quality measured "after the fact"? Naively one
> might think one can set a target like 99.9% (or some such?). A mature shop
> with an establised and "calibrated" process might be able to "call their
> shots" like that. More likely they know what their average process results
> are, and they won't stray significantly. Isn't this related to CMM stuff?
> If you're starting out with a new group/process it's going to be tough.
> Maybe set a goal of no severe bugs found within a particulat time
interval?

Measurement is always "after the fact." The issue isn't when you
do the measurement, the issue is whether what you're doing after
the fact is a bug hunt, or measuring the quality of the product
for purposes of feeding that back to improve process.
 
>
> I would hesitate to presuppose any implementation method or methodology,
> while trying to set objectives and targets. If they are chosen after goals
> are set, and they achieve the goals, then that's great!

As I said up front in my original response, I'm answering this
question from an XP perspective. I'm not trying to be even-handed
since I firmly believe that "standard software engineering" methodologies
have demonstrable problems, both in effectiveness and efficiency.

> Also, a general comment on "bugs" and "issues". Some organizations and/or
> customers are shocked at the numbers of bugs that are actually discovered
> and/or recorded. I would suggest that even pretty good shops have some few
> per KLOC (not popular metric, but that IS what people write!). Boehm
> suggested IIRC that the measured injection rate for software development
> was something like 50 errors per KLOC? Good developers and integrators
> get most of them out, but some remain. That was some 20 years ago, but
> like the saying goes "times may have changed, but people haven't". Does
> anyone have any good numbers, from more recent studies? Wooly anyway?

See the comment in my original post. It's demonstrably possible
to get an almost defect-free product because some teams are
actually doing it. Customer visible defects are *expensive*! One
of the rallying cries in manufacturing in the last several decades
has been "Quality is free", meaning that increasing your quality
has paybacks in terms of support and warrenty service, etc., that
more than pay for the effort.

> I have found it useful to "manage expectations" of (naive?) project and
> corporate management to mention that they should expect some at least
> 5/KLOC bugs/issues. (No way! is the usual response) If you don't have
> tools and processes in place to manage that number (i.e. 1MLOC -> 5000+
> "bugs" or "issues") you will get buried! Dunno what an average cost is,
> but for smallish projects I have used about $500/bug for budgeting in the
> past (i.e. 1MLOC -> $2.5M+ for bug/issue fixing). If you're not planning
> or setting aside something similar (or more), there are likely going to be
> some bitter tears (yours and/or customers)!

There is a naive approach that says: we can do anything we want,
and it'll work out in the end. That demonstrably leads to crap.

> Like that old joke:
> "You want good, cheap, fast? Pick any 2 and call me back!"

The only way you get quality is to pay thorough, detailed attention
to quality in every aspect of the process. There is no free lunch in
this respect. I believe it was Socrates that is supposed to have said:
"The unexamined life is not worth living." That was 2500 years ago,
and it's still true: the unexamined process most likely doesn't do
what you think it does.

The notion that you will inevitably produce a defective product
is one of the most pernicious beliefs in the industry today. It
gets repeated ad nausium by the pundits to excuse the huge
amount of shoddy software that's being ground out by developers
who are being mismanaged by people that think they know better
than the developers how to write software, and continually impose
demonstrably bad processes on them simply because the process
salesman is slickly convincing.

> Sometimes organizations set themselves up for failure by arbitrarily and
> "willfully" driving the software development process out of its
> "compliance limits" (i.e. until something breaks). There's no free lunch.

I'm not sure what you're saying here, other than that every process
has known limitations. You cannot arbitrarilly specify scope, time,
budget and quality, and expect that, by some miracle, it will be
met. It doesn't work in building bridges, and it doesn't work in
building software. That doesn't stop MBA trained managers from
trying.

John Roth
>
> --
> Juhan Leemet
> Logicognosis, Inc.
>

"John Roth" <newsgroups@jhrothjr.com> wrote in message
news:10hnhvemkknag3e@news.supernews.com...
>
> "Juhan Leemet" <juhan@logicognosis.com> wrote in message
> news:pan.2004.08.12.18.20.57.825960@logicognosis.com... 
<snip> 

<snip>
 
popular 
etc.
>
> Actually, any period will not do. There is a growing body
> of evidence that the amount of trouble you have in integration
> is super-linear in the amount of time code has stayed out without
> being integrated. In other words, "integration Hell" is an easily
> predictable consequence of leaving integration until the end
> of the project.

Agreed, and I favor daily integration at a minimum.

That said, YMMV, and I have been on projects where we
did not integrate more often than wly for some period,
with no problems. I will note that:

- we would have liked to integrate more often; factors
out of our control made it impossible
- all developers were familar with each others coding
habits
- we kept in close communication, and
- we planned for integration, by agressively preventing
side effects, name collisions, keeping identifier
scope very limited, etc.

All of which mitigated the problems of infrequent
integration.
 

> Measurement is always "after the fact."

Possibly, but measurement can be -effectively- continuous,
for the best possible results, or it can be delayed until the
information is useless for correcting problems in this project/
iteration.

> The issue isn't when you
> do the measurement, the issue is whether what you're doing after
> the fact is a bug hunt, or measuring the quality of the product
> for purposes of feeding that back to improve process.

Also whether you are measuring frequently enough to
find the bugs when they are still small, or waiting till they
grow up and lay eggs that hatch into more bugs.

<snip>

> The only way you get quality is to pay thorough, detailed attention
> to quality in every aspect of the process. There is no free lunch in
> this respect. I believe it was Socrates that is supposed to have said:
> "The unexamined life is not worth living." That was 2500 years ago,
> and it's still true:

No argument.

> the unexamined process most likely doesn't do
> what you think it does.

Sounds vaguely like my other post ;>

--> a typical problem in many institutions
--> is the inability to see actual differences between
--> what we -say- we do and what we actually do

--
Ron Ruble
raffles2@att.net
Prefix email messages with "Newsgroup" to bypass spam filters