Extreme Programming - How do you make decisions that optimize software quality?

Author

How do you make decisions that optimize software quality?

Andrew

2004-08-11, 3:58 pm

Hi there,

Thanks to all for your feedback on my previous question on unit
testing.

I want to step back and ask the group a broader question. Imagine
you are the VP Engineering at a large software company with the
mandate that you must deliver software at the absolute lowest possible
total cost of development and deployment - that means, low cost to
engineer but with high enough quality that post-release deployment
costs are also low (limited bugs, etc).

1) What are the top 3-5 problems or issues be in your experience that
would strain my ability to be low cost AND high quality?

1a) How do people resolve these 3-5 problems? Tools? Software
lifecyle methodologies? Which of these tools and methods can be
provably shown to mitigate those 3-5 problems or is it just
statistical/anecdotal suggestion?

2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
but could manifest as a problem) that would strain low cost/high
ability?

2a) How do people prevent those risks from arising? Tools?
Methodologies? Can the tools and methods be provably shown to
suppress the risks or is it statistical/anecdotal suggestion?

3) What are the top 3-5 decision tradeoffs that a VP Engineering has
to make to deliver software - i.e. not problems here, but just
decisions that have to be made between deliving in one way vs.
another?

3a) How do people determine the optimal decision for those tradeoffs?

Any input on these questions - even if only for a subsection would be
greatly appreciated. I hope it actually spurns some good discussionn
because my guess is that different people have different opinions.

Phlip

2004-08-11, 8:57 pm

Andrew wrote:

> I want to step back and ask the group a broader question. Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

Release frequently and deploy as often as feasible, so that most of
development occurs within a maintenance context.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

Suppose your strategy were to sort features in business priority, and work
on the most important ones first. They might not lead immediately to a
deployable solution. They might require a few more iterations of
lower-priority work until

Suppose you address that issue by buying a legacy system that comes close to
covering your features. Now you must retrofit tests on it.

Suppose instead you address that issue by writing an entire tool from
scratch (test-first, natch).

The first and last option raise quality, because in-house test-first is
always better than OTS. (Jack of all trades, master of none, etc.)

The middle option lowers quality.

All three options delay feedback regarding external quality. How well does
our deployment channel work?

> 1a) How do people resolve these 3-5 problems? Tools? Software
> lifecyle methodologies? Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

You are asking how to be a Software Engineer... ;-)

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

Unexpected and untested things. Tests should cover every detail from the
programmer's keyboard to the user's platform.

For example, consider these two user stories:

- the appliance shall use a dual-booting RAID

- install the appliance, turn it off, yank either RAID
cable, turn it on, and expect it to boot and work

Now suppose the first story goes long, because Fry's does not carry the RAID
card that supports our appliance and OS. That bumps the second story out of
the iteration.

Now suppose you fly to the customer's site, install the appliance, turn it
on, plant your hands on your hips with a big satisfied grin, and the RAID
cable falls out. The system doesn't boot, and you have to fly home, fix
everything, and fly back.

That's a true story: The one thing they didn't test was the first thing that
happened to them at deployment time.

Now consider this user story (also part of that true story):

- a form on a Web page backs up the server's database

To get that to pass, write a test that claws its way thru all these steps:
Create a sample database, put sample data in it, call production code to
back it up, zilch the database, restore it from the backup, and look for
that sample data.

On the customer site (the second time around), demonstrate the backup
system. Then accidentally erase the production database. You are saved: The
backup is tested, and all you need to do is restore it. (Then quickly
disable the backdoor you used to erase the database!)

> 2a) How do people prevent those risks from arising? Tools?
> Methodologies? Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

By trying to force them to surface. Engineers s

ways to cause the next
problem they plan to solve.

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

I have heard of the acronym ROI, but folks use it somewhat mystifyingly...

> 3a) How do people determine the optimal decision for those tradeoffs?

Emergently. Follow simple rules that constrain decisions, and repeat all
aspects of a project over and over again. This cycle converges on an
optimum, without over-control.

--
Phlip
http://industrialxp.org/community/b...tUserInterfaces

David

2004-08-11, 8:57 pm

Hello Andrew,

On Wed, 11 Aug 2004 19:41:31 UTC, analystresearch2002@yahoo.com (Andrew)
wrote:

> Hi there,
>
> Thanks to all for your feedback on my previous question on unit
> testing.
>
> I want to step back and ask the group a broader question. Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

I would hope that as VP of Engineering, you would include as part of
your responsibilities, a periodic evaluation of your overall costs,
productivity, and conformance/deviations to prescribed standards.
This would help identify your costs, how they fluctuate, and which
measurements might be tunable.

While your responsibility would be very high level, the figures
gathered from teams or groups under you may have varied organizations.
You must also be careful to make changes that will achieve your goals
while not adversely affecting other parameters.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

What are your goals? What is an acceptable cost for all the
development? This should include design, implementation, verification,
validation, release preparation, and training. What are your post
delivery costs? These may include rework time, emergency customer
support, irate customers, and perhaps unexpected impact on
subsequent development or procedures. When must you deliver products?
What classes of service do you offer? For instance, many companies
provide "bronze", "gold", and "platinum" service to customers.
What preparations have you and will you make to accomplish those
service agreements?

Are there particular failures within your organization today?
If not, look closer. I could name perhaps a dozen problems
that could be factors in a large organization. The top 3-5
problems for your hypothetical organization must be determined
so that you will address the proper issues.

> 1a) How do people resolve these 3-5 problems? Tools? Software
> lifecyle methodologies? Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?
>
>
> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

There are more risks than can easily be mentioned. How good is
your current process and what are you and your teams/departments
doing to improve your overall goals?

> 2a) How do people prevent those risks from arising? Tools?
> Methodologies? Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?
>
>
> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

I'll hazard a guess that in many organizations your top problem
is that you were given the goal to unilaterally reduce costs in the
first place. Your second problem would be the unexpected results
of changes that you actually can make. Your third problem would be
the political ramifications and tradeoffs you had to make to accomplish
your goals. (e.g. Reducing documentation effort under your group
may adversely affect another department who has similar or perhaps
conflicting goals.)

> 3a) How do people determine the optimal decision for those tradeoffs?

Any optimal configuration is likely to be for the moment and will
change. Your next set of goals as VP may be different, but affect what
you just (hopefully) accomplished.

> Any input on these questions - even if only for a subsection would be
> greatly appreciated. I hope it actually spurns some good discussionn
> because my guess is that different people have different opinions.

David

John Roth

2004-08-11, 8:57 pm

"Andrew" <analystresearch2002@yahoo.com> wrote in message
news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...
> Hi there,
>
> Thanks to all for your feedback on my previous question on unit
> testing.
>
> I want to step back and ask the group a broader question. Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

I think the first thing is to get an economic model, and since I'm
answering this from an XP perspective, it's going to be the simplest
economic model that could possibly be informative.

Post-deployment costs of defects start with lost productivity
due to the defects, which in turn cascades into lost sales, etc.
Those, in turn, go back to engineering costs in finding and fixing
the defects, which in turn displaces other activities that could
be used to enhance revenue.

In other words, I find it hard to believe that there is such a
thing as a field-reported defect where the economic loss in
the field is less than the economic cost of not letting it out in
the first place. Other people may differ on this, however.

The second limb of this is that code quality directly affects
the ease of adding additional features (and of repairing
defects) in an existing code base. Remember that I'm answering
this from an agile perspective: most of what is delivered will
consist of code that has been extensively modified in the
process of developing the product.

The metric here is velocity: how does the ease (that is to say,
the cost) of adding a feature change over time. If it's stable, then
you've got a code base whose quality is "good enough."

These two measures of quality are related, but not identical.
For may business applications, the second (quality as seen
by the developers) may exceed the quality as needed by
the end users. For life, safety or mission critical applications,
the reverse may be true.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?
>
> 1a) How do people resolve these 3-5 problems? Tools? Software
> lifecyle methodologies? Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

XP teams can deliver substantial projects that have less than
one field reported defect per month. See, for example,

http://martinfowler.com/bliki/VeryLowDefectProject.html

By can I don't mean that they manage to do so as a routine
consequence of using XP! I mean that there are some real
world examples of projects that do it. Regardless of some
proponent's rhetoric, XP is not a magic bullet that cures
every software development ill under the sun. It is, combined
with experianced project management and a dedicated
development staff (and possibly a secret sauce) capable
of getting the job done.

There are several basic processes needed.

1. Executable (meaning repeatable) acceptance tests.
These tests must be accessable to non-technical people
on the customer side (including analysts, QA technicians,
and documentation specialists.) That means that they
can't be written in xUnit or, even worse, some variety
of GUI automation scripting language.

2. Every line of code the developers write must be
integrated that night. The integration and build system
must run all of the passing acceptance tests to date,
and all of the unit tests must run at 100%. The build
must finish with a properly packaged deployable. The
build system may run analysis code like code coverage
metrics; if so, unit test statement and branch coverage
must be at 100%, and acceptance tests coverage and
branch metrics must be fairly high.

3. QA must be done statistically. That is, it is the job
of QA to insure that the product meets quality standards,
not to discover specific defects. A failure to meet
quality standards means that the process needs to be
amended until it can produce a product that meets
quality standards.

The interesting thing about this list is that it's not completely
specific to XP. XP mandates the first two and strongly suggests
the third. Most shops doing standard software engineering
could put in 1 and 3 without seriously affecting their
work flow. Item 2 is not possible unless the shop shifts
to TDD rather than sequential design, code, test and
integrate phases on each module.

The other thing about this is that it doesn't mandate
a lot of expensive tools. The current practice on 1
involves the FIT and FitNesse packages, both of which
are GPL.

Code analyzers (like coverage analyzers) in item 2
may well cost a bit, but they're worth every cent both
from the standpoint of knowing where you're at and
from the standpoint of controlling cowboy coders.

Likewise, some tool support of item 3 may well
be appropriate.

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

The top risk is not paying enough attention to keeping
the code base sufficiently maintainable that you get a
flat velocity curve. Crap code costs money. So does
deteriorating design.

> 2a) How do people prevent those risks from arising? Tools?
> Methodologies? Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

At this point, there are projects that succeed with just about
every imaginable process, and projects that fail with the
exact same processses.

The only thing I know of that's helpful across the board is
to keep believable progress indicators, and the only way
I know of doing that is to keep WIP (work in progress)
to an absolute minimum. WIP is anything that's somewhere
between "detailed requirements analysis started" and
"successfully integrated, all acceptance tests pass."

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

The biggest factor with an XP team is that it's
capable of putting out a fairly predictable amount
of function per iteration. Once a team has geled,
it's not all that easy to add or subtract people, or
change process. Increasing the amount of function
per iteration is a slow process that requires a lot
of attention.

The lever that can be controlled is the actual functionality
that's worked on in any iteration going toward a formal
release. Pulling all of the stakeholders together for a
project is a problem that's fully worthy of a VP's time.

The other issue is that there will be times when the
product needs major restructuring work to maintain
design integrity, and that may not correspond to
actual deliverables. This can be deferred for a while,
but it cannot be ignored or the ability of the team
to deliver at a constant velocity will suffer.

> 3a) How do people determine the optimal decision for those tradeoffs?

I've never figured out how a marketing department comes
up with their estimates.

> Any input on these questions - even if only for a subsection would be
> greatly appreciated. I hope it actually spurns some good discussionn
> because my guess is that different people have different opinions.

You're welcome.

John Roth

Scott Ellsworth

2004-08-12, 3:58 am

In article <43cf64b0.0408111141.4cc1a6dc@posting.google.com>,
analystresearch2002@yahoo.com (Andrew) wrote:

> I want to step back and ask the group a broader question. Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

Alarm bells start ringing here - "absolute lowest possible total cost"
emphasizes low cost twice. This is means _cheap_. That is ok - a first
crack proof of concept is usually better done really cheaply, because it
is going to be completely redone before delivery. Two poorly written,
but working, demos are often better than one quality demo.

That said, if the folks at the top REALLY mean absolute bottom quality,
you may have some frights ahead.

> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

The tradeoff, with vague terms. Get the stakeholders to commit to a
level of quality and a cost, and then you might have a prayer. For
example, ask them how many w

s of downtime a month they want, then
when they go ballistic, point out that they likely want fairly high
quality code, and thus it will take appropriate resources to create.
Get a commitment for the number of bugs (statistically) they are willing
to accept in the code base, and some idea of budget. Then, you can
start planning. Also ask which one is more flexible - budget, time, or
bug count.

Also, if lowest cost is a priority, they are unlikely to pay for top
tier talent. Again, if the project does not justify it, you have
nothing to worry about, but if it does, then you have made a business
case for better employees.

Beware of vague specs and people who refuse to comment on intermediate
builds. You do not always need an "on site customer", but you do need
someone who has approval and rejection authority for features, and you
need the key stakeholders to agree that said person can do that.
Nothing sucks more than hitting the end of a project and being told "the
VP of finance really wanted feature foo, and is thus unhappy with the
product and the high cost", when you gave them the choice of feature foo
or feature bar, and they chose foo.

Finally, lowest cost often implies cheaping out on the physical plant.
Get your developers in offices with doors that close, and get them out
of meetings. This increases productivity and lowers costs over the long
run, though it may be hard to run by the hierarchy police.

Again, these are not always necessary to accomplish great things, but
they help, and thus they reduce the risk of problems.

> 1a) How do people resolve these 3-5 problems? Tools? Software
> lifecyle methodologies? Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

Provable? Don't go down that road, as that boosts your cost by orders
of magnitude. For reasonable cost, you get statistical bug counts.
This is not a bad state, as you can adopt unit tests and continuous
integration to try and get a handle on bug counts, fix rates, and
reversion prevention.

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

If your bug counts are higher than you expect, or your feature set
changes after you have built your acceptence criteria and unit test
harness, you may well be shafted.

> 2a) How do people prevent those risks from arising? Tools?
> Methodologies? Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

Some risks happen - that is why they are risks. By taking control, you
can manage risk, but then you have higher cost as a result. Look at
DeMarco's work, and the people he references to see how you can do some
of that management. (Hint: you can drive the odds of a given failure
arbitrarily close to zero by spending enough and doing enough testing,
but cost goes up exponentially, because these techniques

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

The biggest single decision is who gets to work on the project, and how
you intend to run it. Get buy-in on your high level decisions, and you
are more likely to make the proper low level decisions.

Scott

Ron Ruble

2004-08-12, 4:03 pm

Juhan Leemet

2004-08-12, 4:03 pm

On Wed, 11 Aug 2004 20:20:39 +0000, Phlip wrote:
[liberal snippage]
> Suppose you address that issue by buying a legacy system that comes
> close to covering your features. Now you must retrofit tests on it.

Unfortunately, I get the impression that COTS solutions are hard to come
by, and harder to retrofit. I know several executives that have gone to
acquire products and/or divisions who narrowly escaped buying (hot) "air".
I think this may be related to a general difficulty of arbitrarily
breaking up systems and contracting out parts. Ostensibly that should be
easy, but in practice (inadequate and/or incomplete specs?) it's hard.

> Unexpected and untested things. Tests should cover every detail from the
> programmer's keyboard to the user's platform.

That's a pretty general statement, sort of "motherhood and apple pie", and
therefore (on its own) pretty useless. You cannot test everything, except
for very simple, highly constrained (sub)systems. Your anecdotes were more
useful to give examples of things that should be tested but are
overlooked. I think in practice, one is given (or has to fight for?) a
testing budget, and after some brainstorming one devises a practically
executable (within that budget) test plan. This should include the most
important (i.e. highest risk? by some measure) kinds of error conditions,
and probably also the most annoying (another measure?). BTW, I've seen
testing budgets cut as a project got into financial difficulty. Amazing?

--
Juhan Leemet
Logicognosis, Inc.

Juhan Leemet

2004-08-12, 4:03 pm

On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
> "Andrew" <analystresearch2002@yahoo.com> wrote in message
> news:43cf64b0.0408111141.4cc1a6dc@posting.google.com...

Somewhat open ended and demanding question?
[color=darkred]
> There are several basic processes needed.
>
> 1. Executable (meaning repeatable) acceptance tests. These tests must be
> accessable to non-technical people on the customer side...

I would go further and say that these tests should be "defined" (perhaps
with assistance) by the customer and/or his representatives. In some/many
contractual arrangments that is mandated. Always a good idea.

> 2. Every line of code the developers write must be integrated that
> night. The integration and build system must run all of the passing
> acceptance tests to date, and all of the unit tests must run at 100%.
> The build must finish with a properly packaged deployable. The build
> system may run analysis code like code coverage metrics; if so, unit
> test statement and branch coverage must be at 100%, and acceptance tests
> coverage and branch metrics must be fairly high.

That sounds pretty arbitrary to me: esp. "every night". That's the popular
line, but any period would do, depending on the shop and the process, etc.

viz. the requirements of the built system: as a lawyer used to tell me
"what is the remedy?" Clearly it should be the goal of a built system
to pass all/most tests, etc., but suppose it does not work? (at all?) What
do you do about it? Probably a "jeopardy alert" to the project management
viz. schedule and cost overruns. What else? What can you do, but "reject"
the build contents and redo the build and test at a later time.

> 3. QA must be done statistically...

This will probably be a quality measured "after the fact"? Naively one
might think one can set a target like 99.9% (or some such?). A mature shop
with an establised and "calibrated" process might be able to "call their
shots" like that. More likely they know what their average process results
are, and they won't stray significantly. Isn't this related to CMM stuff?
If you're starting out with a new group/process it's going to be tough.
Maybe set a goal of no severe bugs found within a particulat time interval?

> The interesting thing about this list is that it's not completely
> specific to XP. XP mandates...

I would hesitate to presuppose any implementation method or methodology,
while trying to set objectives and targets. If they are chosen after goals
are set, and they achieve the goals, then that's great!

Also, a general comment on "bugs" and "issues". Some organizations and/or
customers are shocked at the numbers of bugs that are actually discovered
and/or recorded. I would suggest that even pretty good shops have some few
per KLOC (not popular metric, but that IS what people write!). Boehm
suggested IIRC that the measured injection rate for software development
was something like 50 errors per KLOC? Good developers and integrators
get most of them out, but some remain. That was some 20 years ago, but
like the saying goes "times may have changed, but people haven't". Does
anyone have any good numbers, from more recent studies? Wooly anyway?

I have found it useful to "manage expectations" of (naive?) project and
corporate management to mention that they should expect some at least
5/KLOC bugs/issues. (No way! is the usual response) If you don't have
tools and processes in place to manage that number (i.e. 1MLOC -> 5000+
"bugs" or "issues") you will get buried! Dunno what an average cost is,
but for smallish projects I have used about $500/bug for budgeting in the
past (i.e. 1MLOC -> $2.5M+ for bug/issue fixing). If you're not planning
or setting aside something similar (or more), there are likely going to be
some bitter tears (yours and/or customers)!

Like that old joke:
"You want good, cheap, fast? Pick any 2 and call me back!"

Sometimes organizations set themselves up for failure by arbitrarily and
"willfully" driving the software development process out of its
"compliance limits" (i.e. until something breaks). There's no free lunch.

--
Juhan Leemet
Logicognosis, Inc.

John Roth

2004-08-12, 4:03 pm

"Juhan Leemet" <juhan@logicognosis.com> wrote in message
news:pan.2004.08.12.18.20.57.825960@logicognosis.com...
> On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:
>
> Somewhat open ended and demanding question?
>
>
> I would go further and say that these tests should be "defined" (perhaps
> with assistance) by the customer and/or his representatives. In some/many
> contractual arrangments that is mandated. Always a good idea.

That is, of course, what XP suggests is the best way of doing it.

>
> That sounds pretty arbitrary to me: esp. "every night". That's the popular
> line, but any period would do, depending on the shop and the process, etc.

Actually, any period will not do. There is a growing body
of evidence that the amount of trouble you have in integration
is super-linear in the amount of time code has stayed out without
being integrated. In other words, "integration Hell" is an easily
predictable consequence of leaving integration until the end
of the project.

On the other side, daily is "white book" XP. In reality,
XP today recommends that you integrate as soon as you've
gone though a full TDD cycle of 5 to 10 test/edit cycles and then
refactor to design goodness. That is, every 20 to 30 minutes.
I didn't want to be *quite* that extreme on this response,
though!

The experiance in XP shops is that they almost never have
integration problems, even though everyone is allowed
(indeed encouraged) to change anything, anywhere in the
code base that they need to for the current story.

>
> viz. the requirements of the built system: as a lawyer used to tell me
> "what is the remedy?" Clearly it should be the goal of a built system
> to pass all/most tests, etc., but suppose it does not work? (at all?) What
> do you do about it? Probably a "jeopardy alert" to the project management
> viz. schedule and cost overruns. What else? What can you do, but "reject"
> the build contents and redo the build and test at a later time.

If you're doing daily (or more frequent) integration / builds, then
"it doesn't work" isn't a crisis - except for the dummy that broke
the build. (The Microsoft principle - one of the few real crimes that
their developers can commit is to break the build.)

[color=darkred]
> This will probably be a quality measured "after the fact"? Naively one
> might think one can set a target like 99.9% (or some such?). A mature shop
> with an establised and "calibrated" process might be able to "call their
> shots" like that. More likely they know what their average process results
> are, and they won't stray significantly. Isn't this related to CMM stuff?
> If you're starting out with a new group/process it's going to be tough.
> Maybe set a goal of no severe bugs found within a particulat time
interval?

Measurement is always "after the fact." The issue isn't when you
do the measurement, the issue is whether what you're doing after
the fact is a bug hunt, or measuring the quality of the product
for purposes of feeding that back to improve process.

>
> I would hesitate to presuppose any implementation method or methodology,
> while trying to set objectives and targets. If they are chosen after goals
> are set, and they achieve the goals, then that's great!

As I said up front in my original response, I'm answering this
question from an XP perspective. I'm not trying to be even-handed
since I firmly believe that "standard software engineering" methodologies
have demonstrable problems, both in effectiveness and efficiency.

> Also, a general comment on "bugs" and "issues". Some organizations and/or
> customers are shocked at the numbers of bugs that are actually discovered
> and/or recorded. I would suggest that even pretty good shops have some few
> per KLOC (not popular metric, but that IS what people write!). Boehm
> suggested IIRC that the measured injection rate for software development
> was something like 50 errors per KLOC? Good developers and integrators
> get most of them out, but some remain. That was some 20 years ago, but
> like the saying goes "times may have changed, but people haven't". Does
> anyone have any good numbers, from more recent studies? Wooly anyway?

See the comment in my original post. It's demonstrably possible
to get an almost defect-free product because some teams are
actually doing it. Customer visible defects are *expensive*! One
of the rallying cries in manufacturing in the last several decades
has been "Quality is free", meaning that increasing your quality
has paybacks in terms of support and warrenty service, etc., that
more than pay for the effort.

> I have found it useful to "manage expectations" of (naive?) project and
> corporate management to mention that they should expect some at least
> 5/KLOC bugs/issues. (No way! is the usual response) If you don't have
> tools and processes in place to manage that number (i.e. 1MLOC -> 5000+
> "bugs" or "issues") you will get buried! Dunno what an average cost is,
> but for smallish projects I have used about $500/bug for budgeting in the
> past (i.e. 1MLOC -> $2.5M+ for bug/issue fixing). If you're not planning
> or setting aside something similar (or more), there are likely going to be
> some bitter tears (yours and/or customers)!

There is a naive approach that says: we can do anything we want,
and it'll work out in the end. That demonstrably leads to crap.

> Like that old joke:
> "You want good, cheap, fast? Pick any 2 and call me back!"

The only way you get quality is to pay thorough, detailed attention
to quality in every aspect of the process. There is no free lunch in
this respect. I believe it was Socrates that is supposed to have said:
"The unexamined life is not worth living." That was 2500 years ago,
and it's still true: the unexamined process most likely doesn't do
what you think it does.

The notion that you will inevitably produce a defective product
is one of the most pernicious beliefs in the industry today. It
gets repeated ad nausium by the pundits to excuse the huge
amount of shoddy software that's being ground out by developers
who are being mismanaged by people that think they know better
than the developers how to write software, and continually impose
demonstrably bad processes on them simply because the process
salesman is slickly convincing.

> Sometimes organizations set themselves up for failure by arbitrarily and
> "willfully" driving the software development process out of its
> "compliance limits" (i.e. until something breaks). There's no free lunch.

I'm not sure what you're saying here, other than that every process
has known limitations. You cannot arbitrarilly specify scope, time,
budget and quality, and expect that, by some miracle, it will be
met. It doesn't work in building bridges, and it doesn't work in
building software. That doesn't stop MBA trained managers from
trying.

John Roth
>
> --
> Juhan Leemet
> Logicognosis, Inc.
>

Ron Ruble

2004-08-12, 8:57 pm

"John Roth" <newsgroups@jhrothjr.com> wrote in message
news:10hnhvemkknag3e@news.supernews.com...
>
> "Juhan Leemet" <juhan@logicognosis.com> wrote in message
> news:pan.2004.08.12.18.20.57.825960@logicognosis.com...
<snip>[color=darkred]

<snip>
[color=darkred]
popular[color=darkred]
etc.[color=darkred]
>
> Actually, any period will not do. There is a growing body
> of evidence that the amount of trouble you have in integration
> is super-linear in the amount of time code has stayed out without
> being integrated. In other words, "integration Hell" is an easily
> predictable consequence of leaving integration until the end
> of the project.

Agreed, and I favor daily integration at a minimum.

That said, YMMV, and I have been on projects where we
did not integrate more often than w

ly for some period,
with no problems. I will note that:

- we would have liked to integrate more often; factors
out of our control made it impossible
- all developers were familar with each others coding
habits
- we kept in close communication, and
- we planned for integration, by agressively preventing
side effects, name collisions, keeping identifier
scope very limited, etc.

All of which mitigated the problems of infrequent
integration.

[color=darkred]
> Measurement is always "after the fact."

Possibly, but measurement can be -effectively- continuous,
for the best possible results, or it can be delayed until the
information is useless for correcting problems in this project/
iteration.

> The issue isn't when you
> do the measurement, the issue is whether what you're doing after
> the fact is a bug hunt, or measuring the quality of the product
> for purposes of feeding that back to improve process.

Also whether you are measuring frequently enough to
find the bugs when they are still small, or waiting till they
grow up and lay eggs that hatch into more bugs.

<snip>

> The only way you get quality is to pay thorough, detailed attention
> to quality in every aspect of the process. There is no free lunch in
> this respect. I believe it was Socrates that is supposed to have said:
> "The unexamined life is not worth living." That was 2500 years ago,
> and it's still true:

No argument.

> the unexamined process most likely doesn't do
> what you think it does.

Sounds vaguely like my other post ;>

--> a typical problem in many institutions
--> is the inability to see actual differences between
--> what we -say- we do and what we actually do

--
Ron Ruble
raffles2@att.net
Prefix email messages with "Newsgroup" to bypass spam filters

Cristiano Sadun

2004-08-13, 9:01 am

analystresearch2002@yahoo.com (Andrew) wrote in
news:43cf64b0.0408111141.4cc1a6dc@posting.google.com:

> Hi there,
>
> Thanks to all for your feedback on my previous question on unit
> testing.
>
> I want to step back and ask the group a broader question. Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).
>
>
> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

At project level, I'd say bad communication (inter-teams and with
customers), big bang integration, feature creep and "indespensable"
personnel. Depending on the nature of the projects and the teams (for
example, multi-national, with different measurement units, packaged
products vs. customized software etc).

At a level where you overview many projects, the main problem I think is
selecting managers which are focused and share (or can be taught) the
right project-level approach; getting the budget and resources to support
them; continuosly reviewing their result and applying corrections without
destroying the morale, in other words having a clear picture of the state
of each project and the necessity (or not) of corrective intervention.

At these levels, I think overviewing the doing of software is pretty much
subject to the same problems as overviewing many other organized
endeavours.

For example, as a VP you don't usually run any project directly, so you
don't decide directly on the fact that one single employee shouldn't have
the sole knowledge of a piece of system; but you can (and must) instruct
your managers to do so, and have a process that allows you to
periodically check, and the ability to intervene if the check fails etc.

>
> 1a) How do people resolve these 3-5 problems? Tools? Software
> lifecyle methodologies? Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

As I said, the higher you get the less the problems are software-
specific, and the tools (conceptually at least) are more generic: you
must have a process to measure quality (and a clear definition of it);
you must have a process to measure effectiveness (i.e. costs); you must
have a process to gather ideas on corrections and possibilities; you must
a strategy for medium and long term goals, and a process that continuosly
produces and reviews the tactics involved; etc.

> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

Inability to gather data - i.e. have a clear picture of the situation;
inability to find viable solutions; no or wrong strategy.

>
> 2a) How do people prevent those risks from arising? Tools?
> Methodologies? Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

As a VP, I suspect, by hiring submanagers that share (and/or improve)
your view - and make sure you view is sufficiently good. ;)

And gear the organization so that such view can be concretely executed in
daily operations. This part is quite difficult in itself, since at the
same time you must restructure, keep producing results and you don't have
infinite budget.

> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

Once you have the shared view, there's always a problem of costs: so the
final tradeoffs are always between expenditures and benefits. An
implemented organization will always be less than optimal, but
determining where to compromise without affecting too much the overall
result is the tricky part.

For example, if part of your view is that the code should be shared, you
need a code-sharing system. Together with your submanagers, you must
decide which tool technique to spend on vs. the perceived benefits.

>
> 3a) How do people determine the optimal decision for those tradeoffs?

Past experience, budget considerations, consultations with your team and
gut feelings. :)

Cristiano Sadun

2004-08-13, 9:01 am

Cristiano Sadun <cristianoTAKE

un@THIShotmailOUT.com> wrote in
news:Xns954467ABA60F8cristiano

unhotma
i@130.133.1.4:

> Depending on the nature of the projects and the teams (for
> example, multi-national, with different measurement units, packaged
> products vs. customized software etc).
>

...there can be additional major issues.

Ron Ruble

2004-08-13, 9:01 am

"Cristiano Sadun" <cristianoTAKE

un@THIShotmailOUT.com> wrote in message
news:Xns954467ABA60F8cristiano

unhotma
i@130.133.1.4...
<snip>
> ...as a VP you don't usually run any project directly, so you
> don't decide directly on the fact that one single employee shouldn't have
> the sole knowledge of a piece of system; but you can (and must) instruct
> your managers to do so, and have a process that allows you to
> periodically check, and the ability to intervene if the check fails etc.

That would depend on the company. In many firms, coders
are Vice Presidents, PMs are Senior Vice Presidents, etc.

Sometimes they get titles in lieu of raises and benefits, other
times they bill people out by title (common in consultancies).

> As I said, the higher you get the less the problems are software-
> specific, and the tools (conceptually at least) are more generic

Darn right.

--
Ron Ruble
raffles2@att.net
Prefix email messages with "Newsgroup" to bypass spam filters

Ronald E Jeffries

2004-08-13, 9:01 am

On Thu, 12 Aug 2004 16:20:59 -0200, Juhan Leemet
<juhan@logicognosis.com> wrote:

>
>That sounds pretty arbitrary to me: esp. "every night". That's the popular
>line, but any period would do, depending on the shop and the process, etc.

Actually, the period makes a big difference. Imagine that the tests
ran unobtrusively every millisecond. Then whenever I made a mistake,
the computer pings and I back the mistake out before it ever really
gets in.

Imagine on the other hand, that the tests ran only every other w

.
In the course of two w

s, I will inevitably break something, and now
it's integrated into the code line. Bingo, a w

later an alarm goes
off: Something is broken. Who broke it, and what did they do? We begin
debugging ...

Agile methods -- in fact all methods -- thrive on feedback. The sooner
we know something isn't right, the better we'll do.

Or so it seems to me ...

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

Phlip

2004-08-13, 3:57 pm

Juhan Leemet wrote:

> Phlip wrote:

>
> Unfortunately, I get the impression that COTS solutions are hard to come
> by, and harder to retrofit. I know several executives that have gone to
> acquire products and/or divisions who narrowly escaped buying (hot) "air".
> I think this may be related to a general difficulty of arbitrarily
> breaking up systems and contracting out parts. Ostensibly that should be
> easy, but in practice (inadequate and/or incomplete specs?) it's hard.

I responded from the project's point of view, not the VP's. My only
comprehension of executives regards how their contracts and stock options
make rewarding them for decisions made in a project's best interests
absurdly unlikely.

>
> That's a pretty general statement, sort of "motherhood and apple pie", and
> therefore (on its own) pretty useless. You cannot test everything, except
> for very simple, highly constrained (sub)systems. Your anecdotes were more
> useful to give examples of things that should be tested but are
> overlooked. I think in practice, one is given (or has to fight for?) a
> testing budget, and after some brainstorming one devises a practically
> executable (within that budget) test plan. This should include the most
> important (i.e. highest risk? by some measure) kinds of error conditions,
> and probably also the most annoying (another measure?). BTW, I've seen
> testing budgets cut as a project got into financial difficulty. Amazing?

I heard a rumor that as a company went to TDD the testers complained they
weren't finding bugs, so their metrics and rewards were going down.

It's all about incoherent reward structures, folks!

--
Phlip
http://industrialxp.org/community/b...tUserInterfaces

Ilja Preuß

2004-08-13, 3:57 pm

Juhan Leemet wrote:
> On Wed, 11 Aug 2004 20:14:33 -0400, John Roth wrote:

>
> That sounds pretty arbitrary to me: esp. "every night". That's the
> popular line, but any period would do, depending on the shop and the
> process, etc.

To me, "every night" is the minimum - a line that should always be
achievable and that I wouldn't want to cross.

Integration typically can happen much more often. Running all the tests, on
the other hand, may take a considerable amount of time, so you might be
restricted to running some of them only at night.

> viz. the requirements of the built system: as a lawyer used to tell me
> "what is the remedy?" Clearly it should be the goal of a built system
> to pass all/most tests, etc., but suppose it does not work? (at all?)
> What do you do about it? Probably a "jeopardy alert" to the project
> management viz. schedule and cost overruns. What else? What can you
> do, but "reject" the build contents and redo the build and test at a
> later time.

If the build run last night, there can't be that much to fix - so you fix it
and make sure that it's unlikely to happen again.

> Boehm suggested IIRC that the measured injection rate
> for software development was something like 50 errors per KLOC? Good
> developers and integrators
> get most of them out, but some remain. That was some 20 years ago, but
> like the saying goes "times may have changed, but people haven't".

What has changed is the power of the hardware, and therefore the amount of
rapid feedback we can get.

The IDE I am using compiles the system immediately when saving a source
file, and can be configured to run all the unit tests in background every
time a compile succeeds. Minutes after commiting my changes to CVS, I will
get an email wether the integration succeeded.

This makes a *big* difference, if we choose to use that power to our
advantage.

> Does anyone have any good numbers, from more recent studies? Wooly
> anyway?

I've heard XP teams reporting around two bugs reported from the customer in
six months.

Cheers, Ilja

Juhan Leemet

2004-08-14, 3:55 am

On Thu, 12 Aug 2004 15:41:14 -0400, John Roth wrote:
> "Juhan Leemet" <juhan@logicognosis.com> wrote in message
> news:pan.2004.08.12.18.20.57.825960@logicognosis.com...
[color=darkred]
>
> Actually, any period will not do. There is a growing body of evidence
> that the amount of trouble you have in integration is super-linear in
> the amount of time code has stayed out without being integrated. In
> other words, "integration Hell" is an easily predictable consequence of
> leaving integration until the end of the project.

OK, I mis-spoke. It's obviously not "any period" because the bean-counters
would then have us only do one build at the very end = cheapest. OTOH too
frequent can burn up resources running on a treadmill without much result.

Maybe I'm thinking of larger teams which have to be structured in groups.
In that case, I would prefer to have better defined deliverables,
periodically submitted into the system build. I don't believe that every
design (esp. redesign) can be completed within a day. Is that always true?
Personally, I think for a schedule that runs out years, one could have
full system integrations done, say, w

ly. The groups themselves might do
their own "integrations" against a stable baseline (working system) more
frequently.

As I write this, I realize that there is no point in continuing this. If
you're talking "XP speak" and I'm thinking concepts and terminology over
some decades, we're probably not communicating well at all.

> On the other side, daily is "white book" XP...
> The experiance in XP shops is that they almost never have integration
> problems, even though everyone is allowed (indeed encouraged) to change
> anything, anywhere in the code base that they need to for the current
> story.

I guess I'm uncomfortable with the notion that "everyone is encouraged to
change anything". Sounds too much like "programming with your spurs on". I
happen to think there should be some thought given to interfaces and they
should be protected "somewhat". If they can be defined (best) then they
should not be changed arbitrarily, but only after consultation and
thought. I like the idea that everyone is "deputized" to be on the lookout
for quality issues. None of that "not my problem" kind of attitude.
Everyone has to be engaged to deliver good product in a timely manner.

>
> If you're doing daily (or more frequent) integration / builds, then "it
> doesn't work" isn't a crisis - except for the dummy that broke the
> build. (The Microsoft principle - one of the few real crimes that their
> developers can commit is to break the build.)

That makes me really uneasy. Having worked with some M$ documentation on
their development tools and finding that they were only very loosely
cohesive (and incredibly mutable, between revisions, see the OS/2 SDKs).

You're talking out of both sides of your mouth here: on the one hand
"doesn't work isn't a crisis", but at the same time "real crime is
breaking the build". Which is it? One could go either way. I've worked
with "bully boy" management that "kills the messenger" (the M$ way?) and
that has literally turned out to be a bankrupt policy!

If the schedule depended on some functionality being available on a
certain date, and the build is busted, then obviously that functionality
is NOT available, and that deadline has been missed. Now whether that is
the "real deadling" or not... ask the executives and bean counters?

> As I said up front in my original response, I'm answering this question
> from an XP perspective. I'm not trying to be even-handed since I firmly
> believe that "standard software engineering" methodologies have
> demonstrable problems, both in effectiveness and efficiency.

I haven't worked with XP (that I know? maybe some similar principles?). I
suppose I should investigate it to some depth. I get a little put off by
the "golly gee" kind of evangelism that I seem to hear. OTOH, if those
techniques are good and they really work well, then we should all use them.

Personally, I believe a lot of the problems of "standard software
engineering" methodologies are more related to their implementations. I
remember it related that Gane & Sarson did 2 pilot projects of their
(first iteration?) SSADM. One was a success (those that grasped the
concepts and their purpose) and one was a total failure ("hostile" group?
just went through the motions, and generated useless "paperwork"). I'll
bet there are some duds out there that can make a total hash of XP, too.

If you never look at the bugs, it doesn't mean they are not being
generated. How many are really being fixed? I'd prefer to have some idea
of how many there are: say starting from a full system integration, not
the itty bitty ones that arise from the low level "fizz" of programming
and subsystem integration. How does XP deal with bugs/problems?
[color=darkred]
> See the comment in my original post. It's demonstrably possible to get
> an almost defect-free product because some teams are actually doing it.
> Customer visible defects are *expensive*! One of the rallying cries in
> manufacturing in the last several decades has been "Quality is free",
> meaning that increasing your quality has paybacks in terms of support
> and warrenty service, etc., that more than pay for the effort.

I've worked with a small team that delivered near perfect (3 "service
calls": 1 RTFM, 1 modification feasibilty question, and only 1 spelling
mistake bug, working at 400% load for years). The travesty was that
"management" did not recognize what they had and "refused" to reward this
form of behaviour. The project was canned. I believe the IP could have
been sold outright for something like $200K but that seemed to be too much
trouble?!? Others since have commented (incredulously) "that was too
good", perhaps hinting that they would have tried to screw our budget?

The point being that I think I understand what it takes. I've also worked
with teams that have recorded 1400 and 11000 bugs/issues during
integration. I can't remember how many of those "leaked out" into the
customer site/environment, maybe a third? half? There was a lot of yelling
and screaming. These same groups refused to believe there would be bugs.

>
> There is a naive approach that says: we can do anything we want, and
> it'll work out in the end. That demonstrably leads to crap.

I get the impression that the XP process does not record any problems.
Must start at some point? How do you know that you're not "going round in
circles"? Are "busted build" problems recorded? Only customer bugs?

>
> The only way you get quality is to pay thorough, detailed attention to
> quality in every aspect of the process...

When was that not true? I would suggest that those people who couldn't
make "waterfall" methodology work were just doing it really badly. Now
maybe XP is better (but not quite the same thing as a methodology)?
Whatever process you choose, you have to understand the purpose of every
thing you do and all "deliverables", otherwise you're just going through
the motions, and you'll get nowhere (good). The earliest proponents that I
met that wanted to do "spiral development" seemed to suggest that they
would just spin their wheels and we should be happy with whatever they
happen to produce. Bloody hell! Not on my dime. Call your shots.

> The notion that you will inevitably produce a defective product is one
> of the most pernicious beliefs in the industry today. It gets repeated
> ad nausium by the pundits...

I hope you were not interpreting my comments as an example? I think it is
important to understand what is going on in the process. If humans inject
some 50 bugs per KLOC then you need some methods and processes to take
them all out of there. I was reporting some "wishful thinking" kind of
"planning" that imagined "how bad could it be?" some 100 bugs? No sweat!
In actual fact, there could be tons of bugs. If you are not paying
proper attention to quality throughout your process, and you're not
finding and fixing those bugs, you'll get "buried" (in shit?).

I have a feeling that we're probably in "violent agreement", but maybe not
totally viz. methods. I don't believe "one size fits all": XP is the
"silver bullet"? I'm mindful of Brooks and others who warn against any
complacency, to think that you can find one thing that'll do it all.

>
> I'm not sure what you're saying here, other than that every process has
> known limitations. You cannot arbitrarilly specify scope, time, budget
> and quality, and expect that, by some miracle, it will be met. It
> doesn't work in building bridges, and it doesn't work in building
> software. That doesn't stop MBA trained managers from trying.

Again we're agreed. The "bully boy" executive mentality thinks that
software is infinitely mutable and therefore can be beaten out of people.

Can you suggest a good overview of XP, to put it into perspective for me?
I'm not sure I have time to read several books and do some projects before
I can make up my mind about it. I'm a little bit uneasy. Seems too facile.

--
Juhan Leemet
Logicognosis, Inc.

John Roth

2004-08-14, 3:57 pm

"Juhan Leemet" <juhan@logicognosis.com> wrote in message
news:pan.2004.08.14.01.19.02.689313@logicognosis.com...
> On Thu, 12 Aug 2004 15:41:14 -0400, John Roth wrote:
>
>
> OK, I mis-spoke. It's obviously not "any period" because the bean-counters
> would then have us only do one build at the very end = cheapest. OTOH too
> frequent can burn up resources running on a treadmill without much result.

One integration at the end generally leads to "integration hell."
The fact is that you shouldn't even think of optimizing parts of
a process until the entire process is working to your satisfaction.
The bean counter that comes in and makes changes to pieces
of the process without looking at the entire process is a menace,
and should be fired promptly.

Well, current XP practice is to integrate every twenty to thirty
minutes! What can I say - it works and if you tend it properly
it doesn't use that much resource. Remember that this is in
an environment where you're probably running several hundred
unit tests after *every* edit, and you've spent a significant amount
of effort to make sure that the collection of unit tests can run in
under a minute. It's not the same process.

> Maybe I'm thinking of larger teams which have to be structured in groups.
> In that case, I would prefer to have better defined deliverables,
> periodically submitted into the system build.

Delayed integration is poison. If you're working on separate
components, you still need to deal with the fact that your
interface will change just as everything else will.

> I don't believe that every
> design (esp. redesign) can be completed within a day. Is that always true?

Major redesigns take time. It takes some skill to fit them into a
flow and still keep delivering value while implementing the redesign.

> Personally, I think for a schedule that runs out years, one could have
> full system integrations done, say, w

ly. The groups themselves might do
> their own "integrations" against a stable baseline (working system) more
> frequently.
>
> As I write this, I realize that there is no point in continuing this. If
> you're talking "XP speak" and I'm thinking concepts and terminology over
> some decades, we're probably not communicating well at all.

I think we're communicating quite well. You've realized that I have
a very different model of how effective software development works
than you do; that's the first step to starting to understand the differences
and how that affects details.

>
> I guess I'm uncomfortable with the notion that "everyone is encouraged to
> change anything". Sounds too much like "programming with your spurs on".

Yeah, it does sound somewhat like "cowboy coding" doesn't it? The
control is that changes have to be done in very small increments - five
lines in an edit before running your test suite is average. And you have
to make that change work before moving on. You can't leave broken
unit tests behind you, and you can't remove existing tests because "I
can't make that work right now." All of those things are symptoms of
cowboy coders, and in a well functioning XP team, they are very offensive.

> I happen to think there should be some thought given to interfaces and
they
> should be protected "somewhat". If they can be defined (best) then they
> should not be changed arbitrarily, but only after consultation and
> thought.

Programming without thinking about design is a disaster. So is
programming without consultation - that's why pair programming
is a core practice. If you don't know an area, pair with someone
who does.

> I like the idea that everyone is "deputized" to be on the lookout
> for quality issues. None of that "not my problem" kind of attitude.
> Everyone has to be engaged to deliver good product in a timely manner.

Exactly. Can't be emphasized too strongly.

>
> That makes me really uneasy. Having worked with some M$ documentation on
> their development tools and finding that they were only very loosely
> cohesive (and incredibly mutable, between revisions, see the OS/2 SDKs).

Well, I didn't say that the entire M$ process was worthy
of emulation - just that their practice of doing integration,
build, test and package daily was probably the only reason
they manage to get anything out the door.

> You're talking out of both sides of your mouth here: on the one hand
> "doesn't work isn't a crisis", but at the same time "real crime is
> breaking the build". Which is it? One could go either way. I've worked
> with "bully boy" management that "kills the messenger" (the M$ way?) and
> that has literally turned out to be a bankrupt policy!

Bad management is bad management. What more can be said,
unless you want to categorize the ways that management can
be bad.

If you're doing frequent integrations, the failure of a single integration
isn't a crisis because it can be fixed quickly. There isn't that much that
went into it, so tracking down the problem is usually quite easy. The
less frequent the integrations, the more goes into them, and the
more problems surface. Since there were more changes, each of
the problems is correspondingly harder to track down. In other
words, the difficulty in integration is proportional to the square
of the time between integrations (measured in terms of changed
(new, deleted) loc.

> If the schedule depended on some functionality being available on a
> certain date, and the build is busted, then obviously that functionality
> is NOT available, and that deadline has been missed. Now whether that is
> the "real deadling" or not... ask the executives and bean counters?

That's only true if 1) you're doing "big bang" integration at the
end of the project, or 2) it's the last integration before the target
date. Again, if you're doing very frequent small integrations, it's
not that hard to find and fix it.

Another issue here is 'end of project variance." The size of problems
and the amount of time to fix them is not proportional to the amount
of time remaining - you can have a large problem right at the end.

This is also where the general Agile practice of scheduling the
highest business value functionality first comes into play. If you
run into a problem right before the hard ship date, by definition
it's the least valuable functionality that was scheduled for that
release, so it's a viable choice to fall back to the last complete
build, test and package that actually worked. That is, of course,
a business decision, and it's critically dependent on the output
of each iteration really, truely being production quality without
needing additional after the fact testing and bug fixing.

>
>
> I haven't worked with XP (that I know? maybe some similar principles?). I
> suppose I should investigate it to some depth. I get a little put off by
> the "golly gee" kind of evangelism that I seem to hear. OTOH, if those
> techniques are good and they really work well, then we should all use
them.

A lot of people seem to be put off by the level of evangalism.
The other problem is that it's different enough from standard
software development methodology that people have a real
hard time getting theirm minds around it at first look.

> Personally, I believe a lot of the problems of "standard software
> engineering" methodologies are more related to their implementations. I
> remember it related that Gane & Sarson did 2 pilot projects of their
> (first iteration?) SSADM. One was a success (those that grasped the
> concepts and their purpose) and one was a total failure ("hostile" group?
> just went through the motions, and generated useless "paperwork"). I'll
> bet there are some duds out there that can make a total hash of XP, too.

Oh, absolutely.

>
> If you never look at the bugs, it doesn't mean they are not being
> generated. How many are really being fixed? I'd prefer to have some idea
> of how many there are: say starting from a full system integration, not
> the itty bitty ones that arise from the low level "fizz" of programming
> and subsystem integration. How does XP deal with bugs/problems?

It may surprise you that the "hard" answer is that you shouldn't have
bugs on the level you're describing. That's not the same as don't, of
course. The thing to remember is that an XP project is integrating
continuously, and running the full test suite every night. All unit tests
have to run 100%, and all previously passing acceptance tests have
to continue to pass. If not, fixing it is the first order of business the
next
morning. If you want to be real hard-ass about it, you'll also run
coverage monitors to insure that you have 100% statement and
branch coverage over the unit tests. (This isn't real difficult: TDD
normally delivers statement coverage in the high 90s, and branch
coverage in the mid 90s, so getting to 100% is easy, at least if you
keep at it. It becomes easier over time as the developers learn what
they need to pay attention to). This catches the "cowboy coder" who
might like to remove a unit test because he can't make it pass.

If you can get there (and it isn't easy) then all you've got are
field reported bugs, and you schedule those the same way
you do stories. Having to have a separate bug data base is a
process smell.

>
> I've worked with a small team that delivered near perfect (3 "service
> calls": 1 RTFM, 1 modification feasibilty question, and only 1 spelling
> mistake bug, working at 400% load for years). The travesty was that
> "management" did not recognize what they had and "refused" to reward this
> form of behaviour. The project was canned. I believe the IP could have
> been sold outright for something like $200K but that seemed to be too much
> trouble?!? Others since have commented (incredulously) "that was too
> good", perhaps hinting that they would have tried to screw our budget?

Bad management is bad management.

> The point being that I think I understand what it takes. I've also worked
> with teams that have recorded 1400 and 11000 bugs/issues during
> integration. I can't remember how many of those "leaked out" into the
> customer site/environment, maybe a third? half? There was a lot of yelling
> and screaming. These same groups refused to believe there would be bugs.

Yup. As I've mentioned previously, the amount of difficulty you have
in integration is proportional to the square of the number of LOC you're
trying to integrate. I won't go so far as to say you absolutly have to do
continuous integration XP-style, but anyone who designs a project
with "big bang" integration should be sacked.

>
> I get the impression that the XP process does not record any problems.
> Must start at some point? How do you know that you're not "going round in
> circles"? Are "busted build" problems recorded? Only customer bugs?

Well, one of the fundamental XP processes is the executable
acceptance test. This must be in existance before working on
the story (test first!). Teams work to make the acceptance tests
pass (the acceptance test is the requirment). Defects that leak
through that indicate that the acceptance tests are insufficient,
which comes down to how well the requirements analysts are
cooperating with the QA analysts.

XP also believes in appropriate measurements. If you're not
getting a significant number of defects, then process measurements
are a waste of time. If you are, then the process needs to be
inspected with some diligence and reworked until you are
no longer getting defects.

>
> When was that not true? I would suggest that those people who couldn't
> make "waterfall" methodology work were just doing it really badly.

We need to distinguish between a strict phasist waterfall and the way
standard software engineering practice does it. The first is a recognized
prescription for disaster except in very restricted circumstances.

Project managers who can make the second work on a regular basis
have my deepest respect. They are rare, but they do exist.

> Now maybe XP is better (but not quite the same thing as a methodology)?

Well, it's different. It puts a very high value on absolutely minimizing
work in process, all the way from requirements through final packaging.
The less WIP, the less possibility you have for stuff you think is complete
but has lurking problems.

> Whatever process you choose, you have to understand the purpose of every
> thing you do and all "deliverables", otherwise you're just going through
> the motions, and you'll get nowhere (good).

Yep. Some of them aren't exactly obvious in XP, though, which is
a problem.

> The earliest proponents that I
> met that wanted to do "spiral development" seemed to suggest that they
> would just spin their wheels and we should be happy with whatever they
> happen to produce. Bloody hell! Not on my dime. Call your shots.

Yep. While it's possible to deliver less than you promised in XP,
it's the second last resort (the last resort is extending the deadline.)
Either is a symptom of something really wrong with the process
(or a developer suddenly being off sick, of course!)

>
> I hope you were not interpreting my comments as an example? I think it is
> important to understand what is going on in the process. If humans inject
> some 50 bugs per KLOC then you need some methods and processes to take
> them all out of there. I was reporting some "wishful thinking" kind of
> "planning" that imagined "how bad could it be?" some 100 bugs? No sweat!
> In actual fact, there could be tons of bugs. If you are not paying
> proper attention to quality throughout your process, and you're not
> finding and fixing those bugs, you'll get "buried" (in shit?).

If your process produces defects, that's a comment on your process,
not on the nature of reality. One of the few places where I think
that manufacturing production experiance is applicable to software
development is this precise attitude: defects are unacceptable, and
you need to fix the process until it does not produce them. You
should not need end of the line testing except to certify the process.

I have nothing but sympathy for managers that refuse to believe that
your process is going to produce 50 defects per KLOC. What I
find hard to believe is that they simply don't come out and say that
is absolutely unacceptable, and fix your process so it doesn't do
that.

There was a recent talk at Atlanta SPIN (Dec 2003 meeting)

http://www.atlantaspin.org/Archives.html

by Iraj Hirmanpour from the
S.E.I. He had a couple of real eye-opening slides on the difference
in effectiveness between CMM Level 5 organizations and organizations
that had implemented PSP and TSP in the context of an otherwise
standard software engineering process.

I'd have to go back to the presentation, but I believe that the
difference was around two orders of magnitude fewer defects
for the PSP and TSP shops. And that is from the S.E.I, not from
some wild-eyed off-brand process fanatic.

> I have a feeling that we're probably in "violent agreement", but maybe not
> totally viz. methods. I don't believe "one size fits all": XP is the
> "silver bullet"? I'm mindful of Brooks and others who warn against any
> complacency, to think that you can find one thing that'll do it all.

XP certainly isn't a silver bullet. There is no such thing.

>
> Again we're agreed. The "bully boy" executive mentality thinks that
> software is infinitely mutable and therefore can be beaten out of people.
>
> Can you suggest a good overview of XP, to put it into perspective for me?
> I'm not sure I have time to read several books and do some projects before
> I can make up my mind about it. I'm a little bit uneasy. Seems too facile.

Unfortunately, someone else is going to have to do that. I'm one of the
loudmouths on the XP mailing list, and it's really been a while since I
evaluated any of the recent books. The White Book (XP Explained)
is four years old now, and some parts are quite outdated. One book
I'd heartily recommend (although it's not on XP per se) is Lean
Software Development by Mary and Tom Poppendieck. That
gives a lot of process tools without recommending a specific process.

One of the things to watch out for in any XP presentation is that it
is going to take a significant learning curve. Test Driven Development,
emergent design and refactoring are not things that people learn overnight.
You have to find a personal (or team) comfort point between explicit
design and emergent design, and that is going to take time to learn.
You also need to find a balance between depending on the unit
tests and code for design documentation, and where you absolutely
do need some ancilliary documentation.

Executable Acceptance Tests not only take some learning, but there
are specific architectural motifs you have to use for user interface
testing, and that doesn't come overnight either.

John Roth
>
> --
> Juhan Leemet
> Logicognosis, Inc.
>

Ronald E Jeffries

2004-08-14, 3:57 pm

On Fri, 13 Aug 2004 23:19:05 -0200, Juhan Leemet
<juhan@logicognosis.com> wrote:

>On Thu, 12 Aug 2004 15:41:14 -0400, John Roth wrote:
>
>
>OK, I mis-spoke. It's obviously not "any period" because the bean-counters
>would then have us only do one build at the very end = cheapest. OTOH too
>frequent can burn up resources running on a treadmill without much result.

Well, um, I must respectfully disagree. Integrating only at the end is
not in fact cheapest. That integration will take infinitely long, and
every defect put into the system over the course of the whole project
will still be there.

As for too frequent, it depends. If the integration is fully automated
-- and if we're doing it frequently at all wouldn't it be -- then its
just burning integration machine cycles which would otherwise be spent
in the idle loop. And the builds would be throwing off information. So
it might not be wasteful at all.
>
>Maybe I'm thinking of larger teams which have to be structured in groups.
>In that case, I would prefer to have better defined deliverables,
>periodically submitted into the system build. I don't believe that every
>design (esp. redesign) can be completed within a day. Is that always true?
>Personally, I think for a schedule that runs out years, one could have
>full system integrations done, say, w

ly. The groups themselves might do
>their own "integrations" against a stable baseline (working system) more
>frequently.

What size projects are you working on?
>
>As I write this, I realize that there is no point in continuing this. If
>you're talking "XP speak" and I'm thinking concepts and terminology over
>some decades, we're probably not communicating well at all.

You certainly may withdraw on that basis or any other. As for decades,
I wrote my first computer program for pay in the summer of 1961. So
while I bring an XP attitude to the situation, I also bring quite a
few other things I've learned along the way. I might have a favorite
approach, but it's not for want of having tried no others.
>
>I guess I'm uncomfortable with the notion that "everyone is encouraged to
>change anything". Sounds too much like "programming with your spurs on". I
>happen to think there should be some thought given to interfaces and they
>should be protected "somewhat". If they can be defined (best) then they
>should not be changed arbitrarily, but only after consultation and
>thought. I like the idea that everyone is "deputized" to be on the lookout
>for quality issues. None of that "not my problem" kind of attitude.
>Everyone has to be engaged to deliver good product in a timely manner.

The XP notion that triggered the "everyone is encouraged to change
anything" includes everyone on the project working together in a
single workspace, on a single code base, and programming in pairs.

In a less integrated team, I might share the concerns you raise as
well.

>I haven't worked with XP (that I know? maybe some similar principles?). I
>suppose I should investigate it to some depth. I get a little put off by
>the "golly gee" kind of evangelism that I seem to hear. OTOH, if those
>techniques are good and they really work well, then we should all use them.

Yes, you get to decide what you want to learn about. I applaud the
notion of looking into it before dismissing it.
>
>Personally, I believe a lot of the problems of "standard software
>engineering" methodologies are more related to their implementations. I
>remember it related that Gane & Sarson did 2 pilot projects of their
>(first iteration?) SSADM. One was a success (those that grasped the
>concepts and their purpose) and one was a total failure ("hostile" group?
>just went through the motions, and generated useless "paperwork"). I'll
>bet there are some duds out there that can make a total hash of XP, too.

Sure. It all comes down to people.
>
>If you never look at the bugs, it doesn't mean they are not being
>generated. How many are really being fixed? I'd prefer to have some idea
>of how many there are: say starting from a full system integration, not
>the itty bitty ones that arise from the low level "fizz" of programming
>and subsystem integration. How does XP deal with bugs/problems?

There are two levels of tests, programmer tests and customer
acceptance tests. Programmer tests must run perfectly all the time.
Really.

Customer tests are (a) defined by the customer, so they serve as
acceptance tests. Team practices vary but the better practice here
seems to be keeping them running perfectly as well.

Whenever a defect shows up in spite of all the tests, what I do is to
write a customer test that shows the defect, then a programmer test
that shows the defect, then fix the defect. Then I reflect on what
other tests I might be missing, based on what I just learned, and
begin getting those written as well.

The intention that I hold is to have zero defects. Whenever I know of
a defect, I go right after it. One might think that one would make no
progress with this attitude. But what really happens is that one makes
very good progress and it's not taintedas much by defects that will
show up later.
>
>I've worked with a small team that delivered near perfect (3 "service
>calls": 1 RTFM, 1 modification feasibilty question, and only 1 spelling
>mistake bug, working at 400% load for years). The travesty was that
>"management" did not recognize what they had and "refused" to reward this
>form of behaviour. The project was canned. I believe the IP could have
>been sold outright for something like $200K but that seemed to be too much
>trouble?!? Others since have commented (incredulously) "that was too
>good", perhaps hinting that they would have tried to screw our budget?
>
>The point being that I think I understand what it takes. I've also worked
>with teams that have recorded 1400 and 11000 bugs/issues during
>integration. I can't remember how many of those "leaked out" into the
>customer site/environment, maybe a third? half? There was a lot of yelling
>and screaming. These same groups refused to believe there would be bugs.

Yes. Rewards notwithstanding, I prefer to work in the former
situation. And you?
>
>I get the impression that the XP process does not record any problems.
>Must start at some point? How do you know that you're not "going round in
>circles"? Are "busted build" problems recorded? Only customer bugs?

There are no mandatory recording requirements in XP. Since the team
works closely together, and since they fix problems /immediately/,
most of the learning is immediate as well.

Individual teams track and record the information that they find
useful. I like to record huge volumes of raw data: every single test
result, every single code checkin. Then, if I get interested in some
topic, I'll write an analysis that scans back over all the data and
asks about whatever I'm interested in. We might start thinking that
Feature XYZ breaks a lot. So we scan the test results and discover
that it and feature ABC are about tied for first place, with ten times
more breakages than any other feature. Hmm ... let's look into that.

But I trust that the team will get the sense of any trend by paying
attention much faster and better than by looking at a lot of stats.
>
>When was that not true? I would suggest that those people who couldn't
>make "waterfall" methodology work were just doing it really badly. Now
>maybe XP is better (but not quite the same thing as a methodology)?
>Whatever process you choose, you have to understand the purpose of every
>thing you do and all "deliverables", otherwise you're just going through
>the motions, and you'll get nowhere (good). The earliest proponents that I
>met that wanted to do "spiral development" seemed to suggest that they
>would just spin their wheels and we should be happy with whatever they
>happen to produce. Bloody hell! Not on my dime. Call your shots.

Well, there are, in my opinion, some fundamental reasons why waterfall
methods will be inferior to agile methods in /any/ situation where
both are applicable.

Whether agile methods are always applicable is another question.
>
>I have a feeling that we're probably in "violent agreement", but maybe not
>totally viz. methods. I don't believe "one size fits all": XP is the
>"silver bullet"? I'm mindful of Brooks and others who warn against any
>complacency, to think that you can find one thing that'll do it all.

I don't recall anyone mentioning that XP was a silver bullet. However,
if one looks at the principles and values and practices, one is likely
to find things of value. Even the famous Mr Ed is now reduced to
objecting to XP on the grounds that "this isn't new." We didn't say it
was new. We said it was good. Check it out, it might be good. :)
>

So true![color=darkred]
>
>Can you suggest a good overview of XP, to put it into perspective for me?
>I'm not sure I have time to read several books and do some projects before
>I can make up my mind about it. I'm a little bit uneasy. Seems too facile.

Everything is facile in a newsgroup.

Try www.extremeprogramming.org for one angle. Try chromatic's extreme
programming book for brief. Try Kent Beck's white book for the
original source. Try the pink book (Jeffries, Anderson, Hendrickson)
for details of how you do it.

Or, if you'd like to start with a single technique, Beck's, or Astels'
books on Test-Driven Development are nice. But they're not overviews
of XP.

Regards,

--
Ron Jeffries
www.XProgramming.com
I'm giving the best advice I have. You get to decide if it's true for you.

lilburne

2004-08-14, 3:57 pm

Juhan Leemet wrote:
>
> OK, I mis-spoke. It's obviously not "any period" because the bean-counters
> would then have us only do one build at the very end = cheapest. OTOH too
> frequent can burn up resources running on a treadmill without much result.
>
> Maybe I'm thinking of larger teams which have to be structured in groups.
> In that case, I would prefer to have better defined deliverables,
> periodically submitted into the system build. I don't believe that every
> design (esp. redesign) can be completed within a day. Is that always true?
> Personally, I think for a schedule that runs out years, one could have
> full system integrations done, say, w

ly. The groups themselves might do
> their own "integrations" against a stable baseline (working system) more
> frequently.
>

The project I work on is officially built and released twice
a w

. IOW the codebase changes tick-tock, tick-tock,
tick-tock. The cardinal sin for a developer is to break the
toplevel. Tick-tock, tick-tock, tick-tock. Twice a month all
the libraries shared across all products are built and
released tick-tock, tick-tock, tick-tock. All the products
are rebuilt using the the new libraries, tick-tock,
tick-tock, tick-tock.

After each build the full suite of tests are run on all
applications. We have dedicated build machines for this.

That's the official cycle. Underneath that developers submit
their code changes to the test system. The system checks
integrates the individual changes builds the application and
the unit test harness, and then runs all the unit tests. At
days end the system runs all the application regression
tests on the code changes submitted that day.

Beneath that the developers build their own project version
of the application and run the application tests at the end
of each day on their work in progress. The unit tests tend
to be run all the time as they develop code.

Individual projects can take 1 day - 6 months to complete.
Though normally our developers make regular submissions of
work back into the top level.

As each new codebase is created developers can run a
"freshen project" script that brings their project up to
date with the current toplevel codebase.

> As I write this, I realize that there is no point in continuing this. If
> you're talking "XP speak" and I'm thinking concepts and terminology over
> some decades, we're probably not communicating well at all.
>

Well the XP enthusiast here have told me that because our
system is over 12 years old what we do isn't XP.

>
>
> I guess I'm uncomfortable with the notion that "everyone is encouraged to
> change anything". Sounds too much like "programming with your spurs on". I
> happen to think there should be some thought given to interfaces and they
> should be protected "somewhat". If they can be defined (best) then they
> should not be changed arbitrarily, but only after consultation and
> thought. I like the idea that everyone is "deputized" to be on the lookout
> for quality issues. None of that "not my problem" kind of attitude.
> Everyone has to be engaged to deliver good product in a timely manner.
>

I have no problem with someone changing whatever code they
need to so long as:

1) They publish what they are about to do, or speak to the
code's guardian.
2) That their changes do not break any of the
application tests.
3) That their changes do not break the integration builds.

IOW prior consultation, responsibilty for the consequencies,
and don't XXXX up the APIs for the other products.

If they do 1) then 2 and 3 don't usually amount to much.

>
> You're talking out of both sides of your mouth here: on the one hand
> "doesn't work isn't a crisis", but at the same time "real crime is
> breaking the build". Which is it? One could go either way. I've worked
> with "bully boy" management that "kills the messenger" (the M$ way?) and
> that has literally turned out to be a bankrupt policy!

Basically in a properly functioning system there is no
reason for a developer to break the build. The build gets
broke if you submit code without having compiled it first,
or without having ensured that all the tests pass. These are
checks that you ought to do (he says being notorious for
'breaking the build').

However, if the build is broken it is fairly obvious what
broke it, and is either simple to fix or the code change can
be backed out.

> If the schedule depended on some functionality being available on a
> certain date, and the build is busted, then obviously that functionality
> is NOT available, and that deadline has been missed. Now whether that is
> the "real deadling" or not... ask the executives and bean counters?
>

Functionality shouldn't be being submitted just before a
deadline.

CTips

2004-08-14, 3:57 pm

Andrew wrote:
> Hi there,
>
> Thanks to all for your feedback on my previous question on unit
> testing.
>
> I want to step back and ask the group a broader question. Imagine
> you are the VP Engineering at a large software company with the
> mandate that you must deliver software at the absolute lowest possible
> total cost of development and deployment - that means, low cost to
> engineer but with high enough quality that post-release deployment
> costs are also low (limited bugs, etc).

I am the equivalent, though of the software side of a smallish
hardware/software startup. Delivering high function, low bug-count
software with a low time to release is paramount for us

>
> 1) What are the top 3-5 problems or issues be in your experience that
> would strain my ability to be low cost AND high quality?

The ability to find good people. This breaks down into 2 parts
- Finding people who have the right mixture of theory & practice [we
expect people to be skilled C systems programmers, be able to develop
new algorithms and be familiar with the literature in the systems
domain. It also helps if they have skills in at least one other domain].
- Convincing them to re-locate to our area [most of them are pretty
happy where they are!]
I suspect if we were based out of Boston, Austin or the SF Bay area,
some of this problem might go away. Salary, interesting work, work
conditions have never been a concern, that I can think of.

Note that this is predicated on the model which we adopted - we would
only hire a few people, and we would only hire people who we thought
were exceptional (That way, at worst, they would turn out to be merely
good).

Other people make the decision to go with hiring across the skill
spectrum, and then putting processes/tools in place to compensate. Their
answers will be considerably different.

> 1a) How do people resolve these 3-5 problems? Tools? Software
> lifecyle methodologies? Which of these tools and methods can be
> provably shown to mitigate those 3-5 problems or is it just
> statistical/anecdotal suggestion?

We try and make the environment as productive as possible for people.
* Tele-commuting is encouraged - at this point, people are expected to
be in Mon & Wed so that everyone is available in case we need to meet
and discuss things, but even thats not hard and fast.
* Everybody gets a high-performance laptop [e.g. IBM Thinkpad a31 with
max memory]
* People get handed big chunks of work to do - where "do" means
everything including design, architect, implement, and testing.
* We try and make sure that everyone gets to get an interesting project
every so often - specially if someones burning out, or just plain bored
with his current work.
* We pay above market

We had a new hire who had a baby - he came into work 3 days after the
event. We told him to go home and not come back for a couple of w

s
[He was walked to the door by his manager within a couple of hours of it
coming to our attention].

>
> 2) What are the top 3-5 risks (i.e. not guaranteed to be a problem,
> but could manifest as a problem) that would strain low cost/high
> ability?

We have low coverage. Typically, for any program, we have one person
whose working on it, with one person who could take over. We lose both
of them and we could be hurting [though this is compensated somewhat by
the fact that we usually have about 3 months of slack in our schedules]

> 2a) How do people prevent those risks from arising? Tools?
> Methodologies? Can the tools and methods be provably shown to
> suppress the risks or is it statistical/anecdotal suggestion?

Make sure that your people are happy. That way they won't leave.
Make sure that they know that they won't be hurt if they look elsewhere
for alternatives [after all, we are a startup with the usual
ups-and-downs]. That way you won't be blindsided if they actually leave.

>
> 3) What are the top 3-5 decision tradeoffs that a VP Engineering has
> to make to deliver software - i.e. not problems here, but just
> decisions that have to be made between deliving in one way vs.
> another?

We have to take into account the following:
- how important is it to our customer(s)
- who all can do the job
- who all are available (or when will they become available)
- who all want to do the job
- how long will it take each of them.

> 3a) How do people determine the optimal decision for those tradeoffs?
>
>
> Any input on these questions - even if only for a subsection would be
> greatly appreciated. I hope it actually spurns some good discussionn
> because my guess is that different people have different opinions.

Phlip

2004-08-14, 3:57 pm

lilburne wrote:

> The project I work on is officially built and released twice
> a w

. IOW the codebase changes tick-tock, tick-tock,
> tick-tock. The cardinal sin for a developer is to break the
> toplevel. Tick-tock, tick-tock, tick-tock. Twice a month all
> the libraries shared across all products are built and
> released tick-tock, tick-tock, tick-tock. All the products
> are rebuilt using the the new libraries, tick-tock,
> tick-tock, tick-tock.
>
> After each build the full suite of tests are run on all
> applications. We have dedicated build machines for this.
>
> That's the official cycle. Underneath that developers submit
> their code changes to the test system. The system checks
> integrates the individual changes builds the application and
> the unit test harness, and then runs all the unit tests. At
> days end the system runs all the application regression
> tests on the code changes submitted that day.
>
> Beneath that the developers build their own project version
> of the application and run the application tests at the end
> of each day on their work in progress. The unit tests tend
> to be run all the time as they develop code.
>
> Individual projects can take 1 day - 6 months to complete.
> Though normally our developers make regular submissions of
> work back into the top level.
>
> As each new codebase is created developers can run a
> "freshen project" script that brings their project up to
> date with the current toplevel codebase.
>
[color=darkred]
> Well the XP enthusiast here have told me that because our
> system is over 12 years old what we do