Question on the COCOMO model

The company I work for, believe it or not, is just now getting into
formal methods for software development cost estimation.  And as is
unfortunately typical, the younger engineers want to start from
scratch in devising methods for software sizing and cost estimation.
I, on the other hand, having had significant experience with cost
estimation with a former employer, but several years ago, feel there
is an enormous amount of knowledge and insight in the cost estimation
models that have been around for years.  I feel we should look at what
is on the shelf, pick one, use it while we build our own statistical
data base and eventually use it to refine our own approach.  COCOMO
happens to be the model that I am familiar with.  Barry Boehm used to
be a god to me, although I guess his work is somewhat dated now
despite earnest attempts to keep it current.  One of the engineers who
wants to start from scratch tells me he looked at material on the USC
web site and found the following (which I verified does exist):

"We note that the predictive performance of the Bayesian approach
(i.e. COCOMO II.1998) is significantly better than that of the
multiple regression approach (i.e. COCOMO II.1997). COCOMO II.1998
gives predictions that are within 30% of the actuals 71% of the time
where as COCOMO II.1997 gives predictions within 30% of the actuals
only 52% of the time."

Not being the statistician I'd like to be, I'm not sure how to
interpret the above.  It does sound pretty bad to my uninformed ears.
30% of actual???  But I'm wondering what someone with more expertise
might say about this.  If there is anyone out there who would like to
comment, I'm all ears.

Thanks.

"TerryG" <ritpg@hotmail.com> wrote in message
news:7322e1ab.0404091757.5a05e193@posting.google.com... 
> "We note that the predictive performance of the Bayesian approach
> (i.e. COCOMO II.1998) is significantly better than that of the
> multiple regression approach (i.e. COCOMO II.1997). COCOMO II.1998
> gives predictions that are within 30% of the actuals 71% of the time
> where as COCOMO II.1997 gives predictions within 30% of the actuals
> only 52% of the time."
>

Just to make sure we're reading this the same way, I see the operative term
as 'within 30% of actual' as meaning 'actual plus or minus 30%'.  71% of the
time.

I'd want to see if the study being quoted had more detail (what % of the
time was
the estimate within 5%, 10% etc.) I'd also ask the smartypants 'how' he
plans to beat
that performance.

Unless you have a lot of historical data just lying around, I don't know how
you
could develop more accurate estimates.

"TerryG" <ritpg@hotmail.com> wrote in message
news:7322e1ab.0404091757.5a05e193@posting.google.com...
> multiple regression approach (i.e. COCOMO II.1997). COCOMO II.1998
> gives predictions that are within 30% of the actuals 71% of the time
> where as COCOMO II.1997 gives predictions within 30% of the actuals
> only 52% of the time."
>
> Not being the statistician I'd like to be, I'm not sure how to
> interpret the above.  It does sound pretty bad to my uninformed ears.
> 30% of actual???  But I'm wondering what someone with more expertise
> might say about this.  If there is anyone out there who would like to
> comment, I'm all ears.

Scott's reply has the correct interpretation of the sentence.

FYI, a few ws ago, at Dr. Boehm's USC Center for Software Engineering,
there was a report on the latest COCOMO II calibration effort that's in
progress.  Preliminary indications are that it will be much better than the
current calibration.

But the best way to use a tool such as COCOMO, is to do a "local
calibration".  That's simply a matter of fitting an equation to the data
you've collected for your organization.  You can start to do that with a
(bare) minimum of 5 completed projects.  You can download a free calibration
tool from our website:
http://www.softstarsystems.com

I gave a talk about local calibrations at one of the annual COCOMO meetings:
http://www.softstarsystems.com/calibrat.ppt

For more information on the calibration of the COCOMO II model see "Software
Cost Estimation with COCOMO II" (Boehm et al).

Dan Ligett, Softstar Systems, (603) 672-0987

On Sat, 10 Apr 2004 11:07:29 -0400, "Dan Ligett"
<ligett@softstarsystems.com> wrote:
> FYI, a few ws ago, at Dr. Boehm's USC Center for Software Engineering,
> there was a report on the latest COCOMO II calibration effort that's in
> progress.  Preliminary indications are that it will be much better than th
e
> current calibration.

Well, well well. I thought COCOMO was dead, I heard about it at
uni' in the mid 80's and then used it in a tral in the early 90s.
But I'd heard nothing about it since so assumed it had just
quietly died a death. I must pay a visit.

> (bare) minimum of 5 completed projects.  You can download a free calibrati
on
> tool from our website:
>     http://www.softstarsystems.com

I hope the tools have moved on from the rather poor MS-DOS based
ones I used?

Alan G.
Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld

> Well, well well. I thought COCOMO was dead, I heard about it at
> uni' in the mid 80's and then used it in a tral in the early 90s.
> But I'd heard nothing about it since so assumed it had just
> quietly died a death. I must pay a visit.

COCOMO is very healthy.  The 19th COCOMO meeting will be held in LA in
October:
http://sunset.usc.edu/events/upcoming/index.html

> I hope the tools have moved on from the rather poor MS-DOS based
> ones I used?

Ummmm... as co-author of the free MS-DOS WICOMO tool (vintage 1982) and
author of Costar (version 1.0 was on MS-DOS), I think my feelings are hurt
;).  I thought they were *rather good* MS-DOS tools... but they were MS-DOS
tools.

And yes, Costar 7.0 is windows based.  Our Sun and VMS versions weren't very
popular, so only the windows version is available now'days.  You can
download a demo of our COCOMO tool here:
http://www.softstarsystems.com

Dan

I've always interpreted this to mean that the 'actuals' are the
numbers in their (large) project library. If this is the case then
the *predictive* performance may be worse.

You mentioned 'formal methods' below. In software engineering,
'formal methods' usually refers to methods which use mathematical or
formal logical approaches (eg Z) to specification and design - and
this is probably not what you meant. (eg see
http://eis.jpl.nasa.gov/quality/Formal_Methods/)

Andrew
--
Andrew Gabb
email: agabb@tpgi.com.au       Adelaide, South Australia
phone: +61 8 8342-1021, fax: +61 8 8269-3280
-----


TerryG wrote:

> The company I work for, believe it or not, is just now getting into
> formal methods for software development cost estimation.  And as is
> unfortunately typical, the younger engineers want to start from
> scratch in devising methods for software sizing and cost estimation.
> I, on the other hand, having had significant experience with cost
> estimation with a former employer, but several years ago, feel there
> is an enormous amount of knowledge and insight in the cost estimation
> models that have been around for years.  I feel we should look at what
> is on the shelf, pick one, use it while we build our own statistical
> data base and eventually use it to refine our own approach.  COCOMO
> happens to be the model that I am familiar with.  Barry Boehm used to
> be a god to me, although I guess his work is somewhat dated now
> despite earnest attempts to keep it current.  One of the engineers who
> wants to start from scratch tells me he looked at material on the USC
> web site and found the following (which I verified does exist):
>
> "We note that the predictive performance of the Bayesian approach
> (i.e. COCOMO II.1998) is significantly better than that of the
> multiple regression approach (i.e. COCOMO II.1997). COCOMO II.1998
> gives predictions that are within 30% of the actuals 71% of the time
> where as COCOMO II.1997 gives predictions within 30% of the actuals
> only 52% of the time."
>
> Not being the statistician I'd like to be, I'm not sure how to
> interpret the above.  It does sound pretty bad to my uninformed ears.
> 30% of actual???  But I'm wondering what someone with more expertise
> might say about this.  If there is anyone out there who would like to
> comment, I'm all ears.
>
> Thanks.

"Dan Ligett" <ligett@softstarsystems.com> wrote in message news:<c592ja$pch$1@pyrite.mv.net
>...
> "TerryG" <ritpg@hotmail.com> wrote in message
> news:7322e1ab.0404091757.5a05e193@posting.google.com... 
>
> Scott's reply has the correct interpretation of the sentence.
>
> FYI, a few ws ago, at Dr. Boehm's USC Center for Software Engineering,
> there was a report on the latest COCOMO II calibration effort that's in
> progress.  Preliminary indications are that it will be much better than th
e
> current calibration.
>
> But the best way to use a tool such as COCOMO, is to do a "local
> calibration".  That's simply a matter of fitting an equation to the data
> you've collected for your organization.  You can start to do that with a
> (bare) minimum of 5 completed projects.  You can download a free calibrati
on
> tool from our website:
>     http://www.softstarsystems.com
>
> I gave a talk about local calibrations at one of the annual COCOMO meeting
s:
>     http://www.softstarsystems.com/calibrat.ppt
>
> For more information on the calibration of the COCOMO II model see "Softwa
re
> Cost Estimation with COCOMO II" (Boehm et al).
>
> Dan Ligett, Softstar Systems, (603) 672-0987

Thanks for the response, Dan.  COCOMO used to be an important tool for
me.  I found that most of the time it gave me a wrong answer it was
because I missed something or was naively optimistic in setting the
factor values.  I've been away from it for several years and I'm
trying to get a feel for where it stands WRT to the state-of-the-art.
I see lots written on using requirements but I'm left cold by the
absence of the consideration of the many cost drivers built into
COCOMO.  We'll check out your web site and see where it takes us.

BTW, any idea why the USC web site isn't being updated.  It looks like
the last update was in 2000.  Where is Dr. Boehm and what is he doing
these days.  I had a copy of his first book.  Unfortunately I loaned
it to someone and never got it back.  It looked pretty ragged with all
my notes and observations.  I sure wish I had it back.  Live and
learn.

Terry

"TerryG" <ritpg@hotmail.com> wrote in message
news:7322e1ab.0404101941.e4dd7ab@posting.google.com... 
> BTW, any idea why the USC web site isn't being updated.  It looks like
> the last update was in 2000.  Where is Dr. Boehm and what is he doing
> these days.  I had a copy of his first book.  Unfortunately I loaned

Terry,

Dr. Boehm is active on many fronts... I can't keep up with all of his work,
but here are a few clues.

Their website is pretty rich, you may have been looking in a quieter corner.
Here's a good starting point:
http://sunset.usc.edu/cse

Dr. Boehm has a flock of graduate students and research associates at work
on a wide variety of problems.  For a sampling, look here:
http://sunset.usc.edu/cse/pub/research

In addition to the COCOMO II book, Dr. Boehm recently published "Balancing
Agility and Discipline: A guide for the Perplexed" (with Richard Turner).
He has at least one other book in the works.

This site lists some upcoming events (including COCOMO Meetings scheduled
through 2007!!):
http://sunset.usc.edu/cse/pub/event

If you're interested in the proceedings from recent COCOMO meetings, they
are online here:
http://sunset.usc.edu/cse/pub/event/past.html

Dr. Boehm also has a thriving affiliates program -- Softstar Systems is
proud to be one of the sponsors of the work his team is doing at USC:
http://sunset.usc.edu/cse/pub/affiliate/general.html

Terry, I'm not a biographer, but I offer these URLs as a partial answer to
you question.  Enjoy.

Dan

On 9 Apr 2004 18:57:12 -0700, ritpg@hotmail.com (TerryG) wrote:
>"We note that the predictive performance of the Bayesian approach
>(i.e. COCOMO II.1998) is significantly better than that of the
>multiple regression approach (i.e. COCOMO II.1997). COCOMO II.1998
>gives predictions that are within 30% of the actuals 71% of the time
>where as COCOMO II.1997 gives predictions within 30% of the actuals
>only 52% of the time."
>
>Not being the statistician I'd like to be, I'm not sure how to
>interpret the above.  It does sound pretty bad to my uninformed ears.
>30% of actual???  But I'm wondering what someone with more expertise
>might say about this.  If there is anyone out there who would like to
>comment, I'm all ears.

Well, that sounds like a rounding-off of a normal curve, one sigma is
like 33% (compare to 71%), right, so he's saying that if Cocomo
predicts a cost of $1,000,000, odds are one sigma (67%, actually) that
the real cost will be between $700k and $1.3m, and two sigma (95%)
that the real cost will be between $400k and $1.6m, and three sigma
(99%) within $100k and $1.9m.  I daresay the higher ranges occur more
than the lower ones!

I'm not a statistician either, but if I haven't completely garbled
things, that's a simple starting point, adjust accordingly for 71%
instead of 67%, skewed ranges, etc.

Hey, that's not bad, as such things go, since it doesn't account for
requirements changes that sneak in in the meantime.  One big
organization I worked for was happy with their SWAG estimates being
within 100%.  Me, I'm happy if the project gets done at all, and the
estimates and variances are only for tuning the tongue-whipping the
project manager gives and gets every w, and possibly computing
bonuses.

J.

Thanks again, Dan, for the great response.  I don't know if we will
end up going with COCOMO but your enlightenment should at least get us
to consider it.  Frankly, I'm very surprised to hear such poor
accuracy probabilities from COCOMO. A long time ago we used it to give
us leverage in what we knew would be a protracted firm fixed price,
earned value contract negotiation.  We already had a good idea from
our own metrics what the effort was going to be.  But in those days
home grown metrics didn't carry much weight with NAVSEA or anyone
else.  We set up a spreadsheet implementing Boehm's first book and
when we plugged in what we thought were honest parameter values, we
came out very close to the estimate we already had.  When we sat down
for negotiation, we used the spreadsheet and Barry Boehm's reputation
heavily to defend our pricing.  The strategy worked very well.  BTW,
what started out as a $4.5M project ended up with a price of ~$12M.
We did very well with our earned value tracking.  I'd say we did a
couple of things right.  We used and religious maintained Timeline
schedules (that should date me) which were constantly shared with the
customer. And we managed requirements about as well as you can.  It
also helped to have a customer who readily accepted the role of
project team mate rather than adversary.  I don't remember how many
P000s (contract mods) we had but it was quite a few.  That was by far
the most dynamic project I've ever been on.  The requirements changed
enormously over the life of the contract.  Every changed requirement
was documented, packaged, prioritized and negotiated.  And because the
customer could see that our estimates were honest and accurate, we
never had any problems negotiating cost/schedule changes.  It was by
far the most successful project I've ever been on.  Ironically, it was
canceled for convenience.  One of the other contractors over-ran their
budget so badly, the customer had to cancel the entire program.  Even
more ironically, we ended up making more money overall than we would
have had the project completed normally.  Have you ever read the FARS
(federal acquisition regs.) governing compensation of companies with
whom the gov't cancels a contract for convenience.  It's amazing what
you're entititled to.  We effectively got close to what we would have
gotten from the contract while our people were freed up to take on
other work.  We had completed all of our builds, our sell-off test
procs. were written and verified and we had completed integration
testing.  We knew the software was solid as did the customer.
Everything ended up being archived and never used.  That is my only
regret about the project.  Many people had worked very hard and never
got to see the final product in use.  Oh well.

BTW, I'm no expert on the latest theory on project estimation but it
seems to me that one of the keys to successful estimation tools is
that they count whatever they count consistently across projects.  The
downside of using a SLOC based tool is that there are so many
languages these days.  When I was managing projects, we tended to us
only one (Navy standard) language.  It was easy for us to develop a
utility that would count our SLOCs at the end of each earned value
period.  We even, over time, calibrated it to weight SLOC
modifications and deletions.  It worked pretty well.  I see from the
USC web site mention of similar utilities that will deal with the
modern languages.  So maybe the problem has been dealt with.

The good news is that computer language, by its syntactical nature, is
well suited for automatic (therefore, more likely objective)
quantification.  What I haven't figured out yet is what those who
advocate the use of requirements (vice SLOCS) for effort
quantification or progress propose as a consistent means to count
requirements completed.  I know the world has progressed in the area
of documenting and tracing requirments.  But it seems to me that a
requirement just doesn't possess the consistent quantifiability that a
SLOC does.  What seems to be a single requirement to one person may be
two or more requirements to another. Any thoughts?

Of course, neither SLOCS nor requirements seems to me to be a useful
measure of progress when it comes to tasking like formal test proc.
writing but I could be wrong.  Maybe that's where requirements may
actually have an advantage.  The message to me is that SLOCS and
requirements may be reasonable bases for overall project estimation.
But neither satisfies the progress quantification requirements of
earned value reporting for all types of project tasking.  What are
your thoughts on this?

I must say I'm enjoying this dialogue.  There was a time when I really
enjoyed the theory of cost estimation and setting up related
processes.  Unfortunately, I ran into one of those career forks that
took me in a different direction.  And at this point, I guess I am
what I am.  But I can still enjoy this exchange.  Thanks.

"Dan Ligett" <ligett@softstarsystems.com> wrote in message news:<c5bl0g$s12$1@pyrite.mv.net
>...
> "TerryG" <ritpg@hotmail.com> wrote in message
> news:7322e1ab.0404101941.e4dd7ab@posting.google.com... 
>
> Terry,
>
> Dr. Boehm is active on many fronts... I can't keep up with all of his work
,
> but here are a few clues.
>
> Their website is pretty rich, you may have been looking in a quieter corne
r.
> Here's a good starting point:
>     http://sunset.usc.edu/cse
>
> Dr. Boehm has a flock of graduate students and research associates at work
> on a wide variety of problems.  For a sampling, look here:
>     http://sunset.usc.edu/cse/pub/research
>
> In addition to the COCOMO II book, Dr. Boehm recently published "Balancing
> Agility and Discipline: A guide for the Perplexed" (with Richard Turner).
> He has at least one other book in the works.
>
> This site lists some upcoming events (including COCOMO Meetings scheduled
> through 2007!!):
>     http://sunset.usc.edu/cse/pub/event
>
> If you're interested in the proceedings from recent COCOMO meetings, they
> are online here:
>     http://sunset.usc.edu/cse/pub/event/past.html
>
> Dr. Boehm also has a thriving affiliates program -- Softstar Systems is
> proud to be one of the sponsors of the work his team is doing at USC:
>     http://sunset.usc.edu/cse/pub/affiliate/general.html
>
> Terry, I'm not a biographer, but I offer these URLs as a partial answer to
> you question.  Enjoy.
>
> Dan