Software Engineering - The Cone of Uncertainty

Author

The Cone of Uncertainty

Gilligan

2004-12-21, 3:59 pm

Is the cone of uncertainty a myth? Barry Boehm presented the cone of
uncertianty for software estimation and it roughly states that
uncertainty decreases over the life cycle of development and estimates
become more accurate. (Is that an accurate summary?)

Do we just accept the cone of uncertainty to be true because it makes
sense or is it real?

I like to bust myths in computer science. One of the recent myths
busted is that programmers can not test their own software.

What about the cone of uncertainty? I think it is a myth as well (read,
trying to provoke conversation).

gooch

2004-12-22, 4:03 pm

Gilligan wrote:
> Is the cone of uncertainty a myth? Barry Boehm presented the cone of
> uncertianty for software estimation and it roughly states that
> uncertainty decreases over the life cycle of development and
estimates
> become more accurate. (Is that an accurate summary?)
>
> Do we just accept the cone of uncertainty to be true because it makes
> sense or is it real?
>
> I like to bust myths in computer science. One of the recent myths
> busted is that programmers can not test their own software.

While I would agree that a programmer "can" test their own code and I
would even say they can do as good of a job as someone else, In my
experience it is almost always better to have someone else do the
testing. The proble is that the coder is too close to the code and has
too many preconceived notions about the way it operates. The person who
coded the project must take care not to leave out test cases because
they know how the system acts in a given situation. This can be
overcome by writing a carefully thought out test plan and taking your
time with the development of test cases but in my opinion the person
without the preconceived notions about the software will have an easier
time and often be able to perform the test quicker and detect subtle
bugs more easily. As I said above this does not mean that it can't be
done and done well but you have to be aware of the risks and make sure
they are mitigated.
>
> What about the cone of uncertainty? I think it is a myth as well
(read,
> trying to provoke conversation).

In my experience this is the case. I think there are at least two
explanations for this phenomenon. One, as the project progresses the
scope and size of remaining work generally decreases making estimation
much simpler. Two, as the project progresses yopu obtain a wealth of
historical data on which to base you estimations, this will inevitably
lead to a better estimate. The second point is shown well by the
estimation abilities of a good CCM4 or CMMI4 organization. As a matter
of fact this is one of the driving forces behind the metrics collection
performed by these organizations. Historical metrics can be used as a
jumping off point in developing an estimate for a new project,
especially if you have historical data on a project with similar scope.
This is why a good organization can improve their estimates over time
as a metric data on a greater number and wider range of projects
becomes available in their database.

Notice that I specified good organization. There are plenty of CMM(I)3,
CMM(I)4, or CMM()5 organizations that do not truly practice what they
have been assesed at. Often an organization will get assesed at a given
level simply to allow for bidding on certain types of contracts and
then a lot of the good practices developed during the assesment run up
drop by the wayside. Most organizations that stick with a well defined
process and metrics collection for a lengthy period of time will begin
to recognize the benefits and continue to use the process.

Robert Klemme

2004-12-22, 4:03 pm

"gooch" <gooch001@comcast.net> schrieb im Newsbeitrag
news:1103724300.447831.168800@z14g2000cwz.googlegroups.com...

> In my experience this is the case. I think there are at least two
> explanations for this phenomenon. One, as the project progresses the
> scope and size of remaining work generally decreases making estimation
> much simpler. Two, as the project progresses yopu obtain a wealth of
> historical data on which to base you estimations, this will inevitably
> lead to a better estimate.

I'd add #3: with a project progressing you usually get a much clearer
understanding of the problem(s) at hand - especially of their complexity.
This in turn allows for more granular definition of tasks to do and thus
better estimates.

Kind regards

robert

Gilligan

2004-12-22, 4:03 pm

"The proble is that the coder is too close to the code and has
too many preconceived notions about the way it operates. The person who
coded the project must take care not to leave out test cases because
they know how the system acts in a given situation."

The other side of the problem is that a tester is to far from the code
and may not have any notion about the way it operates. Also, the tester
is a downstream consumer of the programmer and the code. If testing
starts before development is finished then the version the tester is
learning may be obsolete by the next milestone. If that is true then
the time spent learning the code, talking to development to learn the
code, and writing the tests are wasted. If testing begins at the end of
development then what is the developer doing? He could be the tester.
If the developer moves on to another task is he going to be able to
give full attention to issues the tester discovers?

"Historical metrics can be used as a
jumping off point in developing an estimate for a new project,
especially if you have historical data on a project with similar
scope."

I agree with your statement.

What is your feel if the above isn't the case, that is, you have lots
of historical data but the new project's scope is completely different?

I think the cone of uncertainty has an "inverse", the cone of
experience. The cone of experience starts at the point and gets wider
as you go.

I think our ability to estimate the unknown does not improve. If it is
unknown, it is unknown. Until we approach the uknown we do not gain
experience. When we finally get to the "end" there is nothing unknown
anymore. But our ability to predict the attributes of the unknown did
not improve. Maybe our courage did. Maybe our confidence to explore
did. But our prediction abilities did not.

Gilligan

2004-12-22, 4:03 pm

Yep. Experience is the key. The quicker you get some experience, the
better.

Gilligan

2004-12-22, 4:03 pm

Yep. Experience is the key. The quicker you get some experience, the
better.

gooch

2004-12-22, 4:03 pm

"The other side of the problem is that a tester is to far from the code
and may not have any notion about the way it operates. Also, the tester
is a downstream consumer of the programmer and the code. If testing
starts before development is finished then the version the tester is
learning may be obsolete by the next milestone. If that is true then
the time spent learning the code, talking to development to learn the
code, and writing the tests are wasted. If testing begins at the end of
development then what is the developer doing? He could be the tester.
If the developer moves on to another task is he going to be able to
give full attention to issues the tester discovers?"

This is all true but at least in my experience someone other than the
coder is more likely to locate subtle bugs.

"What is your feel if the above isn't the case, that is, you have lots
of historical data but the new project's scope is completely
different?"

Agreed. If you have no experience with something it is very hard to
predict the effort required. It is also true, however, that over time a
good engineering team can pool their experiences on projects to develop
reasonable up front estimates even if the previous projects are not
similar to the new project. I think the key here is to have the entire
engineering team involved. I have seen many times where a program
manager or similar person compiles an estimate based on nothing more
than a guess and these are invariably inaccurate. If representatives of
the entire engineering team are involved from the beginning the results
are much more likely to reflect the real world and not simply some
idealized view of it.

Gilligan

2004-12-22, 9:13 pm

"If representatives of
the entire engineering team are involved from the beginning the results
are much more likely to reflect the real world and not simply some
idealized view of it."

Agreed!

I think that during the meeting with everyone a little "exploration"
into the unknown is ocurring!

Thanks for the converstation.

I think the cone of uncertainty is not a myth but its definition maybe
confusing.
The number of uncertain items decrease over the time of working towards
a solution. Once the item is not as uncertain the new estimate will be
closer to the correct value. So, I feel the cone of uncertainty is not
a myth.

Oh, back to the testing thing. A second pair of eyes is always good. I
just hate to see the developer dump his code over the fence, move on to
the next task, and have an unfamiliar tester and a junior programmer
try to finish the product. You know how some programmers never have to
finish, they get to do the fun 80% part and always delegate the rest.

Thanks again.

Geoff (Gilligan)
http://home.att.net/~geoffrey.slink...MaverickQA.html

Alan Gauld

2004-12-23, 9:03 am

On 22 Dec 2004 06:05:00 -0800, "gooch" <gooch001@comcast.net>
wrote:[color=darkred]
> estimates

Yes and no its not a myth. There is a large body of empirical
data which shows that estimation accuracy improves as the project
matures. (Estimates are most always accurate on the day the
software is delivered! :-)

For more on this whole topic you shoud read the excellent book
"Facts & Fallacies of Software Engineering" by Robert Glass. He
covers many of these types of topics with references to the
research data.
[color=darkred]

This is maybe a bit misleading. Programmers can, and always have,
tested their own code. And at the level of testing code they
usually do a reasonable job in my experience. What programmers
are lousy at testing is how the *system* or *application* behaves
in the hands of a user. Users do the most unreasonable things to
applications and no programmer I've ever met would dream of doing
some of those things - myself included!

So end-user testing is always needed. But the end-user tester is
not testing the code. They are testing the functionality as set
out in the requirements - whatever they may look like - and the
actual usability of the end product for the task it was intended
for. Now having user representatives in the development team, as
per JAD style methods, is a good start but that will only be a
single users view, every user will have different opinions and
slightly different ways of working. You need to cater for those
variances. For example I've never (so far) seen a programmer
write a tst that says - "While the hour glass shows on screen
randomly tap the keyboard and play with the mouse" - yet thats
what real users do!

So I'd say that yes, programmers can test code, but no, they
can't test systems. All IMHO of course! :-)

Alan G.
Author of the Learn to Program website
http://www.freenetpages.co.uk/hp/alan.gauld

Gilligan

2004-12-23, 9:03 pm

Good points. Thanks.
The more that come in contact with the system the better!

Sponsored Links