Home > Archive > Extreme Programming > October 2006 > Metrics - How to measure sucess
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Metrics - How to measure sucess
|
|
| dave.johnston83@gmail.com 2006-10-19, 4:01 am |
| Hi - this question maybe better directed at a software dev metrics
fourm but;
My team currently uses a traditional iterative approach to development,
although we are now starting to work on smaller more distributed bits
of work I think that a agile development method would be a good way to
go.
Now generally at the end of project we will review what went well and
what didn't - although the outcome is usually we delivered the software
within the time frame and it worked so it is a success.
What I want to know is has anyone on here used metrics in the past to
measure a successful project using traditional approaches and then
measured another project which was successful using an agile process to
determine the benfits of using agile vs traditional iterative dev in a
given dev environment.
| |
| Paul Hodgetts 2006-10-19, 6:57 pm |
| Dave Johnston wrote:
>What I want to know is has anyone on here used metrics in the past to
>measure a successful project using traditional approaches and then
>measured another project which was successful using an agile process to
>determine the benfits of using agile vs traditional iterative dev in a
>given dev environment.
I was a co-author of a paper that compared a non-agile to and agile
project (using XP primarily) and used a number of metrics to try and
compare them. The paper was eventually published as a chapter in the
book Extreme Programming Perspectives.
The good side is that the second project was a re-implementation of
the first system, within the same company and with a lot of the same
team members. So it's closer to an apples-to-apples comparison. The
down side is that it's still a different context, so there could
certainly be factors other than the agile practices in play. Also,
the second project got cancelled (dot-com crash time), so we had to
use some interpolations of how things would have played out.
In any case, regardless of the results, we used these metrics: elapsed
time to develop a similar feature set, developer months of work to
develop each, defects found, code size (LOC, number of methods), code
metrics (lines per method, cyclomatic complexity).
If I had to measure an agile project nowadays, with another 4 years of
experience, I'd add some tings like: time to deliver a story
(feature), degree and depth of feedback and how it was used, ability
to release at each iteration, some measure of story value delivered
per cost.
The biggest metric, IMHO, is the measure of value delivered per time
and per cost, which gets slippery because of how we measure value, but
we can use something like story (point) delivery per time and cost.
The other one is the ability to punch out a release at any point the
customer says "I want to use it now."
The paper can be found in Agile Logic's resources section at:
http://www.agilelogic.com/resources.html .
Paul
-----
Paul Hodgetts -- CEO, Coach, Trainer, Consultant
Agile Logic -- www.agilelogic.com
Training, Coaching, Consulting -- Agile Processes/Scrum/Lean/XP
Complete solutions for adopting agile processes, Scrum and XP.
| |
| Andy Champ 2006-10-19, 6:57 pm |
| Paul Hodgetts wrote:
>
> The paper can be found in Agile Logic's resources section at:
> http://www.agilelogic.com/resources.html .
>
Paul,
that's one of the best references I;ve seen so far! I'll read it fully
in the office tomorrow. I'm a little cautious though because [quoting
selectively!]
"The V2 project cost a total of 207 developer-months of effort ...
Overall, using the estimated schedule, the V3 project would have cost a
total of 40 developer-months."
Good so far but...
"We note that the V3 team was staffed by senior developers, and their
expertise probably contributed to the productivity gains."
Thanks
Andy
| |
|
| Andy Champ wrote:
> Paul Hodgetts wrote:
[color=darkred]
> Paul,
> that's one of the best references I;ve seen so far! I'll read it fully in
> the office tomorrow. I'm a little cautious though because [quoting
> selectively!]
>
> "The V2 project cost a total of 207 developer-months of effort ...
> Overall, using the estimated schedule, the V3 project would have cost a
> total of 40 developer-months."
>
> Good so far but...
>
> "We note that the V3 team was staffed by senior developers, and their
> expertise probably contributed to the productivity gains."
Uh, they contributed the entire order of magnitude of gain?
I'd like to meet such senior developers!
Maybe they wrote the paper...
--
Phlip
[url]http://www.greencheese.us/Z Land[/url] <-- NOT a blog!!!
| |
| gnoll110 2006-10-20, 9:57 pm |
| Paul Hodgetts wrote:
> The biggest metric, IMHO, is the measure of value delivered per time
> and per cost, which gets slippery because of how we measure value, but
> we can use something like story (point) delivery per time and cost.
> The other one is the ability to punch out a release at any point the
> customer says "I want to use it now."
>
> The paper can be found in Agile Logic's resources section at:
> http://www.agilelogic.com/resources.html .
>
> Paul
> -----
> Paul Hodgetts -- CEO, Coach, Trainer, Consultant
> Agile Logic -- www.agilelogic.com
> Training, Coaching, Consulting -- Agile Processes/Scrum/Lean/XP
> Complete solutions for adopting agile processes, Scrum and XP.
Arrr, delivered value. One can argue that true delivered value is the
only metric.
Do you 'map' the delivered value to increased revenues into the future?
Hows are people tracking this delivery?
Justification by accountant! ;)
Gnoll110
| |
|
| gnoll110 wrote:
> Do you 'map' the delivered value to increased revenues into the future?
> Hows are people tracking this delivery?
Ask the end users if your features are making them more productive, hence
bringing them profit.
Repeat in cycles of a w, not a year.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| John Roth 2006-10-20, 9:57 pm |
|
Phlip wrote:
> Andy Champ wrote:
>
>
>
> Uh, they contributed the entire order of magnitude of gain?
>
> I'd like to meet such senior developers!
>
> Maybe they wrote the paper...
It was also redevelopment of an existing project, with
the people who had worked on the original project.
I'd love to see something substantive (in the 50,000
loc or above category) that was done with two different
developer teams, both of which had come up to speed
on the process they used, and neither of which had
prior experiance on the specific application (but were
both familiar with the domain.)
Failing that, this kind of experiance is at least
useful, if not precisely ideal.
John Roth
>
> --
> Phlip
> [url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
| |
| Isaac Gouy 2006-10-20, 9:57 pm |
| John Roth wrote:
> Phlip wrote:
>
> It was also redevelopment of an existing project, with
> the people who had worked on the original project.
I've been wondering if anyone would mention that ;-)
[color=darkred]
> I'd love to see something substantive (in the 50,000
> loc or above category) that was done with two different
> developer teams, both of which had come up to speed
> on the process they used, and neither of which had
> prior experiance on the specific application (but were
> both familiar with the domain.)
>
> Failing that, this kind of experiance is at least
> useful, if not precisely ideal.
>
> John Roth
>
>
>
| |
| Paul Hodgetts 2006-10-21, 3:58 am |
| Andy Champ wrote:
>Paul Hodgetts wrote:
>
>Paul,
>that's one of the best references I;ve seen so far! I'll read it fully
>in the office tomorrow. I'm a little cautious though because [quoting
>selectively!]
Cautious about what? I'd always recommend caution and never assume a
case study means anything for your project. Even if this comparison
was conducted under the best of research conditions, there's never any
guarantee that the results would translate into your (or anyone
else's) context. One thing I've learn from six years of agile
coaching is that while some patterns of similarity emerge, no two
teams are the same.
We agonized over whether to publish this because it is so anecdotal
and there a lot of variables that were not controlled. But in the end
we figured any data, especially back in 2002, was better than nothing,
so we'd state some caveats and let folks do what they will with it. We
got pretty well beat up about it by some folks back then, but it also
generated some good discussions and further work by others. Barry
Boehm actually cites it in his Balancing Agility and Discipline book
(not that it means anything in particular).
I was primarily pointing out the paper in response to the OP's request
for suggestions of metrics to monitor, not really for the results of
the comparison.
>"The V2 project cost a total of 207 developer-months of effort ...
>Overall, using the estimated schedule, the V3 project would have cost a
>total of 40 developer-months."
>
>Good so far but...
>
>"We note that the V3 team was staffed by senior developers, and their
>expertise probably contributed to the productivity gains."
If I recall correctly, the first project had about a 6:1 junior:senior
ratio, while the second had about a 4:5 ratio. So, really that line
should be more like "...the V3 project was staffed by /significantly
more/ senior developers...". We realized after the first project that
the (former) CTO's approach of low-cost junior labor was actually more
expensive than using more senior people.
To answer another point brought up, the first project had a Microsoft
front end with ASP and VB code, talking via XML to a Java mid-tier
implemented with full-on EJBs and an Oracle back end. The second was
all Java, with a JSP front end, no EJBs (a custom framework similar to
something like Spring and Hibernate) and the same Oracle back end.
So, even though the two systems had a large overlap in functionality,
there was a lot of difference in technology, so it wasn't just a
re-implement. The second system added an entirely new web services
interface that the first did not have, as well as implementing a much
more sophisticated work flow.
Thanks for the interest,
Paul
| |
| gnoll110 2006-10-21, 3:58 am |
| Phlip wrote:
> gnoll110 wrote:
>
>
> Ask the end users if your features are making them more productive, hence
> bringing them profit.
>
> Repeat in cycles of a w, not a year.
>
> --
> Phlip
> [url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
How are they measuring the future benefit (rev. increase, rev
maintained or cost reduced)?
I've mainly worked in places were this was 'paper' money. No one really
checked what happened! I was interested in how 'best practise' places
that really do measure these benefit (profits) do it!
Did this $1M development project really generate/save what was said, in
the initial 3 years? That kinda stuff.
I'ld like to hear 'war stories', with the names remove to protect the
guilty ;)
Worked in non-agile places where the system owner wanted changes ASAP,
then it would sit in User Acceptance Testing for sometimes months
before they started their (the system owner) testing.
Gnoll110
| |
|
| [Tip: Please trim your quotes]
gnoll110 wrote:
[color=darkred]
> How are they measuring the future benefit (rev. increase, rev
> maintained or cost reduced)?
They don't; they measure how the program helps them, right now.
No prognostication; no forecasts. If a project can't achieve any margin of
productivity very soon, it's probably too high risk to bother continuing to
try!
> I've mainly worked in places were this was 'paper' money. No one really
> checked what happened! I was interested in how 'best practise' places
> that really do measure these benefit (profits) do it!
>
> Did this $1M development project really generate/save what was said, in
> the initial 3 years? That kinda stuff.
I didn't read the article. I would ask, after the first month or two:
- how much revenue did the team get selling the program
- how much more productivity did the users _feel_
I don't want to think about job-cost accounting, to rate their actual
productivity boosts. But sometimes this information is readily available; if
the users are sales representatives, for example.
In other situations (such as life-critical software), you indeed delay
releases. However, because such software must use a mockup of its target
situation, to stress-test it, the team could deploy to this situation very
frequently, and treat its measurements as productivity.
> Worked in non-agile places where the system owner wanted changes ASAP,
> then it would sit in User Acceptance Testing for sometimes months
> before they started their (the system owner) testing.
Curiously, if such changes are more rapid, they also tend to become
convergent, as minor feature requests. Not divergent, as major changes in
direction.
--
Phlip
[url]http://www.greencheese.us/ZLand[/url] <-- NOT a blog!!!
|
|
|
|
|