Software Testing - testing and SOX

Author

testing and SOX

JohnSteele

2008-01-16, 7:24 pm

I'm involved with a large multi-year project to implement the next
generation financial system. Our SOX auditors are telling me that they
will need "evidence" that we actually executed test cases. Recording
that a test passed will not do. Since we expect 10s of thousands of
test cases per run, this could be a big problem. Have any of you done
this before? What have your auditors taken as evidence of execution?

Phlip

2008-01-16, 10:27 pm

JohnSteele wrote:

> I'm involved with a large multi-year project to implement the next
> generation financial system. Our SOX auditors are telling me that they
> will need "evidence" that we actually executed test cases. Recording
> that a test passed will not do. Since we expect 10s of thousands of
> test cases per run, this could be a big problem. Have any of you done
> this before? What have your auditors taken as evidence of execution?

Sounds to me you should ask the SOX auditors this question!

--
Phlip

Vladimir Trushkin

2008-01-17, 7:27 pm

On Jan 16, 10:55=A0pm, JohnSteele <john_ste...@steelelogic.com> wrote:
> I'm involved with a large multi-year project to implement the next
> generation financial system. Our SOX auditors are telling me that they
> will need "evidence" that we actually executed test cases. Recording
> that a test passed will not do. Since we expect 10s of thousands of
> test cases per run, this could be a big problem. Have any of you done
> this before? What have your auditors taken as evidence of execution?

A good test management system can help you with that. It may record
the fact that every step in a test has been actually executed (when
and by whom). Another way is having a set of logs generated by your
application, or coverage analysis report. Some data generated by your
system may also play a role of validation means. For example, if your
system produces files or database records the number and the content
of those will be the justification you are looking for. Finally, you
may try to record thousands hours of movies about how testers executed
test cases ;)

I wonder, why do they need it? Is it common with audits to demand such
kind of thing?

----
Best Wishes,
Vladimir

H. S. Lahman

2008-01-17, 7:27 pm

Responding to JohnSteele...

> I'm involved with a large multi-year project to implement the next
> generation financial system. Our SOX auditors are telling me that they
> will need "evidence" that we actually executed test cases. Recording
> that a test passed will not do. Since we expect 10s of thousands of
> test cases per run, this could be a big problem. Have any of you done
> this before? What have your auditors taken as evidence of execution?

I'm with Trushkin. If there are thousands of tests per run you will need
some kind of automation. That means output results from the testing to
be examined after the run. I would expect that an entry in an output log
file for each test executed would be sufficient.

But you will need that internally anyway just to keep track of what is
actually in your test suite. IOW, it will come "for free" with a good
test process.

--
There is nothing wrong with me that could
not be cured by a capful of Drano.

H. S. Lahman
hsl@pathfindermda.com
Pathfinder Solutions
http://www.pathfindermda.com
blog: http://pathfinderpeople.blogs.com/hslahman
"Model-Based Translation: The Next Step in Agile Development". Email
info@pathfindermda.com for your copy.
Pathfinder is hiring:
http://www.pathfindermda.com/about_us/careers_pos3.php.
(888)OOA-PATH

Michael Bolton

2008-01-18, 8:24 am

On Jan 16, 3:55 pm, JohnSteele <john_ste...@steelelogic.com> wrote:
> I'm involved with a large multi-year project to implement the next
> generation financial system. Our SOX auditors are telling me that they
> will need "evidence" that we actually executed test cases. Recording
> that a test passed will not do. Since we expect 10s of thousands of
> test cases per run, this could be a big problem. Have any of you done
> this before? What have your auditors taken as evidence of execution?

To YOUR auditors, it is unlikely to matter what OUR auditors accepted.

"Bring me a rock." (Later...) "No, not that rock. Bring me a
different rock."

---Michael B.

developer

2008-02-28, 10:32 pm

Vladimir Trushkin wrote:
> On Jan 16, 10:55 pm, JohnSteele <john_ste...@steelelogic.com> wrote:
>
>
>
> A good test management system can help you with that. It may record
> the fact that every step in a test has been actually executed (when
> and by whom). Another way is having a set of logs generated by your
> application, or coverage analysis report. Some data generated by your
> system may also play a role of validation means. For example, if your
> system produces files or database records the number and the content
> of those will be the justification you are looking for. Finally, you
> may try to record thousands hours of movies about how testers executed
> test cases ;)
>
> I wonder, why do they need it? Is it common with audits to demand such
> kind of thing?
>

Actually, all you need is a rigorous QA dept. One that will record
their results as they go and test such things as boundary conditions etc...

The job of QA is to prove your software doesn't work. If they fail in
that task, then you have done well in yours.

SOX is

- it won't do anything to prevent the kinds of problems our
politicians told us it would address - weren't the Enron execs still
sentenced to hard time despite SOX not being law when they screwed me
(and many of you) over?

It would be a joke if it weren't for the very real consequences of SOX.
Hey, let's pay some overpriced morons to come in and tell us what to
do even though they have ZERO experience with SOX because it is brand new.

And let's implement a lot of CRAP that will give some politician
somewhere a couple more votes because his constituents are morons.

It was supposed to be about accountability and we already had that. Now
it's more about deniability than anything else.

The lower echelon of a company can go through the motions all day long,
clicking this and that in some crappy web-based application, but does it
mean anything?

No, it does not.

In fact, SOX probably has done more to make corporate malfeasance easier
than if it were never passed.

Sponsored Links